Lewies Blog

Sometimes, I surpass my own geneouse.  Something got done today that was an amazing solution to a difficult problem regarding cyclic redundancy.  The side effect of this solution has sereouse potential to solve some very aged and hard problems that have been left hidden away in a corner.

All of this has to do with the implementation of Http Modules within ASP.Net.  With these special modules, you can catch the process of a web page before the web server processes it in its own normal way.  The power of them is so strong once they are understood that you can make all kinds of tweaks that were never possible before.

So today, I solved an immediate problem, impressed the manager on how I solved it, and collaborated on how we could use the method to solve some past feature requests.  When I left for home, I already had a working solution up and running on my local machine in the office.  What a day.

I’ve been tasked to learn ColdFusion this weekend. If you see any web pages on the internet where the name ends with CFM, those are ColdFusion web pages. I don’t have any software, and a server wasn’t provided for me to work with. I was a bit confused on how I was supposed to learn the stuff. We have a new guy comming in on Tuesday and I need to know ColdFusion by then so I can help him get up to speed.

I did some searching on the internet. I found a product called BlueDragon that can run web pages with ColdFusion tags in them (ColdFusion Markup Language - CFML). It can run on its own on port 8080 by default, and you can also map it to Microsoft’s Internet Information Server (IIS). The best part of the deal is that this server is free.

I found a few websites with ColdFusion tutorials. I also found that Macromedia has a website that I could use as a reference for everything CFML. I always looked down at ColdFusion as if it were almost a joke. When HomeSite started supporting ColdFusion tags, I was busy learning Active Server Pages (ASP). Thank goodness for that, as ASP has caught on greatly compared to ColdFusion.

It seems that ColdFusion has finally caught up with me. The good news is that I already know so many programming, scripting, and markup languages. This is making it easier to catch onto CFML.

I created an account for myself over at SourceForge. This is a website that many people use to provide open-source software on. Many services are available such as defect tracking, forums, tasks, screenshots, and even some source-control using CVS.

The whole reason I’m looking into SourceForge is that I want to go public with my Automagic Podcaster software soon that I use with Dreamy Audio. I’m looking into a couple of things before I do it. Besides places to host the files and collaborate with others, I also need to look into licensing.

Although I wish to distribute the code as free open-source software, I also need to protect it. I want to allow people to modify it, but I don’t want them to package it up and resell it. I also want credit where credit is due. I also don’t want people to redistribute there modified version without stating what they have modified from the version that they received.

There are many licenses out there. I just have to find the rite one that matches my intent.

I’ve started to have a look at the tags in other MP3 sources other then the files that I make myself for DreamyAudio. The purpose is to test the compatability of my software with MP3 files. It’s like throwing a monkey wrench into a running motor. I started off with DarkCompass and found a few bugs that I had fixed quickly.

During this time, I stumbled upon four ID3v2.3+ tags that I hadn’t seen before (PCST, TDES, TGID) in the DarkCompass podcast. I haven’t found any documentation on these tags when googling them, but from the content within, they appear to be related specifically to podcasts. I came upon similar tags (TID, TDS, WFD, TDR) in another podcast that was using ID3v2.2 tags. Thankfully, the 2.2 tags were documented on many sites on the internet.

One thing that grabbed my attention was the content withing the tags of an old podcast that I grabbed off of Odeo. It actually had the URL of my feed in it. I started thinking a bit about this and realized that if I redistributed the MP3 file to friends and such, that eventually it could be traced back to me. It seems that Odeo had dynamically threw in its own ID3 tags that identify my account.

Perhaps in the future, ID3 tags (or any embedded metadata) could become a privacy concern. Imagine if more sensitive data was stored within those tags such as email addresses, phone numbers, or any sort of contact information without your knowledge and you let your friends get a copy and listen to it. Further down the line after friends sending friends sending their friends the MP3 file, someone gets a hold of it who knows how to read all of that data. Now a stranger has your personal data and could sell it or make use of it in malicouse ways.

Ok, so I’m really going over the edge here and blowing a little thing up to infinite proportions. Who cares if the alien Captian Zorgon from the planet TH-87w discovers our planet and designates you as the offender of the careless audio. What if he makes use of a tool specifically crafted for him to carry out his will of removing any source of that aweful sound and targets you just because your email address was in a small tag in the file.

Back to reality. It’s just a little thing really. I’m just a bit unsettled that someone had inserted some information with my account name as part of a url feed into the ID3 tags without my knowing.  Still, maybe it isn’t Odeo that did it.  Maybe iTunes inserted this data in there behind my back when it downloaded the MP3 file from the feed.  If that is the case, iTunes is the culprit for exposing that extra information.

I’ve been having a go at improving the accuracy and format of my DreamyAudio podcast. The accuracy being the part where I read information embedded within the MP3 files.

I decided to challenge myself and display the playtime for a mp3 file encoded with a variable bitrate (VBR). It was tough just trying to detect that the file had a VBR. It’s actually simple now that I figured it out.  You need to detect if an Xing header is present in the audio data a few bytes after the MPEG header.  The tricky part is figuring out what position those bytes are at in the file.
After figuring out if the file has a VBR, I noticed that the estimated playtime is out of wack. It’s based on a constant bitrate (CBR). So now I’m plagued with a problem.

I assume that I could go ahead and read through the individual audio frames and tally the bitrates from each one and divide it by the number of frames found. The thing is, there are thousands of frames. Perhaps I can find a way to read the first few hundred and give an estimate based on the number of bytes remaining.
Bah. No time for it today.

I was looking into a way to cache my generated pages on the server with PHP. I couldn’t find anything to do it on the server-side, but I did find something to prevent a client from asking for the page via client-side caching.

When a web browser requests a page for the first time, it will save a copy into its local cache. If the person visits the web page a second time, the browser will send the date and time when the page in the local cache was originally retrieved. The server checks the date and responds with either a new page, or a message stating that the page hasn’t been modified since the last time the request was made.

This helps out servers to decrease how much bandwidth is being sent accross the internet. Some of us are charged extra if we go over so many gigabytes per month. It also helps my server from doing unnecessary processing of scripts.

I was working out a way to take care of this. PHP has a method to access the client headers individually called getallheaders(). The problem is that this only works on apache web servers. I’m testing things here at home on an Internet Information Server (iis). Although my website is hosted on apache (I think), I still would rather have support for multiple platforms.

I continued to run through some problems until I found a PHP implimation of conditional get. It was pretty easy to impliment and it worked immediately. The method expects a parameter of a date to compare. I read the date that an MP3 file was last modified and passed it in there. The method did the rest of the work telling the web browser that the information had not changed.

I didn’t realize that this term, “Conditional Get”, was coined for this pattern. The PHP implementation referred to a very good article about it called HTTP Conditional Get for RSS Hackers. It goes into detail about the whole process of it all. It’s amazing how much detail you can get into that amounts to just a few lines of code.

Tomorrow I’ll look more into caching data on the server. I found a method (md5_file()) to get a hash of a file that I can use to see if the files contents have changed since the last time I created cached xml documents and images referring to its contents. Reading all of that data from MP3 files constantly could perhaps have a toll on the servers performance that I want to avoid.

Since my interest in recording my dreams has been renewed, I’ve started fixing up the code to my audio blog. I mainly call it “Automagic Podcaster” due to the fact that all the end-user has to do is upload MP3 files onto there site. The code reads all the meta data about the MP3 files that have ID3 tags embedded within them.

Some of the MP3 tagging software that I’ve been using lack features such as Unicode support and chapters. I tried doing translations of my dreams in Japanese, Arabic, Chinease, and Portuguise and saving them as lyrics. Most other languages that I translated using google had problems with some characters. I soon fixed that by using html entities to represent the special characters. I sent an email off to the author of a particular piece of software. I’m hoping that maybe he is already working on somethng similar for the next version of his software.

Anyhow, I’m thinking that soon I may just package everything up, clean the code a little, and look into posting it as open source software. I don’t know what all is involved. I’m using a flash applet to read and play XSPF feeds on my pages to play the mp3 files, so I need to contact the author to see if I can distribute it along with my code and what additional requirements there may be.

So this past weekend I learned a thing or two about new technologies (ORM) for bridging the gap between programming data and database data. I figured out how to solve my problems I had yesterday with inheritance using subclasses within Hibernate.

Today, I spoke with my manager and we discussed that the ORM and MVC technologies that I was looking into was only a first generation approach. We went into a brief history of the second and third generation of this technology. The third generation tools that are available actually do most of the work for you. You visually setup your business objects, and the program creates/updates your database, sets up your data tier, and models your business objects for you. I was just a bit stunned.

My managers argument in support of this technology was that most of my time is spent going through the same process on the database and the business logic. Granted that I did make a ton of little utilities to help me out along the way, there are programs out there that walk through the whole process.

There were some tools that we talked about and I started looking into one called DeKlareit. It took a little getting used to, but by the end of the day, I got most of the main concepts of how it worked and the intelligence behind it. Each time I understood another feature, I was just impressed. Most of it is just visual. I don’t write much code at all except for a few formulas or rules.

It is amazing that my manager is suggesting these kind of tools. I feel like it is taking me out of the classical erra of programming, bypassing the industrial age, and going strait to the modern age (if I was comparing it to Civalization IV, of course). I can not say that I no longer need to worry about the database at all, but I feel that this is a huge step in increasing my productivity. The price of this tool can’t even compete with the number of hours I would be doing the same thing by hand.

Since Friday, I’ve been researching a bit here and there about Object-Relational Mapping (ORM) and Model-View-Controller (MVC) at work and at home. These are technologies that help with manageable programming.

ORM is used to help store objects within a relational database. A database stores data differently then an object. This technique is used to map data between objects and databases. A popular and open-source tool to assist in this mapping for programming in .Net is nHibernate. nHibernate is a port of Hibernate written in JAVA. I have figured out the basics of saving and retrieving objects to a relational database. Though I’m starting to have difficulty with polymorphism in inheritance. I’ll keep up the R&D until I get it rite. So far, the technologies involved are showing a lot of promise. On top of that, I have found many websites offering supporting documentation and techniques that apply to both platforms on JAVA and .Net.

Model-View-Controller (MVC) can work with or without ORM. ORM would only be used with the Model portion of MVC. MVC breaks your application (desktop or web applications) into three parts. Model represents your business elements, or data schemata. “View” represents the final web page, also known as your “presentation layer”. Last we have “Controller” wich is the business logic between your presentation and your data. Business logic can manipulate data, authorize that you do in fact have permission to access that data, and just about everything else. It can be thought of as the API undernieth the presentation itself. Some people claim that it is the glue between the data and the presentation.

I’ve worked with MVC often, but not to its full effect. The rules with MVC pretty much go something like this. If any one part of MVC changes (The model, view, or controller), then the other two parts should not break. That is the part where I am not successful. I usually develop 3 projects initially in my applications. The web or desktop application, a project for business logic, and another project for the data tier.

I believe most of my problems arrise from the stability of the model. As long as the model doesn’t change, the control and view should be able to change more easily without any impact. I’ve originally kept a virtual model in place and manually mapped it to an older model that was practically inherited from another project. This worked great in the first phase as there were very simple requirements and tight deadline.

In this second phase, requirements change every day and are still being gathered. After much research into ORM and MVC, I am wanting to just get a definition of the domain that I can rearchitect the model after. Currently, I am having to rearchitect the older model many times due to its current incompatabilities with the requirements. This still raises a problem though for future architecture. How can a database be architected properly to be able to grow not in data scalability, but in schemata scalability? I understand normalization as such, but the current model already has normalization and needs to be reworked often when changes to requirements occur.

I’ve been making a few objects here and there in SecondLife. Most of them have scripts in them. Over time, I have updated my items with bug fixes and feature additions. I could just drop items on everyones profile, but as more people start buying, this becomes a hassel.

I purchased an auto-upgrade machine that my customers could approach with a license key. If the license key was validated, and a more recent version of the product was available, they would be given the newest version. This is nice, but it requires my customers to guess when a new update is available.

I started looking around and found an item called SL Courier. It allows me to compose a list of peoples names and identification keys along with a list of products to send them. Once I release a new version, I just put the product in the couriers inventory and tell it to deliver to the list of past customers.

This solution seems to be the best for keeping my customers up to date with the latest products. The problem is that I don’t have everyones identification keys. Through my in-world object sales and selling through the web on sites such as SL Exchange, SL Boutique, and SecondServer, I am only able to get a list of names of the people who purchased my products.

At first, I created a customer registration product that the customers could manually rez and say “register”. After about 40 seconds, it would confirm there registration after sending me there name and identification key. It was a little more convienient, but I was finding that a lot of people were not going through the process and that it just took too long.

I decided to make a different attempt. I changed the actual package that my products are delivered in. A customer normally has to rez the package and open it to remove there products. I’ve started working on a little project where I can make that box talk to an in-world server and send me the name of the customer, there identification key, and the name of the package that they are opening.

I went to the internet and found a website that could generate bar codes online. There were a ton to choose from. I liked the look of MaxiCode by UPS, but I found a better resolution of it at another website. I also looked at different types of shipping lables such as those for the most often used UPS, and USPS. I put everything together to make my own version of a shipping label.

When the box is rezzed in the world for the first time, the label gathers the necessary information and encodes a message. The label turns yellow when it sends the registration information out. I spent a good deal of time working within the limitations of LSL script (MD5, XOR, Base64) to make it semi-secure. I also setup the scripts to warn me if I didn’t set the permissions correctly for the next owner. If I give the package to someone, I want to prevent them from looking at the password and the servers id in the scripts. This way, I have a visual cue (the shipping label turns red) and some messages telling me what is wrong and what I have to do.

There is a lot of work into the communications with the in-world server. The label makes a few attempts to contact the server. Each time that it fails to receive a response, it turns cyan and waits a little longer before trying again. This is just in case the sim that the server is on goes down, or its email que is full. If the server can’t be contacted after about an hour, I stop all registration attempts and notify the customer that they should notify the seller of the problem. I also make the label turn purple as a visual cue as to a response problem.

If everything goes well, registration usually takes only a few seconds and the label turns white once it has completed. All the scripts in the label are deleted to prevent duplicate registrations.

The server itself is a beast. I am usually limited to 16kb of memory per script. When I worked on my sim land owner scanner in the past, I learned how to expand that memory and communicate with other scripts within the same prim. I’ve used that experience here and made a few scripts whose main purpose was to store and retrieve data - just like a disk drive. In fact, I called each script “Disk 1″, “Disk 2″, etc.

When a message comes in and is decoded and validated, it is sent to a disk that has the most free space available. Then every 24 hours, a log of all data is sent to my email account. Email messages are limited to about 4 kb that are sent out from SecondLife. It isn’t much, so as I compose the email, I constantly check how large the message is, and how much data I’m about to add to it. If it is too big, then I send out an email and start composing a new one.

Email was a big issue in the past mainly due to the script delay penalty enforced on it. When I used this mehtod in the past, my script would suffer a 20 second delay. A friend of mine in-world had pointed me in the rite direction to make another script to send the email for me. I send a message to that script and then let my original script continue on with its own workload. Instead of a confirmation after a minimum of 40 seconds, I get them in about 5 only because I set my timers to check emails every 5 seconds on both the server and the package.

I am almost on the edge of actually puting my system into production. I have a few tweaks to do before it is ready. Drive management needs the most attention rite now.

It is crazy with how much work you can put into a game. I hope to start profiting later on from it. The materials (prims, scripts, notecards) are unlimited in supply. Your mind of skills and creativity are all that limit you. I can build just about anything from scratch and people actually buy them. So far, I’ve made about $14 this month. I pay less then that in monthly account fees in order to be able to own land (otherwise accounts are free).

Since I own more then 512 square meters of land, I pay an additional monthly fee. My accounts land fees has me paying at $75 per month, and my wifes is as $40/month. Yes, you read rite. So hopefully that $14 starts gradually becomming a bigger number. If we both sold off all our land, except for the last 512 sq. m. each, we would be making a profit every month with just my product line.

Why is land important? Land gives you the ability to have prims. The more land you own, the more prims you can have. Many people who don’t have land, and those that do, go to sandboxes to make there own stuff without a strict prim limit. I consider these places to be laggy. I’ve gotten into problems with one place where I couldn’t get out of it. Half the time you meet up with people that are there to ruin everyones day, and you also run into the occasional experiments that have gone wrong. On top of that, I’m just as much of a hermit in SL (Second Life) as I am in RL (Real Life).

You can also rent land or buildings from other players. You can rez your objects or build new ones there, but there is usually a high priced monthly fee to help the owner pay for there own accounts land costs.

The other reason to own land is to have your own freedom of expression. A perminant place of residence. Someday I’ll write about what I’m doing there. Most of what I do is make new products to sell or assist in selling to other players. For now … I leave you with just the shipping confirmation project. I only need 2 prims for the project and could do that on just 16 sq. m. of land and still have room for one more prim.