The other side of the moon

Up front - if you haven't purchased this laptop yet, then stop now and look for a different brand. The Acer Extensa 4630Z has a great webcam, but will take you through hell when you try to install linux on it.

My tests were with Fedora Core 9, because that's all that I had with me. I may have had better luck with FC10 or Ubuntu, but I do not have those CDs with me, and I have a really slow network connection, so downloading it is not an option.

If you've already bought this laptop, then this post will probably help you out with finding some of the drivers.

First off, the keyboard has some extra keys for the Euro and dollar. These are just above the cursor keys, but I haven't yet figured out the keyboard layout to use them, so forget about that.

The trackpad is standard, but you cannot tap on it - at least the default driver on Fedora won't let you. I tried adding a section to my xorg.conf for the touchpad, but that only slowed it down a lot.

Now the problems.

The laptop has two switches - one for the bluetooth antenna and one for the wifi antenna. The bluetooth switch directly controls the bluetooth antenna, but the wifi switch is just a simple key that triggers an event and it is up to the wifi driver to deal with that. It took me a while to figure this out, since the driver for the wifi card isn't installed by default on FC9. I spent a lot of time playing with acerhk to enable the card, but I guess that isn't needed. If you do use acerhk though, the series id is 3000.

The wifi card has the Atheros AR5B91 chipset. This is chipset isn't listed on the Atheros website or the various linux driver pages, don't be confused by that. The device ID shown by lspci -nn ends with 002a.

This card works with the ath5k driver, and on FC9, you'll have to get that from the linux wireless driver page. Just get the whole tarball and build it locally.

Before you can do that, you'll first need to yum install make, gcc, kernel-headers and kernel-devel.

Do not use the madwifi driver - it does not work with this card.

The ethernet card is a Broadcom card, and works well with the default driver, nothing much to worry about here.

However, I had problems getting dhclient to get a DHCP IP for both cards simultaneously. I think this is a quirk with Fedora, because I've done it successfully with Ubuntu on a Thinkpad. If you want to connect to wireless, and you already have dhclient running, you have to first disconnect from wired, and kill the dhclient process. I found this very stupid and frustrating.

The next thing that did not work was the microphone. There is a built in mic right next to the web cam. Your volume control sees this as Mic. There is another device called Front Mic, which is actually a mic that you plug in to the microphone jack in the front of the laptop.

Installing all the PulseAudio tools helps.

This does not help with Skype though, and the audio quality is very bad. I don't know if this is a property of the system, or with skype, but basically I was getting clicks, scratches and a lot of static with skype - even after setting the audio output devices in Skype to Pulse.

Anyway, the wireless problems did not end there. I managed to get the wifi card detected, and could connect to an unsecured network, but as soon as I tried to connect to a secure network, it failed. It failed with both WEP and WPA, and I have no idea how to debug or fix that.

I now have only one day left to get this box working 100%, so will try with Ubuntu tomorrow if I can get the CD from someone. Will update this post when I'm done.

Update: The Ubuntu story

After my trials with Fedora, I moved to Ubuntu. Got an 8.10 CD, but it appeared to be corrupt (tested on three different drives). Then got an 8.04 CD that worked.

The first problem was when starting the live CD. It correctly detects that it needs the intel display driver, but the driver itself is broken. I had the same problem with my Thinkpad - the screen is garbled once xorg starts up. I had to boot up in Save Graphics Mode (Press F4 at the grub menu), which uses the vesa driver.

The vesa driver only goes up to 1024x768, which looks weird on a widescreen, but it lets you get things started.

I went through the install procedure, which worked without problem. Followed the same instructions as above for getting the ath5k driver, except that this time it set up the ath9k driver. NetworkManager still wouldn't connect to a WPA protected wireless network, but after reading up online, I decided to try wicd, and it worked out of the box.

I then downloaded Skype and tried it out, but had very bad luck. The microphone wouldn't work, and skype crashed when I tried to use the webcam. This was much worse than Fedora. I tried the whole pulseaudio setup, but that didn't help.

I then upgraded the kernel to the latest, but still had no luck.

Finally I started looking for an updated video driver. I found melchiorre's weblog where he has a deb for the intel driver.

I first tried to get this driver using apt hoping for a newer release, but there wasn't anything, so I downloaded the deb from the blog, and installed it.

This worked in the broad sense. I could start up in graphical mode and use 1280x800 resolution - which is good for a 14" wide screen. The only problem is that the mouse pointer kept getting garbled every few seconds. It would reset to normal if I moused over anything that changed the pointer, but within a second or two it was garbled again.

This made it very confusing to determine where the pointer was (it was just a large square with a series of black dashes all over the place), so I switched back to vesa and decided to tolerate 1024x768.

The system was now in a state where it was usable as an internet browsing box, but Skype not working was a big problem, because everyone that my dad needs to communicate with is on Skype (which meant that Ekiga wasn't an option).

The Windows story

Just for completeness, I should mention the situation with Windows XP:

Windows does not detect the wireless card or the display card, so it runs in vesa mode at 1024x768, and there's no network. The only advantage with windows is that anyone in the world can fix it, so my absence won't be the bottleneck when this box breaks (and it will).

The final story

So, in the end, I've decided to leave Windows on half the drive - this will be the part that all service engineers can 'fix' when things break.

The rest of the drive I've split between Fedora 9 and Ubuntu 8.04 - I wish I could have more recent versions, but I'm not that lucky. I'll set Fedora as the default system, and get the wireless network working there, and use OpenOffice from the Ubuntu partition. At some point I might switch the default to Ubuntu once I figure out what works better with this hardware.

Update: 8.10

I finally managed to upgrade the box to Ubuntu 8.10. I had to do a remote upgrade, since I'm now half way around the world from the box. The upgrade is pretty easy. Just follow the upgrade instructions listed on the ubuntu site. You need to go through the section titled "Network upgrade for ubuntu servers".

The upgrade took almost two days to complete because the network was really slow, and it required me to hit "y" a few times, so it could not be automated (the do-release-upgrade script does not accept a -y flag), but other than that, it was really smooth.

Note: If the upgrade terminates partway, you need to resume it within a day. The first time it failed, I resumed after three days, and it restarted from scratch. The second time it failed, I resumed after 6 hours, and it continued from where it had left off.

The only problem with doing a remote upgrade is that I could not easily test a few things. For starters, I did not know if gnome was working correctly, however, this can be tested remotely. ssh to the box, and look at /var/log/Xorg.0.log. Look for errors (anything starting with (EE)). When I checked this, I found errors saying that there were no usable screen configurations.

I also found that /etc/X11/xorg.conf had been replaced by the upgrade. Restoring it from the original did not help, so I replaced the vesa driver with the intel driver (the one that Fedora worked with), and tried again:

   sudo /etc/init.d/gdm restart

This time it worked, and I confirmed with my dad on the other end that the track pad was working as expected.

The other things I could not test were the webcam, microphone and speakers, but I got my dad to test all three using Skype, and we had a successful VoIP chat - no crashes.

This system is now in a fully working condition.

Thanks for all the comments so far, they definitely helped me push on.

One of the cool things about working at Yahoo! is that you get to see and play with a lot of little toys before the rest of the world does. YUI was one such tool.

I started playing with YUI while it was still in version 1, and its API was much different from what it looks like now. Among the toys I'd made was a bookmarklet to add drag and drop to any website. I used it on our internal sites until YUI was made publicly available.

A couple of years ago, I published instructions for adding drag and drop to any website, but that still needed a little technical know-how on the user's part.

So, to remedy all that, and to bring us up to date with YUI 2.6.0, I've rewritten the bookmarklet, and hosted it on yui.bluesmoon.info (no, there's nothing else there). It's far simpler, than the earlier version, and without further ado, here it is: Make Draggable.

Simply drag that link to your bookmarks toolbar, and you're ready to use it.

Now, if you click on the bookmark (in your bookmarks toolbar) when you visit a website, most sections of the page should become draggable. Enjoy yourself rearranging your favourite pages.

Let me know if it doesn't work for you, and let me know of additional features that you'd like to see. Yes, remembering your past state would be cool, but is probably not something I want to do right now.

Here's a simple snippet of PHP code:

        setlocale(LC_TIME, 'de_CH.UTF-8');
        echo strftime("%x");

I run this on RHEL5 and Ubuntu 8.04 and I get different results:

RHEL5:

       2008-10-07

Ubuntu 8.04:

       07.10.2008

So I look through /usr/share/i18n/locales/de_CH for a hint, and I find it.

On RHEL, d_fmt in LC_TIME maps to <u0025><u0059><u002d><u0025><u006d><u002d><u0025><u0064>, which in English, is %Y-%m-%d, while on Ubuntu, it maps to <u0025><u0064><u002e><u0025><u006d><u002e><u0025><u0059>, which in English is, %d.%m.%Y, which is exactly where this discrepancy arises from.

Now I have no idea how to verify which is more recent, because Ubuntu and RHEL do not package things in the same way. Any ideas?

Since my last post on javascript dates, I've been doing a lot more reading and testing. This post is a summary of what I've found.

I read up a few RFCs and ISO specifications. In particular, RFC2822: Internet Message Format, and RFC3339: Date and Time on the Internet: Timestamps, which summarises ISO8601: An internet standard for date and time (Wikipedia link). Below is a summary of what these specifications state.

RFC2822 lists two date formats, in sections 3.3 and 4.3. 3.3 is current while 4.3 is obsolete. RFC3339 lists a third date format based on ISO8601. All these specifications are for date-time representations for use on the Internet.

RFC3339:

2008-09-23T17:39:44-07:00

RFC2822:

Tue, 23 Sep 2008 17:39:44 -0700

(obsolete) Tue, 23 Sep 2008 17:39:44 PDT (might also include 2 digit year)

Both specifications state that the time zone code SHOULD NOT be used, and only the offset should be used. The colon used in the time zone offset is optional in ISO8601, but not in RFC3339. The time zone may be replaced with Z, indicating Zulu time (UTC+0000).

Additionally, RFC2822 lists 10 codes in the obsolete: UT, GMT, EST, EDT, CST, CDT, MST, MDT, PST, PDT. Military time zones (A-Z) are also included in the obsolete list. While other time zone codes have been used by various implementations, they are not defined by a common specification, and SHOULD NOT be used.

My tests with the Date.toString() method have shown the following:

Internet Explorer returns the time zone code for the eight US time zones listed in section 4.3. of RFC2822, and the offset for all other time zones using the UTC+/-HHMM format. The offset is NOT included for the eight US time zones specified above. This is in line with RFC822, but not 2822, which obsoletes 822.

Examples:

PST:

Thu Sep 18 16:47:36 PDT 2008

IST:

Fri Sep 19 10:11:38 UTC+0530 2008

Firefox returns the offset for all time zones, and a time zone string in parentheses for some time zones. The string is different on different platforms:

Windows:

full timezone name, ie, Pacific Standard Time

Linux & Mac OS X:

timezone code, ie, PST

examples:

Windows/Taiwan

Tue Sep 23 2008 10:14:10 GMT+0800

Windows/India

Tue Sep 23 2008 09:42:49 GMT+0530 (India Standard Time)

Mac OS X/US West Coast

Tue Sep 23 2008 17:57:04 GMT-0700 (PDT)

Linux/India

Tue Sep 23 2008 09:42:49 GMT+0530 (IST)

Safari matches the results for Firefox on Mac OS X.

Both Safari and Firefox match the specification by including the offset. The time zone code that they return is considered auxiliary information for display purpose only.

Opera consistently uses the UTC+/-HHMM format on all platforms, and never includes a time zone code. This is probably the strictest adherence to the specification.

So, what does all this mean? Well, to me, it means that I shouldn't spend any more time searching for a browser independent way to pull out the time zone code. It also means that strftime does not provide an easy way to print out an RFC3339 date with the colon in the time zone offset, but we'll figure something out for that :)

Finally, a bit of trivia...

Standard time in the Netherlands was exactly 19 minutes and 32.13 seconds ahead of UTC by law from 1909-05-01 through 1937-06-30. This time zone cannot be represented exactly using the HH:MM format.

Short URL: http://tr.im/dateformats

Ever since I built my javascript strftime implementation, I've been playing around with dates and date formatting in javascript. One of the things that users of the library have been happy about is the ease with which dates can be localised using the library, and it's this particular feature that's been frustrating me a lot when dealing with time zones. Let me get this out of my head.

The javascript engine of various browsers stringify dates in subtly different ways, but different enough to mess up time zone parsing.

Here's what I'm talking about. Tell me what your browser says when you click the following two links:

• Date.toString()
• timezone

Try this snippet of code in your favourite browsers:

var d = new Date();
alert(d.toString());
alert(d.toLocaleString());

I've seen different results on Firefox, IE, Opera and Safari. Firefox is the worst offender with different results across platforms. The following are my results:

Firefox 2 & 3 on Linux and Mac OS X, and Safari on Mac OS X

Thu Sep 18 2008 19:20:23 GMT-0700 (PDT)

Firefox 2 & 3 on Windows

Thu Sep 18 2008 19:20:23 GMT-0700 (Pacific Daylight Time)

Google Chrome

Thu Sep 18 2008 19:20:23 GMT-0700 (Pacific Daylight Time) However it displays GST instead of GMT if your timezone is set to GMT

Opera 9.52 on Linux and Mac OS X

Thu Sep 18 2008 19:20:23 GMT-0700

IE 6, 7, 8 on Windows

Thu Sep 18 19:20:23 PDT 2008

I've only tested this in the PDT time zone, but the results are diverse enough to make parsing out the time zone annoying (except for Opera, where it's impossible). This is what I came up with:

d.toString().replace(/^.*:\d\d( GMT[+-]\d+)? \(?([A-Za-z ]+)\)?\d*$/, '$2').replace(/[a-z ]/g, '');

I derived this regex over a couple of iterations. It started out taking care of the Firefox on Mac & Linux and Safari case:

d.toString().replace(/^.* \(([^)]+)\)$/, '$1');

Next, I added support for Firefox on Windows by getting rid of all lowercase letters and spaces:

d.toString().replace(/^.* \(([A-Za-z ]+)\)$/, '$1').replace(/[a-z ]/g, '');

Finally, I added in support for IE, by checking for a string of characters that fell between seconds and year:

d.toString().replace(/^.*:\d\d( GMT[+-]\d+)? \(?([A-Za-z ]+)\)?\d*$/, '$2').replace(/[a-z ]/g, '');

This is by no means fool proof. It could break with time zones and browsers that I haven't encountered yet, so I'm asking everyone for help in figuring this out.

Short of asking all browser authors to be consistent in their Date.toString() implementations, how do I figure out the timezone string for a date? I've even thought about building a table of offsets to timezone strings, but there isn't a one to one mapping for that.

I've also noticed that the timezone is always displayed in English, even when I change my system locale. Perhaps this is different for you. Let me know.

And yo, Google, what's up with Greenwich Standard Time?

Short URL: http://tr.im/jsinconsistentdate

I write a lot of code in sed whenever I need to do some kind of filtering, and I realised that there are several patterns that emerge. Sed is a Stream EDitor, and its capabilities are somewhat limited, yet it does provide for some of the more important things required in a programming language. It has sequence, selection, iteration, variables and debugging statements. In this post, I'll go over each of these.

1. Sequence

I've started with sequence because it's always the easiest to explain. Unless branching is involved, a sed script flows from top to bottom. All statements are executed in sequence, and that's pretty much all I have to say about it. Let's move on.

2. Selection

Selection is where things start happening. There are a few ways to execute a statement based on a condition. That condition almost always deals with a pattern in the current input line, but we'll see later how that can be changed. For now, here's how you do selection:

   /pattern/ command

  s/pattern/replace/
  t label

  s/pattern/replace/
  T label

The first is a simple "execute this command if the current pattern space matches /pattern/". That's akin to saying if(line.match(/pattern/)) { command; } in more common programming languages. Command could even be a block of commands enclosed in braces like this:

   /pattern/ {
    command1
    command2
    command3
  }

Let's take a few examples. We'll assume that sed is called without arguments, so each line is printed once by default.

1. If the line starts with "hello", add "world" after it:

      /^hello/ s/^hello/hello world/

2. If the current line number is 3, print out the line twice:

      3 p

   Since each line is printed once by default, the p prints it a second time.
3. If the line starts with "next", swap it with the next line and print both out:

      /^next\>/ {
        N
        s/\(.*\)\n\(.*\)/\2\n\1/
     }

The second and third type of selection are similar, and basically say branch to a label if the previous replace command succeeded (t) or failed (T). These make more sense when looking at iteration, so that's what we'll do now.

3. Iteration

Things always get interesting when you iterate. You can execute the same set of statements over a group of data without knowing in advance what that data is. The b, t and T commands come in play here, along with labels defined with the : command, similar to other programming languages. We'll look at some common loop types from other languages:

1. While(condition) {...} (loop executed 0 or more times)

      :loopstart
      /condition/ {
         command1
         command2
         command3
         b loopstart
      }

   
   For example, while the input contains ==, append the contents of the file named equals.txt:

      :loopstart
      /==/ {
         s/==//
         r equals.txt
         b loopstart
      }

   
   We can also do this with the T command:

      :loopstart
      s/==//
      T loopend
         r equals.txt
         b loopstart
      :loopend

   Though it's a little more clumsy this way because you need two labels. The first method is the code pattern that I use for a while loop.
2. Do {...} While(condition) (loop executed 1 or more times)

      :loopstart
         command1
         command2
         command3
      /condition/ b loopstart

   This is almost the same as the first loop, except that the condition is tested at the end of the block of statements. Let's take the same example, but this time, we read in the file at least once:

      :loopstart
         r equals.txt
      /==/ {
         s/==//
         b loopstart
      }

   In this case, using the t command makes it less messy since we need to do the replacement anyway:

      :loopstart
         r equals.txt
         s/==//
      t loopstart

3. The third type of loop is a for loop, which is harder because you can't really do math in sed. Still, if one tries, one can figure out weird ways to count. In this case, we use the hold space:

   
      # We want to print the current line 10 times:
   
      # 1. Grab the current line into the hold space
      h
      # 2. Replace the pattern space with = based on what we want to count to
      c \
   ==========
      # 3. Print the line as long as there are = left:
      :loopstart
         s/^=//
         T loopend
         x
         p
         x
      b loopstart

   
   In this code, we need to constantly swap between the pattern space and the hold space, since all our operations are done in the pattern space. Which brings us to variables.

4. Variables

Well, make that variable, since sed has only one piece of memory that can hold something, and that's called the hold space. The good news though, is that it has no size limit - well, theoretically at least. This means that using your own delimiters, you could store anything in there. JSON anyone? I generally use the newline character as a delimiter, since that's unlikely to show up more than once in a single line of input, but you can use anything that you think is unique to your application. Here's one way to do it:

   # 1. Swap the hold and pattern space
   x
   # 2. Set the pattern space to the value of your variable using the s, c, i, a, g or G commands:
   s/$/\nfoo\n/
   G
   # 3. Swap the hold and pattern space again
   x

   # The hold space now contains:
   # prev value of hold space\n
   # foo\n
   # the input line\n

And you can use these values later using the s/// command:

   # 1. Append the current line to the hold space
   H
   # 2. Pull the hold space into the pattern space
   g
   # At this point the pattern space has the newline separated list of
   # variables followed by a newline and the current input line
   # We can use these variables if we know their position, and replace
   # them into the input line:

   # 3. Append the previous input line to the first word of the current input line:
   s/.*\n.*\n\(.*\)\n\([A-Za-z][A-Za-z]*\)\>/\2\1/

   # Now the input line has been modified, and the hold space remains the same

5. Debugging

Finally, we come to debugging, which is extremely useful when writing code in such a strange language. sed has two commands that make debugging possible, though I won't say easy. The = command prints out the current line number of the input file. Note that this is not the line number of the sed script, but of the input that the script reads. The l command prints out the current input line in a visually unambiguous way. It's up to you to scatter your code with these lines to figure out what's happening internally. You can always swap the pattern and hold spaces and use the l command to find out what's in your hold space.

Apart from the above, sed also has methods to read and write files. We've seen reading above with the r command, and similarly, writing is handled by the w command. There are also R and W commands, but you can read the manual to figure those out. I'll leave sed here.

Update: should have mentioned this a long time ago, but someone wrote tetris in sed

On Friday I upgraded Ubuntu to 8.04. There are a bunch of problems with it.

Right at the start I noticed that my display wasn't working correctly. It was very jerky - even just drawing windows on screen was slow. Scrolling was impossible.

It turns out that while they chose to install the fglrx driver for my ATI card, they also installed the Mesa OpenGL drivers. The two aren't compatible. You'd think that there would be some simple package conflicts with this, but somehow the upgrade got both in. I got rid of the Mesa OpenGL package and reinstalled the fglrx package and display worked.

The next thing I noticed was that Firebug wouldn't work anymore. Hardy comes with the brand new Firefox 3, except that the version they chose isn't the latest, but is an older buggy version. The default installed is Firefox 3.0b5, while the latest available from getfirefox.com is 3.0rc1. The weird numbering is because it's still pre-release, but there have been significant fixes between 3.0b5 and 3.0rc1 that, for starters, make firebug work. Since most people who'd use Firefox 3 at this point in time are web developers, they'd also use Firebug, so picking a version that doesn't work with firebug just doesn't make sense.

I downloaded the new Firefox from getfirefox.com and got Firebug 1.1 from getfirebug.com and things worked.

Oh, the older version of Firefox also crashes regularly.

The third problem was video. mplayer said that it couldn't find a compatible xvideo device. This was easy to fix. /etc/mplayer/mplayer.conf had vo set to xv. In other words, the default video output device was xvideo, which doesn't appear to be supported. Why they'd use an unsupported output device by default is beyond my understanding, but I changed it to gl and it worked.

The better default would have been to just not put anything there. mplayer's fairly good at detecting these things at runtime. It's a small startup cost to pay, but it's better than having no video at all, and people who really want it to load faster will figure out the config options anyway.

Next is seahorse - the daemon that caches GPG and SSH keys. It wasn't starting up correctly. Instead, I noticed that gpg-agent was running and seahorse was running as a subprocess of gpg-agent. Why is this a problem? It's a problem because seahorse caches keys so that you don't have to enter your passphrase every time. Without this, I'd have to type in my GPG passphrase for every email I send so that it could be signed. While this is good on a multi-user system where you want things to be secure, I have a single-user box, and this just gets in the way.

The problem was in /etc/X11/Xsession.d/. There's a file called 90gpg-agent which starts up gpg-agent and overrides seahorse. I deleted this file and restarted X, and seahorse started up correctly.

Finally, and this is the most annoying thing I've come across so far.

My network interfaces are not recognise unless there is an active network available on an interface at boot-up time. Let me explain this with an example.

At home, I have a wireless network, while at work, I have a wired network. Now, I boot up my machine at home, and it detects the wireless network, and connects to it (eth1), and it also brings up lo, however, since I'm not connected to a wired network, my wired ethernet interface - eth0 - suddenly becomes non-existent, and there's no way to bring it up. It's as if the kernel hasn't detected its presence.

The only way to get the wired interface detected is to plug in an ethernet cable, and reboot the machine. Of course, when I do this, I also disable the wireless interface, so now that no longer exists, even if I re-enable it after booting.

This happened rarely with the older version, but for the most part, all my network interfaces were visible when I booted up, whether they were connected or not. This is a definite step backwards.

I haven't looked for a fix for this yet, but will post here once I find one.


On the plus side though, suspend now works quite well, and resume is quite fast. Of course, there's no point suspending if I'm going to need a different network interface when I resume.


A few more annoyances:

- My weather applets suddenly started showing the temperature in Fahrenheit instead of Celsius, which is what I'd set it to
- Shift+Space doesn't insert a space any more - it does nothing.

I fixed the space problem by going to Keyboard Preferences, then choosing Layouts, and under my selected Layout, choose Layout Options. Under that, go to the section on Using space key to input non-breakable space character, and set it to Space key outputs usual space at any level.

A discussion at work got into whether comments are good or not. I have my own opinions about comments, and years of having to dig through other people's code has taught me not to trust them too much. Comments that I've put into my code in the past have generally been stories and anecdotes that I wanted to tell future readers of my code, and seldom about the code itself.

When I have to go through someone else's code, I start off by assuming that their comments were bad and delete all but a few of them. They may not actually be bad, but it saves me the time I'd spend separating bad comments from good ones, and I can always refer to the last version in CVS if I need to check on old comments. In general, I've seen the following types of comments:

1. Useless comments - they explain what a particular piece of code does
2. Obsolete comments - the code changed but the comments didn't
3. Mental notes - they explain why a particular piece of code does what it does
4. Reminders - FIXME/TODO comments

There are also API docs that look like comments, but that's only because the comment syntax in most languages is the only way to add documentation (perl has pod though). I'll ignore this type, but you still need to be careful of API docs that don't match the API or the implementation - they're probably both wrong.

The first type is largely useless. The code is either very obvious in what it does, in which case the comment is not needed, or it's not, in which case it should be rewritten. One of the rules from the Elements of Programming Style goes thus:

Don't comment bad code, rewrite it

It's #63 if you care, while #69 says:

Don't over-comment

The next type is worse than useless, it slows you down in your understanding of the code. Delete and take note of rule #61:

Make sure comments and code agree

The third type of comment is useful in understanding a block of code and the thought process that went into it, and is also useful in knowing how to rewrite/optimise that code. Some developers may prefer to push this kind of comment out to bugzilla and link to it in the code. Either way works. I generally leave these comments around or rewrite them depending on what I'm doing to the code they affect. Rule #62 says:

Don't just echo the code with comments - make every comment count

This is how you make it count.

Reminders, IMO, are better suited to bugzilla with the bug number noted in the code. This is so mainly because bugzilla sends out reminders while your code doesn't. If I can, I'll fix the relevant code, or delete the comment if no longer pertinent (these kinds of comments tend to hang around for several years without anyone noticing).

Oh, and just in case someone wanted to counter #2 by saying that sometimes you need to write obscure code to be efficient, or because it was a smart way to do something, I present to you rules #1, #2 and #5:

Write clearly - don't be too clever

Say what you mean, simply and directly

Write clearly - don't sacrifice clarity for "efficiency"

So, have fun commenting your code, have a dialogue with your readers, or think of it as a mini blog about the code, but make sure you make your comments useful or fun. If you're interested in the quick summary of the Elements of Programming Style, grab the fortune cookie file.

As a follow-up to my last post about Javascript date functions, I went ahead and implemented strftime for javascript.

Have a look at the demo at the link above, and download the code: strftime.js. It's distributed under a BSD license, so use it and give me feedback. Post comments here for now.

• Docs
• demo
• code
• PHP implementation
• Opengroup Specification

The code is documented using doxygen style comments.

You should also check out Stoyan's time input library that lets you do the equivalent of php's strtotime function.

Update: This code is now part of the YUI library and all further development will be done there.

I keep forgetting what I can do with dates in javascript, so this post is just a list of built in date functions. I've also seen a lot of sites that do some terrible things with javascript dates including trying to parse it out of a string, or using getYear(). Don't do that.

I've also seen sites that contain arrays of calendar month names and days of the week. This is locale specific, so do it only if you're willing to make the array easily localised.

To create a new date object:

var d = new Date;

var d = new Date('2008/04/22');

var d = new Date('2008/04/22 01:03:31');

var d = new Date(1208758100000);

Note: don't type that into firefox/firebug while editing a post on blogger - they have a global variable called d which is used to save your post.

Also don't forget the new keyword. Without that you'll just create a string containing the current date.

After this line, d evaluates to a string representation of the date which isn't very useful for anything since it's almost always not what you want. You really want to get various parts of the date out. There's a whole bunch of functions to do that:

d.getYear();         // deprecated function to get year.  don't use this
d.getFullYear();     // get four digit year.  use this
d.getMonth();        // get the month 0 - 11
d.getDate();         // get the day of the month 1 - 31
d.getDay();          // get the day of the week 0-6 (Sunday is 0)
d.getHours();        // get the hour - 0-23.  Note the s in the function name
d.getMinutes();      // get the minutes of the hour - 0-59
d.getSeconds();      // get the seconds of the minute - 0-59
d.getMilliseconds(); // get milliseconds past the second - 0-999
d.getTime();         // get number of milliseconds since the epoch.  integer divide by 1000 to get unix timestamp
d.getTimezoneOffset(); // get offset from GMT in minutes - note capitalisation

All of these functions return the value for your current timezone. To get values in UTC, add UTC after the get in the method name. There are also equivalent set methods, but I've never needed to use them.

One useful method is parse() which takes in a string representation of a date and returns the number of milliseconds since the epoch for that. You can call this directly on the Date class.

A useful page for javascript date functions is quackit.com

I mentioned earlier that you should not use getYear();. This is so because this method is not quite Y2K compliant. It is sort of compliant, but browsers don't implement it in the same way. The standard says that it should return the number of years since 1900, which means 2000 would be 100 and 2008 would be 108. Netscape/Firefox does that correctly (possibly because they defined the standard back in the day), but IE decided to 'fix' it by returning 4 digit years for anything past 1999, which means that 1999 is 99 while 2000 is 2000. Your code gets unwieldy trying to manage it, so it's best to just use getFullYear().

I'd always thought about writing an equivalent to strftime for javascript, but never got down to it. In my mind, it would be something that uses a regex with a callback function, and I'd use the unique property of javascript that allows one to call an object's methods like keys in an associative array. Basically something like this:

var fmt = {'Y': 'getFullYear', 'm': 'getMonth', 'd': 'getDate'};
Date.prototype.strftime = function(str)
{
var d = this;
var ret = str.replace(/%([Ymd])/g, function(m0, m1)
{
return d[fmt[m1]]();
});
d=null;
return ret;
};

I'll have to build on that sometime.

Update: I've built an implementation of strftime in javascript. This implementation is also part of the YUI library.

I was speaking with my dad on Skype yesterday when he told me that his mails weren't getting sent out, they were all stuck in his mail queue. Now before we go on, it's important to understand the outgoing mail set up.

My dad's machine has fetchmail fetching mail from various POP3 servers and sorting them into each user's mail spool, and it has sendmail to send mail using gmail's SMTP servers. This is perfectly okay, since my dad uses gmail.

Now if you go through gmail's configuration documentation, they say that you need SMTP over SSL on port 587 with TLS. They also say to use smtp.googlemail.com, but we'll ignore that, because we're not in the UK.

Setting this up requires a few simple steps.

1. First, create your sendmail certificate:

         cd /usr/share/ssl/certs
         make sendmail.pem
   

   Note that you can run make usage in this directory for help. Also this directory is created by the openssl package, so make sure you have that.
2. Next, define the SMART_HOST as esmtp:smtp.gmail.com and add certificate paths:

         define(`SMART_HOST', `esmtp:smtp.gmail.com')dnl
         define(`CERT_DIR', `/usr/share/ssl/certs')
         define(`confCACERT_PATH', `CERT_DIR')
         define(`confCACERT', `CERT_DIR/ca-bundle.crt')
         define(`confCRL', `CERT_DIR/sendmail.pem')
         define(`confSERVER_CERT', `CERT_DIR/sendmail.pem')
         define(`confSERVER_KEY', `CERT_DIR/sendmail.pem')
         define(`confCLIENT_CERT', `CERT_DIR/sendmail.pem')
         define(`confCLIENT_KEY', `CERT_DIR/sendmail.pem')
   

3. Now add PLAIN and LOGIN to confAUTH_MECHANISMS so that it looks like this:

         define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 PLAIN LOGIN')dnl
   

   This requires the Cyrus SASL library, which you probably already have
4. You also need to create an auth-info file that looks something like this:

         youraddress@gmail.com
         youraddress@gmail.com
         yourgmailpassword
         smtp.gmail.com

   Make sure this file is only readable by root.
5. Now regenerate everything and start sendmail. On RedHat based systems, this is as easy as running /etc/init.d/sendmail restart. On other systems you may have to run make first (or you may not have to use sendmail at all :)

If all went well, you should now be able to send mail using gmail's SMTP server.

Unfortunately, all doesn't go well. Mails don't go out, and in /var/log/maillog, you get errors saying "smtp.gmail.com: No route to host". You can try a traceroute and a ping, and they'll both succeed.

The problem is with that port 587 thing. For whatever reason, sendmail keeps trying port 25 even though 587 is the specified mail submit port, and sometime in the last few days or weeks, gmail stopped accepting mail submits on port 25 (at least that's what it looks like).

So, get back to your config file. This time I didn't know what options to use, but I know more or less what the sendmail.cf syntax means, and how to edit that file, so I edited it directly.

I went down to the line that starts with Mesmtp, and looked for the line below that which said TCP $h, which basically means connect using TCP to the host specified in $h. We need to add the port to this line. Change the line to TCP $h 587 and we're done.

Restart sendmail, and all works.

But this isn't a good solution, because the next time you regenerate sendmail.cf from sendmail.mc, your change will be overwritten.

So, what I did next, was go into /usr/share/sendmail-cf and run grep -r '^Mesmtp' *. The result which stood out was the mailer file. Inside that file, I saw that the TCP $h line was being added by the macro ESMTP_MAILER_ARGS, so, I just needed to add this one line after the SMART_HOST:

      define(`ESMTP_MAILER_ARGS', `TCP $h 587')dnl

and we're done. Restart sendmail, and the config changes are permanent. All works, and mails go out.

I'd been doing some stress testing of my mysql application today, and I was hitting some weird cases. Several transactions were deadlocking - this was expected - but the number of records that got inserted into my table was more than the number that I expected after subtracting errors.

My test was fairly simple:

1. Fork 15 processes
2. Insert and update 100 records in each process, running each INSERT/UPDATE pair inside one transaction
3. ROLLBACK on error

Either the INSERT or the UPDATE was expected to fail due to deadlock, and the whole transaction should have rolled back leaving no record in the table.

Before I go on, I should mention that I was using InnoDB, which does support transactions.

What I expected was that the total number of records in the table + the total number of INSERT/UPDATE aborts due to deadlock should be equal to 1500 (15*100). What I was actually seeing, was completely off. At one point I had 400 errors and 1299 records in the table - that's 199 extra records, and at another point I had 344 errors with 1497 records left over. On inspection of the records, I found that many of them had bad data. In most cases, it looked as though the UPDATE had never run, and in other cases, it looked as though the UPDATE ran on more than one record.

Now, my code looked something like this:

   mysql_query("START TRANSACTION", $db);

   $result = mysql_query("INSERT ...", $db);
   if(!$result)
   {
      log_error($db);
      mysql_query("ROLLBACK", $db);
      return false;
   }

   $result = mysql_query("UPDATE ...", $db);
   if(!$result)
   {
      log_error($db);
      mysql_query("ROLLBACK", $db);
      return false;
   }

   if(mysql_affected_rows($db) != 1)
   {
      log_error("bad update");
      mysql_query("ROLLBACK", $db);
      return false;
   }

   mysql_query("COMMIT", $db);
   return true;

So if anything failed, there should have been no record in the table, yet I was seeing records that did the INSERT, but not the UPDATE, or that did the UPDATE for more than one record. Control was getting in to all the if conditions - I was getting log messages for that, but the ROLLBACK was never taking effect.

Again, before anyone asks, the log_error() function I wrote uses debug_backtrace to log the filename and line number that it was called from.

I stared at the screen for a while wondering what the problem could be. The MySQL docs said a few things:

• InnoDB supports transactions
• AUTOCOMMIT is On by default
• Use START TRANSACTION or BEGIN to turn AUTOCOMMIT OFF until the next ROLLBACK OR COMMIT
• ROLLBACK rolls back everything until the last START TRANSACTION if a transaction is in effect, or the last statement (if AUTOCOMMIT is OFF)
• COMMIT commits the last statement or transaction depending on whether AUTOCOMMIT is OFF and a transaction is active or not

Nothing explained why ROLLBACK wasn't working here.

The only possibility I could think of, was that the START TRANSACTION itself failed, but when I checked the mysql docs, there was no mention of START TRANSACTION ever failing. Anyway, I decided to check, so I changed the start of the code to this:

   if(!mysql_query("START TRANSACTION", $db))
   {
      log_error($db);
      return false;
   }

And immediately I started seeing errors on this line.

The error I got was:

   Lost connection to MySQL server during query

Searching on Yahoo! got me this link on the MySQL reference manual. The second last point on that page answered my question immediately:

You can also encounter this error with applications that fork child processes, all of which try to use the same connection to the MySQL server. This can be avoided by using a separate connection for each child process.

That's exactly what I was doing. I was opening the db connection before forking, so each child inherited a copy of that connection, and as soon as one of them closed it, all the others would fail.

So, I changed my code so that the db connection was opened and closed inside each child process, and now my results are consistent. I still get deadlocks, but the sum of errors seen and records added does add up correctly to the expected total.

Moral of the story: It may be okay to ignore the return values of COMMIT and ROLLBACK, but always check the return value of START TRANSACTION.

Got back home a couple of days ago, and there was a new webcam sitting on my box. It was being used on windows, which I'm naturally allergic to. I decided to try and get it working on my ubuntu laptop.

Now I've set up webcams on linux before, and it was a pain, but it got done. This was over five years ago, so I'd assumed things may have gotten easier by now. They hadn't. New webcams were out, with just as bad support as there was five years ago. All the same, documentation on the web is good if you know where to find it. That's what this doc is for.

The first thing to do was find out what model my webcam was. I knew it was Creative, since that's what was written on the cam itself. Plugged it in, and dmesg just said that a new USB device had been plugged in. Then I ran lsusb, which gave me this:

   Bus 005 Device 001: ID 0000:0000  
   Bus 001 Device 013: ID 041e:405f Creative Technology, Ltd 
   Bus 001 Device 001: ID 0000:0000  
   Bus 002 Device 001: ID 0000:0000  
   Bus 004 Device 001: ID 0000:0000  
   Bus 003 Device 001: ID 0000:0000

So the vendor ID was 041e — which is Creative, and the product ID was 405f, which probably maps onto some name, but that's immaterial.

Most webcams (well, creative ones at least) work with the ov511 driver that comes packaged with most distros, but this cam didn't, so I started searching around for support for this particular product ID. Found out from RastaGeeks that this was a Creative Vista Webcam VF0330 (the VF0330 matched the model number at the back of the cam), and that it was supported by the ov51x-jpeg driver, version 1.5.2 or higher.

The Ubuntu Webcam page had instructions on setting this up, which I followed and met with success.

Now, chances are that you'll find other information as well about this driver. One of the steps I'd followed was to install the ov51x-jpeg-source and module-assistant packages using apt, and then build and install the module using module-assistant. Unfortunately, this installed an older version of ov51x-jpeg, which didn't work with the camera. That led me to believe that the driver didn't work, until I tried the newer version.

If you've done this, then you will need to apt-get remove ov51x-jpeg-modules-2.6.22-14-generic first, then install the new driver and then run depmod -A.