The other side of the moon

Someone on quora asked why people still use editors like emacs and vim when more modern alternatives exist.

There were so many great answers that I didn't need to answer the original question. I couldn't possibly add more to the question of why emacs or vim. Instead, I was reminded of an experience where even emacs/vim weren't options.

Sometime in the mid to late '90s, I visited my sister at university. She was in a biology lab, and they had a single 80386 PC with DOS and Windows 3.1. The computer wouldn't start Windows and they didn't know why and they asked me if I could do anything.

Since I love debugging obscure problems like this, I decided to take a look. It turned out to be a simple case of there not being enough available RAM to start Windows. The box did however have 4MB of RAM, which should have been more than enough to start Windows... except, this was a 386, and for compatibility with older 8 bit software, RAM was split into Conventional memory (640KB), System ROM (640K-1M) and Extended memory (everything above 1MB), and this box wasn't configured to use extended memory (if you remember HIMEM.SYS).

To make matters worse, AUTOEXEC.BAT loaded a bunch of programs at startup that used up a bunch of RAM, which meant that I couldn't even start the basic EDIT program to edit AUTOEXEC.BAT or CONFIG.SYS.

My only option at that point was to fall back to the absolute basics.

COPY CON AUTOEXEC.BAT
COPY CON CONFIG.SYS

The equivalent on unix would be cat. COPY CON on MS-DOS stands for COPY what's on the CONSOLE (the keyboard in this case) to the destination file, overwriting it if it exists.

(See What is copy con? for details)

And I had to be really careful with what I typed because typing in the wrong thing would mean the system might not start up, and I didn't have a boot disk on me (remember I was just visiting with no plan of actually fixing a computer), and if I did have a boot disk, none of this would have been necessary.

Anyway, I managed to build a very basic AUTOEXEC.BAT and CONFIG.SYS from memory (though I cannot remember now what I put into them), which allowed me to reboot the machine with enough RAM to start EDIT which allowed me to further edit the files to reboot with enough RAM to start Windows.

What I learnt from this is that no matter how good a system you may have access to, you need to be prepared to use the absolute minimum available tools. On DOS this was COPY CON. On unix over a slow or lossy network, you might actually have to edit a file by sending single lines of sed. In order to be prepared to do this, you need to do this a lot. It turns out that vim and emacs are really just one step above sed (well technically one step above ed which is half a step above sed) although they are extendable to have all the features of Eclipse or Visual Studio if you like, but even without those extensions, they are far more powerful.

Even while working with Eclipse, I'll find that there are times when I need to quit Eclipse, open my files in Vim, run a few commands to do things that would take me ages to do in Eclipse and then return to Eclipse. I need to use Eclipse because that's what our dev team has standardized on, and it makes it easier when screen sharing with other devs.

If you liked this post, there's a far more fun video of how the JPL team debugged and fixed an issue 15 billion miles away on Voyager 1.

Several discussions I've had with friends and colleagues recently reminded me of an incident we faced several years ago at Yahoo!

Now Yahoo! as a company was made up of many different local offices around the world, each responsible for content in their locale. Since there was a lot of user generated content, this meant users in a particular locale could easily enter content (blog posts, restaurant reviews, etc.) in their local language script.

Everyone was happy!

From about 2005 onwards, the company was looking to unify some of the platforms used around the world. For example, we had something like 4 or 5 different platforms to do ratings and reviews and it didn't make sense to have different architectures, database layouts, BCP setups, and a separate team managing each of these, so we started unifying. Building a common architecture was the easy part. I worked on several of these projects. Getting front end teams to migrate was also not terribly hard. Migrating content though, was tough because each region had content in their own locale and MySQL didn't let you set multiple character encodings on text columns.

So the i18n team started working with teams across Y! to move everything to utf8. The easy part was changing HTTP headers and <meta> tags. Content was a little harder, but doable with iconv(1) since in most cases we knew the source character encoding and the destination was always utf-8. In some cases we had to guess, but it generally worked...

...until at one point we also decided to do it for authentication.

One of the things that was localized was authentication, because it allowed users in, for example, South Korea, to use Hangul characters in their passwords. Usernames were always restricted to just alphanumeric characters and underscores (If I remember correctly).

Passwords are stored, as they should be, salted and hashed, so the character encoding of the database column was always us-ascii, which is compatible with utf-8, so no biggie..., except the input character encoding used by the browser was based on the HTTP headers or META tags of the page, and the transfer encoding was based on the enctype of the login FORM.

Prior to this move, these were all set to a character encoding that made sense locally, so Korea used EUC-KR and China used Big5, and so the hashed passwords used the byte sequences that resulted from treating the input as one of these encodings.

After the move, the user would still type in the same password, but when we converted them to bytes, we used utf-8, which resulted in a different byte sequence than the original encoding, so hashing this new sequence of bytes resulted in a different hash, and users could no longer log in. Well, only users that used non-ASCII characters in their passwords.

I forget what the actual fix was, but there were several options on the table. One was to revert the character encoding changes on the login page and to re-encode all passwords after a successful login. Another was to generate two hashes, one using utf-8 and another using the pre-migration character encoding for the region and to allow a success on either to go through.

When looking at the Core Web Vitals, we often try optimizing each independently of the others, but that's not how users experience the web. A user's web experience is made up of many metrics, and it's important to look at these metrics together for each experience. Real User Measurement (RUM) allows us to do that by collecting operational metrics in conjunction with user actions, the combination of which can tell us whether our pages actually meet the user's expectations.

In this experiment, I decided to look at each of the events in a page's loading cycle, and break that down by when the user tried interacting with the page. For those interactions, I looked at the Interaction to Next Paint, and the rate of rage clicking to get an idea of user experience and whether that experience may have been frustrating or not.

Before I jump into the charts, I should note an important caveat about the data. This analysis was done using RUM data collected by Akamai's mPulse product which collects data at or soon after page load. Not all page views resulted in an interaction before data was collected. Most of the analysis was restricted to page views where we had at least one interaction prior to data collection. We see on average, between 2-25% of beacons collected (across sites) had an interaction. Most sites had a recorded interaction on about 10% of beacons. I also separately looked at data collected during page unload/pagehide and while it captured more interactions, it did not have a noticeable effect on the results.

Each of the following charts is from a different website in mPulse's dataset.

Exploring the chart

Let's now look at the various features of this chart.

The chart shows multiple dimensions of data projected onto a 2D surface, so some parts of it will appear wonky. We'll walk through that in this section.

Event labels

The first thing we'll describe are the events. These are the vertical colored lines with labels to their right. These represent transition events in the page load cycle. The events we include are:

• First Paint (FP)
• First Contentful Paint (FCP)
• Largest Contentful Paint (LCP)
• Time to Visually Ready (TTVR)
• Time to Interactive (TTI)
• Page Load Time

You may have already noticed that in this particular chart, First Paint is _after_ First Contentful Paint, which is counter-intuitive. The reason we see this is that the number of data points with First Paint on them is different from those with First Contentful Paint. Safari and Firefox, for example, support FCP but not FP. When aggregating these points, the same percentile value when applied to two data sets will likely get you values from two different experiences. This effect is more prominent when the sample sizes are different. In general we would not expect the delta to be too far off, and in the data I've looked at, it hasn't been more than 50ms off.

The events to keep an eye on are the Largest Contentful Paint or Time to Visually Ready, the Time to Interactive, and the delta between them. LCP is not currently supported on Safari, so we use boomerang's cross-browser calculation of TTVR in those cases.

Time to Interactive is considered a lab measurement, but `boomerang` measures it in a cross-browser manner during RUM sessions, and passes that data back to mPulse. It is approximately the time when interactions with the page are expected to be smooth due to no more long animation frames and blocking time.

The next thing to note are that these events are positioned on this projection based on when they occurred relative to interactions _as well as_ when they occurred relative to page load time. By definition this means that all interactions should show up after LCP but it may show up differently on the chart due to the projection from multiple dimensions down to two. There's also the fact that TTVR calculations do not stop at first interaction, so on browsers that do not support LCP, we may see interactions before the proxy for that event.

The absolute value of each event is calculated across the entire dataset, even on pages without intereactions, so it might look like events aren't placed where their values dictate they should be, however the percentage of users interacting before & after an event is always correct.

The last label to take note of is the fraction of users that interacted before `boomerang` considered the page to be interactive. In this case, it's 12% of users.

Data distributions

There are a few different distributions shown on this chart, (and even more when we look at the mouseover in the chart above).

The blue area chart is the _population density_. It shows, for every 5% interval of the page load time, how many users first interacted with the page at that point in the page's loading cycle.

The blue dots that trace the population density chart show the median _Interaction to Next Paint_ value for all of those interactions. Keep in mind that INP is not supported on Safari, whereas `boomerang`'s own measurements for TTI do work across browsers.

The vertical position of the red dots shows the _probability_ that interactions at that time resulted in _rage clicks_ while the size of the red dots shows the _intensity_ of these rage clicks. Rage clicks are collected across browsers.

The thin orange line shows Frustration Index for users that interacted within that window.

We also have the median Total Blocking Time for each of these interactions, though that's only visible in the live versions of these charts and not in most of the screenshots posted here.

In this second chart, we see that 59% of users interacted with the site before it became interactive. Its TTI is further from the LCP time compared to the first site.

Insights from the data

When we look at this data across websites, we see the same patterns. Users expect to be able to interact with the site once the page is largely visible, however, the user experience for interactions is sub-optimal until the time to interactive which can be much later in the page's loading cycle.

In most cases we see a high Total Blocking Time in the period between LCP and TTI, resulting in a slow INP, and higher probability of rage clicking.

When looking to optimize a site for user experience, we shouldn't look at each metric in isolation. A really fast LCP is a great first user experience, but it's also a signal to the user that they can proceed with interacting to complete their task. It's important that the rest of the page be ready for those interactions and keep up the good experience.

The elephant in the room

As an aside, has anyone else noticed that these charts almost always look like a sleeping elephant (or maybe a hat)? I've seen very few sites where this isn't the case, so I looked into that pattern.

The population distribution pattern we see is a gradual curve increasing, then a dip that looks like the elephant's neck, then a bump that could be its ears, a sharp dip and long flat region that could be its trunk.

It could well be a Normal distribution if it weren't for the dip and spike right around PLT.

The drop-off after OnLoad is expected. `boomerang.js` sends a beacon on or soon after page load (sites can configure a beacon delay of a few seconds to capture post-onload events). This results in a drop-off in data with interactions after onload. The post onload interactions are on pages that are faster than the average.

The strange pattern is the spike in interactions just at or after onload (it's sometimes at 100% and sometimes at 105%). The dip at 95% & 100% shows up on most, but not all sites, but the spike shows up everywhere.

I looked closer at the data around those buckets and there is very little difference in terms of experience. The page load time, LCP time, TTI time, etc. are all very similar at the 25th and 75th percentile (in other words, the experiences are comparable). The only difference is that more users prefer to interact with the site just after the onload event has fired than just before it. It's not a big delay - about 200-400ms on average across sites, but it does look like some portion of users still wait for the loading indicator to complete before they interact.

Conclusions

In conclusion, I think there's a lot to be learned from looking at when your users interact with your site. Which parts of the page have finished loading when that interaction happens? What's still in flight? What do they experience? Is there too much of a delay between your LCP and the site becoming usable?

A good loading experience needs your page to transition from state to state smoothly without too much delay between states. Looking at the loading Frustration Index can identify pages where this isn't the case.

When comparing different events on the page, look at the aggregate of deltas rather than the delta of aggregates.

And lastly, keep an eye out for that elephant.

References

Glossary on Mozilla Developer Network

• First Paint
• First Contentful Paint
• Largest Contentful Paint
• Interaction to Next Paint

Web Vitals on Google's Web.Dev

• Interaction to Next Paint
• Total Blocking Time
• Largest Contentful Paint
• First Contentful Paint

Implementations in mPulse

• boomerang
• Frustration Index by Tim Vereecke
• Time to Visually Ready in boomerang
• Time to Interactive in boomerang
• Monitoring Interactions
• Frustration Index in mPulse

Other useful links

• Paul Calvano on different performance metrics
• The RUM Archive to do your own RUM analysis.