The other side of the moon

One of the things that drew me to Everybuddy, its USP really, was fallback messaging. I haven't seen any other client (other than eb's offspring -- ayttm and eb-lite) implement this feature, which is why I've never switched to another client.

So, what is fallback messaging?

Consider two friends who communicate via various network oriented means (IM, Email, SMS, etc.), we'll call them bluesmoon and mannu (because they are two people who communicated this way for several years before meeting IRL). Now, said friends are extremely tech savvy, and have accounts on virtually every server that offers free accounts, and then some.

So, you've got them on MSN, Yahoo!, AOL, ICQ... um, ok, not ICQ because ICQ started sucking, Jabber, and that's just IM. They prolly have 3 or 4, maybe 5 accounts on each of these services, ok, maybe just one on AOL. Then they have email accounts. The standard POP3 accounts, 3 gmail accounts, a Yahoo! account for every yahoo ID, and likely no hotmail accounts (even though they have MSN passports) because we all know that hotmail is passé.

These guys also have lj accounts and one or two cellular phones on different service providers.

Ok, now that we have our protagonists well defined, let's set up the scene.

Act 1, scene 1

Mannu and bluesmoon are chatting over, umm, we'll pick MSN to start with. So mannu and bluesmoon are chatting over MSN, when all of a sudden <insert musical score for suspense here> a message pops up:

The MSN server is going down for maintenance, all conversations will now end

Sweet. So MSN decides that we're not allowed to talk anymore.

What are our options? Oh well, Yahoo!'s still online, so switch to Yahoo!. It's much nicer too because you can chat while invisible too. MSN (officially) doesn't let you do that.

So, they switch to Yahoo!, but... what the heck were they chatting about when the server went down? Context lost. They need to start a new conversation, most likely centred around cursing MSN. What's worse is that the earlier conversation was being archived because they may have needed it as a reference later. The new conversation can also be archived, but it's a pain to merge all these different archives later.

Anyway, they plough ahead. The conversation veers back on topic, ... but now the main net connection goes down. The only things that work are websites and email. What do you do? What do you do? Ok, Dennis Hopper I am not, so let's forget I said that.

The easiest option would be for bluesmoon to send a mail to mannu saying, "Hey dude, my net connection went down, gotta end the convo here.", or he could send the same in an SMS. But to do that he's gotta start yet another program and type out stuff out of context again, or worse, type out an SMS that he has to pay for!

So, here's where fallback messaging comes in.

Act 1, Scene 1, Take 2

<jump back to the MSN chat>

Where were we? Oh yeah, the MSN server goes down. Now, what if the chat client we were using was smart enough to figure this out, and do something about it. What's that something you say? Switch to the next available service. So, in this case, the chat program would automatically and seamlessly switch to using Yahoo!

There's several user centric pluses here. The people chatting do not need to know that a server went down, leave alone care about it and figure out what to do. Archives will be maintained across sessions. The context of the conversation will be preserved. Mannu and bluesmoon can go on chatting as if nothing happened.

If all the IM protocols go down, the chat client could switch to Email or SMS. Of course, mannu should have to tell it explicitly to use one of these, because the conversation will no longer be online. There's gonna be delays between sending the message and getting a response, so the chatters need to know about this.

So, how does your chat client know that you have buddies on multiple services, and about their email address and phone number?

Well, the chat client would have to group accounts on various services into a single contact. This kind of grouping also has other benefits.

Two people chatting with each other now don't have to think about user names and different services and what not. Mannu wants to chat with bluesmoon, he just selects bluesmoon from his buddy list. He doesn't have to care whether bluesmoon has an account on MSN, Yahoo!, AOL or whatever. Why should he care? So, I'd be chatting one to one with another person, without caring about what happens behind the scenes. Isn't that what makes for a good play?

Well, at some point mannu would have to care about services and user names, because he'd actually have to manually add and group all these accounts into one. Perhaps he could also set preferences of the order in which to fallback. That's all a one time set up. For the continuous ease of use to follow, I'd say it's worth it.

Final questions...

Is this really possible? Yeah, sure it is. You can thank Torrey Searle for that. Torrey implemented everybuddy to do just this, and threw in loads of sanity checking - thanks for that dude. It's what drew me to use and then work on the project for so long.

So, is this really possible? Probably not until IM companies decide that the network is just a transport, and it's the value a user derives from using that transport that makes them choose one service over another. It's why we choose the Mumbai-Pune expressway over NH4 that runs through the ghats, even though there's a toll.

Update I did a talk on fallback messaging at Linux Bangalore 2004

A colleague of mine deleted a source file he'd been working on for over a week.

How do you undelete a file on a UFS partition? I'd done it before on ext2, I'd also recovered lost files from a damaged FAT32 partition (one that no OS would recognise), heck, I'd even recovered an ext3 file system that had been overwritten by NTFS. Why should undelete be tough?

Well, the tough part was that in all earlier instances, I had a spare partition to play on, _and_ I had root (login, not sudo) on the box, so could effectively boot in single user mode and mount the affected partition read-only. Couldn't do that here. I'd have to work on a live partition. sweet.

The first thing to do was power down the machine (using the power switch) to prevent any writes to the disk via cron or anything else. We then set about trying to figure out our strategy. A couple of websites had tools that could undelete files, but they'd have to be installed on the affected partition, so that was out of the question.

Now the machine has two partitions, one for / and one for /home. / is significantly smaller than /home, but has enough space to play around 100MB at a time. Decided to give it a try, copying /home to /tmp 10MB at a time.

Command:

dd if=/dev/ad0s1e of=deleted-file bs=1024k count=10

searched through the 10MB file for a unique string that should have been in the file. No match. Next 10 MB:

dd if=/dev/ad0s1e of=deleted-file bs=1024k count=10 skip=10

This was obviously going to take all night, so we decided to script it. (the code is broken into multiple lines for readability, we actually had it all on one line).

for i in 10 20 30 40 50 60 70 80 90; do
    dd if=/dev/ad0s1e of=deleted-file bs=1024k count=10 skip=$i;
    grep unique-string deleted-file && echo $i
done

We'd note down the numbers that hit a positive and then go back and get those sections again. Painful.

Ran through without luck. Had to then go from 100 to 190, so scripted that too with an outer loop:

for j in 1 2 3 4 5 6 7 8 9; do
    for i in 00 10 20 30 40 50 60 70 80 90; do
        dd ..... of=deleted-file ... skip=$j$i; ...

The observant reader will ask why we didn't just put in an increment like i=$[ $i+10 ]
Well, that runs too fast, and we wouldn't be able to break out easily. We'd have to do a Ctrl+C for every iteration to break out. This way the sheer pain of having to type in every number we wanted was enough to keep the limits low. That wasn't the reason. We did it because it would also be useful when we had to test only specific blocks that didn't have successive numbers.

IAC, the number of loops soon increased to 3, and the script further evolved to this:

for s in 1 2 3 4; do
    for j in 0 1 2 3 4 5 6 7 8 9; do
        for i in 00 10 20 30 40 50 60 70 80 90; do
            dd if=/dev/ad0s1e of=deleted-file bs=1024k count=10 skip=$s$j$i &>/dev/null;
            grep unique-string deleted-file && echo $s$j$i
        done
    done
done

Pretty soon hit a problem when grep turned up an escape sequence that messed up the screen. Also decided that we may as well save all positive hit files instead of rebuilding them later, so... broke out of the loops, and changed the grep line to this:

grep -q unique-string deleted-file-$s$j$i || rm deleted-file-$s$j$i

Were reasonably happy with the script to leave it to itself now. Might have even changed the iteration to an auto-increment, except there was no point changing it now since what we had would work for the conceivable future (going into the 10's place would be as easy as changing s to 10 11 12... and we didn't expect to have to go much further than 12 because the partition didn't have that much used space).

We finally hit some major positives between 8700 and 8900. Then started the process of extracting the data. 10MB files are too big for editors, and contains mostly binary data that we could get rid off. There was also going to be a lot of false positives because the unique (to the project) string also showed up in some config files that hadn't been deleted.

First ran this loop:

for i in deleted-file-*; do strings $i | less; done

and tried to manually search for the data. Gave up very soon and changed it to this:

for i in deleted-file-*; do echo $i; strings $i | grep unique-string; done

This showed us all lines where unique-string showed up so we could eliminate files that had no interesting content.

We were finally left with 3 files of 10MB each and the task of extracting the deleted file from here.

The first thing was to find out where in the file the code was. We first tried this:

less deleted-file-8650

search for the unique string and scroll up to the start of the text we wanted. Ctrl+G told us the position into the file that we were at (as a percent of the total). Then scroll to the end and again find the percent.

Now, we were reading 10 blocks of 1MB so using the percentage range, could narrow that down to 1 block.

Again got a percentage value within this 1MB file, and now swapped the block size and count a bit. So we went from 1 block of 1024k to 256 blocks of 4k each. Also had to change the offset from 8650 to 256 times that much. bc came in handy here.

I should probably mention that at this point we'd taken a break and headed out to Guzzler's Inn for a couple of pitchers and to watch the olympics. 32/8 was a slightly hard math problem on our return. Yathin has a third party account of that.

We finally narrowed down the search to 2 2K sections and one 8K section, with about 100 bytes of binary content (all ASCII NULLs) at the end of one of the 2K sections. This section was the end of the file. Used gvim to merge the files into one 12K C++ file complete with copyright notice and all.

If you plan on doing this yourself, then change the three for loops to this:

i=10;
while [ $i -lt 9000 ]; do
    dd ...
    i=$[ $i+10 ];
done

Secondly, you could save a lot of time by using grep -ab right up front so you'd get an actual byte count of where to start looking, and just skip the rest. Some people have suggested doing the grep -ab right on the filesystem, but that could generate more data than we could store (40GB partition, and only 200MB of space to store it on).

What does one write in a first post? A new beginning? A fresh start? The first step? It has always been hard for me to start a new project. Be it an article a large work of programming or cleaning my room. It is hard to start. The first step is always the hardest. It is easy to proceed if the project is interesting enough.

The question that begs to be answered with this project is why? I already have a journal, so why create a separate blog? What do I hope to achieve? How will this be different? Do I plan on maintaining two publishing outlets?

My journal - Blue Swiss Cheese, or The blues side of the moon - is, and remains to be a journal of what I've been up to. A collection of writings aimed at the community nature of livejournal. A space to jot down informal thoughts, birthday wishes, opinions and reports of meetings with interesting persons.

On the other side of the moon, I hope to publish articles of a more technical or documentary nature. I'd like to move my restaurant reviews here, my series on web standards, and my travelogues too. I'd like to categorise articles, although I'm still not sure how. This space will have more articles rather than posts, language will, I hope, be more formal, or less casual at the least.

Well, so much for an editorial, if I may call this one. Let the writing begin.