The other side of the moon

I've been using a GeForce MX 2 for well over a year. It worked quite well with RH8, FC1 and Knoppix. I needed to use the proprietary drivers provided by Nvidia to get hardware acceleration though.

Motherboard: ASUS A7N266

A couple of months ago, upgraded to FC2, and the nvidia driver wouldn't work anymore. I had to run back to Bangalore, and since no one at home really needed hardware acceleration, I switched back to the free nv driver from X (well, I was using x.org now).

This December... well, yesterday actually, I decided to try out 3ddesktop, but of course, this requires hardware acceleration. So I started. Went through a lot to get it to work, and the details are boring. However, what I learnt could help others, so I'll document that.

The problem:

When starting X with the nvidia driver, the screen blanked out and the system froze. Pushing the reset button is the only thing that worked.

Solutions and Caveats:

Get the latest NVIDIA drivers and try.

At the time of writing, the latest drivers from the nvidia site are in the 1.0-6629 package. This doesn't work with the GeForce MX 2, and many other older chips, so if you try to use it, you'll spend too much time breaking your head for nothing. Instead, go for the 1.0-6111 driver, which does work well...

On kernels below 2.6.5 that is. FC2 ships with a modified 2.6.5 kernel that has a forced 4K stack and CONFIG_REGPARM turned on. The NVIDIA drivers are (or were) compiled with 8K stacks and do not work with CONFIG_REGPARM turned on. I'd faced similar problems when I first used the nvidia driver, and recompiling my kernel with 8K stacks fixed the problem.

Searching the net, I came across dozens of articles that spoke about 4K stacks v/s 8K stacks in the 2.6 kernel, but also said that from 5xxx onwards, the nvidia driver supported 4K stacks and CONFIG_REGPARM.

I tried getting prebuilt kernels (smaller download) with 16K stacks, but it didn't help, so finally decided to downlad the entire 32MB kernel source for 2.6.10.

While compiling, I came across this thread on NV News (pretty much the best resource for nvidia issues on linux). In short, the 6111 driver wouldn't work with kernels above 2.6.5 or something like that. I needed to patch the kernel source.

The patch was tiny enough: in arch/i386/mm/init.c, add a single line:
EXPORT_SYMBOL(__VMALLOC_RESERVE);
after the __VMALLOC_RESERVE definition.

Stopped compilation, made the change and restarted compilation.

Also had to rebuild the NVIDIA driver package, again as documented in that thread:

- extract the sources with the command : ./NVIDIA-Linux-x86-1.0-6111-pkg1.run --extract-only
- in the file "./NVIDIA-Linux-x86-1.0-611-pkg1/usr/src/nv/nv.c" replace the 4 occurences of
'pci_find_class' by 'pci_get_class'
- repack the nvidia installer with the following command:

sh ./NVIDIA-Linux-x86-1.0-6111-pkg1/usr/bin/makeself.sh --target-os Linux --target-arch x86 NVIDIA-Linux-x86-1.0-6111-pkg1 NVIDIA-Linux-x86-1.0-6111-pkg2.run "NVIDIA Acclerated Graphics Driver for Linux-x86 1.0-6111" ./nvidia-installer

The new installer is called "NVIDIA-Linux-x86-1.0-6111-pkg2.run"

With these changes, the driver compiled successfully and I was able to insert it.

I had a minor problem when rebooting. usbdevfs has become usbfs, so a change has to be made in /etc/rc.sysinit. Change all occurences of "usbdevfs usbdevfs" to "usbfs none".

Once you've done this, X should start with acceleration on.

3ddesktop is slow, but it starts up. Tux racer works well.

What I think is really cool about this solution, is that I did not have to make a single post to a single mailing list or forum. All the information I needed was already on the net. It was just a matter of reading it, understanding what it said, and following those instructions. For example, there were many threads on screen blanking with the 6629 driver, and somewhere in there was mentioned that the new driver didn't support older hardware, but that the 6111 did. That was the key that brought me the solution. I knew the 6111 didn't work out of the box, because I'd already tried it, but now I could concentrate on threads about the 6111 exclusively, only looking for anything that sounded familiar.

A little over a year ago, at my previous company, I had to change my second harddisk on my PC. It was a bit of an experience, because the service engineer who came to do the job had never encountered linux before, but seemed to think that he could deal with it just like he did windows.

The engineer put in the new hard disk as a secondary master (my old one was a secondary slave to the CDD).

He then booted using a Win 95 diskette... hmm... what's this? Then started some norton disk copy utility. It's a DOS app that goes into a graphics mode... why?

Then started transferring data... hey, wait a minute, I don't have any NTFS partitions. Hit reset! Ok, cool down for a minute. I've got three ext3 partitions. So, now it's time to assess the damage.

Boot into linux - hmm, /dev/hdd1 (/home) won't mount, down to root shell. Get /home out of /etc/fstab and reboot. Ok, runlevel 3 again. Check other partitions - hdd5 (/usr) ... good, hdd6 (/usr/share) ... good... everything else is on hda... good. all my data, is in /home ... not good

So, I start trying to figure out how to recover. google... no luck. google again... one proprietary app, and a couple of howtos on recovering deleted files from ext2 partitions... no good. google again, get some docs on the structure of efs2, and find a util called e2salvage which won't build. time to start fooling around myself.

start by reading man pages. tune2fs, e2fsck, debugfs, mke2fs... so I know that mke2fs makes backups of the superblock, but where are they?

mke2fs -n /dev/hdd1... ok, that's where
dd if=/dev/hdd5 of=superblock bs=4096 count=2
hmm, so that's what a superblock looks like
dd if=/dev/hdd5 of=superblock2 bs=4096 count=2 skip=32768
hey, that's not a superblock. Ok, try various combinations, finally get this:
dd if=/dev/hdd5 of=superblock2 bs=1024 count=8 skip=131071
that's 32768*4-1
Ok, so that's where the second superblock is.

Check hdd1 - second superblock blown away as well. Look for the third... 98304*4-1=393215.. ok, that's good. should I dd it to the first? Hmm, no, e2fsck can do that for me... but, I shouldn't work on the original partition. Luckily I have 30GB of free space to play with, and /home is just 6GB.

dd if=/dev/hdd1 of=/mnt/tmp/home.img
cp home.img bak.home.img

Now I start playing with home.img.

The instructions in e2salvage said to try e2fsck before trying e2salvage, so I try that.

e2fsck home.img
no use... can't find superblock
e2fsck -b 32768 -B 4096 home.img
works... starts the fsck, and gets rid of the journal. this is gonna take too long if I do it manually, so I quit, and restart with:
e2fsck -b 32768 -B 4096 -y home.img
The other option would have been to -p(reen) it, but that wouldn't give me any messages on stdout, so I stuck with -y(es to all questions).

2 passes later it says, ok, got whatever I could.

mount -oloop,ro home.img /mnt/home
yippeee, it mounted
cd /mnt/home; ls
lost+found
ok, so everything's in lost+found, and it will take me ages to sift through all this. Filenames might give me some clues.
find . -type f | less
Ok, scroll, scroll, scroll... hmm, this looks like my home directory... yes.
cp -a \#172401 /mnt/home/philip
scroll some more, find /usr/share/doc (which I keep in /home/doc and symlink from /usr/share/doc). move it back to /usr/share/doc. find jdk1.1.8 documentation... pretend I didn't see that.

find moodle home - yay. find yabb home - yay again. Ok, find a bit more that's worth saving, and copy it over. Many files in each of these directories are corrupted, including mailboxes, and some amount of test data, but haven't found anything serious missing.

All code was in CVS anyway, so rebuilt from there where I had to.

Now decided to try e2salvage anyway, on the second copy of hdd1. It wouldn't compile. Changed some code to get it to compile, it ran, found inodes, directories and the works, then segfaulted. The program tries to read from inode 2, which doesn't exist on my partition, and then it tries to printf that inode without checking the return value.

I'd have fixed that, but the result is used in further calculations, so I just left it at that. The old hard disk was taken away, so I don't have anything to play with anymore.

It'll take me a little while to figure out all that was lost, but so far it doesn't look like anything serious.

One of the things that drew me to Everybuddy, its USP really, was fallback messaging. I haven't seen any other client (other than eb's offspring -- ayttm and eb-lite) implement this feature, which is why I've never switched to another client.

So, what is fallback messaging?

Consider two friends who communicate via various network oriented means (IM, Email, SMS, etc.), we'll call them bluesmoon and mannu (because they are two people who communicated this way for several years before meeting IRL). Now, said friends are extremely tech savvy, and have accounts on virtually every server that offers free accounts, and then some.

So, you've got them on MSN, Yahoo!, AOL, ICQ... um, ok, not ICQ because ICQ started sucking, Jabber, and that's just IM. They prolly have 3 or 4, maybe 5 accounts on each of these services, ok, maybe just one on AOL. Then they have email accounts. The standard POP3 accounts, 3 gmail accounts, a Yahoo! account for every yahoo ID, and likely no hotmail accounts (even though they have MSN passports) because we all know that hotmail is passé.

These guys also have lj accounts and one or two cellular phones on different service providers.

Ok, now that we have our protagonists well defined, let's set up the scene.

Act 1, scene 1

Mannu and bluesmoon are chatting over, umm, we'll pick MSN to start with. So mannu and bluesmoon are chatting over MSN, when all of a sudden <insert musical score for suspense here> a message pops up:

The MSN server is going down for maintenance, all conversations will now end

Sweet. So MSN decides that we're not allowed to talk anymore.

What are our options? Oh well, Yahoo!'s still online, so switch to Yahoo!. It's much nicer too because you can chat while invisible too. MSN (officially) doesn't let you do that.

So, they switch to Yahoo!, but... what the heck were they chatting about when the server went down? Context lost. They need to start a new conversation, most likely centred around cursing MSN. What's worse is that the earlier conversation was being archived because they may have needed it as a reference later. The new conversation can also be archived, but it's a pain to merge all these different archives later.

Anyway, they plough ahead. The conversation veers back on topic, ... but now the main net connection goes down. The only things that work are websites and email. What do you do? What do you do? Ok, Dennis Hopper I am not, so let's forget I said that.

The easiest option would be for bluesmoon to send a mail to mannu saying, "Hey dude, my net connection went down, gotta end the convo here.", or he could send the same in an SMS. But to do that he's gotta start yet another program and type out stuff out of context again, or worse, type out an SMS that he has to pay for!

So, here's where fallback messaging comes in.

Act 1, Scene 1, Take 2

<jump back to the MSN chat>

Where were we? Oh yeah, the MSN server goes down. Now, what if the chat client we were using was smart enough to figure this out, and do something about it. What's that something you say? Switch to the next available service. So, in this case, the chat program would automatically and seamlessly switch to using Yahoo!

There's several user centric pluses here. The people chatting do not need to know that a server went down, leave alone care about it and figure out what to do. Archives will be maintained across sessions. The context of the conversation will be preserved. Mannu and bluesmoon can go on chatting as if nothing happened.

If all the IM protocols go down, the chat client could switch to Email or SMS. Of course, mannu should have to tell it explicitly to use one of these, because the conversation will no longer be online. There's gonna be delays between sending the message and getting a response, so the chatters need to know about this.

So, how does your chat client know that you have buddies on multiple services, and about their email address and phone number?

Well, the chat client would have to group accounts on various services into a single contact. This kind of grouping also has other benefits.

Two people chatting with each other now don't have to think about user names and different services and what not. Mannu wants to chat with bluesmoon, he just selects bluesmoon from his buddy list. He doesn't have to care whether bluesmoon has an account on MSN, Yahoo!, AOL or whatever. Why should he care? So, I'd be chatting one to one with another person, without caring about what happens behind the scenes. Isn't that what makes for a good play?

Well, at some point mannu would have to care about services and user names, because he'd actually have to manually add and group all these accounts into one. Perhaps he could also set preferences of the order in which to fallback. That's all a one time set up. For the continuous ease of use to follow, I'd say it's worth it.

Final questions...

Is this really possible? Yeah, sure it is. You can thank Torrey Searle for that. Torrey implemented everybuddy to do just this, and threw in loads of sanity checking - thanks for that dude. It's what drew me to use and then work on the project for so long.

So, is this really possible? Probably not until IM companies decide that the network is just a transport, and it's the value a user derives from using that transport that makes them choose one service over another. It's why we choose the Mumbai-Pune expressway over NH4 that runs through the ghats, even though there's a toll.

Update I did a talk on fallback messaging at Linux Bangalore 2004