The other side of the moon

A little over a year ago, at my previous company, I had to change my second harddisk on my PC. It was a bit of an experience, because the service engineer who came to do the job had never encountered linux before, but seemed to think that he could deal with it just like he did windows.

The engineer put in the new hard disk as a secondary master (my old one was a secondary slave to the CDD).

He then booted using a Win 95 diskette... hmm... what's this? Then started some norton disk copy utility. It's a DOS app that goes into a graphics mode... why?

Then started transferring data... hey, wait a minute, I don't have any NTFS partitions. Hit reset! Ok, cool down for a minute. I've got three ext3 partitions. So, now it's time to assess the damage.

Boot into linux - hmm, /dev/hdd1 (/home) won't mount, down to root shell. Get /home out of /etc/fstab and reboot. Ok, runlevel 3 again. Check other partitions - hdd5 (/usr) ... good, hdd6 (/usr/share) ... good... everything else is on hda... good. all my data, is in /home ... not good

So, I start trying to figure out how to recover. google... no luck. google again... one proprietary app, and a couple of howtos on recovering deleted files from ext2 partitions... no good. google again, get some docs on the structure of efs2, and find a util called e2salvage which won't build. time to start fooling around myself.

start by reading man pages. tune2fs, e2fsck, debugfs, mke2fs... so I know that mke2fs makes backups of the superblock, but where are they?

mke2fs -n /dev/hdd1... ok, that's where
dd if=/dev/hdd5 of=superblock bs=4096 count=2
hmm, so that's what a superblock looks like
dd if=/dev/hdd5 of=superblock2 bs=4096 count=2 skip=32768
hey, that's not a superblock. Ok, try various combinations, finally get this:
dd if=/dev/hdd5 of=superblock2 bs=1024 count=8 skip=131071
that's 32768*4-1
Ok, so that's where the second superblock is.

Check hdd1 - second superblock blown away as well. Look for the third... 98304*4-1=393215.. ok, that's good. should I dd it to the first? Hmm, no, e2fsck can do that for me... but, I shouldn't work on the original partition. Luckily I have 30GB of free space to play with, and /home is just 6GB.

dd if=/dev/hdd1 of=/mnt/tmp/home.img
cp home.img bak.home.img

Now I start playing with home.img.

The instructions in e2salvage said to try e2fsck before trying e2salvage, so I try that.

e2fsck home.img
no use... can't find superblock
e2fsck -b 32768 -B 4096 home.img
works... starts the fsck, and gets rid of the journal. this is gonna take too long if I do it manually, so I quit, and restart with:
e2fsck -b 32768 -B 4096 -y home.img
The other option would have been to -p(reen) it, but that wouldn't give me any messages on stdout, so I stuck with -y(es to all questions).

2 passes later it says, ok, got whatever I could.

mount -oloop,ro home.img /mnt/home
yippeee, it mounted
cd /mnt/home; ls
lost+found
ok, so everything's in lost+found, and it will take me ages to sift through all this. Filenames might give me some clues.
find . -type f | less
Ok, scroll, scroll, scroll... hmm, this looks like my home directory... yes.
cp -a \#172401 /mnt/home/philip
scroll some more, find /usr/share/doc (which I keep in /home/doc and symlink from /usr/share/doc). move it back to /usr/share/doc. find jdk1.1.8 documentation... pretend I didn't see that.

find moodle home - yay. find yabb home - yay again. Ok, find a bit more that's worth saving, and copy it over. Many files in each of these directories are corrupted, including mailboxes, and some amount of test data, but haven't found anything serious missing.

All code was in CVS anyway, so rebuilt from there where I had to.

Now decided to try e2salvage anyway, on the second copy of hdd1. It wouldn't compile. Changed some code to get it to compile, it ran, found inodes, directories and the works, then segfaulted. The program tries to read from inode 2, which doesn't exist on my partition, and then it tries to printf that inode without checking the return value.

I'd have fixed that, but the result is used in further calculations, so I just left it at that. The old hard disk was taken away, so I don't have anything to play with anymore.

It'll take me a little while to figure out all that was lost, but so far it doesn't look like anything serious.