The other side of the moon

This is an old story. Old enough that the guilty parties have either learnt from their mistakes or have moved on to other things.

It was 1999. I was an intern at an ERP/CRM company called Thirdware Solutions Pvt. Ltd. or TSPL for short. If you'd asked me a year earlier, I wouldn't have heard of this company, and I wouldn't have imagined ever working in this space. However the people who interviewed us made a convincing argument that as interns we'd be able to work on fringe technologies that they couldn't spare the rest of their full time engineers for. The web was one of these. It was very new to Indian businesses, the dot com bubble was at its peak. I'd been dabbling with HTML, JavaScript, and CSS for about 3 years and was interested in learning more about the underlying protocols. So I joined them.

I largely consider this a good decision. After the initial training on what the company did and how it did it, I was given some freedom to explore. In about about a week I'd reformatted my windows box and installed RedHat Linux 5.2. Reading through all the HOWTOs that I could find, I managed to set up daemons for DNS, DHCP, SMTP, HTTP and POP3.

At the time the company had a single Pentium box that would connect to the internet over dialup. They used a commercial HTTP & Email proxy on this box that only allowed 3 people to connect at a time, so people in the office developed an honour system where each would connect, send & receive email, and then disconnect as soon as possible. If anyone was expecting large customer requests, they'd let the rest of the office know and people would stay off the network. I took it upon myself to "fix" this. With the blessing of our network administrator and the company CEO, I wrote a little Java app that proxied ports 25 and 110. I couldn't figure out HTTP proxying yet, but POP3 & SMTP were ok. I just had to give everyone a new "proxy email address" that they'd use when connecting to the proxy and that would be translated to the actual address when going out to the server.

We left the web throttled at 3 users since no one in the office needed to access the web, and email was the most critical use of the internet.

This worked out quite well, so the leadership team started to trust me a fair bit. I do not think that any other company would have trusted an intern with the kinds of decisions they let me make following that, but it leads directly into how we avoided a fairly bad security situation.

A few months later we decided to make our ERP/CRM system available over the web. A full rewrite would take over a year, but we found something called Citrix App Server, that ran on Windows NT and would make any desktop application available to someone over the web taking care of basic authentication. We tested it out locally and it worked well on our LAN, so we now had to make it available to our customers. Except, this wasn't happening over a 56K dialup network that only allowed 3 users through at a time.

We ended up speaking to the top ISPs in India at the time, and got a great deal from one of them to put our Windows NT box inside their datacenter, on their always on network.

A few weeks later we locked the hard drive. No, this is not a security thing. This is a process of moving the drive's arm to the outer most "locked" position, so that significant vibrations would not result in the head hitting and damaging the disk platters. We did this because the next step was me and our network admin sitting in the backseat of the company president's car with our Windows NT Pentium 6 Tower PC resting across our laps while our company president drove us down the length of Mumbai trying to avoid as many potholes as possible.

We made it to the other end and when I powered the host back up and unlocked the drive (automatically on boot up), it still ran, so we were happy. We went into an unmarked building, carried the box to a floor with security guards outside the door, and a keypad entry. Inside, there were closets of blades and a few minitowers and tower hosts sitting on the bottom shelf of a rack. We were told to put our box next to the others, and then the guy who ran the datacenter said the magic words.

Start up your server, and set the Administrator password to "password"

I glanced over at the other boxes, and they all had stickers on them saying "Administrator/password"

The three of us from TSPL looked at each other, and our president told me to decide. I asked the datacenter guy why he needed that. He said that sometimes they need to shutdown the boxes so they can move them to a different power strip. I asked him if it would be sufficient to give him an account that only had local access and could only reboot the box. He thought about it for a bit and said yes.

So I created a new account that required a physically attached keyboard for login, and all it had was the ability to reboot the box. Our app was set up to start up automatically on boot, so we weren't worried about someone having to start it. DC guy physically locked the box to a rack, showed us that he was keeping they key, and we headed back to the office.

We now needed to test our setup, so we asked everyone in the office to let us use the internet connection. We tried accessing our app, and it worked!

Since I had Admin access to our box, I was also able to open the "Network Neighbourhood" of our box in the datacenter. On that network, I saw all the other hosts that were in the datacenter. They had names identifying them from India's largest IT companies. These were companies I'd initially though of interning at.

I looked at our president and grinned, and he looked back and said, "Send me a safe summary report when you're done" and walked off to his office.

I double clicked on one of the other big boxes and was prompted for a username and password to connect to it.

You can probably guess what happened next ;)