The other side of the moon

[philiptellis] /bb|[^b]{2}/
Never stop Grokking

Thursday, April 13, 2017

A tale of datacenter security

This is an old story. Old enough that the guilty parties have either learnt from their mistakes or have moved on to other things.

It was 1999. I was an intern at an ERP/CRM company called Thirdware Solutions Pvt. Ltd. or TSPL for short. If you'd asked me a year earlier, I wouldn't have heard of this company, and I wouldn't have imagined ever working in this space. However the people who interviewed us made a convincing argument that as interns we'd be able to work on fringe technologies that they couldn't spare the rest of their full time engineers for. The web was one of these. It was very new to Indian businesses, the dot com bubble was at its peak. I'd been dabbling with HTML, JavaScript, and CSS for about 3 years and was interested in learning more about the underlying protocols. So I joined them.

I largely consider this a good decision. After the initial training on what the company did and how it did it, I was given some freedom to explore. In about about a week I'd reformatted my windows box and installed RedHat Linux 5.2. Reading through all the HOWTOs that I could find, I managed to set up daemons for DNS, DHCP, SMTP, HTTP and POP3.

At the time the company had a single Pentium box that would connect to the internet over dialup. They used a commercial HTTP & Email proxy on this box that only allowed 3 people to connect at a time, so people in the office developed an honour system where each would connect, send & receive email, and then disconnect as soon as possible. If anyone was expecting large customer requests, they'd let the rest of the office know and people would stay off the network. I took it upon myself to "fix" this. With the blessing of our network administrator and the company CEO, I wrote a little Java app that proxied ports 25 and 110. I couldn't figure out HTTP proxying yet, but POP3 & SMTP were ok. I just had to give everyone a new "proxy email address" that they'd use when connecting to the proxy and that would be translated to the actual address when going out to the server.

We left the web throttled at 3 users since no one in the office needed to access the web, and email was the most critical use of the internet.

This worked out quite well, so the leadership team started to trust me a fair bit. I do not think that any other company would have trusted an intern with the kinds of decisions they let me make following that, but it leads directly into how we avoided a fairly bad security situation.

A few months later we decided to make our ERP/CRM system available over the web. A full rewrite would take over a year, but we found something called Citrix App Server, that ran on Windows NT and would make any desktop application available to someone over the web taking care of basic authentication. We tested it out locally and it worked well on our LAN, so we now had to make it available to our customers. Except, this wasn't happening over a 56K dialup network that only allowed 3 users through at a time.

We ended up speaking to the top ISPs in India at the time, and got a great deal from one of them to put our Windows NT box inside their datacenter, on their always on network.

A few weeks later we locked the hard drive. No, this is not a security thing. This is a process of moving the drive's arm to the outer most "locked" position, so that significant vibrations would not result in the head hitting and damaging the disk platters. We did this because the next step was me and our network admin sitting in the backseat of the company president's car with our Windows NT Pentium 6 Tower PC resting across our laps while our company president drove us down the length of Mumbai trying to avoid as many potholes as possible.

We made it to the other end and when I powered the host back up and unlocked the drive (automatically on boot up), it still ran, so we were happy. We went into an unmarked building, carried the box to a floor with security guards outside the door, and a keypad entry. Inside, there were closets of blades and a few minitowers and tower hosts sitting on the bottom shelf of a rack. We were told to put our box next to the others, and then the guy who ran the datacenter said the magic words.

Start up your server, and set the Administrator password to "password"

I glanced over at the other boxes, and they all had stickers on them saying "Administrator/password"

The three of us from TSPL looked at each other, and our president told me to decide. I asked the datacenter guy why he needed that. He said that sometimes they need to shutdown the boxes so they can move them to a different power strip. I asked him if it would be sufficient to give him an account that only had local access and could only reboot the box. He thought about it for a bit and said yes.

So I created a new account that required a physically attached keyboard for login, and all it had was the ability to reboot the box. Our app was set up to start up automatically on boot, so we weren't worried about someone having to start it. DC guy physically locked the box to a rack, showed us that he was keeping they key, and we headed back to the office.

We now needed to test our setup, so we asked everyone in the office to let us use the internet connection. We tried accessing our app, and it worked!

Since I had Admin access to our box, I was also able to open the "Network Neighbourhood" of our box in the datacenter. On that network, I saw all the other hosts that were in the datacenter. They had names identifying them from India's largest IT companies. These were companies I'd initially though of interning at.

I looked at our president and grinned, and he looked back and said, "Send me a safe summary report when you're done" and walked off to his office.

I double clicked on one of the other big boxes and was prompted for a username and password to connect to it.

You can probably guess what happened next ;)

Wednesday, May 27, 2015

Velocity Santa Clara 2015 -- My List

At Velocity SC 2015, these are the talks that I'd really like to see if only I could be in more than one room at a time.

Wednesday, May 27

09:00 Service Workers by Pat Meenan

Exciting new technology currently available in Chrome

11:00 Building performant SPAs by Chris Love

Since we're doing a lot of SPA work for boomerang at the moment, I'm very interested in performance best practices for SPAs

13:30 Metrics Metrics Everywhere by Tammy & Cliff

Tammy and Cliff are my colleagues at SOASTA, and this talk is based on a lot of the data that I've been working to collect over the last few years. I'm torn between this and the next one also at the same time.

13:30 Self-healing systems by Todd & Matt

Todd & Matt are also colleagues at SOASTA, and this talk is about the infrastructure we've developed to collect the metrics that are covered in the talk that Tammy & Cliff are doing. I really wish I could be at both.

15:30 Linux Perf Tools by Brendan Gregg

Always interested in tools to analyse linux performance.

Thursday, May 28

I haven't listed the keynotes here because that's the only track at the time and I don't need to choose which room to be in.

13:45 LinkedIn's use of RUM by Ritesh Maheshwari

LinkedIn uses a modified version of boomerang, and I'm keen to know what they've done.

13:45 Stream processing and anomaly detection by Arun Kejriwal

Very interesting topic, something that I'm very interested in.

13:45 Design & Performance by Steve Souders

Steve's talks are always educational

14:40 Visualising Performance Data by Mark Zeman

Again, this is something I'm working on at the moment, so very interested.

14:40 Failure is an Option by Ian Malpass

Etsy's devops talks are always educational.

16:10 Crafting performance alerting tools by Allison McKnight

I'm very interested in crafting alerts from RUM data.

Friday, May 29

09:00 RUM at MSN by Paul Roy

14:25 Missing Bandwidth by Bill Green

14:25 Winning Arguments with Performance Data by Buddy Brewer

17:05 All talks at this time slot

This last slot is unfortunate. Every talk at this slot is interesting and by good speakers.

Thursday, January 08, 2015

IE throws an "Invalid Calling Object" Exception for certain iframes

On a site that uses boomerang, I found a particular JavaScript error happen very often:
TypeError: Invalid calling object
This only happens on Internet Explorer, primarily IE 11, but I've seen it on versions as old as 9. I searched through stack overflow for the cause of this error, and while many of the cases sounded like they could be my problem, further investigation showed that my case didn't match any of them. The code in particular that threw the exception was collecting resource timing information for all resources on the page. Part of the algorithm involves drilling into iframes on the page, and this error showed up on one particular iframe. There are a few things to note:
   ("performance" in frame) === true;

   frame.hasOwnProperty("performance") === false;
The latter is not a surprise since hasOwnProperty("performance") is not supported for window objects on IE (I've seen this before when investigating JSLint problems.) There was no problem accessing frame.document, but accessing frame.performance threw an exception.
    frame.performance;    // <-- throws "TypeError: Invalid calling object" with error code -2147418113

    frame["performance"]; // <-- throws "TypeError: Invalid calling object" with error code -2147418113
In fact, frame.<anything except document> would throw the same exception. So I looked at the iframe's document object some more, and found this:
    frame.document.pathname === "/xxx/yyy/123/4323.pdf";
The frame was pointing to a PDF document, and while IE was creating a reference to hold the performance object of this document, it prevented any attempts to access this reference. I tested Chrome and Firefox, and they both create and populate a frame.performance object for PDF documents.