With most technology, it’s a given that there’s almost always More Than One Way To Do It (unless you worship Python). There are always those situations where choices must be made, and different people use different yardsticks to decide. Some try to minimize “cost,” either up-front development cost or long-term engineering cost. The smarter ones have recognized the concept of “Technology Debt” as addressed by several observers. As a leader in Operations, however, I tend to subscribe to my own rule: the 4 a.m. rule.

Simply put, the 4 a.m. rule is this: 

Never adopt any solution which you couldn’t understand immediately upon being awoken to fix it at 4 a.m. 

There’s a very simple reason to adhere to this rule whenever possible — as I’ve previously mentioned, things fall apart. Systems all break: complex ones and simple ones alike. Sooner or later, people need to fix them and the more byzantine the operation of the system, the harder it will be. 

The simplest way possible to survive the 4 a.m. test is to only build very simple systems. A totally simple system is sometimes just the ticket to solve the problem, and where it is adequate, go with it. Interesting problems occasionally have extremely elegant solutions, and making them more complex is just bad mojo. 

Still, you’ll much more often find a place where more complexity is necessary to achieve your desired goal. In these circumstances, it can be tricky to pass the 4 a.m. test. This is where two strategies are necessary: documentation and transparency.

Documentation deserves a whole separate discussion, but the part that’s important at 4 a.m. is a complete lack of subtlety: 

• Recovery instructions: you’ll have bleary eyes, so these must be as simple as “if this, do that”
• Architecture diagrams: simple pictures with bright colors and clearly labeled lines detailing what talks to what and why. And don’t make me load Visio at 4 a.m. ever. 
• If it’s needed, and can fail, it should be mentioned, but in as few words as necessary. This is not the time for flowery prose.

Transparency is quite a bit harder. This is about exposing as much as possible to someone observing the system. A few places are crucial:

• Error messages: For the love of god people, make sure every message requires absolutely no prior knowledge and is clear and unambiguous even out of context. 
• Simple dependencies: Nothing is harder to discover than extremely complex webs of services. If you ever see an design with recursive dependency, run like heck.
• Change logging: the first question you should ask when something is broken is “what changed.” Keep a record of even the boring stuff - you never know when it’ll save your bacon.

Remember as a cardinal rule: 

complexity is a vice: use it sparingly and explain it simply enough for 4 a.m.

Just the other day, I stumbled on Yet Another Startup CDN Provider, SimpleCDN. The CDN business has become very crowded as facilities-based carriers (AT&T and Level), start-ups (Limelight, Peer1, Panther Express), and the established leader (Akamai) engaged in a free-for-all over price. SimpleCDN’s value proposition is straightforward: pay a one-time up-front fee based on object size, instead of monthly based on usage. The big question I’m inclined to want answered is: how could this possibly work as a business?

This is a pretty unusual concept, so I decided to check it out myself. They let anyone sign up on their portal and start delivering without so much as a sales call. I wanted to ask some questions, so I went looking for real-world contact information. 

Here’s where the first problems became obvious: 

- The only address they provide (the Highland Park, IL one) appears to be a Mailboxes Etc location. 

- They also list some “NOC” mailing addresses on their contact pages. If they seem familiar, it’s because they’re major carrier neutral data centers. Perhaps they host something at these locations, but they sure don’t have people there. 

- There was, however, a 24/7 support address. So, I sent them some email.

I received an almost-immediate response (on a Sunday night) from Frank Wilson, who described himself as “Chief Engineer of SimpleCDN, and also a co-founder.” He offered to answer any questions, so I asked:

I’m assuming your data center bills you for power and space every month, and I’m sure you’re paying someone for transit monthly, so how exactly does this work over the long run? On the surface, it seems like you’re banking on the demand for content dropping dramatically over time, and the marginal cost to store it approaching zero. 

Assuming your cost for delivery is the 7 cents per GB transferred (which implies Cogent-level transit costs). At the 1MB = 1 credit = ~14GB xfer’d price point you’ve set, you’re collecting that one-time fee sufficiently to cover 14000 times while at the 1GB = 35 credits = 500GB which would only cover around 500 downloads. 

Days later, with no answer to my question, I decided to do a bit of poking around. Here are a few things I discovered:

1) The entire SimpleCDN operation seems to be hosted in a single Chicago location. Looking at looking glass servers from Europe, Asia, South America, and Australia reveals the exact same set of servers in Chicago serving everything. They own AS46177, but don’t seem to be using it.

2) All of their routes come via a self-declared ISP called Resisoft (AS32205). Don’t bother trying to click that link — they don’t seem to be a working web site. It isn’t clear if they’re troubled, or just host troublesome companies. It seems like most of their transit is Hurricane Electric, which isn’t exactly premium bandwidth. 

3) Frank Wilson seems to be the only visible employee of SimpleCDN–he answers their support email, manages their network, and markets their services. Yes, you heard that right: from sites from the technical web-hosting to stock boards to commenting on industry sites he pops up, promoting SimpleCDN. Sometimes he identifies as an employee, other times he talks about SimpleCDN in the third person, as if he weren’t affiliated. I find this very shifty.

Here’s what I think is happening: SimpleCDN is using current revenue from those initial “one-time” payments to cover their current bandwidth and hosting bills. As long as usage remains reasonable, and they keep growing their customer base, they can probably keep up with their costs for a little while. 

Except, this just isn’t a very good business model. Most of these customers will pay a few dollars a few times a year. The transaction charges eat alive a small company — even Amazon’s EC2 products aren’t really designed around someone using 5 instance-hours a month. And, all it takes is one really big download (Firefox 4, maybe?) to call their bluff.

So, worst case, they’re some sort of scam that will collect a bunch of pre-paid hosting charges and disappear. Best case, they have a naive business model which will cause their collapse if they get even moderately popular. I suppose I can understand why someone currently using S3 as a CDN (seriously, guys, why?) might consider this an appealing alternative, but it’s pretty hard to imagine that Frank’s business will be here in 2 years. 

Oh, and to Frank: I’d love to see a business plan that proves me wrong. I’m always interested in seeing legitimate competitors in the space, so show the world some evidence that SimpleCDN is more than just a great big promise that can’t go the distance!

In general, these sorts of documents contain all of the super-secret (or just embarrassing) details that make up daily life in Operations. They’re almost never distributed very far — even large service providers (say, Verizon) tend to have a sanitized version they give their customers. Interestingly, however, a sanitized but pretty juicy example emerged from Amazon in response to their recent S3 outage. 

Here’s a break-down by phase. This is the “detection” phase — someone, likely someone in a Network Operations Center (since this is Sunday morning) — starts seeing big red lights. Detection is all about finding out something is wrong, and defining how serious it is and who needs to fix it.

At 8:40am PDT, error rates in all Amazon S3 datacenters began to quickly climb and our alarms went off. By 8:50am PDT, error rates were significantly elevated and very few requests were completing successfully. By 8:55am PDT, we had multiple engineers engaged and investigating the issue. Our alarms pointed at problems processing customer requests in multiple places within the system and across multiple data centers. While we began investigating several possible causes, we tried to restore system health by taking several actions to reduce system load. We reduced system load in several stages, but it had no impact on restoring system health.

At this point, it’s pretty clear that they had a major system event going on. I’d imagine cell phones or pagers (depending on how retro they are out in Seattle) were ruining Sunday morning all over Washington state. The next phase is “investigation” — basically, determining the proximate cause of the problem.

At 9:41am PDT, we determined that servers within Amazon S3 were having problems communicating with each other. As background information, Amazon S3 uses a gossip protocol to quickly spread server state information throughout the system. This allows Amazon S3 to quickly route around failed or unreachable servers, among other things. When one server connects to another as part of processing a customer’s request, it starts by gossiping about the system state. Only after gossip is completed will the server send along the information related to the customer request. On Sunday, we saw a large number of servers that were spending almost all of their time gossiping and a disproportionate amount of servers that had failed while gossiping. With a large number of servers gossiping and failing while gossiping, Amazon S3 wasn’t able to successfully process many customer requests.

I notice that the times moved from 5-minute rounding to 1-minute rounding. You get that level of detail from log analysis, and from the sort of really clever network and system monitoring technology that used to be the domain of really big players with lots of money. So, we’re an hour into a major outage and it’s likely that this has been escalated both technically (to the most senior engineers who know the system) and to the management of the business that owns the system (Amazon Web Services). 

At 10:32am PDT, after exploring several options, we determined that we needed to shut down all communication between Amazon S3 servers, shut down all components used for request processing, clear the system’s state, and then reactivate the request processing components.

Okay, almost another hour is gone and I’d imagine all the “easy” options are exhausted. Now, they’re trying more high-impact solutions. This one sounds suspiciously like “bounce the <insert process here> and see if it comes up clean,” which is one of those embarrassing-but-effective solutions you end up using when you just don’t know what else to do sometimes.

By 11:05am PDT, all server-to-server communication was stopped, request processing components shut down, and the system’s state cleared. By 2:20pm PDT, we’d restored internal communication between all Amazon S3 servers and began reactivating request processing components concurrently in both the US and EU.

At 2:57pm PDT, Amazon S3’s EU location began successfully completing customer requests. The EU location came back online before the US because there are fewer servers in the EU. By 3:10pm PDT, request rates and error rates in the EU had returned to normal. At 4:02pm PDT, Amazon S3’s US location began successfully completing customer requests, and request rates and error rates had returned to normal by 4:58pm PDT.

So, a full bounce of that subsystem took almost 4 hours to show results. You can imagine those were 4 pretty tense hours. Some companies use a conference bridge to manage big incidents, others use web chats or VoIP systems. I’m sure a bunch of people were all working very hard to move this along quickly, and it still took quite a while, but you can almost imagine the relief that flooded the whole team at 2:57pm when EU came back up. By around 5pm, the whole system was back up and normal, and there wasn’t much left to do except the paperwork.

Which brings us to the last part of their message:

We’ve now determined that message corruption was the cause of the server-to-server communication problems. More specifically, we found that there were a handful of messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect. We use MD5 checksums throughout the system, for example, to prevent, detect, and recover from corruption that can occur during receipt, storage, and retrieval of customers’ objects. However, we didn’t have the same protection in place to detect whether this particular internal state information had been corrupted. As a result, when the corruption occurred, we didn’t detect it and it spread throughout the system causing the symptoms described above. We hadn’t encountered server-to-server communication issues of this scale before and, as a result, it took some time during the event to diagnose and recover from it.

You can be sure that as soon as they were on a path to get the system stable, they were investigating how it got unstable in the first place. This is a pretty in-depth problem statement, and it’s impressive how transparent Amazon is about it. I doubt I’d ever be allowed to report to the public on any of my outages like this alone, and I suspect Amazon is no different: at some point during that investigation, Amazon’s PR people started working on wording for public announcements. At the same time, technical teams were working to figure out how to make sure it never happens again.

During our post-mortem analysis we’ve spent quite a bit of time evaluating what happened, how quickly we were able to respond and recover, and what we could do to prevent other unusual circumstances like this from having system-wide impacts. Here are the actions that we’re taking: (a) we’ve deployed several changes to Amazon S3 that significantly reduce the amount of time required to completely restore system-wide state and restart customer request processing; (b) we’ve deployed a change to how Amazon S3 gossips about failed servers that reduces the amount of gossip and helps prevent the behavior we experienced on Sunday; (c) we’ve added additional monitoring and alarming of gossip rates and failures; and, (d) we’re adding checksums to proactively detect corruption of system state messages so we can log any such messages and then reject them.

This is fascinating. Certainly, (a) is an obvious concern — I know I’d be pulling out my hair if a key system I managed took 4 hours to restart. I’d speculate that this was simply a contingency nobody had thought about yet, and thus probably was a pretty painfully manual process. Amazon (and all the large operators) rely on quite a bit of automation to manage large fleets of servers, but when the unexpected happens, it doesn’t always work as planned. Item (b) looks like it’s addressing the root cause, item (d) may also play into this. Item (c) is that perennial favorite of ops: monitoring and alarming. These are all just the sort of things I’d expect to see if I were in that situation.

In summary, very interesting data about an increasingly important service on the internet. At the same time, a rare view into what goes on in the dark where SysAdmins and Network Engineers roam. Incidentally, there are technical details about the system that failed, which is a technology Amazon calls Dynamo.

The second event I’ll be at will be GigaOM’s Structure 08. Cloud computing is really leveling the playing field, giving small start-ups access to world-class operational assets… which to me only underscore the importance of having brilliant Ops folks to run those systems. I’m eager to see what sort of discussions emerge.

If you happen to be at either, give me a buzz in the comments, and I’ll try and catch up with you. 

DFT operates quite a few facilities in Northern Virginia. Most familiar to me would be the former AOL Gainesville Technology Center, sold to them in 2005. While this facility was big, the real capacity of a data center is less space (”raised floor square feet”) but power (”megawatts of critical load”). DFT has subsequently acquired several other sites, and will open a new facility roughly twice the size and with four times the power of the former AOL site, with several more in their pipeline.

That’s quite a bit of  new hosting capacity, and a clear sign that the large facility shortage may be over. To put it in perspective, the Lenior, NC or Quincy, WA sites mentioned in my earlier post as “really big” are in the same ballpark, size-wise, as one of their new centers. And, while we don’t really know that Google or Microsoft has filled out their space, DFT brags that all their earlier sites are leased with terms averaging 8 years. Their still-under-construction ACC4 site was 43.8% pre-leased in August.

So, who’s using all that space? Their IPO tells a bit of these normally secretive details. Facebook is making a big new expansion, leasing 10,000 square feet of space. Most of the rest of the space goes to a few major users, according to their registration statement. The same statement indicates that their undeveloped property represents 187MW of additional capacity. That said, they probably want to acquire some new customers, because “As of August 1, 2007, our two largest tentants, Microsoft and Yahoo!, accounted for 86.0% of our annualized rent”.

This is a distinctly different business model than that of either the major network providers (Level3 or Verizon), or the major Carrier Neutral providers (Equinix or Switch & Data). They consider themselves wholesalers to major providers, rather than a retail provider, because they allow the sort of large capacity blocks to be purchased that otherwise can’t be gotten without building it yourself.

So, will they ultimately suffer the fate of Exodus or Genuity?  They do seem to be in the right geographic locations (NoVA, Chicago, Santa Clara). They seem to have the right sizes and scale (new facilities over 30MW), with individual units of capacity in the 2-4MW range, for their target market. I’m not sure there are 10 more Microsoft, Yahoo, or Googles out there looking for space, but if they’re opening up their doors to Facebook at reasonably pricing, perhaps there’s enough large scale demand left. With Amazon chasing the small end, and DFT chasing the high end, this may make a very interesting business outlook for the ever-turbulent retail data center business.

If there’s any one thing that’s been disappointing about this time around, it’s been the continual stream of nastiness by people who claim to be current or former AOL employees in public forums and online rumors sites. Isn’t it bad enough that we’re going through this — do we need to rip on each other as well? Difficult situations produce different responses from people, and I understand the desire to vent, I just wish it were less self-destructive.

And that’s all I have to say about the topic. I’ll be back to my regularly-scheduled … uhm … “posting schedule” when the grumpy wears off.

So, 365 Main screwed up big time? Not really. Their design was horribly flawed? Not so much. They have eight rooms full of servers, and ten generators — enough for every room to have one, plus two extra generators for contingencies. This type of UPS is a Diesel Rotary UPS: utility power makes a flywheel spin, the flywheel runs a generator, the generator supplies power to the computers. When utility power goes away, there’s a brief (to humans) pause while the diesel engine starts up make power for the computers. As long as the diesel spins up before the flywheel spins down, power keeps flowing. Proponents of this design like to emphasize how it’s simple, and thus pretty reliable, and in its defense, no part of the Rotary system seemed to fail in this case. For completeness, the other kind of UPS uses lots and lots of a batteries.

What did fail was the diesel engine’s controller. What makes the electricity is an enormous diesel engine, so naturally there’s quite a bit of support equipment to keep it running that had to be checked. During the investigation, issues with other systems (exhaust) were uncovered. Ultimately, though, they found a flaw that could be reproduced in a critical part of a system that should never fail. Yet, it did fail in 30% of the cases, and that was enough to bring down all sorts of different products, not to mention causing a highly public issue for 365 Main.

I’m not affiliated in any way with 365 Main, but I do have to say I’m impressed with how they have handled the aftermath of the incident. They were open and honest about the incident, provided lots of public information about their investigation, even to non-customers like me. Heck, they even ignored the idiotic rumors that Valleywag was tricked into posting as “news.” Nicely done, guys.

So, what makes companies like Google, Yahoo, Microsoft, and AOL build their own data center facilities, when the vast majority of companies end up leasing space from carrier-neutral colocation providers like Equinix, or telecommunications providers like Verizon Business or Level3? The tongue in cheek answer would be ‘because they can’, but building a data center facility is as much about control as it is about anything else. Being able to control key elements such as physical security, power and network access, space assignment policy, and general access to the space makes it compelling to own and operate a space. In addition, in recent years, the market for large spaces in the leasing market has dried up, adding both an availability benefit (when you build 250,000 square feet of space, you know it’s there for you) and a cost benefit (no competing with Google for the last big cage in a facility). Of course, owning a space locks in your cost basis in a way which leasing doesn’t, but for a business on the grow, there’s not really a question of whether the space will be used or not.

And yet, the Google, Microsoft, Yahoo, and AOL’s of the world don’t exclusively use owned space to host their servers: at least some of their footprint remains in leased space. Part of this is necessary in order to build out network connectivity in a desired way. You might not be able to get every peer you want to follow you to the middle-of-nowhere, but you can certainly get to them in San Jose or Washington D.C. There are also certain situations where being present in a specific place is the most important consideration for technical, legal, or contractual reasons.

So, what drives the massive centers out to the boonies? It certainly isn’t a proliferation of talent in those areas. In the past, land prices were a primary consideration, but in the last two years, the single most important factor seems to be power. Let’s take a look at a little something from the Department of Energy about Residential Electricity Prices, understanding that commercial/industrial pricing follows the same trend, but is likely a bit lower. The following map illustrates average prices in centers per kWh (caveat, this data is 4 years old. Here’s an ugly but new table of energy prices by State):

You’ll see some interesting facts — the states with the largest populations also have some of the highest energy costs. Compare 5.81 cents per kWh in Kentucky against New York’s pricely 14.31 cents, and you’ll see that there’s a tremendous incentive to locate in the lower cost states: Washington, Idaho, North Dakota, Nebraska, Tennessee, Kentucky, West Virginia. Of course, these are averages — most of those big high profile deals involved low-cost energy. The Microsoft deal, for example, was rumored to include sub-2 cent power. Add to this that several of these states receive power from hydroelectric (Bonneville Power Administration, or Tennessee Valley Authority) which have highly fixed costs, rather than burning a fuel whose cost could increase markedly in the future.

Cheap power is great, but without fiber to connect into, a data center is just a gigantic resistance heater. There are certainly plenty of public networks out there, but the larger providers seem to get especially interested in so-called Dark Fiber. A well-documented story about Google discussed their appetite for Dark Fiber and expected they’d start their own ISP. What’s much more likely is they just wanted to build their own backbone to connect up their numerous facilities. Dark fiber are individual fibers within an already laid cable that haven’t been connected or used yet. While the market has tightened with the consolidation to a few providers (okay, Level3 bought everybody: Wiltel, Progress, ICG, Telcove, Looking Glass, and Broadwing), major providers still can get access to the raw material they need to build their own network, using technologies such as Dense Wavelength Division Multiplexing to cram breathtaking amounts of traffic onto a single fiber. Trace the lines on, say, the Level3 network map and you can plot locations for almost all of those new gigantic datacenters. There’s no coincidence that Montana or North Dakota have no fiber and hence no data centers.

Next time: so, what’s actually in one of those gigantic Google data centers.