In general, these sorts of documents contain all of the super-secret (or just embarrassing) details that make up daily life in Operations. They’re almost never distributed very far — even large service providers (say, Verizon) tend to have a sanitized version they give their customers. Interestingly, however, a sanitized but pretty juicy example emerged from Amazon in response to their recent S3 outage. 

Here’s a break-down by phase. This is the “detection” phase — someone, likely someone in a Network Operations Center (since this is Sunday morning) — starts seeing big red lights. Detection is all about finding out something is wrong, and defining how serious it is and who needs to fix it.

At 8:40am PDT, error rates in all Amazon S3 datacenters began to quickly climb and our alarms went off. By 8:50am PDT, error rates were significantly elevated and very few requests were completing successfully. By 8:55am PDT, we had multiple engineers engaged and investigating the issue. Our alarms pointed at problems processing customer requests in multiple places within the system and across multiple data centers. While we began investigating several possible causes, we tried to restore system health by taking several actions to reduce system load. We reduced system load in several stages, but it had no impact on restoring system health.

At this point, it’s pretty clear that they had a major system event going on. I’d imagine cell phones or pagers (depending on how retro they are out in Seattle) were ruining Sunday morning all over Washington state. The next phase is “investigation” — basically, determining the proximate cause of the problem.

At 9:41am PDT, we determined that servers within Amazon S3 were having problems communicating with each other. As background information, Amazon S3 uses a gossip protocol to quickly spread server state information throughout the system. This allows Amazon S3 to quickly route around failed or unreachable servers, among other things. When one server connects to another as part of processing a customer’s request, it starts by gossiping about the system state. Only after gossip is completed will the server send along the information related to the customer request. On Sunday, we saw a large number of servers that were spending almost all of their time gossiping and a disproportionate amount of servers that had failed while gossiping. With a large number of servers gossiping and failing while gossiping, Amazon S3 wasn’t able to successfully process many customer requests.

I notice that the times moved from 5-minute rounding to 1-minute rounding. You get that level of detail from log analysis, and from the sort of really clever network and system monitoring technology that used to be the domain of really big players with lots of money. So, we’re an hour into a major outage and it’s likely that this has been escalated both technically (to the most senior engineers who know the system) and to the management of the business that owns the system (Amazon Web Services). 

At 10:32am PDT, after exploring several options, we determined that we needed to shut down all communication between Amazon S3 servers, shut down all components used for request processing, clear the system’s state, and then reactivate the request processing components.

Okay, almost another hour is gone and I’d imagine all the “easy” options are exhausted. Now, they’re trying more high-impact solutions. This one sounds suspiciously like “bounce the <insert process here> and see if it comes up clean,” which is one of those embarrassing-but-effective solutions you end up using when you just don’t know what else to do sometimes.

By 11:05am PDT, all server-to-server communication was stopped, request processing components shut down, and the system’s state cleared. By 2:20pm PDT, we’d restored internal communication between all Amazon S3 servers and began reactivating request processing components concurrently in both the US and EU.

At 2:57pm PDT, Amazon S3’s EU location began successfully completing customer requests. The EU location came back online before the US because there are fewer servers in the EU. By 3:10pm PDT, request rates and error rates in the EU had returned to normal. At 4:02pm PDT, Amazon S3’s US location began successfully completing customer requests, and request rates and error rates had returned to normal by 4:58pm PDT.

So, a full bounce of that subsystem took almost 4 hours to show results. You can imagine those were 4 pretty tense hours. Some companies use a conference bridge to manage big incidents, others use web chats or VoIP systems. I’m sure a bunch of people were all working very hard to move this along quickly, and it still took quite a while, but you can almost imagine the relief that flooded the whole team at 2:57pm when EU came back up. By around 5pm, the whole system was back up and normal, and there wasn’t much left to do except the paperwork.

Which brings us to the last part of their message:

We’ve now determined that message corruption was the cause of the server-to-server communication problems. More specifically, we found that there were a handful of messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect. We use MD5 checksums throughout the system, for example, to prevent, detect, and recover from corruption that can occur during receipt, storage, and retrieval of customers’ objects. However, we didn’t have the same protection in place to detect whether this particular internal state information had been corrupted. As a result, when the corruption occurred, we didn’t detect it and it spread throughout the system causing the symptoms described above. We hadn’t encountered server-to-server communication issues of this scale before and, as a result, it took some time during the event to diagnose and recover from it.

You can be sure that as soon as they were on a path to get the system stable, they were investigating how it got unstable in the first place. This is a pretty in-depth problem statement, and it’s impressive how transparent Amazon is about it. I doubt I’d ever be allowed to report to the public on any of my outages like this alone, and I suspect Amazon is no different: at some point during that investigation, Amazon’s PR people started working on wording for public announcements. At the same time, technical teams were working to figure out how to make sure it never happens again.

During our post-mortem analysis we’ve spent quite a bit of time evaluating what happened, how quickly we were able to respond and recover, and what we could do to prevent other unusual circumstances like this from having system-wide impacts. Here are the actions that we’re taking: (a) we’ve deployed several changes to Amazon S3 that significantly reduce the amount of time required to completely restore system-wide state and restart customer request processing; (b) we’ve deployed a change to how Amazon S3 gossips about failed servers that reduces the amount of gossip and helps prevent the behavior we experienced on Sunday; (c) we’ve added additional monitoring and alarming of gossip rates and failures; and, (d) we’re adding checksums to proactively detect corruption of system state messages so we can log any such messages and then reject them.

This is fascinating. Certainly, (a) is an obvious concern — I know I’d be pulling out my hair if a key system I managed took 4 hours to restart. I’d speculate that this was simply a contingency nobody had thought about yet, and thus probably was a pretty painfully manual process. Amazon (and all the large operators) rely on quite a bit of automation to manage large fleets of servers, but when the unexpected happens, it doesn’t always work as planned. Item (b) looks like it’s addressing the root cause, item (d) may also play into this. Item (c) is that perennial favorite of ops: monitoring and alarming. These are all just the sort of things I’d expect to see if I were in that situation.

In summary, very interesting data about an increasingly important service on the internet. At the same time, a rare view into what goes on in the dark where SysAdmins and Network Engineers roam. Incidentally, there are technical details about the system that failed, which is a technology Amazon calls Dynamo.

DFT operates quite a few facilities in Northern Virginia. Most familiar to me would be the former AOL Gainesville Technology Center, sold to them in 2005. While this facility was big, the real capacity of a data center is less space (“raised floor square feet”) but power (“megawatts of critical load”). DFT has subsequently acquired several other sites, and will open a new facility roughly twice the size and with four times the power of the former AOL site, with several more in their pipeline.

That’s quite a bit of  new hosting capacity, and a clear sign that the large facility shortage may be over. To put it in perspective, the Lenior, NC or Quincy, WA sites mentioned in my earlier post as “really big” are in the same ballpark, size-wise, as one of their new centers. And, while we don’t really know that Google or Microsoft has filled out their space, DFT brags that all their earlier sites are leased with terms averaging 8 years. Their still-under-construction ACC4 site was 43.8% pre-leased in August.

So, who’s using all that space? Their IPO tells a bit of these normally secretive details. Facebook is making a big new expansion, leasing 10,000 square feet of space. Most of the rest of the space goes to a few major users, according to their registration statement. The same statement indicates that their undeveloped property represents 187MW of additional capacity. That said, they probably want to acquire some new customers, because “As of August 1, 2007, our two largest tentants, Microsoft and Yahoo!, accounted for 86.0% of our annualized rent”.

This is a distinctly different business model than that of either the major network providers (Level3 or Verizon), or the major Carrier Neutral providers (Equinix or Switch & Data). They consider themselves wholesalers to major providers, rather than a retail provider, because they allow the sort of large capacity blocks to be purchased that otherwise can’t be gotten without building it yourself.

So, will they ultimately suffer the fate of Exodus or Genuity?  They do seem to be in the right geographic locations (NoVA, Chicago, Santa Clara). They seem to have the right sizes and scale (new facilities over 30MW), with individual units of capacity in the 2-4MW range, for their target market. I’m not sure there are 10 more Microsoft, Yahoo, or Googles out there looking for space, but if they’re opening up their doors to Facebook at reasonably pricing, perhaps there’s enough large scale demand left. With Amazon chasing the small end, and DFT chasing the high end, this may make a very interesting business outlook for the ever-turbulent retail data center business.

So, 365 Main screwed up big time? Not really. Their design was horribly flawed? Not so much. They have eight rooms full of servers, and ten generators — enough for every room to have one, plus two extra generators for contingencies. This type of UPS is a Diesel Rotary UPS: utility power makes a flywheel spin, the flywheel runs a generator, the generator supplies power to the computers. When utility power goes away, there’s a brief (to humans) pause while the diesel engine starts up make power for the computers. As long as the diesel spins up before the flywheel spins down, power keeps flowing. Proponents of this design like to emphasize how it’s simple, and thus pretty reliable, and in its defense, no part of the Rotary system seemed to fail in this case. For completeness, the other kind of UPS uses lots and lots of a batteries.

What did fail was the diesel engine’s controller. What makes the electricity is an enormous diesel engine, so naturally there’s quite a bit of support equipment to keep it running that had to be checked. During the investigation, issues with other systems (exhaust) were uncovered. Ultimately, though, they found a flaw that could be reproduced in a critical part of a system that should never fail. Yet, it did fail in 30% of the cases, and that was enough to bring down all sorts of different products, not to mention causing a highly public issue for 365 Main.

I’m not affiliated in any way with 365 Main, but I do have to say I’m impressed with how they have handled the aftermath of the incident. They were open and honest about the incident, provided lots of public information about their investigation, even to non-customers like me. Heck, they even ignored the idiotic rumors that Valleywag was tricked into posting as “news.” Nicely done, guys.

So, what makes companies like Google, Yahoo, Microsoft, and AOL build their own data center facilities, when the vast majority of companies end up leasing space from carrier-neutral colocation providers like Equinix, or telecommunications providers like Verizon Business or Level3? The tongue in cheek answer would be ‘because they can’, but building a data center facility is as much about control as it is about anything else. Being able to control key elements such as physical security, power and network access, space assignment policy, and general access to the space makes it compelling to own and operate a space. In addition, in recent years, the market for large spaces in the leasing market has dried up, adding both an availability benefit (when you build 250,000 square feet of space, you know it’s there for you) and a cost benefit (no competing with Google for the last big cage in a facility). Of course, owning a space locks in your cost basis in a way which leasing doesn’t, but for a business on the grow, there’s not really a question of whether the space will be used or not.

And yet, the Google, Microsoft, Yahoo, and AOL’s of the world don’t exclusively use owned space to host their servers: at least some of their footprint remains in leased space. Part of this is necessary in order to build out network connectivity in a desired way. You might not be able to get every peer you want to follow you to the middle-of-nowhere, but you can certainly get to them in San Jose or Washington D.C. There are also certain situations where being present in a specific place is the most important consideration for technical, legal, or contractual reasons.

So, what drives the massive centers out to the boonies? It certainly isn’t a proliferation of talent in those areas. In the past, land prices were a primary consideration, but in the last two years, the single most important factor seems to be power. Let’s take a look at a little something from the Department of Energy about Residential Electricity Prices, understanding that commercial/industrial pricing follows the same trend, but is likely a bit lower. The following map illustrates average prices in centers per kWh (caveat, this data is 4 years old. Here’s an ugly but new table of energy prices by State):

You’ll see some interesting facts — the states with the largest populations also have some of the highest energy costs. Compare 5.81 cents per kWh in Kentucky against New York’s pricely 14.31 cents, and you’ll see that there’s a tremendous incentive to locate in the lower cost states: Washington, Idaho, North Dakota, Nebraska, Tennessee, Kentucky, West Virginia. Of course, these are averages — most of those big high profile deals involved low-cost energy. The Microsoft deal, for example, was rumored to include sub-2 cent power. Add to this that several of these states receive power from hydroelectric (Bonneville Power Administration, or Tennessee Valley Authority) which have highly fixed costs, rather than burning a fuel whose cost could increase markedly in the future.

Cheap power is great, but without fiber to connect into, a data center is just a gigantic resistance heater. There are certainly plenty of public networks out there, but the larger providers seem to get especially interested in so-called Dark Fiber. A well-documented story about Google discussed their appetite for Dark Fiber and expected they’d start their own ISP. What’s much more likely is they just wanted to build their own backbone to connect up their numerous facilities. Dark fiber are individual fibers within an already laid cable that haven’t been connected or used yet. While the market has tightened with the consolidation to a few providers (okay, Level3 bought everybody: Wiltel, Progress, ICG, Telcove, Looking Glass, and Broadwing), major providers still can get access to the raw material they need to build their own network, using technologies such as Dense Wavelength Division Multiplexing to cram breathtaking amounts of traffic onto a single fiber. Trace the lines on, say, the Level3 network map and you can plot locations for almost all of those new gigantic datacenters. There’s no coincidence that Montana or North Dakota have no fiber and hence no data centers.

Next time: so, what’s actually in one of those gigantic Google data centers.

There’s a clear analogy to the music industry here, in my mind. News consumers have a choice: paying for a product or getting it for free. Unlike in the case of Napster, however, those reading articles on the web aren’t stealing them — they’re just ‘paying’ with ad-support. So, the ad support isn’t generating enough revenue to pay for the content, and this is adversely hurting the legacy business.

My first and foremost sentiment is pretty blunt: too bad. Newspapers used to have a monopoly on the news, more or less. Then there was radio — some suggested the end of print. But of course, it didn’t happen. A few decades later, television was forecast to be the end of the written word. More recently, 24-hour news networks and the continual news cycle are the raven of the broadsheet. Now, it’s internet readership and citizen journalism.

I’m of the opinion that the internet is a more potent foe because of the combination of two elements: immediacy (e.g., the news is there, right away) and ubiquity (an awful lot of people have access to it via one form or another). It’d be easy to believe that all those people who used to pay for newspapers are simply reading online — I know it’s happened to me.

I think, however, that this is too easy an explanation: circulation is down much more than page views are up. Also, if reader who defect are as high-value as some suggest, why aren’t their page views more valuable from an advertising perspective online.

Whatever the cause of the declines, it’s clear to me that a drastic action like this won’t drive users back to their subscriptions (online or off). Newspapers are far from the only reliable news source of record: the 24-hour news networks are well-established, entrenched nationwide, and have earned a respectable modicum of trust.

Taking away articles from AOL, Yahoo, or Google will simply result in licensing deals being struct with CNN, MSNBC, and Fox. And, in my mind, that’s probably what will happen anyway. The amount of energy with which newspapers cover local news has declined substantially, classifieds no longer hold a candle to online advertising, and the appeal of anytime information have combined to wean readers away from the morning news. Cutting them off will only accelerate a transition to 24-hour news sources that’s been happening for 20 years.

So, how should a newspaper respond? For the slew of papers who package a thin sliver of journalism around a thick package of wire stories and advertising, there may be no home. Major papers of record probably stand the best chance since they have a brand which resonates nationwide. To cover their costs, they will need to tighten their belts, improve their advertising model, and rethink partnerships. The biggest question in my mind is whether groups like the AP are a good idea online — they seem to be enabling competition in ways which probably aren’t profitable for the papers which own the AP.

And Citizen Journalism, while exotic and new, probably won’t change the world all that much. Although it is great for scandal…