In the spirit of the recently released movie adaptation of the popular book The Hunger Games, I have a suggestion for how Google can structure registration next year.

There will be an open registration period (perhaps a week) where developers can put their names into a drawing. If Google wishes to provide multiple entries for those who have previously attended, have contributed to projects, or are otherwise important to them, they can do so.

At a given date and time, Google will conduct the Google IO Reaping. A certain number of names will be drawn and receive email notification that they have an opportunity to purchase a ticket–I’d suggest they have 24-48 hours, so there’s no rush. Everyone who receive an invitation from the reaping gets a ticket if they pay for one.

If seats don’t get sold as tickets, they go back into the pool and re-reaped. The goal is that only the people who signed up for the reaping can be named as Google Tributes. Of course, people sometimes will need to cancel, and we’ll put those back into the pool to be re-reaped, and if someone else takes the seat the original purchaser gets a refund.

This process avoids a huge sign-up rush. It also gives everyone in every time zone an equal chance to make it in. And, finally, it restores an opportunity for Google to give previous participants an edge while keeping things fair.

What do you think, Google? May the odds be EVER in IO’s favor?

The subject was about the convergence between CDN services and Cloud Computing services. Enjoy!

No effort of this nature happens as an individual effort, and this was no exception. Given the challenging circumstances at AOL in 2010, it was especially exciting to see people band together to work on such an exciting project. I’m glad we were able to embrace some of the best of current technology at AOL, and make it ours. I’m looking forward to seeing our participation in World IPv6 Day as well.

Go AOL!

Contact: Carl.Coppadge at corp.aol.com or AIM: carlcoppadge11

Location: Dulles, VA

MySQL Database Administrator

AOL’s People Networks Operations, which operates AIM and ICQ, has an opening for a Sr. MySQL Database Administrator. This position would be responsible for managing large, highly scaled production MySQL databases as well as pre-production (QA/Dev/Staging) environments.

Key Responsibilities:

• All aspects of MySQL deployment, operation,and design to ensure high reliability and performance
• Collaborate with developers and architects to create high-performance, cost-effective designs
• Participate in an on-call rotation as part of a 24×7 operations team to resolve urgent production issues
• Ensuring data integrity with good process and a keen eye for detecting errors and misuses of data

  Desired Skills for this Position:

• Knowledge of database architecture concepts as well as MySQL-specific implementations
• Significant experience with MySQL in a high-volume production environment
• Experience with open source ETL packages and methods for large scale data migration
• Diverse technical background with awareness of concepts in networking, Linux, and storage
• Bachelor-level degree in Engineering, Computing, or Sciences orequivalent experience
• Replication: managing replication delay, scaling replication throughput, and designing resilient systems
• Configuration: tuning InnoDB, managing memory usage, and tuning file systems for maximum throughput
• Reliability: backups under load, failover strategies, and recovery from replication issues
• MySQL: familiarity with Percona and Google patches

  Our perfect candidate can manage many rapidly changing projects while maintaining professionalism and poise. Our environment is both highly exciting and highly demanding — those who thrive here adopt a “work smarter, not harder” attitude.

There’s much that can be said about how to do this right, but a few tips I use myself in these situations:

• Timely: tell your customers what you’re certain of as early as reasonable
• Unbiased: don’t speculate or editorialize, just the facts
• Clear: avoid using unnecessary jargon or technical terms
• Regular: keep communicating until the issue is over

With most technology, it’s a given that there’s almost always More Than One Way To Do It (unless you worship Python). There are always those situations where choices must be made, and different people use different yardsticks to decide. Some try to minimize “cost,” either up-front development cost or long-term engineering cost. The smarter ones have recognized the concept of “Technology Debt” as addressed by several observers. As a leader in Operations, however, I tend to subscribe to my own rule: the 4 a.m. rule.

Simply put, the 4 a.m. rule is this: 

Never adopt any solution which you couldn’t understand immediately upon being awoken to fix it at 4 a.m. 

There’s a very simple reason to adhere to this rule whenever possible — as I’ve previously mentioned, things fall apart. Systems all break: complex ones and simple ones alike. Sooner or later, people need to fix them and the more byzantine the operation of the system, the harder it will be. 

The simplest way possible to survive the 4 a.m. test is to only build very simple systems. A totally simple system is sometimes just the ticket to solve the problem, and where it is adequate, go with it. Interesting problems occasionally have extremely elegant solutions, and making them more complex is just bad mojo. 

Still, you’ll much more often find a place where more complexity is necessary to achieve your desired goal. In these circumstances, it can be tricky to pass the 4 a.m. test. This is where two strategies are necessary: documentation and transparency.

Documentation deserves a whole separate discussion, but the part that’s important at 4 a.m. is a complete lack of subtlety: 

• Recovery instructions: you’ll have bleary eyes, so these must be as simple as “if this, do that”
• Architecture diagrams: simple pictures with bright colors and clearly labeled lines detailing what talks to what and why. And don’t make me load Visio at 4 a.m. ever. 
• If it’s needed, and can fail, it should be mentioned, but in as few words as necessary. This is not the time for flowery prose.

Transparency is quite a bit harder. This is about exposing as much as possible to someone observing the system. A few places are crucial:

• Error messages: For the love of god people, make sure every message requires absolutely no prior knowledge and is clear and unambiguous even out of context. 
• Simple dependencies: Nothing is harder to discover than extremely complex webs of services. If you ever see an design with recursive dependency, run like heck.
• Change logging: the first question you should ask when something is broken is “what changed.” Keep a record of even the boring stuff – you never know when it’ll save your bacon.

Remember as a cardinal rule: 

complexity is a vice: use it sparingly and explain it simply enough for 4 a.m.

In general, these sorts of documents contain all of the super-secret (or just embarrassing) details that make up daily life in Operations. They’re almost never distributed very far — even large service providers (say, Verizon) tend to have a sanitized version they give their customers. Interestingly, however, a sanitized but pretty juicy example emerged from Amazon in response to their recent S3 outage. 

Here’s a break-down by phase. This is the “detection” phase — someone, likely someone in a Network Operations Center (since this is Sunday morning) — starts seeing big red lights. Detection is all about finding out something is wrong, and defining how serious it is and who needs to fix it.

At 8:40am PDT, error rates in all Amazon S3 datacenters began to quickly climb and our alarms went off. By 8:50am PDT, error rates were significantly elevated and very few requests were completing successfully. By 8:55am PDT, we had multiple engineers engaged and investigating the issue. Our alarms pointed at problems processing customer requests in multiple places within the system and across multiple data centers. While we began investigating several possible causes, we tried to restore system health by taking several actions to reduce system load. We reduced system load in several stages, but it had no impact on restoring system health.

At this point, it’s pretty clear that they had a major system event going on. I’d imagine cell phones or pagers (depending on how retro they are out in Seattle) were ruining Sunday morning all over Washington state. The next phase is “investigation” — basically, determining the proximate cause of the problem.

At 9:41am PDT, we determined that servers within Amazon S3 were having problems communicating with each other. As background information, Amazon S3 uses a gossip protocol to quickly spread server state information throughout the system. This allows Amazon S3 to quickly route around failed or unreachable servers, among other things. When one server connects to another as part of processing a customer’s request, it starts by gossiping about the system state. Only after gossip is completed will the server send along the information related to the customer request. On Sunday, we saw a large number of servers that were spending almost all of their time gossiping and a disproportionate amount of servers that had failed while gossiping. With a large number of servers gossiping and failing while gossiping, Amazon S3 wasn’t able to successfully process many customer requests.

I notice that the times moved from 5-minute rounding to 1-minute rounding. You get that level of detail from log analysis, and from the sort of really clever network and system monitoring technology that used to be the domain of really big players with lots of money. So, we’re an hour into a major outage and it’s likely that this has been escalated both technically (to the most senior engineers who know the system) and to the management of the business that owns the system (Amazon Web Services). 

At 10:32am PDT, after exploring several options, we determined that we needed to shut down all communication between Amazon S3 servers, shut down all components used for request processing, clear the system’s state, and then reactivate the request processing components.

Okay, almost another hour is gone and I’d imagine all the “easy” options are exhausted. Now, they’re trying more high-impact solutions. This one sounds suspiciously like “bounce the <insert process here> and see if it comes up clean,” which is one of those embarrassing-but-effective solutions you end up using when you just don’t know what else to do sometimes.

By 11:05am PDT, all server-to-server communication was stopped, request processing components shut down, and the system’s state cleared. By 2:20pm PDT, we’d restored internal communication between all Amazon S3 servers and began reactivating request processing components concurrently in both the US and EU.

At 2:57pm PDT, Amazon S3′s EU location began successfully completing customer requests. The EU location came back online before the US because there are fewer servers in the EU. By 3:10pm PDT, request rates and error rates in the EU had returned to normal. At 4:02pm PDT, Amazon S3′s US location began successfully completing customer requests, and request rates and error rates had returned to normal by 4:58pm PDT.

So, a full bounce of that subsystem took almost 4 hours to show results. You can imagine those were 4 pretty tense hours. Some companies use a conference bridge to manage big incidents, others use web chats or VoIP systems. I’m sure a bunch of people were all working very hard to move this along quickly, and it still took quite a while, but you can almost imagine the relief that flooded the whole team at 2:57pm when EU came back up. By around 5pm, the whole system was back up and normal, and there wasn’t much left to do except the paperwork.

Which brings us to the last part of their message:

We’ve now determined that message corruption was the cause of the server-to-server communication problems. More specifically, we found that there were a handful of messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect. We use MD5 checksums throughout the system, for example, to prevent, detect, and recover from corruption that can occur during receipt, storage, and retrieval of customers’ objects. However, we didn’t have the same protection in place to detect whether this particular internal state information had been corrupted. As a result, when the corruption occurred, we didn’t detect it and it spread throughout the system causing the symptoms described above. We hadn’t encountered server-to-server communication issues of this scale before and, as a result, it took some time during the event to diagnose and recover from it.

You can be sure that as soon as they were on a path to get the system stable, they were investigating how it got unstable in the first place. This is a pretty in-depth problem statement, and it’s impressive how transparent Amazon is about it. I doubt I’d ever be allowed to report to the public on any of my outages like this alone, and I suspect Amazon is no different: at some point during that investigation, Amazon’s PR people started working on wording for public announcements. At the same time, technical teams were working to figure out how to make sure it never happens again.

During our post-mortem analysis we’ve spent quite a bit of time evaluating what happened, how quickly we were able to respond and recover, and what we could do to prevent other unusual circumstances like this from having system-wide impacts. Here are the actions that we’re taking: (a) we’ve deployed several changes to Amazon S3 that significantly reduce the amount of time required to completely restore system-wide state and restart customer request processing; (b) we’ve deployed a change to how Amazon S3 gossips about failed servers that reduces the amount of gossip and helps prevent the behavior we experienced on Sunday; (c) we’ve added additional monitoring and alarming of gossip rates and failures; and, (d) we’re adding checksums to proactively detect corruption of system state messages so we can log any such messages and then reject them.

This is fascinating. Certainly, (a) is an obvious concern — I know I’d be pulling out my hair if a key system I managed took 4 hours to restart. I’d speculate that this was simply a contingency nobody had thought about yet, and thus probably was a pretty painfully manual process. Amazon (and all the large operators) rely on quite a bit of automation to manage large fleets of servers, but when the unexpected happens, it doesn’t always work as planned. Item (b) looks like it’s addressing the root cause, item (d) may also play into this. Item (c) is that perennial favorite of ops: monitoring and alarming. These are all just the sort of things I’d expect to see if I were in that situation.

In summary, very interesting data about an increasingly important service on the internet. At the same time, a rare view into what goes on in the dark where SysAdmins and Network Engineers roam. Incidentally, there are technical details about the system that failed, which is a technology Amazon calls Dynamo.

The second event I’ll be at will be GigaOM’s Structure 08. Cloud computing is really leveling the playing field, giving small start-ups access to world-class operational assets… which to me only underscore the importance of having brilliant Ops folks to run those systems. I’m eager to see what sort of discussions emerge.

If you happen to be at either, give me a buzz in the comments, and I’ll try and catch up with you. 

DFT operates quite a few facilities in Northern Virginia. Most familiar to me would be the former AOL Gainesville Technology Center, sold to them in 2005. While this facility was big, the real capacity of a data center is less space (“raised floor square feet”) but power (“megawatts of critical load”). DFT has subsequently acquired several other sites, and will open a new facility roughly twice the size and with four times the power of the former AOL site, with several more in their pipeline.

That’s quite a bit of  new hosting capacity, and a clear sign that the large facility shortage may be over. To put it in perspective, the Lenior, NC or Quincy, WA sites mentioned in my earlier post as “really big” are in the same ballpark, size-wise, as one of their new centers. And, while we don’t really know that Google or Microsoft has filled out their space, DFT brags that all their earlier sites are leased with terms averaging 8 years. Their still-under-construction ACC4 site was 43.8% pre-leased in August.

So, who’s using all that space? Their IPO tells a bit of these normally secretive details. Facebook is making a big new expansion, leasing 10,000 square feet of space. Most of the rest of the space goes to a few major users, according to their registration statement. The same statement indicates that their undeveloped property represents 187MW of additional capacity. That said, they probably want to acquire some new customers, because “As of August 1, 2007, our two largest tentants, Microsoft and Yahoo!, accounted for 86.0% of our annualized rent”.

This is a distinctly different business model than that of either the major network providers (Level3 or Verizon), or the major Carrier Neutral providers (Equinix or Switch & Data). They consider themselves wholesalers to major providers, rather than a retail provider, because they allow the sort of large capacity blocks to be purchased that otherwise can’t be gotten without building it yourself.

So, will they ultimately suffer the fate of Exodus or Genuity?  They do seem to be in the right geographic locations (NoVA, Chicago, Santa Clara). They seem to have the right sizes and scale (new facilities over 30MW), with individual units of capacity in the 2-4MW range, for their target market. I’m not sure there are 10 more Microsoft, Yahoo, or Googles out there looking for space, but if they’re opening up their doors to Facebook at reasonably pricing, perhaps there’s enough large scale demand left. With Amazon chasing the small end, and DFT chasing the high end, this may make a very interesting business outlook for the ever-turbulent retail data center business.