The second event I’ll be at will be GigaOM’s Structure 08. Cloud computing is really leveling the playing field, giving small start-ups access to world-class operational assets… which to me only underscore the importance of having brilliant Ops folks to run those systems. I’m eager to see what sort of discussions emerge.

If you happen to be at either, give me a buzz in the comments, and I’ll try and catch up with you. 

DFT operates quite a few facilities in Northern Virginia. Most familiar to me would be the former AOL Gainesville Technology Center, sold to them in 2005. While this facility was big, the real capacity of a data center is less space (“raised floor square feet”) but power (“megawatts of critical load”). DFT has subsequently acquired several other sites, and will open a new facility roughly twice the size and with four times the power of the former AOL site, with several more in their pipeline.

That’s quite a bit of  new hosting capacity, and a clear sign that the large facility shortage may be over. To put it in perspective, the Lenior, NC or Quincy, WA sites mentioned in my earlier post as “really big” are in the same ballpark, size-wise, as one of their new centers. And, while we don’t really know that Google or Microsoft has filled out their space, DFT brags that all their earlier sites are leased with terms averaging 8 years. Their still-under-construction ACC4 site was 43.8% pre-leased in August.

So, who’s using all that space? Their IPO tells a bit of these normally secretive details. Facebook is making a big new expansion, leasing 10,000 square feet of space. Most of the rest of the space goes to a few major users, according to their registration statement. The same statement indicates that their undeveloped property represents 187MW of additional capacity. That said, they probably want to acquire some new customers, because “As of August 1, 2007, our two largest tentants, Microsoft and Yahoo!, accounted for 86.0% of our annualized rent”.

This is a distinctly different business model than that of either the major network providers (Level3 or Verizon), or the major Carrier Neutral providers (Equinix or Switch & Data). They consider themselves wholesalers to major providers, rather than a retail provider, because they allow the sort of large capacity blocks to be purchased that otherwise can’t be gotten without building it yourself.

So, will they ultimately suffer the fate of Exodus or Genuity?  They do seem to be in the right geographic locations (NoVA, Chicago, Santa Clara). They seem to have the right sizes and scale (new facilities over 30MW), with individual units of capacity in the 2-4MW range, for their target market. I’m not sure there are 10 more Microsoft, Yahoo, or Googles out there looking for space, but if they’re opening up their doors to Facebook at reasonably pricing, perhaps there’s enough large scale demand left. With Amazon chasing the small end, and DFT chasing the high end, this may make a very interesting business outlook for the ever-turbulent retail data center business.

So, 365 Main screwed up big time? Not really. Their design was horribly flawed? Not so much. They have eight rooms full of servers, and ten generators — enough for every room to have one, plus two extra generators for contingencies. This type of UPS is a Diesel Rotary UPS: utility power makes a flywheel spin, the flywheel runs a generator, the generator supplies power to the computers. When utility power goes away, there’s a brief (to humans) pause while the diesel engine starts up make power for the computers. As long as the diesel spins up before the flywheel spins down, power keeps flowing. Proponents of this design like to emphasize how it’s simple, and thus pretty reliable, and in its defense, no part of the Rotary system seemed to fail in this case. For completeness, the other kind of UPS uses lots and lots of a batteries.

What did fail was the diesel engine’s controller. What makes the electricity is an enormous diesel engine, so naturally there’s quite a bit of support equipment to keep it running that had to be checked. During the investigation, issues with other systems (exhaust) were uncovered. Ultimately, though, they found a flaw that could be reproduced in a critical part of a system that should never fail. Yet, it did fail in 30% of the cases, and that was enough to bring down all sorts of different products, not to mention causing a highly public issue for 365 Main.

I’m not affiliated in any way with 365 Main, but I do have to say I’m impressed with how they have handled the aftermath of the incident. They were open and honest about the incident, provided lots of public information about their investigation, even to non-customers like me. Heck, they even ignored the idiotic rumors that Valleywag was tricked into posting as “news.” Nicely done, guys.

So, what makes companies like Google, Yahoo, Microsoft, and AOL build their own data center facilities, when the vast majority of companies end up leasing space from carrier-neutral colocation providers like Equinix, or telecommunications providers like Verizon Business or Level3? The tongue in cheek answer would be ‘because they can’, but building a data center facility is as much about control as it is about anything else. Being able to control key elements such as physical security, power and network access, space assignment policy, and general access to the space makes it compelling to own and operate a space. In addition, in recent years, the market for large spaces in the leasing market has dried up, adding both an availability benefit (when you build 250,000 square feet of space, you know it’s there for you) and a cost benefit (no competing with Google for the last big cage in a facility). Of course, owning a space locks in your cost basis in a way which leasing doesn’t, but for a business on the grow, there’s not really a question of whether the space will be used or not.

And yet, the Google, Microsoft, Yahoo, and AOL’s of the world don’t exclusively use owned space to host their servers: at least some of their footprint remains in leased space. Part of this is necessary in order to build out network connectivity in a desired way. You might not be able to get every peer you want to follow you to the middle-of-nowhere, but you can certainly get to them in San Jose or Washington D.C. There are also certain situations where being present in a specific place is the most important consideration for technical, legal, or contractual reasons.

So, what drives the massive centers out to the boonies? It certainly isn’t a proliferation of talent in those areas. In the past, land prices were a primary consideration, but in the last two years, the single most important factor seems to be power. Let’s take a look at a little something from the Department of Energy about Residential Electricity Prices, understanding that commercial/industrial pricing follows the same trend, but is likely a bit lower. The following map illustrates average prices in centers per kWh (caveat, this data is 4 years old. Here’s an ugly but new table of energy prices by State):

You’ll see some interesting facts — the states with the largest populations also have some of the highest energy costs. Compare 5.81 cents per kWh in Kentucky against New York’s pricely 14.31 cents, and you’ll see that there’s a tremendous incentive to locate in the lower cost states: Washington, Idaho, North Dakota, Nebraska, Tennessee, Kentucky, West Virginia. Of course, these are averages — most of those big high profile deals involved low-cost energy. The Microsoft deal, for example, was rumored to include sub-2 cent power. Add to this that several of these states receive power from hydroelectric (Bonneville Power Administration, or Tennessee Valley Authority) which have highly fixed costs, rather than burning a fuel whose cost could increase markedly in the future.

Cheap power is great, but without fiber to connect into, a data center is just a gigantic resistance heater. There are certainly plenty of public networks out there, but the larger providers seem to get especially interested in so-called Dark Fiber. A well-documented story about Google discussed their appetite for Dark Fiber and expected they’d start their own ISP. What’s much more likely is they just wanted to build their own backbone to connect up their numerous facilities. Dark fiber are individual fibers within an already laid cable that haven’t been connected or used yet. While the market has tightened with the consolidation to a few providers (okay, Level3 bought everybody: Wiltel, Progress, ICG, Telcove, Looking Glass, and Broadwing), major providers still can get access to the raw material they need to build their own network, using technologies such as Dense Wavelength Division Multiplexing to cram breathtaking amounts of traffic onto a single fiber. Trace the lines on, say, the Level3 network map and you can plot locations for almost all of those new gigantic datacenters. There’s no coincidence that Montana or North Dakota have no fiber and hence no data centers.

Next time: so, what’s actually in one of those gigantic Google data centers.