Really Big Data Centers
While most of my time these days is spent contemplating software and application considerations, I’d like to take a moment to address a topic which only occasionally gets the attention it deserves: the role of a high quality data center. While a few folks may think that networking and data center infrastructure are dead arts, I’m quite confident that there is still significant work going on in this space. Case in point: Yahoo, Microsoft, Google, Google, and (shockingly) Google are building massive new data centers taking advantage of all of the latest features to increase density and automation and reduce cost. At the end of the day, scale wins, and these facilities (which have price tags in the half-a-billion dollar range) have scale. Not to be outdone, incidentally, AOL has built a few big data centers — and sold them too.
So, what makes companies like Google, Yahoo, Microsoft, and AOL build their own data center facilities, when the vast majority of companies end up leasing space from carrier-neutral colocation providers like Equinix, or telecommunications providers like Verizon Business or Level3? The tongue in cheek answer would be ‘because they can’, but building a data center facility is as much about control as it is about anything else. Being able to control key elements such as physical security, power and network access, space assignment policy, and general access to the space makes it compelling to own and operate a space. In addition, in recent years, the market for large spaces in the leasing market has dried up, adding both an availability benefit (when you build 250,000 square feet of space, you know it’s there for you) and a cost benefit (no competing with Google for the last big cage in a facility). Of course, owning a space locks in your cost basis in a way which leasing doesn’t, but for a business on the grow, there’s not really a question of whether the space will be used or not.
And yet, the Google, Microsoft, Yahoo, and AOL’s of the world don’t exclusively use owned space to host their servers: at least some of their footprint remains in leased space. Part of this is necessary in order to build out network connectivity in a desired way. You might not be able to get every peer you want to follow you to the middle-of-nowhere, but you can certainly get to them in San Jose or Washington D.C. There are also certain situations where being present in a specific place is the most important consideration for technical, legal, or contractual reasons.
So, what drives the massive centers out to the boonies? It certainly isn’t a proliferation of talent in those areas. In the past, land prices were a primary consideration, but in the last two years, the single most important factor seems to be power. Let’s take a look at a little something from the Department of Energy about Residential Electricity Prices, understanding that commercial/industrial pricing follows the same trend, but is likely a bit lower. The following map illustrates average prices in centers per kWh (caveat, this data is 4 years old. Here’s an ugly but new table of energy prices by State):
You’ll see some interesting facts — the states with the largest populations also have some of the highest energy costs. Compare 5.81 cents per kWh in Kentucky against New York’s pricely 14.31 cents, and you’ll see that there’s a tremendous incentive to locate in the lower cost states: Washington, Idaho, North Dakota, Nebraska, Tennessee, Kentucky, West Virginia. Of course, these are averages — most of those big high profile deals involved low-cost energy. The Microsoft deal, for example, was rumored to include sub-2 cent power. Add to this that several of these states receive power from hydroelectric (Bonneville Power Administration, or Tennessee Valley Authority) which have highly fixed costs, rather than burning a fuel whose cost could increase markedly in the future.
Cheap power is great, but without fiber to connect into, a data center is just a gigantic resistance heater. There are certainly plenty of public networks out there, but the larger providers seem to get especially interested in so-called Dark Fiber. A well-documented story about Google discussed their appetite for Dark Fiber and expected they’d start their own ISP. What’s much more likely is they just wanted to build their own backbone to connect up their numerous facilities. Dark fiber are individual fibers within an already laid cable that haven’t been connected or used yet. While the market has tightened with the consolidation to a few providers (okay, Level3 bought everybody: Wiltel, Progress, ICG, Telcove, Looking Glass, and Broadwing), major providers still can get access to the raw material they need to build their own network, using technologies such as Dense Wavelength Division Multiplexing to cram breathtaking amounts of traffic onto a single fiber. Trace the lines on, say, the Level3 network map and you can plot locations for almost all of those new gigantic datacenters. There’s no coincidence that Montana or North Dakota have no fiber and hence no data centers.
Next time: so, what’s actually in one of those gigantic Google data centers.
October 21st, 2007 at 11:26 pm
[...] shortage may be over. To put it in perspective, the Lenior, NC or Quincy, WA sites mentioned in my earlier post as “really big” are in the same ballpark, size-wise, as one of their new centers. And, [...]