Contact: Carl.Coppadge at corp.aol.com or AIM: carlcoppadge11

Location: Dulles, VA

MySQL Database Administrator

AOL’s People Networks Operations, which operates AIM and ICQ, has an opening for a Sr. MySQL Database Administrator. This position would be responsible for managing large, highly scaled production MySQL databases as well as pre-production (QA/Dev/Staging) environments.

Key Responsibilities:

• All aspects of MySQL deployment, operation,and design to ensure high reliability and performance
• Collaborate with developers and architects to create high-performance, cost-effective designs
• Participate in an on-call rotation as part of a 24×7 operations team to resolve urgent production issues
• Ensuring data integrity with good process and a keen eye for detecting errors and misuses of data

  Desired Skills for this Position:

• Knowledge of database architecture concepts as well as MySQL-specific implementations
• Significant experience with MySQL in a high-volume production environment
• Experience with open source ETL packages and methods for large scale data migration
• Diverse technical background with awareness of concepts in networking, Linux, and storage
• Bachelor-level degree in Engineering, Computing, or Sciences orequivalent experience
• Replication: managing replication delay, scaling replication throughput, and designing resilient systems
• Configuration: tuning InnoDB, managing memory usage, and tuning file systems for maximum throughput
• Reliability: backups under load, failover strategies, and recovery from replication issues
• MySQL: familiarity with Percona and Google patches

  Our perfect candidate can manage many rapidly changing projects while maintaining professionalism and poise. Our environment is both highly exciting and highly demanding — those who thrive here adopt a “work smarter, not harder” attitude.

If there’s any one thing that’s been disappointing about this time around, it’s been the continual stream of nastiness by people who claim to be current or former AOL employees in public forums and online rumors sites. Isn’t it bad enough that we’re going through this — do we need to rip on each other as well? Difficult situations produce different responses from people, and I understand the desire to vent, I just wish it were less self-destructive.

And that’s all I have to say about the topic. I’ll be back to my regularly-scheduled … uhm … “posting schedule” when the grumpy wears off.

So, what makes companies like Google, Yahoo, Microsoft, and AOL build their own data center facilities, when the vast majority of companies end up leasing space from carrier-neutral colocation providers like Equinix, or telecommunications providers like Verizon Business or Level3? The tongue in cheek answer would be ‘because they can’, but building a data center facility is as much about control as it is about anything else. Being able to control key elements such as physical security, power and network access, space assignment policy, and general access to the space makes it compelling to own and operate a space. In addition, in recent years, the market for large spaces in the leasing market has dried up, adding both an availability benefit (when you build 250,000 square feet of space, you know it’s there for you) and a cost benefit (no competing with Google for the last big cage in a facility). Of course, owning a space locks in your cost basis in a way which leasing doesn’t, but for a business on the grow, there’s not really a question of whether the space will be used or not.

And yet, the Google, Microsoft, Yahoo, and AOL’s of the world don’t exclusively use owned space to host their servers: at least some of their footprint remains in leased space. Part of this is necessary in order to build out network connectivity in a desired way. You might not be able to get every peer you want to follow you to the middle-of-nowhere, but you can certainly get to them in San Jose or Washington D.C. There are also certain situations where being present in a specific place is the most important consideration for technical, legal, or contractual reasons.

So, what drives the massive centers out to the boonies? It certainly isn’t a proliferation of talent in those areas. In the past, land prices were a primary consideration, but in the last two years, the single most important factor seems to be power. Let’s take a look at a little something from the Department of Energy about Residential Electricity Prices, understanding that commercial/industrial pricing follows the same trend, but is likely a bit lower. The following map illustrates average prices in centers per kWh (caveat, this data is 4 years old. Here’s an ugly but new table of energy prices by State):

You’ll see some interesting facts — the states with the largest populations also have some of the highest energy costs. Compare 5.81 cents per kWh in Kentucky against New York’s pricely 14.31 cents, and you’ll see that there’s a tremendous incentive to locate in the lower cost states: Washington, Idaho, North Dakota, Nebraska, Tennessee, Kentucky, West Virginia. Of course, these are averages — most of those big high profile deals involved low-cost energy. The Microsoft deal, for example, was rumored to include sub-2 cent power. Add to this that several of these states receive power from hydroelectric (Bonneville Power Administration, or Tennessee Valley Authority) which have highly fixed costs, rather than burning a fuel whose cost could increase markedly in the future.

Cheap power is great, but without fiber to connect into, a data center is just a gigantic resistance heater. There are certainly plenty of public networks out there, but the larger providers seem to get especially interested in so-called Dark Fiber. A well-documented story about Google discussed their appetite for Dark Fiber and expected they’d start their own ISP. What’s much more likely is they just wanted to build their own backbone to connect up their numerous facilities. Dark fiber are individual fibers within an already laid cable that haven’t been connected or used yet. While the market has tightened with the consolidation to a few providers (okay, Level3 bought everybody: Wiltel, Progress, ICG, Telcove, Looking Glass, and Broadwing), major providers still can get access to the raw material they need to build their own network, using technologies such as Dense Wavelength Division Multiplexing to cram breathtaking amounts of traffic onto a single fiber. Trace the lines on, say, the Level3 network map and you can plot locations for almost all of those new gigantic datacenters. There’s no coincidence that Montana or North Dakota have no fiber and hence no data centers.

Next time: so, what’s actually in one of those gigantic Google data centers.

The most obvious and easy example to quote would be those who have accepted Steve Jobs as their personal savior: Mac users. I’m not talking about the iPod buyers who swoon over silouettes dancing to indie tunes, but the dyed-in-the-wool Macheads who keep arguing that their migration from 68000 to PowerPC was an “investment.” Yup, you know who I’m talking about.

Full Disclosure: I use a Mac Laptop and a Mac workstation. I also use a PC desktop and a PC laptop. And I like FreeBSD. And Linux. I can still call bullshit, right?

Yes, there are obviously great things about that platform. But even things I’d see as serious drawbacks (low-single-digit market share means limited commercial software) get spun as a positive (low-single-digit market share means limited malware). They tolerate wildly out of spec pricing (examine the cost of a Mac Pro, the desktop, Scandalous!) as justified due to good design cues. And they put up a hue and cry if anyone doubts the superiority of their technology. They’d seem just like Linux users, except some people listen to them.

Within AOL (or any large company which builds its own software), we get the same sort of syndrome. It goes like this: we have a technology problem, we build a technology solution, others have the same problem, an industry standard to solve that problem emerges, and we’re left with our highly prioprietary fix. There is then tremendous resistance to giving up our known, loved, and trusted solution for what the rest of the crowd is doing. Either a wrenching change happens, or we remain permanently out of step with the rest of the world. It isn’t that the solution itself is necessarily better, but having lived with it over our heads for so long, we’ve begun to emphathize with that solution.

So, what’s the solution here? Recognize that there’s some value in familiarity, but also that familiarity should also breed some contempt. Continually re-evaluate key decisions to make certain they’re still appropriate. Challenge technology creators, be they vendors, internals, or “the street” to prove their value against the competition. And, remember always that going your own way doesn’t make you a pioneer so much as it makes you a loner. Don’t be the last angry user of, say, NeXT — the final phase of this syndrome is the one where you become a joke.

DISCLAIMER: This is also not a negative reflection on IT staff. Certainly, there a distinction between the roles, but I mean absolutely no disrespect of any kind to the helpful and hardworking people who keep computing systems running everywhere.

To grossly oversimplify, Operations represents the activities of an organization which directly produce the product or render the service to the customer. For a manufacturing company, Operations is what makes the widgets. For an airline, Operations is flying planes. And for an Internet portal, Operations is — you guessed it — providing that portal to those users all over the Internet.

A Technology Operations group employees the people who run the hardware, software, networks and systems which actually deliver the product that is offered. At AOL, Operations are the people who make sure that you can log in to the AOL service or send an Instant Message. We fight the ground war on spam and viruses in your inbox. When Mapquest gets updated roads in your area, Netscape.com gets a new look, or AOL.com gets new features, we’re the ones deploying the applications — probably in the dead of night.

In addition to helping when new things go out, we also play a large role in figuring out how to solve problems. We’ve seen a lot of things scale, so we can give developers pointers to what will and won’t work. As product managers draw up their requirements, Development and Operations work together to make sure that we can get the product out on time, ensure it grows in a healthy fashion, and can be supported easily. There’s nothing worse than making a popular new product that breaks every time it gets a healthy crowd, and Operations works full time to make sure that doesn’t happen.

A lot of the same things apply for an IT organization, but with a different sort of products. First, the vast majority of IT products are off the shelf applications from commercial software vendors. The landscape of IT full of products like Exchange, SAP, Oracle Financials, and Peoplesoft. While some products have moved towards a ’software as a service’ model (think Salesforce.com), most companies still run a lot of off the shelf software. That isn’t to say that this is easy by any means — these applications have the potential to be even more complex than the ones that Operations people run.

The key difference is that IT shops rarely are part of the software development life cycle. Commercial support, training, and documentation are available for most of these applications. There are mature user communities which provide opportunities to hire people experienced in these applications. In addition, the customers for those applications are different: they’re all employees too. This has the effect of limiting their number: a few thousand employees using Exchange versus many millions of Mapquest users.

While I can search a jobs site and find people with experience in all of the common platforms of IT, the best-case Operations resume probably contains 15% of the technology we use. You can’t really find a Certified Administrator for a platform which hasn’t been written. You’ve got to find people who can think on their feet, parley well with developers, and impersonate QA when nobody else has bothered to test whatever we’re being handed. Oh, and don’t let things break, or it’ll be reported everywhere from the blogosphere to the Wall Street Journal.

Sound interesting? Ready for that sort of challenge? We’re always looking for people ready to run some of the world’s largest web properties. Drop me a line if you’re interested – my contact info is in About Jacob.

First stop, AOL Video, at least partly because our department runs most of those products. Video Search returned the 1 result for my search (Dana Carvey Gerald Ford SNL), sadly enough a link to Google Video. Oh, well, at least I found it. Enjoy!

A once-dominant technology company in the midst of a changing marketplace makes a dramatic shift to their business model. In January, they announce that they will give away their product for free. By February, the sharks are already circling, looking to shop them to the highest bidder. November of that year, they are acquired in much the way prophesized. And, of course, then the exodus/bloodbath begins, with most of their employees heading out in all directions.

A once-dominant technology company in the midst of a changing marketplace makes a dramtic shift to their business model. In August, they announce that they will give away their product for free. By December, the sharks are already circling, looking to shop them to the highest bidder. The denouement has not yet been reached, yet the exodus/bloodbath has already begun, with many employees heading/pushed out.
Ouch, too soon for me, too. Still, if there’s ever a call for a Joseph Campbell for Web 2.0, ring me up. Happy New Years, folks!

1. Denial

This is by far the most popular stage to get mired in these days. Recent stories, however, are making it tougher to deny what’s going on. Yes, it’s really happening. Yes, it’s going on right now. It isn’t a rumor anymore — it’s time to deal with it.
2. Anger

This is a stage where emotional outbursts, perhaps in the form of posts or comments on some site known for publishing rumors, are common. Resist the temptation to blog during this period, especially if you’re Dossy.

3. Bargaining

The prototypical bargaining tends to be with a higher power. The fact that many of your colleagues are reaching this phase may explain why your manager is looking so tired. You may wonder if change could be delayed until a project ends, or perhaps avoided altogether (e.g., Can we skip the layoffs if we don’t have a holiday part).

Sometimes this bargaining takes the form of a deal with yourself. I’m sure right now, at least a few employees are counting the days until bonuses are paid out. Others, who recently received layoff notices, may be considering whether to look for some opportunity to stay. Remember that you have to live with any deal you make, especially if it is with yourself.
4. Depression

Can’t… even… manage… to… blog…

5. Acceptance

Yup, things are changing, and that’s how it has to be. The reality is that every company, every society, every person is continually changing and this is simply part of the human condition.

Many of us are very wrapped up in the AOL Experience — we’ve been living the company day in, day out for years. This can lead to a certain loss of perspective: it is, after all, just a company that provides a service. We’ll do our best, and deal with what is to come next.

And, if what you really need is a dose of schadenfreude, just think that Yahoo employees will probably be doing the exact same thing really soon.