The most obvious and easy example to quote would be those who have accepted Steve Jobs as their personal savior: Mac users. I’m not talking about the iPod buyers who swoon over silouettes dancing to indie tunes, but the dyed-in-the-wool Macheads who keep arguing that their migration from 68000 to PowerPC was an “investment.” Yup, you know who I’m talking about.

Full Disclosure: I use a Mac Laptop and a Mac workstation. I also use a PC desktop and a PC laptop. And I like FreeBSD. And Linux. I can still call bullshit, right?

Yes, there are obviously great things about that platform. But even things I’d see as serious drawbacks (low-single-digit market share means limited commercial software) get spun as a positive (low-single-digit market share means limited malware). They tolerate wildly out of spec pricing (examine the cost of a Mac Pro, the desktop, Scandalous!) as justified due to good design cues. And they put up a hue and cry if anyone doubts the superiority of their technology. They’d seem just like Linux users, except some people listen to them.

Within AOL (or any large company which builds its own software), we get the same sort of syndrome. It goes like this: we have a technology problem, we build a technology solution, others have the same problem, an industry standard to solve that problem emerges, and we’re left with our highly prioprietary fix. There is then tremendous resistance to giving up our known, loved, and trusted solution for what the rest of the crowd is doing. Either a wrenching change happens, or we remain permanently out of step with the rest of the world. It isn’t that the solution itself is necessarily better, but having lived with it over our heads for so long, we’ve begun to emphathize with that solution.

So, what’s the solution here? Recognize that there’s some value in familiarity, but also that familiarity should also breed some contempt. Continually re-evaluate key decisions to make certain they’re still appropriate. Challenge technology creators, be they vendors, internals, or “the street” to prove their value against the competition. And, remember always that going your own way doesn’t make you a pioneer so much as it makes you a loner. Don’t be the last angry user of, say, NeXT — the final phase of this syndrome is the one where you become a joke.

DISCLAIMER: This is also not a negative reflection on IT staff. Certainly, there a distinction between the roles, but I mean absolutely no disrespect of any kind to the helpful and hardworking people who keep computing systems running everywhere.

To grossly oversimplify, Operations represents the activities of an organization which directly produce the product or render the service to the customer. For a manufacturing company, Operations is what makes the widgets. For an airline, Operations is flying planes. And for an Internet portal, Operations is — you guessed it — providing that portal to those users all over the Internet.

A Technology Operations group employees the people who run the hardware, software, networks and systems which actually deliver the product that is offered. At AOL, Operations are the people who make sure that you can log in to the AOL service or send an Instant Message. We fight the ground war on spam and viruses in your inbox. When Mapquest gets updated roads in your area, Netscape.com gets a new look, or AOL.com gets new features, we’re the ones deploying the applications — probably in the dead of night.

In addition to helping when new things go out, we also play a large role in figuring out how to solve problems. We’ve seen a lot of things scale, so we can give developers pointers to what will and won’t work. As product managers draw up their requirements, Development and Operations work together to make sure that we can get the product out on time, ensure it grows in a healthy fashion, and can be supported easily. There’s nothing worse than making a popular new product that breaks every time it gets a healthy crowd, and Operations works full time to make sure that doesn’t happen.

A lot of the same things apply for an IT organization, but with a different sort of products. First, the vast majority of IT products are off the shelf applications from commercial software vendors. The landscape of IT full of products like Exchange, SAP, Oracle Financials, and Peoplesoft. While some products have moved towards a ’software as a service’ model (think Salesforce.com), most companies still run a lot of off the shelf software. That isn’t to say that this is easy by any means — these applications have the potential to be even more complex than the ones that Operations people run.

The key difference is that IT shops rarely are part of the software development life cycle. Commercial support, training, and documentation are available for most of these applications. There are mature user communities which provide opportunities to hire people experienced in these applications. In addition, the customers for those applications are different: they’re all employees too. This has the effect of limiting their number: a few thousand employees using Exchange versus many millions of Mapquest users.

While I can search a jobs site and find people with experience in all of the common platforms of IT, the best-case Operations resume probably contains 15% of the technology we use. You can’t really find a Certified Administrator for a platform which hasn’t been written. You’ve got to find people who can think on their feet, parley well with developers, and impersonate QA when nobody else has bothered to test whatever we’re being handed. Oh, and don’t let things break, or it’ll be reported everywhere from the blogosphere to the Wall Street Journal.

Sound interesting? Ready for that sort of challenge? We’re always looking for people ready to run some of the world’s largest web properties. Drop me a line if you’re interested – my contact info is in About Jacob.

Big Step 2 is the moment where a web application outgrows the bounds of being hosted in a single physical location. The warning signs that this moment is approaching include the moment that you realize how much money you lost because your colocation provider had an outage, or when you glance at the Google Analytics report for your site and realize that half your users are half a world away from your servers.

But wait, some are saying, lots of great internet companies got really big without ever going into multiple locations. Even AOL was just in one dinky little data center for half its life. Can’t we just grow bigger in one place, and maybe write a check to some data center insurance scheme like SunGard.

The answer is hard maybe.

Multi-site may not be necessary for everyone. It can be expensive, it can force wrenching decisions, and it can force architectural changes that may not be worth a little performance degradation or risk of catastrophe.

Think about how hard it would be to go multi-site, consider the risk versus benefit, and decide for yourself whether you want to go down the route. I think it makes sense for most major sites with a broad geographic focus to consider it for performance reasons, and if your users’ data is important at all to your business, you better have a backup plan that goes that voodoo doll you keep on your bedside table.

Here are the main reasons why I like a bunch of smaller sites over one big site with a disaster site somewhere:

1) cheaper hosting costs

Okay, this one is pretty counterintuitive. Everyone talks about economies of scale, so why should smaller be cheaper? It comes down to demand. The industry is booming again, and large spaces in attractive facilities aren’t easy to come by anymore. If you’re looking for 100,000+ square feet of data center space, you’ll probably end up building something new, perhaps in the middle of no-where.

On the other hand, getting a cage large enough for a few dozen racks is still pretty practical. By spreading out your purchase, you reduce your buying power, but by using the same provider in a bunch of locations, you can probably get most of it back — especially if you’re also buying your transit from them, too.

2) less redundant gear

When you build one big site, you then need someplace to go when the next big hurricane/earthquake/flood/fire/power outage/alien invasion/godzilla attack hits. You’ll probably want to put this other place far enough away that the effects of the bad thing that knocked out your primary site don’t affect your emergency site.

Next, you may just want to put some gear in there. You can economize, and buy replacements when the disaster hits, but then you’ll be down a few days. That may be okay, but if you’re an online business, chances are it isn’t. So now you’re stuck buying a clone of all your normal gear. Plus, you need to make sure every time you buy a widget for site A, you get one for site B. And, to make things even more fun, you’ll need to make sure all that user data from site A end up at site B. That could be pretty tricky or expensive.

You’ve just doubled your operating costs, with very little to show for it aside from a warm feeling in case godzilla attacks your data center. On the other hand, let’s say you have six locations, with just enough extra gear so that if one of the six is attacked by aliens, the rest can handle the extra traffic. We’re talking 2N versus N+1 redundancy. You’ve probably cut your costs significantly by doing this.

3) better user performance

The more locations you have, the more likely that one of them will be close to your end user from a network perspective. You’ll need some sort of GSLB that does localization to make this work correctly, but it can be a big plus, especially if your application does something chatty. You might also be able to take advantage of better costs for network on a local basis (e.g., domestic vs international). In addition to putting fewer miles between you and the end user, you’re also probably putting fewer moving parts. This probably helps with reliability as well, but it certainly provides fewer choke points between you and the web, which will also likely have some performance benefits.

So, with all of this, you might be convinced that it makes sense to go with a distributed operating environment of smaller sites, glued together across geographies using cool networking technologies. But wait — I’ve just talked about the good parts, there are some things that get a lot harder in this environment as well. I’ll cover that next time, in part 2 of Big Step 2.

So, why does complexity spike at this point?

Up until this point, there are a lot of great shortcuts that you can use to make your application simple:
- You can count on having a single file system that’s always going to be there.
- IPC and shared memory work for tying together applications or systems.
- Your users always come back to the same web server process, so it is easy to use java or PHP sessions to store user data.
- There is no need to be concerned about network equipment like server load balancers.

Do any of your applications take advantage of any of the things mentioned above? Would you know how to re-engineer around them? Here are a few of the typical ways people deal with the above issues:

- Using NFS or some sort of other shared file system to try to maintain one single file system’s worth of stuff
- Moving from things like shared memory to network communication
- Using ’sticky’ load balancers to keep those users coming back to the same server

Most of these will allow you to expand to a second, third, or fortieth server. Eventually, however, most of them will hit their maximum as well. At very least, most of the interim solutions begin to cause problems when you reach Big Step 2, which I’ll discuss next time.

There were many different paths. People often speak about the AOL walled garden, but it wasn’t just a wall that kept people out. The garden wasn’t even of the same earth as the rest of the web. Proprietary technology and systems combined with a custom clients provided a universe that was totally alien to the rest of the web.

For years, we built and maintained those systems, which enjoyed a substantial technical head start over the Web technology evolving at would-be competitors like Yahoo. Still, AOL fielded hundreds of developers on their custom platform, while thousands of people at companies all over the world evolved the state of the art in web technology: sooner or later they would catch up.

The day probably came sometime before AOL acquired Netscape in November of 1998, but wasn’t acknowledged until much later. In fact, the defining moment where AOL decided to tear down the wall and move content out of FDO occured as part of a program called Big Bowl. Over the course of 2003 and 2004, starting with the News Channel, AOL migrated their content from legacy – but reliable – systems into web technology.

The platform of choice was AOLserver, an Open Source web server acquired with the purchase of Navisoft. A platform was created using this system which launched Digital City. In 2001, that platform extended to Mapquest and AIM Today, and in 2002 was extended to Netscape.com.

As channels began to migrate to this new platform, AOLserver hit the big time. The group working on this unified platform swelled in size, and began to take on other work as well. Some of these migrations went smoothly, but others showed signs of growing pains. Many from the legacy system complained that web technology was inefficient and inferior to the older systems. Still, the pace of migration continued, driven largely by the ability to put more and higher value ads on pages that were web than pages which were FDO.

The jewel in the crown, of course, was AOL.com. Several efforts were made to redesign this page, including the My AOL Startpage product. This was a flash-based personalized site launched during 2004 on ADP. While some users liked the flexible UI and neat features, it suffered on narrowband connections, had usability and accessibility challenges, and was generally considered a failure.

The next attempt on AOL.com was even bigger — in addition to doing a new portal in 2005, we were going to make AOL content available via a free portal. This was something which had been hotly debated for years, but seemed to be happening for-real this time.

The crux of the plan involved a new top-level portal with Yahoo-like features combined with open and accessible content from all the channels converted during the Big Bowl process. There would also be new free features, such as AIM Mail, a free clone of the AOL Webmail product which was popular with AOL.com users.

The new AOL.com launched over several months in the summer of 2005. It was considered a success at the time, and received widespread praise. Other releases followed in its wake, including a Video Portal and a social networking site called AIMPages in 2006.

At this time, however, it was clear that the future of the company was the web, and only the web. By mid-2006, it was widely rumored AOL would rush away from Access and dial-up and all that proprietary software which had defined the experience for 20 years. The sleeping giants were beginning to awaken, and that the AOL Web software was not built on our core, trusted technologies angered them.

Over the course of 2006, nearly every single architect of the ADP platform and the successful technologies that made AOL.com Portal a reality have moved on to other things.
Here lies the problem: we are cut off from the safety of a walled garden due to market forces. Yet, when things become rough, we resort to the same technologies which kept us behind those walls for so many years. Amazing things have been done with web technology over the last few years — heck, YouTube built a $1.6 billion business out of things you could learn in the Web Development section of a Barnes and Noble.

The future is the web, and the web is open. Is there a place for proprietary systems? Of course – many of the market leaders keep their lead (Google) by having secret sauce that goes beyond that can be found on the shelf. At the core of their excellence, however, is a spirit which embraces the potential of the web as a good thing. Until we’re able to set aside our sentimentality for the ways of the past, and truly embrace the future, we will fall short in what we hope to achieve.

I’m ready for the future — how about you?