Why Amazon Hired a Car Mechanic to Run Its Cloud Empire

Raw Text

James Hamilton on his boat, Dirona , docked at the Wakiki Yacht Club in Honolulu, Hawaii. Photo: Kent Nishimura/Wired

James Hamilton

On a rainy Monday in August 2011, a 10-million-watt transformer exploded in northern Virginia, sending an enormous voltage spike across the power grid. The surge hit an Amazon data center in Ashburn, Virginia, knocking out the facility’s main source of power, and about 15 minutes later, James Hamilton just happened to pull into the parking lot.

It was a serendipitous moment. Hamilton is the Distinguished Engineer who oversees the increasingly complex design of the data-center empire that drives Amazon Web Services , or AWS — the nothing-less-than-revolutionary collection of online services that provide computing power to companies across the globe, including names such as Netflix , Pinterest , and Dropbox . The Ashburn facility is part of that AWS empire. When it goes down, services like NetFlix are in danger of going down, and Hamilton is the man who works to ensure this doesn’t happen.

When Hamilton and his team drove up, the data center’s backup generators had kicked in, and they were idling away. But for some reason, the power wasn’t getting to the servers inside the facility, and the machines had sapped most of the juice from the battery system that temporarily keeps them going during a blackout. “We arrived just as the servers were starting to come down,” Hamilton remembered during a recent speech at an Amazon conference in Las Vegas. “Super annoying. Super annoying.”

“He’s really sharp,” says Adrian Cockcroft, the director of cloud architecture at Netflix, Amazon’s most high-profile customer. “He’s been around a long time, built a lot of interesting stuff.”

“Super annoying” is putting things mildly. But Hamilton’s job is to face such situations with a cool head — and figure out how to avoid them in the future. He and his team eventually realized the switches that were supposed to route backup power into the facility weren’t properly designed for Amazon’s world, where downtime like this can never happen. Today, the company programs its own electrical gear using PLCs, or programmable logic controllers. “We have multiple switch gear providers,” he said in Las Vegas. “We have gone through PLC programming on all of them and we do code reviews on all of our switch gear.”

When Amazon’s EC2 and S3 web services arrived in 2006, they changed the computing business. Big server makers — most noticeably Sun Microsystems — had been tinkering with the idea of selling computer power in much the same way that utility companies sell power or water, but much to everyone’s surprise, Amazon — a seller of books and CDs — managed to produce a set of services that the market was willing to pay for. Seven years later, the company is running a multi-billion dollar cloud computing business that runs as much as one percent of the internet , and everyone from Rackspace to HP is looking to duplicate its success.

But with this enormous success comes a whole new set of computing problems, and James Hamilton is one of the key thinkers charged with solving such problems, striving to rethink the data center for the age of cloud computing. Much like two other cloud computing giants — Google and Microsoft — Amazon says very little about the particulars of its data center work, viewing this as the most important of trade secrets, but Hamilton is held in such high regard, he’s one of the few Amazon employees permitted to blog about his big ideas , and the fifty-something Canadian has developed a reputation across the industry as a guru of distributing systems — the kind of massive online operations that Amazon builds to support thousands of companies across the globe.

“He’s really sharp,” says Adrian Cockcroft, the director of cloud architecture at Netflix, Amazon’s most high-profile customer. “He’s been around a long time, built a lot of interesting stuff.”

For Netflix, Amazon’s pay-as-you go computing services are the ideal. Netflix gets a big spike of movie-watchers hitting its website on Sunday night, and then by Monday morning, most of those people are gone. With Amazon, it can grab the resources it needs when it needs them and let them go when it doesn’t. But Cockcroft and company must also trust that Amazon can keep its massive operation up and running at all times — and keep prices well below the cost of doing it yourself. That’s where Hamilton comes in.

Inside the Hacker Mind: James Hamilton On Apple

When Apple says it’s building a 100-acre solar farm that can feed clean energy into the massive data center it operates in Maiden, North Carolina, most people can’t help but view it as a major victory for the future of God’s green Earth. But not James Hamilton.

“I just can’t make the math work,” he wrote last year in the wake the ongoing press coverage of Apple’s solar array , “and find myself wondering if these large solar farms are really somewhere between a bad idea and pure marketing, where the environmental impact is purely optical.”

Hamilton isn’t a marketer. He’s an engineer. He points out that a solar farm would have to be ridiculously large — and take out a ridiculously large number of trees — in order to significantly reduce the strain that a data center the size of Apple’s puts on the environment. Though it spans 100 acres, he argues, Apple’s solar farm likely provides only about 4 percent of the power needed to run the data center.

According to his math, if you wanted to power the entire 500,000-square-foot facility, you’d need a 181-million-square-feet solar farm. “There are many ways to radically reduce aggregate data center environmental impact without as much land consumption,” Hamilton says. “I look first to increasing the efficiency of power distribution, cooling, storage, networking and server and increasing overall utilization [as] the best routes to lowering industry environmental impact.”

Photo: Kent Nishimura/Wired

From Ferraris to Databases

A kind of Bohemian Grove for geeks, it’s the sort of place where you can sit down with “the people who invented databases”

When we met James Hamilton this past November in Las Vegas — a day before his talk on the Ashburn data center meltdown — he, his wife Jennifer, and his cat Spitfire had just spent 11 days on the open sea, travelling from San Francisco to Hawaii on their yacht, the Dirona. Hamilton, you see, is not your typical data-center engineer. He’s also the quintessential boatman — he even lives on a boat, which is only occasionally berthed in Bell Harbor Marina near the AWS headquarters in Seattle — and in a past life, he worked as an auto mechanic who specialized in Lamborghinis and Ferraris.

In the ’80s, he left cars for databases, doing some seminal work on IBM’s DB2 database and later on Microsoft’s SQL Server. NetFlix’s Adrian Cockroft met Hamilton several years ago at a conference in Asilomar, California, called the High Performance Transaction Systems Workshop. A kind of Bohemian Grove for geeks, it’s the sort of place where you can sit down with “the people who invented databases,” Cockcroft says, calling Hamilton “one of the fundamental people in that space.” The invitation-only event was co-founded by Jim Gray, the legendary database guru Hamilton is often compared to. Gray shared Hamilton’s love for the ocean — sadly, he vanished at sea off the northern California coast six years ago — and both pushed the boundaries of database research at IBM before moving to Microsoft.

But while at Microsoft, Hamilton caught the data center bug, working as an architect in a research group called called the Data Center Futures team, and in 2009, he moved to Amazon, where he does similar work — although many of the particulars remain a mystery, with both Amazon and Microsoft reluctant to reveal the secrets of their data centers. When we ask Microsoft’s David Gauthier, a director in the company’s data center group, if he knew James Hamilton, he brightens for an instant — “Yeah, I definitely spent a lot of time working with James before he left and went over to the other side” — before declining to say anything more.

But occasionally we get a whiff of how Hamilton is changing the data-center world. While at Microsoft, according to contemporary reports , he was one of the driving forces behind Microsoft’s move to modular data centers — where computing facilities are more efficiently pieced together using self-contained building blocks . And over a beer at Las Vegas, Hamilton confirmed that, in an effort to cut costs across its massive data-center empire, Amazon now designs its own servers .

On his blog, Hamilton provides at least a small window into his approach to data-center design — not to mention his take on massive diesel engines . He has endorsed the idea of ultra-low-power servers built around chips not unlike the one in your cellphone , and when half of the New Orleans Superdome went black earlier this month in the middle of Super Bowl XLVII, he cranked out a detailed analysis of how pro sports stadiums could prevent this kind of thing by borrowing a page from the Amazon playbook.

Over the past few years, Hamilton has swung by the University of Washington to give a series of “spectacular talks” about data-center design and efficiency, says Ed Lazowska, a computer science professor at the university. But Lazowska makes a point of saying this only begins to describe Hamilton’s work, which he calls remarkably “broad and deep,” the same words he uses in discussing the late Jim Gray. “Normally, as you get broader, you get uselessly shallow,” says Lazowska, “and the interesting thing about James and Jim … is that they’re both broad and deep.”

Renaissance Hacker

About four years ago, James and Jennifer Hamilton sold their house and their car and most of their worldly possessions, and they moved onto the Dirona . Now, when he’s berthed in Seattle, Hamilton bikes to Amazon headquarters, does his shopping via Amazon Prime, and picks up his mail at the local UPS store. But he’s untethered. Sometimes, he takes the boat to Hawaii — and works from there.

In short, he hacked his life. He rethought it and turned it into something new. He’s the sort of person who’s willing to hack anything, and that’s why Amazon wants him.

Amazon has built data centers around the globe, everywhere from Brazil to Singapore to Ireland. It spent $4.5 billion on technology and content last year, and while the company doesn’t say how much it spends on data centers, its AWS build-out now accounts for the majority of the $1.6 billion bump in technology and content spending it saw in 2012. With billions of dollars at stake, you need people who can keep those data centers running — and keep them running as efficiently as possible.

This talent, says Ed Lazowska, is hard to find. “There are some small number of people at the big companies — at Microsoft and Amazon and Google — who understand where you need to focus. Where do you put your effort if your goal is to run one of these data centers as cost-effectively as possible?”

James Hamilton is one of those people.

Cade Metz contributed to this story.

Go Back to Top. Skip To: Start of Article.

Single Line Text

James Hamilton on his boat, Dirona , docked at the Wakiki Yacht Club in Honolulu, Hawaii. Photo: Kent Nishimura/Wired. James Hamilton. On a rainy Monday in August 2011, a 10-million-watt transformer exploded in northern Virginia, sending an enormous voltage spike across the power grid. The surge hit an Amazon data center in Ashburn, Virginia, knocking out the facility’s main source of power, and about 15 minutes later, James Hamilton just happened to pull into the parking lot. It was a serendipitous moment. Hamilton is the Distinguished Engineer who oversees the increasingly complex design of the data-center empire that drives Amazon Web Services , or AWS — the nothing-less-than-revolutionary collection of online services that provide computing power to companies across the globe, including names such as Netflix , Pinterest , and Dropbox . The Ashburn facility is part of that AWS empire. When it goes down, services like NetFlix are in danger of going down, and Hamilton is the man who works to ensure this doesn’t happen. When Hamilton and his team drove up, the data center’s backup generators had kicked in, and they were idling away. But for some reason, the power wasn’t getting to the servers inside the facility, and the machines had sapped most of the juice from the battery system that temporarily keeps them going during a blackout. “We arrived just as the servers were starting to come down,” Hamilton remembered during a recent speech at an Amazon conference in Las Vegas. “Super annoying. Super annoying.” “He’s really sharp,” says Adrian Cockcroft, the director of cloud architecture at Netflix, Amazon’s most high-profile customer. “He’s been around a long time, built a lot of interesting stuff.” “Super annoying” is putting things mildly. But Hamilton’s job is to face such situations with a cool head — and figure out how to avoid them in the future. He and his team eventually realized the switches that were supposed to route backup power into the facility weren’t properly designed for Amazon’s world, where downtime like this can never happen. Today, the company programs its own electrical gear using PLCs, or programmable logic controllers. “We have multiple switch gear providers,” he said in Las Vegas. “We have gone through PLC programming on all of them and we do code reviews on all of our switch gear.” When Amazon’s EC2 and S3 web services arrived in 2006, they changed the computing business. Big server makers — most noticeably Sun Microsystems — had been tinkering with the idea of selling computer power in much the same way that utility companies sell power or water, but much to everyone’s surprise, Amazon — a seller of books and CDs — managed to produce a set of services that the market was willing to pay for. Seven years later, the company is running a multi-billion dollar cloud computing business that runs as much as one percent of the internet , and everyone from Rackspace to HP is looking to duplicate its success. But with this enormous success comes a whole new set of computing problems, and James Hamilton is one of the key thinkers charged with solving such problems, striving to rethink the data center for the age of cloud computing. Much like two other cloud computing giants — Google and Microsoft — Amazon says very little about the particulars of its data center work, viewing this as the most important of trade secrets, but Hamilton is held in such high regard, he’s one of the few Amazon employees permitted to blog about his big ideas , and the fifty-something Canadian has developed a reputation across the industry as a guru of distributing systems — the kind of massive online operations that Amazon builds to support thousands of companies across the globe. “He’s really sharp,” says Adrian Cockcroft, the director of cloud architecture at Netflix, Amazon’s most high-profile customer. “He’s been around a long time, built a lot of interesting stuff.” For Netflix, Amazon’s pay-as-you go computing services are the ideal. Netflix gets a big spike of movie-watchers hitting its website on Sunday night, and then by Monday morning, most of those people are gone. With Amazon, it can grab the resources it needs when it needs them and let them go when it doesn’t. But Cockcroft and company must also trust that Amazon can keep its massive operation up and running at all times — and keep prices well below the cost of doing it yourself. That’s where Hamilton comes in. Inside the Hacker Mind: James Hamilton On Apple. When Apple says it’s building a 100-acre solar farm that can feed clean energy into the massive data center it operates in Maiden, North Carolina, most people can’t help but view it as a major victory for the future of God’s green Earth. But not James Hamilton. “I just can’t make the math work,” he wrote last year in the wake the ongoing press coverage of Apple’s solar array , “and find myself wondering if these large solar farms are really somewhere between a bad idea and pure marketing, where the environmental impact is purely optical.” Hamilton isn’t a marketer. He’s an engineer. He points out that a solar farm would have to be ridiculously large — and take out a ridiculously large number of trees — in order to significantly reduce the strain that a data center the size of Apple’s puts on the environment. Though it spans 100 acres, he argues, Apple’s solar farm likely provides only about 4 percent of the power needed to run the data center. According to his math, if you wanted to power the entire 500,000-square-foot facility, you’d need a 181-million-square-feet solar farm. “There are many ways to radically reduce aggregate data center environmental impact without as much land consumption,” Hamilton says. “I look first to increasing the efficiency of power distribution, cooling, storage, networking and server and increasing overall utilization [as] the best routes to lowering industry environmental impact.” Photo: Kent Nishimura/Wired. From Ferraris to Databases. A kind of Bohemian Grove for geeks, it’s the sort of place where you can sit down with “the people who invented databases” When we met James Hamilton this past November in Las Vegas — a day before his talk on the Ashburn data center meltdown — he, his wife Jennifer, and his cat Spitfire had just spent 11 days on the open sea, travelling from San Francisco to Hawaii on their yacht, the Dirona. Hamilton, you see, is not your typical data-center engineer. He’s also the quintessential boatman — he even lives on a boat, which is only occasionally berthed in Bell Harbor Marina near the AWS headquarters in Seattle — and in a past life, he worked as an auto mechanic who specialized in Lamborghinis and Ferraris. In the ’80s, he left cars for databases, doing some seminal work on IBM’s DB2 database and later on Microsoft’s SQL Server. NetFlix’s Adrian Cockroft met Hamilton several years ago at a conference in Asilomar, California, called the High Performance Transaction Systems Workshop. A kind of Bohemian Grove for geeks, it’s the sort of place where you can sit down with “the people who invented databases,” Cockcroft says, calling Hamilton “one of the fundamental people in that space.” The invitation-only event was co-founded by Jim Gray, the legendary database guru Hamilton is often compared to. Gray shared Hamilton’s love for the ocean — sadly, he vanished at sea off the northern California coast six years ago — and both pushed the boundaries of database research at IBM before moving to Microsoft. But while at Microsoft, Hamilton caught the data center bug, working as an architect in a research group called called the Data Center Futures team, and in 2009, he moved to Amazon, where he does similar work — although many of the particulars remain a mystery, with both Amazon and Microsoft reluctant to reveal the secrets of their data centers. When we ask Microsoft’s David Gauthier, a director in the company’s data center group, if he knew James Hamilton, he brightens for an instant — “Yeah, I definitely spent a lot of time working with James before he left and went over to the other side” — before declining to say anything more. But occasionally we get a whiff of how Hamilton is changing the data-center world. While at Microsoft, according to contemporary reports , he was one of the driving forces behind Microsoft’s move to modular data centers — where computing facilities are more efficiently pieced together using self-contained building blocks . And over a beer at Las Vegas, Hamilton confirmed that, in an effort to cut costs across its massive data-center empire, Amazon now designs its own servers . On his blog, Hamilton provides at least a small window into his approach to data-center design — not to mention his take on massive diesel engines . He has endorsed the idea of ultra-low-power servers built around chips not unlike the one in your cellphone , and when half of the New Orleans Superdome went black earlier this month in the middle of Super Bowl XLVII, he cranked out a detailed analysis of how pro sports stadiums could prevent this kind of thing by borrowing a page from the Amazon playbook. Over the past few years, Hamilton has swung by the University of Washington to give a series of “spectacular talks” about data-center design and efficiency, says Ed Lazowska, a computer science professor at the university. But Lazowska makes a point of saying this only begins to describe Hamilton’s work, which he calls remarkably “broad and deep,” the same words he uses in discussing the late Jim Gray. “Normally, as you get broader, you get uselessly shallow,” says Lazowska, “and the interesting thing about James and Jim … is that they’re both broad and deep.” Renaissance Hacker. About four years ago, James and Jennifer Hamilton sold their house and their car and most of their worldly possessions, and they moved onto the Dirona . Now, when he’s berthed in Seattle, Hamilton bikes to Amazon headquarters, does his shopping via Amazon Prime, and picks up his mail at the local UPS store. But he’s untethered. Sometimes, he takes the boat to Hawaii — and works from there. In short, he hacked his life. He rethought it and turned it into something new. He’s the sort of person who’s willing to hack anything, and that’s why Amazon wants him. Amazon has built data centers around the globe, everywhere from Brazil to Singapore to Ireland. It spent $4.5 billion on technology and content last year, and while the company doesn’t say how much it spends on data centers, its AWS build-out now accounts for the majority of the $1.6 billion bump in technology and content spending it saw in 2012. With billions of dollars at stake, you need people who can keep those data centers running — and keep them running as efficiently as possible. This talent, says Ed Lazowska, is hard to find. “There are some small number of people at the big companies — at Microsoft and Amazon and Google — who understand where you need to focus. Where do you put your effort if your goal is to run one of these data centers as cost-effectively as possible?” James Hamilton is one of those people. Cade Metz contributed to this story. Go Back to Top. Skip To: Start of Article.