I used to work on Google's indexing system, and I guess for lot of Google's workloads, total machine performance-per-Watt is the key metric. Many server workloads are largely coordinating the disk controller and the network controller, with maybe some heavy integer and a bit of fp processing in the middle, so your normal pure CPU performance-per-Watt benchmarks can be misleading. ("A supercomputer is a device for turning compute-bound problems into I/O-bound problems." --Seymour Cray, possibly apocryphal) POWER8 uses a lot of Watts, but it also has high I/O throughput, so for many server workloads it could beat Intel chips in full system performance-per-Watt. I would be really interested in seeing an I/O-per-Watt CPU benchmark.
Many of Google's workloads are embarrassingly parallel. I used to work on Google's indexing system, and one of the binaries I worked with had a bit over a million threads (across many processes and machines) running at any moment, with most of those threads blocked on I/O. POWER8 has a fair number of hardware threads per core, which should help with the heavily multithreaded style of programming used in many Google projects.
Google's datacenters use enormous amounts of power. Several of their locations are former Aluminum smelting plants, because Aluminum smelting also uses enormous amounts of power so the power lines are already in place. My team luckily happened to sit next to one of the guys who designs Google's datacenters and we happened to overhear him say something over the phone about not being able to make sense about enough power to power a small town just disappearing from our usage. One of the guys on my team asked him when this power reduction occurred, and if it had happened before. We worked out that the huge swings in power usage happened when we shut down the prototype for the new indexing system. Most of Google's datacenters are largely empty space because they are limited by the power lines and cooling capacity.
In another instance, we discovered a mistake had been made in measuring the maximum power drawn by one of the generations of Google servers. Google had a program designed to max out the systems, and they plugged a server into a power meter wall-wart and ran this program for a while. The maximum power usage was under-estimated due to a combination of the office where the measurement was made being cooler than a datacenter (electrical resistance of most conductors has a positive temperature coefficient in the range of temperatures found in working servers), the machine not being allowed sufficient time to warm up, and/or the indexing system being more highly tuned than the program designed to maximize server utilization. (I like to think it was mostly the latter, but I suspect the first two were the main contributors.) The end result was that cooling was under-provisioned in one of the datacenters. During a heatwave in the area occupied by the datacenter used for most of the indexing process, the datacenter began to overheat, so one of the guys on my team was getting temperature updates every 10 to 15 minutes from a guy actually in the datacenter, and adjusting the number of processes running the indexing system up and down accordingly in order to match indexing speed to the cooling capacity. When you're really truly maxing out that many machines 24/7, some machines will break every day, so the indexing system (as most Google systems) are tolerant of processes just being killed either by software fault, hardware fault, or the cross-machine scheduler.
Through a combination of realizing smart phones were really going to take off and they were all ARM powered and realizing how important total machine performance-per-Watt is to some major server purchasers lead me to invest in ARM Holdings in mid 2009. (This does not constitute investment advice. The rest of the market has now realized what I realized in 2009 and I don't feel I have more insight than the market at this point.)
Many of Google's workloads are embarrassingly parallel. I used to work on Google's indexing system, and one of the binaries I worked with had a bit over a million threads (across many processes and machines) running at any moment, with most of those threads blocked on I/O. POWER8 has a fair number of hardware threads per core, which should help with the heavily multithreaded style of programming used in many Google projects.
Google's datacenters use enormous amounts of power. Several of their locations are former Aluminum smelting plants, because Aluminum smelting also uses enormous amounts of power so the power lines are already in place. My team luckily happened to sit next to one of the guys who designs Google's datacenters and we happened to overhear him say something over the phone about not being able to make sense about enough power to power a small town just disappearing from our usage. One of the guys on my team asked him when this power reduction occurred, and if it had happened before. We worked out that the huge swings in power usage happened when we shut down the prototype for the new indexing system. Most of Google's datacenters are largely empty space because they are limited by the power lines and cooling capacity.
In another instance, we discovered a mistake had been made in measuring the maximum power drawn by one of the generations of Google servers. Google had a program designed to max out the systems, and they plugged a server into a power meter wall-wart and ran this program for a while. The maximum power usage was under-estimated due to a combination of the office where the measurement was made being cooler than a datacenter (electrical resistance of most conductors has a positive temperature coefficient in the range of temperatures found in working servers), the machine not being allowed sufficient time to warm up, and/or the indexing system being more highly tuned than the program designed to maximize server utilization. (I like to think it was mostly the latter, but I suspect the first two were the main contributors.) The end result was that cooling was under-provisioned in one of the datacenters. During a heatwave in the area occupied by the datacenter used for most of the indexing process, the datacenter began to overheat, so one of the guys on my team was getting temperature updates every 10 to 15 minutes from a guy actually in the datacenter, and adjusting the number of processes running the indexing system up and down accordingly in order to match indexing speed to the cooling capacity. When you're really truly maxing out that many machines 24/7, some machines will break every day, so the indexing system (as most Google systems) are tolerant of processes just being killed either by software fault, hardware fault, or the cross-machine scheduler.
Through a combination of realizing smart phones were really going to take off and they were all ARM powered and realizing how important total machine performance-per-Watt is to some major server purchasers lead me to invest in ARM Holdings in mid 2009. (This does not constitute investment advice. The rest of the market has now realized what I realized in 2009 and I don't feel I have more insight than the market at this point.)