Information’s Carbon Footprint

I noticed last week that Chad mentioned the Intel – EMC research that will reduce server power consumption for the Atmos Storage System.

For those unfamiliar with the software internals of the Atmos system, I wrote a primer on the overall architecture last year. This architecture allows a service provider to globally distribute petabytes of information around the world as part of "one global Atmos configuration". This architecture is also the reason why Atmos is referred to as cloud-optimized storage; content stored in one geographic location is dynamically copied/migrated within a cloud to other geographic locations based on the nature of the content itself.

Service providers that deploy a cloud-based storage system could in theory generate a fairly significant carbon footprint when storing petabytes of content. Their purchasing decisions will be partly based on vendor ability to minimize power consumption.

Atmos to date has used disk spin-down as an answer to minimizing power consumption. Imagine an Atmos deployment with three systems: one in Santa Clara, one in NYC, and one in Shanghai.  Atmos has central configuration monitoring and knows the "capabilities" of the Atmos hardware in each location. For example, Atmos might know that the system in Shanghai is spin-down capable, and that the system in NYC is not.

Assume a video file is stored on the Santa Clara system. Assume that the service provider has configured a policy which states "all video files need to be mirrored".  At this point Atmos could mirror a second copy to either Shanghai or NYC.

But what if the video file was stored with additional metadata that stated "green = true". In other words, what if the service provider wanted to store a second copy but minimize the kwh required to store the second copy.  When Atmos detects the green keyword, it triggers the policy which mirrors to Shanghai.

So how does the Intel / EMC research play into this?  Today's Atmos product features disk spin-down. In twelve months, the Intel / EMC collaboration should enable Atmos to manage power at the server level (in addition to disk). Server power will be controlled by taking advantage of Intel's power management tools for its Nehalem processors.

It becomes an interesting thought experiment, therefore, to extend our service provider example by adding a fourth location: Iceland. Imagine that the new Iceland Atmos hardware supports spin-down AND Nehalem power management. What happens when a video file gets stored to Santa Clara with the "green = true" metadata? Does Atmos put the second copy in Shanghai or Iceland?

When it comes to power consumption, what I'd like to see is the addition of more mathematical policy evaluation within Atmos, as well as the ability to dynamically reflect the prevailing electric rates for each geography. For example, how big (number of disks and servers) are the systems in Shanghai/Iceland, how full (capacity) are they, how often are they read (spun-up/spun-down ratio), etc. 

One of my co-workers has spent a good deal of time fine-tuning these equations. They can result in answers that reflect the kilowatt cost of storing a file of a certain size, and the estimated percentage of spin-down savings that the file is likely to see.  On a strictly power-consumption basis, Iceland may end up being the destination data center, given the additional capability for server-based savings.

However, if China has lower electric rates, a "cost-to-power" policy may determine that a given file may cost less to store in China than Iceland.

It's fun to imagine the possibilities here. If it sounds pie-in-the-sky, I can assure you that it's not. (It's more like "head-in-the-clouds").

These types of algorithms and capabilities will eventually be commonplace, and it goes beyond policies for electricity. There will be cloud policies for compliance, performance, and perhaps even information value.

The technology is enabled by attaching metadata to objects, which in part explains why there have been so many discussions about object storage this year.

Steve

http://stevetodd.typepad.com

Twitter: @SteveTodd

EMC Intrapreneur