One of the more common IT themes in the last few years has been the “cleaning up” of IT silos that have sprung up across an organization (the MD Anderson use case was about IT silos across departments in a hospital). In most cases these use cases are first attacked via the introduction of virtualization that runs on top of a homogeneous infrastructure.
One common “IT silo” use case is geographically distributed backups. When geographically distributed sysadmins are given the responsibility to back up their silo, they can often resort to whatever method works best for them. In addition, geographic distance can serve as an obstacle to centralize backup processes and methods (especially for technology deployments in remote areas).
And it’s hard to get more remote than Alaska.
The Alaska Department of Fish and Game has IT gear spread throughout the state. There are dozens of remote offices, with four geographically distributed data centers. There are two primary data centers (P) located in Anchorage and Juneau, and two smaller, secondary (S) data centers operating in Kodiak and Fairbanks. The map below highlights the location of ADF&G’s four data centers.
In 2007, ADF&G’s backup infrastructure was a hodgepodge of tape drives and formats, different backup software, different procurement procedures, different backup processes and methods, and no formal testing methodology to verify backup integrity.
To make matters worse, WAN connectivity to Kodiak and Fairbanks was inconsistent. The State of Alaska has a centralized technology group that runs the state WAN, and ADF&G had no visibility into the saturation levels at any given time (resulting in unpredictable data transfer guarantees). The technology group had also decided to deploy ESX servers at every data center, running hundreds of VMs. Attempting to back up this many VMs across a spotty WAN connection would be challenging.
ADF&G’s infrastructure manager Corey Kos was responsible for solving the backup problem across his department’s IT silos. His strategy was as follows:
“Mandate the introduction of a common, scalable, state-wide platform and consolidated management for backup”.
Tall order!
To make a long story short, ADF&G decided to buy 2 Data Domain 510 systems (5.7 TB prior to dedup) for Kodiak and Fairbanks, and 2 DD565 systems (16.2 TB) for the larger data centers in Anchorage and Juneau. The dozens of remote offices (most are only active in the summer) would use Windows DFS to replicate to one of the four data centers, where data was then backed up to the local Data Domain system.
The Data Domain systems replicate to each other across the state WAN. Data Domain’s ability to minimize the amount of WAN replication traffic “sealed the deal” for USD&F:
”The lack of visibility into our WAN connections between our four main data centers made it difficult to architect this part of our solution. But the EMC Data Domain appliances replicate data so efficiently that even over our worst WAN links we can get all of our data from our two most remote data centers by noon of the next day.”
The solution as a whole is diagrammed below:
All tape has been eliminated in every office. Data can be recovered for any locale. The offices have increased the number of virtual machines they are able to run, and the backup systems can handle the new capacity. For ADF&G, backup is something about which they “no longer have to think about or manage”.
I’d love to check out this configuration personally. One of the goals of ADF&G is to “optimize public participation in Alaskan fish and wildlife pursuits”. Sounds like fun.
More detail on this use case can be found here.
Steve
Twitter: @SteveTodd
