If you’re uploading 50 or 75 terabytes (TB) of data to the cloud, not only do you have to pay to send all that data using your network bandwidth, it could also take an outrageously long amount of time. Is here another way to move an extremely large amount of data to the cloud? Google Transfer Appliance can do the job and also save time and money.
Amazon Web Services tried to come up with a solution for this problem, named Snowball, but their data limit is 50 or 80TB. Google Transfer Appliance, on the other hand, comes in two much larger sizes: 100TB and 480TB.
What is Google Transfer Appliance?
Google Transfer Appliance is a high-capacity storage server leased to you from Google. After you receive it, you connect it to your network, load the data, then ship it to a facility where the data is uploaded to Google Cloud Storage.
The two sizes available, as mentioned, are 100TB and 480TB, although the appliances can potentially store even more depending on the compression ratio and deduplication of your data. For example, Google claims that the 100TB option can potentially store 200TB, while the 480TB one could host 1 petabyte of data.
Using this option, you can securely transfer up to a petabyte of data on one single appliance paying a fraction of what you would uploading this much data via a high-speed Internet connection, with either a rackable or standalone appliance depending on your needs.
In fact, a petabyte of data would take about three years to upload with a typical network bandwidth of 100Mbps. Google Transfer Appliance, instead, lets you capture this same amount of data in less than 25 days after you receive the appliance. Then, it’ll take about another 25 days until this data is accessible in Google Cloud Storage.
Besides the speed, one of the key advantages of this product is the cost savings from not having to consume any outbound network bandwidth to make your data accessible on the cloud storage.
As of right now, this product is in the beta stages. If you’d like to participate, you can fill out Google’s Transfer Appliance beta application form.
Who would benefit from Google Transfer Appliance?
Google released a helpful table for you to see approximate upload times based on your dataset, size, and network speeds.
If you have more than 60TB of data or need more than one week to upload your data, Google recommends that you use the Transfer Appliance. However, don’t forget to consider that the predicted time for your data to be uploaded to the cloud using the Transfer Appliance is a total of 50 days, depending on the size, so factor this in before deciding.
How does it work?
First, you’ll receive a Transfer Appliance in the mail, then you’ll configure it, and connect it to your network. The Transfer Appliance utilities capture and stream data to the appliance. After this, files can be attached directly or mounted over network shares as long as they can be accessed as a file in a POSIX compliant file system.
According to Google’s blog post, “all captured data is deduplicated, compressed and encrypted with an industry standard AES 256 algorithm using a password and passphrase that you specify” before being stored on the Transfer Appliance.
This password and passphrase are then used when the data is uploaded to Google Cloud Platform to unencrypt, uncompress, and reconstruct the data. Be careful, though. There is no way to recover the data on the Transfer Appliance if you forget the password or passphrase.
After you’ve completed all of this, make sure that there is no corruption in the data and that everything transferred completely by running data integrity checks. If all goes well, it’s time to ship the Transfer Appliance back to Google.
Once Google receives the appliance, they copy the encrypted data to your Google Cloud Storage staging bucket. Right now, the uploaded data is still compressed, deduplicated, and encrypted. So, Google will send you an email as soon as your data has finished uploading and you can begin rehydrating it.
To make the data usable again, you rehydrate it, or unencrypt, uncompress, and reconstruct the data. Simply run the Transfer Appliance Rehydrator application and choose your Cloud Storage destination bucket.
This Rehydrator instance that you use is a virtual appliance that runs as a Google Compute Engine instance on GCP. Then, once again, run those integrity checks to make sure all is well.
From here, the Transfer Appliance is securely wiped and reimaged to be used again in the future.
Pricing
With this much data, nothing is cheap. While you might be saving a lot of money by not having to pay for any outbound network bandwidth consumption, it’s still a costly experience.
It’s important to understand how many days you’ll need the Transfer Appliance for because Google charges a usage fee that includes a certain number of free days of onside usage, but you’ll incur a late fee if the appliance is kept for longer than the allotted free days.
With the 100TB appliance, you have 10 days to upload your data, whereas with the 480TB one you are given 25 free days. A nice inclusion to this rule is that only weekdays are counted toward these free days.
One important consideration is that you will have to pay for the shipping fees of your Transfer Appliance. Of course, these vary greatly depending on your location and shipping options. According to Google, though, “shipping one 480TB appliance is typically 70% cheaper than shipping five 100TB appliances.”
They offer to work with you so you can find the best, lowest-cost option for your needs, but this could run from about $500-$900 depending on your appliance size.
Google also saves you some costs by not making you pay data ingest fees to move the data from the Transfer Appliance to Google Cloud Storage. However, once the data is located in the Cloud Storage, you do have to pay the associated monthly storage and operations costs for the data, of course.
If you’d like more information about pricing, including a chart and an example, check out Google Transfer Application Pricing.
If you have an extremely large amount of data that you’d like to upload to Google Cloud Storage, this appears to be the most economical option for now.
Thanks for the article!
You made an interesting research on how GCP transfer appliance works and I should add that 43 days for data validation is really impressing. Seems like Amazon has got a new competitor.