Backblaze is transitioning from using 4 TB hard drives to 6 TB hard drives in the Storage Pods we will be deploying over the coming months. With over 10,000 hard drives needing to be purchased over the next several months, the choice of which 6TB hard drive to use is critical. Let’s take a look at how we’re navigating this transition.
We started the process in September when we purchased and deployed our first 6 TB hard drives. We purchased Western Digital (WD60EFRX) and Seagate (STBD6000100) hard drives. We deployed two Storage Pods, each with 45 drives from each brand. The Western Digital Storage Pod was designated UL796 and the Seagate Storage Pod was designated UL800. Each Pod was identical in its design and configuration except for the hard drives used.
In a previous post we described how we set up and load tested our Storage Pods. Each of the 6 TB-drive Pods passed their set up and load testing without incident and were deployed, first the Western Digital Pod and a few days later the Seagate. For comparison purposes we also deployed Storage Pod UL838, which had 4 TB HGST drives installed.
Into the Fire
Backblaze currently receives 130 TB of data from our customers to store each day. Data arrives in similarly sized encrypted blocks of data. On any given day, there are 20-40 Storage Pods accepting these data blocks as they arrive. As a block arrives it is passed to a Storage Pod, if that Pod is busy, the data block is passed to the next Pod in line. Over the course of each day this results in all the available Storage Pods being given the same opportunity to accept data at the same rate.
The amount of customer data Backblaze stores each day.
Newly installed Storage Pods accept data for the most part unencumbered until they reach 80% full. At that point a Storage Pod will reduce the amount of data it receives each day. New Storage Pods come on line on a regular basis so that arriving data always has a place to go without delay.
When UL796, UL800 and UL 838, came online we recorded various statistics with regards to how much data they loaded. Here’s a sample of the relevant storage data collected on one of those days.
|Date||Pod ID||Data Stored||Free Space||Total Space||Status|
|9/17/2014||UL796||48.87 TB||162.27 TB||211.14 TB||Active|
|9/17/2014||UL800||29.16 TB||181.98 TB||211.14 TB||Active|
|9/17/2014||UL838||6.81 TB||133.95 TB||140.76 TB||Active|
Since the three Storage Pods did not start accepting data on the same day, there could be some differences in data stored by day. The following chart aligns the three Storage Pods over the same calendar days, e.g. Day 1 is Sep. 19th, Day 2 is Sep. 20th, etc.
Observe that the different Storage Pods filled up at different rates. Here is how quickly each Pod reached 80% capacity.
|Pod ID||Drive Brand||80% Full||Total Space|
|UL796||Western Digital||33 days||169 TB|
|UL800||Seagate||42 days||169 TB|
|UL838||HGST||23 days||113 TB|
This translates to the following average data stored per day for each Storage Pod.
|Pod ID||Drive Brand||Data Loaded|
|UL796||Western Digital||5.12 TB/day|
The Western Digital hard drive loaded data faster than the Seagate and even edged out the HGST.
Let’s review the Seagate and Western Digital drives so far:
- Initial reliability (how many drives failed) – No failures.
- Running reliability (3 months) – No failures
- SMART Stats (3 months) – No error conditions recorded for the 5 stats that we utilize.
- Hard Drive Cost – about the same.
- Energy Use – The Seagate drives were 7200 rpm and used slightly more electricity than the Western Digital drives which were 5400 rpm. This small difference adds up when you place 45 drives in a Storage Pod and then stack 10 Storage Pods in a cabinet.
- Loading speed – Edge to Western Digital, by a little over 1 TB per day on average.
Our goal is to find hard drive models that are reliable and cost effective in our environment. The “One Pod” test has proven over the years to be a good starting point to eliminate those hard drives models that are obviously incompatible with our environment.
Next Step: Scaling the Test
Based on the results, we have ordered 230 Western Digital drives to fill 5 Storage Pods (with 5 spare drives). These will be installed, load tested and deployed shortly. Assuming the Western Digital drives continue to perform as well in the 5 Storage Pods, we’ll move forward with using the Western Digital 6TB drives in our Storage Pods over the coming months.
Is Seagate Shutout?
The Seagate 6TB drives performed well, albeit they loaded a little slower. They also use a little more electricity than we’d like. Still they are not shut out. We really like to have multiple qualified hard drives to order and use in our Storage Pods. Diversification is good. To that end we expect to order additional Seagate 6 TB hard drives over the coming months, build them into Storage Pods and monitor how they perform. Assuming they perform well, the 6 TB Seagate hard drives would be added to our list of qualified drives.
Victory is Fleeting
Today the Western Digital hard drives are first in line to be our choice for 6 TB drives. Of course, we just ordered 45 HGST 8 TB Helium hard drives for testing. Their unit price is still a too high for cost effective deployment, but thanks to the return of a relentlessly decreasing hard drive pricing curve, it could be just a matter of time. In fact, Seagate is now beginning to ship their 8 TB shingled magnetic recording (SMR) hard drives for a reported $260 a drive. Availability is spotty at the moment, but is certain to improve over the coming months. Using either the HGST or the Seagate 8TB drives means a 360 TB Storage Pod is imminent – we can hardly stand the wait…