![\"Backblaze](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2015\/04\/seagate-3tb.jpg\")
<\/p>\n<\/p>\n
Beginning in January 2012, Backblaze deployed 4,829 Seagate 3TB hard drives, model ST3000DM001, into Backblaze Storage Pods. In our experience, 80% of the hard drives we deploy will function at least four years<\/a>. As of March 31, 2015, just 10% of the Seagate 3TB drives deployed in 2012 are still in service. This is the story of the 4,345 Seagate 3TB drives that are no longer in service.<\/p>\n In November 2011, Backblaze, like everyone else who used hard drives, was reeling from the effects of the Thailand drive crisis<\/a>. Prices had jumped 200-300% for hard drives and supplies were tight. The 3TB drives we normally used from HGST (formerly Hitachi) were difficult to find, but we still needed to buy 500-600 drives a month to run our online backup business. The 3TB drives we were able to find in decent quantity were from Seagate and we bought as many as we could. We purchased internal drives and also external USB drives, from which we removed the enclosed hard drive. The model number of the drive, ST3000DM001, was the same for both the internal and external drives.<\/p>\n Here is a chart of our Seagate drive purchases from November 2011 through December 2012.<\/p>\n Looking back on 2012, it is safe to say that if we did not purchase the Seagate 3TB drives, our business would have been dramatically affected. We estimated that our costs would have been at least $1.14 million more<\/a>, making our goal of keeping our price at $5\/month for unlimited storage<\/a> difficult, at best. In other words, the ability to purchase, at a reasonable price, the nearly 5,000 Seagate 3TB drives that we needed during 2012 was instrumental in meeting our business objectives.<\/p>\n Beginning in January 2012, we deployed 4,829 Seagate 3TB drives as shown below.<\/p>\n We would expect the Seagate 3TB drives to follow the bathtub-shaped failure rate curve<\/a> described in our study on hard drive<\/a> life expectancy. Instead, the Seagate drives failure model was quite different.<\/p>\n In annual terms, 2.7% or the drives failed in 2012, 5.4% failed in 2013, and 47.2% failed in 2014.<\/p>\n As of March 31, 2015, 1,423 of the 4,829 deployed Seagate 3TB drives had failed; that\u2019s 29.5% of the drives.<\/p>\n Let\u2019s take a minute to describe what happens when a drive in a Storage Pod fails. When a drive fails, no data is compromised since we distribute data redundantly across multiple drives. Simply, the bad drive is replaced and the system is tested and rebuilt. During the entire process, the data is safe and available for file recovery as needed.<\/p>\n If during the rebuilding process, a second drives fails, the data is migrated to another Storage Pod where it is safe and available and the Storage Pod with the second failed drive is taken offline. Once offline, technicians go through a series of steps to assess the health of the system.<\/p>\n One of the health assessment steps can be to remove all the drives from the Storage Pod for testing. There are two different tests. The first test is similar to \u201cadvanced\u201d reformatting and takes about 20 minutes. The second basically writes and reads all the sectors on the drive and takes several hours. Only if a drives passes both tests can it be reformatted and reused.<\/p>\n The first sign of trouble was in May of 2013, when 27 drives failed. This was about 0.5% of the Seagate drives deployed at the time, a small number, but worth paying attention to. In June, there were 25 failures and in July there were 29, but it was in July 2013 that the failing drives issue came to the forefront.<\/p>\n During July and August 2013, three Storage Pods, all with Seagate drives, had drive failures. In all three cases, each time a drive was replaced and the rebuilding process restarted, additional drive failures would occur. At this point, all of the hard drives in each of the three Storage Pods were removed and scheduled for further testing. The Storage Pods themselves had new drives installed and went back into service.<\/p>\n The drives from the three Pods were removed and tested, as noted above, and about half of the drives from the three Storage Pods failed the first test. The remaining \u201cgood\u201d drives were subjected to the second test and about 50% failed that test. The results were eye-opening. It was decided that all of the drives from the three Storage Pods would be removed from service and not redeployed.<\/p>\n Over the next several months, Seagate hard drives failed in noticeable quantities: 31 in October 2013, 68 in November, 70 in December and the upward trend continued in 2014. The only saving grace was that in nearly all cases, once a failed drive was replaced, the system would rebuild without incident. Any time a drive gave the least sign of trouble, it was removed and tested. Failing either of the external tests meant the drive was removed from service and placed with the \u201csuspect\u201d drives. The \u201csuspect\u201d pile was getting larger by the day.<\/p>\n The failure count continued to rise and in the Spring of 2014 we had decided that if a Storage Pod with Seagate 3TB drives showed any type of drive failure<\/a> we would 1) immediately migrate all the data and then 2) remove and test all the drives in the Storage Pod<\/a>.<\/p>\n In July alone, 189 hard drives failed and another 273 were removed from service. The total, 462, was 11.4% of the Seagate 3TB drives operational on July 1st<\/sup>, 2014.<\/p>\n To be clear, a drive is marked \u201cfailed\u201d because it failed in operation or during a rebuilding process. Drives marked \u201cremoved\u201d are those that were removed from a Storage Pod that contained failed drives. When the \u201cremoved\u201d drives were tested, nearly 75% of them failed one of the two tests done after removal. It could be argued that 25% of the \u201cremoved\u201d drives were still good, even though they were assigned to the removed category, but these drives were never reinstalled.<\/p>\n The Seagate 3TB drives purchased from November 2011 through December 2012 were failing at very high rates throughout 2014. If we look at the Seagate 3TB drives deployed during 2012, here is their status as of March 31, 2015.<\/p>\n Only 251 of the 4,190 Seagate 3TB hard drives deployed in 2012 are still in service as of March 31, 2015. Breaking it down:<\/p>\n As a reminder, about 75% of the \u201cremoved\u201d drives failed one of the bench tests once they were removed from a Storage Pod.<\/p>\n The first thing to consider is whether this was a systemic issue on our part. Let\u2019s start with comparing the Seagate drives to other 3TB drives deployed in 2012.<\/p>\n Given that the drives were deployed into the same environment, the Seagate 3TB drives didn\u2019t fare as well.<\/p>\n A second thing to consider is the model of the Storage Pod. In 2012, Version 2.0 was the only Storage Pod deployed; Version 3.0 was not used until February 2013. So all of the 3TB drives deployed in 2012 were installed in a Storage Pod 2.0<\/a> system. In the case of Seagate, the 3TB drives installed in 2012 performed reasonably well during the first and second years of operation; 2.7% of the drives in service failed in 2012 and a total of 7.7% of the drives deployed in 2012 had failed through the end of 2013. It was in 2014 that the drives seemed to \u201chit the wall.\u201d<\/p>\n Conversely, as noted above, we also deployed 2,511 HGST drives, all into Version 2.0 Storage Pods. To date, they have not shown any signs of \u201chitting the wall,\u201d with just 4.1% of the drives failing as of March 31, 2015.<\/p>\n A third thing to consider was the use of \u201cexternal\u201d drives. Did the \u201cshucking\u201d of external drives inflate the number of drive failures? Consider the following chart.<\/p>\nNovember 2011: The Thailand Drive Crisis<\/h2>\n
![\"Seagate](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2015\/04\/blog_seagate_purchases.png\")
<\/a><\/p>\nOur New Reality in the Face of the Thailand Drive Crisis<\/h3>\n
![\"blog_seagate_3tb_deploy\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2015\/04\/blog_seagate_3tb_deploy.jpg\")
<\/a><\/p>\nThe Slide to Failure<\/h2>\n
![\"Failed](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2015\/04\/blog_seagate_3tb_bad_curve.jpg\")
<\/p>\nDrive Failure Replacement and Testing<\/h2>\n
The Harbinger<\/h2>\n
Hitting the Wall<\/h2>\n
Digging In<\/h2>\n
![\"Seagate](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2015\/04\/blog_seagate_3tb_2012_failures.jpg\")
<\/a><\/p>\n![\"blog_seagate_status\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2015\/04\/blog_seagate_status1.png\")
<\/a><\/p>\nThoughts and Theories<\/h2>\n
Theory: The Backblaze System<\/h4>\n
![\"blog_seagate_status\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2015\/04\/blog_seagate_status.png\")
<\/a><\/p>\nTheory: Storage Pod 2.0<\/h4>\n
Theory: Shucking External Drives<\/h4>\n
![\"Seagate](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2015\/04\/blog_seagate_ext_int.png\")
<\/a><\/p>\n