![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/01\/0-Drive_Stats_Header_2021_Annual.png\")
<\/p>\n<\/p>\n
In 2021, Backblaze added 40,460 hard drives and as of December 31, 2021, we had 206,928 drives under management. Of that number, there were 3,760 boot drives and 203,168 data drives. This report will focus on our data drives. We will review the hard drive failure rates for 2021, compare those rates to previous years, and present the lifetime failure statistics for all the hard drive models active in our data center as of the end of 2021. Along the way, we share our observations and insights on the data presented and, as always, we look forward to you doing the same in the comments section at the end of the post.<\/p>\n
At the end of 2021, Backblaze was monitoring 203,168 hard drives used to store data. For our evaluation, we removed 409 drives from consideration which were used for either testing purposes or drive models for which we did not have at least 60 drives. This leaves us with 202,759 hard drives to analyze for this report.<\/p>\n
![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/01\/1-Table_2021_annual_AFR.png\")
<\/p>\n
The Old Guy Rules:<\/strong> For 2021, the 6TB Seagate (model: ST6000DX000) had the lowest failure rate of any drive model, clocking in with an annualized failure rate (AFR) of 0.11%. This is even more impressive when you consider that this 6TB drive model is the oldest in the fleet with an average age of 80.4 months. The number of drives, 886, and 2021 drive days, 323,390, are on the lower side, but after nearly seven years in operation, these drives are thumbing their nose at the tail end of the bathtub curve<\/a>.<\/p>\n The Kids Are Alright:<\/strong> Two drive models are new for 2021 and both are performing well. The 16TB WDC drive cohort (model: WUH721816ALE6L0) has an average age of 5.06 months and an AFR of 0.14%. While the 16TB Toshiba drive cohort (model: MG08ACA16TE) has an average age of 3.57 months and an AFR of 0.91%. In both cases, the number of drive days is on the lower side, but these two drive models are off to a good start.<\/p>\n <\/p>\n AFR stands for annualized failure rate. This is different from an annual failure rate in which the number of drives is the same for each model (cohort) throughout the annual period. In our environment, drives are added and leave throughout the year. For example, a new drive installed in Q4 might contribute just 43 days, while a drive that failed in July might contribute 186 days, while drives in continuous operation for the year could contribute 365 days each. We count the number of drive days each drive contributes throughout the period and annualize the total using this formula:<\/p>\n The Patient Is Stable:<\/strong> Last quarter, we reported<\/a> on the state of our 14TB Seagate drives (model: ST14000NM0138) provisioned in Dell storage servers. They were failing at a higher than expected rate and everyone\u2014Backblaze, Seagate, and Dell\u2014wanted to know why. The failed drives were examined by fault analysis specialists and in late Q3 it was decided as a first step to upgrade the firmware for that cohort of drives still in service. The results were that the quarterly failure rate dropped from 6.29% in Q3 to 4.66% in Q4, stabilizing the rapid rise in failures we\u2019d seen in Q2 and Q3. The 19 drives that failed in Q4 were shipped off for further analysis. We\u2019ll continue to follow this process over the coming quarters.<\/p>\n The AFR for 2021 for all drive models was 1.01%, which was slightly higher than the 0.93% we reported for 2020. The next section will compare the data from the last three years.<\/p>\n The chart below compares the AFR for each of the last three years. The data for each year is inclusive of that year only and for the active drive models present at the end of each year.<\/p>\n Digging a little deeper, we can aggregate the different drive models by manufacturer to see how failure rates per manufacturer have fared over the last three years.<\/p>\n Note that for the WDC data, a blank value means we did not have any countable WDC drives in our data center in that quarter.<\/p>\n The AFR Stayed Low in 2021:<\/strong> In 2021, the AFR for all drives was 1.01%. This was slightly higher than 2020 at 0.93%, but a good sign that the drop in 2020 from 1.83% in 2019 was not an anomaly. What\u2019s behind the 1.01% for 2021? Large drives, as seen below:<\/p>\n The AFR for larger drives, defined here as 12TB, 14TB, and 16TB drives, are all below the 2021 AFR of 1.01% for all drives. The larger drives make up 69% of the total drive population, but more importantly, they total 66% of the drive days total, while only producing 57% of the drive failures.<\/p>\n The larger drives are also the newer drives, which tend to fail less versus older drives. In fact, the oldest large drive has an average age 33 months, while the youngest \u201csmall\u201d (4TB, 6TB, 8TB, and 10TB) drive has an average age of 44.9 months.<\/p>\n In summary, the lower AFR for the larger drives is a major influence in keeping the overall AFR for 2021 low.<\/p>\n Drive Model Diversity Continues:<\/strong> In 2021, we added two new drive models to our farm with no models retired. We now have a total of 24 different drive models in operation. That\u2019s up from a low point of 14 in 2019 and 22 in 2020. The chart below for “Backblaze Hard Drive Population Percentage by Manufacturer” examines the changing complexion of our drive farm as we look at the number of models from each manufacturer we used over the past six years.<\/p>\n When we first started, we often mixed and matched drive models, mostly out of financial necessity\u2014we bought what we could afford. As we grew, we bought and deployed drives in larger lots and drive homogeneity settled in. Over the past few years, we have gotten more comfortable with mixing and matching again, enabled by our Backblaze Vault architecture. A Vault is composed of sixty tomes, with each tome being 20 drives. We make each tome the same drive model, but each of the tomes within a vault can have different drive models, and even different drive sizes. This allows us to be less reliant on any particular drive model, so the more drive models the better.<\/p>\n Drive Vendor Diversity Continues, Too:<\/strong> When looking at the chart above for “Backblaze Hard Drive Population by Model Count per Manufacturer Over Time,” you might guess that we have increased the percentage of Seagate drives over the last couple of years. Let\u2019s see if that\u2019s true.<\/p>\n It appears the opposite is true, we have lowered the percentage of Seagate drives in our data centers, even though we have added additional Seagate models.<\/p>\n Why is it important to diversify across multiple manufacturers? Flexibility, just like increasing the number of models. Having relationships with all the primary hard drive vendors gives us the opportunity to get the resources we need in a timely fashion. The fact that we can utilize any one of several different models from these vendors adds to that flexibility.<\/p>\n The chart below shows the lifetime annualized failure rates of all the drive models in production as of December 31, 2021.<\/p>\n The lifetime AFR for all the drives listed above is 1.4% and continues to go down year over year. At the end of 2020, the AFR was 1.54% and at the end of 2019, the AFR stood at 1.62%.<\/p>\n When looking at the chart above, several of the drives have a fairly wide confidence interval (>0.5). In these cases, we do not really have enough information about the drive\u2019s performance to be reasonably confident (>95%) in the AFR listed. This is typically the case with lower drive counts or newer drives.<\/p>\n <\/p>\n We’ll be covering our annual failure rates for our SSD drives in a separate post in the next few weeks. We realized that combining the analysis of our data drives and our boot drives in one post was confusing. Stay tuned.<\/p>\n<\/div>\n The complete data set used to create the information used in this review is available on our Hard Drive Test Data<\/a> page. You can download and use this data for free for your own purpose. All we ask are three things: 1) you cite Backblaze as the source if you use the data, 2) you accept that you are solely responsible for how you use the data, and 3) you do not sell this data to anyone; it is free.<\/p>\n\nAFR, What Does That Mean?<\/h5>\n
Comparing Drive Stats for 2019, 2020, and 2021<\/strong><\/h2>\n
![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/01\/2-Table_2021_three_year_AFR_comparison.png\")
<\/p>\n![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/01\/3-Chart_2021_quarterly_by_MFG-2.png\")
<\/p>\nTrends for 2021<\/strong><\/h3>\n
![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/01\/4-Table_2021_drive_size_comparison.png\")
<\/p>\n![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/01\/5-Chart_2021_model_count_2.png\")
<\/p>\n![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/01\/6-Chart_2021_population_by_MFG.png\")
<\/p>\nLifetime Hard Drive Stats<\/strong><\/h2>\n
![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/01\/7-chart_2021_lifetime_AFR.png\")
<\/p>\nObservations and Caveats<\/strong><\/h3>\n
\nLooking for SSD Numbers?<\/h5>\n
The Hard Drive Stats Data<\/strong><\/h2>\n