![\"blog_smart-stats\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/10\/blog_smart-stats.jpg\")
<\/p>\n<\/p>\n
I\u2019ve shared a lot of Backblaze data about hard drive failure statistics<\/a>. While our system handles a drive failing, we prefer to predict drive failures, and use the hard drives\u2019 built-in SMART metrics to help. The dirty industry secret? SMART stats are inconsistent from hard drive to hard drive.<\/p>\n With nearly 40,000 hard drives and over 100,000,000GB<\/a> of data stored for customers, we have a lot of hard-won experience. See which five of the SMART stats are good predictors of drive failure below. And see the data we have started to analyze from all of the SMART stats to see which other ones predict failure.<\/p>\n Every disk drive includes Self-Monitoring, Analysis, and Reporting Technology (SMART<\/a>), which reports internal information about the drive. Initially, we collected a handful of stats each day, but at the beginning of 2014 we overhauled our disk drive monitoring to capture a daily snapshot of all of the SMART data for each of the 40,000 hard drives we manage. We used Smartmontools<\/a> to capture the SMART data.<\/p>\n But, before we dig into the data, we first need to define what counts as a failure.<\/p>\n Backblaze counts a drive as failed when it is removed from a Storage Pod<\/a> and replaced because it has 1) totally stopped working, or 2) because it has shown evidence of failing soon.<\/p>\n A drive is considered to have stopped working when the drive appears physically dead (e.g. won\u2019t power up), doesn\u2019t respond to console commands or the RAID system tells us that the drive can\u2019t be read or written.<\/p>\n To determine if a drive is going to fail soon we use SMART statistics as evidence to remove a drive before it fails catastrophically or impedes the operation of the Storage Pod<\/a> volume.<\/p>\n From experience, we have found the following five SMART metrics indicate impending disk drive failure:<\/p>\n We chose these five stats based on our experience and input from others in the industry because they are consistent across manufacturers and they are good predictors of failure.<\/p>\n We compiled and placed online our list of all the SMART stats across all hard drives we use<\/a>. For each stat we display the failure rate charts based on the raw and normalized values we recorded. Remember, this is raw data and since different disk drive manufacturers report SMART stats differently, be careful how you use this.<\/p>\n There are over 70 SMART statistics available, but we use only five. To give some insight into the analysis we\u2019ve done, we\u2019ll look at three different SMART statistics here. The first one, SMART 187, we already use to decide when to replace a drive, it\u2019s really a test of the analysis. The other two are SMART stats we don\u2019t use right now, but have potentially interesting correlations with failure<\/p>\n Backblaze uses this one. This first chart shows the failure rates by number of errors. Because this is one of the attributes we use to decide whether a drive has failed, there has to be a strong correlation:<\/p>\n The next question you might ask is: How many drives fall into each of those ranges? That\u2019s answered by the next chart: This looks at the full time range of the study, and counts \u201cdrive years.\u201d Each day that a drive is in one of the ranges counts as 1\/365 of a drive year for that range. Those fractions are all added up to produce the chart above. It shows that most of the daily samples show drives without errors.<\/p>\n For SMART 187, the data appears to be consistently reported by the different manufacturers, the definition is well understood, and the reported results are easy to decipher: zero is good, above zero is bad. For Backblaze this is a very useful SMART stat.<\/p>\n Backblaze does not use this one. We\u2019re not sure whether this is because cycling the power is bad for the drive, or because working on the Pods is bad for the drives, or because \u201cnew\u201d drives have flaws that are exposed during the first few dozen power cycles and then things settle down.<\/p>\n Most of our drives have very few power cycles. They just happily sit in their Storage Pods holding your data. If one of the drives in a Storage Pod fails, we cycle down the entire Storage Pod to replace the failed drive. This only takes a few minutes and then power is reapplied and everything cycles back up. Occasionally we power cycle a Storage Pod for maintenance and on rare occasions we\u2019ve had power failures, but generally, the drives just stay up.<\/p>\n As a result, the correlation of power cycles to failure is strong, but the power cycles may not be the cause of the failures because of our limited number of power cycles for a drive (less than 100) and also considering the variety of other possible failure causes during that period. In addition to reporting the raw value, drives also report a \u201cnormalized\u201d value in the range from 253 (the best) down to one (the worst). The drive is supposed to know what its design criteria and failure modes are, then interpret the raw value and tell you whether it\u2019s good or bad. Unfortunately, with the Power_Cycle_Count, the drives all say the value is 100, which doesn\u2019t lead to a very useful chart.<\/p>\n As shown, SMART 12 does not produce a useful normalized value, it doesn\u2019t think that power cycling is a problem at all.<\/p>\n You may ask whether there is a correlation of power cycle count with failures because power cycle count correlates with age, and age correlates with failures. The answer is no. The correlation of power cycle count with age is very weak: 0.05. New drives can have higher power cycle counts, and old drives can have low power cycle counts.<\/p>\n Because Backblaze does not power-cycle our drives very often, this SMART stat is not very useful to us in determining the potential failure of a drive. It also does not answer the age-old question that asks whether turning off your computer every night is better or worse for the disk\u2014that mystery remains.<\/p>\n Backblaze does not use this one. And a lot of the drives have a zero value, presumably meaning \u201cno problem:\u201d<\/p>\n Unlike the Power_Cycle_Count, the scaled value for this the Raw_Read_Error_Rate does what it\u2019s supposed to do: failure rates are higher for lower normalized values, although it\u2019s not a nice smooth progression: For Backblaze to use this SMART stat we\u2019d like to have a better sense of the values as reported by each vendor. While a value above zero is not good, the inconsistency of the reported values above zero is wildly inconsistent as seen in the charts above using normalized values. Since the manufacturers don\u2019t tell us what their attribute values could be, this SMART stat is not very useful, especially across multiple drive manufacturers.<\/p>\n Backblaze uses SMART 5, 187, 188, 197, and 198 for determining the failure or potential failure of a hard drive. We would love to use more\u2014ideally the drive vendors would tell us exactly what the SMART attributes mean. Then we, and the rest of the storage community, could examine the data and figure out what\u2019s going on with the drives.<\/p>\n In the meantime, at Backblaze, we\u2019ll continue gathering data and working to correlate it as best we can. One thing we are looking at is to break down each SMART stat by the drive model, but there are challenges with how drive manufacturers change drive model numbers and how firmware changes occur within a given model. We\u2019ll see if there is anything interesting and let you know.<\/p>\n Remember, you can find charts like the ones above for all of the SMART attributes on our website here<\/a>. If you see something interesting there and figure out what it means, or know more yourself, be sure to let us know.<\/p>\n","protected":false},"excerpt":{"rendered":"S.M.A.R.T.<\/h2>\n
What Is a Failure?<\/h2>\n
\n
The Other SMART Stats<\/h2>\n
Choosing the Right Stats to Use<\/h3>\n
SMART 187: Reported_Uncorrect<\/h3>\n
\nNumber 187 reports the number of reads that could not be corrected using hardware ECC. Drives with zero uncorrectable errors hardly ever fail. This is one of the SMART stats we use to determine hard drive<\/a> failure; once SMART 187 goes above zero, we schedule the drive for replacement.<\/p>\n![\"blog-chart-smart-stats-187a\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-187a.jpeg\")
<\/a><\/p>\n
\n![\"blog-chart-smart-stats-187b\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-187b.jpg\")
<\/a><\/p>\nSMART 12: Power_Cycle_Count<\/h3>\n
\nThe number of times the power was turned off and turned back on correlates with failures:
\n![\"blog-chart-smart-stats-12a\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-12a.jpeg\")
<\/a><\/p>\n
\n![\"blog-chart-smart-stats-12b\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-12b.jpg\")
<\/a><\/p>\n![\"blog-chart-smart-stats-12c\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-12c.jpg\")
<\/a><\/p>\nSMART 1: Read_Error_Rate<\/h3>\n
\nThe Wikipedia entry for this one says \u201cThe raw value has different structure for different vendors and is often not meaningful as a decimal number.\u201d So the numeric values here probably don\u2019t count anything directly, but it\u2019s clear from the failure rate chart that they have something to do with drives failing, and that non-zero values are worse. Once the value goes above zero, bigger is not worse, though.<\/p>\n![\"blog-chart-smart-stats-1a\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-1a.jpeg\")
<\/a><\/p>\n![\"blog-chart-smart-stats-1b\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-1b.jpg\")
<\/a><\/p>\n
\n![\"blog-chart-smart-stats-1c\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-1c.jpeg\")
<\/a><\/p>\n![\"blog-chart-smart-stats-1d\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2014\/11\/blog-chart-smart-stats-1d.jpeg\")
<\/a><\/p>\nTell Us What They Mean<\/h2>\n