Hard Drive Stats for Q3 2016: Less is More

November 15th, 2016

Hard Drive Stats - 8TB Drives

In our last report for Q2 2016, we counted 68,813 spinning hard drives in operation. For Q3 2016 we have 67,642 drives, or 1,171 fewer hard drives. Stop, put down that Twitter account, Backblaze is not shrinking. In fact, we’re growing very nicely and are approaching 300 petabytes of data under our management. We have fewer drives because over the last quarter we swapped out more than 3,500 2 terabyte (TB) HGST and WDC hard drives for 2,400 8 TB Seagate drives. So we have fewer drives, but more data. Lots more data! We’ll get into the specifics a little later on, but first, let’s take a look at our Q3 2016 drive stats.

Backblaze hard drive reliability stats for Q3 2016

Below is the hard drive failure data for Q3 2016. This chart is just for the period of Q3 2016. The hard drive models listed below are data drives, not boot drives. We only list drive models that have 45 or more of that model deployed.

Q3 2016 hard drive failure rate chart

A couple of comments on the chart:

  • The models that have an annualized failure rate of 0.00% had zero hard drive failures in Q3 2016.
  • The “annualized failure rate” is computed as follows: ((Failures)/(Drive Days/365)) * 100. Therefore, consider the number of “Failures” and “Drive Days” before reaching any conclusions about the failure rate.

Less is more: The move to 8 TB drives

In our Q2 2016 drive stats post we covered the beginning of our process to migrate the data on our aging 2 TB hard drives to new 8 TB hard drives. At the end of Q2, the migration was still in process. All of the 2 TB drives were still in operation, along with 2,720 of the new 8 TB drives – the migration target. In early Q3, that stage of the migration project was completed and the “empty” 2 TB hard drives were removed from service.

We then kicked off a second wave of migrations. This wave was smaller but continued the process of moving data from the remaining 2 TB hard drives to the 8 TB based systems. As each migration finished we decommissioned the 2 TB drives and they stopped reporting daily drive stats. By the end of Q3, we had only 180 of the 2 TB drives left – four Storage Pods with 45 drives each.

The following table summarizes the shift over the 2nd and 3rd quarters.

Migration from 2TB hard drives to 8TB hard drives

As you can see, during Q3 we “lost” over 1,100 hard drives from Q2, but we gained about 12 petabytes of storage. Over the entire migration project (Q2 and Q3) we added about 900 total drives while gaining 32 petabytes of storage.

Drive migration and hard drive failure rates

A four-fold storage density increase takes care of much of the math in justifying the migration project. Even after factoring drive cost, migration costs, drive recycling, electricity, and all the other incidentals, the migration still made economic sense. The only wildcard was the failure rates of the hard drives in question. Why? The 2 TB HGST drives had performed very well. Drive failure is to be expected, but our costs go up if the new drives fail at twice or three times the rate of the 2 TB drives. With that in mind let’s take a look at the failure rates of the drives involved in the migration project.

Comparing Drive Failure Rates

The Seagate 8 TB drives are doing very well. Their annualized failure rate compares favorably to the HGST 2 TB hard drives. With the average age of the HGST drives being 66 months, their failure rate was likely to rise, simply because of normal wear and tear. The average age of the Seagate 8 TB hard drives is just 3 months, but their 1.6% failure rate during the first few months bodes well for a continued low failure rate going forward.

What about the 60 drive Storage Pods?

In Q3 we deployed 2,400 eight TB drives into two Backblaze Vaults. We used 60 drive Storage Pods in each vault. In other words, each Backblaze Vault had 1,200 hard drives and each hard drive was 8 TB. That’s 9.6 petabytes of storage in one Backblaze Vault.

Each Backblaze Vault has 9.6 petabytes of storage

As a reminder, each Backblaze Vault consists of 20 Storage Pods logically grouped together to act as one storage system. Storage Pods are spread out across a data center in different cabinets, on different circuits and on different network switches to maximize data durability. Backblaze Vaults are the backbone that powers both our cloud backup and B2 cloud storage services.

60 drive storage pod

Our Q3 switch to 60-drive Storage Pods signals the end of the line for our 45-drive systems. They’ve had a good long run. We put together the history of our Storage Pods for anyone who is interested. Over the next couple of years, all of our 45 drive Storage Pods will be replaced by 60 drive systems. Most likely this will be done as we migrate from 3 TB and 4 TB drives to larger hard drives. I hear 60 TB HAMR drives are just around the corner, although we might have to wait for the price to drop a bit.

Cumulative hard drive failure rates by model

Regardless of the drive size or the Storage Pod used, we’ll continue to track and publish our data on our hard drive test data web page. If you’re not into wading through several million rows of hard drive data, the table below shows the annualized drive failure rate over the lifetime of each of the data drive models we currently have under management. This is based on data from April 2013 through September 2016 for all data drive models with active drives as of September 30, 2016.

Hard Drive Stats

Hard drive stats webinar: Join Us!

Want more details on our Q3 drive stats? View the webinar below in the Backblaze BrightTALK channel. You will need to subscribe to the Backblaze channel to view the webinar, but you’ll only have to do that once.

Recap

Less is more! The migrations are finished for the moment, although we are evaluating the migration from 3 TB drives to 10 TB drives. First though, we’d like to give our data center team a chance to catch their breath. The early returns on the Seagate 8 TB drives look good. The 1.6% failure rate at the 3-month point is the best we’ve seen from any Seagate drive we’ve used at the same average age. We’ll continue to track this going forward.

Next time we’ll cover our Q4 drive stats, along with a recap of the lifetime performance of every data drive we’ve used past and present. That should be fun.

Looking for the tables, charts, and images from this post? You can download them from Backblaze B2 as a ZIP file (2.3 MB).

Andy Klein

Andy Klein

Andy has 20+ years experience in technology marketing. He has shared his expertise in computer security and data backup at the Federal Trade Commission, Rootstech, RSA and over 100 other events. His current passion is to get everyone to back up their data before it's too late.
  • Denis Snow

    just a little of information of my experience with Seagate.
    invested to the most referred as “the best” maximum 8Tb.
    got very special data relocated, used 4 times, and failure
    attaching power the device does not start up, no sound…
    Let Seagate Customer Service know, they even did not
    apologise and rudely asked for £500 to start a recovery.
    Let Seller clarify issue, Maplin took drive and returned,
    only said that of I open the drive warranty would be void.
    Data specialist with a certification statement was asked.
    So, I decided just to refund back from criminal trumpists.

    P.S. Do Not Buy Seagate at all. Invest in Toshiba or WD.
    PS. Know that buying Seagate you time trap for all data.

    • Okc Dave

      Can you restate that in English? :)

  • kbarb

    I’m just curious . . . how loud is that drive center anyway, with all those tens of thousands of drives spinning away.

  • OnlyMe999

    What do you do with your old drives? Do you destroy them, sell them on ebay after wiping them, sell them to the manufacturer for them to sell them as refurbished drives?

    • stephendt0

      I’m curious about this. Getting rid of 4000+ drives is no easy task.

  • Tentative Infidel

    Freakin’ awesome data tables. The awesomeness is simply awesome in a slightly awesome sort of way.

  • No WDC 8TB because it’s less than 45?
    Failure means there are at least N bad sector or as far as the disk no longer detected?

  • Chris Carruthers

    Your articles are my go-to for HD stats. Have been since you released the first plans for the Pod. Andy thanks for publishing the work. Happy nerds all around. I do share Damian’s sentiment about the text tables instead of images though ;-) … Either way, love it and keep up the great work!

  • Gabe Anguiano

    I truly appreciate these blog posts and stats but wanted to provide some feedback.

    The first chart is hard to understand, and I did read this: https://f001.backblazeb2.com/file/Backblaze_Blog/hard-drive-stats/computing_failure_rates+copy.pdf

    1) Is Drive Count the number of drives in operation at the end of the quarter or the total number drives used since inception (including failed)?

    2) Drive days, is this the total number since inception? I’m guessing so, since the numbers don’t jive for the 3 month period (drive count * 90 days).

    3) Drive Failures, is this number of failures during the 3 month period or total or for the period of a year? The comment that “The models that have an annualized failure rate of 0% had zero hard drive failures in Q3…” makes it sound like it’s for the 3 month period. If so, then this is a quarterly failure rate not yearly.

    QFR = (Failures within Quarter) / (Cumulative Drive Days in Operation within Quarter) / 65 * 100

    Possible statistics that would be helpful are:

    Mean drive failure point (e.g. 3.5yrs ). The quality of this value would improve with time.

    Mean drive read or write bytes.

    Stats that are adjusted to the drive size, for instance drives that are 2TB are an apple compared to oranges that are 8TB. I would assume a 8TB failure more catastrophic than a 2TB failure.

    Stats broken out by drive technology. I was researching SMR for a NAS which led me to here.

    Thanks again,

    Gabe

  • Wilfried

    Hi Andy,
    I admit I didn’t read all your posts and all comments, but I am wondering a bit about the statistics. If I assume you chosed bigger drives to fill them with data, then I would consider this too. Doesn’t the double capacity result in twice as much accesses (in average). In my opinion this should result in a failurerate/TB to obsere the greater capacity of newer drives too.

    Wilfried

  • Lukas

    Hi Andy, thanks for always posting the harddrive stats with some meaningful paragraphs.

    One thing that i want to ask, did you still publish entire raw smart log .csv files?
    I’m really interested to observe the old drives failure rates (e.g 2 years old Seagate 4TB), because recently Backblaze got a bunch a new drive, that might still working strong.

    I’m afraid that seagate’s lifetime still short like the 3TB version, which running quite good for the first 2 years, and fails horribly on the third year.

  • 한대희

    Hi, I’m currently implementing survival analysis on 2016 Q3 data. What bothers me is that some drives do not have record even though they do not have failure before.
    For example, by considering only the indicator for failure, some drive has
    07-01 07-02 07-03 07-04 07-05 07-06
    0 NA NA NA 0 1

    Does these records are only the matter of records? (Like missing data of S.M.A.R.T data) Or is there any other reason for missing data? (Like scheduled maintenance)

    Thank you

  • xcopy

    Hi Andy,

    Thanks for (consistently) publishing this data.

    It’s increasingly hard for home users to get real hard drive reliability data today. It seems as if the manufacturers are more interested in engineering their products to barely make the warranty date before failure, as opposed to producing products that customers can rely upon.

    My experience with Seagate has not been good. They have always failed early and often (nearly 100% failure record for me) but I know my sample size is very small compared to yours, and I would be willing to reconsider them if they’ve changed their ways. However, it appears that the Seagate 6tb drives you’re using are no longer available, or is the “st6000DX000” a placeholder for something else?

    Here’s a link to the only reference I could find, indicating it’s a desktop drive which may not be suitable for a NAS.
    http://www.newegg.com/Product/Product.aspx?Item=N82E16822178737

    I’ve been thinking of buying the WD 6TB Red NAS drives, but it looks like the failure numbers are the highest you have. Is this just another WD product to avoid, or is there something else that could have caused the increased number of failures?

    Lastly, any thoughts on the WD Red Pro line vs HGST 6TB drives?

    Thanks, and I’m rooting for you guys.

    • Tom Gabriele

      I don’t remember exactly, but I think it was mentioned in one of these reports a while back that they weren’t observing a significant difference in failures between consumer/desktop drives and enterprise/NAS drives, so they just buy whichever is cheapest.

    • coolspot

      If you have a 100% failure rate with Seagate … it must be you, because even Backblaze doesn’t come close to this number!

      • xcopy

        Actually, I think you need to read a little more carefully before replying. I said “nearly” 100%, and that’s not unusual from what I read from others. I’ve got one or two Seagate drives now (out of 35+ drives), but I don’t trust them, and Seagate has had more problems than anyone else in my 30 years of hard drive usage, but that’s my experience.

        And as for “it must be you”, what can anyone say to such an ignorant comment?

        • Brian Nichols

          Well… I’m a professional software engineer who’s been building systems since the IBM XT days (back when Seagate was king of the 40mb MFM drives). So I hope I don’t suffer the same slam as #xcopy did..

          I will add to his comment, however, and say that I personally will NEVER put another Seagate drive in ANY of my systems. My experience has run par with his, it seems. WD use to be awesome, but now they’ve gone down hill (albeit slowly). I panicked somewhat when they purchased Hitachi’s HDD division, but it seems they kept their manufacturing technology. The HGST drives (specifically the UltraStore) seems to be the best drive, so far, for my money. I have 8 spinning 24/7 since 2014 with 0% issues… they replaced 3 WD blacks, and 2 WD Greens (which seem to have the worst failure rate, in my experience).

          I hope the HGST reputation holds, as I am getting ready to purchase 4 more.

        • Okc Dave

          Obviously you’re getting defensive instead of accepting that you’re creating a bad environment that causes excessive failures.

          Further you’re either lying or an idiot because nobody would keep buying Seagates for 30 years if they kept having them fail prematurely.

    • Bernhard Schindler

      The problem with applying this data to desktop users is that these are enterprise drives. If you look up the model numbers, you’ll see that they’re not what the desktop user would usually buy. While it “may” be that these two segments share some or all properties, it’s not necessarily so. Even from one revision / size to another, there can be substantial changes (which is also highlighted by Backblaze if you read their comments).

      From anecdotal evidence, I can only share the same experience you have had: Desktop Seagate plates seem to fail more often than other manufacturers. This has already been the case 10 years ago and still seems to be.

      Also worth mentioning, there has been a substantial consolidation in the HDD market going on for quite some time. HGST is now a subsidiary of WD. Only Seagate, WD and Toshiba remain as “parents” on the market. The rest are just brands in a portfolio of the aforementioned.

      However, in the end, the info here is better suited for companies that calculate in replacement ratios rather than anybody seeking that bullet-proof HDD – which doesn’t exist. No matter if a drive has an annual failure rate of 1% or 2%, if that is YOUR particular drive in your desktop, you’re screwed. Only back-ups help. Best on another machine or external drive. Employing RAID 1 is no backup.

  • Techwolf Lupindo

    I notice that Temperature and Vibration isolation is missing from the report. Those two missing items can make a difference in drive reliability. Are they available?

  • John S. Mills

    Andy, I realise your use of harddrives is much different from mine, but I’m still curious about your opinion. What are your thoughts on reliability on the drives you use for deep cold storage? That is, drives that get popped into a workstation for backups, and then be put back on a shelf (roomtemp, 50% humidity, little variance). While these drives are tested for a number of hours before being deployed to weed out the doa’s, the main reason of concern for me is the longevity of the magnetic recording.

    I would like to start using 8-10TB drives, but this means the amount of magnetic particles storing a bit is much lower than the current 2TB drives (3 and 4 platter models iirc). Sofar these old drives have held up for +10 years, but the storage density of the new models makes me nervous. And unfortunately, I have found very little information about this on the internet.

    So I’m curious about your view on this.

    • ZeDestructor

      I discussed this with a few people over the past few months, and my personal conclusion is that bitrot is too real an issue these days to have truly cold storage outside of tapes: you simply need the regular scrubs with an amount of redundancy (even on the backups).

      For me, the solution of choice right now is to mirror my main ZFS array to a secondary off-site box and rely on snapshots to contain any accidental deletion/cryptolocker/whatever failures to the main/live copy only.

      • John S. Mills

        For us, this would be unaffordable, alas. We’re currently backing up raw footage of 200TB on harddisks onsite and another duplicate offsite.
        How bad do you estimate the bitrot issue is? Would a parity system (using something like Multipar for Windows to create a percentage of parity files on the same disk) be enough or would a modern Seagate Ironwolf 10TB after residing on the shelf for 5 years have thousands of flipped bits?

        • ZeDestructor

          Heh.. there’s a reason a lot of recoverable stuff isn’t backed up for me – that would be price.

          As for how prevalent bitrot is, I have no clue how prevalent it is, but I do know it hit me twice. The first time was by far the more annoying: it hit my filetable and that ruined tons of data. The second time, it ruined a few family pictures, but that was fine cause my 4 other copies were intact. 2 disks out of 12.

          I say that’s enough evidence to say that bitrot is real and has happened to me, but useless for predicting prevalence.

          that said, it could be something else, but bitrot fits the symptoms by far the best.

          EDIT: As for stuff like multipar, it could work. I personally just leave it all up to ZFS because ZFS has several design choices that make it work just fine to counter bitrot: regular scrubs, ability to repair bad blocks from parity data, all while being completely transparent to any users/clients not following dmesg all day.

  • Seriously, folks, why don’t you post *normal* tables? Can’t copy HDD model names from damn images into Amazon or Newegg.

    • Damian:

      “Looking for the tables, charts, and images from this post? You can download them from Backblaze B2 as a ZIP file (2.3 MB).” The link is -> https://f001.backblazeb2.com/file/Backblaze_Blog/hard-drive-stats/Q3_2016_Drive_Stats_Materials.zip.

      • All I’m saying is build tables in HTML instead of images for everyone’s convenience.

        • inhumantsar

          Thank you for taking a bunch of time (you didn’t have to take) to generate a report (which your competitors could use) that I find so useful that I base my purchasing decisions on it.

          By the way, I’m too lazy to type out four hundred characters worth of model numbers so could you code it into HTML instead of just screenshotting your spreadsheets?

          FTFY. Asshat.

          • Jani

            He may have come across rude, but he’s got a point there, which you seem to have completely missed and instead even somehow took it personally. What an asshat.

          • inhumantsar

            Didn’t miss it or take it personally, I mocked it.

        • inhumantsar

          oh man, apparently i opened your profile in a tab and forgot about it. just rediscovered that now and saw that you posted the exact same entitled comment on another one of these reports.

          sorry for calling you an asshat, but really.

          • Com Ment

            Nothing wrong with asking 2 times for the same improvement when the latter hasn’t been implemented.

    • I’m sure they’d get more inbound organic search engine traffic if people Googling those part numbers stumbled on the site. This is a WordPress blog after all.

    • gavingreenwalt

      Can’t share an html table with someone easily. I can copy/paste an image into a chat, web forum or social media post though.

      • Okc Dave

        because you’ve never seen a keyboard with a Print Screen Key? I suggest that your computer literacy should start there rather than telling other people something that a web link can do just fine without taking up more server bandwidth every time it loads.

        • gavingreenwalt

          Yeah people just have to hit print screen, open it up, crop it, upload it to a file hosting site and then link it. Or they can right click and hit copy.

    • FredWallace18

      Fortunately for you, it takes all of 5 seconds to type a model number into one of these sites from these statistics they’re giving you for free.

      • cyrylthewolf

        @Damian Nowak ^BOOM.

  • Earl Garber

    It would be interesting if failures were broken down into categories, i.e. mechanical, electronic, and media.

    • How would you get this data from most failed harddrives? And different manufacturer firmware certainly handles its faults differently.

      Would be nice to get the data if there was a practical way to get the failure mode.

    • coolspot

      What does it matter – a failure is a failure? Most people won’t repair their hard drives, so such a detail isn’t relevant to the big picture?

      • Børre Børresen

        failure is a failure.
        but if its like heads or engine that goes to smitterns compare to say dead sectors i would have chosen the later one as an consumer because then i might save the rest of the disk to an replacement.
        if first 2 things happens its dead… very dead.
        and to save whats left it then needs a trip to a place like IBAS and then you get a 2000$+++ bill afterwards

      • Okc Dave

        It matters because you can track which bit of tech is the weakest link and whether they’ve changed it in successive models.

  • Matt Viverette

    Are you selling the 45-drive pods at a discount (without the drives, of course)? I know you’ve shared some of the designs, but they’re probably still useful machines.

    • Andy Klein

      We do have some empty chassis (V1 – V3?), is that interesting? Or do you want the rest of the electronics.?

      • ZeDestructor

        I don’t know about Matt, but for I’m personally perfectly fine with just the empty chassis (electrnics, especially the PSUs would be nice though).

        Direct wire is a must though, cause I expect to use em with SAS HBAs and expanders. On that note, have you considered direct wiring to a custom port multiplier chassis instead of boilting the multiplers directly under the drives?

      • Matt Viverette

        Yes, that’s interesting to me. I’d want the backplanes, though. Seems like they are an integral part of the chassis.

      • Sondre Engebråten

        This would be really interesting to me as well. I can’t really justify the cost of a new gen 6 pod for personal use, but a used one might fit the budget. Please get in touch if you have more left and an idea of what you would like to get for them.

        I would use them as a personal NAS solution, as such I am interested in at least some of the electronics as well.

        Best
        Sondre

      • John S. Mills

        Andy, out of curiosity, what did you do with the 2TB HGSTs? Put to other use, scrub & sold or melted down in Mount Doom?

  • Bill Waslo

    Now I see this. I just bought and installed 2 of the WD Red WD20EFX drives in a RAID. 8.2% annualized failure rate!?! Should have bought the cheaper HGST…..

    • You’re probably going to be fine! Our environment is different than most…homes. Just make sure you have a good backup strategy ;)

  • gcstang

    Not sure I’m reading the tables correctly but in the 8TB range is it saying the Seagate is more reliable than the HGST ? I’m looking at purchasing some drives for a personal NAS and don’t want to have to turn around and purchase more drives, I’d like them to last a while (reliable, etc…)

    Thank you for your posts

    • At the moment, yes – but we have less of the HGST drives in production. In all likelihood you’re fine either way since our environment is much different than what you’d likely have at home.

      • John S. Mills

        Are you purchasing the new types from Seagate (Ironwolf, Barracuda Pro) with helium? The ones listed in your tables are becoming hard to find already, and the new series are technically different, so I wonder how well the new Seagates compare.

  • Vince

    Interestingly the WD20 series we found so unreliable they almost all failed and the last few we removed before they also did. Whereas the WD10 series are ridiculously reliable and so few have failed we can’t recall it happening.

    So far experience with WD30 good, not enough of the WD40 or WD60 to know long term but the WD60 series has a high initial dead on arrival count, but if they survive a few weeks they seem OK.

    We are cautiously optimistic about trying the Seagate 8TB and 10TB series and your initial finds seem favourable so we might just give it a whirl – thanks for continuing to publish your data.

  • Brandon Kruse

    We have 3,024 8TB Seagate drives (non-SMR/non-archive) and have had a slightly higher overall failure rate than the 6TB. We’ve had them for about 9 months now. We’ve had a significantly higher INITIAL failure rate (failure within less than a week), but a lower annualized failure rate for the last 9-months of data that we have. This could have just been a batch problem with the drives we received, as they were very close to first off the line.

    I suspect as we continue to operate, that the annualized failure rate will win out past the 6TB drives, but the initial failures put it a little behind.

  • morganf

    In the last table, the 2TB WD drive has the upper AFR interval listed as “0.136” instead of “13.6%”

    • Andy Klein

      Formatting error – I’ll fix it shortly. Thanks.