Hard Drive Stats for Q2 2016

By | August 2nd, 2016

Hard Drive Reliability

Q2 2016 saw Backblaze: introduce 8TB drives into our drive mix, kickoff a pod-to-vault migration of over 6.5 Petabytes of data, cross over 250 Petabytes of data stored, and deploy another 7,290 drives into the data center for a total of 68,813 spinning hard drives under management. With all the ins-and-outs let’s take a look at how our hard drives fared in Q2 2016.

Backblaze hard drive reliability for Q2 2016

Below is the hard drive failure data for Q2 2016. This chart is just for the period of Q2 2016. The hard drive models listed below are data drives (not boot drives), and we only list models which have 45 or more drives of that model deployed.

Q2 2016 Hard Drive Failures RatesA couple of observations on the chart:

  1. The models that have an annualized failure rate of 0.00% had zero hard drive failures in Q2 2016.
  2. The annualized failure rate is computed as follows: ((Failures)/(Drive Days/365)) * 100. Therefore consider the number of “Failures” and “Drive Days” before reaching any conclusions about the failure rate.

Later in this post we’ll review the cumulative statistics for all of our drives over time, but first let’s take a look at the new drives on the block.

The 8TB hard drives have arrived

For the last year or so we kept saying we were going to deploy 8TB drives in quantity. We did deploy 45 8TB HGST drives, but deploying these drives en masse did not make economic sense for us. Over the past quarter, 8TB drives from Seagate became available at a reasonable price, so we purchased and deployed over 2,700 in Q2 with more to come in Q3. All of these drives were deployed in Backblaze Vaults with each vault using 900 drives, that’s 45 drives in each of the 20 Storage Pods that form a Backblaze Vault.

Yes, we said 45 drives in each storage pod, so what happened to our 60 drive Storage Pods? In short, we wanted to use the remaining stock of 45 drive Storage Pods before we started using the 60 drive pods. We have built two Backblaze Vaults using the 60 drive pods, but we filled them with 4- and 6TB drives. The first 60 drive Storage Pod filled with 8TB drives (total 480TB) will be deployed shortly.

Hard Drive Migration – 85 Pods to 1 Vault

One of the reasons that we made the move to 8TB drives was to optimize storage density. We’ve done data migrations before, for example, from 1TB pods to 3TB and 4TB pods. These migrations were done one or two Storage Pods at a time. It was time to “up our game.” We decided to migrate from individual Storage Pods filled with HGST 2TB drives, average age 64 months, to a Backblaze Vault filled with 900 8TB drives.

Backblaze Data Migration

We identified and tagged 85 individual Storage Pods to migrate from. Yes, 85. The total amount of data to be migrated was about 6.5PB. It was a bit sad to see the 2TB HGST drives go as they have been really good over the years, but getting 4 times as much data into the same space was just too hard to resist.

The first step is to stop all data writes on the donor HGST 2TB Storage Pods. We then kicked off the migration by starting with 10 Storage Pods. We then added 10 to 20 donor pods to the migration every few hours until we got to 85 pods. The migration process is purposely slow as we want to ensure that we can still quickly read files from the 85 donor pods so that data restores are not impacted. The process is to copy a given RAID-array from a Storage Pod to a specific “Tome” in a Backblaze Vault. Once all the data in a given RAID-array has been copied to a Tome, we move on to the next RAID-array awaiting migration and continue the process. This happens in parallel across the 45 Tomes in a Backblaze Vault.

We’re about 50% of the way through the migration with little trouble. We did have a Storage Pod in the Backblaze Vault go down. That didn’t stop the migration, as vaults are designed to continue to operate under such conditions, but more on that in another post.

250 Petabytes of data stored

Recently we took a look at the growth of data and the future of cloud storage. Given the explosive growth in data as a whole it’s not surprising that Backblaze added another 50PB of customer data over the last 2 quarters and that by mid-June we had passed the 250 Petabyte mark in total data stored. You can see our data storage growth below:

Backblaze Data Managed

Back in December 2015, we crossed over the 200 Petabyte mark and at that time predicted we would cross 250PB in early Q3 2016. So we’re a few weeks early. We also predicted we would cross 300PB in late 2016. Given how much data we are adding with B2, it will probably be sooner, we’ll see.

Cumulative hard drive failure rates by model

In the table below we’ve computed the annualized drive failure rate for each drive model. This is based on data from April 2013 through June 2016.

Q2 2016 Cumulative Hard Drive Failure Rates

Some people question the usefulness of the cumulative Annualized Failure Rate. This is usually based on the idea that drives entering or leaving during the cumulative period skew the results because they are not there for the entire period. This is one of the reasons we compute the Annualized Failure Rate using “Drive Days”. A Drive Day is only recorded if the drive is present in the system. For example, if a drive is installed on July 1st and fails on August 31st, it adds 62 drive days and 1 drive failure to the overall results. A drive can be removed from the system because it fails or perhaps it is removed from service after a migration like the 2TB HGST drives we’ve covered earlier. In either case, the drive stops adding Drive Days to the total, allowing us to compute an Annualized Failure Rate over the cumulative period based on what each of the drives contributed during that period.

As always, we’ve published the Q2 2016 data we used to compute these drive stats. You can find the data files along with the associated documentation on our hard drive test data page.

Which hard drives do we use?

We’ve written previously about our difficulties in getting drives from Toshiba and Western Digital. Whether it’s poor availability or an unexplained desire not to sell us drives, we don’t have many drives from either manufacturer. So we use a lot of Seagate drives and they are doing the job very nicely. The table below shows the distribution of the hard drives we are currently using in our data center.

Q2 2016 Hard Drive Distribution

Recap

The Seagate 8TB drives are here and are looking good. Sadly we’ll be saying goodbye to the HGST 2TB drives, but we need the space. We’ll miss those drives, they were rock stars for us. The 4TB Seagate drives are our workhorse drives today and their 2.8% annualized failure rate is more than acceptable for us. Their low failure rate roughly translates to an average of one drive failure per Storage Pod per year. Over the next few months expect more on our migrations, a look at the day in the life of a data center tech, and an update of the “bathtub” curve, i.e. hard drive failure over time.

Andy Klein

Andy Klein

Director of Product Marketing at Backblaze
Andy has 20+ years experience in technology marketing. He has shared his expertise in computer security and data backup at the Federal Trade Commission, Rootstech, RSA and over 100 other events. His current passion is to get everyone to back up their data before it's too late.
Category:  Cloud Storage
  • HARD DISK PROTECTOR INDIA

    HD Protector is really well-done and meets all the demands concerning Hard Disks. I have recommended HD Protector Software for all data storage computers.
    visit Here..
    https://www.harddiskprotector.com/

  • strawhat64

    Thanks to the 2014 study I decided to buy a Seagate ST4000DM000 and it has been almost 2 years since then. I usually have my PC on for around 16 hours a day. I have a certain problem from time to time where the drive will disconnect itself when it tries to read data or on startup but unfortunately the smart tests I try with various programs show nothing. The temporary fix for this problem is to simply change SATA cable, I believe I have done this 4-5 times so far so I am not sure if I should classify the drive as bad or if the problem lies elsewhere. Perhaps it could be because my PC is old? It is over 7 years old, or could it be a PSU issue? Those are undetectable after all.

    If I get enough money again I believe I will go for a Hitachi one this time. You pay more but you get what you pay for in this case.

    All in all I am not particularly unhappy with the 4TB Seagate I bought but since I do not know why the problem occurs I will not trust Seagate too much. I once bought a 3 TB Seagate drive and it failed within 2-3 months. I remember it was surprisingly cheap as well.

  • AZZAZ RACHID

    hellos. I am a PhD student in Statistics at High National School of Statistics and Applied Economics-Algeria-.I’ve been following your works since 2015, i always read your updates . because My research theme is on the statistical models in reliability. where the practical part of my thesis deals with the application of a mixed model on HDD reliability Data in order to get more modeling flexibility , so I have prepared your data by serial numbers in order to follow every hard drive separately . Note i want to use your data, but I need a written permission from your laboratory. This result as a request of my supervisor who insist on the source and reliability of the data used in my research. So I have the honor to ask you to help me by sending me an authenticate authorization which conclude the permission and notes on the reliability of your data.i will send you all results that i will reach it , and a copy of my phd thesis when i finish it.
    thank you for your helps ,
    Thank you for your good work
    warmest greetings from me and my supervisor.

  • indyconservative

    What’s with all the deleted comments below?

  • schabernack

    There is an error in your table here…

    http://i.imgur.com/PQN7Bs5.jpg

  • Braed

    Holy shit that’s a lot of drives, and a lot of data. 250PB makes my PC storage look microscopic haha.
    Nice article! :)

  • AlainCl

    The Seagate 8Tb stats looked decent, and I ended up being unable to turn down a $199 special on the external 8Tb Seagate Backup Plus Hub to use to back up my internal and external drive.

    After a few weeks no problems, quiet action, though a little slower than I’d hoped. As always fingers crossed.

  • So what would you recommend for a new Synology NAS system?

    • Qualified Expert

      You can use this HDD from Hitachi:
      HGST Deskstar NAS 3.5-Inch 4TB 7200RPM SATA III 64MB Cache Internal Hard Drive Kit 0S03664

      So far I’ve been using it in my QNAP NAS very quiet and fast response as well.
      HTH..

  • WoodyTX66

    A number of years ago, drives with an odd number of platters had a poor reputation for durability (750 GB, 1.5 TB, etc). I see the 3TB drives are less than stellar, but does this reputation continue with other odd-platter sizes?

  • Nigel Fisher

    Comparison of runtime and failure rate by model in a readable chart:

    http://i.imgur.com/AIT9Oe7.png

    http://i.imgur.com/eaxVrnZ.png

    • Qualified Expert

      Nigel,

      Does this means WD is better than Hitachi due to the lower rate of failure ?

      • Nigel Fisher

        WD shows a much higher failure rate as well as lower sample size than Hitachi in those images. Please note the coloring of the legend.

  • schabernack

    Love these articles and reports. Very very helpful. Since I wanna get a local backup, I wanted to get the “best” 4TB HDD. Seems the HGST HMS5C4040ALE640 is no more available to buy. Is the HMS5C4040BLE640 the follow up?

    • Qualified Expert

      Hi Schaber,
      I’m using the HGST Deskstar NAS 3.5-Inch 4TB 7200RPM SATA III 64MB Cache Internal Hard Drive Kit 0S03664 from Amazon, it is a good drive in terms of performance, pricing and the noise level.

      • schabernack

        Thanks for getting back. I think I might get one of these for my NAS.

  • Hanno Hugenberg

    Are the burn-in/test-cycle failing drives added to the total failure count or are only production capable drives counted?
    If the burn-in/test drives failures are included, what is their percentage of the total drive failures?

  • Wow! Western Digital failure rates just shocked me!
    I can’t believe that WD failure rates are so high, also surprised by Seagate 6TB drive, the failure rate seems to be to good to be true.

    • FelipeGR

      Same here, in my experience, I’ve had about 50/50 WD/Seagate failures, so it probably depends on if you happen to get the unlucky drives, or a bad lot. Obviously there’s been recorded cases like some models of Seagate that had excessive failures, or WD Greens failing when used in a NAS or other heavy duty environments (some external HDD users).

      Since a number of those are 2TB drives, I wonder if they are simply older drives failing due to age.

  • Good to note I was aware of this tech data of yours a long time before becoming a client. As a photographer, I had some troubles a couple years ago using consumer line of hard disks both from Seagate and WD. I didn’t hear about HGST before reading this blog, to be honest.

    Now I’m into a Seagate surveillance line that is attending me ok but I’m buying an HGST (…)BLE640 4TB drive and I think it’s gonna be a nice experience to work with it. Happy to see this choice is Backblaze’s second most used and one of the lowest failure rates.

    I’m also impressed by the 5+ years of the 2TB drives, I was used to a year and a half with my consumer drives.

    Great jobs guys, happy to become a client of Backblaze B2.

  • Robert Hollar

    Can you post detailed data on your enclosures? Id like to also inquire about vibration mesurments. From the looks of this your publishing half truths especially on seagates end. By using non vibration rated desktop drives in what im guessing is going to be a server enclosure.

  • Caleb Stone

    Google had some data made public a couple of years ago that showed most HDDs fail within the first 18 months.
    After that time (or maybe it was 24) period most Drives seem to be invincible.
    Do you have made similar experiences?
    Nice data anyway, thx for sharing.

  • James Moore

    A thought occurred to me: if you laid the drives in board-to-board pairs, so the platters spun in opposite directions, would that reduce vibrations, hence operating temperatures?

  • James Moore

    just had a “2001” moment reading this:

    “My God, it’s full of porn!”

  • How about adding a column to your table, Age at failure (days). You could add the confidence intervals, too.

    • Andy Klein

      We sometimes publish a drive age table, I’ll try to add in the next time. The cumulative table has confidence internals, but we don’t add them to the quarterly table as the time span is too short to give a decent range.

  • TechTony

    Some things I wonder about…
    With the 8TB drives, don’t they have more spare blocks to use than lower capacity drives? Perhaps when the 8TB drives run out of spare blocks, then the statistics will become much different (i.e., perhaps they are already failing but not dead yet). Does Backblaze monitor and analyze the SMART data?

  • ManOnTheHill

    Have you done any positional analysis – that is, are there places in the pod that are harder on drives than other places? The analysis would be a little complicated (lots of confounding factors that would hide the actual failure rates by slot), but it might give interesting insights into ways to improve the physical design of the pods.

  • GuitarJam

    I wouldn’t trust a 8TB drive. I don’t care who made it.

    • jp

      I keep data on two drives and maintain a local and off-site backup. The 8tb archive are good enough for my media server

      • GuitarJam

        I have four 2TB drives in a RAID. if one fails everything is recovered. putting everything on one 8TB drive is stupid

        • Robert

          Obviously you dont trust the 2TB drives you have either so whats your point? If someone has 2TB drives in a raid then obviously they would have the 8TB drives in a RAID also. You can’t compare apples and oranges, the comparison has to be equal.

  • Paul Reay

    Is it fair to stack up failure rates of drives with differing capacities, perhaps the larger drives haven’t failed because they haven’t had to overwrite data as frequently as smaller ones

  • Joshua Brown

    These storage metrics are awesome, glad you guys keep publishing these. Since the big three have begun selling their new 10TB drives, when, if ever, do you guys expect to start deploying them?

  • Chris Parkin

    Great post but what are your thoughts on using the Seagate 8TB Archive drives?

    • Andy Klein

      The Seagate 8TB Archive drives use SMR technology to store data. To simplify, SMR allows for data tracks to overlap at different levels. This is great for recording large blocks of data and reading them back – archiving surveillance data for example. It becomes an issue when you delete data and then want to rewrite new data into the blank space. To do that, the overlapping data above must be “moved” so you can delete and write the new data.
      You can choose to simply break the pointers to the data, but never physically delete the data. This uses more space, but is more time efficient. Our system tries to optimize space, by recovering deleted space for reuse, so using SMR drives would be less than optimal for us.

  • BT

    Built a few systems lately and have moved away almost entirely from WD after a stint of 4 bad drives in a month (all purchased new and used for about 1-3 months a piece) and have started recommending HGST drives. Seagates will always be on my bad list for their noise levels and “crunchy” read noises that freaks me out to no end. Thanks for the updated list Backblaze!

  • Leigh de Paor

    What will you be doing with the retired 2TB drives? Do you have a second-hand drive store?

    • James Moore

      probably not a good idea, since we’re talking about user data. Also consider their age and the conditions they’ve been through (heat). You couldn’t warrant that kind of risk in hardware or data integrity, or the slim chance that old user data might be reconstructed and used for nefarious purposes.

      (have had experience in data recovery and secure, permanent deletion – which is only practically achievable through shredding the platters)

      • Leigh de Paor

        Thanks for the response James, I was hoping to provoke a discussion.

        Some people may consider buying these drives despite age and use conditions if the price was right.
        Permanent deletion of data on drives using self encryption only requires a simple change of the key but would require the implementer to have considered this in advance and understand how to change the keys. Of course this approach can also save on the expensive platter shredding process.

        See http://www.seagate.com/gb/en/solutions/security/data-center-security/ for more information.
        “Seagate Instant Secure Erase (ISE) is designed to protect data on hard disc drives by instantly resetting the drive back to factory settings and changing the encryption key so that any data remaining on the drive is cryptographically erased. This means all data on the drive is permanently and instantly unreadable.”

        • James Moore

          nice, I didn’t know about this (having pretty much just worked with COTS drives), but it doesn’t appear that ISE is available on retail hardware.

          • Leigh de Paor

            I was hoping that Seagate might migrate the technology across the full range, but the idea is sound and could be done using software based FDE.
            Simply removing the disks from the system used to encrypt them would render the data practically unrecoverable.

          • bmurphr1

            Forensic studies have shown that the right people could still copy an image of the previously stored files on the drives using the right tools, even after using something like DBAN to perform a seven-layer wipe that just write garbage to the drive and then zeroes it all out to make the drive blank like it came out of the box. Granted we’re talking millions of dollars worth of equipment just to recover the data from one of these drives, and the length of time it would take to rebuild an image from an old hard drive, but storage platters have “memory” in the sense that you could examine each bit and tell if it was a 1 or a 0 at least 5-10 cycles ago and use that same consistency to rebuild the entire image of the drive. Your average Drive Savers-type facility can’t even do this, but a higher tech company could. The drive could be sold obviously, but I’m not sure if I would sell an entire RAID pool’s worth of disks to the same person just out of security. Selling the drives as individual drives somewhere like eBay and swapping in drives from different pools for large orders could mitigate a lot of the issues and make this much more challenging. It would be a great way to recoup some of the losses of buying new HDs and keeping the facilities running tightly, but if security is your number one concern against data theives there’s only one solution…destruction. Hammers all the way to mill presses that just push HDs through mills like paper-shredders that jab holes through the entire drive would be the answer to that one…just search YouTube for hard drive destruction.

          • Leigh de Paor

            Yes bmurph1, if anyone was determined and rich enough (NSA for example) they could recover data pools from a magnetic HDD, however the residual field strength would be very, very low. If the disks had been encrypted using FDE then even after spending billions trying to recover the 1’s and 0’s then they’d have to figure out the decryption keys etc. Theoretically if they had unlimited Quantum computing power they could do this but it’s so far fetched as to be impossible to all intents and purposes.
            There’s also the RAID stripes and the second layer of user encryption on the data which would mean that recovering the data would require all the drives in the particular raid set, hugely expensive machines and theoretical physicists on tap followed up by unlimited fast computing power to decrypt a double layer of encryption, assuming the 1’s and 0’s recovered were meaningful. So the key to being able to re-use disks is Full Disk Encryption. On SSD devices the residual magnetic effect is not an issue but many of these are now using 256 AES encryption as a standard. – Check out “BestCrypt” volume encryption for an example.

          • bmurphr1

            You underestimate the power of the government…if they wanted they could subpoena all of the hard drives leaving that facility because they would be an easy target compared to some place like Dropbox or Google Drive of all places. There are so many vulnerabilities with placing the drive in someone’s hand that it’s ridiculous, not to mention the access to all sorts of gadgets that the layman has never seen or even heard of in his entire life. I can even bring up a simple argument that stands out…look at the Pegasus attacks. Vulnerabilities that existed for years before anyone outside of a small group of people had access to were discovered, and not only hid it from Apple but the world as a whole until they were discovered by complete accident. Using any FDE is pointless as RAM can be “frozen in time” and then read back using a special machine that can read 99%+ of the entire RAM storage set before it has time to thaw out and return back to normal…by then they have an image of the entire memory mappings and the FDE password or decryption string just sits there…all of that information is already public knowledge and one of the reasons why TrueCrypt was deprecated. They couldn’t cipher a password in memory to the point that it didn’t eat up CPU cycles all of the time just maintaining a salted copy of the original password without using up real horsepower to do so. BitLocker is even easier to bypass.

            Nothing like feeding your drive to a metal grinder and pulverizing it until it looks like a shark mistook it for chewing gum.

          • Leigh de Paor

            Would you not then be in contempt of court or charged with perverting the course of justice for shredding drives that have been the subject of a subpoena? Nobody but an idiot would argue that shredding the drives is the only real 100% guarantee against data recovery, however to all practicality a single or double layer of 256 bit encryption with key deletion is currently unrecoverable.

          • bmurphr1

            You’re gasping for air at this point for air right about now…search for “freezing RAM bypasses all disk based encryption” and have at it. Courts could subpoena a hard drive any time but still never gain access to it’s content because those court systems don’t have the abilities the higher-up organizations do, especially if it’s related to terrorism but can be done if caught the right way. It sounds like you’ve never taken a cryptography course in college, otherwise you’d know some of these things. A dedicated machine that is designed to destroy the hard drive by sheer brute force combined with a heck of a jolt of magnetism could virtually render any hard drive useless. I’ll even give you a hint why I know these hard drive destroyers exist…I used one at my previous job when I replaced 2TB hard drives with the exact same model. Super degaussing magnetic wand plus mechanically trashed and 20 seconds later you have a dead hard drive with no hopes of ever reading any data off of it. Even the NSA does this, and that’s why it’s a popular method of permanently decommission hard drives.

          • Gesta Non Hats

            “a jolt of magnetism could virtually render any hard drive useless”

            Unless It’s a SSD(?)

          • Robert

            A SSD is not a HARD DRIVE, its a solid state drive.

          • Gesta Non Hats

            ouch…. lil loud
            But quite my point, SSD’s are impervious to magnetic erasure as well as …free wheelin use of caps.

          • James Moore

            no, they’re not. Trust me on that.

          • Gesta Non Hats

            Could you explain, as I like to learn the nuts n bolts of how’s n why’s

    • Expecting an interesting discussion? You got it… had some joy reading what came after, thanks hehe

  • BraveLabrador

    Seagate seems to have picked up the pace and actually besting WD. Good for them.
    But for now, HGST is by far the way to go.

    • GuitarJam

      Seagate is garbage

      • Wibla

        The numbers disagree with you.

  • Antonio Spadim

    Hello, just a curiosity or even a newbie question, when you move this huge amount of date could you describe what are the software tools you use to perform this migration, to control the progress of the migration, and to ensure that the data that was migrated is ok on new pod? Thank you for all and keep with awesome job! =)

    • Andy Klein

      All our migration tools are home-grown.

      • Antonio Spadim

        ok tks

  • Thrift Man

    What are your thoughts on the future use of SSDs over the traditional spinning discs. Do you perform regular post mortems on your failed drives and if so, are you able to determine the rate of mechanical failures which SSDs would eliminate?

    • Andy Klein

      That’s a whole blog article all by itself, but to keep it short, SSDs are still too expensive per GB to replace the HDDs in our environment.

  • Techwolf Lupindo

    Interesting stats there. But it has me wondering about what temp you run the drives at. From this articial and comments, it appears Blackblaze is going for densest data storage, meaning the drives are pack in very tight and therefore poor cooling. It does not matter if the data center is a cool 70 degrees if the heat generated by the drives is not taken away and therefore the drives literally cook at 120 plus degrees. My experience is some drives have low failure rate at high temps, but high at cool temps, but the ones that fail at high temp have zero failures at cool temps.
    Another question is does Blackblaze employed vibration isolation. I have read quiet a few stories of hard drives fail and the replacement failed soon after only to discover it was the next door drive that was vibrating a lot causing the failures of next door drives.

    • Andy Klein

      Heat: The drive stats we pull include the max temperature of each drive (SMART 194, I think). The last time I looked through the data only two of the drives had ever exceeded the manufacturer’s range, both failed. Most of the drives live at about 80 degrees F.
      Vibration: Over the years, we’ve worked hard to reduce individual drive vibration, but it can happen. Our detection software is pretty good at finding the real culprit, but if we have a question we remove both drives, test them off-line and let the good one go back into service.

  • Paul van den Bergen

    Another interesting way to display the disk failure performance would be a plot of time to failure for a given model…. it would expose things like early failure populations versus “normal” cumulative failure modes – what ever normal is…

  • disqus_00YDCZxqDV

    always enjoy reading your detailed tech articles. And well done for being an independent tech firm doing hard core shit.

  • Looks like HGST are still great drives.

    • Emmanuel Goldstein

      Too bad they’re almost impossible to find.

      • GuitarJam

        I have 3. buy them online

  • George

    Is it weird I really like reading these Backblaze storage stats posts? Keep it up! Onto a non-rhetorical question: with the capacity of hard drives continuing to increase, would it make sense to also measure failure in terms of failures/TB or failures/platter?

    • It’s not weird :P And doing analysis on failure by platter or per TB would probably be more trouble than its worth. There may come a time when we want to get more granular, but since we’re just reporting on or stats, the failure/drive is a fine component for us. Though someone could definitely take our raw stats and probably calculate those numbers.

  • Hannes Fischer

    Terrific, can you tell us a bit more about energy consumption/TB/year of storage too? Wonder if you are working on smart algoritm to turn off those drives from spinning that have cold data onto them… this probably would save you tons in electricity bills.

    • Andy Klein

      Good thought, but the “backup” nature of the data that is stored with us, means we keep the drives spinning to ensure we can provide appropriate data restore times. If we were providing cold storage we could cycle down the drives as restore time is measured in hours.

  • Jim_Feeley

    Great work and article. Thanks. You write at the end, “The 4TB Seagate drives are our workhorse drives today and their 2.8% annualized failure rate is more than acceptable for us.”

    Do you have a sense of what failure rate starts to concern you?
    Also, you’re drives are spinning 24/7, right? In my personal (ie- anecdotal at best) experience, drives seem to fail at startup. Any sense of how Backblaze’s experiences might map to those of us who turn off our drives once in a while? No worries if not. Thanks again for publishing your findings.

    • Andy Klein

      We’ve lived with failures rates of some drives over 20%, 2.8% means our data center techs spend less time changing drives. Every day on average 3-4 drives fail in our data center, so our techs are used to doing it and we staff accordingly.

      • SonyAD

        In your experience, is it more hours of spinning or spin-up/spin-down cycles that wear drives out faster? I my home PC I keep the drives spinning constantly (power saving options deactivated) regardless of them being used or not and I’ve had stellar reliability (knock on wood) but I’m wondering whether I wasn’t just lucky and my usage profile is actually wearing them out faster.

        My thinking is that if the drive has to spin down, it has to park the heads. Which it does on the platter, which wears out the heads and might also create debris.

        • Chris Moore

          I don’t know if all drives do it, but some drives actually park the heads off the platter on a ramp when they are not actually engaged in a read or write operation.
          Initial start is the most likely time to detect a failure in the motor / bearing seize, which is usually and immediate and total loss of data. If you keep the drive spinning and check for bad blocks regularly, you will have warning that the drive is eminent and you can move your data elsewhere.

    • jnix1985

      They also use consumer grade drives, not enterprise grade drives. Keep that in mind when reading these ;)

      2.8% failure rate on drives that are not designed to spin 24/7 is pretty good.

      • Sai

        Conversely, 20% even for consumer grade drives is HORRIBLE.

        • James Moore

          bear in mind that they’re densely packed and in cases with fans – which are also prone to failure.

  • ssphillips

    What do you do with the old drives that you take out of service? Do you sell them?

    • Andy Klein

      All of the removed drives are wiped clean of data (multiple pass), assuming they still work. Then some of the drives are re-purposed for testing and burn in of new pods, some are kept as spares to replace failed drives of the same model, and some are recycled.

    • bobjr94

      The recycled ones probably end up on amazon. 2 months ago i got an amazon warehouse deal 2TB drive. – Used, Like New, In Original Packaging – SMART reported like 26,000 hours powered on. I was expecting a new return, maybe 0 to 20 hours.

      • Alexander Bukh

        imagined just that, thanks for sharing

  • awesome post again!

  • Joshua Manor

    Would you please run the numbers for a pod filled with the new Samsung PM1633a 15.36TB SSDs.
    How many would fit? 100+?

    http://www.theverge.com/circuitbreaker/2016/8/1/12342696/samsung-pm1633a-ssd-15tb-storage-drive-specs-price?

    • Andy Klein

      Right now 60 would fit as there are 60 drive connectors in Pod. Even with that you’d get about 900TB of storage for about $600,000 USD. ($10K/drive * 60 drives). That works out to about $0.66/GB, while we currently pay $0.03 to $0.04/GB, so the premium is a bit high for the SSDs right now. Still, it’s fun to dream.

      • Dan Neely

        With your current pods anyway. OTOH since this is clearly a fantasy exercise it’s reasonable to assume you’d be using a new pod designed for the smaller drives. (Scaling it down should be cheap compared to the cost of filling it anyway.) The PM1633a is a 15mm drive (roughly double the thickness of a standard 2.5″ laptop hdd); which gives it about 1/6th the volume of a 3.5″ drive. So unless you slimmed down to a 1U design packing a few hundred drives in should be doable.

        • Emmanuel Goldstein

          You still have to deal with getting those drives connected, cabled, adding expansion cards, etc.

          • Wibla

            You could make a high-density 2U pod for 2.5″ “double height” (15mm) drives , but I suspect the backplanes etc would get pricey and/or require special design and manufacture.

      • Joshua Manor

        So cool. A 1PB Pod has to sound awesome. The 10K can’t last, but $0.03 might be a few years off ;)

    • We could fit in a bunch, but would need to completely redesign the pod to house SSDs and would be paying way more than what we currently get when comparing price to storage density. That said, SSD Backblaze Pods are not out of the question, the price of them just has to drop a bit more before it’s feasible for us.

  • idmah

    Please paste tables as TXT files or something with the ability to select txt. So we can copy and search for the models listed.