{"id":64061,"date":"2016-08-02T07:55:15","date_gmt":"2016-08-02T14:55:15","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=64061"},"modified":"2021-08-12T06:28:41","modified_gmt":"2021-08-12T13:28:41","slug":"hard-drive-failure-rates-q2-2016","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/hard-drive-failure-rates-q2-2016\/","title":{"rendered":"Hard Drive Stats for Q2 2016"},"content":{"rendered":"

\"Hard<\/a><\/p>\n

For the most recent hard drive reliability<\/a> statistics, as well as the raw hard drive test data, visit Hard Drive Data and Stats<\/a>.<\/div>\n

Q2 2016 saw Backblaze introduce 8TB drives into our drive mix, kick off a Pod to Vault migration of over 6.5PB of data, cross over 250 petabytes<\/a> of data stored, and deploy another 7,290 drives into the data center for a total of 68,813 spinning hard drives under management. With all the ins and outs, let\u2019s take a look at how our hard drives fared in Q2 2016.<\/p>\n

Backblaze Hard Drive Reliability for Q2 2016<\/h2>\n

Below is the hard drive failure data for Q2 2016. This chart is just for the period of Q2 2016. The hard drive models listed below are data drives (not boot drives), and we only list models which have 45 or more drives of that model deployed.<\/p>\n

\"Q2<\/a>A couple of observations on the chart:<\/p>\n

    \n
  1. The models that have an annualized failure rate of 0.00% had zero hard drive failures in Q2 2016.<\/li>\n
  2. The annualized failure rate is computed as follows: ((Failures)\/(Drive Days\/365)) * 100. Therefore consider the number of failures and drive days before reaching any conclusions about the failure rate.<\/li>\n<\/ol>\n

    Later in this post, we\u2019ll review the cumulative statistics for all of our drives over time, but first, let\u2019s take a look at the new drives on the block.<\/p>\n

    The 8TB Hard Drives Have Arrived<\/h3>\n

    For the last year or so we kept saying we were going to deploy 8TB drives in quantity. We did deploy 45 8TB HGST drives<\/a>, but deploying these drives en masse<\/em> did not make economic sense for us. Over the past quarter, 8TB drives from Seagate became available at a reasonable price, so we purchased and deployed over 2,700 in Q2 with more to come in Q3. All of these drives were deployed in Backblaze Vaults with each Vault using 900 drives, that’s 45 drives in each of the 20 Storage Pods that form a Backblaze Vault<\/a>.<\/p>\n

    Yes, we said 45 drives in each Storage Pod, so what happened to our 60 drive Storage Pods<\/a>? In short, we wanted to use the remaining stock of 45 drive Storage Pods before we started using the 60 drive Pods. We have built two Backblaze Vaults using the 60 drive Pods, but we filled them with 4TB and 6TB drives. The first 60 drive Storage Pod filled with 8TB drives (total 480TB) will be deployed shortly.<\/p>\n

    Hard Drive Migration\u201485 Pods to One Vault<\/h3>\n

    One of the reasons that we made the move to 8TB drives was to optimize storage density<\/a>. We\u2019ve done data migrations before, for example, from 1TB Pods to 3TB and 4TB Pods. These migrations were done one or two Storage Pods at a time. It was time to up our game. We decided to migrate from individual Storage Pods filled with HGST 2TB drives<\/a>, average age 64 months, to a Backblaze Vault filled with 900 8TB drives.<\/p>\n

    \"Backblaze<\/a><\/p>\n

    We identified and tagged 85 individual Storage Pods to migrate from. Yes, 85. The total amount of data to be migrated was about 6.5PB. It was a bit sad to see the 2TB HGST drives go as they have been really good over the years, but getting four times as much data into the same space was just too hard to resist.<\/p>\n

    The first step is to stop all data writes on the donor HGST 2TB Storage<\/a> Pods. We then kicked off the migration by starting with 10 Storage Pods. We then added 10 to 20 donor Pods to the migration every few hours until we got to 85 Pods. The migration process is purposely slow as we want to ensure that we can still quickly read files from the 85 donor Pods so that data restores are not impacted. The process is to copy a given RAID-array from a Storage Pod to a specific Tome in a Backblaze Vault. Once all the data in a given RAID-array has been copied to a Tome, we move on to the next RAID-array awaiting migration and continue the process. This happens in parallel across the 45 Tomes in a Backblaze Vault.<\/p>\n

    We\u2019re about 50% of the way through the migration with little trouble. We did have a Storage Pod in the Backblaze Vault go down. That didn\u2019t stop the migration, as Vaults are designed to continue to operate under such conditions, but more on that in another post.<\/p>\n

    250 Petabytes of Data Stored<\/h3>\n

    Recently we took a look at the growth of data and the future of cloud storage<\/a>. Given the explosive growth in data as a whole, it’s not surprising that Backblaze added another 50PB of customer data over the last two quarters and that by mid-June we had passed the 250PB mark in total data stored. You can see our data storage growth<\/a> below:<\/p>\n

    \"Backblaze<\/a><\/p>\n

    Back in December 2015, we crossed over the 200PB mark and at that time predicted we would cross 250PB<\/a> in early Q3 2016. So, we\u2019re a few weeks early. We also predicted we would cross 300PB in late 2016. Given how much data we are adding with Backblaze B2<\/a>, it will probably be sooner; we\u2019ll see.<\/p>\n

    Cumulative Hard Drive Failure Rates by Model<\/h3>\n

    In the table below, we\u2019ve computed the annualized drive failure rate for each drive model. This is based on data from April 2013 through June 2016.<\/p>\n

    \"Q2<\/a><\/p>\n

    Some people question the usefulness of the cumulative annualized failure rate. This is usually based on the idea that drives entering or leaving during the cumulative period skew the results because they are not there for the entire period. This is one of the reasons we compute the annualized failure rate using drive days. A drive day is only recorded if the drive is present in the system. For example, if a drive is installed on July 1st and fails on August 31st, it adds 62 drive days and one drive failure to the overall results. A drive can be removed from the system because it fails, or perhaps it is removed from service after a migration, like the 2TB HGST drives we\u2019ve covered earlier. In either case, the drive stops adding drive days to the total, allowing us to compute<\/a> an annualized failure rate over the cumulative period based on what each of the drives contributed during that period.<\/p>\n

    As always, we’ve published the Q2 2016 data we used to compute these drive stats.<\/a> You can find the data files along with the associated documentation on our hard drive test data page<\/a>.<\/p>\n

    Which Hard Drives Do We Use?<\/h3>\n

    We\u2019ve written previously about our difficulties in getting drives from Toshiba and Western Digital. Whether it\u2019s poor availability or an unexplained desire not to sell us drives, we don\u2019t have many drives from either manufacturer. So we use a lot of Seagate drives and they are doing the job very nicely. The table below shows the distribution of the hard drives we are currently using in our data center.<\/p>\n

    \"Q2<\/a><\/p>\n

    Recap<\/h2>\n

    The Seagate 8TB drives are here and are looking good. Sadly we\u2019ll be saying goodbye to the HGST 2TB drives, but we need the space. We\u2019ll miss those drives, they were rock stars for us. The 4TB Seagate drives are our workhorse drives today and their 2.8% annualized failure rate is more than acceptable for us. Their low failure rate roughly translates to an average of one drive failure per Storage Pod<\/a> per year. Over the next few months, expect more on our migrations, a look at the day in the life of a data center tech, and an update of the bathtub curve, i.e. hard drive failure<\/a> over time.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Q2 2016 saw Backblaze introduce 8TB drives into our drive mix, kick off a pod-to-vault migration of over 6.5 Petabytes of data, cross over 250 Petabytes of data stored, and deploy another 7,290 drives into the data center for a total of 68,813 spinning hard drives under management. With all the ins-and-outs let\u2019s take a look at how our hard drives fared in Q2 2016.<\/p>\n","protected":false},"author":12,"featured_media":64121,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[7,457],"tags":[468],"class_list":["post-64061","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-storage","category-hard-drive-stats","tag-b2cloud","entry"],"acf":[],"yoast_head":"\nHard Drive Failure Rates: The Results from 68,813 Hard Drives<\/title>\n<meta name=\"description\" content=\"We take a close look at the hard drive failure rates from 68,813 hard drives that are used in the Backblaze data center.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.backblaze.com\/blog\/hard-drive-failure-rates-q2-2016\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Hard Drive Failure Rates: The Results from 68,813 Hard Drives\" \/>\n<meta property=\"og:description\" content=\"We take a close look at the hard drive failure rates from 68,813 hard drives that are used in the Backblaze data center.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/\" \/>\n<meta property=\"og:site_name\" content=\"Backblaze Blog | Cloud Storage & Cloud Backup\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/backblaze\" \/>\n<meta property=\"article:published_time\" content=\"2016-08-02T14:55:15+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-08-12T13:28:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2016\/07\/blog-drive-stats-q2-2016.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1440\" \/>\n\t<meta property=\"og:image:height\" content=\"810\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Andy Klein\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@backblaze\" \/>\n<meta name=\"twitter:site\" content=\"@backblaze\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andy Klein\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Hard Drive Failure Rates: The Results from 68,813 Hard Drives","description":"We take a close look at the hard drive failure rates from 68,813 hard drives that are used in the Backblaze data center.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.backblaze.com\/blog\/hard-drive-failure-rates-q2-2016\/","og_locale":"en_US","og_type":"article","og_title":"Hard Drive Failure Rates: The Results from 68,813 Hard Drives","og_description":"We take a close look at the hard drive failure rates from 68,813 hard drives that are used in the Backblaze data center.","og_url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/","og_site_name":"Backblaze Blog | Cloud Storage & Cloud Backup","article_publisher":"https:\/\/www.facebook.com\/backblaze","article_published_time":"2016-08-02T14:55:15+00:00","article_modified_time":"2021-08-12T13:28:41+00:00","og_image":[{"width":1440,"height":810,"url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2016\/07\/blog-drive-stats-q2-2016.jpg","type":"image\/jpeg"}],"author":"Andy Klein","twitter_card":"summary_large_image","twitter_creator":"@backblaze","twitter_site":"@backblaze","twitter_misc":{"Written by":"Andy Klein","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/#article","isPartOf":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/"},"author":{"name":"Andy Klein","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/9ac7e0bf0bd16852f8bfef352ce5fa8c"},"headline":"Hard Drive Stats for Q2 2016","datePublished":"2016-08-02T14:55:15+00:00","dateModified":"2021-08-12T13:28:41+00:00","mainEntityOfPage":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/"},"wordCount":1254,"commentCount":114,"publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2016\/07\/blog-drive-stats-q2-2016.jpg","keywords":["B2Cloud"],"articleSection":["Cloud Storage","Hard Drive Stats"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/","name":"Hard Drive Failure Rates: The Results from 68,813 Hard Drives","isPartOf":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/#primaryimage"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2016\/07\/blog-drive-stats-q2-2016.jpg","datePublished":"2016-08-02T14:55:15+00:00","dateModified":"2021-08-12T13:28:41+00:00","description":"We take a close look at the hard drive failure rates from 68,813 hard drives that are used in the Backblaze data center.","breadcrumb":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/#primaryimage","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2016\/07\/blog-drive-stats-q2-2016.jpg","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2016\/07\/blog-drive-stats-q2-2016.jpg","width":1440,"height":810},{"@type":"BreadcrumbList","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/hard-drive-failure-rates-q2-2016\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Hard Drive Stats for Q2 2016"}]},{"@type":"WebSite","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","name":"Backblaze Cloud Solutions Blog","description":"Cloud Storage & Cloud Backup","publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization","name":"Backblaze","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Backblaze"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/backblaze","https:\/\/x.com\/backblaze","https:\/\/www.youtube.com\/user\/Backblaze","https:\/\/en.wikipedia.org\/wiki\/Backblaze"]},{"@type":"Person","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/9ac7e0bf0bd16852f8bfef352ce5fa8c","name":"Andy Klein","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/04\/andy.jpg","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/04\/andy.jpg","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/04\/andy.jpg","caption":"Andy Klein"},"description":"Andy Klein is the former Principal Cloud Storage Storyteller at Backblaze. He has over 25 years of experience in technology marketing and during that time, he has shared his expertise in cloud storage and computer security at events, symposiums, and panels at RSA, SNIA SDC, MIT, the Federal Trade Commission, and hundreds more. He currently writes and rants about drive stats, Storage Pods, cloud storage, and more.","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/author\/andy\/"}]}},"jetpack_featured_media_url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2016\/07\/blog-drive-stats-q2-2016.jpg","_links":{"self":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/64061","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/comments?post=64061"}],"version-history":[{"count":0,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/64061\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media\/64121"}],"wp:attachment":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media?parent=64061"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/categories?post=64061"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/tags?post=64061"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}