{"id":84006,"date":"2018-07-17T07:59:15","date_gmt":"2018-07-17T14:59:15","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=84006"},"modified":"2024-02-28T12:26:22","modified_gmt":"2024-02-28T20:26:22","slug":"cloud-storage-durability","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/cloud-storage-durability\/","title":{"rendered":"Backblaze Durability Calculates at 99.999999999% — And Why It Doesn\u2019t Matter"},"content":{"rendered":"

\"Dials<\/p>\n

One of the most often talked about, but least understood, metrics in our industry is the concept of \u201cdata durability.\u201d It is often talked about in that nearly everyone quotes some number of nines, and it is least understood in that no one tells you how they actually computed the number or what they actually mean by it.<\/p>\n

It strikes us as odd that so much of the world depends on the concept of RAID<\/a> and Encodings, but the calculations are not standard or agreed upon. Different web calculators allow you to input some variables but not the correct or most important variables. In almost all cases, they obscure the math behind how they spit out their final numbers. There are a few research papers, but hardly a consensus. There just doesn’t seem to be an agreed upon standard calculation of how many “9s” are in the final result. We\u2019d like to change that.<\/p>\n

In the same spirit of transparency that leads us to publish our hard drive<\/a> performance stats, open source our Reed-Solomon Erasure Code,<\/a> and generally try to share as much of our underlying architecture<\/a> as practical, we\u2019d like to share our calculations for the durability of data stored with us.<\/p>\n

We are doing this for two reasons:<\/p>\n

    \n
  1. We believe that sharing, where practical, furthers innovation in the community.<\/li>\n
  2. Transparency breeds trust. We\u2019re in the business of asking customers to trust us with their data. It seems reasonable to demonstrate why we\u2019re worthy of your trust.<\/li>\n<\/ol>\n

    11 Nines Data Durability for Backblaze B2 Cloud Storage<\/a><\/h2>\n

    At the end of the day, the technical answer is \u201c11 nines.\u201d That\u2019s 99.999999999%. Conceptually, if you store 1 million objects in B2 for 10 million years, you would expect to lose 1 file. There\u2019s a higher likelihood of an asteroid destroying Earth<\/a> within a million years, but that is something we\u2019ll get to at the end of the post.<\/p>\n

    How to Calculate Data Durability<\/a><\/h3>\n

    Amazon\u2019s CTO put forth the X million objects over Y million years metaphor in a blog post<\/a>. That\u2019s a good way to think about it — customers want to know that their data is safe and secure.<\/p>\n

    When you send us a file or object, it is actually broken up into 20 pieces (\u201cshards\u201d). The shards overlap so that the original file can be reconstructed from any combination of any 17 of the original 20 pieces. We then store those pieces on different drives that sit in different physical places (we call those 20 drives a \u201ctome\u201d) to minimize the possibility of data loss. When one drive fails, we have processes in place to \u201crebuild\u201d the data for that drive. So, to lose a file, we have to have four drives fail before we had a chance to rebuild the first one.<\/p>\n

    The math on calculating all this is extremely complex. Making it even more interesting, we debate internally whether the proper calculation methodology is to use the Poisson distribution (the probability of continuous events occurring) or Binomial (the probability of discrete events). We spent a shocking amount of time debating this and believe that both arguments have merits. Rather than posit one absolute truth, we decided to publish the results of both calculations (spoiler alert: either methodology tells you that your files are safe with Backblaze).<\/p>\n

    The math is difficult to follow unless you have some facility with advanced statistics. We\u2019ll forgive you if you want to skip the sections entirely, just click here<\/a>.<\/p>\n

    Poisson Distribution<\/h3>\n

    When dealing with the probability of X number of events occuring in a fixed period of time, a good place to start is the Poisson distribution<\/a>.[1]<\/sup><\/a><\/p>\n

    For inputs, we use the following assumptions:[2]<\/sup><\/a><\/p>\n

      \n
    1. The average rebuild time to achieve complete parity for any given B2 object with a failed drive is 6.5 days.<\/strong> A given file uploaded to Backblaze is split into 20 \u201cshards\u201d or pieces<\/a>. The shards are distributed across multiple drives in a way that any drive can fail and the file is fully recoverable — a file is not lost unless four drives were to fail in a given vault before they could be \u201crebuilt.\u201d This rebuild is enabled through our Reed-Solomon Erasure Code<\/a>. Once one drive fails, the other shards are used to \u201crebuild\u201d the data on the original drive (creating, for all practical purposes, an exact clone of the original drive).\n

      The rule of thumb we use is that for every 1 TB needed to be rebuilt, one should allow 1 day. So a 12 TB drive would, on average, be rebuilt after 12 days. In practice, that number may vary based on a variety of factors, including, but not limited to, our team attempting to clone the failed drive before starting the rebuild process. Based on whatever else may be happening at a given time, a single failed drive may also not be addressed for one day. (Remember, a single drive failure has a dramatically different implication than a hypothetical third drive failure<\/a> within a given vault — different situations would call for different operational protocols.) For the purposes of this calculation, and a desire to provide simplicity where possible, we assumed an average of a one day lag time before we start the rebuild.<\/li>\n

    2. The annualized failure rate of a drive is 0.81%.<\/strong> For the trailing 60 days while we were writing this post, our average drive failure rate was 0.81%. Long time readers of our blog will also note that hard drive failure rates in our environment have fluctuated over time. But we also factor in the availability of data recovery services including, but not limited to, those offered by our friends at DriveSavers. We estimate a 50% likelihood of full (100%) data recovery from a failed drive that\u2019s sent to DriveSavers. That cuts the effective failure rate in half to 0.41%.<\/li>\n<\/ol>\n

      For our Poisson calculation<\/a>, we use this formula:<\/p>\n

      \"Poisson<\/p>\n

      The values for the variables are:<\/p>\n