Farming hard drives: how Backblaze weathered the Thailand drive crisis

October 9th, 2012

The rain started in August and by mid-October 2011, violent floods in Thailand had crippled the factories that helped produce nearly half of the world’s hard drives. As an online backup company, Backblaze fills more than 50 TB of new drives every day. To survive this crisis without raising prices or compromising service, Backblaze deployed every last employee, as well as friends and family, to acquire drives in what became known internally as “drive farming”. What follows is how we did it.

Good morning, your costs just tripled!
Brian, our Buyer, gasped, “$349 for a 3TB internal Hitachi hard drive!” How could that be? The last drive order he placed just two weeks ago on NewEgg was at $129 per drive. He rechecked the SKU three times. He found Seagate and Western Digital prices were similar. What was going on? Over the next few days prices spiked in all of our traditional purchasing channels, everyone was quoting ridiculously high numbers. One “friend” offered us 100 drives, non-returnable, for a mere $568 each – we said no. In a matter of days the cost of the hard drives that are core to our business had jumped over 200% meaning the cost of providing our customers unlimited storage was going to skyrocket as well. We needed a plan.

We didn’t plan for rain in Thailand
We started drive farming in November 2011. The reason was simple, the supply of the 3TB hard drives used in our Storage Pods had dried up – or more correctly was under water. The tragic flooding in Thailand began in August 2011 and by early-October had submerged houses, schools and factories. Over 800 people died and many more were homeless and hungry, with over 1 million people thrown out of work. As the water receded, the human cost of the flooding was obvious and the economic impact was slowly coming into focus. In late-October, it was estimated that up to 50% of the worldwide hard drive manufacturing capacity was lost or damaged. The impact of the lost capacity was immediate as hard drive prices nearly tripled overnight.

Wanted: a really good idea
We huddled in the conference room. CTO Brian stood at the whiteboard listing retailers, store locations and employee names. While we had a storage pod buffer in our datacenter that could last a few months, some industry analysts were already predicting that the drive shortage could last well over a year. Backblaze is committed to providing unlimited data backup, but with our normal channels charging usury prices for the hard drives core to our business, we needed a miracle. We got two: Costco and Best Buy. On Brian’s whiteboard he listed every Costco and Best Buy in the San Francisco Bay Area and then some. We would go to each location and buy as many 3 TB drives as possible.

Shucking is a good thing
The going rate for a 3TB external drive at Costco or Best Buy was $169. Internal drives, like those inside your computer and the ones we typically use in our storage pods, were at least $100 more and were becoming nearly impossible to find, having been bought by HP, Apple and Dell for their systems. Our “choice” to use external drives meant they had to be “shucked” – a process where the internal drive is removed from its case – before being deployed into a Backblaze storage pod. In addition to the price of the drive there were now extra costs to shuck the drives, extra costs to recycle the shucked parts, and once shucked we couldn’t return a bad drive, but it was still less expensive than buying internal drives even if we could find them. Problem solved, Costco here we come.

“Aaargh, there the treasure be”
Backblaze employees Yev and Ken stared exhausted into the car trunk at their treasure – fifty-two 3 TB disk drives. That would be enough for 2 maybe 3 days at the current burn rate they thought as they drove back to the office to unload their bounty. They hoped Damon, KC and the other Backblaze employees would have similar success and we’d add a few more weeks’ worth of hard drives to our stockpile.

The economics of storage pods
Backblaze provides online backup for unlimited data at $5/month. We can do this because we have implemented our own backup and data storage management software on our own Storage Pods. These pods are built from commercially available parts such as 3TB internal hard drives from Seagate or Western Digital, motherboards from Supermicro, etc. Each Storage Pod holds 45 drives or 135 TB of data. We can reliably store and retrieve data at up to 25 times lower than the cost of other services such as Amazon S3 by using our own purposed-designed cloud storage.

Did you really, really, really mean 2?
The “Two Drive Limit” signs started appearing in retail stores in mid-November. At first we didn’t believe them, but we quickly learned otherwise. Sometimes, we talked our way into more, but we heard “2 is the limit” a lot. We started doing “drive math”: 2 drives a day per store, times 3 stores per day, times 5 “farmers”, times 7 days a week is 210 drives. That would be sufficient, but in reality it didn’t work out that way. Stores were stocking out of drives on a regular basis and we really couldn’t farm every day, but we kept at it. One Wednesday afternoon, after working all day at Backblaze, Yev circled the San Francisco Bay hitting local Costco and Best Buy stores – 10 stores, 46 drives, 212 miles on his Nissan.

Family and friends to the rescue
In late November, Brian was banned from buying drives on Costco’s online store, and Billy was banned from purchasing them at any Costco in the Bay Area. Other Backblaze employees were also asked to leave stores empty handed. Other vendors, along with the seasonal demand from Christmas were rapidly draining supplies. It was time to get creative, again. At an impromptu drive crisis meeting, fueled by stale pizza and good beer, we decided we needed to scale drive farming and the idea of friends and family drive farming was born. Emails and text messages were sent, phones calls were made, Facebook posts were posted, tweets were tweeted, you name it – the call went out to friends and family – buy hard drives and send them to Backblaze, NOW.

Simon says, “Raise prices” – Backblaze doesn’t play Simon says.
While we were drive farming other vendors reacted more traditionally. Intel warned that the drive crisis could lower its quarterly revenue by about $1 billion, give or take $300 million. EMC notified its Velocity channel partners, “…our Q1 2012 HDD list prices will rise between 5-15% over Q4 2011 levels.” HP announced that, “…HP will be forced to increase the prices that we charge for certain disk drives”. Dell sent us a nicely worded email explaining that “HDD costs have increased…and Dell’s list pricing has been adjusted as a result.” So while we pondered the economics of renting a truck and driving it across the country buying drives along the way, others raised prices – we would not do that.

Drive farming at work
Stores throughout the rest of the country were also limiting hard drive availability, but now we had more farmers. Drives started to trickle in, 2 at a time. It was cheaper to buy external drives at a store in Iowa and have Yev’s dad, Boris, ship them to California than it was to buy internal drives through our normal channels. Nearly every day the postman would deliver packages of drives from friends and family around the country. Tim would pile the boxes up on a table in the office and every week the van from our pod assembly company would pick them up. A few weeks later the Storage Pods, filled with 45 drives in each, would arrive in the datacenter. There, Sean and Guido would deploy the pods to store the petabytes of data backed up by our customers.

Holiday drive(s)
On Christmas Eve, Gleb, our CEO, stopped by a friend’s house to pick up eighty 3TB drives his friend had acquired when an online site forgot to limit the quantity he could order. It had taken the FedEx guy nearly 30 minutes to unload them and carry them up to his friend’s apartment. As Gleb was lugging boxes of hard drives to his car to take back to Backblaze, he realized that the hard drives he was loading were worth more than the car he was loading them into.

More stories
There are many great adventures from our days as drive farmers, here are just a few…

Getting back to normal
Drive farming continued throughout January and into mid-February of 2012. By then we had farmed 5.5 Petabytes of data storage. Eventually smaller distributors and resellers appeared offering decent prices on internal drives – no more shucking. Throughout the crisis, we didn’t raise our prices, we continued our policy of unlimited storage, and we didn’t throttle our customers’ backup speeds. Prices have slowly decreased over the past several months as Thailand has gotten back on its feet and production shifted to other factories. We recently purchased another 2,000 internal drives to add to our stockpile. Lower prices are nice.

Thank you
We want to say thank you to our friends and family who helped us through the drive crisis; Boris, Cara, Jim, Susan, Dave, Meng, Mike, Ben, Evelyn, Katherine, Randy, Ramey, Lise, Pete, Paul, Vladik, Tasha, Hakan, Alec, Alla, Katia, Leya, Yan, Jessica, Sergey, Mary, John, Eugenia, Penny, Sergey, Rebecca, Tim and many others. We’d also like to thank the retailers who kept the price of external drives reasonable – even Costco.

Yet, even though we at Backblaze weathered the drive challenge it pales in comparison to the challenges faced by the people of Thailand as they continue to recover from the devastating floods that have ravaged their country. Our hearts go out to them. If you wish to help, please consider a donation through Give2Asia or the relief organization of your choice.

Epilogue
On July 25th of this year, Backblaze took $5M in venture funding. At the same time, Costco was offering 3TB external drives for $129 about $30 less than we could get for internal drives. The limit was five drives per person. Needless to say, it was a deal we couldn’t refuse. Old habits die hard.

References:

  • EMC and HP quotes: http://www.storagenewsletter.com/news/miscellaneous/emc-and-hp-increasing-hdd-prices.
  • Intel earnings warning: http://www.businesswire.com/news/home/20111212005709/en/Intel%E2%80%99s-Fourth-Quarter-Revenue-Expectations-Due-Hard-Disk
  • Andy Klein

    Andy Klein

    Andy has 20+ years experience in technology marketing. He has shared his expertise in computer security and data backup at the Federal Trade Commission, Rootstech, RSA and over 100 other events. His current passion is to get everyone to back up their data before it's too late.
    Andy Klein

    Latest posts by Andy Klein (see all)

    • Bartz

      That is awesome!

    • Tim

      Unfortunately it was this behavior, in mass, that caused Seagate to cripple firmware on their external USB drives to not accept ACPI mode (the SATA-USB adapters typically use the IDE command set) so modern drives can no longer be shucked.
      Thanks, Backblaze.

      • calmdownbro

        Why would you even buy Seagate drives? Haven’t you read the Reliability post of Backblaze? (Which is the same experience I had with Seagate drives throughout the years.) Seagate, never again.

        • Tim

          You are joking, right? Seagate is the worlds largest OEM of hard drives. They wouldn’t be if they were unreliable. The reason Backblaze has rampant Seagate failures is because they’re putting drives not designed for the vibration of a multi-drive cabinet into a multi-drive cabinet. 99.999% of end users of Seagate drives will not use them in such a scenario…and if they do, Seagate has enterprise-class drives that are designed to do so.
          Take Backblazes report for what it is, reliability of drives in a SPECIFIC application. It is in no way comparable to a desktop, laptop, portable or even a small NAS enclosure, simply due to the excessive harmonic vibration in the Backblaze Pods.
          Backblaze clearly states this in their testing methodoly and disclosure, but morons just skip to the charts without knowing what they mean.

          • calmdownbro

            “but morons just skip to the charts without knowing what they mean.”

            Calling others moron sure makes you look smarter.
            They also stated that the new pods have anti-vibration built in.
            Anyhow. I had quite a few drives since my very first computer.
            And guess what? Drum roll please… Only Seagates died.
            About 6-8 drives, mostly desktop, and two laptop ones.

            Sure I thrown out WD as well. Because it was too old/too small, but it was still working.

            Now it would be cool if Backblaze would start checking their comments and remove paid Seagate comments/posters. Thanks!

            • Nicholas Oldroyd

              Key word: *New* pods.

              The point is all those 3TB Seagates that are failing are running from 2.0 pods, not even 3.0 or 4.0 pods.

              Check out the Seagate 4TB disks that are running from newer pods and you will see what may be the impact of lower vibration.

              The data available is inconclusive whether the disk are bad or whether they were improperly used given their design specifications.

            • calmdownbro

              Another post, another hard burn for Seagate:
              https://www.backblaze.com/blog/best-hard-drive/

              I am not a fanboy of any brand, just read the posts, and also had bad experiences with Seagate drives. That’s all.

              Ps (not a direct answer to your comment).: An average desktop computer generates more vibration than a storage pod, I bet. I seen desktop PCs vibrating all over the place because of the ultra-cheap case, the cheap components and everything.

            • Joe

              Whatever you have against the other poster, manufacturers also have periods of bad luck. As a large buyer of disks (research computing), several OEMs told me they were only later told certain disk models had to re-source parts post-flood. No model change. IBM is very notable for having a bomb of a product and selling their (otherwise excellent) hard drive unit to Hitachi.

              Your anecdotal experience, small sample size and utter lack of methodology make your numbers meaningless.

              There are plenty of flaws in the backblaze reliability “studies.” They’re just raw number dumps with some stabs at reasoning. They lack any information on the workloads, environmentals, or deployment of each model of disk.

              And cmon, no need to call the poster a shill, some people are storage admins. We exist and have opinions. We just have them on odd things like arrays, spindles, flash and interconnects… Seagate’s a pretty active R&D machine like any other major tech company. They do put out some good stuff, even if they did have a stumble with the 2TB spindles.

            • Chris Moore

              I have no idea what your background is, but I have been working in hardware support for more than 20 years and the brand that I have seen fail most frequently, in a catastrophic fashion, is Western Digital. They all fail, it is just a matter of when. The vibration only accelerates the process.

          • Designer_Gayle

            Because they are a big company… they must be the BEST drives.
            Huh?
            What kind of logic is that?

            *PLEASE* don’t ever apply for a job at my company.

        • Matthew F

          So if Seagate is so bad, why did Backblaze keep buying them? Their entire test and claims were flawed like no tomorrow.

          • http://www.backblaze.com/ YevP

            They’re inexpensive and work in our use-case! Our system is designed for hard drive failure, so the the price of the drives matters a lot more to us than the failure rate, though if the failure rate was extremely high, we’d take a look at other options. We buy lots of stuff though, they just happen to win on price a lot.

      • Rath

        Unfortunately, Seagate’s retarded for doing that in the first place. Internal drives should not be more expensive than external. This isn’t rocket science, its common sense.

    • Rockdirector

      I’ve been shucking for years, and it’s GTK large companies have done it too, Tim’s warning below notwithstanding (for example, WD portable drives no longer even have internal SATA connections — the USB bridge is integrated into the drive itself.)

      Ppl should note that it usually results in voiding of the warranty (mfgs tend to sequester the serials of the drives shipped in enclosures) although with many warranty replacements being refurb drives it’s sort of a moot point these days. Do you knowingly trust data to a refurb? I can’t do it.

    • BobBobson

      It’s impressive that while some companies (*cough*rogers*cough*) raise their prices every chance they get even without a good/any reason, you refused to do so despite needing to.

      That said, I do have one issue with this post, specifically this passage:

      > In addition to the price of the drive there were now extra costs to shuck the drives, extra costs to recycle the shucked parts

      Instead of spending money to recycle the enclosures, did it occur to you to simply post them on classified ads (Kijiji, eBay Classifieds, Craig’s List, etc.) to offer them for free to people? You could easily have gotten rid of them without spending any money, and provided a lot of people with enclosures that they can use. At the very least, you could have reduced how many needed to be recycled.

      • Markus Hess

        Most of the external cases cannot be opened nondestructive, so no chance to use them any more after removing the drive.

        • Chris Moore

          I have opened many and never found one that couldn’t be opened without damage once you figure out how it is meant to be opened. The manufacturers make them to be opened without damage so they can fix them and return them to customers as “refurbished” drives. When you get a drive on a warranty exchange, it is a refurbished unit which is one that they received, fixed and shipped back out.