{"id":92724,"date":"2019-09-24T17:06:53","date_gmt":"2019-09-25T00:06:53","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=92724"},"modified":"2021-07-23T12:45:38","modified_gmt":"2021-07-23T19:45:38","slug":"origins-of-the-pod","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/origins-of-the-pod\/","title":{"rendered":"Interview With the Podfather: Tim Nufire on the Origins of the Pod"},"content":{"rendered":"<p><a href=\"\/blog\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg\" data-rel=\"lightbox-gallery-WCSyxJtY\" data-rl_title=\"\" data-rl_caption=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-92734 size-full\" title=\"\" src=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg\" alt=\"Tim, the Podfather, working in data center\" width=\"1440\" height=\"820\" srcset=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg 1440w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter-300x171.jpg 300w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter-1024x583.jpg 1024w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter-768x437.jpg 768w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter-560x319.jpg 560w\" sizes=\"auto, (max-width: 1440px) 100vw, 1440px\" \/><\/a><\/p>\n<p id=\"bzdropcap\">It\u2019s early 2008, the concept of cloud computing is only just breaking into public awareness, and the economy is in the tank. Despite this less-than-kind business environment, five intrepid Silicon Valley veterans quit their jobs and pooled together $75K to launch a company with a simple goal: provide an easy-to-use, cloud-based <a href=\"https:\/\/www.backblaze.com\/cloud-storage\/solutions\/backup-and-archive\">backup service<\/a> at a price that no one in the market could beat &#8212; $5 per month per computer.<\/p>\n<p>The only problem: both hosted storage (through existing cloud services) and purchased hardware (buying servers from Dell or Microsoft) were too expensive to hit this price point. Enter Tim Nufire, aka: <em>The Podfather<\/em>.<\/p>\n<p style=\"margin-bottom: 10px;\">Tim led the effort to build what we at Backblaze call the <a href=\"\/blog\/petabytes-on-a-budget-v2-0revealing-more-secrets\/\">Storage Pod<\/a>: The physical hardware our company has relied on for data storage for more than a decade. On the occasion of the decade anniversary of the open sourcing of our Storage Pod 1.0 design, we sat down with Tim to relive the twists and turns that led from a crew of backup enthusiasts in an apartment in Palo Alto to a company with four data centers spread across the world holding 2100 storage pods and closing in on an <a href=\"\/blog\/exabyte-unlocked\/\">exabyte<\/a> of storage.<\/p>\n<p style=\"text-align: center; margin: 6px 0 18px 0;\">\u2723\u00a0\u00a0 \u2723\u00a0\u00a0 \u2723<\/p>\n<p class=\"e-interview\"><strong>Editors:<\/strong> So Tim, it all started with the $5 price point. I know we did market research and that was the price at which most people shrugged and said they&#8217;d pay for <a href=\"https:\/\/www.backblaze.com\/cloud-backup\/personal\" target=\"_blank\" rel=\"noopener noreferrer\">backup<\/a>. But it was so audacious! The tech didn\u2019t exist to offer that price. Why do you start there?<\/p>\n<p><strong>Tim Nufire:<\/strong> It was the pricing given to us by the competitors, they didn\u2019t give us a lot of choice. But it was never a challenge of <em>if<\/em> we should do it, but <em>how<\/em> we would do it. I had been managing my own backups for my entire career; I cared about backups. So it\u2019s not like backup was new, or particularly hard. I mean, I firmly believe Brian Wilson\u2019s (Backblaze\u2019s Chief Technology Officer) top line: You read a byte, you write a byte. You can read the byte more gently than other services so as to not impact the system someone is working on. You might be able to read a byte a little faster. But at the end of the day, it\u2019s an execution game not a technology game. We simply had to out execute the competition.<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> Easy to say now, with a company of 113 employees and more than a decade of success behind us. But at that time, you were five guys crammed into a Palo Alto apartment with no funding and barely any budget and the competition &#8212; Dell, HP, Amazon, Google, and Microsoft &#8212; they were huge! How do you approach that?<\/p>\n<p><strong>TN:<\/strong> We always knew we could do it for less. We knew that the math worked. We knew what the cost of a 1 TB hard drive was, so we knew how much it <em>should<\/em> cost to store data. We knew what those markups were. We knew, looking at a Dell 2900, how much the margin was in that box. We knew they were overcharging. At that time, I could not build a desktop computer for less than Dell could build it. But I could build a server at half their cost.<\/p>\n<p>I don\u2019t think Dell or anyone else was being irrational. As long as they have customers willing to pay their hard margins, they can\u2019t adjust for the potential market. They have to get to the point where they have no choice. We didn\u2019t have that luxury.<\/p>\n<p>So, at the beginning, we were reluctant hardware manufacturers. We were manufacturing because we couldn\u2019t afford to pay what people were charging, not because we had any passion for hardware design.<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> Okay, so you came on at that point to build a cloud. Is that where your title comes from? Chief Cloud Officer? The pods were a little ways down the road, so Podfather couldn\u2019t have been your name yet. \u2026<\/p>\n<p><strong>TN:<\/strong> This was something like December, 2007. <a href=\"\/blog\/author\/gleb\/\">Gleb<\/a> (Budman, the Chief Executive Officer of Backblaze) and I went snowboarding up in Tahoe, and he talked me into joining the team. \u2026 My title at first was all wrong, I never became the VP of Engineering, in any sense of the word. That was never who I was. I held the title for maybe five years, six years before we finally changed it.<a href=\"\/blog\/author\/tnufire\/\"> Chief Cloud Officer<\/a> means nothing, but it fits better than anything else.<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> It does! You built the cloud for Backblaze with the Storage Pod as your water molecule (if we\u2019re going to beat the cloud metaphor to death). But how does it all begin? Take us back to that moment: the <em>podception<\/em>.<\/p>\n<p><strong>TN:<\/strong> Well, the first pod, per se, was just a bunch of USB drives strapped to a shelf in the data center attached to two Dell 2900 towers. It didn\u2019t last more than an hour in production. As soon as it got hit with load, it just collapsed. Seriously! We went live on this and it lasted an hour. It was a complete meltdown.<\/p>\n<p>Two things happened: The bus was completely unstable, so the USB drives were unstable. Second, the DRDB (Distributed Replicated Block Device) &#8212; which is designed to protect your data by live mirroring it between the two towers &#8212; immediately fell apart. You implement a DRDB not because it works in a well-running situation, but because it covers you in the failure mode. And in failure mode it just unraveled &#8212; in an hour. It went into a split-brain mode under the hardware failures that the USB drives were causing. A well-running DRDB is fully mirrored, and split-brained mode is when the two sides simply give up and start acting autonomously because they don\u2019t know what the other side is doing and they\u2019re not sure who is boss. The data is essentially inconsistent at that point because you can choose A or B but the two sides are not in agreement.<\/p>\n<p>While the USB specs say you can connect something like 256 or 128 drives to a hub, we were never able to do more than like, five. After something like five or six, the drives just start dropping out. We never really figured it out because we abandoned the approach. I just took the drives out and shoved them inside of the Dells, and those two became pods number 0 and 1. The Dells had room for 10 or 8 drives apiece, and so we brought that system live.<\/p>\n<p>That was what the first six years of this company was like, just a never-ending stream of those kind of moments &#8212; mostly not panic inducing, mostly just: you put your head down and you start working through the problems. There\u2019s a little bit of adrenaline, that feeling before a big race of an impending moment. But you have to just keep going.<\/p>\n<figure id=\"attachment_92736\" aria-describedby=\"caption-attachment-92736\" style=\"width: 1440px\" class=\"wp-caption aligncenter\"><a href=\"\/blog\/wp-content\/uploads\/2019\/09\/tim-video-pod40_edit.jpg\" data-rel=\"lightbox-gallery-WCSyxJtY\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-92736\" src=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2019\/09\/tim-video-pod40_edit.jpg\" alt=\"Tim working on Storage Pod\" width=\"1440\" height=\"820\" srcset=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-video-pod40_edit.jpg 1440w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-video-pod40_edit-300x171.jpg 300w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-video-pod40_edit-1024x583.jpg 1024w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-video-pod40_edit-768x437.jpg 768w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-video-pod40_edit-560x319.jpg 560w\" sizes=\"auto, (max-width: 1440px) 100vw, 1440px\" \/><\/a><figcaption id=\"caption-attachment-92736\" class=\"wp-caption-text\">Tim welcoming another Storage Pod to the family.<\/figcaption><\/figure>\n<p class=\"e-interview\"><strong>E:<\/strong> Wait, so this wasn\u2019t in testing? You were running this live?<\/p>\n<p><strong>TN:<\/strong> Totally! We were in friends-and-family beta at the time. But the software was all written. We didn\u2019t have a lot of customers, but we had launched, and we managed to recover the files: whatever was backed up. The system has always had self healing built into the client.<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> So where do you go from there? What\u2019s the next step?<\/p>\n<p><strong>TN:<\/strong> These were the early days. We were terrified of any commitments. So I think we had leased a half cabinet at the 365 Main facility in San Francisco, because that was the most we could imagine committing to in a contract: We committed to a year\u2019s worth of this tiny little space.<\/p>\n<p>We had those first two pods &#8212; the two Dell Towers (0 and 1) &#8212; which we eventually built out using external exclosures. So those guys had 40 or 45 drives by the end, with these little black boxes attached to them.<\/p>\n<p>Pod number 2 was the plywood pod, which was another moment of sitting in the data center with a piece of hardware that just didn\u2019t work out of the gate. This was Chris Robertson\u2019s prototype. I credit him with the shape of the basic pod design, because he\u2019s the one that came up with the top loaded 45 drives design. He mocked it up in his home woodshop (also known as a garage).<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> Wood in a data center? Come on, that\u2019s crazy, right?<\/p>\n<p><strong>TN:<\/strong> It was what we had! We didn\u2019t have a metal shop in our garage, we had a woodshop in our garage, so we built a prototype out of plywood, painted it white, and brought it to the data center. But when I went to deploy the system, I ended up having to recable and rewire and reconfigure it on the fly, sitting there on the floor of the data center, kinda similar to the first day.<\/p>\n<p>The plywood pod was originally designed to be 45 drives, top loaded with port multipliers &#8212; we didn\u2019t have backplanes. The port multipliers were these little cards that took one set of cables in and five cables out. They were cabled from the top. That design never worked. So what actually got launched was a fifteen drive system that had these little five drive enclosures that we shoved into the face of the plywood pod. It came up as a 15 drive, traditionally front-mounted design with no port multipliers. Nothing fancy there. Those boxes literally have five SATA connections on the back, just a one-to-one cabling.<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> What happened to the plywood pod? Clearly it\u2019s cast in bronze somewhere, right?<\/p>\n<p><strong>TN:<\/strong> That got thrown out in the trash in Palo Alto. I still defend the decision. We were in a small one-bedroom apartment in Palo Alto and all this was cruft.<\/p>\n<figure id=\"attachment_92675\" aria-describedby=\"caption-attachment-92675\" style=\"width: 1200px\" class=\"wp-caption aligncenter\"><a href=\"\/blog\/wp-content\/uploads\/2019\/09\/ten-years-1-woodenpod.jpg\" data-rel=\"lightbox-gallery-WCSyxJtY\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-92675\" style=\"max-width: 60%;\" src=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2019\/09\/ten-years-1-woodenpod.jpg\" alt=\"Wooden pod\" width=\"1200\" height=\"1600\" srcset=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/ten-years-1-woodenpod.jpg 1200w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/ten-years-1-woodenpod-225x300.jpg 225w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/ten-years-1-woodenpod-768x1024.jpg 768w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/ten-years-1-woodenpod-1152x1536.jpg 1152w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/ten-years-1-woodenpod-560x747.jpg 560w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/a><figcaption id=\"caption-attachment-92675\" class=\"wp-caption-text\">The plywood pod, RIP.<\/figcaption><\/figure>\n<p class=\"e-interview\"><strong>E:<\/strong> Brutal! But I feel like this is indicative of how you were working. There was no looking back.<\/p>\n<p><strong>TN:<\/strong> We didn\u2019t have time to ask the question of whether this was going to work. We just stayed ahead of the problems: Pods 0 and 1 continued to run, pod 2 came up as a 15 drive chassis, and runs.<\/p>\n<p>The next three pods are the first where we worked with Protocase. These are the first run of metal &#8212; the ones where we forgot a hole for the power button, so you\u2019ll see the pried open spots where we forced the button in. These are also the first three with the port-multiplier backplane. So we built a chassis around that, and we had <em>horrible<\/em> drive instability.<\/p>\n<p>We were using the Western Digital Green, 1 TB drives. But we couldn\u2019t keep them in the RAID. We wrote these little scripts so that in the middle of the night, every time a drive dropped out of the array, the script would put it back in. It was this constant motion and churn creating a very unstable system.<\/p>\n<p>We suspected the problem was with power. So we made the octopus pod. We drilled holes in the bottom, and ran it off of three PSUs beneath it. We thought: \u201cIf we don\u2019t have enough power, we\u2019ll just hit it with a hammer.\u201d Same thing on cooling: \u201cWhat if it\u2019s getting too hot?\u201d So we put a box fan on top and blew a lot of air into it. We were just trying to figure out what it was that was causing trouble and grief. Interestingly, the array in the plywood pod was stable, but when we replaced the enclosure with steel, it became unstable as well!<\/p>\n<figure id=\"attachment_92742\" aria-describedby=\"caption-attachment-92742\" style=\"width: 1200px\" class=\"wp-caption aligncenter\"><a href=\"\/blog\/wp-content\/uploads\/2019\/09\/podfather1.jpg\" data-rel=\"lightbox-gallery-WCSyxJtY\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-92742\" style=\"max-width: 60%;\" src=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2019\/09\/podfather1.jpg\" alt=\"Storage Pod with fan\" width=\"1200\" height=\"1600\" srcset=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather1.jpg 1200w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather1-225x300.jpg 225w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather1-768x1024.jpg 768w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather1-1152x1536.jpg 1152w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather1-560x747.jpg 560w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/a><figcaption id=\"caption-attachment-92742\" class=\"wp-caption-text\">Early experiments in pod cooling.<\/figcaption><\/figure>\n<p>We slowly circled in on vibration as the problem. That plywood pod had actual disk enclosure with caddies and good locking mechanisms, so we thought the lack of caddies and locking mechanisms could be the issue. I was working with Western Digital at the time, too, and they were telling me that they also suspected vibration as the culprit. And I kept telling them, \u2018They are hard drives! They should work!\u2019<\/p>\n<p>At the time, Western Digital was pushing me to buy enterprise drives, and they finally just gave me a round of<a href=\"\/blog\/enterprise-drive-reliability\/\"> enterprise drives<\/a>. They were worse than the consumer drives! So they came over to the office to pick up the drives because they had accelerometers and lot of other stuff to give us data on what was wrong, and we never heard from them again.<\/p>\n<p>We learned later that, when they showed up in an office in a one bedroom apartment in Palo Alto with five guys and a dog, they decided that we weren\u2019t serious. It was hard to get a call back from them after that \u2026 I\u2019ll admit, I was probably very hard to deal with at the time. I was this ignorant wannabe hardware engineer on the phone yelling at them about their hard drives. In hindsight, they were right; the chassis needed work.<\/p>\n<p>But I just didn\u2019t believe that vibration was the problem. It\u2019s just 45 drives in a chassis. I mean, I have a vibration app on my phone, and I stuck the phone on the chassis and there\u2019s vibration, but it\u2019s not like we\u2019re trying to run this inside a race car doing multiple Gs around corners, it was a metal box on a desk with hard drives spinning at 5400 or 7200 rpm. This was not a seismic shake table!<\/p>\n<p>The <a href=\"\/blog\/history-hard-drives\/\">early hard drives<\/a> were secured with EPDM rubber bands. It turns out that real rubber (latex) turns into powder in about two months in a chassis, probably from the heat. We discovered this very quickly after buying rubber bands at Staples that just completely disintegrated. We eventually got better bands, but they never really worked. The hope was that they would secure a hard drive so it couldn\u2019t vibrate its neighbors, and yet we were still seeing drives dropping out.<\/p>\n<p>At some point we started using clamp down lids. We came to understand that we weren\u2019t trying to isolate vibration between the drives, but we were actually trying to mechanically hold the drives in place. It was less about vibration isolation, which is what I thought the rubber was going to do, and more about stabilizing the SATA connector on the backend, as in: You don\u2019t want the drive moving around in the SATA connector. We were also getting early reports from Seagate at the time. They took our chassis and did vibration analysis and, over time, we got better and better at stabilizing the drives.<\/p>\n<p>We started to notice something else at this time: The Western Digital drives had these model numbers followed by extension numbers. We realized that drives that stayed in the array tended to have the same set of extensions. We began to suspect that those extensions were manufacturing codes, something to do with which backend factory they were built in. So there were subtle differences in manufacturing processes that dictated whether the drives were tolerant of vibration or not. Central Computer was our dominant source of hard drives at the time, and so we were very aggressively trying to get specific runs of hard drives. We only wanted drives with a certain extension. This was before the Thailand drive crisis, before we had a real sense of what the supply chain looked like. At that point we just knew some drives were better than others.<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> So you were iterating with inconsistent drives? Wasn\u2019t that insanely frustrating?<\/p>\n<p><strong>TN:<\/strong> No, just gave me a few more gray hairs. I didn\u2019t really have time to dwell on it. We didn\u2019t have a choice of whether or not to grow the storage pod. The only path was forward. There was no plan B. Our data was growing and we needed the pods to hold it. There was never a moment where everything was solved, it was a constant stream of working on whatever the problem was. It was just a string of problems to be solved, just \u201cwheels on the bus.\u201d If the wheels fall off, put them back on and keep driving.<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> So what did the next set of wheels look like then?<\/p>\n<p><strong>TN:<\/strong> We went ahead with a second small run of steel pods. These had a single Zippy power supply, with the boot drive hanging over the motherboard. This design worked until we went to 1.5TB drives and the chassis would not boot. Clearly a power issue, so Brian Wilson and I sat there and stared at the non-functioning chassis trying to figure out how to get more power in.<\/p>\n<p>The issue with power was not that we were running out of power on the 12V rail. The 5V rail was the issue. All the high end, high-power PSUs give you more and more power on 12V because that\u2019s what the gamers need &#8212; it\u2019s what their CPUs and the graphics card need, so you can get a 1000W or a 1500W power supply and it gives you a ton of power on 12V, but still only 25 amps on 5V. As a result, it\u2019s really hard to get more power on the 5V rail, and a <a href=\"\/blog\/life-and-times-of-a-backblaze-hard-drive\/\">hard drive<\/a> takes 12V and 5V: 12V to spin the motor and 5V to power the circuit board. We were running out of the 5V.<\/p>\n<p>So our solution was two power supplies, and Brian and I were sitting there trying to visually imagine where you could put another power supply. Where are you gonna put it? We can put it were the boot drive is, and move the boot drive to the side, and just kind of hang the PSU up and over the motherboard. But the biggest consequence with this was, again, vibration. Mounting the boot drive to the side of a vibrating chassis isn\u2019t the best place for a boot drive. So we had higher than normal boot drive failures in those nine.<\/p>\n<figure id=\"attachment_92743\" aria-describedby=\"caption-attachment-92743\" style=\"width: 1200px\" class=\"wp-caption aligncenter\"><a href=\"\/blog\/wp-content\/uploads\/2019\/09\/podfather3.jpg\" data-rel=\"lightbox-gallery-WCSyxJtY\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-92743\" style=\"max-width: 60%;\" src=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2019\/09\/podfather3.jpg\" alt=\"Storage Pod power supply\" width=\"1200\" height=\"1600\" srcset=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather3.jpg 1200w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather3-225x300.jpg 225w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather3-768x1024.jpg 768w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather3-1152x1536.jpg 1152w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather3-560x747.jpg 560w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/a><figcaption id=\"caption-attachment-92743\" class=\"wp-caption-text\">Tim holding the second power supply in place to show where it should go.<\/figcaption><\/figure>\n<p>So the next generation, after pod number 8, was the beginning of Storage Pod 1.0. We were still using rubber bands, but it had two power supplies, 45 drives, and we built 20 of them, total. Casey Jones, as our designer, also weighed in at this point to establish how they would look. He developed the faceplate design and doubled down on the deeper shade of red. But all of this was expensive and scary for us: We\u2019re gonna spend $10 grand!? We don\u2019t have much money. We had been two years without salary at this point.<\/p>\n<figure id=\"attachment_92744\" aria-describedby=\"caption-attachment-92744\" style=\"width: 2000px\" class=\"wp-caption aligncenter\"><a href=\"\/blog\/wp-content\/uploads\/2019\/09\/podfather2.jpg\" data-rel=\"lightbox-gallery-WCSyxJtY\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-92744\" src=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2019\/09\/podfather2.jpg\" alt=\"Storage Pod faceplates\" width=\"2000\" height=\"444\" srcset=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather2.jpg 2000w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather2-300x67.jpg 300w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather2-1024x227.jpg 1024w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather2-768x170.jpg 768w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather2-1536x341.jpg 1536w, https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/podfather2-560x124.jpg 560w\" sizes=\"auto, (max-width: 2000px) 100vw, 2000px\" \/><\/a><figcaption id=\"caption-attachment-92744\" class=\"wp-caption-text\">Casey Jones&#8217; faceplate vent design versions, with the final first generation below.<\/figcaption><\/figure>\n<p>We talked to Ken Raab from Sonic Manufacturing, and he convinced us that he could build our chassis, all in, for less than we were paying. He would take the task off my plate, I wouldn\u2019t have to build the chassis, and he would build the whole thing for less than I would spend on parts \u2026 and it worked. He had better backend supplier connections, so he could shave a little expense off of everything and was able to mark up 20%.<\/p>\n<p>We fixed the technology and the human processes. On the technology side, we were figuring out the hardware and hard drives, we were getting more and more stable. Which was required. We couldn\u2019t have the same failure rates we were having on the first three pods. In order to reduce (or at least maintain) the total number of problems per day, you have to reduce the number of problems per chassis, because there\u2019s 32 of them now.<\/p>\n<p>We were also learning how to adapt our procedures so that the humans could live. By \u201cthe Humans,\u201d I mean me and Sean Harris who joined me in 2010. There are physiological and psychological limits to what is sustainable and we were nearing our wits end.\u2026 So, in addition to stabilizing the chassis design, we got better at limiting the type of issues that would wake us up in the middle of the night.<\/p>\n<p class=\"e-interview\"><strong>E:<\/strong> So you reached some semblance of stability in your prototype and in your business. You\u2019d been sprinting with no pay for a few years to get to this point and then \u2026 you decide to give away all your work for free? You open sourced Storage Pod 1.0 on September 9th, 2009. Were you a nervous wreck that someone was going to run away with all your good work?<\/p>\n<p><strong>TN:<\/strong> Not at all. We were dying for press. We were ready to tell the world anything they would listen to. We had no shame. My only regret is that we didn\u2019t do more. We open sourced our design before anyone was doing that, but we didn\u2019t build a community around it or anything.<\/p>\n<p>Remember, we didn\u2019t want to be a manufacturer. We would have killed for someone to build our pods better and cheaper than we could. Our hope from the beginning was always that we would build our own platform until the major vendors did for the server market what they did in the personal computing market. Until Dell would sell me the box that I wanted at the price I could afford, I was going to continue to build my chassis. But I always assumed they would do it <a href=\"\/blog\/backblaze-is-faster-than-the-sound-of-panic-after-a-hard-drive-failure\/\">faster than a<\/a> decade.<\/p>\n<p>Supermicro tried to give us a complete chassis at one point, but their problem wasn\u2019t high margin; they were targeting too high of performance. I needed two things: Someone to sell me a box and not make too much profit off of me, and I needed someone who would wrap hard drives in a minimum performance enclosure and not try to make it too redundant or high performance. Put in one RAID controller, not two; daisy chain all the drives; let us suffer a little! I don\u2019t need any of the hardware that can support SSDs. But no matter how much we ask for barebones servers, no one\u2019s been able to build them for us yet.<\/p>\n<p>So we\u2019ve continued to build our own. And the design has iterated and scaled with our business. So we\u2019ll just keep iterating and scaling until someone can make something better than we can.<\/p>\n<p class=\"e-interview\" style=\"margin-bottom: 24px;\"><strong>E:<\/strong> Which is exactly what we\u2019ve done, leading from Storage Pod 1.0 to 2.0, 3.0, 4.0, 4.5, 5.0, to 6.0 (if you want to learn more about these generations, check out our <a href=\"\/blog\/backblaze-storage-pod-museum\/\" target=\"_blank\" rel=\"noopener noreferrer\">Pod Museum<\/a>), preparing the way for more than 800<a href=\"\/blog\/150-petabytes-of-cloud-storage\/\"> petabytes<\/a> of data in management.<\/p>\n<figure id=\"attachment_88030\" aria-describedby=\"caption-attachment-88030\" style=\"width: 4032px\" class=\"wp-caption aligncenter\"><a href=\"\/blog\/wp-content\/uploads\/slider5\/StoragePodMuseum.jpeg\" data-rel=\"lightbox-gallery-WCSyxJtY\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-88030\" src=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/slider5\/StoragePodMuseum.jpeg\" alt=\"Storage Pod Museum\" width=\"4032\" height=\"3024\" \/><\/a><figcaption id=\"caption-attachment-88030\" class=\"wp-caption-text\">The Backblaze Storage Pod Museum in San Mateo, California<\/figcaption><\/figure>\n<p style=\"text-align: center; margin: 12px 0 18px 0;\">\u2723\u00a0\u00a0 \u2723\u00a0\u00a0 \u2723<\/p>\n<p>But while Tim is still waiting to pass along the official Podfather baton, he\u2019s not alone. There was the early help from <a href=\"\/blog\/author\/brianw\/\">Brian Wilson,<\/a> Casey Jones, Sean Harris, and a host of others, and then in 2014,<a href=\"\/blog\/how-backblaze-buys-hard-drives\/\"> Ariel Ellis<\/a> came aboard to wrangle our supply chain. He grew in that role over time until he took over the responsibility over charting the future of the Pod via <a href=\"\/blog\/backblaze-labs-hardware-design\/\" target=\"_blank\" rel=\"noopener noreferrer\">Backblaze Labs<\/a>, becoming the <em>Podson<\/em>, so to speak. Today, he\u2019s sketching the future of Storage Pod 7.0, and &#8212; provided no one builds anything better in the meantime &#8212; he\u2019ll tell you all about it on our blog.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Backblaze Storage Pod has reached a ten-year milestone. We thought it would be a great opportunity to talk to the father of the pod, Tim Nufire, about the origins of the pod and its evolution into the game-changing cloud storage icon it has become.<\/p>\n","protected":false},"author":144,"featured_media":92734,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[7,23],"tags":[468],"class_list":["post-92724","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-storage","category-storage-pod","tag-b2cloud","entry"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>A Decade of Storage Pods: The Road to an Exabyte of Cloud Storage<\/title>\n<meta name=\"description\" content=\"10 years after open sourcing the Storage Pod, Backblaze deploys 2100 pods and is closing in on an exabyte of storage. Read about the journey to get here.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.backblaze.com\/blog\/origins-of-the-pod\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A Decade of Storage Pods: The Road to an Exabyte of Cloud Storage\" \/>\n<meta property=\"og:description\" content=\"10 years after open sourcing the Storage Pod, Backblaze deploys 2100 pods and is closing in on an exabyte of storage. Read about the journey to get here.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/\" \/>\n<meta property=\"og:site_name\" content=\"Backblaze Blog | Cloud Storage &amp; Cloud Backup\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/backblaze\" \/>\n<meta property=\"article:published_time\" content=\"2019-09-25T00:06:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-07-23T19:45:38+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1440\" \/>\n\t<meta property=\"og:image:height\" content=\"820\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Patrick Thomas\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@backblaze\" \/>\n<meta name=\"twitter:site\" content=\"@backblaze\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Patrick Thomas\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"19 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"A Decade of Storage Pods: The Road to an Exabyte of Cloud Storage","description":"10 years after open sourcing the Storage Pod, Backblaze deploys 2100 pods and is closing in on an exabyte of storage. Read about the journey to get here.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.backblaze.com\/blog\/origins-of-the-pod\/","og_locale":"en_US","og_type":"article","og_title":"A Decade of Storage Pods: The Road to an Exabyte of Cloud Storage","og_description":"10 years after open sourcing the Storage Pod, Backblaze deploys 2100 pods and is closing in on an exabyte of storage. Read about the journey to get here.","og_url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/","og_site_name":"Backblaze Blog | Cloud Storage &amp; Cloud Backup","article_publisher":"https:\/\/www.facebook.com\/backblaze","article_published_time":"2019-09-25T00:06:53+00:00","article_modified_time":"2021-07-23T19:45:38+00:00","og_image":[{"width":1440,"height":820,"url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg","type":"image\/jpeg"}],"author":"Patrick Thomas","twitter_card":"summary_large_image","twitter_creator":"@backblaze","twitter_site":"@backblaze","twitter_misc":{"Written by":"Patrick Thomas","Est. reading time":"19 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/#article","isPartOf":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/"},"author":{"name":"Patrick Thomas","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/7939165675bc36f0862dbe0b25d3657f"},"headline":"Interview With the Podfather: Tim Nufire on the Origins of the Pod","datePublished":"2019-09-25T00:06:53+00:00","dateModified":"2021-07-23T19:45:38+00:00","mainEntityOfPage":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/"},"wordCount":3935,"commentCount":2,"publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg","keywords":["B2Cloud"],"articleSection":["Cloud Storage","Storage Pod"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/","name":"A Decade of Storage Pods: The Road to an Exabyte of Cloud Storage","isPartOf":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/#primaryimage"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg","datePublished":"2019-09-25T00:06:53+00:00","dateModified":"2021-07-23T19:45:38+00:00","description":"10 years after open sourcing the Storage Pod, Backblaze deploys 2100 pods and is closing in on an exabyte of storage. Read about the journey to get here.","breadcrumb":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/#primaryimage","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg","width":1440,"height":820,"caption":"Tim, the Podfather, working in data center"},{"@type":"BreadcrumbList","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/origins-of-the-pod\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Interview With the Podfather: Tim Nufire on the Origins of the Pod"}]},{"@type":"WebSite","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","name":"Backblaze Cloud Solutions Blog","description":"Cloud Storage &amp; Cloud Backup","publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization","name":"Backblaze","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Backblaze"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/backblaze","https:\/\/x.com\/backblaze","https:\/\/www.youtube.com\/user\/Backblaze","https:\/\/en.wikipedia.org\/wiki\/Backblaze"]},{"@type":"Person","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/7939165675bc36f0862dbe0b25d3657f","name":"Patrick Thomas","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/patrick_thomas-e1569451539653-150x150.png","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/patrick_thomas-e1569451539653-150x150.png","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/patrick_thomas-e1569451539653-150x150.png","caption":"Patrick Thomas"},"description":"Patrick Thomas is the Vice President of Marketing at Backblaze. He has managed all aspects of content development and strategy across a number of industries, including literary publishing, gaming, and tech. He has developed and edited New York Times bestsellers and Wall Street Journal books of the year, and written for National Geographic and the San Francisco Chronicle. He loves nothing more than learning, and Backblaze\u2019s steady beat of innovation feeds that love every day. LinkedIn: Patrick Thomas.","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/author\/patrick\/"}]}},"jetpack_featured_media_url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2019\/09\/tim-working-in-datacenter.jpg","_links":{"self":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/92724","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/users\/144"}],"replies":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/comments?post=92724"}],"version-history":[{"count":0,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/92724\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media\/92734"}],"wp:attachment":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media?parent=92724"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/categories?post=92724"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/tags?post=92724"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}