We thought ten people would care; instead a million people read our Storage Pod 1.0 blog post where we open sourced the Backblaze Storage Pod design and introduced the world’s most cost-efficient way to store big data. The interest grew when we published our Petabytes on a Budget: Revealing More Secrets blog post that announced Storage Pod 2.0, which doubled the amount of storage and reduced the price. Since then several companies have built businesses selling Storage Pods inspired by Backblaze to hundreds of organizations around the world who are storing hundreds of petabytes of data on their own Storage Pods. Today we introduce Backblaze Storage Pod 3.0 which stores more data, costs less, is more reliable, and is easier to service.
For Storage Pod 3.0 we redesigned the chassis and upgraded many of the components. Most of the changes are aimed at improving the reliability and flexibility of the Storage Pod. The full parts list is in Appendix A at the end of this post.
Here are the highlights:
- 180 terabytes of storage. With the availability of 4 TB hard drives a Storage Pod can now be configured to store 180 TB (45 x 4 TB). As a bonus the same chassis and components can be used with any capacity of 3.5” hard drives.
- Anti-vibration drive bay assemblies. There are now 3 assemblies, one for each row of 15 drives. Each assembly is designed to lock down a row of drives in place. These assemblies replace the “drive bands” around each drive. This saves nearly an hour during Pod assembly and makes drive replacement easier as well.The key advantage of the drive bay assemblies is to reduce vibration. These assemblies not only keep the drives still, they also keep them firmly seated in the backplanes. Over the past several months we have tested different models of drives in the new drive bay assemblies and we have seen a dramatic improvement in overall system performance along with lower drive failure rates.
- Upgraded motherboard. We now use the Supermicro MBD-X9SCL-F motherboard which replaces the previous model. The new motherboard adds a host of advanced processing features from Intel, as well as upgrades the PCIe slots to double throughput. Note that while our currently specified SATA cards do not take advantage of this increased throughput, it’s nice to know we can use it in the future.
- More motherboard choices. We added standoffs to the chassis to provide better support for Micro ATX motherboards while still supporting the Standard ATX form factor. Specifically the new standoffs support the outer edge of the Micro ATX boards.
- CPU. We upgraded the CPU to a 2nd generation Intel Core i3-2100 processor to replace the end of lifed (EOL) i3-540 model. This also gets us a little bump in clock speed (3.06 to 3.1 GHz), lower power usage (65 versus 73 watts), and more supported RAM (32GB up from 16GB).
- Memory. We changed memory suppliers, so now the memory is certified by Supermicro.
- Boot drive options. With Storage Pod 3.0, boot drives can now be 2.5” or 3.5” and we allow a second 2.5” drive to be attached for use in a redundant RAID1 boot volume. Boot-up drives can also be traditional HD or SSD drives. Backblaze switched to 2.5” boot drives because they are less expensive and are more reliable, but we didn’t want to eliminate support for 3.5” drives in case anyone needs a higher capacity 3.5” drive.
- Backplanes. While we continue to use the same backplanes (Sil3726 chipset) as with Pod 2.0, there is another backplane based on the newer Silicon Image Sil3826 chipset that can be used. If you do use the Sil3826 based backplanes, you’ll notice that boot up can take a long time due to a large number of time-out/retry errors during the boot process. Eventually the boot up process will succeed. To fix this, you can use these instructions to update the backplane driver in your Linux kernel so you can use the Sil3826 based backplanes.
- SATA cables. We replaced our SATA cable vendor with Nippon Labs. If you decide to use another vendor, look for cables that are SATA II or SATA III compliant and test them extensively. While any SATA II or SATA III cable should work we have found quality control problems with a number of vendors. In all cases it was the connectors, not the cables, which were defective.
- Metal standoffs in the chassis. We replaced the plastic standoffs with metal components that can be manufactured as part of the chassis to reduce the cost and assembly time.
- Improved airflow. The vent design was improved to increase airflow through the pod. We’ve never really had a problem with heat in the pods, and we’d like to keep it that way.
- Chassis rivets. We replaced many of the screws with rivets. This simplifies the manufacturing process and saves times during the assembly process. Note, if you end up buying a case from Protocase (more on them later), they continue to use screws versus rivets. The two manufacturing processes produce the same basic chassis regardless.
- Costs Less. For Storage Pod 2.0, the price for the components without drives was $1,984.00. For Pod 3.0 the price is $1,942.59, or $37.41 less — or about a 1.9% decrease in the cost (see Appendix A for a parts breakdown). The main component in the total cost will be the hard drives. The lingering effects from the Thailand drive crisis and consolidation in the drive industry have meant that even today hard drive prices are higher than they were when Pod 2.0 was introduced back in July of 2011 but here’s one way you can save a little on the price of hard drives.
Backblaze currently has over 450 Storage Pods deployed and manages nearly 50 petabytes of data. There are also many Storage Pods being used by organizations around the globe. Along the way we’ve learned a few things:
- Firmware revisions matter. Watch out, manufacturers update hardware and upgrade firmware without changing their model number. These updates are intended to fix bugs but in the process new bugs can be introduced. Read the release notes carefully and downgrade the firmware to the version you’ve tested whenever possible. When updates are unavoidable, test them thoroughly before deploying them.
- Let your vendors do the testing. Components like memory, PCIe cards and hard drives are modular and should all play well together. Unfortunately, the specs that allow for this interoperability (PCIe, SATA, DDR3, etc) are as complicated as are the components themselves. Because of this, use ‘certified’ components whenever possible. This will minimize problems and avoid finger pointing between vendors if problems do arise.
- Don’t make random changes to our design. It might be tempting to try a build a pod out of the spare parts lying around your office but don’t do it. The components specified in the parts list in Appendix A are known to work well together. We believe in iteration and experimentation but don’t reinvent the wheel unless you have to.
- There is more to power than just Watts. ATX power supplies deliver power at several voltages or ‘rails’ (12V, 5V, 3.3V, etc). Each vendor imposes unique limits on the amount of power you can draw off of each rail and unused power on one rail cannot be used on another. In particular, most high end power supplies are designed to deliver most of their power on the 12V rail because that is what high end gamer PCs use. Unfortunately, hard drives draw a lot of power off the 5V rail and can easily overwhelm a high wattage power supply. You will hit serious problems if power requirements for each component are not met so be careful if you don’t use the power supplies we recommend.
- Keep it simple. For hard drives, SATA cards, SATA cables, backplanes, etc. you should use the same vendor, part number and/or model number as much as possible. By keeping things simple you reduce the number of variables that need to be considered if things go wrong. If you do mix components do it intentionally and for a good reason like comparing the performance of 3 different hard drives. And when you do this, make sure you’ve set things up in a way that allows you to draw clear conclusions. For example: with hard drives, if you want to compare the performance of 3 different models you should arrange them so that each RAID array is homogeneous. You should also take care to make sure each array is spread across all backplanes and SATA cards so that you don’t have IO hotspots which could taint your results.
- Things change. Just when you get comfortable with a part, it will be discontinued or upgraded. We buy in quantity, we buy spares, and we have substitutes ready to go at any time. We also look for parts that have long-term support policies. We realize you may not be able to do these things, so be prepared when you upgrade one component that something may break.