Storage Pod 4.0: Direct Wire Drives – Faster, Simpler and Less Expensive

March 19th, 2014

blog-pod40-header
For the first time since the original Storage Pod, Backblaze is announcing a completely redesigned approach with the introduction of the first “direct wire” Storage Pod. This new Storage Pod performs four times faster, is simpler to assemble, and delivers our lowest cost per gigabyte of data storage yet. And, once again, it’s open source.

The Original Storage Pod

In order to provide unlimited online backup for $5 per month when Backblaze first started in 2007, we needed to figure out the least expensive way to “attach a hard drive to the Internet.” This meant cost-efficiently attaching as many drives as possible to a single motherboard. We tried USB hubs, daisy-chaining Firewire, and various other approaches. In the end, we found one that worked: port multiplier backplanes.

We used these port multipliers to design a system with 9 five-drive NAS backplanes that connected via 3 SATA cards to the motherboard. The results were incredibly dense storage, currently 180TB in a 4U rack, from off the shelf consumer commodity parts. This design has served us through Storage Pod versions 1.0, 2.0, and 3.0, and today stores 100 petabytes of customer data in the Backblaze cloud.

However, the port multiplier backplanes had three key issues:

  1. They were one of the least reliable hardware components,
  2. When they had an issue, they affected 5 drives at once, and
  3. They were not completely a commodity part, thus making them somewhat difficult to buy (especially for someone building a single Storage Pod on their own.)

The New Storage Pod Design

Fast forward nearly 5 years as we introduce Storage Pod 4.0 and a completely new way of attaching 45 drives to the motherboard. The 5-drive backplanes are gone, replaced by individual direct wire SATA/Power connectors. In concert with this we replaced the three 4-port SATA cards with two high performance 40-port HighPoint Rocket 750 cards delivering SATA 3 throughput. Simpler to build and maintain, and seriously fast.


The New Storage Cost – $0.051 per GB

When we designed Storage Pods 2.0 and 3.0 our goal was to reduce the total cost per GB to store data. Pod 2.0 was great, but when the Thailand Drive Crisis pushed up hard drive prices we struggled to lower the cost of Storage Pod 3.0. With Storage Pod 4.0 we are pleased to announce it delivers 180TB of storage for just $0.051 per GB — our lowest cost per GB to date.

What’s New?

Here’s a video highlighting some of the new features in Storage Pod 4.0.

We summarized all of the changes below and as always we’ve included a parts list at the end of the post.

    Screaming Fast SATA Cards — Storage Pod 4.0 uses two HighPoint Rocket 750 cards replacing the previous 3 SATA cards. Each Rocket 750 supports 40 directly connected SATA device channels via ten 6 Gbps Mini-SAS ports. The Rocket 750 card connects to the motherboard via an 8x PCIe 2.0 connection that supports 8 parallel “lanes” to the motherboard. Currently we are seeing a 4 to 5 times performance improvement in throughput when using this set up versus Storage Pod 3.0.
    Raid Array Cards
    Backblaze has never had a problem with data throughput speed based on how we use a Storage Pod in production, but the increased throughput does dramatically reduce the amount of time it takes to sync the RAID arrays and burn in the system. Typically, it takes 5-6 days to do these activities when we use 4TB disk drives. With Storage Pod 4.0, this drops down to 1-2 days, so we can get 4.0 pods into service much quicker.
    Direct Wire Hard Drive Connections — In previous Storage Pods, there were 9 backplanes, each supporting 5 hard drives. There were 9 SATA cables from the SATA cards to the backplanes and the backplane multiplexed to 5 different drives. With Storage Pod 4.0 each of the 45 drives has its own direct wire connector that provides power and a SATA connection for each drive. Each SATA connection goes directly back to the SATA card, the Rocket 750. Fifteen of the connectors are affixed to one of three cross supports in the chassis.
    pod_3_with_backplanes pod_4_with_direct_wire
    The direct wire connectors have multiple benefits:
  • They are fundamentally simpler in design than the 5-drive backplanes.
  • By replacing the 5-drive backplanes, all of the set up and management of the 45 drives is done on the motherboard or in the SATA cards.
  • The direct wire connectors are less likely to break or burn out than the previous backplanes.
  • When a 5-drive backplane went down, we had 5 drives go offline. Now if a connector goes down, only 1 drive will go offline.
    New SATA cables — Each of the Rocket 750 Cards has 10 ports with Mini-SAS connectors. The new SATA cables start with a Mini-SAS connector and fan out to 4 individual cables each of which terminates in the direct wire connector. Here is a link showing the mapping between the Mini-SAS ports on each Rocket 750 card to the individual direct wire connectors. The new cables are designed to provide both SATA and power to each hard drive. Power is provided via Molex connectors attached to a wiring harness that is connected to, and part of, the Power Supply wiring harness.
    One Power Supply — Previous Storage Pod versions had two power supplies, which totaled nearly 25% of the cost of the Storage Pod chassis (not including the drives). The two power supplies were not for failover purposes, we just needed the power they created. For Storage Pod 4.0 there is only one power supply to power everything. It generates enough power to run all 45 drives, the motherboard and all other components. Having one power supply saves on the cost and complexity, reduces the weight, and increases reliability of Storage Pod 4.0.
    pod-side-by-side-40-560
    Thinner Chassis and New Side Rails — One of the byproducts of replacing the 5-drive backplanes is the reduction of the width of the chassis by 1/2 inch from 17 5/8 to 17 1/8. This allows us to use side rails for mounting Storage Pods in our cabinets. The previous L-bracket rails were known to be sharp in all the wrong places and more than once drew blood from our Operations staff. For them, this is the most important feature of Storage Pod 4.0.
    Faster CPU – We’ve upgrade the CPU to an i3-3240 (Ivy Bridge) from an i3-2100 (Sandy Bridge) CPU. This makes processing faster and when combined with the Rocket 750 cards provided the boost in throughput of 4 to 5 times we’ve experienced. The i3-2120 and the i3-3220 have also been tested in this environment and seem to work fine, but given they are currently either the same price or a little more expensive as the i3-3240 we are currently using that CPU.
    Boot Drive – The boot drive was moved from the side to the rear of the chassis. This was done to allow for mounting holes for the new side rails on each side of the chassis. This also makes the cabling a bit easier. The new location can support either one or two 2.5” boot drives.

More Storage?

Storage Pod 4.0 uses forty-five 4 TB hard drives for a storage capacity of 180 TB, the same as Storage Pod 3.0. Why didn’t we use 5 TB or even the 6 TB drives in Storage Pod 4.0? In short, these higher capacity drives are not yet widely available and are not cost effective for Backblaze at the moment.

For example, as of March 17, 2014 an HGST 6 TB internal hard drive listed on Amazon was $745.67 or about $0.124 per GB with an availability of 5 drives.

5 TB drives are also hard to find, with only LaCie’s external 5TB drive showing up on Newegg on March 17, 2014. The list price for this external drive is $449.99, making the cost about $0.09 per GB, although that’s the cost of the entire unit, not just the hard drive.

Currently 4 TB internal hard drives that we use are readily available for $160.00 or less in many places, that’s about $0.04 per GB.

While we’re on the topic, here are the 4 TB hard drives we currently use in our Storage Pods. We have tested these extensively and are satisfied with their performance.

  • HGST (formerly Hitachi) — HDS5C4040ALE630
  • Seagate – ST4000DM000

Although we like the HGST drives, as of March 1st, the availability of this HGST model is becoming limited as its looks like they are being EOL’d. We currently have the Seagate 4 TB drives installed in about 50 storage pods and they are performing well. We expect to try out the Western Digital WD40EFX (RED) drives in the near future.

Over the next few months we expect the price of the 5 TB and 6TB drives to decrease. At some point we’ll buy 50 drives (45 plus spares) and install them in a Storage Pod to see how they perform in our environment. As we’ve seen previously, some drives do better than others in our Storage Pods. If we were able to use the 6 TB drives for example we’d have 270 TB of storage in a 4U rack. As always we’ll keep you posted.

Buying a Storage Pod

If you are interested in buying a Storage Pod we recommend you talk to the folks at 45 Drives, a division of Protocase. They are very capable of building you a Backblaze inspired Storage Pod that can be customized if desired. They also can sell you a Storinatorâ„¢ their 180TB storage solution. The Storinator is similar to Storage Pod 4.0 in many ways, and is readily available for $5,403 for the non-redundant version and $6,888 for the redundant version. This does not include the price of the drives, tax, or shipping. While the 45 Drives system does not include hard drives, they test each system they build with drives before they ship the unit to you. Then it’s up to you to buy and install the hard drives. Remember that while the 45 Drives products are inspired by Backblaze, we do not build, test, or warrant them. Since we don’t sell the Storage Pods and don’t make money off any ancillary services, we don’t provide support for people who buy them from 45 Drives or other manufacturers.

Assembling Your Own Storage Pod

For those of you who want to assemble your own Storage Pod there is a parts list at the end of this post. You can order the 4U metal box from 45 Drives and start from there. Their 4U metal box includes the 3 metal “cross supports” to mount the direct wire connectors (see the parts list) that you can also purchase from 45 Drives. Once you have the metal box and the direct wire connectors, the other parts can be sourced as noted in the parts list or you can add in your own motherboard, memory, SATA cards, etc. to meet your needs. If you go this route, here is a wiki page, where you can learn more about the assembly process. Backblaze does not provide support or warranty to anyone who wishes to use the design to build or assemble a Storage Pod.

The Price of Storage Pod 4.0

In truth, there are actually three different prices for a 180 TB Backblaze Storage Pod 4.0:

  1. The price Backblaze contractually pays to build about 25 Storage Pods a month.
  2. The price you would pay to purchase a Backblaze inspired Storage Pod or a Storinator from 45 Drives.
  3. The price you would pay to purchase the parts and build your own Storage Pod.

In our previous Storage Pod blog posts we published the Backblaze prices, but people soon learned that they had to pay more for many of the parts, sometimes a lot more. We buy parts in bulk and as such get better pricing on many of the parts. So the real cost of building your own Backblaze Storage Pod was higher than expected. So to be clear, here are the approximate costs for a 4.0 Storage Pod with 180 TB of storage i.e. forty-five 4TB hard drives;

    • Backblaze – $9,305
    • Make Your Own – $10,587
    • 45 Drives – $12,603

    The Backblaze price of Storage Pod 4.0 of $9,305 for the complete 180TB system is about 13% less than the cost of Storage Pod 3.0 at $10,700 for an 180TB system. Of course these prices are not guaranteed as things like hard drive shortages can change prices pretty quickly.

    The Cost per Gigabyte — Onward and Downward

    A Storage Pod costs Backblaze $9,305 for 180TB of storage which translates into $0.0517 per GB. This is our lowest price per GB ever as you can see below:

    pod-4-cost-per-GB-comparison

    Over the next few months we’ll continue to look for ways to decrease the various costs and with the expected increase in storage capacities such as 5 and 6 TB drives we’re sure the cost will drop below $0.05 per GB.

    Appendix A — Parts List

    This is the parts list for Storage Pod 4.0, not including the 45 hard drives. The prices listed are from the sources noted and are as of the date of the publication of this blog post. Taxes and shipping are not included. You may find better prices with a little more research. Good luck.

    Item
    Qty
    Price
    Total
    4U Custom Case
    Includes case, anti-vibration assemblies, cross supports, etc.
    www.45drives.com
    1
    $872.00
    $872.00
    850 Watt Power Supply
    Zippy PSL-6850P Power Supply
    www.circotech.com
    1
    $355.99
    $355.99
    Motherboard
    Supermicro MBD-X9SCL-F (MicroATX)
    www.amazon.com
    1
    $151.99
    $151.99
    8GB DDR3 RAM
    Kingston 4GB – KVR16E11S8 DDR3 1600 (PC3 12800)
    www.newegg.com
    2
    $44.90
    $89.80
    CPU
    Intel Core i3 processor i3-3240
    www.newegg.com
    1
    $119.00
    $119.00
    Case Fan
    Dynatron 120x120x25mm 2600RPM Case Fan
    www.newegg.com
    6
    $9.99
    $59.94
    SATA Card
    Rocket 750 40 port PCI2.0 SATA III HBA CARD
    www.amazon.com
    2
    $688.20
    $1,376.40
    Direct Wire SATA Cable 4/24
    Direct wire cable SATA/Power 4 connectors – 24″ length
    www.45drives.com
    3
    $17.25
    $51.75
    Direct Wire SATA Cable 4/30
    Direct wire cable SATA/Power 4 connectors – 30″ length
    www.45drives.com
    8
    $17.25
    $138.00
    Direct Wire SATA Cable 1/24
    Direct wire cable SATA/Power 1 connector – 24″ length
    www.45drives.com
    3
    $17.25
    $51.75
    On/Off Switch
    FrozenCPU ele-302 Bulgin Vandal Momentary LED Power Switch 12″ 2-pin
    www.frozencpu.com
    1
    $24.95
    $24.95
    Boot Drive
    WD Blue WD5000LPVX 500GB 5400 RPM Internal
    www.amazon.com
    1
    $50.92
    $50.92
    Slide Rails
    Supermicro MCP-290-00053-0N Side Rails – 1 pair
    www.provantage.com
    1
    $62.45
    $62.45
    Screw: 4-40 X 3/16 Phillips: Secure Boot drive (B00916VMBU)
    www.amazonsupply.com
    4
    $0.027
    $0.11
    Standoff: Round 6-32 X 1/4 Dia X 5/16 Lng: Motherboard mount (15TSP054B)
    www.grainger.com
    1
    $0.39
    $0.39
    Screw: 6-32 X 3/16 Phillips PAN SST: Motherboard standoff (B00916VMBU)
    www.amazonsupply.com
    1
    $0.71
    $0.71
    Screw: 4-40 X 5/16 Phillips PAN ZPS: Connection Brackets (B000MN6TCI)
    www.amazonsupply.com
    30
    $0.04
    $1.20
    Screw: 4-40 X 5/16 Phillips PAN ZPS: Direct Wire Connectors (B000MN6TCI)
    www.amazonsupply.com
    90
    $0.04
    $3.60
    Acousti Anti-Vibration Silicone Washers AP-1003W-C (8 pack)
    www.quietpcusa.com
    8
    $1.25
    $5.00
    Foam Tape, 1″ x 36yds x 1/16 in black: Vibration lids
    www.shop3m.com
    .25
    $19.25
    $4.84
     
    TOTAL:  $3,387.28

    Notes:

    1. Screws and other small parts are generally purchased in packages (e.g. 10 or 100), but the price list assumes you will only pay for the quantity you actually need.

     

    Andy Klein

    Andy Klein

    Andy has 20+ years experience in technology marketing. He has shared his expertise in computer security and data backup at the Federal Trade Commission, Rootstech, RSA and over 100 other events. His current passion is to get everyone to back up their data before it's too late.
    Andy Klein

    Latest posts by Andy Klein (see all)

    • bossturbo

      Very cool, and great info. Thanks guys! Also, I’m going to switch to you guys for my cloud storage needs, dig it!

    • Grnch

      Wow, are these guys serious? $6,888???

      Not knocking you Backblaze guys, your Storage Pod design is awesome, and this 4.0 version rocks! :)

      But those 45Drives people are completely insane, if they think the case with no drives costs $6,888.

      You can get a refurbished Thumper off of eBay for $1,000 with much better specs and WITH DRIVES INCLUDED. Check this one out, for example:

      http://www.ebay.com/itm/Sunfire-X4540-2x-AMD-Opteron-2435-Six-core-2-6Ghz-32GB-16TB-Sata-Storage-Server-/161353413595?pt=COMP_EN_Servers&hash=item259169b3db

      Let’s compare…

      45Drives Storinator: 2 CPU cores @ 3.4GHz, 8GB RAM, Redundant PSUs, 45 drive bays, No drives – $6,888

      Sun x4540: 12 (yes, twelve) CPU cores @ 2.6GHz, 32GB RAM, Redundant PSUs, 48 drive bays, 16 x 1TB drives – $999

      You get quadruple the RAM and CPU, a better build quality of the case, and as a bonus you get 16 hard drives that you can sell off on eBay, so the empty Thumper really ends up costing you something like $500.

      No offense to the 45D guys, but old school Sun engineers were something else. Even just comparing pictures from Google Images you can see that the Storinator case looks flimsier than the Thumper case.

      Sure, the Thumper is a refurb, but does that really justify a 14x price difference? Is the Storinator really 14x better? Really?

      I don’t think it’s better than even a single refurbished Thumper (with its superior CPU/RAM/case), let alone 14 of them!

      • ctn

        Used Server you can only buy one of (and you have to deal with Oracle for support)
        vs.
        Brand new server that can be mass-produced on the cheap.

        Obviously a business that finds servers on eBay isn’t going to last very long…

        Also, I looked at an x4540 11.5TB (Sun didn’t sell barebones, and replacement drives cost your first-born) about six years ago for work, they costed $34,995 new!

        Have a look at page 60 of http://www.absol.co.za/medialib/Downloads/Home/AboutUs/News/Just%20the%20Facts.pdf

        • Grnch

          Yeah, sure, but my point is that those prices and those times are long gone.

          In this era of dirt cheap cloud storage, you can no longer command such ridiculous prices, and yes, $7k for an empty cage is just as ridiculous today as Sun’s prices were back in the day, when adjusted for technology progress and performance difference of Sun’s hardware.

          You claim it’s a brand new server that can be mass produced on the cheap, but why isn’t that reflected in the price? I’m sorry, but $7k ain’t cheap for an empty shell with no storage.

          • http://www.fullmoonblog.com/ Flatlinebb

            “In this era of dirt cheap cloud storage”
            Where do you think this cloud storage comes from? You think cloud storage providers, like Amazon or Backblaze just outsource their storage to another cloud storage provider? At some point, someone has to build an actual physical server with the physical storage. BB is interested in storage capacity per sq ft, not server chassis quality. Their Pods have the highest storage density in the industry, so who cares how flimsy the cases are? Google’s servers barely even have cases – they are are just mostly exposed motherboards (http://www.epanorama.net/newepa/wp-content/uploads/2009/05/google_server.jpg) And yet Google runs the world.

    • Tharun

      May i know which switches and routers used in your datacenter

    • Tony Yu

      Despite the spec list looks pretty ghetto compare to other big companies, it looks very interesting. Since I know the i3 does a good job in those HEAVY loaded machines, I think I could build one for lot less price but of course with lot less storage. Only problem is that what system you guys use in those racks? Please tell if you will. Thank you in advance!

      • http://www.fullmoonblog.com/ Flatlinebb

        If you are asking about the OS, it’s Debian Linux. At least according to the job descriptions.

    • Gumby

      Any idea where someone can purchase the sata connectors? I dont mean the whole pre wired case part that 45drives sells, just the Santa connector that connects to the drive on one side and then power/data on the other.

    • abslc

      Are the design files for the chassis available for download? I can’t seem to find a download link like in previous versions.

    • Pavel Ni

      Some friendly advice here.

      1. You surely have the most expensive power button of any server appliance I saw so far.

      2. You can order a custom aluminium (save on air shipment) 4U enclosures in China or Russia for less than $200 each in any sheet metal shop even in quantities of less than 100 units.

      3. Do you really need a boot hard drive? You can get a single chip, preflashed, bootable USB flash drives specially made for x86 appliances for as cheap as $2.

      4. Backplanes – you can get cheap and reliable backplanes for around 10 USD each, without port multiplexers.

      5. CPU/RAM/Motherboard/SATA cards – the whole X86 server part can be thrown away and replaced with a two FGPA board design that can run both NIC, sata host controllers, RAID, Accelerated file system, and a cheap x86 IP capable of running Linux or *BSD.

      6. Don’t ever use anything made by Silicon Image. Storage products is a side business for them. They are not committed to it.

      7. Last one: I can recommend contra-rotating fans, so the appliance will still be functional and need no service in case of single fan failure.

      • calmdownbro

        I was also shocked to see the boot drivers (a small SSD would also do, and is also more reliable), let alone the power button. Agreed on the rest of the points as well.

      • jeffmacdonald

        These were really interesting suggestions. I’d be curious to hear backblazes feed back about why they would or wouldn’t work well for them.

    • Billco

      I still don’t understand your insistence on direct-connect SATA, when SAS expanders are quite mainstream. You can buy a Supermicro 45-bay JBOD chassis for much less than this, where all you need to do is populate the drives and connect the (included) SAS cables. Redundant power is included too. Then all you need is a $200 HBA and 1-2 SFF-8088 cables per JBOD. I’ve built beefy head units that easily handle a rack full of JBODs, each pushing out upwards of 2 TB/sec (bytes not bits)… sure you spend more on the “brain” since a lowly i3 won’t keep up with 100+ drives, so you drop an extra $1000 on dual hexa-core xeons – once! Then you only have one boot drive, one set of configs to worry about. If total throughput is not an issue, you can daisy-chain the JBODs and have even more drives per head unit.

      I tell ya, SAS expanders are trivial to work with and you don’t need to reinvent the wheel. If you really want to stick with the top-loading layout, I bet you could reuse Supermicro’s 16-bay backplanes and mount them along the bottom, but with the added benefit of a built-in expander, so you’d only need one cable going to each backplane. Way cleaner, cheaper and more versatile than these Rocketpoint cards since you have the freedom to choose any cheap HBA – even the weird Supermicro AOC ones with the backwards PCBs.

    • Michael

      I must second about the SAS topology. Even if you stay with your current build (1 Mainboard for each chassis). You can easily get a 8 port SAS controller for <200$ (e.g. Supermicro AOC or IBM M1015) and another two 36 port SAS expanders for <400$ each (e.g. Chenbro, Intel or HP) and this gives you 48 to 64 usable ports for 350$ for each POD.

      If you wanna save more money you could use e.g. 2 chassis with 1 mainboard. All you needed is instead of a controller with 8 internal ports, one with 8 external for the second chassis. Simply connect the chassis without mainboard to the chassis with mainboard by two (or even one) sas cables. You can even save more money by cascading the sas expanders and leaving out the controller with external ports. Just watch out for the maximum number of drives for the SAS Controller.

      Maybe this can help you reduce the cost for POD 4.1 or 5

    • http://strongallies.com Drew

      If you are using consumer based drives without TLER have you had any problem with drives spontaneously falling out of the RAID arrays? What RAID levels are you using? What is the point of putting 45 hard drives capable of 100MB+ (4500MB) throughput each into a system that is only at most connected to the network at 2Gbps (250MB)

      • Andrew Zammit

        This is for nearline storage. Throughput is not a priority; with backup systems $/GB/sqft is more important.

    • Paul Malishev

      Won’t 2.5″ drives be more cost effective? Probably you can put more drives in a smaller (3U? 2U?) case.

    • Ehsan Farahani

      hi
      thanks for your review. i have some questions about your storage pod.
      1. What is the maximum speed of reading and writing on local storage pod and network?
      2. which used nas server software?

      3. which used software for management nas server?

      best regards

    • Random

      PXE boot servers are better