A Behind-the-Scenes Look at Designing the Next Storage Pod

By  /  September 1st, 2015

Backblaze Labs Storage Pod Design
Tim holds up a drive beam and shows the design to Matt who is attending the meeting courtesy of a telepresence robot. They are discussing the results of a prototype Storage Pod developed using Agile development methodology applied to hardware design. The Storage Pod they are discussing was the result of the fifth sprint (iteration) of prototyping. Based on the evaluation by the Scrum (project) team, the prototype pod is either sent to the datacenter for more testing and hopefully deployment, or it is decided that more work is required and the pod is salvaged for usable parts. Learning from failure, as well as from success, is a key part of the Agile methodology.

Prototyping Storage Pod Design

Backblaze Labs uses the Scrum framework within the Agile methodology to manage the Storage Pod prototyping process. To start, we form a Scrum team to create a product backlog containing the tasks we want to accomplish with the project: redesign drive grids, use tool-less lids, etc. We then divide the tasks into four-week sprints, each with the goal of designing, manufacturing, and assembling one to five production-worthy Storage Pods.

Let’s take a look at some of the tasks on the product backlog that the Scrum team is working through as they develop the next generation Storage Pod.

Drive Grids Become Drive Beams

Each Storage Pod holds 45 hard drives. Previously we used drive grids to hold the drives in place so that they connect to the backplanes. Here’s the process of how drive grids became drive brackets that became drive beams.

Current Storage Pod 4.5 Design: Each pod has an upper and lower drive grid. Hard drives are placed in each “bay” of the grid and then a drive cover presses lightly on each row of 15 hard drives to secure them in place.
Storage Pod Drive Grids
Storage Pod Scrum – Sprint 1 Design: This drive bracket design allowed drives to be easily installed and removed, but didn’t seem to hold the drives well. A good first effort, but a better design was needed.
Drive Guides Sprint 2
Storage Pod Scrum – Sprint 3 Design: Sprint 3 introduced a new drive beam design. On the plus side, the drives were very secure. On the minus side, the beams sat too high in the chassis, so it was nearly impossible to remove a drive with your fingers.
Drive Beams Sprint 3
Storage Pod Scrum – Sprint 5 Design: The height of the drive beams was shortened in Sprint 5, making them low enough so that the hard drives could be removed by using your fingers.
Drive Beams Sprint 5
Operations Impacts the Design
Having Operations personnel on the Scrum team helped get the design of the drive beams right. For the first two sprints, the hard drives installed in the prototypes for testing purposes were small capacity drives. These drives were ¾ inch wide. Once installed, the gap between the drives provided enough space to use your fingers to remove the drives. Matt from Operations suggested that we use large capacity hard drives, like we do in production, to test the prototypes. These drives are one inch wide. When we did this in Sprint 3, we discovered there was no longer enough room between the drives our for fingers. This led to lowering the height of the drive beams in Sprint 5.

Drive Guides

In conjunction with the introduction of the new drive beam design from Sprint 3, we needed a way to keep the drives aligned and immobile within the beams. Drive guides were the proposed solution. These guides are screwed or inserted into the four mounting holes on each drive. The drive then slides into cutouts on the drive beam securing the drive in place. There were three different drive guide designs:

Sprint 2 – The first design for drive guides was to use screws in the four mounting holes on each drive. The screw head was the right size to slide into the drive beam. Inserting 180 screws into the 45 drives (45 x 4) during assembly was “time expensive” so other solutions were examined.
Sprint 3 – The second design for drive guides was plastic “buttons” that fit into each of the four drive-mounting screw holes. This was a better solution, but when the height of the drive beams was lowered in Sprint 4, the top two drive guide buttons were now above the drive beams with nothing to attach to.
Drive Guide Buttons Sprint 3
In Sprint 5, full-length drive guides replaced the buttons, with one guide attached to each side of the drive. The full-length guides allowed the drive to be firmly seated even though the drive beams were made shorter. They had the added benefit of taking much less time to install on the assembly line.
Drive Rails
Time Versus Money: 3D Printing
One lesson learned as we proceeded through the sprints was that we could pay extra to get the parts needed to complete the sprint on time. In other words, we could trade money for time. One example is the drive guides. About halfway through Sprint 5 we created the design for the new full-length drive guides. We couldn’t procure any in time to finish the sprint, so we made them. In fact, we printed them using a 3D printer. At $3 each to print ($270 for all 45 drives), the cost is too high for production, but for prototyping we spent the money. Later, a plastics manufacturer will make these for $0.30 each in a larger quantity – trading time to save money during production.

Drive Lids

Drive lids were introduced in Storage Pod 3.0 to place a small amount of pressure on the top of the hard drives, minimizing vibrations as they sat in the drive bay. The design worked well, but the locking mechanism was clumsy and the materials and production costs were a bit high.

Sprint 2 – We wanted to make it easier to remove the drive lids yet continue to apply the lid pressure evenly across the drives. This design looked nice, but required 6 screws to be removed in order to access a row of drives. In addition, the screws could cross-thread and leave metal shards in the chassis. A better solution was needed.
Screw Down Drive Lids
Sprint 4 – The candidate design was tool-less; there were no screws. This drive lid flexed slightly when pressed so as to fit into cutouts on the drive beams. The Scrum team decided to test multiple designs and metals to see if any performed better and have the results at the next sprint.
Drive Lids Sprint 4
Sprint 5 – The team reviewed the different drive lids and selected three different lids to move forward. Nine copies of each drive lid were made, and three Storage Pods were given to operations for testing to see how each of the three drive lids performed.
Drive Lids Sprint 5

Backplane Tray

In Storage Pod 4.5, 3.0, and prior versions, the backplanes were screwed to standoffs attached to the chassis body itself. Assembly personnel on the Scrum team asked if backplanes and all their cables, etc. could be simplified to make assembly easier.

Sprint 5 – The proposed solution was to create a backplane tray on which all the wiring could be installed and then the backplanes attached. The resulting backplane module could be pre-built and tested as a unit and then installed in the pod in minutes during the assembly process.
Backblaze Tray Sprint 5
Sprint 7 – The design of the backplane module is still being finalized. For example, variations on the location of wire tie downs and the wiring routes are being designed and tested.
Backblaze Tray Sprint 7
Metal Benders Matter
One thing that may not be obvious is that the actual Storage Pod chassis, the metal portion, is fabricated from scratch each sprint. Finding a shop that can meet the one-week turnaround time for metal manufacturing required by the sprint timeline is time-consuming, but well worth it. In addition, even though these shops are not directly part of the Scrum team, they can provide input into the process. For example, one of the shops told us that if we made a small design change to the drive beams they could fabricate them out of a single piece of metal and lower the unit production cost by up to half.

The Scrum team

Our Storage Pod Scrum team starts with a Scrum Master (Ariel/Backblaze), a designer (Petr/Evolve), and a Product Owner (Tim/Backblaze). Also included on the team are representatives from assembly (Edgar/Evolve and Greg/Evolve), operations (Matt/Backblaze), procurement (Amanda/Evolve), and an executive champion (Rich/Evolve). At the end of each four-week sprint, each person on the team helps evaluate the prototype created during that sprint. The idea is simple: identify and attack issues during the design phase. A good example of this thinking is the way we are changing the Storage Pod lids.

Current Storage Pods require the removal of twelve screws to access the drive bay and another eight screws to access the CPU bay. The screws are small and easy to lose, and they take time to remove and install. Operations suggested tool-less lids be part of the Scrum product backlog and they were introduced in Sprint 3. A couple of iterations (Sprints) later, we have CPU bay and drive bay lids that slide and latch into place and are secured with just two thumb screws each that stay attached to the lid.

Tool-less Lids Sprint 3

Going From Prototype to Production

One of the supposed drawbacks of using Scrum is that is it is hard to know when you are done. Our goal for each sprint is to produce a production-worthy Storage Pod. In reality, this will not be the case, since each time we introduce a major change to a component (drive beams, for example) things break. This is just like doing software development, where new modules often destabilize the overall system until the bugs are worked out. The key is for the Scrum team to capture any feedback and categorize it as: 1) must fix, 2) new idea to save for later, or 3) never mind.

Here’s the high-level process the Scrum team uses to move the Storage Pod design from prototype to production.

  1. Manufacture one to five Storage Pods chassis per sprint until the original product backlog tasks are addressed and, based on input from manufacturing, assembly and operations, the group is confident in the new design.
  2. The Scrum team is responsible for the following deliverables:
    • Final product specifications
    • Bill of materials
    • Final design files
    • Final draft of the work instructions (Build Book)
  3. Negotiate production pricing to manufacture the chassis. This will be done based on a minimum six-month production run.
  4. Negotiate production pricing to acquire the components used to assemble the Storage Pod: power supplies, motherboards, hard drives, etc. This will be done based on a minimum six-month production run.

Once all of these tasks are accomplished the design can be placed into production.

The Storage Pod Scrum is intended to continue even after we have placed Storage Pods into production. We start with a product backlog, build and test a number of prototypes and then go into production with a feature complete, tested unit. At that point, any new ideas generated along the way would then be added to the product backlog and the sprints for the next Storage Pod production unit would begin.

Scrum Process Diagram

Will the ongoing Storage Pod Scrum last forever? We started the Scrum with the intention of continuing as long as one or more of the following are being accomplished:

  • We are improving product reliability
  • We are reducing the manufacturing cost
  • We are reducing the assembly cost
  • We are reducing the maintenance cost
  • We are reducing the operating cost

The Next Storage Pod – Not Yet

Where are we in the process of creating the next Storage Pod version to open source? We’re not done yet. We have a handful of one-off pods in testing right now and we’ve worked through most of the major features in the product backlog, so we’re close.

In the meantime, if you need a Storage Pod or you’re just curious, you can check out https://www.backblaze.com/storage-pod.html. There you can learn all about Storage Pods, including their history, the Storage Pod ecosystem, how to build or buy your very own Storage Pod and much more.

Andy Klein

Andy Klein

Andy has 20+ years experience in technology marketing. He has shared his expertise in computer security and data backup at the Federal Trade Commission, Rootstech, RSA and over 100 other events. His current passion is to get everyone to back up their data before it's too late.
  • Stoatwblr

    Devil’s advocate here….

    Based on the 4.0 vs 4.5 design changes

    Why not switch back to the rocketport cards and use larger dumb (as in no port multiplier) backplanes with 4 SFF-8087 sockets onboard? That would massively simplify the wiring over the individual socket design whilst simultaneously reducing the electronics/mechanical component count and assembly time dramatically, plus allow the narrower chassis (or would allow 16 drives in the wider one for a 48 drive count, but I’m trying to keep this inside existing metalwork to reduce engineering issues.

    Taking it one step further, if you switched from 10 to 12 socket rocketport cards you would only need to use one of them, further reducing component count and costs.

  • Jeppelelle

    Is there a reason to why you still only have 3 rows of drives? Why not add another one? Take a look at the Dell XA90, it has 6 rows for a total of 90 drives per 4U, you guys basically have half the density right now. Is weight or power an issue for your racks or why not add another row of drives per storage pod?

    • Right now it’s a bit of a power draw but we’re constantly trying out different configurations!

  • Дмитрий Шивцов

    Why images on right are so small and not lightboxed? Opening them in new tab is so uncomfortable :(

  • asposium

    for the PCB could you use a single multilayer board that covers the entire area of the drives.
    make the bottom of the board either contact free or a ground plain
    then use surface mount components for drive connectors
    any vias aren’t coated onto the ground plain
    result, the pcb can be mounted direct onto the case work, no standoffs required
    if connectors cannot be surface mount
    then incorporate standoffs into the case
    spot weld bent metal tabs
    a single PCB would save a lot of cabling.
    a common power connection and SATA connections along one edge in a convenient location

    • Stoatwblr

      “then use surface mount components for drive connectors”

      I suspect this would lead to long-term reliability issues as drive vibration stressed the PCB lands, unless the SM sockets incorporated some form of mechanical isolation.

  • r3muxd

    What kind of drives do you guys use?

  • Ben Jolly

    Andy, being that the new drive beam is far more secure and presumably introducing less vibration, does that then remove the need for the drive lid?

    Also, in developing the full length drive sliders, where clips considered as is typically the case in these arrangements, making something off the shelf available and perhaps reducing costs?

    • Andy Klein

      Good question about the drive lids. I’m not sure we considered leaving out the drive lids, but it is worth running the experiment in the lab. I’ll propose it to the Scrum team.
      We looked at drive rails with clips, but they either had to be attached by screws or they were more expensive then the $0.30/each (or less) we’ll pay for the one’s we spec’d. Also, most clipped from the top, and there was nothing for the clip to attach to when we lowered the drive beams. So in our case it was cheaper to have our own made.