Tim holds up a drive beam and shows the design to Matt, who is attending the meeting courtesy of a telepresence robot. They are discussing the results of a prototype Storage Pod developed using Agile development methodology applied to hardware design. The Storage Pod they are discussing was the result of the fifth sprint (iteration) of prototyping. Based on the evaluation by the Scrum (project) team, the prototype Pod is either sent to the data center for more testing and hopefully deployment, or it is decided that more work is required and the Pod is salvaged for usable parts. Learning from failure, as well as from success, is a key part of the Agile methodology.
Prototyping Storage Pod Design
Backblaze Labs uses the Scrum framework within the Agile methodology to manage the Storage Pod prototyping process. To start, we form a Scrum team to create a product backlog containing the tasks we want to accomplish with the project: redesign drive grids, use tool-less lids, etc. We then divide the tasks into four-week sprints, each with the goal of designing, manufacturing, and assembling one to five production-worthy Storage Pods.
Let’s take a look at some of the tasks on the product backlog that the Scrum team is working through as they develop the next generation Storage Pod.
Drive Grids Become Drive Beams
Each Storage Pod holds 45 hard drives. Previously we used drive grids to hold the drives in place so that they connect to the backplanes. Here’s the process of how drive grids became drive brackets that became drive beams.
|Operations Impacts the Design
Having Operations personnel on the Scrum team helped get the design of the drive beams right. For the first two sprints, the hard drives installed in the prototypes for testing purposes were small capacity drives. These drives were ¾ inch wide. Once installed, the gap between the drives provided enough space to use your fingers to remove the drives. Matt from Operations suggested that we use large capacity hard drives, like we do in production, to test the prototypes. These drives are one inch wide. When we did this in Sprint 3, we discovered there was no longer enough room between the drives our for fingers. This led to lowering the height of the drive beams in Sprint 5.
In conjunction with the introduction of the new drive beam design from Sprint 3, we needed a way to keep the drives aligned and immobile within the beams. Drive guides were the proposed solution. These guides are screwed or inserted into the four mounting holes on each drive. The drive then slides into cutouts on the drive beam securing the drive in place. There were three different drive guide designs:
|Time Versus Money: 3D Printing
One lesson learned as we proceeded through the sprints was that we could pay extra to get the parts needed to complete the sprint on time. In other words, we could trade money for time. One example is the drive guides. About halfway through Sprint 5, we created the design for the new full-length drive guides. We couldn’t procure any in time to finish the sprint, so we made them. In fact, we printed them using a 3D printer. At $3 each to print ($270 for all 45 drives), the cost is too high for production, but for prototyping we spent the money. Later, a plastics manufacturer will make these for $0.30 each in a larger quantity—trading time to save money during production.
Drive lids were introduced in Storage Pod 3.0 to place a small amount of pressure on the top of the hard drives, minimizing vibrations as they sat in the drive bay. The design worked well, but the locking mechanism was clumsy and the materials and production costs were a bit high.
In Storage Pod 4.5, 3.0, and prior versions, the backplanes were screwed to standoffs attached to the chassis body itself. Assembly personnel on the Scrum team asked if backplanes and all their cables, etc. could be simplified to make assembly easier.
|Metal Benders Matter
One thing that may not be obvious is that the actual Storage Pod chassis, the metal portion, is fabricated from scratch each sprint. Finding a shop that can meet the one-week turnaround time for metal manufacturing required by the sprint timeline is time-consuming, but well worth it. In addition, even though these shops are not directly part of the Scrum team, they can provide input into the process. For example, one of the shops told us that if we made a small design change to the drive beams they could fabricate them out of a single piece of metal and lower the unit production cost by up to half.
The Scrum Team
Our Storage Pod Scrum team starts with a Scrum master (Ariel/Backblaze), a designer (Petr/Evolve), and a product owner (Tim/Backblaze). Also included on the team are representatives from Assembly (Edgar/Evolve and Greg/Evolve), Operations (Matt/Backblaze), Procurement (Amanda/Evolve), and an executive champion (Rich/Evolve). At the end of each four-week sprint, each person on the team helps evaluate the prototype created during that sprint. The idea is simple: identify and attack issues during the design phase. A good example of this thinking is the way we are changing the Storage Pod lids.
Current Storage Pods require the removal of twelve screws to access the drive bay and another eight screws to access the CPU bay. The screws are small and easy to lose, and they take time to remove and install. Operations suggested tool-less lids be part of the Scrum product backlog and they were introduced in Sprint 3. A couple of iterations (Sprints) later, we have CPU bay and drive bay lids that slide and latch into place and are secured with just two thumb screws each that stay attached to the lid.
Going from Prototype to Production
One of the supposed drawbacks of using Scrum is that is it is hard to know when you are done. Our goal for each sprint is to produce a production-worthy Storage Pod. In reality, this will not be the case, since each time we introduce a major change to a component (drive beams, for example) things break. This is just like doing software development, where new modules often destabilize the overall system until the bugs are worked out. The key is for the Scrum team to capture any feedback and categorize it as: 1) must fix, 2) new idea to save for later, or 3) never mind.
Here’s the high-level process the Scrum team uses to move the Storage Pod design from prototype to production.
- Manufacture one to five Storage Pods chassis per sprint until the original product backlog tasks are addressed and, based on input from Manufacturing, Assembly, and Operations, the group is confident in the new design.
- The Scrum team is responsible for the following deliverables:
- Final product specifications
- Bill of materials
- Final design files
- Final draft of the work instructions (Build Book)
- Negotiate production pricing to manufacture the chassis. This will be done based on a minimum six-month production run.
- Negotiate production pricing to acquire the components used to assemble the Storage Pod: power supplies, motherboards, hard drives, etc. This will be done based on a minimum six-month production run.
Once all of these tasks are accomplished the design can be placed into production.
The Storage Pod Scrum is intended to continue even after we have placed Storage Pods into production. We start with a product backlog, build, and test a number of prototypes and then go into production with a feature complete, tested unit. At that point, any new ideas generated along the way would then be added to the product backlog and the sprints for the next Storage Pod production unit would begin.
Will the ongoing Storage Pod Scrum last forever? We started the Scrum with the intention of continuing as long as one or more of the following are being accomplished:
- We are improving product reliability
- We are reducing the manufacturing cost
- We are reducing the assembly cost
- We are reducing the maintenance cost
- We are reducing the operating cost
The Next Storage Pod: Not Yet
Where are we in the process of creating the next Storage Pod version to open-source? We’re not done yet. We have a handful of one-off Pods in testing right now and we’ve worked through most of the major features in the product backlog, so we’re close.
In the meantime, if you need a Storage Pod or you’re just curious, you can check out https://www.backblaze.com/b2/storage-pod.html. There you can learn all about Storage Pods, including their history, the Storage Pod ecosystem, how to build or buy your very own Storage Pod and much more.