Recent Outages: Why We Accelerated Registry Changes

Looking Out for Our Team, Customers, and Community

Earlier in February, Backblaze experienced two unexpected service disruptions. We take this very seriously and we apologize for any disruptions that you may have experienced as a result of these instances. We would also like to explain what happened and what steps we have taken to avoid such interruptions in the future.

What Happened:

Backblaze B2 Cloud Storage can serve a variety of use cases, including hosting content. Sometimes cloud storage services are used to host malicious content. The use of malicious content by any of our customers violates Backblaze policy. Any time we receive notice that an account is serving such content involving our cloud storage system, our Compliance team investigates and instigates appropriate actions as needed. (For reference, our preferred reporting channel is an email to reportphishing@backblaze.com.)

Sometimes domain registrars notice or receive these complaints and then reach out to the involved parties with a notice and mandate that the content be removed or the account will be suspended. That is what happened in both outage cases mentioned above: Our registrar notified us of malicious content from certain customers and requested that we remove specific links for these customers from Backblaze B2 in order to avoid suspension of the Backblaze B2 domain.

On February 12, due to a variety of factors—including a notice from a third-party registrar outside of our established communication process—the Backblaze B2 domain was suspended and we worked with our registrar to bring it back online shortly thereafter. This prompted us to review and refine our operating procedures (for example, we expanded our response team and hardened our escalation process) to coordinate more effectively with our third-party registrar, and to move up exploration of developer-friendly registrar solutions more fitting for current and future needs.

On February 17, a similar scenario occurred again. In this case, we acted in line with our new processes, yet despite our removal of the content as they had requested, the domain was still suspended by our third-party registrar. The domain was brought back up shortly thereafter, but for anyone experiencing outages for their Backblaze B2-hosted data who were unable to log in as a result of the suspension, we believe that any outage is too long.

What We’ve Changed

After the second domain suspension, we accelerated our decision to switch registrars—with determination to migrate as fast as possible. We opted to move to Cloudflare, whom we already partner closely with as part of the Bandwidth Alliance and whom we see as offering proven, enterprise-worthy service to a wide array of businesses and organizations. Beyond their technology, one of the key benefits of working with our partners at Cloudflare is that we’ll have a direct line of communication in the event of any future content takedown requests.

We started the process of switching our registration to Cloudflare on February 18 and completed the process on February 23.

We truly appreciate the clear communication and suggestions from our customers and others who were experiencing issues—and yet we aim to make any such future communication unnecessary. If you have any additional questions, please feel free to post them in the comments below and we’ll respond as we’re able!

print

About Jeremy Milk

Jeremy Milk is a storybuilder who heads the Backblaze Product Marketing team. He's spent more than two decades honing his craft in product and consumer goods marketing leadership roles at companies including Intuit, WePay (acquired by JPMorgan Chase), and The Clorox Company. Outside the office, he can often be found near a soccer field, on a running trail, or fueling on coffee and tacos. Follow him on LinkedIn or Twitter.