Extended Maintenance
(Thursday 09/07/2017 2-6pm Pacific)

By | September 5th, 2017

As part of our Backblaze maintenance window (regularly Thursday 2-3pm Pacific), we are doing maintenance on some core Backblaze systems and may experience longer downtime affecting Backblaze B2 and Backblaze Computer Backup systems (both Personal and Business).

The planned extended maintenance will be from Thursday September 7th, from 2-6pm Pacific. We are anticipating that most systems will come back sooner, but want folks to plan accordingly in case the outage does last the full term.

During this time new Backblaze Computer Backup and Backblaze B2 account creation may be affected.
Additionally, during this time, some users may not be able to:

  • Log in to www.backblaze.com
  • Access and utilize Backblaze B2
  • Create Personal Backup and Business Backup Restores
  • Start downloading newly created restores (restores already downloading should continue)
  • Continue uploads from their computer (data will continue to be scanned and uploads will resume automatically after the maintenance)
  • Access and utilize the Backblaze Mobile Apps

Not all Backblaze customers will be affected during the maintenance window, but it is best to assume you will not have access to the Backblaze services as noted above during the maintenance window.

All Backblaze services are expected to resume once maintenance is complete.

We will be updating this post with information as we go through the system maintenance, as well as updating our Twitter feeds for more information.

*Update 4:00 PM* Maintenance is running on schedule.
*Update 5:51 PM* Maintenance is taking longer than expected, though some services have resumed.
*Update 9:07 PM* Maintenance still underway. Ops and DC teams working to get systems online.
*Update 9:51 PM* Systems coming back online. Thank you for your patience.
*Update 10:55 PM* Most systems should be back online. If you are still experiencing issues, please contact support, and they will try to help. We apologize for this extended outage as we upgraded our systems and will use it as a learning opportunity to make sure we get better moving forward. If you’re still seeing maintenance pages, please clear your cache and try again.

Yev

Yev

Chief Smiles Officer at Backblaze
Yev enjoys speed-walking on the beach. Speed-dating. Speed-writing blog posts. The film Speed. Speedy technology. Speedy Gonzales. And Speedos. But mostly technology. He also runs social for Backblaze.

Follow Yev on:
Twitter: @YevP | LinkedIn: Yev Pusin | Google+: Yev Pusin
  • Dave Higbee

    How do we move forward from this knowing that high availability of the B2 object store is disregarded so easily?

  • Jaymes

    Maintenance is maintenance; just send your customers a quick email to warn them! :)

  • karl

    Large maintenance windows do not affect my use case for B2. I only use for cold data backups. B2 is still maturing, so mission critical applications should be using Google, Azure or AWS — for now. I also use other cloud providers — why not it’s so cheap.

    I calculated this month’s up-time assuming no further down time.

    30 days in September
    This month’s up-time is 98.75%

    Remember, Backblaze is a one data-centre operation. With the new data centre coming online (not sure if already is) perhaps core services will switch to that during maintenance windows / failure. Duplicating B2 data across both data centres might be within Backblaze’s service plan in the future — would be an interesting incite to know this.

    Due to my personal use case being cold backup, I do not require high availability, so on other services I reduce my redundancy and availability for cheaper storage.

  • Kris

    It’s Sunday morning.

    Are we not yet done?

    • Hi Kris – sorry for the late reply, I believe you may have been cached, if you reload the page that might do the trick!

  • Dariusz

    BB has customers outside of US as well. So peak, off-peak maintenance window discussion is relative. At least if done over weekends it would be less disruptive for majority of customer base in all geographies.

    • Thank you for the suggestion, it’s tough to hammer out the perfect time for everyone, but I’m making sure that we review all of the suggestions.

  • Chainsman

    You really, really stink at this.

    Maintenance windows should be in the early hours of the morning in the US Eastern Time zone. I’m flabbergasted that you don’t already know this. Literally everyone else already knows this.

    I believe that you mean well, but your naivete suggests some dangerously bad decisions are being made in other, perhaps more critical, areas of your business.

    Sure, you publish hard drive failure data, and your B2 object storage is the cheapest next to Amazon Glacier, but you need to clean up your act and start acting seriously.

    The fact that your “scheduled maintenance window” extended hours longer than expected is no surprise to me and shouldn’t be any surprise to anyone else familiar with BackBlaze.

    Finally, the availability of your employees should have nothing to do with the scheduling of this maintenance. I’m also flabbergasted that you don’t already know this.

    Amateurs.

    • Thanks for the feedback, I’m making sure that our managing team sees all of it so that we can get better! Sorry for the long outage, and thank you for the patience.

      • Chainsman

        Thank you for replying so thoughtfully and I hope your day gets better!

        • We’re coming back online now so it’ll get better soon! I’m taking this very seriously so we do learn from it. This was definitely not how we expected this upgrade to go.

          • Chainsman

            Good luck. I hope you’ll survive this.

  • Gray Lucas

    I’m a recent ex-Crashplan customer with two days left of my BB trial, also in the tech industry for 15+ years and have been in your shoes more times than I like to admit.
    Obviously this sort of outage truly is unacceptable, I think that’s abundantly clear. The key for me moving forward will be to see if this nightmare ever happens again. Fool me once- you know the rest of the story.
    BUT, I would wager a bet Yev, that you and your engineers are more stressed, anxious and upset about this extended outage than anyone badgering on social media. Not saying people don’t have a right to be upset- they certainly do, but sh*t happens. It’s what Backblaze learns as a company from this horrid night that will make or break the future for the company – I hope for greener pastures and better decisions moving forward.
    +1, these sort of outages should NEVER happen during business hours, period!
    +1, I sincerely hope this NEVER happens again.
    But lastly, chin up Yev, there’s a beer with your name on it after this is all over. Hang in there BB team, I know behind the scenes you’re working your tails off to get us back online, keep on keeping on.

    • Hi Gray! Systems are coming back online now. The large influx of folks over the last few weeks put our back against the wall so “ripping the band-aid off” as they say was our best bet to make sure we’re ready to continue receiving the incoming masses. We know it sucks and you’re right, our Ops and Datacenter teams are having a pretty bad night, but we’ll be buying them a round tomorrow. This is never good for anyone and we know the timing wasn’t great, but we wanted to make sure we had enough awake personnel to take care of things if they went wrong, and it turns out we needed them! Thank you for the patience, this is definitely not the norm! And also welcome aboard – hopefully you can get over our first impression!

  • Sharan Harkisoon

    It’s shenanigans like this that will make users look elsewhere. Some of us actually care about our customers and set maintenance windows appropriately. Taking down your entire service to increase capacity sounds real fishy.

    • Sorry to hear that Sharan, we have had a lot of server load since one of our competitors left the market – we’re trying to adjust but an unexpected influx of this magnitude required this maintenance. We’re working to get things back up as quickly as possible.

  • Matthew Moog

    I’m just going to add to the fray here but on the other side of the fence – Backblaze as well as a number of other “big name” cloud based storage solutions are all feeling the effects of what was arguably the most popular cloud based backup solution shutting down. The amount of data that Backblaze is taking in is likely staggering for the majority of us admins to even comprehend.

    Speaking from an IT Consultant standpoint, I would much rather go to my clients proactively and let them know that we’ve experienced an unexpected event that is potentially going to impact performance and reliability of our service offerings and as a result we are going to take an unexpected maintenance window to get ahead of this as opposed to explaining to my client that our services have stumbled because we decided to “wait it out”.

    Add to that, Backblaze isn’t “online” storage. It’s “nearline” storage. I would question any of the admins screaming in the comments about their practice for managing data and expectations if a couple of hour outage with a “nearline” storage provider is causing this much pain that they feel a need to make threats in the comments on a post explaining why this is happening.

    I’ll also add one more thought as an IT Consultant – the WORST thing that someone can do is get management riled up in a critical time. This simply causes management to come down on the techs. let the techs tech. It’s their job. Don’t add stress to it by throwing a tantrum and stomping your feet at management. It will just introduce the likelihood of said techs making mistakes as they rush to meet an arbitrary deadline that has now been set. believe me – they’re plenty aware that there is a service outage and are likely already freaking out trying to get services restored before their clients begin stomping their feet and throwing a tantrum like 6 a year old that didn’t get their ice cream. Oh…. Never mind. Too late.

    Semi-related: Really? Backblaze made it pretty clear in their posting that there is a massive influx of data and users occurring and you feel that voicing your dislike and threatening to go to another provider is going to phase them? C’mon. Seriously. They’re trying to proactively get ahead of this and your upset about that. Just as you would have been upset if they went down due to being reactionary instead of proactive. Maybe they can shove everyone who complains onto a pod with the slowest disk and slowest bandwidth available to it but is super stable. But then they’ll just complain about the speed.

    Thanks Backblaze! Great service at darn reasonable prices and thank you for being proactive and working to get ahead of this instead of sitting on it until it causes an incredibly long and protracted outage!

    Edit to add this: Looks like the majority of comments are from people who created an account just to complain about the outage. Incredible.

  • Ed

    This is unbelievable. Now not scheduled to come back until 12:30am ET. I have a hurricane coming and backups can’t run.

    • Ed we are so sorry for the maintenance taking longer – we know you’re trying to move quickly and so are we. Hoping everything comes back soon.

      • Ed

        are you still on schedule for a 12:30am ET return to normal?

        • Chainsman

          Nope, still not available at 01:33 ET, even though they claimed it would be back (at the latest update) at 01:30 ET.

          Amateurs.

          It’s insultingly stupid to call this a “scheduled maintenance period.”

          What a joke.

          Stop blaming this on CrashPlan refugees. If you designed the service properly, an influx of new customers would not AND SHOULD NOT have mattered to your EXISTING customers.

        • Ed – no we missed that window unfortunately. But we are back online now and systems are rolling back to normal. You should be good to go soon.

  • Andy Chan

    Really upset customer here. I lost all my files more than a week ago.

    First of all, I have problem to order the backup drive. Finally, figured out my Ghostery plugin was the root cause but your customer service was not friendly at all. Come on! I was very frustrated and why would I have to troubleshoot different browsers and disable my plugins to get my backup that I paid for!?!?!

    After the backup drive created for a few days, I got an email told me there was a disruption and I have to start the restore to hard drive ALL OVER AGAIN! That added additional days before the hard drive can be shipped out.

    Finally, Fedex came to my door steps with the hard drive. I connected the drive and login to BB to get my security code. You know what? SCHEDULED MAINTENANCE!!!!!! and still on going…

    Upset customer still can’t restore the files after more than a week.

    • So sorry about that Andy. We just recently learned about the Ghostery issue affecting payments (they’ve updated their service with a fix now). We are working hard to get the system back online so you can get your access code and get to your data. We are sorry it has taken this long for you to receive it, and we’re doing all we can tonight to get you access.

  • Lanstar

    I’m a new B2 customer. I also manage a datacenter for a Midwest university.

    Let me add my voice to what looks like many others…

    Maintenance windows that require down time don’t happen during normal waking hours and don’t happen on work days at all. Saturday night / Sunday morning – starting after midnight Pacific would be an example of something I’d probably approve (with lots of notice to my campus) if I were asked about scheduling service interrupting maintenance. Doing service interrupting maintenance on a Thursday afternoon is just tone deaf, amateurish and reeks of a decision made by a “bean counter” more worried about salary expense than customer relations.

    If you guys want to stem this customer relations disaster, you need to offer something of value to your customers by way of apology… and then NEVER do this again.

    • Thank you for the feedback, we’ll make sure that our managing team sees it.

  • Crispin Munns

    It’s 1pm in Sydney now, just sayin’… How’s it going?

    A daytime MW just screams of something hitting the fan, not getting services back at the end of the planned MW also looks like a seat of the pants type deal, I feel sorry for the techs.
    I’d guess that Crashplan announcing their withdrawal from the consumer cloud market has seen a LOT of new users and a sudden influx of data on Backblaze, they’ve probably got capacity problems now and are scrambling to install some recently delivered hardware.

    Assuming they got no prior warning of this from Crashplan (making BackBlaze look bad is good for them) I can partly sympathise. I’m sure there were some hastily raised PO’s two weeks ago…

    • This maintenance was unplanned and due to unexpected and severe load on most of our systems – we needed to expand capacity and increase stability quickly – so yes, we’re doing what we can and our Ops Techs are dealing with a lot of pressure. We know it’s not ideal, and thank you for understanding Crispin!

      • Crispin Munns

        Feel for you and the techs, hang in there! It’s only data, no one is going to die.

  • Mel Nathanson

    I just started my trial today and started my initial backup. TWO weeks?? Really? It is now 10:08 on the east coast, 7:08 your time and the maintenance was supposed to be done by 6:00 PM, your time. This is not really the best time to be doing maintenance. It should be scheduled in off hours, not when people are trying to use it

    • Mel, we apologize for the long maintenance and unforeseen issues which are causing the downtime to increase. Our management team is aware and we apologize for the inconvenience.

  • Reed Zhao

    I feel so upset to continue to be a B2 customer. Whoever comes out with the idea of maintenance during peak business hour just says: I don’t care.

    • Sorry Reed, we’re sending all feedback to management and hope to avoid this in the future.

      • Reed Zhao

        Now, it is 10:00pm est.(1 hour past the announced schedule) Our app is half-broken without images stored on B2. We had to send push notification to users explaining the situation.

        A year ago I persuaded my team to choose B2 over other storage solutions for the lower cost and acceptable reliability, I did not know there is an *official* maintenance window of 2~3PM. If I knew it, we would stick with S3.

        B2 is an API service. It is meant to be running 24/7 autonomously.

        • DH4

          I can confirm from multiple regions it is still down. Over 5.5 hours of downtime and counting.

          • Joshua

            Their CTO should be fired for incompetence. What kind of a company schedules downtime for peak internet traffic hours? They should throw in some free service, as compensation.

          • Hi, I’m Brian, the CTO.

            > What kind of a company schedules downtime for peak internet traffic hours?

            The kind of company that got blindsided this week with an unholy number of CrashPlan refugees and needed to upgrade a central account server. We apologize for the outage, and now that it is behind us and we can plan for the growth ahead, we really hope you will stay with us as a customer.

            > Their CTO should be fired for incompetence

            It was my call, I accept all responsibility. The board of directors has read your comment and they can ask for my resignation if they like.

          • Chainsman

            Thanks for your thoughtful response. I wish you well.

        • Crispin Munns

          You can’t get too angry at BB for your own SPOF

          • Reed Zhao

            You have a point about SPOF. If I had to pay S3 as a redundancy to B2, there is no point of using B2 at all. Just like I don’t have two phone numbers from different carriers in case one would fail, I choose the carriers with better reliability.

          • The Great Gak

            Brand A may be more reliable than Brand B, but, as has been borne out repeatedly in recent years, neither is immune to failures. The only way to increase the durability of your storage is to treat multiple (and by that I mean a lot more than two) cloud providers the same way each one treats its internal parts. It costs more, but what’s your data worth to you?

          • Reed Zhao

            You all have a point. Our expectation is that as long as B2 is not *significantly* worse than other providers, we are okay.

            ——– From Backblaze B2 SLA ———-
            Backblaze will make commercially reasonable efforts to ensure that B2 Cloud Storage is available and able to successfully process requests during at minimum 99.9% of each calendar month.
            ——————— https://www.backblaze.com/company/sla.html

            Error 1. Treat B2 API users the same way as home backup users.
            We don’t have the same tolerance level to downtime. A home user can wait a few hours to backup his laptop, but our back-end service can not be stopped.

            Error 2. No email blast out ahead of time.
            I only found out the maintenance notice on twitter after the service is down.

            Both errors put BackBlaze leagues behind its competition.

          • Reed Zhao

            @thegreatgak:disqus Downtime and data loss are two different things. I have a backup copy, but that can’t makeup for downtime.

          • The Great Gak

            “I have a backup copy, but that can’t makeup for downtime.” — What I said applies equally to uptime. If failure of a single provider is intolerable, the data has to be spread among several in a way that it can be reconstructed on-the-fly if one fails, just as would happen in a RAID. This problem will not be solved with a single provider as things stand today.

    • Joseph

      Agreed. Two times this has happened too. Do the maintenance in the middle of the night!

  • Joshua

    This is completely unacceptable. Not only are you down, but you are still down after maintenance window. Apparently you get what your pay for. Should have stayed with Crashplan; twice the price but they never went down in all the years I had them.

    • Sorry for the extended outage. As with a lot of networking and infrastructure-related things, we try to move quickly but we have to be careful. We apologize for any inconvenience.

  • Saffron Walden

    This is NOT a good business idea. Maintain your servers at 3AM Sunday, not Thursday afternoon when CUSTOMERS need your services!

    • We apologize for the maintenance work at an inopportune time for you. Most of our maintenance is transparent, this one required more hands on deck, so had to occur when most of our folks were available. We’re sorry for any inconvenience.

      • DH4

        1. It’s called overtime. Should already be calculated into the costs of running such a service.

        2. You obviously have some design flaws in your system if a maintenance takes your service down.

        • Joshua

          If Amazon went down for even 5 minutes, they would lose millions of dollars a minute in sales.

        • Thank you for your feedback, I’ll make sure it gets to the right people.

      • The Great Gak

        Pro tip from someone who’s been up in the wee hours of the morning for this stuff more times than he can count: You do maintenance at the times that are least inconvenient for your customers, not when it’s most convenient for your employees.

        I signed up yesterday to evaluate the service, dumped in a few gigabytes of data this afternoon just before all of this started and got taken by surprise by it. Not good.

        • Thank you for your feedback, I’ll make sure it gets to the right folks.

      • Juan

        Newsflash: make your techs come in during off-peak hours. We all work in tech and know how this works. Your customers aren’t soccer moms who are clueless about maintenance. If this would have been an unplanned outage, then fine. But a PLANNED outage because your techs cant work off-peak hours. I’m out.

        • We’re trying to get systems back online as quickly as possible. We’re sorry to see you go Juan.

  • Juan

    Maintenance during business hours?!? B2 sounded great but with the problems I’ve been having over the past week and now this, maybe its time I look elsewhere!

    • icodeforlove

      Yeah i don’t get it… They really need to figure out how to properly deploy changes.

      Maintenance windows are so early 2000’s. Especially when they are offering API’s.

    • Some B2 services should remain online during maintenance. If you’re having issues, please ping B2 support -> https://help.backblaze.com/hc/en-us/requests/new.

  • DH4

    Why would you do this maintenance during peak hours? And why are your systems not designed with high availability in mind?

    For a service like B2 you should be able to roll out updates without kicking customers offline. I guess you get what you pay for.

    • Most B2 services remain online during our maintenance periods. If you’re having issues, please ping support -> https://help.backblaze.com/hc/en-us/requests/new.

      • DH4

        I have uptime monitors in-place and can verify from multiple locations B2 has been down since the beginning of the maintenance period.

        • DH4

          I think we will need to migrate away from B2, we just can’t have the service going down during peak business hours than this. Not only that, but it was on purpose and only a 2 day warning in a blog post.

          There should have been at least an email sent out with more than a 2 day notice. I guess we’ll just have to pay the extra cost for a more reliable cloud storage provider.

          I knew about this outage a couple hours before it happened and only because Google Now happened to put it at the top of my feed today.

        • This is same for our services as well. All uploading and downloading is not functioning at the moment. I have lodged a ticket.

      • Reed Zhao

        B2 is completely down for us. No image on our website! We have 2 million+ images on B2. This is very very bad.

        Try this https://f001.backblazeb2.com/file/pix-treb/E3915406/01.jpg