What Happened: Adobe Creative Cloud Update Bug

February 14th, 2016

Adobe Creative Cloud Update
On February 11th, Adobe released an update to Creative Cloud (v 3.5.0.206) which, as they put it, “…in some scenarios the application may incorrectly remove files with user writeable permissions from the system root directory.” You can find their release notes here. This meant that if you were an Adobe Creative Cloud customer on Mac with auto-update turned on (or happened to download that version), as soon as you signed in to Creative Cloud, files from folders within your root directory could have been removed. We typically saw this occur to the user’s top-most (when alphabetically sorted) hidden folder, or to a folder that had a space at the front of its name (a semi-common practice to force a folder to always appear at the top of the list).

That update of Adobe Creative Cloud has since been pulled, and version 3.5.1.209 is available and is supposed to offer a fix.

What To Do:
If you were not running Adobe Creative Cloud on the Mac or did not update to Adobe Creative Cloud v3.5.0.206, you have nothing to worry about. If you did, and you received the Backblaze bzvol pop-up alert, you’re in luck because the file that was deleted has been re-created by Backblaze. Backblaze backups were never in jeopardy during this event.

If you were not running Backblaze or did not receive this alert, Adobe Creative Cloud may have deleted a file or folder on your computer. It’s difficult to tell exactly which data may have been removed, but you can open the root directory on your Mac and try to look towards the top for any folders that are empty. This would have occurred to only one folder (that we know of) so the top-most hidden folder or the first folder with a space as the first character would have been affected. If you do see empty folder and have a local backup, or another backup system, you may be able to restore that data from them. If not, you may want to contact Adobe for any help on this issue.

Why Was Backblaze Involved:
Backblaze keeps a hidden folder called .bzvol at the top of each drive we see connected to your system. It helps us keep track of your hard drives and keep them backed up. You can read our CTO Brian’s description of it and why we placed it there in this reddit thread. In a lot of cases, when an Adobe Creative Cloud customer hit the above bug, the first hidden folder on their main hard drive’s root folder, was indeed that .bzvol. Fortunately for us and why we were able to identify this issue so quickly, is that we have a pop-up that fires whenever we detect the file .bzvol missing.
bzvol
Normally, we get reports of this error very rarely. Suddenly we started seeing a rapid increase in support tickets and tweets related to the .bzvol pop-up. We quickly opened an internal investigation into the issue.

What Backblaze Found:
February 10th –
Wednesday night we started getting support tickets relating to the .bzvol file being removed from computers. Normally the pop-up sends people to this (which we have since edited to highlight this current issue): bzvol webpage. The problem was, the folks on Mac kept reporting that our fix did not work and that they kept getting the error. Our support team contacted our lead Mac developer for help trying to troubleshoot and figure out what was causing this surge.

February 11th –
We continued to get a lot of tweets like Rob’s


They hit the .bzvol error repeatedly and our standard fix wasn’t helping. At this point support was receiving hundreds of tickets and chat requests (much higher than our normal ticket totals) and we were seeing a lot of tweets coming in as well. The weird part was, none of our machines were affected, which meant it wasn’t a Mac issue across the board.

At about 3PM we created an FAQ with a few instructions on how to solve our original .bzvol issues. This would temporarily fix it for some users, but many kept having this issue appear after they rebooted their machines.

On Thursday, about mid-day we caught a break. Our head designer Casey got the .bzvol error message. He told the support team, who in turn told our lead Mac developer that we had a live use-case in the office. After a couple of hours our lead Mac developer posted the following message in our support team’s Slack channel:
Damon
screen_shot_2016-02-11_at_3.26.54_pm
And we were off to the races. We circled back with a lot of the twitter folks (like Rob and Tony) asking if they used Adobe’s Creative Cloud, and they all responded in the affirmative. Support also started asking in chats and we quickly found that yes, all of these were Adobe users.

We contacted Adobe’s support and opened a case; actually we opened two separate cases. We also continued working on a work-around on our end while trying to find what was actually occurring and how we could stop it. We also opened this thread on Adobe’s forums.

This is also when we started tweeting about the bug and trying to get people’s attention and explaining that their backups may be affected. At this point we did not know that the Adobe Creative Cloud bug would also affect non-Backblaze customers. Below are a series of tweets:


More testing revealed that it wasn’t just the .bzvol folder that could have it’s contents deleted by Adobe, but any folder in the Mac root directory that was alphabetically “higher” on the list:

At this point we updated our help article: “bzvol FAQ” with a workaround. We created a “sacrificial folder” in the root directory which the bug would eat every time someone signed in to their Creative Cloud (even if they were still using the buggy version). We did not yet know that it could have also potentially affected customers with a space in front of their folder names (which is more common than hidden folders). We also created videos showing this happening:

Side-Note: our lead Mac developer, support team, and I were all working from different locations. The lead Mac developer was working from home, the support team was in the office (some of them were also working remotely) and I was in San Diego at a conference. We used Slack to coordinate and it worked pretty well for a real-time event like this.

February 12th –
By this point many folks in the office were trying to help debug this problem, including our CTO. We kept testing and realized that this was occurring upon “sign-in” to Creative Cloud and that folders with spaces at the front of them on the root directory were also susceptible:

At about 10AM we were contacted by Ars Technica, and I explained what was going on, what we had found, and who it was affecting. Since the situation was fluid we exchanged a lot of emails and a quick call to keep Dan Goodin abreast of what we were finding.

At 12PM I received emails from Adobe’s PR who wanted to make sure we were on the same page with how we were addressing the issue. They told me at this time that they were pulling that update and working on a fix for it.

At that time we published our blog post and sent an email to all our customers explaining that if they used Adobe Creative Cloud they might have been subject to data loss. We also let them know that Adobe had pulled the update.

Adobe also posted to their blog (Adobe blog) explaining that it was an issue affecting some folks.

As a backup company we take data loss seriously, and when we tried to ask who exactly may have been susceptible, they did not have much information, however we assume that anyone on a Mac that updated or was auto-updated to the affected version of Creative Cloud, and signed in, may have lost data.

We continued to field questions throughout the day and updated our help article with more concrete information as it became available.

February 13th –
There was still some confusion out in the field and in some of the articles whether or not this was strictly an issue for Backblaze users. So we ran the experiment of signing in to Adobe’s Creative Cloud without having Backblaze installed on the computer, and determined the issue could affect anyone, not just Backblaze users:

We updated our help articles to reflect that anyone on Mac may have been affected and updated our communications to reflect that.


We urged potentially affected folks to check the folders towards the top their root directory for empty folders.

By mid-afternoon Adobe had started testing their fix and began rolling it out that night.

February 14th –
Adobe updated their blog with a link to their fix (version 3.5.1.209) and contacted us letting us know that folks should update. We’ve disseminated that information.

TL/DR Recap:

  • Adobe had a buggy version of Creative Cloud (3.5.0.206) that came out on Wednesday which deleted some data inside root folders for Mac users.
  • Backblaze acted as an early alert canary because we trigger a pop-up when our files are deleted from customer machines in order preserve backup integrity.
  • Backblaze spent a lot of time debugging the issue and told Adobe and the public about it once we realized what was going on and who it was affecting.
  • Adobe fixed the issue on Sunday with version (3.5.1.209). All users of Adobe Creative Cloud can now update their software.
  • Backblaze customers who were affected can follow the instructions here: bzvol FAQ and restore their files from their Backblaze backup.
  • Non-Backblaze customers who may have been affected can check other backup methods for any data that may have been removed. If you have any Adobe-related questions, please contact Adobe Support.

Many Backblaze personnel, especially support, engineering and marketing all spent uncounted hours tracking down, debugging, creating a workaround and keeping our customers up to date on the issue. It has been a long few days.

Happy Valentine’s Day.

Yev

Yev

Chief Smiles Officer at Backblaze
Yev enjoys speed-walking on the beach. Speed-dating. Speed-writing blog posts. The film Speed. Speedy technology. Speedy Gonzales. And Speedos. But mostly technology. He also runs social for Backblaze.

Follow Yev on:
Twitter: @YevP | LinkedIn: Yev Pusin | Google+: Yev Pusin
Yev

Latest posts by Yev (see all)

  • louis925

    Never use Adobe stuff again….

  • Ken Simpson

    This certifies that Adobe can’t write reliable software. They ought to completely outsource the revolting Adobe Application Manager to someone who can do it right. I mean, deleting the contents of the first folder on the drive sorted alphabetically as some kind of strategy? What?

  • Nan

    Has anyone had any luck recovering removed files? This hit me hard and I lost a ton of important working files.

  • Mike McKeller

    Yev – thanks for your excellent work on this. Am I understanding correctly that the buggy update only deleted files from hidden folders? The top unhidden folder in my root directory is “Applications”, and so far I haven’t noticed anything missing in there. Thanks.

    • Mike, thanks! From what we could gather it would have been either the top-most hidden folder, or the top folder with a space in the name. I’m afraid that’s all we were able to find on our end. For a more concrete explanation of what actually happened you’d have to ask Adobe, we may have missed something on our end, and you’d want to be doubly sure that you’re not missing anything!

  • ikon819

    This will probably be one of the most unusual posts. I’ve been reading BackBlaze blogs for several years and find them interesting and informative. However, I don’t use BackBlaze myself, because I have a very robust backup system that I’ve had in place since before BackBlaze existed. I have 5 copies of everything, and I do mean everything, with at least 1 or 2 copies off site at all times. Nevertheless, I wanted to post so that I can express how impressed I am with how BackBlaze conducts itself. This kind of ethical and open conduct needs to be encouraged, even by non-users. I tip my hat to everyone at BackBlaze — well done!

    • Thanks @disqus_DDjVIvNXGh:disqus! We love hearing that and we’re glad that you have a great backup strategy in place! You’re doing far better than the average computer user, kudos!

  • Jesse Kaufman

    Coming from someone in the IT as well as design industry, this kind of transparency, THOROUGH reporting, and dedication to customers is rare! i always recommend Backblaze when people ask about off-site backup, and now I’ll push it even more!

  • Diirge

    I was affected and didn’t even know it. I happened to see the Tweet about it and immediately checked Backblaze and then rand the terminal command. It had deleted it and I was very happy to have seen that tweet.

    • Glad to hear it! We tried to be really quick on social media about it once we realized it wasn’t a “normal” issue. We waited to fire an email to everyone until we had more information though, can’t be emailing all the users willy-nilly!

      • Jesse Kaufman

        Could not have done a better job, in my opinion! :)

  • iestynx

    Thanks for all your hard work. I find it astounding that Adobe has still to issue an apology.

  • 3901

    I’m not on Twitter or any blogs so I didn’t received all this communication. All I could see was Backblaze was not backing up my boot drive. The only “fix” I received was the pop-up from Backblaze that didn’t work and the .bzvol directory kept disappearing. Backblaze was the only program affected on my computer so there was no reason to think it was someone other than Backblaze causing the problem and I lost confidence in your program. Because of this I switched back to CrashPlan and I bet I’m not the only one to leave Backblaze. I appreciate the long hours you spent trying to solve this problem, but If you would have had better communication other than the useless pop-up, I probably would have stayed with Backblaze.

    • Marvin (FatherStorm) Francois

      I’m not sure how they could have better accounted for some other company’s software. Not only was the popup not useless, had it not fired, a lot of users would have walked in to work tuesday, turned on their macs, opened adobe CC and potentially nuked rather more important files. Blaming them for not communicating better is like blaming Taylor Swift because her song was playing in your car when the brakes failed.

    • Yozman

      I assume you don’t have any email address either?

      I don’t have any macs being backed up by backblaze (in fact I don’t have any macs period) and yet I still received two emails regarding this. If you take backup seriously why not read the emails you receive from the service that is storing your data? Aside from email, how did you expect to be contacted? Singing telegram?

      • 3901

        Yeah, I have email and read everything I receive. Unfortunately, the only email I received regarding this issue wasn’t sent until 10:03PM last night (2/14). Since I don’t sit at my computer 24/7, I didn’t see this email until this morning (2/15). By then, it was too late and I already switched back to CrashPlan. Save your smartass replies for someone else.

        • Yozman

          Check again they sent out an email on Friday, I personally have all backblaze emails labeled and marked as important so they float to the top, I’d recommend doing that for your crashplane emails to avoid making foolish remarks in the future.

        • 3_14159

          @3901: I found your post a bit off putting. You noticed that your boot drive was not being backed up, so your first reaction is to lose confidence in the program you’re using and switch to another? What about contacting the company’s Tech Support department and asking if they have any ideas, perhaps work with them to track down the issue? Particularly given BackBlaze’s reputation for service and transparency, it seems to me this would have been a more prudent course of action.

          I say this particularly because, apparently, the only way to reach you is by email, which is hardly a real-time method of communication. Not only does it take time for emails to make their way across the Internet, it takes even more time for a server to send out thousands up thousands of them.

          If you’re not going to participate in any of the instant messaging systems (which is certainly your right & priviledge) perhaps you need to temper your expectations about notifications.

          I agree with you that you are probably not the only person to jump to another program, and that’s unfortunate for BB. Hopefully, upon reflection you will realize that BB did everything that could be expected (including sending you an email) and will consider using their service again.

    • Sorry about that @3901:disqus. Unfortunately for us you weren’t the only one that left. We had an email sent to ALL of our customers and a blog-post out on the 2nd day of this event. I’m sorry you missed that communication. CrashPlan is a good group as well, we’re glad you’re still backing up online! If you ever want to switch back, we’ll be happy to have you :)

  • Ian

    A great job to all your team from start to finish. I am a very happy customer to know you have our backs!! Thanks for keeping at it until a solution was found. I had the same popup window come up dozens of times over the couple of days and couldn’t figure out what was causing it so great to know all sorted now. No more popups :-)

    • Glad to hear you’re all sorted Ian :)

  • Great job on the speedy resolution and pushing Adobe to act fast as well

    • Thanks! We take data loss very seriously!

  • Send Adobe a bill for the extra time you put in and see if they pay it.

    • Hah, we might try and calculate it :P