{"id":103776,"date":"2021-12-20T13:10:04","date_gmt":"2021-12-20T21:10:04","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=103776"},"modified":"2025-12-11T13:38:54","modified_gmt":"2025-12-11T21:38:54","slug":"our-response-to-the-log4j-vulnerability","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/","title":{"rendered":"Our Response to the Log4j Vulnerability"},"content":{"rendered":"

\"\"<\/p>\n

When the director of the Cybersecurity and Infrastructure Agency calls a vulnerability \u201cone of the most serious I\u2019ve seen in my entire career, if not the most serious,\u201d<\/a> ears perk up.<\/p>\n

The director was referring to the Apache Log4j vulnerability that was discovered this month. Some more colorful phrases used to describe the Log4j incident include: \u201ca grave threat,\u201d<\/a> \u201ca design failure of catastrophic proportions,\u201d<\/a> something that will \u201chaunt the internet for years.\u201d<\/a><\/p>\n

The vulnerability proceeded to set off five-alarm fires in IT, security, and operations departments around the world. Or should have, at least. Researchers estimate at least 840,000 attacks<\/a> have since been launched via the vulnerability since it was discovered. That is to say, if you\u2019re using a software or cloud vendor that hasn\u2019t<\/em> made some kind of statement or taken corrective action, you should be asking them why not.<\/p>\n

At Backblaze, we made the decision to take our servers temporarily offline in order to hunt down potential threats, apply the appropriate security patches, and test those patches to help prevent our systems from being compromised. This post explains why we made the decision, outlines the actions we took to meet our objective of securing customer data as well as our environment, and provides more insight into our process.<\/p>\n

\n

What Is the Log4j Vulnerability?<\/strong><\/p>\n

As reported by ArsTechnica<\/a>, a zero-day vulnerability was discovered in the Apache Log4j logging library that enables attackers to take control of vulnerable servers. Though it may not be an immediately recognizable name, Log4j is widely used throughout the world by companies like Apple, Twitter, and Tesla as well as the game Minecraft. The library allows developers to easily log application events. The Cybersecurity & Infrastructure Security Agency<\/a> (CISA) urged users to apply patches immediately to address the vulnerabilities.<\/p>\n<\/div>\n

Our Decision<\/strong><\/h2>\n

Upon learning of the Log4j vulnerability, our team took swift action to investigate and assess available options to address the potential impacts since Log4j is leveraged widely in our environment. As part of our investigation, our internal team used a nondestructive form of the exploit to confirm our vulnerability. We also noted close to 80,000 unsuccessful Log4j exploit attempts on our sites in a 12-hour period. The level of activity, along with our success using the exploit (albeit with internal knowledge of our own systems), was very concerning to us.<\/p>\n

Although we were not aware of any unauthorized access to our systems due to the Log4j vulnerability, out of an abundance of caution, we decided it was in our customers\u2019 best interest to take systems offline until they could be patched. The decision to take our systems offline was not one we took lightly. However, our Incident Management Guidelines are quite clear. In a crisis where tradeoffs must be made, our descending list of priorities (all of which are very important to us) is as follows:<\/p>\n

    \n
  1. Health & Safety.<\/li>\n
  2. Data Integrity & Confidentiality.<\/li>\n
  3. Service Availability.<\/li>\n
  4. Service Performance.<\/li>\n<\/ol>\n

    Protecting customer data integrity is second only to health and safety and above service availability. That said, the decision to temporarily bring all services down was unprecedented in the 14-year history of Backblaze. This was an extraordinary case where we made a decision to take a necessary action to address an imminent risk of a vulnerability with a Common Vulnerability Scoring System (CVSS) score of 10.0<\/a>\u2014the highest possible score. We believe that we needed to take preventative steps to protect customer data by temporarily taking our services offline until the security patching process was complete.<\/p>\n

    What Actions Have We Taken?<\/strong><\/h2>\n

    A recap of recent actions is outlined below:<\/p>\n

      \n
    • Upon learning of the Log4j vulnerability, our Security team took immediate action to investigate.<\/li>\n
    • Based on our assessment of the potential threat, we decided to temporarily take our services offline to apply a security patch to prevent our systems from potentially being compromised.<\/li>\n
    • We announced our systems had been taken offline at 5:20 p.m. PT on December 10, 2021.*<\/li>\n
    • We announced our systems were back online and functioning normally at 3:01 a.m. PT on December 11, 2021.<\/li>\n
    • Based on our investigation, we also determined that there was no evidence of our systems being compromised or unauthorized access to customer data or files due to the Log4j vulnerability.<\/li>\n<\/ul>\n

      *We decided not to announce downtime publicly until after our systems were offline to avoid any elevation of priority to those targeting our services. Accordingly, we did not make a public announcement until after the servers were disconnected.<\/p>\n

      Was Backblaze Compromised?<\/strong><\/h2>\n

      We have not found any evidence of system compromise or unauthorized access to customer data or files at this time.<\/p>\n

      Next Steps<\/strong><\/h2>\n

      As is part of our incident response process, we always look for ways to do better and identify areas for improvement. In this case, two top priorities moving forward would be to improve how we can apply security patches faster and reduce downtime.<\/p>\n

      Thank you to our customers for your understanding as we navigated this challenging incident.<\/p>\n","protected":false},"excerpt":{"rendered":"

      Learn more about the recent Apache Log4j vulnerability and the actions we took in response.<\/p>\n","protected":false},"author":170,"featured_media":103779,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":"","jetpack_post_was_ever_published":false},"categories":[131],"tags":[469],"class_list":["post-103776","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-backblaze-bits","tag-consumerbackup","entry"],"acf":[],"yoast_head":"\nOur Response to the Log4j Vulnerability and Our Risk Assessment<\/title>\n<meta name=\"description\" content=\"The Apache Log4j vulnerability was an important discovery. Here\u2019s how we\u2019ve been handling this vulnerability and keeping our customers safe through it.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Our Response to the Log4j Vulnerability and Our Risk Assessment\" \/>\n<meta property=\"og:description\" content=\"The Apache Log4j vulnerability was an important discovery. Here\u2019s how we\u2019ve been handling this vulnerability and keeping our customers safe through it.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/\" \/>\n<meta property=\"og:site_name\" content=\"Backblaze Blog | Cloud Storage & Cloud Backup\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/backblaze\" \/>\n<meta property=\"article:published_time\" content=\"2021-12-20T21:10:04+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-11T21:38:54+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/bb-bh-Log4j-Response-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1440\" \/>\n\t<meta property=\"og:image:height\" content=\"821\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Mark Potter\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@backblaze\" \/>\n<meta name=\"twitter:site\" content=\"@backblaze\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mark Potter\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Our Response to the Log4j Vulnerability and Our Risk Assessment","description":"The Apache Log4j vulnerability was an important discovery. Here\u2019s how we\u2019ve been handling this vulnerability and keeping our customers safe through it.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/","og_locale":"en_US","og_type":"article","og_title":"Our Response to the Log4j Vulnerability and Our Risk Assessment","og_description":"The Apache Log4j vulnerability was an important discovery. Here\u2019s how we\u2019ve been handling this vulnerability and keeping our customers safe through it.","og_url":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/","og_site_name":"Backblaze Blog | Cloud Storage & Cloud Backup","article_publisher":"https:\/\/www.facebook.com\/backblaze","article_published_time":"2021-12-20T21:10:04+00:00","article_modified_time":"2025-12-11T21:38:54+00:00","og_image":[{"width":1440,"height":821,"url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/bb-bh-Log4j-Response-1.png","type":"image\/png"}],"author":"Mark Potter","twitter_card":"summary_large_image","twitter_creator":"@backblaze","twitter_site":"@backblaze","twitter_misc":{"Written by":"Mark Potter","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/#article","isPartOf":{"@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/"},"author":{"name":"Mark Potter","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/fa0c9c069ba1626a7d08daa9eea36005"},"headline":"Our Response to the Log4j Vulnerability","datePublished":"2021-12-20T21:10:04+00:00","dateModified":"2025-12-11T21:38:54+00:00","mainEntityOfPage":{"@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/"},"wordCount":834,"commentCount":1,"publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/bb-bh-Log4j-Response-1.png","keywords":["ConsumerBackup"],"articleSection":["Backblaze Bits"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/","url":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/","name":"Our Response to the Log4j Vulnerability and Our Risk Assessment","isPartOf":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/#primaryimage"},"image":{"@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/bb-bh-Log4j-Response-1.png","datePublished":"2021-12-20T21:10:04+00:00","dateModified":"2025-12-11T21:38:54+00:00","description":"The Apache Log4j vulnerability was an important discovery. Here\u2019s how we\u2019ve been handling this vulnerability and keeping our customers safe through it.","breadcrumb":{"@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/#primaryimage","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/bb-bh-Log4j-Response-1.png","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/bb-bh-Log4j-Response-1.png","width":1440,"height":821},{"@type":"BreadcrumbList","@id":"https:\/\/www.backblaze.com\/blog\/our-response-to-the-log4j-vulnerability\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Our Response to the Log4j Vulnerability"}]},{"@type":"WebSite","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","name":"Backblaze Cloud Solutions Blog","description":"Cloud Storage & Cloud Backup","publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization","name":"Backblaze","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Backblaze"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/backblaze","https:\/\/x.com\/backblaze","https:\/\/www.youtube.com\/user\/Backblaze","https:\/\/en.wikipedia.org\/wiki\/Backblaze"]},{"@type":"Person","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/fa0c9c069ba1626a7d08daa9eea36005","name":"Mark Potter","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/Mark-Potter-5x7-closeup-copy-150x150.jpg","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/Mark-Potter-5x7-closeup-copy-150x150.jpg","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/Mark-Potter-5x7-closeup-copy-150x150.jpg","caption":"Mark Potter"},"description":"Mark Potter is Backblaze's chief information security officer. He brings experience from over 29 years working in information security governance, risk management, regulatory compliance, and data protection and privacy program design and implementation to Backblaze. He is an IAPP Fellow of Information Privacy and holds over 30 security, privacy, and risk management certifications.","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/author\/mark\/"}]}},"jetpack_featured_media_url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2021\/12\/bb-bh-Log4j-Response-1.png","_links":{"self":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/103776","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/users\/170"}],"replies":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/comments?post=103776"}],"version-history":[{"count":0,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/103776\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media\/103779"}],"wp:attachment":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media?parent=103776"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/categories?post=103776"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/tags?post=103776"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}