{"id":112085,"date":"2025-05-06T09:37:44","date_gmt":"2025-05-06T16:37:44","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=112085"},"modified":"2025-05-06T09:40:21","modified_gmt":"2025-05-06T16:40:21","slug":"one-simple-change-that-made-our-exabyte-scale-storage-faster","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/","title":{"rendered":"One Simple Change That Made Our Exabyte-Scale Storage Faster"},"content":{"rendered":"\n
\"A<\/figure>\n\n\n\n
<\/div>\n\n\n\n

When you’re moving exabytes of data, every network request, every CPU cycle, every byte matters. Recently, I had the chance to revisit a part of our system that’s been quietly humming along for years. With one small rethink, we helped give our download performance a serious boost.<\/p>\n\n\n\n

The idea was almost laughably simple: combine two separate requests into one. But when you’re operating at massive scale, even a “simple” change can make a huge difference.<\/p>\n\n\n\n

\n

Curious how we think about performance at scale?<\/h4>\n

From our new series on engineering innovations, check out Analyzing Performance at Exabyte Scale<\/a> and What Powers the Performance of Backblaze<\/a> for a deeper dive into the engineering principles that drive our storage platform.\n<\/p><\/div>\n\n\n\n

The challenge: Why we had 40 requests per download<\/strong><\/h2>\n\n\n\n

Before the change, downloading a file meant:<\/p>\n\n\n\n

    \n
  • A “download coordinator” pod would reach out across the 20 pods that make up a Vault to grab metadata.<\/li>\n\n\n\n
  • Once it had those, it would figure out where the needed bytes lived.<\/li>\n\n\n\n
  • Then it would go back and request the actual data.<\/li>\n<\/ul>\n\n\n\n

    That meant 40 separate requests<\/strong> just to get the ball rolling on every download.<\/p>\n\n\n\n

    The fix: Smarter reads with half the overhead<\/strong><\/h2>\n\n\n\n

    At some point, it clicked for me: why were we doing this in two steps? The original setup only pulled the bare minimum of data. But what if we just grabbed everything we needed at once? There wasn\u2019t a good reason not to. So I refactored the process so that a pod could grab both the shard header and the data in a single request.<\/p>\n\n\n\n

    Now:<\/p>\n\n\n\n

      \n
    • The coordinator still orchestrates the work.<\/li>\n\n\n\n
    • The receiving pod reads the header, figures out what it needs, and pulls the data\u2014all internally. By shifting this responsibility to the receiving pod, we eliminate a network round trip per pod\u201420 round trips in total. <\/li>\n\n\n\n
    • The combined result is sent back to the coordinator in a single step.<\/li>\n<\/ul>\n\n\n\n

      After the fix, we\u2019re still reading the same amount of data from disk, so disk I\/O remains unchanged, but network performance improved significantly. Instead of kicking off 40 network operations, we’re down to about half that. Less traffic, less overhead, faster performance.<\/p>\n\n\n\n

      It was a simple fix, but the project required a significant amount of software engineering work as well. By shifting responsibilities to the “receiving pod” the coordinator needed to learn to perform lots of just-in-time reasoning about the nature of the download, which required rethinking how we architected portions of the download code.<\/p>\n\n\n\n

      Why it didn’t just instantly double download performance<\/strong><\/h2>\n\n\n\n

      If you’re thinking, “shouldn’t that make downloads twice as fast?”\u2014not quite.<\/p>\n\n\n\n

      Here’s why: Big files get broken into “stripes” during download, and my change only optimizes the first stripe request. Smaller files (a big chunk of our traffic) see the full benefit because they often fit into a single stripe. For larger files, though, the improvement only affects a small part of the overall download, so the impact is more limited.<\/p>\n\n\n\n

      How we measured the impact<\/strong><\/h2>\n\n\n\n

      Measuring the real-world effect turned out to be trickier than I expected. Our download traffic isn’t steady; it’s spiky. Under normal conditions, our system wasn’t hitting capacity limits which made it hard to clearly see changes in download performance.\u00a0<\/p>\n\n\n\n

      But in our dedicated performance testing environment, where we could send a controlled load of downloads, the improvement was crystal clear. With this change, our system could handle a much higher peak load\u2014great news for handling things like backup surges, AI training runs, and large enterprise downloads. <\/p>\n\n\n\n

      Beyond download performance: System-wide benefits<\/strong><\/h3>\n\n\n\n

      One of the coolest side effects? This doesn’t just help customer downloads. It also speeds up internal operations like vault recomputing data drives and server-side copies.<\/p>\n\n\n\n

      By freeing up CPU cycles that used to be wasted on multiple requests, we open the door for better performance everywhere. And hey, maybe even some minor energy savings\u2014less CPU load means less heat, less power.<\/p>\n\n\n\n

      What this taught me about optimization<\/strong><\/h2>\n\n\n\n

      When you’re trying to optimize a massive system, it’s tempting to chase performance with complicated solutions: more threads, smarter caches, fancier hardware.<\/p>\n\n\n\n

      But sometimes, the real win is just about thinking differently. Questioning assumptions. Asking, “Wait, why are we doing it this way?”<\/p>\n\n\n\n

      For me, this project was a great reminder that even at exabyte scale, the simplest solution can be the most impactful.<\/p>\n","protected":false},"excerpt":{"rendered":"

      When you’re operating at exabyte scale, even one extra network request adds up. A simple tweak\u2014combining two steps into one\u2014cut download overhead in half and boosted performance. When you question assumptions, you can find big wins in small, smart changes.<\/p>\n","protected":false},"author":216,"featured_media":112019,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[7,434,438,483],"tags":[468],"class_list":["post-112085","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-storage","category-featured-1","category-featured-cloud-storage","category-tech-lab","tag-b2cloud","entry"],"acf":[],"yoast_head":"\nHow We Improved Download Performance With A Simple Change<\/title>\n<meta name=\"description\" content=\"Learn how a simple change halved network overhead and boosted download performance across our exabyte-scale system.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How We Improved Download Performance With A Simple Change\" \/>\n<meta property=\"og:description\" content=\"Learn how a simple change halved network overhead and boosted download performance across our exabyte-scale system.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/\" \/>\n<meta property=\"og:site_name\" content=\"Backblaze Blog | Cloud Storage & Cloud Backup\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/backblaze\" \/>\n<meta property=\"article:published_time\" content=\"2025-05-06T16:37:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-06T16:40:21+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/03\/bb-bh-Five-Surprising-Factors-That-Affect-Cloud-Performance.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1440\" \/>\n\t<meta property=\"og:image:height\" content=\"820\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Jerry Sha\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@backblaze\" \/>\n<meta name=\"twitter:site\" content=\"@backblaze\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jerry Sha\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How We Improved Download Performance With A Simple Change","description":"Learn how a simple change halved network overhead and boosted download performance across our exabyte-scale system.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/","og_locale":"en_US","og_type":"article","og_title":"How We Improved Download Performance With A Simple Change","og_description":"Learn how a simple change halved network overhead and boosted download performance across our exabyte-scale system.","og_url":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/","og_site_name":"Backblaze Blog | Cloud Storage & Cloud Backup","article_publisher":"https:\/\/www.facebook.com\/backblaze","article_published_time":"2025-05-06T16:37:44+00:00","article_modified_time":"2025-05-06T16:40:21+00:00","og_image":[{"width":1440,"height":820,"url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/03\/bb-bh-Five-Surprising-Factors-That-Affect-Cloud-Performance.png","type":"image\/png"}],"author":"Jerry Sha","twitter_card":"summary_large_image","twitter_creator":"@backblaze","twitter_site":"@backblaze","twitter_misc":{"Written by":"Jerry Sha","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/#article","isPartOf":{"@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/"},"author":{"name":"Jerry Sha","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/5764886eec9c6eb96046e0133cc8ad91"},"headline":"One Simple Change That Made Our Exabyte-Scale Storage Faster","datePublished":"2025-05-06T16:37:44+00:00","dateModified":"2025-05-06T16:40:21+00:00","mainEntityOfPage":{"@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/"},"wordCount":748,"commentCount":0,"publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/03\/bb-bh-Five-Surprising-Factors-That-Affect-Cloud-Performance.png","keywords":["B2Cloud"],"articleSection":["Cloud Storage","Featured","Featured-Cloud Storage","Tech Lab"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/","url":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/","name":"How We Improved Download Performance With A Simple Change","isPartOf":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/#primaryimage"},"image":{"@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/03\/bb-bh-Five-Surprising-Factors-That-Affect-Cloud-Performance.png","datePublished":"2025-05-06T16:37:44+00:00","dateModified":"2025-05-06T16:40:21+00:00","description":"Learn how a simple change halved network overhead and boosted download performance across our exabyte-scale system.","breadcrumb":{"@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/#primaryimage","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/03\/bb-bh-Five-Surprising-Factors-That-Affect-Cloud-Performance.png","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/03\/bb-bh-Five-Surprising-Factors-That-Affect-Cloud-Performance.png","width":1440,"height":820,"caption":"A decorative image showing various towers and a cloud."},{"@type":"BreadcrumbList","@id":"https:\/\/www.backblaze.com\/blog\/one-simple-change-that-made-our-exabyte-scale-storage-faster\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/"},{"@type":"ListItem","position":2,"name":"One Simple Change That Made Our Exabyte-Scale Storage Faster"}]},{"@type":"WebSite","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","name":"Backblaze Cloud Solutions Blog","description":"Cloud Storage & Cloud Backup","publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization","name":"Backblaze","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Backblaze"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/backblaze","https:\/\/x.com\/backblaze","https:\/\/www.youtube.com\/user\/Backblaze","https:\/\/en.wikipedia.org\/wiki\/Backblaze"]},{"@type":"Person","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/5764886eec9c6eb96046e0133cc8ad91","name":"Jerry Sha","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/05\/profile-150x150.jpg","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/05\/profile-150x150.jpg","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/05\/profile-150x150.jpg","caption":"Jerry Sha"},"description":"As a senior software engineer on the Storage Engineering team at Backblaze, Jerry brings 15 years of experience building reliable and scalable systems. When he\u2019s not designing infrastructure that keeps petabytes of data safe, he\u2019s either scaling a rock wall or tinkering with the latest tech just for the fun of it.","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/author\/jerry-sha\/"}]}},"jetpack_featured_media_url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/03\/bb-bh-Five-Surprising-Factors-That-Affect-Cloud-Performance.png","_links":{"self":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/112085","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/users\/216"}],"replies":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/comments?post=112085"}],"version-history":[{"count":0,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/112085\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media\/112019"}],"wp:attachment":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media?parent=112085"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/categories?post=112085"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/tags?post=112085"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}