{"id":112050,"date":"2025-04-23T09:45:08","date_gmt":"2025-04-23T16:45:08","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=112050"},"modified":"2025-04-23T09:45:10","modified_gmt":"2025-04-23T16:45:10","slug":"analyzing-performance-at-exabyte-scale","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/","title":{"rendered":"Analyzing Performance at Exabyte Scale"},"content":{"rendered":"\n
\"A<\/figure>\n\n\n\n
<\/div>\n\n\n\n

When you\u2019re operating a data storage platform at exabyte scale, even small inefficiencies become big problems. With billions of files flowing through our systems, performance isn’t something we think about after the fact\u2014it\u2019s something we constantly chase, measure, and optimize.<\/p>\n\n\n\n

But before you can improve cloud performance, you have to know where to look. When we were working on improving small file uploads<\/a>, I was tasked with taking a closer look at our file upload pipeline to see if we could make it faster.\u00a0<\/p>\n\n\n\n

The path from that general idea to hitting a clear performance goal taught me a lot\u2014not just about our systems, but about how to approach performance work in a principled, strategic way. Here\u2019s how it unfolded, and what you can apply to your own environment.<\/p>\n\n\n\n

Step one: Define the problem<\/h2>\n\n\n\n

The initial ask from our Product team was pretty familiar: \u201cCan we make uploads faster?\u201d It’s a fair question, but not a very actionable one. So we worked with our Product team to define our success criteria. Here are some of the questions we asked to get to specific, actionable goals:<\/p>\n\n\n\n

\u201cWhat do we mean by faster? Do we want to improve latency or throughput?\u201d<\/p>\n\n\n\n

\u201cDo we want to improve all uploads? Just big files? Just small files?
\u201cWhat qualifies as a small upload?\u201d<\/p>\n\n\n\n

After some back and forth, we landed on a clear, measurable target: Process file uploads of 1MB or less via our B2 API in under 40 milliseconds. That specificity made a huge difference:<\/p>\n\n\n\n

    \n
  • With a goal of 40 milliseconds, we had a stopping point<\/strong>. We would know when we\u2019d done enough.<\/li>\n\n\n\n
  • We had a bar to measure against and a way to identify what was worth optimizing<\/strong>. If something took two milliseconds, we could leave it alone. If it took 30, it became the focus.<\/li>\n\n\n\n
  • We could scope effort<\/strong>. There\u2019s a big difference between getting something under 40 milliseconds versus 200. <\/li>\n<\/ul>\n\n\n\n

    Step two: Use the right tools for the job if you possibly can<\/h2>\n\n\n\n

    Analyzing performance without proper tooling means doing a lot of heavy lifting by hand. We had to drop custom instrumentation throughout the stack, create metric-collecting objects, and pass them all the way down the call stack so we could get timing data from different parts of the upload path.<\/p>\n\n\n\n

    The upload flow touches more than 20 storage pods and services, so we also built a lightweight sampling system to keep from flooding our metrics pipeline. The data went into an open-source search and analytics suite, and from there we built dashboards to try to make sense of it all.<\/p>\n\n\n\n

    It was time-consuming. Painfully so. But it worked. <\/p>\n\n\n\n

    I could now compare fast and slow uploads, identify patterns, and\u2014most importantly\u2014see where time was actually being spent. That\u2019s how we discovered that fsync<\/code> was dominating our performance profile, captured in the screenshot below. We measured each sub-operation that comprises our drive write operations, and grouped them by the total time they took to complete. You can see the process fsync<\/code> sub-operation dominates in every group. Removing or optimizing around it offered a 10x speedup. But it took weeks of manual effort to get to that insight.<\/p>\n\n\n\n

    \"\"
    Drive write operations grouped by the time they took to complete.<\/figcaption><\/figure>\n\n\n\n

    Enter: Tracing at scale<\/h3>\n\n\n\n

    Eventually, we brought in more powerful tooling, including an open source distributed tracing system. It was a game changer.<\/p>\n\n\n\n

    What used to take dozens of lines of code and a lot of custom wiring now took a single annotation. More importantly, it gave us something we couldn\u2019t get otherwise: a way to see activity across services, systems, and pods\u2014all in one view.<\/p>\n\n\n\n

    It allowed us to correlate events happening across different physical machines, trace performance end-to-end, and understand the impact of specific changes in real time.<\/p>\n\n\n\n

    At one point, we were debating whether a particular optimization would get us across the finish line. This optimization allows the slowest shard to store asynchronously if and only if all others had been successfully and durably stored. This prevents a single slow shard from slowing down the entire upload. Thanks to the new tooling, we didn\u2019t have to guess\u2014we could see<\/em> that once we flipped the switch, we\u2019d hit our 40ms goal (and it would help all other uploads as well, not just small uploads). That let us focus on getting that one feature ready for production, confident that it would move the needle.<\/p>\n\n\n\n

    \"\"
    Visualization demonstrating one of our upload optimizations, this time for a slower upload. The first 19 shards to complete were stored successfully and durably, so we stop waiting for the last shard, return a 200 status code to the customer (indicated by the dotted line), and allow the 20th shard to finish asynchronously. <\/figcaption><\/figure>\n\n\n\n

    Step three: Optimize with intent<\/h2>\n\n\n\n

    One of the biggest lessons I learned through this process is that you can spend weeks optimizing the wrong thing if you\u2019re not careful. That\u2019s why measurement has to come first.<\/p>\n\n\n\n

    Don\u2019t guess. Instrument. Don\u2019t tweak randomly. Set a baseline and track deltas. Performance work is iterative. You\u2019ll fix one bottleneck, only to reveal the next one lurking beneath it. That\u2019s the job.<\/p>\n\n\n\n

    In hindsight, one of the smartest things we did was setting a clear performance goal at the start of the project. It didn\u2019t just help us focus\u2014it told us when we were done<\/em>. You can optimize forever. Knowing when to stop is just as important as knowing where to begin.<\/p>\n\n\n\n

    Step four: Tool up for the future<\/h2>\n\n\n\n

    The tracing tool has made life a lot easier, but it\u2019s not the only tool we use. Our analytics suite still plays a big role when we want to analyze aggregate data, or need the flexibility to slice and dice data. The two complement each other nicely.<\/p>\n\n\n\n

    There\u2019s no one-size-fits-all solution\u2014it\u2019s more like a toolbox. And like any good toolbox, it keeps growing with our needs.<\/p>\n\n\n\n

    Advice from the trenches<\/h2>\n\n\n\n

    If you\u2019re running distributed systems or chasing performance in your own stack, here\u2019s what I\u2019d suggest:<\/p>\n\n\n\n

      \n
    • Start with a clear goal.<\/strong> Know exactly what \u201cfaster\u201d means, and write it down.<\/li>\n\n\n\n
    • Measure before you optimize.<\/strong> Otherwise, you\u2019re flying blind.<\/li>\n\n\n\n
    • Pick the right tool for the job.<\/strong> Tracing, metrics, logs\u2014they all have their place.<\/li>\n\n\n\n
    • Don\u2019t wait to build your tools.<\/strong> Invest in observability early.<\/li>\n\n\n\n
    • Know when to stop.<\/strong> The ROI of performance work diminishes fast if you\u2019re not careful.
      <\/li>\n<\/ul>\n\n\n\n

      And maybe give your helper methods better names than DoSomeWork<\/code>. Or don\u2019t. It makes the code reviews a little more entertaining.<\/p>\n","protected":false},"excerpt":{"rendered":"

      Before you can improve performance, you have to know where to look.<\/p>\n","protected":false},"author":215,"featured_media":110181,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[7,438,483],"tags":[468],"class_list":["post-112050","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-storage","category-featured-cloud-storage","category-tech-lab","tag-b2cloud","entry"],"acf":[],"yoast_head":"\nHow to Analyze and Improve Cloud Performance at Exabyte Scale<\/title>\n<meta name=\"description\" content=\"Before you can improve cloud performance, you have to know where to look. Get lessons on analyzing cloud performance at exabyte scale.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Analyze and Improve Cloud Performance at Exabyte Scale\" \/>\n<meta property=\"og:description\" content=\"Before you can improve cloud performance, you have to know where to look. Get lessons on analyzing cloud performance at exabyte scale.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/\" \/>\n<meta property=\"og:site_name\" content=\"Backblaze Blog | Cloud Storage & Cloud Backup\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/backblaze\" \/>\n<meta property=\"article:published_time\" content=\"2025-04-23T16:45:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-23T16:45:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2023\/10\/bb-bh-Cloud-Performance.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1440\" \/>\n\t<meta property=\"og:image:height\" content=\"820\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Tim Johnson\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@backblaze\" \/>\n<meta name=\"twitter:site\" content=\"@backblaze\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Tim Johnson\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Analyze and Improve Cloud Performance at Exabyte Scale","description":"Before you can improve cloud performance, you have to know where to look. Get lessons on analyzing cloud performance at exabyte scale.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/","og_locale":"en_US","og_type":"article","og_title":"How to Analyze and Improve Cloud Performance at Exabyte Scale","og_description":"Before you can improve cloud performance, you have to know where to look. Get lessons on analyzing cloud performance at exabyte scale.","og_url":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/","og_site_name":"Backblaze Blog | Cloud Storage & Cloud Backup","article_publisher":"https:\/\/www.facebook.com\/backblaze","article_published_time":"2025-04-23T16:45:08+00:00","article_modified_time":"2025-04-23T16:45:10+00:00","og_image":[{"width":1440,"height":820,"url":"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2023\/10\/bb-bh-Cloud-Performance.png","type":"image\/png"}],"author":"Tim Johnson","twitter_card":"summary_large_image","twitter_creator":"@backblaze","twitter_site":"@backblaze","twitter_misc":{"Written by":"Tim Johnson","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/#article","isPartOf":{"@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/"},"author":{"name":"Tim Johnson","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/89868aff6d31053d82c557c0582122aa"},"headline":"Analyzing Performance at Exabyte Scale","datePublished":"2025-04-23T16:45:08+00:00","dateModified":"2025-04-23T16:45:10+00:00","mainEntityOfPage":{"@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/"},"wordCount":1111,"commentCount":0,"publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/10\/bb-bh-Cloud-Performance.png","keywords":["B2Cloud"],"articleSection":["Cloud Storage","Featured-Cloud Storage","Tech Lab"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/","url":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/","name":"How to Analyze and Improve Cloud Performance at Exabyte Scale","isPartOf":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/#primaryimage"},"image":{"@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/10\/bb-bh-Cloud-Performance.png","datePublished":"2025-04-23T16:45:08+00:00","dateModified":"2025-04-23T16:45:10+00:00","description":"Before you can improve cloud performance, you have to know where to look. Get lessons on analyzing cloud performance at exabyte scale.","breadcrumb":{"@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/#primaryimage","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/10\/bb-bh-Cloud-Performance.png","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/10\/bb-bh-Cloud-Performance.png","width":1440,"height":820,"caption":"A decorative image showing a cloud in the foreground and various mocked up graphs in the background."},{"@type":"BreadcrumbList","@id":"https:\/\/www.backblaze.com\/blog\/analyzing-performance-at-exabyte-scale\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Analyzing Performance at Exabyte Scale"}]},{"@type":"WebSite","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","name":"Backblaze Cloud Solutions Blog","description":"Cloud Storage & Cloud Backup","publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization","name":"Backblaze","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Backblaze"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/backblaze","https:\/\/x.com\/backblaze","https:\/\/www.youtube.com\/user\/Backblaze","https:\/\/en.wikipedia.org\/wiki\/Backblaze"]},{"@type":"Person","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/89868aff6d31053d82c557c0582122aa","name":"Tim Johnson","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/04\/Tim-Johnson-headshot-150x150.png","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/04\/Tim-Johnson-headshot-150x150.png","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2025\/04\/Tim-Johnson-headshot-150x150.png","caption":"Tim Johnson"},"description":"Tim Johnson is a principal software engineer at Backblaze focused on storage, durability, and performance. He previously served as a frontend web developer for YouTube.","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/author\/tim-johnson\/"}]}},"jetpack_featured_media_url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/10\/bb-bh-Cloud-Performance.png","_links":{"self":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/112050","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/users\/215"}],"replies":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/comments?post=112050"}],"version-history":[{"count":0,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/112050\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media\/110181"}],"wp:attachment":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media?parent=112050"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/categories?post=112050"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/tags?post=112050"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}