{"id":109959,"date":"2023-10-05T09:15:00","date_gmt":"2023-10-05T16:15:00","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=109959"},"modified":"2024-08-14T12:01:15","modified_gmt":"2024-08-14T19:01:15","slug":"overload-to-overhaul-how-we-upgraded-drive-stats-data","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/overload-to-overhaul-how-we-upgraded-drive-stats-data\/","title":{"rendered":"Overload to Overhaul: How We Upgraded Drive Stats Data"},"content":{"rendered":"\n
\"A<\/figure>\n\n\n\n
<\/div>\n\n\n\n

This year, we\u2019re celebrating 10 years of Drive Stats<\/a>. Coincidentally, we also made some upgrades to how we run our Drive Stats reports. We reported on how an attempt to migrate triggered a weeks-long recalculation of the dataset, leading us to map the architecture of the Drive Stats data.\u00a0<\/p>\n\n\n\n

This follow-up article focuses on the improvements we made after we fixed the existing bug (because hey, we were already in there), and then presents some of our ideas for future improvements. Remember that those are just ideas<\/em> so far\u2014they may not be live in a month (or ever?), but consider them good food for thought, and know that we’re paying attention so that we can pass this info along to the right people.<\/p>\n\n\n\n

Now, onto the fun stuff.\u00a0<\/p>\n\n\n\n

Quick Refresh: Drive Stats Data Architecture<\/h2>\n\n\n\n

The podstats generator runs on every Storage Pod<\/a>, what we call any host that holds customer data, every few minutes. It\u2019s a C++ program that collects SMART stats<\/a> and a few other attributes, then converts them into an .xml file (\u201cpodstats\u201d). Those are then pushed to a central host in each datacenter and bundled. Once the data leaves these central hosts, it has entered the domain of what we will call Drive Stats.<\/a>\u00a0\u00a0<\/p>\n\n\n\n

Now let\u2019s go into a little more detail: when you\u2019re gathering stats about drives, you\u2019re running a set of modules with dependencies to other modules, forming a data-dependency tree. Each time a module \u201cruns\u201d, it takes information, modifies it, and writes it to a disk. As you run each module, the data will be transformed sequentially. And, once a quarter, we run a special module that collects all the attributes for our Drive Stats reports, collecting data all the way down the tree. <\/p>\n\n\n\n

Here\u2019s a truncated diagram of the whole system, to give you an idea of what the logic looks like:<\/p>\n\n\n\n

\"A
An abbreviated logic map of Drive Stats modules. <\/figcaption><\/figure>\n\n\n\n

As you move down through the module layers, the logic gets more and more specialized. When you run a module, the first thing the module does is check in with the previous module to make sure the data exists and is current. It caches the data to disk at every step, and fills out the logic tree step by step. So for example, drive_stats<\/code>, being a \u201cper-day\u201d module, will write out a file such as \/data\/drive_stats\/2023-01-01.json.gz<\/code> when it finishes processing. This lets future modules read that file to avoid repeating work.<\/p>\n\n\n\n

This work deduplication process saves us a lot of time overall\u2014but it also turned out to be the root cause of our weeks-long process when we were migrating Drive Stats to our new host. We fixed that by implementing versions to each module.\u00a0\u00a0<\/p>\n\n\n\n

While You\u2019re There\u2026 Why Not Upgrade?<\/h2>\n\n\n\n

Once the dust from the bug fix had settled, we moved forward to try to modernize Drive Stats in general. Our daily report still ran quite slowly, on the order of several hours, and there was some low-hanging fruit to chase.<\/p>\n\n\n\n

Waiting On You, failures_with_stats<\/code><\/h3>\n\n\n\n

First things first, we saved a log of a run of our daily reports in Jenkins<\/a>. Then we wrote an analyzer to see which modules were taking a lot of time. failures_with_stats<\/code> was our biggest offender, running for about two hours, while every other module took about 15 minutes.<\/p>\n\n\n\n

\"An
Not quite<\/em> two hours.<\/figcaption><\/figure>\n\n\n\n

Upon investigation, the time cost had to do with how the date_range<\/code> module works. This takes us back to caching: our module checks if the file has been written already, and if it has, it uses the cached file. However, a date range is written to a single file. That is, Drive Stats will recognize \u201cMonday to Wednesday\u201d as distinct from \u201cMonday to Thursday\u201d and re-calculate the entire range. This is a problem for a workload that is essentially doing work for all of time, every day.\u00a0\u00a0<\/p>\n\n\n\n

On top of this, the raw Drive Stats data, which is a dependency for failures_with_stats<\/code>, would be gzipped onto a disk. When each new query triggered a request to recalculate all-time data, each dependency would pick up the podstats file from disk, decompress it, read it into memory, and do that for every day of all time. We were picking up and processing our biggest files every day, and time continued to make that cost larger.<\/p>\n\n\n\n

Our solution was what I called the \u201cDate Range Accumulator.\u201d It works as follows:<\/p>\n\n\n\n