{"id":110674,"date":"2024-01-05T09:43:26","date_gmt":"2024-01-05T17:43:26","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=110674"},"modified":"2025-12-11T13:29:02","modified_gmt":"2025-12-11T21:29:02","slug":"new-open-source-tool-for-consistency-in-cassandra-migrations","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/new-open-source-tool-for-consistency-in-cassandra-migrations\/","title":{"rendered":"New Open Source Tool for Consistency in Cassandra Migrations"},"content":{"rendered":"\r\n
\"A<\/figure>\r\n\r\n\r\n\r\n
\u00a0<\/div>\r\n\r\n\r\n\r\n

Sometimes you find a problem because something breaks, and other times you find a problem the good way\u2014by thinking it through before <\/em>things break. This is a story about one of those bright, shining, lightbulb moments when you find a problem the good way.<\/p>\r\n\r\n\r\n\r\n

On the Backblaze Site Reliability Engineering (SRE) team, we were thinking through an upcoming datacenter migration in Cassandra<\/a>. We were running through all of the various types of queries we would have to do when we had the proverbial \u201caha\u201d moment. We discovered an inconsistency in the way Cassandra handles lightweight transactions (LWTs).<\/p>\r\n\r\n\r\n\r\n

If you\u2019ve ever tried to do a datacenter migration in Cassandra and something got corrupted in the process but you couldn\u2019t figure out why or how\u2014this might be why. I\u2019m going to walk through a short intro on Cassandra, how we use it, and the issue we ran into. Then, I\u2019ll explain the workaround, which we open sourced.\u00a0<\/p>\r\n\r\n\r\n\r\n

\r\n

Get the Open Source Code<\/h4>\r\n

You can download the open source code from our Git repository<\/a>. We\u2019d love to know how you\u2019re using it and how it\u2019s working for you\u2014let us know in the comments.<\/p>\r\n<\/div>\r\n\r\n\r\n\r\n

How We Use Cassandra<\/h2>\r\n\r\n\r\n\r\n

First, if you\u2019re not a Cassandra dev, I should mention that when we say \u201cdatacenter migration\u201d it means something slightly different in Cassandra than what it sounds like. It doesn\u2019t mean a data center migration in the physical sense (although you can use datacenter migrations in Cassandra when you\u2019re moving data from one physical data center to another). In the simplest terms, it involves moving data between two Cassandra or Cassandra-compatible database replica sets within a cluster.<\/p>\r\n\r\n\r\n\r\n

And, if you\u2019re not familiar with Cassandra at all, it\u2019s an open-source, NoSQL, distributed database management system. It was created to handle large amounts of data across many commodity servers, so it fits our use case\u2014lots of data, lots of servers.\u00a0<\/p>\r\n\r\n\r\n\r\n

At Backblaze, we use Cassandra to index filename to location for data stored in Backblaze B2, for example. Because it\u2019s customer data and not just analytics, we care more about durability and consistency than some other applications of Cassandra. We run with three replicas in a single datacenter and \u201cbatch\u201d mode to require writes to be committed to disk before acknowledgement rather than the default \u201cperiodic.\u201d<\/p>\r\n\r\n\r\n\r\n

Datacenter migrations are an important aspect of running Cassandra, especially on bare metal<\/a>. We do a few datacenter migrations per year either for physical data moves, hardware refresh<\/a>, or to change certain cluster layout parameters like tokens per host that are otherwise static.\u00a0<\/p>\r\n\r\n\r\n\r\n

What Are LWTs and Why Do They Matter for Datacenter Migrations in Cassandra?<\/h2>\r\n\r\n\r\n\r\n

First of all, LWTs are neither lightweight nor transactions, but that\u2019s neither here nor there. They are <\/em>an important feature in Cassandra. Here\u2019s why.\u00a0<\/p>\r\n\r\n\r\n\r\n

Cassandra is great at scaling. In something like a replicated SQL cluster, you can add additional replicas for read throughput, but not writes. Cassandra scales writes (as well as reads) nearly linearly with the number of hosts\u2014into the hundreds. Adding nodes is a fairly straightforward and \u201cautomagic\u201d process as well, with no need to do something like manual token range splits. It also handles individual down nodes with little to no impact on queries. Unfortunately, these properties come with a trade-off: a complex and often nonintuitive consistency model that engineers and operators need to understand well.<\/p>\r\n\r\n\r\n\r\n

In a distributed database like Cassandra, data is replicated across multiple nodes for durability and availability.\u00a0<\/p>\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n
\u00a0<\/div>\r\n\r\n\r\n\r\n

Although databases generally allow multiple reads and writes to be submitted at once, they make it look to the outside world like all the operations are happening in order, one at a time. This property is known as serializability, and Cassandra is not serializable. Although it does have a \u201clast write wins\u201d system, there\u2019s no transaction isolation and timestamps can be identical.\u00a0<\/p>\r\n\r\n\r\n\r\n

It\u2019s possible, for example, to have a row that has some columns from one write and other columns from another write. It\u2019s safe if you\u2019re only appending additional rows, but mutating existing rows safely requires careful design. Put another way, you can have two transactions with different data that, to the system, appear to have equal priority.\u00a0<\/p>\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n
\u00a0<\/div>\r\n\r\n\r\n\r\n

How Do LWTs Solve This Problem?<\/h2>\r\n\r\n\r\n\r\n

As a solution for cases where stronger consistency is needed, Cassandra has a feature called \u201cLightweight Transactions\u201d or LWTs. These are not really identical to traditional database transactions, but provide a sort of \u201ccompare and set\u201d operation that also guarantees pending writes are completed before answering a read. This means if you\u2019re trying to change a row\u2019s value from \u201cA\u201d to \u201cB\u201d, a simultaneous attempt to change that row from \u201cA\u201d to \u201cC\u201d will return a failure. This is accomplished by doing a full\u2014not at all lightweight\u2014Paxos round complete with multiple round trips and slow expensive retries in the event of a conflict.

In Cassandra, the minimum consistency level for read and write operations is ONE<\/strong>, meaning that only a single replica needs to acknowledge the operation for it to be considered successful. This is fast, but in a situation where you have one down host, it could mean data loss, and later reads may or may not show the newest write depending on which replicas are involved and whether they\u2019re received the previous write. For better durability and consistency, Cassandra also provides various quorum levels that require a response from multiple replicas, as well as an ALL<\/strong> consistency that requires responses from every replica.<\/p>\r\n\r\n\r\n\r\n

\r\n

Cassandra Is My Type of Database<\/h4>\r\n

Curious to know more about consistency limitations and LWTs in Cassandra? Christopher Batey\u2019s presentation at the 2016 Cassandra Summit<\/a> does a good job of explaining the details.<\/p>\r\n<\/div>\r\n\r\n\r\n\r\n

\r\n
https:\/\/www.youtube.com\/watch?v=wcxQM3ZN20c<\/div>\r\n<\/figure>\r\n\r\n\r\n\r\n
\u00a0<\/div>\r\n\r\n\r\n\r\n

The Problem We Found With LWTs During Datacenter Migrations<\/h2>\r\n\r\n\r\n\r\n

Usually we use one datacenter in Cassandra, but there are circumstances where we sometimes stand up a second datacenter in the cluster and migrate to it, then tear down the original. We typically do this either to change num_tokens<\/code>, to move data when we\u2019re refreshing hardware, or to physically move to another nearby data center.<\/p>\r\n\r\n\r\n\r\n

The TL:DR<\/h3>\r\n\r\n\r\n\r\n

We reasoned through the interaction between LWTs\/serial<\/code> and datacenter migrations and found a hole\u2014there\u2019s no guarantee of LWT correctness during a topology change (that is, a change to the number of replicas) large enough to change the number of replicas needed to satisfy quorum. It turns out that combining LWTs and datacenter migrations can violate consistency guarantees in subtle ways without some specific steps and tools to work around it.<\/p>\r\n\r\n\r\n\r\n

The Long Version<\/h3>\r\n\r\n\r\n\r\n

Let\u2019s say you are standing up a new datacenter, and you need to copy an existing datacenter to it. So, you have two datacenters\u2014datacenter A, the existing datacenter, and datacenter B, the new datacenter. Let\u2019s say datacenter A has three replicas you need to copy for simplicity\u2019s sake, and you\u2019re using quorum writes to ensure consistency.<\/p>\r\n\r\n\r\n\r\n

\r\n

Refresher: What is Quorum-Based Consistency in Cassandra?<\/h4>\r\n

Quorum consistency in Cassandra is based on the concept that a specific number of replicas must participate in a read or write operation to ensure consistency and availability\u2014a majority (n\/2 +1) of the nodes must respond before considering the operation as successful. This ensures that the data is durably stored and available even if a minority of replicas are unavailable.<\/p>\r\n<\/div>\r\n\r\n\r\n\r\n

You have different types of quorum you can choose from, and here\u2019s how those defaults make a decision:\u00a0<\/p>\r\n\r\n\r\n\r\n