{"id":87225,"date":"2019-01-10T08:00:06","date_gmt":"2019-01-10T16:00:06","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=87225"},"modified":"2025-12-12T07:06:38","modified_gmt":"2025-12-12T15:06:38","slug":"wide-partitions-in-apache-cassandra-3-11","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/wide-partitions-in-apache-cassandra-3-11\/","title":{"rendered":"How We Optimized Storage and Performance of Apache Cassandra at Backblaze"},"content":{"rendered":"

\"Guest<\/a><\/p>\n

\n

Backblaze uses Apache Cassandra, a high-performance, scalable distributed database to help manage hundreds of petabytes<\/a> of data. We engaged the folks at The Last Pickle to use their extensive experience to optimize the capabilities and performance of our Cassandra 3.11 cluster, and now they want to share their experience with a wider audience to explain what they found. We agree; enjoy!<\/p>\n

— Andy<\/p>\n<\/div>\n

Wide Partitions in Apache Cassandra 3.11<\/h2>\n

by Mick Semb Wever, Consultant, The Last Pickle<\/a><\/p>\n

Wide partitions in Cassandra can put tremendous pressure on the Java heap and garbage collector, impact read latencies, and can cause issues ranging from load shedding and dropped messages to crashed and downed nodes.<\/p>\n

While the theoretical limit on the number of cells per partition has always been two billion cells, the reality has been quite different, as the impacts of heap pressure show. To mitigate these problems, the community has offered a standard recommendation for Cassandra users to keep partitions under 400MB, and preferably under 100MB.<\/p>\n

However, in version 3 many improvements were made that affected how Cassandra handles wide partitions. Memtables, caches, and SSTable components were moved off-heap, the storage engine was rewritten in CASSANDRA-8099<\/a>, and Robert Stupp made a number of other improvements listed under CASSANDRA-11206<\/a>.<\/p>\n

While working with Backblaze<\/a> and operating a Cassandra version 3.11 cluster, we had the opportunity to test and validate how Cassandra actually handles partitions with this latest version. We will demonstrate that well designed data models can go beyond the existing 400MB recommendation without nodes crashing through heap pressure.<\/p>\n

Below, we walk through how Cassandra writes partitions to disk in 3.11, look at how wide partitions impact read latencies, and then present our testing and verification of wide partition impacts on the cluster using the work we did with Backblaze.<\/p>\n

The Art and Science of Writing Wide Partitions to Disk<\/h2>\n

First we need to understand what a partition is and how Cassandra writes partitions to disk in version 3.11.<\/p>\n

Each SSTable contains a set of files, and the (\u2013Data.db<\/code>) file contains numerous partitions.<\/p>\n

The layout of a partition in the \u2013Data.db<\/code> file has three components: a header, followed by zero or one static rows, which is followed by zero or more ordered Clusterable<\/a> objects. The Clusterable object in this file may either be a row or a RangeTombstone that deletes data with each wide partition containing many Clusterable objects. For an excellent in-depth examination of this, see Aaron\u2019s blog post Cassandra 3.x Storage Engine<\/a>.<\/p>\n

The \u2013Index.db<\/code> file stores offsets for the partitions, as well as the IndexInfo<\/code> serialized objects for each partition. These indices facilitate locating the data on disk within the \u2013Data.db<\/code> file. Stored partition offsets are represented by a subclass of the RowIndexEntry<\/a>. This subclass is chosen by the the ColumnIndex<\/a> and depends on the size of the partition:<\/p>\n