Lukka
Lukka Builds a Predictable Data Layer for Unpredictable Analytics
Lukka
Jakub Fijolek, Head of Infrastructure, Lukka
active data
monthly reads
ingestion processes
Lukka continuously ingests data from more than 50 blockchains, preserving every block and keeping complete transaction histories available at low latency. Its compute runs across AWS and two on-premises data centers, but storing data in AWS made cross-environment access costly and unpredictable, especially with billions of small objects and irregular retrieval patterns.
Lukka moved nearly 3PB of active blockchain data to B2, using it as shared object storage for AWS Frankfurt and on-premises clusters. S3 compatibility allowed 200+ Java ingestion processes, Spark, Hive-style partitioning, and AWS SDKs to keep working. During migration, two-way event notifications synchronized writes between S3 and B2 so live ingestion could continue while producers and consumers moved in stages.
B2 provides comparable access latency and throughput with simpler, more predictable economics. Jobs can run both on prem and in the cloud wherever compute is available without being forced into AWS to avoid egress charges. More than nine petabytes of monthly off-AWS reads fits within B2’s included free egress, helping engineers run analytics and power agentic workflows without adding storage-cost approvals to every decision.
Lukka’s blockchain nodes continually surface new blocks, which extractors capture and preserve as raw JSON or binary objects in B2. Transformers then normalize those records into Lukka’s canonical format, identifying transactions, addresses, assets, and values. Inserter services route the normalized data through AWS MSK into application-specific datasets and tables. Apache Spark workers running in AWS Frankfurt and Lukka’s Warsaw and Katowice data centers share the workload, accessing the same B2 data layer. Hive-compatible paths and partitioning help Spark target only the relevant objects instead of scanning entire datasets.
.png)
.svg)
Lukka is an enterprise data company serving institutions that need normalized, auditable digital asset data for compliance, investigations, pricing, valuation, accounting, tax, and portfolio management.
For Lukka’s blockchain analytics products, old data is not cold data. A customer may ask about a highly active exchange address or one that last moved funds years ago—and either request can require the address’s complete transaction history. Meanwhile, networks such as Solana can add a new block roughly every 250 milliseconds. Lukka must preserve every block, ingest new data continuously, and make the full history available at low latency. That combination rules out traditional archival tiers and creates billions of small, frequently accessed objects.
We have a blend of archival and fast-retrieval needs. With Backblaze, we know roughly how much data and traffic we have, without having to decide which partitions or files belong on which tier.
Jakub Fijolek, Head of Infrastructure, Lukka
AWS worked well when compute and storage stayed inside its ecosystem. Lukka, however, operates a distributed model: Spark workloads can run in AWS Frankfurt or in its Warsaw and Katowice data centers. Pulling large datasets from S3 to on-premises compute created substantial and hard-to-predict egress charges. Tiered storage did not solve the problem. Standard storage was expensive at Lukka’s scale, while infrequent-access tiers added retrieval charges to workloads whose access patterns could change with every customer request.
Lukka’s migration could not pause continuous blockchain ingestion. After the initial bulk load, the team:
This let Lukka migrate producers and consumers in stages while the live pipeline kept moving. Today, B2 supports active ingestion and processing—not simply backup or archive storage.
For us, S3 compatibility was a prerequisite. We are still using the AWS SDK almost everywhere. In Spark, we changed the connection classes, endpoints, paths, and region—and Backblaze just worked.
Jakub Fijolek, Head of Infrastructure, Lukka
B2 now gives Lukka’s three compute locations a shared data layer. Services are replicated across the environment, and queued work is picked up wherever capacity is available. Because jobs no longer need to be forced into AWS to avoid egress, Lukka can use its on-premises infrastructure as a stable baseline and burst in the cloud when needed. More than nine petabytes of monthly reads outside AWS remains within B2’s 3x free-egress allowance.
It’s hard to stress how important pricing predictability is for a data company. We can simplify workflows and let people process data independently without worrying that a job will cause an unexpected bill spike.
Jakub Fijolek, Head of Infrastructure, Lukka
That flexibility matters even more as Lukka expands agentic analytics. Agents make data access less predictable than analyst-authored queries: one request may retrieve megabytes, another terabytes. With retrieval volume and object counts largely removed from the cost equation, Lukka can measure and manage compute—the variable Spark exposes precisely—without making every data job carry a cloud-storage pricing model.
.png)
.png)
