lakeFS Integration with Backblaze B2

lakeFS is an open-source data version control platform that provides Git-like branching, commits, merges, and rollbacks for data stored in object storage. You can configure a self-managed lakeFS deployment to use Backblaze B2 Cloud Storage as its underlying object store through the Backblaze B2 S3-Compatible API.

With this configuration, lakeFS manages the versions and metadata for your repositories while Backblaze B2 stores the underlying data objects. This arrangement can help teams create reproducible datasets, isolate experimental changes, and return to an earlier data state without creating a complete copy for every branch.

This guide covers a self-managed lakeFS deployment that uses one Backblaze B2 bucket as its S3-compatible blockstore. It does not cover installing or administering the lakeFS metadata database.

Prerequisites

Before you begin, you must have the following items:

• A Backblaze account with Backblaze B2 Cloud Storage enabled

• A Backblaze B2 bucket, noting the endpoint host and region

• A Backblaze B2 application key that is restricted to the bucket

  Select Read and Write, and select Allow List All Bucket Names.

  Save the keyID and matching applicationKey when you create it; Backblaze shows the applicationKey only once.

• Administrative access to a self-managed lakeFS installation

• A supported lakeFS metadata database, such as PostgreSQL

• Access to the lakeFS configuration file or the environment variables used to run the lakeFS server

For production deployment requirements, see the lakeFS installation documentation.

Configure lakeFS to Use Backblaze B2

Add the Backblaze B2 connection to the blockstore section of your lakeFS configuration file. Retain the database, authentication, and other settings required by your existing lakeFS deployment.

database:
  type: "postgres"
  postgres:
    connection_string: "<postgres-connection-string>"

auth:
  encrypt:
    secret_key: "<lakefs-auth-secret>"

blockstore:
  type: "s3"
  default_namespace_prefix: "s3://<bucket-name>/lakefs/"
  signing:
    secret_key: "<lakefs-blockstore-signing-secret>"
  s3:
    region: "<b2-region>"
    endpoint: "https://s3.<b2-region>.backblazeb2.com"
    force_path_style: true
    discover_bucket_region: false
    credentials:
      access_key_id: "<b2-application-key-id>"
      secret_access_key: "<b2-application-key>"

Replace the placeholders with the values for your environment:

The following settings are important when you use Backblaze B2:

• force_path_style: true directs lakeFS to use path-style S3 requests.

• discover_bucket_region: false prevents lakeFS from attempting to discover the bucket region through an Amazon S3-specific workflow.

• region should match the region in the Backblaze B2 bucket endpoint.

• endpoint must be the S3 endpoint for the bucket's region, not the Backblaze B2 Native API URL. Do not include the bucket name in this value.

The location and loading order of lakeFS configuration files depend on your installation. See the lakeFS server configuration reference for details.

Configure lakeFS with Environment Variables

If you run lakeFS in a container, you can provide the equivalent blockstore settings as environment variables:

LAKEFS_BLOCKSTORE_TYPE="s3"
LAKEFS_BLOCKSTORE_DEFAULT_NAMESPACE_PREFIX="s3://<bucket-name>/lakefs/"
LAKEFS_BLOCKSTORE_SIGNING_SECRET_KEY="<lakefs-blockstore-signing-secret>"
LAKEFS_BLOCKSTORE_S3_REGION="<b2-region>"
LAKEFS_BLOCKSTORE_S3_ENDPOINT="https://s3.<b2-region>.backblazeb2.com"
LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE="true"
LAKEFS_BLOCKSTORE_S3_DISCOVER_BUCKET_REGION="false"
LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID="<b2-application-key-id>"
LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY="<b2-application-key>"

Store credentials and lakeFS secrets in the secret-management system used by your deployment. Do not commit them to source control.

After you update the configuration, restart the lakeFS server by using the process appropriate for your deployment.

Create a lakeFS Repository

Follow the lakeFS instructions to create your first repository. In Storage Namespace, enter an unused location within your Backblaze B2 bucket (for example, s3://<bucket-name>/lakefs/training-data).

Assign each lakeFS repository a unique storage namespace. Do not use the same bucket prefix for multiple repositories.

Verify the Integration

Upload a small test object to your repository and commit the change by using lakectl:

lakectl fs upload -s ./test.txt lakefs://<repository>/main/test.txt
lakectl commit lakefs://<repository>/main -m "Add test object"

The lakeFS quickstart starts its own local lakeFS instance with a local blockstore, so it does not test your Backblaze B2 configuration. In the Backblaze web console, open the bucket and confirm that lakeFS created objects beneath the repository's storage namespace.

Repeat the test with a file larger than 32 MB to verify large-file behavior in your environment.

Do not expect the objects stored in Backblaze B2 to use the same names or directory layout shown in the lakeFS repository. lakeFS manages the physical object layout. Access and modify repository data through lakeFS.

Import Existing Backblaze B2 Data

Creating a lakeFS repository does not automatically add existing objects from the bucket to the repository. To begin tracking existing data, follow the lakeFS instructions to import data. Before you import, make sure the following are true:

• The credentials configured for the lakeFS server can list and read the objects in the source location.

• The source location is in the same Backblaze B2 region as the repository.

• You run the import as the lakeFS administrator user (lakeFS Community) or as a member of the SuperUsers group (lakeFS Enterprise).

For the source location, enter the Backblaze B2 path. For the destination, choose a path in the repository that is empty, because the import deletes any existing objects under the destination prefix. The UI and lakectl import import directories only. To import a single object, use the lakeFS HTTP API or an API client (for example, s3://<bucket-name>/existing-data/).

The source path must not overlap the storage namespace that lakeFS uses to manage the repository. Import does not copy data. Imported objects stay at their original location in Backblaze B2, and lakeFS records pointers to them. After importing the data, make changes through lakeFS. Changes made directly to the original source location are not reflected in the lakeFS metadata and may cause inconsistencies. An object you edit at the source is served with its new content while lakeFS still reports its old size and checksum, and an object you delete at the source can no longer be read through lakeFS.

Troubleshooting

Use the following information to resolve common issues with the integration.

Resolve Authentication Errors

If lakeFS returns an authentication or access-denied error, verify the following items:

• The application key ID is configured as access_key_id.

• The application key is configured as secret_access_key.

• You are using a manually created application key, not the master application key.

• The key has access to the selected bucket.

• The key can list bucket names and list, read, write, and delete files.

Resolve Bucket or Endpoint Errors

If lakeFS cannot find or connect to the bucket, verify the following items:

• The endpoint is the S3 endpoint, not a Backblaze B2 Native API URL.

• The endpoint does not include the bucket name or a bucket path.

• The configured region matches the region in the endpoint.

• force_path_style is set to true.

• discover_bucket_region is set to false.

Resolve Large Upload or TLS Errors

Backblaze B2 does not impose a 32 MB object-size limit. If small uploads work but larger uploads end with a TLS or SSLEOFError, verify the path-style and region settings and inspect any reverse proxy, ingress controller, or load balancer between the client and lakeFS.

Request-body limits or connection timeouts in those components can interrupt an upload before it reaches Backblaze B2. For example, the lakeFS documentation notes that the NGINX Ingress Controller limits request bodies to 1 MiB by default. Increase the applicable request-size and timeout limits for your deployment, and test the same file directly against the Backblaze B2 S3 endpoint to isolate the failing component.

Resolve Missing Existing Objects

Objects that were already in the bucket do not automatically appear in a new lakeFS repository. Import the objects by using the lakeFS import workflow. Do not point a new repository's storage namespace at an existing populated prefix and expect lakeFS to discover the objects automatically.

Next Steps

After you verify the integration, use the lakeFS documentation to continue working with the repository. The examples in the linked quickstart pages use a sample repository named quickstart. Replace lakefs://quickstart/... in the commands with your own repository, such as lakefs://training-data/....

• Create a branch: Create an isolated branch where you can modify data without affecting the main branch.

• Commit and merge changes: Commit your changes and merge them into another branch.

• Roll back changes: Restore repository data to an earlier state.

• Work with data locally: Access and manage lakeFS repository data from your local environment.

• Explore more integrations: Learn about other Backblaze integrations and developer examples at Backblaze Labs.

Additional Resources

• lakeFS website

• lakeFS GitHub repository

• lakeFS documentation

• lakeFS server configuration reference

• lakeFS on-premises installation

• lakeFS data import guide

• lakeFS CLI reference (lakectl)

• About Backblaze B2 Cloud Storage

• Backblaze B2 S3-Compatible API

• Backblaze B2 S3-Compatible application keys

• Create and manage Backblaze B2 buckets

• Backblaze Labs