Use Backblaze B2 as an Artifact Store with ZenML

ZenML automatically tracks, versions, and stores every output your pipeline steps produce. These outputs are called artifacts, and they are written to whichever artifact store is registered in your active ZenML stack. By default, ZenML uses your local filesystem. Switching to Backblaze B2 Cloud Storage means your artifacts are stored in the cloud, accessible to teammates, and no longer tied to a single machine.

ZenML connects to Backblaze B2 through its B2 artifact store flavor, which uses Backblaze B2's S3-compatible API under the hood. No custom connector code is required.

By the end of this guide, you will have:

• A Backblaze B2 bucket and application key configured for S3-compatible access

• ZenML installed with the Backblaze B2 integration

• A ZenML stack registered with Backblaze B2 as its artifact store

• A working pipeline that stores its artifacts in Backblaze B2

Prerequisites

Make sure the following prerequisites are in place before you begin.

• Python 3.10 or later
  For new environments, Backblaze recommends Python 3.11 or later because Python 3.10 reaches end of life in October 2026.

• pip (comes with Python)

• Backblaze B2 enabled on your account.

• A Backblaze B2 bucket noting the endpoint URL.

• A Backblaze B2 application key scoped to the bucket you just created.
  Save the keyID and matching applicationKey when you create it; Backblaze shows the applicationKey only once.

Install ZenML and the S3 Integration

ZenML is a Python package installed via pip. The S3 integration is a separate add-on that provides the artifact store flavor used to connect to any S3-compatible storage service, including Backblaze B2. You need to install both before you can register Backblaze B2 as an artifact store.

1. Run the following command to install ZenML:

   pip install zenml[local]

2. ZenML manages integration dependencies through its own CLI. Run the following command to install the Backblaze B2 artifact store integration and its dependencies:

   zenml integration install b2 -y

   The -y flag skips the confirmation prompt. This installs the s3fs and boto3 libraries that ZenML uses under the hood to communicate with Backblaze B2.

3. Initialize ZenML in your project directory. This creates a local database that tracks your stacks, runs, and artifacts.

   zenml init

   Note

   Run zenml init at the root of your project, not inside a subdirectory. ZenML uses this location to find your pipeline files when running remotely.

Set Your Backblaze B2 Credentials as Environment Variables

ZenML's Backblaze B2 artifact store reads credentials from environment variables at runtime. Storing them this way keeps your keys out of your code and configuration files, which matters both for security and for sharing your stack configuration with teammates who use their own credentials.

Set the following environment variables in your terminal session before running any ZenML commands:

macOS / Linux

export B2_APPLICATION_KEY_ID=YOUR_APPLICATION_KEY_ID
export B2_APPLICATION_KEY=YOUR_APPLICATION_KEY

Windows Command Prompt

set B2_APPLICATION_KEY_IDAWS_ACCESS_KEY_ID=YOUR_APPLICATION_KEY_ID
set B2_APPLICATION_KEYAWS_SECRET_ACCESS_KEY=YOUR_APPLICATION_KEY

Windows PowerShell

$env:B2_APPLICATION_KEY_IDAWS_ACCESS_KEY_ID="YOUR_APPLICATION_KEY_ID"
$env:B2_APPLICATION_KEYAWS_SECRET_ACCESS_KEY="YOUR_APPLICATION_KEY"

Note

ZenML's B2 integration reads Backblaze's standard environment variable names. If you use the generic S3 artifact store instead, use AWS-style variable names. However, this guide uses the dedicated Backblaze B2 flavor.

Register Backblaze B2 as an Artifact Store

Now that your credentials are set, you can register your Backblaze B2 bucket as a named artifact store in ZenML. Registering it gives it a name you can reference when building stacks, and records the bucket path and endpoint so ZenML knows where and how to connect.

1. Run the following command, replacing the placeholder values with your bucket name and endpoint URL:

   zenml artifact-store register b2_artifact_store \
     --flavor=b23 \
     --path=s3://your-bucket-name \
     --client_kwargs='{"endpoint_url": "https://s3.us-west-004.backblazeb2.com"}'

   --flavor=b2s3 tells ZenML to use the Backblaze B2 artifact store integration.

   --path is the root path inside your bucket where ZenML stores artifacts. It uses the s3:// URI scheme because Backblaze B2 exposes an S3-compatible API.

   --client_kwargs passes the Backblaze B2 endpoint URL to the underlying boto3 client, overriding the default AWS endpoint. Use the exact endpoint URL shown for your bucket, such as https://s3.us-east-005.backblazeb2.com or https://s3.us-west-004.backblazeb2.com.

2. Run the following command to verify the artifact store was registered correctly:

   zenml artifact-store describe b2_artifact_store

   You should see the store listed with its flavor, path, and endpoint URL.

Register a Stack with Backblaze B2

A ZenML stack is a named collection of infrastructure components that your pipelines run on. Every stack requires at least an orchestrator (which runs your steps) and an artifact store (which stores your outputs). In this step, you'll create a new stack that uses Backblaze B2 as its artifact store.

1. Run the following command to register the stack:

   zenml stack register b2_stack \
     --orchestrator=default \
     --artifact-store=b2_artifact_store \
     --set

   The --orchestrator=default flag uses ZenML's built-in local orchestrator, which runs pipeline steps on your machine. The --set flag makes this stack active immediately, so any pipeline you run from this point forward uses Backblaze B2 for artifact storage.

2. Run the following command to confirm the active stack:  

   zenml stack describe

   The output should show b2_stack as the active stack, with b2_artifact_store listed as the artifact store component.

Write and Run a Pipeline

With your stack active, any ZenML pipeline you run will automatically store its artifacts in Backblaze B2. You do not need to change your pipeline code to make this work. The artifact store is wired in at the stack level. In this step, you write a simple pipeline to confirm the end-to-end connection.

1. Create a new file called run.py and paste the following Python code:

   import pandas as pd
   from zenml import step, pipeline
   
   @step
   def load_data() -> pd.DataFrame:
       """
       Loads a sample dataset. In a real pipeline, replace this
       with a database query, API call, or file load.
       """
       return pd.DataFrame({
           'product_id':   [1, 2, 3, 4, 5],
           'product_name': ['Widget A', 'Widget B', 'Gadget C', 'Tool D', 'Part E'],
           'price':        [9.99, 14.99, 24.99, 4.99, 2.49],
       })
   
   @step
   def summarize_data(data: pd.DataFrame) -> dict:
       """
       Produces a summary of the dataset. ZenML stores the
       returned dict as a versioned artifact in Backblaze B2.
       """
       return {
           'num_rows':    len(data),
           'avg_price':   round(data['price'].mean(), 2),
           'total_value': round(data['price'].sum(), 2),
       }
   
   @pipeline
   def b2_pipeline():
       data    = load_data()
       summary = summarize_data(data)
   
   if __name__ == '__main__':
       b2_pipeline()

2. Run the following command to run the pipeline:

   python run.py

   ZenML will execute both steps in sequence and print a log showing each step's status. When the run completes, the data DataFrame and the summary dict have both been serialized and written to your Backblaze B2 bucket as versioned artifacts.

Verify the Artifacts in Backblaze B2

Before treating this setup as production-ready, confirm that ZenML actually wrote artifact files to your bucket. This step closes the loop between what ZenML reports locally and what Backblaze B2 actually received.

1. Sign in to the Backblaze web console.

2. In the left navigation menu under B2 Cloud Storage, click Buckets.

3. Click Browse Files next to your bucket.

4. ZenML writes artifacts to the path that you configured for the artifact store. In a typical run, you should see folders for step outputs, such as load_data and summarize_data, along with a logs folder. Navigate into these folders to find serialized files for each artifact that the pipeline produced, including the DataFrame and summary dictionary.

5. Run the following command to list artifact versions tracked in the current ZenML database:

   zenml artifact list

   You should see entries for the load_data and summarize_data outputs, each with a version number, type, and URI pointing to your Backblaze B2 bucket.

Note

If no artifacts appear in the bucket, check that the correct stack is active (zenml stack describe) and that your environment variables are still set. Environment variables set with export or set apply only to the current terminal session.

Troubleshooting

If something goes wrong, check the terminal output from python run.py first. ZenML prints step-level logs including any errors. This table lists common issues and their fixes:

Next Steps

Now that the core integration is working, here are several ways to extend this setup:

• Use real data: Replace the sample load_data step with a query against a database or a call to an external API. ZenML handles the serialization automatically for pandas DataFrames, NumPy arrays, and many other common types.

• Add more steps: Chain additional steps to your pipeline for model training, evaluation, or export. Every step output is automatically versioned and stored in Backblaze B2.

• Inspect runs: Use zenml artifact list or your configured ZenML dashboard to review stored artifacts and lineage.

• Keep credentials out of commands: Instead of passing keys on the command line, continue using B2_APPLICATION_KEY_ID and B2_APPLICATION_KEY in your environment, or store them in a ZenML secret and reference the secret when you register the artifact store.

• Share the stack: Export your stack configuration with zenml stack export b2_stack b2_stack.yaml so teammates can import it with zenml stack import and use the same Backblaze B2 bucket.

• Upgrade to ZenML Pro: When you need a centrally deployed ZenML server, team collaboration, and a hosted dashboard, ZenML Pro provides these on top of the same open-source framework. See zenml.io for details.

• Explore more integrations: To learn about other Backblaze integrations and developer examples, visit Backblaze Labs.

Resources

• ZenML Documentation: Full reference for pipelines, stacks, steps, and integrations

• ZenML S3 Artifact Store Reference: Configuration parameters, authentication methods, and advanced options for the S3 artifact store flavor

• ZenML Installation Guide: Full installation instructions including Docker and Jupyter options

• ZenML S3 Artifact Store Reference: Advanced S3-compatible configuration shared by the B2 flavor

• Backblaze B2 S3-Compatible API Reference: Endpoint reference and authentication details

• Backblaze Labs: Backblaze B2 integrations and developer examples