Export Data Pipelines to Backblaze B2 with Mage OSS

This guide walks you through building a data pipeline in Mage OSS that exports a dataset to Backblaze B2 Cloud Storage using Mage's native BackblazeB2 storage class, which is built on Backblaze B2's S3-Compatible API.

Note

The BackblazeB2 storage class is not yet available in a stable Mage OSS release. Until it ships, use the boto3-based approach to export data to Backblaze B2.

By the end of this guide, you will have:

• A running Mage OSS instance on your local machine

• A Backblaze B2 bucket configured for S3-compatible access

• An application key with the correct permissions

• Your B2 credentials stored in Mage's io_config.yaml instead of hardcoded in block code

• A complete Mage pipeline that loads, transforms, and exports data to your bucket using the BackblazeB2 class

Version Note
The BackblazeB2 storage class was added to Mage OSS after version 0.9.79, so it ships in the next release (expected to be 0.9.80 or 0.10.0). On Mage 0.9.79 or earlier, from mage_ai.io.backblaze_b2 import BackblazeB2 fails with ModuleNotFoundError. Until the next stable release is published, pull Mage's bleeding-edge image (docker pull mageai/mageai:alpha), which is built from the development branch and already includes the class. See the Backblaze B2 integration reference for the canonical, up-to-date documentation.

Prerequisites

Make sure the following prerequisites are in place before you begin.

• Docker Desktop (macOS/Windows) or Docker Engine (Linux). This is required to run Mage OSS in a container.

• Backblaze B2 enabled on your account.

• A Backblaze B2 bucket, noting the endpoint URL shown on the bucket page (for example, s3.us-west-004.backblazeb2.com).

• A Backblaze B2 application key scoped to that bucket. Create a standard application key, not the master application key: the master key is not S3-compatible and will not work here. Copy the keyID and applicationKey when the key is created, because the applicationKey is shown only once.

Install and Start Mage OSS

Mage OSS runs as a Docker container that mounts your local project directory, so your pipeline files persist between sessions.

1. Open a terminal and create a new directory for your Mage project, then change into it:

   mkdir mage-b2-project
   cd mage-b2-project

2. Run the following command to pull the Mage image and start the server. Replace my_project with your Mage project name:

   # macOS / Linux
   docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
     /app/run_app.sh mage start my_project

   # Windows Command Prompt
   docker run -it -p 6789:6789 -v "%cd%:/home/src" mageai/mageai ^
     /app/run_app.sh mage start my_project

   # Windows PowerShell
   docker run -it -p 6789:6789 -v ${PWD}:/home/src mageai/mageai `
     /app/run_app.sh mage start my_project

   The -v flag mounts your current directory into the container, so your pipeline files are saved to your local filesystem and survive container restarts.

3. After the container is running, open your browser and navigate to http://localhost:6789. You should see the Mage dashboard. Your project will appear in the left sidebar.

Add Your Backblaze B2 Credentials

Mage stores connection credentials in a file named io_config.yaml at the root of your project. The BackblazeB2 class reads your B2 application key from this file, which keeps credentials out of your block code and makes them easy to rotate.

1. In the Mage UI, expand the left side of the screen to view the file browser.

2. Scroll down and click the file named io_config.yaml.

3. Under the default profile, add your Backblaze B2 credentials. You can keep multiple profiles; add these under whichever one you plan to use:

   version: 0.1.1
   default:
     B2_APPLICATION_KEY_ID: your_key_id
     B2_APPLICATION_KEY: your_application_key
     # Optional. The endpoint defaults to the us-west-004 region. If your bucket
     # is in another region, uncomment and set this to that region's endpoint
     # (shown on the B2 bucket page), for example us-east-005 or eu-central-003:
     # B2_ENDPOINT_URL: https://s3.us-east-005.backblazeb2.com

4. Save the file.

Note
For backwards compatibility, the class also falls back to AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_ENDPOINT when the corresponding B2_* keys are not set, so an existing AWS-style profile keeps working.

Application Key Permissions

Before continuing, confirm that your application key grants the following capabilities on your bucket. Missing any of these will cause the pipeline to fail at runtime.

• readFiles

• writeFiles

• listFiles

If your key is restricted to a single bucket, it may also need the listAllBucketNames capability for S3 SDK compatibility.

Create a New Pipeline

Mage pipelines are built from individual blocks (loaders, transformers, and exporters) that each do one thing. This modular structure makes it easy to reuse blocks across pipelines and test each step independently. In this step, you'll create the pipeline shell that the blocks will live in.

1. In the Mage UI, click New pipeline in the left sidebar.

2. Select Standard (batch) as the pipeline type.

3. Give the pipeline a name (for example, export_to_b2).

4. Click Create Pipeline.

You will be taken to the pipeline editor, which shows a blank canvas ready for blocks.

Block Types

The next three steps walk you through adding one block at a time. Here's what each one does before you build it:

Add a Data Loader Block

The data loader is the starting point of every Mage pipeline. It defines what data enters the pipeline. For this guide, a simple loader is created that generates a sample dataset, so you can verify the full export flow without needing a live data source.

1. In the pipeline editor, click + Data Loader.

2. Select Python, and select Generic (no template). Mage will create a new block with a default template.

3. Clear the template and paste the following Python code. This generates a small Pandas DataFrame that represents a set of product records:

   import pandas as pd
   
   @data_loader
   def load_data(*args, **kwargs):
       data = {
           'product_id':   [1, 2, 3, 4, 5],
           'product_name': ['Widget A', 'Widget B', 'Gadget C', 'Tool D', 'Part E'],
           'category':     ['Widgets', 'Widgets', 'Gadgets', 'Tools', 'Parts'],
           'price':        [9.99, 14.99, 24.99, 4.99, 2.49],
           'in_stock':     [True, False, True, True, False],
       }
       return pd.DataFrame(data)

4. Click Run block at the top right of the block. Mage will execute the code and display a preview of the output DataFrame below the block.

5. Confirm that you see five rows of product data before continuing.

Add a Transformer Block

Transformer blocks are where you clean, filter, rename, or enrich data before it leaves the pipeline. Even a minimal transformer is good practice, because it keeps your logic separated from loading and exporting and gives you a place to add validation later.

1. In the pipeline editor, click + Transformer.

2. Select Python, and select Generic (no template) to add a transformer block connected to your loader.

3. Enter the following Python code. This transformer normalizes column names to lowercase and drops any rows where the price field is null:

   import pandas as pd
   
   @transformer
   def transform(data: pd.DataFrame, *args, **kwargs):
       data.columns = [col.lower() for col in data.columns]
       data = data.dropna(subset=['price'])
       return data

4. Click Run block. The output should show the same five rows with lowercase column names (product_id, product_name, and so on).

5. If you see any errors, check that the upstream loader block ran successfully first. Mage requires upstream blocks to pass data before a transformer can execute.

Add a Data Exporter Block for Backblaze B2

The data exporter connects your pipeline to Backblaze B2 using the BackblazeB2 storage class, which reads the credentials you saved in io_config.yaml. Because credentials live in the config file rather than in block code, you can rotate them without touching your pipeline.

1. In the pipeline editor, click + Data Exporter.

2. Select Python, and select Generic (no template). This gives you a blank exporter to fill in.

3. Replace the template with the following Python code, updating bucket_name and object_key for your bucket:

   from mage_ai.settings.repo import get_repo_path
   from mage_ai.io.config import ConfigFileLoader
   from mage_ai.io.backblaze_b2 import BackblazeB2
   from os import path
   from pandas import DataFrame
   
   if 'data_exporter' not in globals():
       from mage_ai.data_preparation.decorators import data_exporter
   
   
   @data_exporter
   def export_to_b2(df: DataFrame, **kwargs) -> None:
       config_path = path.join(get_repo_path(), 'io_config.yaml')
       config_profile = 'default'  # Change if you use a different profile
   
       bucket_name = 'your-bucket-name'
       object_key = 'exports/products.csv'  # .parquet, .json, and .hdf5 also work
   
       BackblazeB2.with_config(ConfigFileLoader(config_path, config_profile)).export(
           df,
           bucket_name,
           object_key,
           index=False,  # keyword args pass through to the CSV/Parquet writer
       )
       print(f'Exported {len(df)} rows to {bucket_name}/{object_key}')

   The export method infers the file format from the object_key extension, so writing CSV, Parquet, or JSON is just a matter of the extension you choose. Extra keyword arguments (such as index=False) are forwarded to the underlying Pandas writer.

4. Click Run block. Mage will execute the exporter and print a confirmation line showing how many rows were uploaded. If the block completes without an error, your data is now in Backblaze B2.

Verify the Export in Backblaze B2

Before treating this pipeline as production-ready, confirm that the file actually landed in your bucket with the expected content. This step closes the loop between what Mage reports and what Backblaze B2 actually received.

1. Sign in to the Backblaze web console.

2. In the left navigation menu under B2 Cloud Storage, click Buckets.

3. Click Browse Files next to your bucket.

4. In the file browser, look for the path you set as object_key in the exporter block. For example, if you used exports/products.csv, you should see a folder called exports containing products.csv, with a recent modification timestamp.

5. Click the file name to view its details, then click Download to save a copy locally.

6. Open the file and verify that the rows match what you saw in the Mage transformer output.

If the file is not present, check the exporter block's output log for errors. Common causes are a wrong endpoint URL, swapped keyID and applicationKey values, or a bucket-restricted key missing the listAllBucketNames capability.

Schedule the Pipeline (Optional)

Running a pipeline manually is useful for testing, but most production use cases require automated scheduling. Mage OSS includes a built-in scheduler that can run your pipeline nightly, hourly, or on a custom cron schedule, without any external tooling.

1. In the Mage UI, click Triggers, or navigate to your pipeline and select the Triggers tab at the top of the editor.

2. Click Add Trigger, and select Schedule.

3. Configure the trigger as follows:    

   • Trigger Name: Enter a descriptive label (for example, nightly_b2_export).

   • Frequency: Select every hour, every day, every week, or custom cron.

   • Start date / time: Enter the date and time for the first run.

   • Status: Select Active to run automatically.

4. Click Save Changes, then toggle the trigger status to Active.

5. Monitor run history and logs from the Runs tab in the pipeline editor.

Mage will run the pipeline on your chosen schedule and write a new file to Backblaze B2 each time.

Troubleshooting

If something goes wrong, start with the exporter block's output log in the Mage UI. Most issues fall into one of these categories:

Next Steps

Now that the core integration is working, here are several ways to extend this pattern:

• Use real data: Replace the sample loader block with a query against your database or a call to an external API. Mage OSS includes prebuilt loader templates for PostgreSQL, MySQL, BigQuery, Snowflake, and more.

• Use Parquet instead of CSV: Parquet files are smaller and faster to query downstream. Change the object_key extension from .csv to .parquet and the BackblazeB2 exporter will handle the rest.

• Add date partitioning: Append a timestamp to object_key so each daily run creates its own file instead of overwriting the previous one.

• Store credentials securely: Use Mage's built-in Secrets manager (Settings > Secrets) to store keyID and applicationKey as named secrets, then reference them with kwargs['env_vars']['B2_KEY_ID'] in your block code.

• Upgrade to Mage Pro: When you are ready for team collaboration, multi-environment orchestration, and AI-assisted debugging, Mage Pro imports your OSS pipelines via Git. See mage.ai for details.

Resources

• Mage OSS on GitHub: Source code, issues, and release history.

• Mage OSS Documentation: Full reference for pipeline setup, block types, scheduling, and connectors.

• Mage Backblaze B2 Integration: Reference for the BackblazeB2 storage class, credential keys, and endpoint overrides.

• Backblaze B2 S3-Compatible API Reference: Endpoint reference, supported operations, and authentication details.