Use NVIDIA AIStore with Backblaze B2

NVIDIA AIStore (AIS) is a distributed storage system designed for AI and machine learning workloads. You can use AIStore with Backblaze B2 Cloud Storage to keep datasets and generated artifacts in Backblaze B2 while caching frequently accessed data close to your compute resources.

The Backblaze AIStore samples use Backblaze B2 as the system of record. AIStore connects to an existing Backblaze B2  bucket through the Backblaze S3-Compatible API, caches data retrieved from Backblaze B2, and writes completed output back to Backblaze B2.

Backblaze provides sample workflows that demonstrate how to:

• Fine-tune NVIDIA Isaac GR00T using robotics datasets stored in B2.

• Curate video datasets with NVIDIA NeMo Curator and Cosmos-Embed1 while storing the source data and curated output in B2.

How AIStore Works with Backblaze B2

AIStore can use an S3-compatible bucket as remote backend storage. When an object is not already cached, AIStore retrieves it from the remote bucket and stores a local copy. Later requests for the same object can be served from the AIStore cache instead of retrieving the object from B2 again.

This behavior is useful for AI workloads that repeatedly stage the same datasets for processing or training.

The two Backblaze samples use AIStore differently:

The GR00T workflow prefetches a robotics dataset from B2 into AIStore and then copies a verified version to local disk. GR00T reads the local copy during training.

The video-curation workflow prefetches source videos into AIStore, and NeMo Curator reads the cached videos through AIStore's S3 interface.

In both workflows, completed output is uploaded through AIStore to B2.

Prerequisites

Before you begin, you must have the following items:

• A Backblaze account with Backblaze B2 Cloud Storage enabled

• A Backblaze B2 bucket, noting the endpoint host and region

• A Backblaze B2 application key restricted to the bucket with the following capabilities:

  • listBuckets

  • listFiles

  • readFiles

  • writeFiles

• Your bucket's S3-Compatible API endpoint and region

• Docker with Docker Compose v2

• curl

• uv

• Python 3.11 or later for the sample tooling

The GPU and software requirements differ between the GR00T and video-curation workflows. Review the requirements for the workflow that you want to run before configuring your GPU host.

Note: The Docker Compose configuration included with these samples is intended for a single-host demonstration. It runs one AIStore proxy and one target without AIStore authentication and binds the services to 127.0.0.1. Do not expose this configuration to an untrusted network.

Configure NVIDIA AIStore to Use Backblaze

Both samples use the same basic B2 environment configuration.

From the sample repository, copy the environment template:

cp .env.example .env

Open .env and configure the following values:

B2_ENDPOINT=https://s3.<region>.backblazeb2.com

B2_REGION=<region>

B2_BUCKET=<bucket-name>

B2_KEY_ID=<application-key-id>

B2_APP_KEY=<application-key>

AIS_ENDPOINT=http://127.0.0.1:51080

The endpoint and region must match the B2 bucket.

Keep .env private and do not commit it to source control.

You can optionally set AIS_CACHE_DIR to store the AIStore cache on a mounted local disk:

AIS_CACHE_DIR=/mnt/nvme/ais

Set this value before you start AIStore.

Train NVIDIA Isaac GR00T with AIStore and Backblaze B2

The AIStore B2 GR00T Robotics Training sample fine-tunes NVIDIA Isaac GR00T N1.7 using robot episodes stored in B2.

The workflow performs the following operations:

1. Uploads a demonstration dataset to B2.

2. Prefetches the dataset from B2 into AIStore.

3. Copies and verifies the cached dataset in a local staging directory.

4. Runs GR00T against the local staged data.

5. Writes checkpoints locally during training.

6. Uploads completed checkpoints through AIStore to B2.

AIStore is used during dataset staging. GR00T itself reads from local disk during training. As a result, AIStore can reduce repeated B2 reads and staging time, but it does not make the GR00T training computation itself faster.

GR00T Requirements

In addition to the common requirements, the current sample requires:

• Git and Git LFS.

• Linux x86-64 running Ubuntu 22.04 or 24.04.

• CUDA 12.8 toolkit.

• An NVIDIA driver compatible with CUDA 12.8.

• FFmpeg 4 through 7 shared libraries required by the pinned TorchCodec version.

• An NVIDIA GPU with 48 GB VRAM recommended for the default tuning configuration.

• Access to the gated nvidia/Cosmos-Reason2-2B model on Hugging Face.

The sample application uses Python 3.11 or later. make setup-gpu installs the upstream GR00T trainer separately using Python 3.10.

Clone and Set Up the GR00T Sample

Clone the repository:

git clone https://github.com/backblaze-b2-samples/aistore-b2-gr00t-robotics-training.git

cd aistore-b2-gr00t-robotics-training

On the GPU host, configure the CUDA 12.8 environment:

export CUDA_HOME=/usr/local/cuda-12.8

export PATH="$CUDA_HOME/bin:$PATH"

nvcc --version

The command should report CUDA release 12.8.

On Ubuntu 22.04 or 24.04, install the system FFmpeg package if necessary:

sudo apt-get update

sudo apt-get install -y ffmpeg

Install the sample dependencies:

make setup

This command creates the locked sample environment, clones the pinned Isaac GR00T release, and downloads the included demonstration dataset through Git LFS.

The demonstration dataset contains five SO-101 robot episodes, two camera views, and LeRobot v2.1 metadata.

Create and configure .env as described in Configure NVIDIA AIStore to Use Backblaze.

Start AIStore

Start the local AIStore cluster and attach the B2 bucket:

make up

The command starts one AIStore proxy and one target and attaches the existing B2 bucket through the S3-Compatible API.

Successful output includes a line similar to:

attached s3://<bucket> -> https://s3.<region>.backblazeb2.com (region <region>) at http://127.0.0.1:51080

AIStore uses its S3 backend with your B2 endpoint. An AWS account is not required.

Upload and Verify the Demonstration Dataset

Upload the demonstration dataset to B2:

make seed

The sample stores the dataset under:

datasets/cube_to_bowl_5/

The upload also creates a _SUCCESS.json publication record after the files are successfully written.

Verify the durable copy in B2:

make verify PREFIX=datasets/cube_to_bowl_5 ARGS="--direct"

Verification checks the size and SHA-256 hash of the files recorded in the publication manifest.

Install the GR00T Training Environment

Install the upstream GR00T training environment:

make setup-gpu

After your Hugging Face account has access to nvidia/Cosmos-Reason2-2B, authenticate on the training host:

hf auth login

Use an account that has access to the gated model.

Run the GR00T Workflow

First, display the staging, training, and checkpoint destinations without running the job:

make run ARGS="--dry-run"

The output includes the stage:, train:, and sync: commands that the sample will use.

Create a run name:

RUN_NAME="demo-$(date -u +%Y%m%dT%H%M%SZ)"

Run the 20-step validation workflow:

make run ARGS="--run-name $RUN_NAME --max-steps 20 -- --global-batch-size 1 --save-steps 10 --save-total-limit 2"

The workflow stages and verifies the dataset before starting GR00T. During training, checkpoints are written locally and completed checkpoints are uploaded through AIStore to B2.

The sample publishes checkpoints at:

checkpoints/<run-name>/checkpoint-N/

The 20-step run is intended to validate dataset staging, GPU training, checkpoint publication, and B2 integrity. It is not intended to produce a useful trained robot policy.

Checkpoints contain model weights and optimizer state and can require substantial local and B2 storage even during a short run.

Verify GR00T Checkpoints

Verify the checkpoints directly against B2:

make verify PREFIX="checkpoints/$RUN_NAME/checkpoint-10" ARGS="--direct"

make verify PREFIX="checkpoints/$RUN_NAME/checkpoint-20" ARGS="--direct"

You can also verify the run-level publication:

make verify PREFIX="checkpoints/$RUN_NAME" ARGS="--direct"

Each completed checkpoint has its own _SUCCESS.json record. Verification streams the files listed in the record and validates their sizes and SHA-256 hashes.

The sample avoids publishing a checkpoint while its files might still be changing. A checkpoint is uploaded only after a newer checkpoint exists or after the training process exits successfully. If training fails or is interrupted, the newest checkpoint is not published since it may be incomplete.

Use NVMe Storage for GR00T

Cloud GPU instances often provide a small root disk and a separate local NVMe volume.

To place the AIStore cache on NVMe storage, add the following setting to .env before running make up:

AIS_CACHE_DIR=/mnt/nvme/ais

You can also place the staged GR00T dataset on NVMe:

make run ARGS="--stage-dir /mnt/nvme/stage --max-steps 20 -- --global-batch-size 1 --save-steps 10 --save-total-limit 2"

Compare Dataset Staging Methods

The GR00T sample includes a benchmark that compares direct B2 staging with cold and warm AIStore staging:

make bench ARGS="--stagers 4 --repeats 3"

The benchmark compares:

• b2-direct — Reads the dataset directly from B2.

• ais-cold — Removes the selected cached objects and stages them through AIStore from B2.

• ais-warm — Populates the AIStore cache before staging.

The generated reports include staging time, throughput, latency, B2 GET counts, and bytes transferred.

Run the benchmark when the AIStore cluster is otherwise idle.

Curate Videos with AIStore and Backblaze B2

The AIStore B2 Cosmos Video Curation sample reads source videos stored in B2, caches them through AIStore, processes them using NVIDIA NeMo Curator, and writes the resulting clips and Cosmos-Embed1 embeddings back to B2.

The workflow performs the following operations:

1. Uploads a demonstration video dataset to B2.

2. Prefetches the input videos into AIStore.

3. Runs NeMo Curator on a GPU host.

4. Creates video clips and Cosmos-Embed1 embeddings.

5. Stores the generated output locally while the workflow runs.

6. Uploads the completed output through AIStore to B2.

7. Publishes a _SUCCESS.json record for the completed output.

8. Caption generation is not included in this sample.

Video Curation Requirements

In addition to the common requirements, the current video-curation sample requires:

• Linux x86-64.

• An NVIDIA GPU.

• An NVIDIA driver compatible with CUDA 13.

• CUDA 12 toolkit for the Curator native video extensions.

• FFmpeg and ffprobe 8.0 or later.

• The encoder and decoder used by your selected video configuration.

• For NVENC, the NVIDIA video-encoding runtime library that matches the installed driver.

Clone and Set Up the Video Curation Sample

Clone the repository:

git clone https://github.com/backblaze-b2-samples/aistore-b2-cosmos-video-curation.git

cd aistore-b2-cosmos-video-curation

Create and configure .env as described in Configure the B2 Connection.

Install the sample environment:

make setup

Start AIStore and attach the B2 bucket:

make up

Upload the Demonstration Videos

Upload the pinned demonstration dataset:

make seed

The sample copies 50 demonstration videos, approximately 60 MB in total, to:

raw/

Running make seed again skips files whose contents already match.

Install NeMo Curator

Display the workflow without processing any data:

make run ARGS="--dry-run"

Install the NeMo Curator GPU environment:

make setup-curator

The sample installs Curator in its own environment and verifies the pinned components used by the workflow.

Run the Video Curation Workflow

Run the curation workflow:

make run

For a full run, the sample prefetches the source videos into AIStore before Curator starts processing them. NeMo Curator creates clips and Cosmos-Embed1 embeddings locally. The sample then uploads the completed output through AIStore.

Output is written to a generated prefix similar to:

curated/<run-id>/

Record the run ID reported by the command.

If the Curator process fails or is interrupted, its local output is not published as a completed B2 result.

Verify Curated Output

Verify the output through AIStore:

make verify PREFIX=curated/<run-id>

To verify the durable copy directly in B2:

make verify PREFIX=curated/<run-id> ARGS="--direct"

The completed prefix contains a _SUCCESS.json record. Verification checks the size and SHA-256 value of each published object.

Use NVMe Storage for Video Curation

To place the AIStore cache on local NVMe storage, configure the following value in .env before starting AIStore:

AIS_CACHE_DIR=/mnt/nvme/ais

You can also place Curator's working files on NVMe storage:

make run ARGS="--work-dir /mnt/nvme/work --video-limit 10"

Curate Your Own Videos

Use a new prefix when you upload a new revision of a source dataset.

For example:

set -a

. ./.env

set +a

uv run --locked python scripts/seed_b2.py \

  --local-dir ./my_videos \

  --prefix raw/revision-001

Run Curator against the new prefix:

make run ARGS="--input-prefix raw/revision-001 --video-limit 5"

Avoid overwriting an existing dataset revision in place. AIStore may continue serving objects that are already present in its cache.

Compare Video Read Performance

The video-curation sample also includes a benchmark for comparing direct B2 reads with cold and warm AIStore reads:

make bench ARGS="--prefix raw/ --limit 50 --workers 16"

The generated reports include wall-clock time, throughput, latency, and measured B2 GET requests and bytes transferred.

The sample does not publish benchmark performance numbers. Run the benchmark on your own environment when the AIStore cluster is otherwise idle.

Use a Shared AIStore Cluster

The Docker Compose configuration is intended for local, single-host testing.

For Kubernetes environments or GPU platforms that do not provide a Docker daemon inside the workload, deploy AIStore separately. For example, you can deploy with ais-k8s and configure the B2 application key as the cluster’s AWS backend credentials, then configure the sample to use the remote AIStore proxy.

Set the AIS_ENDPOINT value in .env to the AIStore proxy address and attach the B2 bucket with:

make attach

The sample repositories include additional operational guidance for remote and shared AIStore deployments.

Keep the AIStore cluster private to the workload and use the appropriate AIStore authentication, network controls, and TLS configuration for shared environments.

Troubleshooting

AIStore cannot attach or list the B2 bucket

Check the following values in .env:

• B2_ENDPOINT

• B2_REGION

• B2_BUCKET

• B2_KEY_ID

• B2_APP_KEY

Also verify that the application key has the required bucket capabilities.

Port 51080 is already in use

Configure different AIStore proxy and target ports in .env, and update AIS_ENDPOINT to match.

GR00T seeding reports Git LFS pointer files

Install Git LFS and run:

make setup

again.

GR00T reports a CUDA toolkit mismatch

Set the CUDA 12.8 environment explicitly:

export CUDA_HOME=/usr/local/cuda-12.8

export PATH="$CUDA_HOME/bin:$PATH"

nvcc --version

Verify that nvcc reports CUDA 12.8.

GR00T returns a 401 or 403 from Hugging Face

Confirm that your Hugging Face account has access to nvidia/Cosmos-Reason2-2B and authenticate on the GPU host:

hf auth login

GR00T GetBatch returns fewer objects than expected

An object might have been evicted from AIStore after the prefetch operation.

Start a new run or use parallel S3 GET staging:

make run ARGS="--stage-method get --max-steps 20 -- --global-batch-size 1 --save-steps 10 --save-total-limit 2"

Stop AIStore

When you are finished with the local sample, stop the AIStore services:

make down

Data stored in B2 remains available.

For datasets that you plan to update, use a new immutable B2 prefix for each revision rather than overwriting an existing prefix. This prevents previously cached objects from being mistaken for the new dataset revision.

Additional Resources

• AIStore B2 GR00T Robotics Training repository

• Pinned NVIDIA Isaac GR00T release

• Cosmos-Reason2-2B model access

• NVIDIA AIStore

• Backblaze B2 S3-Compatible API

• Sample licenses and dataset attribution

• Related video curation sample

• Backblaze Labs