Cloud Replication

The Backblaze Cloud Replication feature enables the copying of data from a source bucket to a destination bucket through the Backblaze B2 Native API.

Backblaze's Cloud Replication feature provides for replicating one or many objects to one or multiple buckets within or between regions. This feature can help to serve multiple use cases, like ensuring data set copies are geographically distributed to support disaster recovery objectives, reducing concentration risk, serving compliance needs, or helping to move data closer to end users for faster access irrespective of edge infrastructure.

What Cloud Replication Provides

The Cloud Replication feature allows you to automate the following:

Key Notes

Key areas to keep in mind when using the Cloud Replication feature:

Note that the following are not replicated:

Lifecycle rules apply to replica files in the same manner as they apply to source files. When replicating files between buckets with different lifecycle rules, be aware of the effect that the different lifecycle rules will have on your files.

Example 1:

In this situation, you might have a file in the source bucket which is 15 days old. If you replicate that file to the destination bucket, the file will become hidden. Specifically, the file will become hidden the next time the system applies the lifecycle rules. Lifecycle rules are applied once per day.

Example 2:

In this situation, you might have a file in the source bucket which reaches an age of 11 days. Due to the source bucket's lifecycle rules, this file will become hidden after 10 days and then deleted on the 11th day. However, if you replicate that file to the destination bucket before it is deleted, the file will remain visible in the destination bucket until it is 30 days old, even though the original file was already deleted from the source bucket.

Creating a Cloud Replication Rule with the B2 Native API

Using the API

There are three steps to leveraging the Cloud Replication feature:

  1. Create the Cloud Replication Rule
  2. Add a new file or a new version of a file
  3. Check the replication status of the file by looking at the source and destination metadata

Note on Multiple Rules

Cloud Replication allows the creation of two rules each bucket. This means that each file can be used as a source in two rules. Adding any further rules will result in an error. Destination buckets can have more than two buckets as the holding bucket for many replicated sources.

1. Creating a Replication Rule

In common with all B2 Native API operations, you must first authenticate via b2_authorize_account. Use the following functions to manage Cloud Replication rules.

Using the APIs

There are three steps to leveraging the Cloud Replication feature:

To create a replication rule, include replicationConfiguration when you call b2_create_bucket or b2_update_bucket. At least one of asReplicationSource or asReplicationDestination is required, but they can also both be present.

For each replication, the user has to apply one rule for setting up source with asReplicationSource section and second rule for setting up the destination with asReplicationDestination section.

Example of a Source rule:

{
    "accountId": "12f634bf3cbz",
    "bucketId": "e1256f0973908bfc71ed0c1z",
    "replicationConfiguration": {
        "asReplicationSource": {
            "replicationRules": [
                {
                    "destinationBucketId": "3f46fe8276c62b414506021y",
                    "fileNamePrefix": "",
                    "includeExistingFiles": false,
                    "isEnabled": true,
                    "priority": 1,
                    "replicationRuleName": "replication-us-east"
                }
            ],
            "sourceApplicationKeyId": "00512f95cf4dcf0000000004z"
        },
    }
}

Example of a destination configuration, which is deployed on the destination account. The source of this bucket is set up by source bucket owner in source configuration, so destination only needs source key ID to map the two buckets:

{
	"accountId": "12f634bf3cbz",
	"bucketId": "e1256f0973908bfc71ed0c1z",
	"replicationConfiguration": {
		"asReplicationDestination": {
			"sourceToDestinationKeyMapping": {
				"00512f95cf4dcf0000000004z": "00512f95cf4dcf0000000004y",
			}
		}
	}
}

The source replication configuration is set within the asReplicationSource section, and the destination replication app key mapping is set within the asReplicationDestination section.

Cloud Replication rules created by the information above can then be turned off by calling b2_update_bucket setting the isEnabled parameter to false. To delete a rule, call b2_list_buckets, if necessary, to obtain the bucket's list of replication rules, remove the rule from the replicationRules JSON array, and call b2_update_bucket, supplying the modified replicationRules. Note, the API layer allows the rules to be turned off or deleted even when created from the Web UI.

When creating a replication rule, you must provide an Authentication Key that has read-access for the source bucket and write-access for the destination bucket. Also, keep in mind that failures may occur due to billing issues. Make sure the source and destination are set up to accommodate the fees associated with Cloud Replication. Contact support if you run into any issues.

Call b2_list_buckets to retrieve all of the replication rules associated with one or more buckets. Please keep in mind that deleted Replication Rules will not be returned.

Each replication rule has the following fields:

Note on Authentication Keys:

To successfully create a working replication rule, the authentication keys must have specific rights.

For Source buckets, the Authentication Key specified must have the following rights:

For Destination buckets, the Authentication Key specified must have the following rights:

Note on File Name Prefix:

The fileNamePrefix parameter allows API calls to specify a prefix to restrict replication to files with matching names, this feature is not available in the Web User Interface.

2. Adding a New File (or Version)

There is no difference to the process of uploading files to bucket(s) with Cloud Replication rules. But keep the following in mind:

3. Check Replication Status

File Info Metadata

To see the replication status and if the file is a replica itself, call b2_get_file_info. The field replicationStatus provides the given file's replication status. Each file supports two rules that can be defined.

The replicationStatus may have one of the following values:

Existing File Replication

When a replication rule is configured to include existing files, the replica files are created with the same upload timestamp as their source file. This has several ramifications:

Account Setup

Cross-Regional Replication and Authenticated Accounts

To create a Replication Rule that copies from the US East to US West, you must connect two separate accounts. You will need to login into the accounts separately to manage the files. When you set up a Cloud Replication Rule in the Web Console for the source account, you can authenticate to an account in the destination region and select a destination bucket.