The Backblaze Cloud Replication feature provides for replicating one or many objects to one or multiple buckets within or between regions. This feature can help to serve multiple use cases, like ensuring data set copies are geographically distributed to support disaster recovery objectives, reducing concentration risk, serving compliance needs, or helping to move data closer to end users for faster access irrespective of edge infrastructure.
Cloud Replication lets you copy data from a source bucket to a destination bucket using the Backblaze web UI or the Native API. The S3-Compatible API does not include Cloud Replication at this time. However, after you create a replication rule, it applies to all files regardless of whether they are created using the Native API or the S3-Compatible API.
You can use Cloud Replication to automate the following processes:
- Configuring a rule to replicate data between a source bucket and destination bucket.
- Setting automatic retries for any transient errors that are encountered during replication.
- Adding a replication status field to every source file and destination file.
To create a Cloud Replication rule using the Backblaze web UI, click here.
Replication times can vary based on the number of files you replicate. Initial replication of a large data set usually takes longer than a subsequent replication. For subsequently uploaded files, replication begins when you upload a new file to a replication-enabled bucket. For existing files, replication occurs each day at midnight UTC. If the replication fails with a non-fatal error, the replication retries after midnight UTC the following day.
You can check the status of your replication rule from the source account on a file-by-file basis. For more information, see Check the Status of a Cloud Replication Rule.
Replicating Data to a Different Region
Backblaze B2 Cloud Storage was designed for an account to tie to a single region. Therefore, to replicate data to a different region, you must create an additional account for that different region. Setting up this additional account is quick and easy, and it does not require that you speak with a Sales or Support resource.
Metadata on newly uploaded objects is replicated at the time of replication. However, subsequent changes to the metadata in the source bucket are not replicated to the destination bucket.
Deleting and Pausing Replication Rules
When you create a replication rule, it automatically replicates per the logic of the rule. To stop the replication, you must pause or delete the rule.
On the Cloud Replication page, the options to either “Pause” or “Delete” the rule are displayed for the rule you created.
Pausing the rule stops the replication while retaining the rule that you created. Pausing the rule also allows you to resume replicating files at any time from the point at which you pause the rule without your files requiring re-evaluation.
Deleting the rule also stops the replication rule and is a good option if you no longer intend to replicate to the destination bucket. If you delete the rule, you must recreate the entire rule again.
Managing Backblaze Accounts
You can create a Group and enable Backblaze B2 for the group. Invite your source account and your destination account to this group. Create a replication rule in the source account to replicate the objects that are in that account’s bucket to a bucket in the destination account. Billing for storage and egress fees for downloaded files are charged to the group. There are no egress fees for data replication. For more information about Groups, see Create and Manage Groups.
Lifecycle Rules for Replicated Files
Lifecycle Rules apply to replica files in the same manner as they apply to source files. When you replicate files between buckets with different lifecycle rules, be aware of the following effects that the different lifecycle rules have on your files.
- You have a source bucket with Lifecycle Rules that hides files after 30 days and deletes them five days later.
- You have a destination bucket with Lifecycle Rules that hides files after 10 days and deletes them one day later.
In this situation, you may have a file in the source bucket that is 15 days old. If you replicate that file to the destination bucket, the file becomes hidden. Specifically, the file becomes hidden the next time the system applies the Lifecycle Rules. Lifecycle Rules are applied once per day.
- You have a source bucket with Lifecycle Rules that hides files after 10 days and deletes them one day later.
- You have a destination bucket with Lifecycle Rules that hides files after 30 days and deletes them five days later.
In this situation, you may have a file in the source bucket that reaches an age of 11 days. Due to the source bucket's Lifecycle Rules, this file becomes hidden after 10 days and deleted on the 11th day. However, if you replicate that file to the destination bucket before it is deleted, the file remains visible in the destination bucket until it is 30 days old even though the original file was deleted from the source bucket.
Existing File Replication
When a replication rule is configured to include existing files, the replica files are created with the same upload timestamp as their source file.
- Backblaze does not guarantee any ordering of individual file version replications. While the replication of existing files is ongoing, it is possible that a filename-based query will have different results between the source and destination buckets. This situation is temporary, and both buckets will provide the same response after the existing files have finished replicating. Consider disabling Lifecycle Rules in the destination bucket if this is an issue.
- Existing files are replicated to the destination bucket regardless of the configuration of the destination bucket's Lifecycle Rules. It is possible that some newly created file replicas will be hidden or deleted when Lifecycle Rules are next applied (typically every 24 hours).
- Replica files are created with the same upload timestamp as their source file. If the destination bucket has a default Object Lock retention period set, the retention period is calculated based on the timestamp that is inherited from the source file.
Cloud Replication is not built to serve every use case. The following list provides cases in which the feature is not recommended or is limited to specific scenarios:
- Setting manual data caps is not recommended.
This can cause unexpected results. If a data cap is set below the 10 GB limit, authenticating to this account does not return any buckets. In this instance, the replication rule should be deleted and the data cap removed. You will then need to recreate the exact replication rule.
- Replication chaining is not possible.
All destination files are seen as replicants and cannot be used for new replications.
- Multiple replication rules
Only two rules can be set per bucket for replication.
- File IDs are not replicated.
Therefore, Veeam backup replication is not currently supported.
- Size limitations
Files with metadata greater than 2,048 bytes are not replicated.
- Deleted files
Deleted files are not replicated.
- SSE-C or encryption managed by the client is not supported.
Backblaze supports encryption (which you must set up on a bucket at the time of creation). SSE-C, or encryption managed by the client, is not supported.
- Metadata changes are not replicated.
If you need to change a file’s metadata, but you do not change the body, the metadata change on the replicated file is ignored.
- Existing vs new data replication
In the API, you can enable the replication of all existing files in a bucket. If this is not enabled, the bucket is set up with New Data replication, where only the new files that were uploaded after the rule was set up are replicated.
For rules that you configure in the Backblaze web UI, all rules are set to existing data, meaning the parameter is set to 'true' as part of the automatic settings that are configured.
- The following items are not replicated:
- Replica Files: Any files that are replicated cannot be set as the source of a new replication.
- SSE-C files: Encrypted files that are managed by the user are not replicated.
- Hide Markers: Hide markers are not replicated. This means that the files that are hidden in a source are not hidden in the destination. Hide markers can be applied to the destination after replication completes.
One of the following Cloud Replication statuses is displayed after you create and run a replication rule:
This status indicates that the file was successfully replicated to the destination bucket. For multiple rules, this means both rules completed successfully.
This status indicates that this file is in the process of being replicated or is waiting to be replicated. For multiple rules, this means at least one rule is in process.
This status indicates that the file encountered an error that cannot be resolved. Files with this status are not attempted again. The only way to attempt it again is to create a new replication rule. For multiple rules, this status means at least one rule failed. You must check the destination bucket to understand which rule failed.
For files that are the result of a replication, the replication status returns “replica.” Replica files cannot be used for more replication rules.
A failed state is a non-recoverable state of the file. This state is not expected to be resolved, and this file will not be re-evaluated in its current state.
The following causes are possible:
Ensure that the permissions that you created for the replication rule are still present. The source bucket must have read access for the authentication key and write access for the destination authentication key. The keys are created when the Backblaze web UI completes successfully. If you remove or change the properties of how these are set up, it may cause failures.
- Object Lock and Encryption
If you enabled Object Lock on the source bucket, you must also enable Object Lock on the destination bucket. Similarly, SSE-C encryption, managed by the client, is not supported. These may also return failed states.
- Part Files
Cloud Replication for large files is done only after the large files are finalized (following a call to b2_finish_large_file). After all of the parts of the file are uploaded, all of the part files are queued for replication. Until the final b2_finish_large_file is run, the part files are ignored by the replication process.
If you believe that your file is a valid file and you want to retry it, the best way is to upload the file again, creating a new version. This triggers the file to be re-evaluated.
You should keep the following key areas in mind when you use the Cloud Replication feature:
- Backblaze does not charge service or egress fees for replication. Standard Backblaze B2 rates apply to replicated data storage.
- Backblaze data storage costs do not differ from one region to another, and you do not incur any service or egress fees. Standard Backblaze B2 rates apply to replicated data storage.
- Backblaze B2 was designed for an account to tie to a single region. Therefore, to replicate data to a different region, you must create an additional account for that different region. Setting up this additional account is quick and easy, and does not require that you speak with a Sales or Support resource.
- When you configure a replication source, you must provide an app key that has read access to the bucket. Similarly, for destination buckets, you must provide an app key that has write access to the destination bucket.
- If you set the replication configuration for a source bucket that references a destination bucket that does not exist, the configuration attempt will fail.
- Ensure that both the source account and the destination account are configured correctly for billing. If they are not, you will be unable to save the related Cloud Replication rule.
- You cannot use a master key as the app key for Cloud Replication.
- If you set Object Lock on a source bucket, you must also enable Object Lock on the destination bucket or the replication returns a failed status.
- To centrally manage your Backblaze B2 accounts, you can create a Group and enable Backblaze B2 for the group. Invite your source account and your destination account to this group. Create a replication rule in the source account to replicate the objects that are in that account’s bucket to a bucket in the destination account. Billing for storage and egress fees for downloaded files are charged to the group. There are no egress fees for data replication. For more information about Groups, see Create and Manage Groups.NoteGroups apply only to consumption-based accounts. The Groups feature is currently not supported for B2 Reserve accounts.
- Replication times can vary based on the number of files you replicate. Initial replication of a large data set usually takes longer than a subsequent replication. For subsequently uploaded files, replication begins when you upload a new file to a replication-enabled bucket. For existing files, replication occurs each day at midnight UTC. If the replication fails with a non-fatal error, the replication retries after midnight UTC the following day.
- Currently, you can apply only two rules for the same bucket. If you create a third rule on the same bucket, the rule fails when you attempt to create the bucket.
- You can check the status of your replication rule only on the source account and only on a file-by-file basis. For more information, see Create and Manage Cloud Replication Rules.
For assistance with Cloud Replication, contact the Support team.