Lifecycle Rules

Lifecycle Rules enable the Backblaze B2 Cloud Storage service to automatically hide and delete older versions of files that are stored in Backblaze B2. This capability is available for the Backblaze web console, the command-line interface (CLI), and the API. You can apply Lifecycle Rules to some or all of the files in a Backblaze B2 bucket, and each bucket has its own Lifecycle Rules. For example, you can set up rules to delete old versions of files 30 days after you upload a newer version.

You can set up to 100 Lifecycle Rules on one bucket.

Lifecycle Rules API Operations

The following API calls are related to Lifecycle Rules:

• b2_create_bucket - This operation creates a new bucket.
• b2_update_bucket - This operation changes the settings on a bucket.
• b2_list_buckets - This operation lists all of the buckets in your account, including a list of Lifecycle Rules for each bucket.
• b2_delete_bucket - This operation deletes a specified bucket and returns a list of Lifecycle Rules for the bucket being deleted.

When you create a new bucket with b2_create_bucket or update an existing bucket with b2_update_bucket, you can specify Lifecycle Rules using the optional lifecycleRules parameter. This parameter takes a JSON array where you specify up to four properties that determine the Lifecycle Rules for the bucket:

• fileNamePrefix (required)
• daysFromHidingToDeleting (optional)
• daysFromUploadingToHiding (optional)
• daysFromStartingToCancelingUnfinishedLargeFiles (optional)

You can set optional values to null so that part of the rule is not applied. You cannot set all of the optional values to null because then the rule does nothing. Unspecified properties are treated as having a null value.

You cannot set daysFromHidingToDeleting, daysFromUploadingToHiding, or daysFromStartingToCancelingUnfinishedLargeFiles to zero. When you set a value for these properties, you must specify a positive number.

The following example shows a JSON array that specifies all four properties in a Lifecycle Rule:

[
  {
     "daysFromHidingToDeleting": 30,
     "daysFromUploadingToHiding": 7,
     "daysFromStartingToCancelingUnfinishedLargeFiles": 3,
     "fileNamePrefix": "backup/"
  }
]

fileNamePrefix

This property specifies the files in the bucket to which the Lifecycle Rule applies. For example, a value of working/ applies to all of the files in the folder named "working." Any file name that begins with this prefix is subject to the rule. A prefix of an empty string ("") means that the rule applies to all of the files in the bucket. When you set a custom rule through the Web UI that applies to all files in a bucket, the fileNamePrefix must be left blank.

Overlapping Prefixes

You cannot use multiple rules on a bucket if the rules have identical fileNamePrefix properties. For example, you cannot have two rules on a bucket that both use the prefix "". Likewise, you cannot have two rules on a bucket that both use the prefix media/.

It is possible to have multiple rules on a bucket that use different prefixes. This allows you to specify a set of files with one rule and specify a subset of those files with another rule. For example, you can have one rule that has the prefix "" and another rule that has the prefix review/. In that case, the prefix review/ is considered to overlap with the prefix "". This is because the prefix "" specifies "all of the files in the bucket" and the prefix review/ specifies "all of the files in the bucket's review folder."

If you use multiple rules on the same bucket and the prefixes on those rules overlap, then you must fulfill the following conditions:

1. If you specify a non-null value for the daysFromStartingToCancelingUnfinishedLargeFiles property in one rule, then you must leave the daysFromStartingToCancelingUnfinishedLargeFiles property unspecified or be null in all of the other rules.
2. If you specify a non-null value for either the property daysFromHidingToDeleting or the property daysFromUploadingToHiding in one rule, then in all other rules you must leave the properties daysFromHidingToDeleting and daysFromUploadingToHiding unspecified or null. You cannot set all three daysFrom parameters to unspecific or null.

The following examples show how multiple rules are used on the same bucket. Both valid and invalid examples are provided.

Valid Example A

First Rule:

"daysFromStartingToCancelingUnfinishedLargeFiles": 2,
"daysFromHidingToDeleting": 1,
"fileNamePrefix": "logs/"

Second Rule:

"daysFromUploadingToHiding": 1,
"fileNamePrefix": "videos/"

These two rules can work together on the same bucket because the prefixes are different, and they do not overlap.

Valid Example B

First Rule:

"daysFromStartingToCancelingUnfinishedLargeFiles": 2,
"fileNamePrefix": ""

Second Rule:

"daysFromHidingToDeleting": 1,
"daysFromUploadingToHiding": null,
"fileNamePrefix": "logs/"

Although these two prefixes are different, the prefix logs/ overlaps with the prefix "". But there is no conflict because while the first rule uses a non-null value for daysFromStartingToCancelingUnfinishedLargeFiles, the second rule leaves that value unspecified. And while the second rule uses a non-null value for daysFromHidingToDeleting, the first rule leaves both daysFromHidingToDeleting and daysFromUploadingToHiding unspecified.

Valid Example C

First Rule:

"daysFromStartingToCancelingUnfinishedLargeFiles": 2,
"daysFromHidingToDeleting": null,
"daysFromUploadingToHiding": null,
"fileNamePrefix": "files/"

Second Rule:

"daysFromStartingToCancelingUnfinishedLargeFiles": null,
"daysFromHidingToDeleting": 2,
"daysFromUploadingToHiding": 1,
"fileNamePrefix": "files/edited/"

This is a more complicated example that demonstrates the valid use of multiple rules on the same bucket. The prefix files/edited/ overlaps with the prefix files/. There is no conflict with daysFromStartingToCancelingUnfinishedLargeFiles because while the first rule assigns a non-null value to that property, the second rule gives the property a null value. And although the second rule uses non-null values for both daysFromHidingToDeleting and daysFromUploadingToHiding, the other rule uses null for both of those properties.

Invalid Example A

First Rule:

"daysFromStartingToCancelingUnfinishedLargeFiles": 2,
"fileNamePrefix": ""

Second Rule:

"daysFromHidingToDeleting": 1,
"fileNamePrefix": ""

Third Rule:

"daysFromUploadingToHiding": 1,
"fileNamePrefix": ""

These three Lifecycle Rules cannot be used together because they have identical prefixes.

Invalid Example B

First Rule:

"daysFromStartingToCancelingUnfinishedLargeFiles": 2,
"fileNamePrefix": ""

Second Rule:

"daysFromHidingToDeleting": 1,
"daysFromUploadingToHiding": 1,
"daysFromStartingToCancelingUnfinishedLargeFiles": 3,
"fileNamePrefix": "logs/"

Here, the prefixes are different but logs/ overlaps with "". There is a conflict because both rules specify a non-null value for daysFromStartingToCancelingUnfinishedLargeFiles.

Invalid Example C

First Rule:

"daysFromStartingToCancelingUnfinishedLargeFiles": 2,
"daysFromHidingToDeleting": 1,
"daysFromUploadingToHiding": null,
"fileNamePrefix": "files/"

Second Rule:

"daysFromStartingToCancelingUnfinishedLargeFiles": null,
"daysFromHidingToDeleting": null,
"daysFromUploadingToHiding": 1,
"fileNamePrefix": "files/edited/"

This is a more complicated example of a conflict. The prefix files/edited/ overlaps with the prefix files/. Although daysFromStartingToCancelingUnfinishedLargeFiles uses a non-null value in the first rule, the second rule assigns null to that property, so those properties are not in conflict. The conflict occurs due to daysFromHidingToDeleting having a non-null value in the first rule while the second rule uses a non-null value with daysFromUploadingToHiding. This example can be made valid by changing the second rule to assign null to daysFromUploadingToHiding.

daysFromHidingToDeleting

Files are automatically hidden when they are replaced with a new version of the file, or they can be explicitly hidden using b2_hide_file. The daysFromHidingToDeleting property causes hidden files that you specify to be automatically deleted after a number of days. Because files are automatically hidden when replaced by a newer version, if you set this property to 10, then the older version is deleted 10 days after you upload a newer version of the file.

Valid values are null or numbers one and greater. Null means that no files are deleted based on this rule.

If you back up your files to Backblaze B2 using software that uploads files when they change, Backblaze B2 retains old versions of the file. This is helpful because the old versions can be used if the original file is accidentally deleted. On the other hand, keeping old versions forever can clutter things. For an application like this, you may want a Lifecycle Rule that keeps old versions in a backup/ folder for 30 days and then deletes them, as in the following example:

[
  {
     "daysFromHidingToDeleting": 30,
     "daysFromUploadingToHiding": null,
     "daysFromStartingToCancelingUnfinishedLargeFiles": null,
     "fileNamePrefix": "backup/"
  }
]

daysFromUploadingToHiding

This value causes the specified files to be automatically hidden after a designated number of days. This applies to all of the copies of the file, even the most current version. For example, setting this property to 30 hides all of the files in the specified folder 30 days after you upload the files. This includes the most current version of the file. Use this option only if you want to hide all of the versions of the specified files.

Valid values are null or numbers one and greater. Null means that no files are hidden based on this rule.

This setting can be useful for things like server log files that should be deleted after a certain time period. The following example shows a rule that keeps files in the logs/ folder for seven days and then hides them. They are then deleted one day after hiding:

[
  {
     "daysFromHidingToDeleting": 1,
     "daysFromUploadingToHiding": 7,
     "daysFromStartingToCancelingUnfinishedLargeFiles": null,
     "fileNamePrefix": "logs/"
  }
]

daysFromStartingToCancelingUnfinishedLargeFiles

This setting cancels any unfinished large file versions after a given number of days. This can be useful for an automatic clean-up of any interrupted large file uploads. Using this Lifecycle Rule is similar to calling b2_cancel_large_file on the unfinished large files.

During Lifecycle Rule processing, unfinished large files cannot hide other files, or be hidden by other files.

File Versions

By default, Backblaze B2 retains all of the files that you upload and all of the different versions of the files that you upload. While Backblaze B2 retains the different versions, it "hides" the older versions to keep the file list easier to view, as shown in the following example:

There are two indicators that there are multiple versions of the same file.

• The file name has a (2) at the end. This is the file version count and, in this case, it means that there are two versions of the file. The date/time stamp that is shown is for the most recent version of the file. When you click the file version count (the 2), the list expands to show all of the file versions.
• The "missing" file information icon is displayed.

The older version, V2 in the example, is hidden. All of the versions of a given file are available to download, delete, or keep. If the file names are different, they are stored as different files, so myfile-version1.doc is not the same as myfile-version2.doc, even if the content is identical.

There is no limit to the number of versions you can have of a given file. By default, older versions are kept forever, unless you delete them or you stop using the Backblaze B2 service. Since all of the versions of a given file are kept, you can use Lifecycle Rules to help manage the older versions. Every Lifecycle Rule includes an implicit rule: when the oldest version of a file is a hide marker, the marker is deleted independent of its age.

The file version rules apply to all of the files in the bucket, unless otherwise specified. The four options for file versions work as follows:

• Keep all versions of the file (default)
  This option removes all of the Lifecycle Rules from the bucket and keeps all of the versions of a given file until you explicitly delete them.
• Keep only the last version of the file
  This rule keeps only the most current version of a file. The previous version of the file is "hidden" for one day and then deleted.

  [
     {
     "daysFromHidingToDeleting": 1,
     "daysFromUploadingToHiding": null,
     "fileNamePrefix": ""
     }
  ]

• Keep prior versions for this number of days
  This rule deletes the older versions of a file after a specified number of days. The number of days can be one or greater. The most current version of a file is always kept unless it is explicitly deleted.

  [
    {
     "daysFromHidingToDeleting": 30,
     "daysFromUploadingToHiding": null,
     "fileNamePrefix": ""
    }
  ]

• Use custom lifecycle rules
  This rule allows you to construct a rule that applies to one or more files using the following properties:
  • fileNamePrefix
  • daysFromUploadingtoHiding
  • daysFromHidingtoDeleting
  • daysFromStartingToCancelingUnfinishedLargeFiles

Lifecycle Rules in the Command-Line Tool

To use Lifecycle Rules with the Backblaze B2 CLI, you need to run version 0.7.0 of the Backblaze B2 command-line tool. Use the following commands to create and manage Lifecycle Rules in the CLI.

b2 create-bucket [–bucketInfo <json>] [–lifecycleRules <json>] <bucketName> [allPublic | allPrivate]

b2 update-bucket [–bucketInfo <json>] [–lifecycleRules <json>] <bucketName> [allPublic | allPrivate]

b2 get-bucket <bucketName>

The b2 get-bucket command returns all of the properties for the named bucket.

Lifecycle Rules and Replica Files

Lifecycle Rules apply to replica files in the same manner as they apply to source files. When you replicate files between buckets with different Lifecycle Rules, the different rules may affect your files.

Example 1:

• You have a source bucket with Lifecycle Rules that hides files after 30 days and then deletes them five days later.
• You have a destination bucket with Lifecycle Rules that hides files after 10 days and then deletes them one day later.

In this situation, you might have a file in the source bucket that is 15 days old. If you replicate that file to the destination bucket, the file becomes hidden. Specifically, the file becomes hidden the next time the system applies the rule. Lifecycle Rules are applied once per day.

Example 2:

• You have a source bucket with Lifecycle Rules that hides files after 10 days and then deletes them one day later.
• You have a destination bucket with Lifecycle Rules that hides files after 30 days and then deletes them five days later.

In this situation, you might have a file in the source bucket that reaches an age of 11 days. Due to the source bucket's Lifecycle Rules, this file becomes hidden after 10 days and is then deleted on the 11th day. However, if you replicate that file to the destination bucket before it is deleted, the file remains visible in the destination bucket until it is 30 days old, even though the original file was already deleted from the source bucket.

Setting Times for Lifecycle Rules

Lifecycle Rules are applied once per day. The fewest number of days that you can set for a rule is one. Rules are applied at the next daily run after that number of days has passed.

For example, if you upload the first version of file.txt on May 2, and then you upload a new version on May 9 at 06:00, there are now two versions: the May 2 version and the May 9 version. The May 2 version is hidden by the May 9 version, so it became hidden at the time the May 9 version was uploaded.

If there is a Lifecycle Rule that deletes files one day after they are hidden, and the Lifecycle Rules are applied at 01:00 on May 10, then nothing happens yet. It has been 19 hours since the May 2 version was hidden. Because a full day has not yet passed, nothing happens.

The next day, on May 11, when the Lifecycle Rule is applied, something does happen. If it runs at 01:00, it has now been 43 hours since the May 2 version was hidden, so the rule applies and the old version is deleted.

Lifecycle Rules are based on the upload timestamp of the file. When migrating from other cloud providers, you can maintain existing Lifecycle Rules by customizing the upload timestamp to set the timestamp in the past. For more information about custom upload timestamps, see b2_upload_file and b2_start_large_file.

Object Lock and Lifecycle Rules

Lifecycle Rules potentially delete specific versions of files. However, if the file version to which the rule applies has Object Lock enabled, then the deletion does not occur.

Existing File Replication

When a replication rule is configured to include existing files, the replica files are created with the same upload timestamp as their source file. This has several ramifications:

• Backblaze does not guarantee any ordering of individual file version replications. While replication of existing files is ongoing, it is possible that a filename-based query will have different results between the source and destination buckets. This situation is temporary, and both buckets provide the same response after the existing files are finished replicating. Consider disabling Lifecycle Rules in the destination bucket if this will be an issue.
• Existing files are replicated to the destination bucket regardless of the destination bucket's Lifecycle Rule configuration. It is possible that some newly created file replicas will be hidden or deleted when Lifecycle Rules are next applied (typically every 24 hours).
• Replica files are created with the same upload timestamp as their source file. If the destination bucket has a default Object Lock retention period set, then the retention period is calculated based on the timestamp that was inherited from the source file.

Integrations and Lifecycle Rules

Application developers are encouraged to look at Lifecycle Rules to use in their applications that integrate into Backblaze B2. This is especially true for applications that perform "sync" or "backup" operations to Backblaze B2. For example, the user interface of the application can allow the user/admin to specify the number of days that older versions of files should be kept in support of data retention rules in an organization.

Lifecycle Rule Examples

If you back up your log files to Backblaze B2 once per day and these files are stored in a folder named "log," then you can set up your Lifecycle Rules to manage the log files based on your data retention policies.

Example 1:

fileNamePrefix: log/
daysFromUploadingtoHiding: 7
daysFromHidingtoDeleting: null

Result:

The files in the log/ folder are hidden seven days after you upload them to Backblaze B2. This means that all of the versions of every file in the log/ folder are hidden after seven days. The hidden files are still available for download using their specific file_id and continue to count as part of your Backblaze B2 storage amount.

Example 2:

fileNamePrefix: log/
daysFromUploadingtoHiding: 7
daysFromHidingtoDeleting: 30

Result:

The files in the log/ folder are hidden seven days after you upload them to Backblaze B2. After the files are hidden for 30 days, they are permanently deleted. Until they are deleted, the files are still available for download and count as part of your Backblaze B2 storage amount.

Example 3:

fileNamePrefix: log/
daysFromUploadingtoHiding: null
daysFromHidingtoDeleting: 180

Result:

The files in the log/ folder are deleted 180 days after they have been either explicitly hidden or they have been replaced by a file of the same name. Until the files are deleted, all of the versions of a file are available for download and count as part of your Backblaze B2 storage amount.