Introducing Backblaze B2 Lifecycle Rules

January 10th, 2017

Lifecycle Rules enable the B2 service to automatically hide and/or delete older versions of files stored in B2. This capability is available for the Web GUI, CLI, and API. Lifecycle Rules are applied to some or all of the files in a B2 Bucket with each Bucket having its own Lifecycle Rules. Let’s take a look at how you can use this capability to help manage your B2 storage space.

Overview

By default, Backblaze B2 will keep all the files you upload and all of the different versions of a file you upload. While B2 will keep the different versions, it will “hide” older versions to keep the file list easier to view as shown in the B2 file list (from the Web GUI) below.

There are two indicators that we have multiple versions of the same file. First the file name has a (2) at the end. This is the file version count and in this case means there are two (2) versions of the file. The date/time stamp shown is for the most recent version of the file. The second indicator is the “missing” file information icon on the right.

When you click on the file version count (i.e. the 2), the list will expand to show all of the file versions.

The older version, V2 in the example, is hidden. All versions of a given file are available to download, delete or just keep around as desired. If the file names are different at all, they are stored as different files, so myfile-version1.doc is not the same as myfile-version2.doc, even if the content is identical.

There is no limit to the number of versions you can have of a given file. Also, by default, the length of time older versions stay in B2 is forever, unless you delete them or you stop using the B2 service.

Since all versions of a given file are kept around you can use Lifecycle Rules to help manage the older versions. Let’s see how.

Lifecycle Rules and the Web GUI

To use Lifecycle Rules you will to have an active Backblaze B2 account and at least one B2 Bucket. It would help to have a few files in your B2 account as well. After logging into your B2 account and clicking “Buckets” on the B2 Cloud Storage menu, you’ll see a list of your buckets. For the “Bucket” of interest select the “Bucket Settings” link as shown in the example below.

The “Bucket Settings” pop-up screen, similar to the one below will appear.


Previously, this screen just contained the settings for making a bucket Public or Private. The additional items are for managing Lifecycle Rules.

Let’s cover the more common ways to use Lifecycle Rules.

File Versions

The file version rules apply to all the files in the bucket, unless otherwise specified. The four options for File Versions work as follows:

  • Keep All Versions (Default): Keeps all versions of a given file.
  • Keep Only Last Version: This keeps only the most current version of a file. The previous version of the file is “hidden” for one day and then deleted.
  • Keep Prior Versions for this number of Days: Older versions of a file are deleted after the specified number of days. The number of days can be 1 or greater. The most current version of a file is always kept unless it is explicitly deleted.
  • Lifecycle Rules Below – Allows you to construct a rule that applies to one or more files. The three parameters are covered in the next section.

Most common situations can be handled using one of the first three options.

Constructing a Lifecycle Rule in the Web GUI

Before you start constructing Lifecycle Rules, here are a few things to know:

  1. You are not allowed to create two or more rules that apply to the same file or files.
  2. Remember that the old version of a file is automatically hidden when a new version of the same name is uploaded.
  3. Read the “file versions” documentation to understand how file versioning works in B2.
  4. Only use this option if you know what you are doing. You are giving B2 the ability to hide and then permanently delete your files. Make sure this is something you want to do. Once deleted, files can not be restored by Backblaze. Please be careful.

There are three variables you need to enter to construct a Lifecycle Rule. All three are required for each rule added.

  • fileNamePrefix: This specifies which files in the B2 Bucket the rules applies to. For example, a value of “workinprocess/” will apply to all files in the “workinprocess” directory/folder. A value of “” (open quote, close quote, no space) means the rule applies to all the files in the bucket.
  • daysFromUploadingtoHiding: Setting this value causes the specified files to be hidden automatically after the specified number of days. This applies to all copies of the file, even the most current version. For example, setting this variable to 30 will hide all of the files in the specified directory/folder 30 days after they have been uploaded – this includes the most current version of the file. Only use this option if you want to hide all versions of the files specified. Valid values are “null” or numbers 1 and greater. Null, means no files are hidden based on this rule.
  • daysFromHidingtoDeleting: Setting this value causes the specified hidden files to be deleted automatically after the specified number of days. Remember that files are automatically hidden when they are replaced with a newer version. For example, if you set this variable to 10, then 10 days after a newer version of the file is uploaded, the older version will be deleted. Valid values are “null” or numbers 1 and greater. Null, means no files are deleted based on this rule.

Lifecycle Rule Examples

Let’s assume that you back up your “log” files once a day to B2 and that these files are stored in a directory/folder named log. Below are sample Lifecycle Rules you could set-up to manage the log files in B2 based on your data retention policies.

Example 1:

    fileNamePrefix: log/
    daysFromUploadingtoHiding: 7
    daysFromHidingtoDeleting: null

Result: The files in the log/ folder will be hidden 7 days after they are uploaded to B2. This means the current version, as well as any previous versions of every file in the log/ folder, will be hidden after 7 days. The hidden files are still available for download using their specific file_id and continue to count as part of your B2 storage amount.

Example 2:

    fileNamePrefix: log/
    daysFromUploadingtoHiding: 7
    daysFromHidingtoDeleting: 30

Result: The files in the log/ folder will be hidden 7 days after they are uploaded to B2. After the files have been hidden for 30 days, they will be permanently deleted. Until they are deleted, the files are still available for download and count as part of your B2 storage amount.

Example 3:

    fileNamePrefix: log/
    daysFromUploadingtoHiding: null
    daysFromHidingtoDeleting: 180

Result: The files in the log/ folder will be deleted 180 days after they have been either 1) explicitly hidden, or 2) been replaced by a file of the same name (i.e. they are an older version of the file – remember older file versions are automatically hidden). Until the files are deleted, all versions of a file are available for download and count as part of your B2 storage amount.

Lifecycle Rules for the CLI and API

Both the B2 CLI and B2 API can utilize Lifecycle Rules.

For the B2 API, the following calls work with Lifecycle Rules:

  • b2_create_bucket
  • b2_update_bucket
  • b2_list_buckets

Each API call has an optional parameter named lifecycleRules that requires the three variables noted above in the following JSON format:
{
    “daysFromHidingToDeleting”: 30,
    “daysFromUploadingToHiding”: null,
    “fileNamePrefix”: “backup/”
}

In addition, b2_delete_bucket will return the list of lifecycleRules for the bucket being deleted.

B2 Integrations and Lifecycle Rules

Application developers are encouraged to look at Lifecycle Rules to use in their applications that integrate into B2. This is especially true for applications that perform Sync or Backup operations to B2. For example, the user interface of the application could allow the user/admin to specify the number of days older versions of files should be kept in support of data retention rules in an organization.

B2 Command Line Tool

The B2 Command Line Tool provides command line access to the B2 service. To use Lifecycle Rules with the B2 CLI, you will need to be running version 0.7.0 of the B2 Command Line Tool. To upgrade to this version you can follow the “Get Command-Line Tool” instructions on the Backblaze website. The following commands utilize Lifecycle Rules in the CLI.

  • b2 create-bucket [–bucketInfo <json>] [–lifecycleRules <json>] <bucketName> [allPublic | allPrivate]
  • b2 update-bucket [–bucketInfo <json>] [–lifecycleRules <json>] <bucketName> [allPublic | allPrivate]
  • b2 get-bucket <bucketName>

The “b2 get-bucket” command will return all the variables for the named bucket.

Summary

Lifecycle Rules are very handy in helping manage multiple versions of files you are storing in B2. These rules can help save storage space by automatically deleting older, unneeded versions of files and can make the B2 file list easier to navigate and manage. Since Lifecycle Rules will delete older file versions, one recommendation is to start with a large number of days for “daysFromHidingtoDeleteing” and reduce this number as you become comfortable with this feature. The easiest way to do this is to use the “Keep Prior Versions for this number of days:” option in the WebGUI.

There is more information on Lifecycle Rules in the B2 documentation section of the Backblaze website. The website also has code examples for the API calls, information in the B2 Command Line tool and more. Let us know if you find anything there we need to update or you feel is missing.

[ED-Updated on 1/11/2016 with reference to file version documentation along with enhancing specifics regarding hidden files].

Andy Klein

Andy Klein

Andy has 20+ years experience in technology marketing. He has shared his expertise in computer security and data backup at the Federal Trade Commission, Rootstech, RSA and over 100 other events. His current passion is to get everyone to back up their data before it's too late.
Andy Klein

Latest posts by Andy Klein (see all)

  • Pingback: Four Tips To Help Photographers and Videographers Get The Most From B2()

  • Pingback: B2 for Beginners: Inside the B2 Web interface()

  • theqin

    Do we need to enable lifecycle rules if backing up time machine backups to Backblaze? How does Backblaze work generally if there are separate data and database files that need to kept in sync, but their relationship is only known to the software itself. Is it possible that we end up in an inconsistent state because only one file was uploaded at not the other, but we didn’t choose to keep past versions?

  • photon

    I’m curious about why there isn’t a lifecycle rule for number of versions? I was expecting to see it. My thinking is that it could be used in conjunction with ‘days from hiding to deleting’ or by itself to increase the granularity of file retention decisions.

  • Robson Sobral

    Hi!

    And what happens to deleted files? Are they influenced by Lifecycle?

    Oh! THe link to docs has the wrong TLD: `.net`, instead of `.com`.

  • Stefan Seidel

    “By default, Backblaze B2 will keep all the files you upload and all of the different versions of a file you upload.” – Does that mean it counts towards my storage quota? I have automated backups going to B2 and of course the backup software will by itself delete old files. Do I actively need to disable “lifecycle” for all my buckets?

    • Andy Klein

      Lifecycle Rules do not need to be disabled, they do not do anything unless they are activated. Backblaze B2 will keep all file versions unless it is told to delete them. If your backup application is programmed to delete old versions from B2 then B2 will delete them. The easiest thing way to check is to log into the B2 WebGUI and see if you have multiple versions or not. It is also possible that the backup application you are using has options you can set such as “keep only current” or “keep all”. And yes, all of the different versions of a file you have stored in B2 count towards your storage total.