Integrate Rclone with Backblaze B2
    • Dark
      Light

    Integrate Rclone with Backblaze B2

    • Dark
      Light

    Article summary

    Rclone is a third-party integration that you can use with Backblaze B2 Cloud Storage, and it is available for most platforms.

    View the following short tutorial to get up and running with Rclone and Backblaze B2 quickly.


    Enable Backblaze B2

    Before you begin: You must have a Backblaze B2 Cloud Storage account. If you already have a Backblaze account and the left navigation menu contains a B2 Cloud Storage section, your account is already enabled for Backblaze B2.

    1. Sign in to the Backblaze web console.
    2. In the user menu in the upper-right corner of the page, select My Settings.
    3. Under Enabled Products, select the checkbox to enable B2 Cloud Storage.
    4. Review the Terms and Conditions, and click OK to accept them. 

    Create a Bucket

    1. Sign in to the Backblaze web console.
    2. In the left navigation menu under B2 Cloud Storage, click Buckets.
    3. Click Create a Bucket.
    4. Enter a name for your bucket. Bucket names must be at least six characters and globally unique.
      A message is displayed if your bucket name is already in use.
    5. Select a privacy setting: Private or Public. Files that are in a private bucket require authentication to perform an action, for example, downloading. Public buckets do not require authentication so you can easily share files. You can change a bucket's privacy settings at any time.
    6. If applicable, enable a Backblaze B2 server-side encryption key.
    7. Enable Object Lock to restrict a file from being modified or deleted for a specified period of time.
    8. Click Create a Bucket, and copy the value that is in the Endpoint field; you will need this value later.
    9. Click Lifecycle Settings to control how long to keep the files in your new bucket.
    Note
    If this is your first time creating a public bucket, complete the following tasks to ensure that you have the correct permissions to create a public bucket:

    1. Verify your email address.
    2. Have a payment history on file, or use the credit card form to pay a small fee that is credited to your account balance.

    Create an Application Key

    Application keys control access to your Backblaze B2 Cloud Storage account and the buckets that are contained in your account.

    1. Sign in to the Backblaze web console.
    2. In the left navigation menu under B2 Cloud Storage, click Application Keys.
    3. Click Add a New Application Key, and enter an app key name.
      You cannot search an app key by this name; therefore, app key names are not required to be globally unique.
    4. Select All or a specific bucket in the Allow Access to Bucket(s) dropdown menu.
    5. Optionally, select your access type (Read and Write, Read Only, or Write Only).
    6. Optionally, select the Allow List All Bucket Names checkbox (required for the B2 Native API b2_list_buckets and the S3-Compatible API S3 List Buckets operations).
    7. Optionally, enter a file name prefix to restrict application key access only to files with that prefix. Depending on what you selected in step #4, this limits application key access to files with the specified prefix for all buckets or just the selected bucket.
    8. Optionally, enter a positive integer to limit the time, in seconds, before the application key expires. The value must be less than 1000 days (in seconds).
    9. Click Create New Key, and note the resulting keyID and applicationKey values.
    Note
    When you create a new app key, the response contains the actual key string, for example N2Zug0evLcHDlh_L0Z0AJhiGGdY. You can always find the keyID on this page, but for security, the applicationKey appears only once. Make sure you copy and securely save this value elsewhere.

    Create and Install Rclone

    Download Rclone here or, depending on your operating system, install it from your system's default repository:

    • Linux: yum install rclone
    • Mac: brew install rclone
    • Windows: choco install rclone
    • Debian: apt install rclone

    Configure Rclone

    1. Open a command prompt or terminal window, and enter the following command:
      rclone config

      You will see your currently configured accounts (if any) and a list of the menu commands.

    2. To create a new configuration, enter n for new, and enter a name for the configuration. A list of cloud accounts is shown. Each account has an alias and a number.
      This name is used in every command so short descriptive names are best. This guide uses "b2."
    3. Choose the number or alias of the account you want to use.
    4. Enter your App key and Key ID, and press Enter at the hard_delete prompt to accept the default.
    5. Press Enter again to skip the advanced configuration page.
      You can edit the configuration at any time to review or change options.
    6.  Review the configuration summary screen, and press Enter to accept it.
    7. To test the configuration, enter the following command to list all of the buckets in your account:
      rclone lsd b2:

    Use the Rclone Text-Based GUI to Browse Buckets and Files

    Rclone has a text-based GUI inspired by Norton Commander, called ncdu.

    1.  Enter the following command to browse the buckets that are in your account.
      rclone ncdu b2:

      If the ncdu command encounters difficulty with files or directory structures it displays errors. Press Ctrl+L on Linux or MacOS, or resize the DOS window to clear the screen.

    2. Navigate to a bucket, and press Enter to browse the files in the bucket.
    3. Press to see a list of all of the commands.

    Use the Rclone Web GUI to Configure or Browse Buckets

    Rclone also has a web interface. The Rclone web interface is experimental.

    1. Enter the following command to access the Rclone web interface:
      rclone rcd --rc-web-gui
    2.  Select Configs in the left navigation menu to browse, edit, or add new configs.
    3.  Select Explorer in the left navigation menu, and select an account to browse buckets.

    Useful Commands

    • lsd - Lists buckets/directories (depending on whether you are at the top level in your account or inside a bucket).
    • ls - Lists files.
    • mkdir - Creates a bucket/directory. Buckets are automatically “private.”
    • copy - Copies files.
    • sync - Syncs files. This creates a mirror copy so both sides match.

    Examples

    The following examples show commands to perform common tasks.

    Note:
    The following examples use sample directory and configuration names.

    List Directories

    rclone lsd b2:

    List Buckets and Files

    rclone ls b2:my-bucket

    Create a New Bucket

    rclone mkdir b2:new-unique-bucket

    Create a New Folder

    rclone mkdir b2:my-bucket/new folder

    Copy or Sync Files

    rclone copy myfiles.zip b2:mybucket
    rclone sync C:\Downloads b2:mybucket/Downloads

    Copy or Sync from Backblaze B2 to a Local File System

    rclone copy b2:mybucket/myfiles.zip /home/mfarace
    rclone sync b2:mybucket/downloads /home/mfarace/downloads

    Fast List Option

    By default, Rclone lists the contents of a Backblaze B2 bucket one directory at a time (one directory per API call). But, with the --fast-list option, it retrieves a listing of all of the files in the bucket, consuming just one API call per 1,000 files. The --fast-list option uses more memory, but it consumes fewer API transactions. Therefore, Backblaze strongly recommends that you use this option with every Rclone command.

    You can set an environment variable so you do not need to include the flag every time you enter an Rclone command:

    export RCLONE_FAST_LIST=1

    For more information about the fast-list option, see the Rclone guide here.

    Performance

    The default number of threads for file transfers is four. You can change the number of threads with the --transfers command. The speed of your Internet connection determines how much the --transfers command helps your file transfer speed. For most people, 10-20 threads is sufficient.

    Example:

    rclone --transfers 10 copy Ubuntu.iso b2:mybucket

    If you have an Internet connection that is 100 Mbps or faster, you can try 50-100 threads. More threads use more memory. If you are transferring files over a long distance, increasing the number of threads helps compensate for latency.

    Rclone Mount

    Rclone Mount allows you to mount a bucket from your account to a local directory. To use Rclone Mount, you must install a Filesystem in Userspace (FUSE) layer on your operating system. FUSE is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code.

    Install Rclone Mount

    You can install Rclone Mount using the following steps based on your operating system.

    MacOS

    For Catalina (10.15) use OSXFuse 3.10.6. Starting with OSXFuse 3.11, there are changes for OSX 11 / BigSur and Intel Silicon chip support that cause problems for OSX 10.x.

    Use Homebrew to install Rclone Mount using the brew install rclone command or download the Mac DMG.

    Example:

    On Mac, use the “mount2” command, instead of “mount.”  Commands otherwise work exactly the same as on Linux systems.

    rclone mount2 b2:mybucket /Users/mfarace/b2bucket

    Linux

    For Linux, the FUSE file system is bundled with s3fs in linux repositories, so install s3fs-fuse, and then install Rclone.

    Example:

    sudo apt-get install s3fs-fuse rclone
    sudo apt-get install rclone

    On Linux systems, you must mount the bucket to an empty directory. Rclone Mount runs in the foreground by default. On Linux systems, the --daemon option runs Rclone Mount as a background process.

    Use case #1 - Mount a read-only bucket

    mkdir /mnt/b2bucket
    rclone mount b2:fuse-mf1 /mnt/b2bucket --daemon

    Use case #2 - Mount a bucket for writes

    rclone mount --vfs-cache-mode writes b2:fuse-mf1 /mnt/b2bucket --daemon

    Applications expect to be able to write into the middle of a file; they cannot do so without this option.

    Windows

    For Windows, install winfsp for the same functionality as FUSE has on Mac/Linux. You can choose github or a chocolatey repository as safe sources to install from.

    Example:

    Windows does not have the option to run Rclone Mount in the background. You can give it a volume name and assign it a drive letter, which is not available on Mac or Linux. If you want to mount the bucket to a directory, the directory must not exist, and also must not be at the top level of a drive. It must be a non-existent sub-directory. All other options work the same.

    rclone mount b2:fuse-mf1 Z: -o volname=B2bucket
    rclone mount b2:fuse-mf1 C:\Users\mfarace\non-existent-directory

    Adjust Performance for Rclone Mount

    If you experience performance issues, adjust the Rclone Mount options as shown in the following example:

    Vfs-cache-mode:          disabled  (read only mode)
    
    
    
    Vfs-read-chunk-size:     128 MB    (read objects from the buckets in chunks)
    
    
    
    Dir-cache-time:          5 mins    (limits how often a bucket is scanned)
    
    
    
    Vfs-cache-poll-interval: 1 min     (when to check cache for file changes)
    
    
    
    Vfs-cache-max-age:       1 hour

    “VFS Cache Mode” is disabled by default. Enabling VFS Cacha Mode uses memory to improve file operations.

    Additional options:

    --vfs-cache-mode=minimal   (minimal read/write buffering to disk, but not enough to do seeks in a file that is open, and if an upload fails, it won’t retry).
    
    
    
    --vfs-cache-mode=writes     (writes are buffers but reads are not)
    
    
    
    --vfs-cache-mode=full       (reads and writes are buffered)
    
    
    
    --vfs-cache-mode full       (no equal sign on Windows)

    Vfs cache mode is important when you copy large files to and from your mounted bucket. If you are on Windows, the following example shows the suggested command:

    rclone --vfs-cache-mode full --transfers 40 mount b2:fuse-mf1 Z:

    The following options can also be adjusted as needed to improve performance:

    --buffer-size        VFS cache size buffered in memory
    
    
    
    --vfs-read-ahead      VFS cache size buffered to disk
    
    
    
    --dir-cache-time 60m   Use to minimize bucket reads on buckets that rarely change.

    Advanced Options

    --bw-limit           Limits the maximum bandwidth rclone can use.
    
    
    --no-modtime          Don't read/write the modification time (can speed things up)
    
    
    --use-server-modtime   Use the server’s recorded time of the file modification

    Was this article helpful?