Contents x
Integrate Rclone with Backblaze B2
    • Print
    • Dark
      Light
    Contents

    Integrate Rclone with Backblaze B2

      • Print
      • Dark
        Light
      Article Summary

      Rclone is a third-party integration that you can use with Backblaze B2 Cloud Storage, and it is available for most platforms.

      Enable Backblaze B2

      Before you begin: You must have a Backblaze B2 Cloud Storage account. You can sign up here. If you already have a Backblaze account and the left navigation menu contains a B2 Cloud Storage section, your account is already enabled for Backblaze B2.

      1. Sign in to your Backblaze account.
      2. In the left navigation menu under Account, click My Settings.
      3. Under Enabled Products, select the checkbox to enable B2 Cloud Storage.
      4. Review the Terms and Conditions, and click OK to accept them. 

      Create a Bucket

      1. Sign in to your Backblaze account.
      2. In the left navigation menu under B2 Cloud Storage, click Buckets.
      3. Click Create a Bucket.
      4. Enter a name for your bucket.
        Bucket names must be at least six characters and globally unique. A message is displayed if your bucket name is already in use.
      5. Select a privacy setting: Private or Public.
        Files that are in a private bucket require authentication to perform an action, for example, downloading. Public buckets do not require authentication so you can easily share files. You can change a bucket's privacy settings at any time.
      6. If applicable, enable a Backblaze B2 server-side encryption key.
      7. Enable Object Lock to restrict a file from being modified or deleted for a specified period of time.
      8. Click Create a Bucket, and copy the value that is in the Endpoint field; you may need this value for other processes.
      9. Click Lifecycle Settings to control how long to keep the files in your new bucket.

      Create an Application Key

      Application keys control access to your Backblaze B2 Cloud Storage account and the buckets that are contained in your account.

      1. Sign in to your Backblaze account.
      2. In the left navigation menu under Account, click Application Keys.
      3. Click Add a New Application Key, and enter an app key name.
            You cannot search an app key by this name; therefore, app key names are not required to be globally unique.
      4. Select All or a specific bucket in the Allow Access to Bucket(s) dropdown menu.
      5. Optionally, select your access type (Read and Write, Read Only, or Write Only).
      6. Optionally, select the Allow List All Bucket Names checkbox (required for the B2 Native API b2_list_buckets and the S3-Compatible API S3 List Buckets operations).
      7. Optionally, enter a file name prefix to restrict application key access only to files with that prefix. Depending on what you selected in step #4, this limits application key access to files with the specified prefix for all buckets or just the selected bucket.
      8. Optionally, enter a positive integer to limit the time, in seconds, before the application key expires. The value must be less than 1000 days (in seconds).
      9. Click Create New Key, and note the resulting keyID and applicationKey values.
      Note
      When you create a new app key, the response contains the actual key string, for example N2Zug0evLcHDlh_L0Z0AJhiGGdY. You can always find the keyID on this page, but for security, the applicationKey appears only once. Make sure you copy and securely save this value elsewhere.

      Create and Install Rclone

      Download Rclone here or, depending on your operating system, install it from your system's default repository:

      • Linux: yum install rclone
      • Mac: brew install rclone
      • Windows: choco install rclone
      • Debian: apt install rclone

      Configure Rclone

      1. Open a command prompt or terminal window, and enter the following command:
        rclone config

        You will see your currently configured accounts (if any) and a list of the menu commands.

      2. To create a new configuration, enter n for new, and enter a name for the configuration. A list of cloud accounts is shown. Each account has an alias and a number.
        This name is used in every command so short descriptive names are best. This guide uses "b2."
      3. Choose the number or alias of the account you want to use.
      4. Enter your App key and Key ID, and press Enter at the hard_delete prompt to accept the default.
      5. Press Enter again to skip the advanced configuration page.
        You can edit the configuration at any time to review or change options.
      6.  Review the configuration summary screen, and press Enter to accept it.
      7. To test the configuration, enter the following command to list all of the buckets in your account:
        rclone lsd b2:

      Use the Rclone Text-Based GUI to Browse Buckets and Files

      Rclone has a text-based GUI inspired by Norton Commander, called ncdu.

      1.  Enter the following command to browse the buckets that are in your account.
        rclone ncdu b2:

        If the ncdu command encounters difficulty with files or directory structures it displays errors. Press Ctrl+L on Linux or MacOS, or resize the DOS window to clear the screen.

      2. Navigate to a bucket, and press Enter to browse the files in the bucket.
      3. Press ? to see a list of all of the commands.

      Use the Rclone Web GUI to Configure or Browse Buckets

      Rclone also has a web interface. The Rclone web interface is experimental.

      1. Enter the following command to access the Rclone web interface:
        rclone rcd --rc-web-gui
      2.  Select Configs in the left navigation menu to browse, edit, or add new configs.
      3.  Select Explorer in the left navigation menu, and select an account to browse buckets.

      Useful Commands

      • lsd - Lists buckets/directories (depending on whether you are at the top level in your account or inside a bucket).
      • ls - Lists files.
      • mkdir - Creates a bucket/directory. Buckets are automatically “private.”
      • copy - Copies files.
      • sync - Syncs files. This creates a mirror copy so both sides match.

      Examples

      The following examples show commands to perform common tasks.

      Note:
      The following examples use sample directory and configuration names.

      List Directories

      rclone lsd b2:

      List Buckets and Files

      rclone ls b2:my-bucket

      Create a New Bucket

      rclone mkdir b2:new-unique-bucket

      Create a New Folder

      rclone mkdir b2:my-bucket/new folder

      Copy or Sync Files

      rclone copy myfiles.zip b2:mybucket
      rclone sync C:\Downloads b2:mybucket/Downloads

      Copy or Sync from Backblaze B2 to a Local File System

      rclone copy b2:mybucket/myfiles.zip /home/mfarace
      rclone sync b2:mybucket/downloads /home/mfarace/downloads

      Fast List Option

      By default, Rclone lists the contents of a Backblaze B2 bucket one directory at a time (one directory per API call). But, with the --fast-list option, it retrieves a listing of all of the files in the bucket, consuming just one API call per 1,000 files. The --fast-list option uses more memory, but it consumes fewer API transactions. Therefore, Backblaze strongly recommends that you use this option with every Rclone command.

      You can set an environment variable so you do not need to include the flag every time you enter an Rclone command:

      export RCLONE_FAST_LIST=1

      For more information about the fast-list option, see the Rclone guide here.

      Performance

      The default number of threads for file transfers is four. You can change the number of threads with the --transfers command. The speed of your Internet connection determines how much the --transfers command helps your file transfer speed. For most people, 10-20 threads is sufficient.

      Example:

      rclone --transfers 10 copy Ubuntu.iso b2:mybucket

      If you have an Internet connection that is 100 Mbps or faster, you can try 50-100 threads. More threads use more memory. If you are transferring files over a long distance, increasing the number of threads helps compensate for latency.

      Rclone Mount

      Rclone Mount allows you to mount a bucket from your account to a local directory. To use Rclone Mount, you must install a Filesystem in Userspace (FUSE) layer on your operating system. FUSE is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code.

      Install Rclone Mount

      You can install Rclone Mount using the following steps based on your operating system.

      MacOS

      For Catalina (10.15) use OSXFuse 3.10.6. Starting with OSXFuse 3.11, there are changes for OSX 11 / BigSur and Intel Silicon chip support that cause problems for OSX 10.x.

      Use Homebrew to install Rclone Mount using the brew install rclone command or download the Mac DMG.

      Example:

      On Mac, use the “mount2” command, instead of “mount.”  Commands otherwise work exactly the same as on Linux systems.

      rclone mount2 b2:mybucket /Users/mfarace/b2bucket

      Linux

      For Linux, the FUSE file system is bundled with s3fs in linux repositories, so install s3fs-fuse, and then install Rclone.

      Example:

      sudo apt-get install s3fs-fuse rclone
      sudo apt-get install rclone

      On Linux systems, you must mount the bucket to an empty directory. Rclone Mount runs in the foreground by default. On Linux systems, the --daemon option runs Rclone Mount as a background process.

      Use case #1 - Mount a read-only bucket

      mkdir /mnt/b2bucket
      rclone mount b2:fuse-mf1 /mnt/b2bucket --daemon

      Use case #2 - Mount a bucket for writes

      rclone mount --vfs-cache-mode writes b2:fuse-mf1 /mnt/b2bucket --daemon

      Applications expect to be able to write into the middle of a file; they cannot do so without this option.

      Windows

      For Windows, install winfsp for the same functionality as FUSE has on Mac/Linux. You can choose github or a chocolatey repository as safe sources to install from.

      Example:

      Windows does not have the option to run Rclone Mount in the background. You can give it a volume name and assign it a drive letter, which is not available on Mac or Linux. If you want to mount the bucket to a directory, the directory must not exist, and also must not be at the top level of a drive. It must be a non-existent sub-directory. All other options work the same.

      rclone mount b2:fuse-mf1 Z: -o volname=B2bucket
      rclone mount b2:fuse-mf1 C:\Users\mfarace\non-existent-directory

      Adjust Performance for Rclone Mount

      If you experience performance issues, adjust the Rclone Mount options as shown in the following example:

      Vfs-cache-mode:          disabled  (read only mode)



Vfs-read-chunk-size:     128 MB    (read objects from the buckets in chunks)



Dir-cache-time:          5 mins    (limits how often a bucket is scanned)



Vfs-cache-poll-interval: 1 min     (when to check cache for file changes)



Vfs-cache-max-age:       1 hour

      “VFS Cache Mode” is disabled by default. Enabling VFS Cacha Mode uses memory to improve file operations.

      Additional options:

      --vfs-cache-mode=minimal   (minimal read/write buffering to disk, but not enough to do seeks in a file that is open, and if an upload fails, it won’t retry).



--vfs-cache-mode=writes     (writes are buffers but reads are not)



--vfs-cache-mode=full       (reads and writes are buffered)



--vfs-cache-mode full       (no equal sign on Windows)

      Vfs cache mode is important when you copy large files to and from your mounted bucket. If you are on Windows, the following example shows the suggested command:

      rclone --vfs-cache-mode full --transfers 40 mount b2:fuse-mf1 Z:

      The following options can also be adjusted as needed to improve performance:

      --buffer-size        VFS cache size buffered in memory



--vfs-read-ahead      VFS cache size buffered to disk



--dir-cache-time 60m   Use to minimize bucket reads on buckets that rarely change.

      Advanced Options

      --bw-limit           Limits the maximum bandwidth rclone can use.


--no-modtime          Don't read/write the modification time (can speed things up)


--use-server-modtime   Use the server’s recorded time of the file modification
      Was this article helpful?
      Related articles
      What's Next