Integrate Rclone with Backblaze B2
Rclone is a third-party integration that you can use with Backblaze B2 Cloud Storage, and it is available for most platforms.
Enable Backblaze B2
Before you begin: You must have a Backblaze B2 Cloud Storage account. You can sign up here. If you already have a Backblaze account and the left navigation menu contains a B2 Cloud Storage section, your account is already enabled for Backblaze B2.
- Sign in to your Backblaze account.
- In the left navigation menu under Account, click My Settings.
- Under Enabled Products, select the checkbox to enable B2 Cloud Storage.
- Review the Terms and Conditions, and click OK to accept them.
Create a Bucket
- Sign in to your Backblaze account.
- In the left navigation menu under B2 Cloud Storage, click Buckets.
- Click Create a Bucket.
- Enter a name for your bucket.
Bucket names must be at least six characters and globally unique. A message is displayed if your bucket name is already in use.
- Select a privacy setting: Private or Public.
Files that are in a private bucket require authentication to perform an action, for example, downloading. Public buckets do not require authentication so you can easily share files. You can change a bucket's privacy settings at any time.
- If applicable, enable a Backblaze B2 server-side encryption key.
- Enable Object Lock to restrict a file from being modified or deleted for a specified period of time.
- Click Create a Bucket, and copy the value that is in the Endpoint field; you may need this value for other processes.
- Click Lifecycle Settings to control how long to keep the files in your new bucket.
Create an Application Key
Application keys control access to your Backblaze B2 Cloud Storage account and the buckets that are contained in your account.
- Sign in to your Backblaze account.
- In the left navigation menu under Account, click Application Keys.
- Click Add a New Application Key, and enter an app key name.
You cannot search an app key by this name; therefore, app key names are not required to be globally unique.
- In the Allow Access to Bucket(s) dropdown menu, select All or a specific bucket.
- Select your access type (for example, Read and Write).
- Select the optional Allow List All Bucket Names checkbox (required for the Backblaze B2 S3-compatible API List Buckets operation).
- Click Create New Key, and note the resulting keyID and applicationKey values.
Create and Install Rclone
Download Rclone here or, depending on your operating system, install it from your system's default repository:
yum install rclone
brew install rclone
choco install rclone
apt install rclone
- Open a command prompt or terminal window, and enter the following command:
You will see your currently configured accounts (if any) and a list of the menu commands.
- To create a new configuration, enter n for new, and enter a name for the configuration. A list of cloud accounts is shown. Each account has an alias and a number.
This name is used in every command so short descriptive names are best. This guide uses "b2."
- Choose the number or alias of the account you want to use.
- Enter your App key and Key ID, and press Enter at the hard_delete prompt to accept the default.
- Press Enter again to skip the advanced configuration page.
You can edit the configuration at any time to review or change options.
- Review the configuration summary screen, and press Enter to accept it.
- To test the configuration, enter the following command to list all of the buckets in your account:
rclone lsd b2:
Use the Rclone Text-Based GUI to Browse Buckets and Files
Rclone has a text-based GUI inspired by Norton Commander, called ncdu.
- Enter the following command to browse the buckets that are in your account.
rclone ncdu b2:
If the ncdu command encounters difficulty with files or directory structures it displays errors. Press Ctrl+L on Linux or MacOS, or resize the DOS window to clear the screen.
- Navigate to a bucket, and press Enter to browse the files in the bucket.
- Press ? to see a list of all of the commands.
Use the Rclone Web GUI to Configure or Browse Buckets
Rclone also has a web interface. The Rclone web interface is experimental.
- Enter the following command to access the Rclone web interface:
rclone rcd --rc-web-gui
- Select Configs in the left navigation menu to browse, edit, or add new configs.
- Select Explorer in the left navigation menu, and select an account to browse buckets.
lsd- Lists buckets/directories (depending on whether you are at the top level in your account or inside a bucket).
ls- Lists files.
mkdir- Creates a bucket/directory. Buckets are automatically “private.”
copy- Copies files.
sync- Syncs files. This creates a mirror copy so both sides match.
The following examples show commands to perform common tasks.
rclone lsd b2:
List Buckets and Files
rclone ls b2:my-bucket
Create a New Bucket
rclone mkdir b2:new-unique-bucket
Create a New Folder
rclone mkdir b2:my-bucket/new folder
Copy or Sync Files
rclone copy myfiles.zip b2:mybucket
rclone sync C:\Downloads b2:mybucket/Downloads
Copy or Sync from Backblaze B2 to a Local File System
rclone copy b2:mybucket/myfiles.zip /home/mfarace
rclone sync b2:mybucket/downloads /home/mfarace/downloads
Fast List Option
By default, Rclone lists the contents of a Backblaze B2 bucket one directory at a time (one directory per API call). But, with the
--fast-list option, it retrieves a listing of all of the files in the bucket, consuming just one API call per 1,000 files. The
--fast-list option uses more memory, but it consumes fewer API transactions. Therefore, Backblaze strongly recommends that you use this option with every Rclone command.
You can set an environment variable so you do not need to include the flag every time you enter an Rclone command:
For more information about the fast-list option, see the Rclone guide here.
The default number of threads for file transfers is four. You can change the number of threads with the
--transfers command. The speed of your Internet connection determines how much the
--transfers command helps your file transfer speed. For most people, 10-20 threads is sufficient.
rclone --transfers 10 copy Ubuntu.iso b2:mybucket
If you have an Internet connection that is 100 Mbps or faster, you can try 50-100 threads. More threads use more memory. If you are transferring files over a long distance, increasing the number of threads helps compensate for latency.
Rclone Mount allows you to mount a bucket from your account to a local directory. To use Rclone Mount, you must install a Filesystem in Userspace (FUSE) layer on your operating system. FUSE is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code.
Install Rclone Mount
You can install Rclone Mount using the following steps based on your operating system.
For Catalina (10.15) use OSXFuse 3.10.6. Starting with OSXFuse 3.11, there are changes for OSX 11 / BigSur and Intel Silicon chip support that cause problems for OSX 10.x.
Use Homebrew to install Rclone Mount using the
brew install rclone command or download the Mac DMG.
On Mac, use the “mount2” command, instead of “mount.” Commands otherwise work exactly the same as on Linux systems.
rclone mount2 b2:mybucket /Users/mfarace/b2bucket
For Linux, the FUSE file system is bundled with s3fs in linux repositories, so install s3fs-fuse, and then install Rclone.
sudo apt-get install s3fs-fuse rclone
sudo apt-get install rclone
On Linux systems, you must mount the bucket to an empty directory. Rclone Mount runs in the foreground by default. On Linux systems, the
--daemon option runs Rclone Mount as a background process.
Use case #1 - Mount a read-only bucket
rclone mount b2:fuse-mf1 /mnt/b2bucket --daemon
Use case #2 - Mount a bucket for writes
rclone mount --vfs-cache-mode writes b2:fuse-mf1 /mnt/b2bucket --daemon
Applications expect to be able to write into the middle of a file; they cannot do so without this option.
Windows does not have the option to run Rclone Mount in the background. You can give it a volume name and assign it a drive letter, which is not available on Mac or Linux. If you want to mount the bucket to a directory, the directory must not exist, and also must not be at the top level of a drive. It must be a non-existent sub-directory. All other options work the same.
rclone mount b2:fuse-mf1 Z: -o volname=B2bucket
rclone mount b2:fuse-mf1 C:\Users\mfarace\non-existent-directory
Adjust Performance for Rclone Mount
If you experience performance issues, adjust the Rclone Mount options as shown in the following example:
Vfs-cache-mode: disabled (read only mode)
Vfs-read-chunk-size: 128 MB (read objects from the buckets in chunks)
Dir-cache-time: 5 mins (limits how often a bucket is scanned)
Vfs-cache-poll-interval: 1 min (when to check cache for file changes)
Vfs-cache-max-age: 1 hour
“VFS Cache Mode” is disabled by default. Enabling VFS Cacha Mode uses memory to improve file operations.
--vfs-cache-mode=minimal (minimal read/write buffering to disk, but not enough to do seeks in a file that is open, and if an upload fails, it won’t retry).
--vfs-cache-mode=writes (writes are buffers but reads are not)
--vfs-cache-mode=full (reads and writes are buffered)
--vfs-cache-mode full (no equal sign on Windows)
Vfs cache mode is important when you copy large files to and from your mounted bucket. If you are on Windows, the following example shows the suggested command:
rclone --vfs-cache-mode full --transfers 40 mount b2:fuse-mf1 Z:
The following options can also be adjusted as needed to improve performance:
--buffer-size VFS cache size buffered in memory
--vfs-read-ahead VFS cache size buffered to disk
--dir-cache-time 60m Use to minimize bucket reads on buckets that rarely change.
--bw-limit Limits the maximum bandwidth rclone can use.
--no-modtime Don't read/write the modification time (can speed things up)
--use-server-modtime Use the server’s recorded time of the file modification