{"id":109809,"date":"2023-09-21T09:13:41","date_gmt":"2023-09-21T16:13:41","guid":{"rendered":"https:\/\/www.backblaze.com\/blog\/?p=109809"},"modified":"2024-08-23T16:50:29","modified_gmt":"2024-08-23T23:50:29","slug":"big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade","status":"publish","type":"post","link":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/","title":{"rendered":"Big Performance Improvements in Rclone 1.64.0, but Should You Upgrade?"},"content":{"rendered":"\n
\"A<\/figure>\n\n\n\n
<\/div>\n\n\n\n

Rclone is an open source, command line tool for file management, and it\u2019s widely used to copy data between local storage and an array of cloud storage services<\/a>, including Backblaze B2 Cloud Storage<\/a>. Rclone has had a long association with Backblaze\u2014support for Backblaze B2 was added back in January 2016<\/a>, just two months before we opened Backblaze B2\u2019s public beta<\/a>, and five months before the official launch<\/a>\u2014and it\u2019s become an indispensable tool for many Backblaze B2 customers. If you want to explore the solution further, check out the Integration Guide<\/a>. <\/p>\n\n\n\n

Rclone<\/a> v1.64.0, released last week<\/a>, includes a new implementation of multithreaded data transfers, promising much faster data transfer of large files between cloud storage services. <\/p>\n\n\n\n

Does it deliver? Should you upgrade? Read on to find out!<\/p>\n\n\n\n

Multithreading to Boost File Transfer Performance<\/h2>\n\n\n\n

Something of a Swiss Army Knife for cloud storage, rclone<\/a> can copy files, synchronize directories, and even mount remote storage as a local filesystem. Previous versions of rclone were able to take advantage of multithreading<\/a> to accelerate the transfer of \u201clarge\u201d files (by default at least 256MB), but the benefits were limited.\u00a0<\/p>\n\n\n\n

When transferring files from a storage system to Backblaze B2, rclone would read chunks of the file into memory in a single reader thread, starting a set of multiple writer threads to simultaneously write those chunks to Backblaze B2. When the source storage was a local disk (the common case) as opposed to remote storage such as Backblaze B2, this worked really well\u2014the operation of moving files from local disk to Backblaze B2 was quite fast. However, when the source was another remote storage\u2014say, transferring from Amazon S3 to Backblaze B2, or even Backblaze B2 to Backblaze B2\u2014data chunks were read into memory by that single reader thread at about the same rate as they could be written to the destination, meaning that all but one of the writer threads were idle.<\/p>\n\n\n\n

What\u2019s the Big Deal About Rclone v1.64.0?<\/h2>\n\n\n\n

Rclone v1.64.0 completely refactors multithreaded transfers. Now rclone starts a single set of threads, each of which both reads a chunk of data from the source service into memory, and then writes that chunk to the destination service, iterating through a subset of chunks until the transfer is complete. The threads transfer their chunks of data in parallel, and each transfer is independent of the others. This architecture is both simpler and much, much faster.<\/p>\n\n\n\n

Show Me the Numbers!<\/h2>\n\n\n\n

How much faster? I spun up a virtual machine (VM) via our compute partner, Vultr<\/a>, and downloaded both rclone v1.64.0 and the preceding version, v1.63.1. As a quick test, I used Rclone\u2019s copyto<\/code><\/a> command to copy 1GB and 10GB files from Amazon S3 to Backblaze B2, like this:<\/p>\n\n\n\n

rclone --no-check-dest copyto s3remote:my-s3-bucket\/1gigabyte-test-file b2remote:my-b2-bucket\/1gigabyte-test-file<\/code><\/pre>\n\n\n\n
Note that I made no attempt to \u201ctune\u201d rclone for my environment by setting the chunk size or number of threads. I was interested in the out of the box performance. I used the --no-check-dest<\/code> flag so that rclone would overwrite the destination file each time, rather than detecting that the files were the same and skipping the copy.<\/p>\n\n\n\n
I ran each copyto<\/code> operation three times, then calculated the average time. Here are the results; all times are in seconds:<\/p>\n\n\n\n
Rclone version<\/strong><\/td>1GB<\/strong><\/td>10GB<\/strong><\/td><\/tr>
1.63.1<\/td>52.87<\/td>725.04<\/td><\/tr>
1.64.0<\/td>18.64<\/td>240.45<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
As you can see, the difference is significant! The new rclone transferred both files around three times<\/strong> faster than the previous version.<\/p>\n\n\n\n
So, copying individual large files is much<\/em> faster with the latest version of rclone. How about migrating a whole bucket containing a variety of file sizes from Amazon S3 to Backblaze B2, which is a more typical operation for a new Backblaze customer? I used rclone\u2019s copy<\/code><\/a> command to transfer the contents of an Amazon S3 bucket\u20142.8GB of data, comprising 35 files ranging in size from 990 bytes to 412MB\u2014to a Backblaze B2 Bucket:<\/p>\n\n\n\n
rclone --fast-list --no-check-dest copyto s3remote:my-s3-bucket b2remote:my-b2-bucket<\/code><\/pre>\n\n\n\n
Much to my dismay, this command failed, returning errors related to the files being corrupted in transfer, for example:<\/p>\n\n\n\n
2023\/09\/18 16:00:37 ERROR : tpcds-benchmark\/catalog_sales\/20221122_161347_00795_djagr_3a042953-d0a2-4b8d-8c4e-6a88df245253: corrupted on transfer: sizes differ 244695498 vs 0<\/code><\/pre>\n\n\n\n
Rclone was reporting that the transferred files in the destination bucket contained zero bytes, and deleting them to avoid the use of corrupt data.<\/p>\n\n\n\n
After some investigation, I discovered that the files were actually being transferred successfully, but a bug in rclone 1.64.0<\/a> caused the app to incorrectly interpret some successful transfers as corrupted, and thus delete the transferred file from the destination. <\/p>\n\n\n\n
I was able to use the --ignore-size<\/code> flag to workaround the bug by disabling the file size check so I could continue with my testing:<\/p>\n\n\n\n
rclone --fast-list --no-check-dest --ignore-size copyto s3remote:my-s3-bucket b2remote:my-b2-bucket<\/code><\/pre>\n\n\n\n
\n
A Word of Caution to Control Your Transaction Fees<\/h4>\n
Note the use of the --fast-list<\/code><\/a> flag. By default, rclone\u2019s method of reading the contents of cloud storage buckets minimizes memory usage at the expense of making a \u201clist files\u201d call for every subdirectory being processed. Backblaze B2\u2019s list files API, b2_list_file_names<\/code><\/a>, is a class C transaction, priced at $0.004 per 1,000 with 2,500 free per day. This doesn\u2019t sound like a lot of money, but using rclone with large file hierarchies can generate a huge<\/strong> number of transactions. Backblaze B2 customers have either hit their configured caps<\/a> or incurred significant transaction charges on their account when using rclone without the --fast-list<\/code> flag.<\/p>\n\n
We recommend you always use --fast-list<\/code> with rclone if at all possible. You can set an environment variable so you don\u2019t have to include the flag in every command:<\/p>\n\n
export RCLONE_FAST_LIST=1<\/code><\/p>\n<\/div>\n\n\n\n
Again, I performed the copy operation three times, and averaged the results:<\/p>\n\n\n\n
Rclone version<\/strong><\/td>2.8GB tree<\/strong><\/td><\/tr>
1.63.1<\/td>56.92<\/td><\/tr>
1.64.0<\/td>42.47<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Since the bucket contains both large and small files, we see a lesser, but still significant, improvement in performance with rclone v1.64.0\u2014it\u2019s about 33% faster than the previous version with this set of files.<\/p>\n\n\n\n
So, Should I Upgrade to the Latest Rclone?<\/h2>\n\n\n\n
As outlined above, rclone v1.64.0 contains a bug<\/a> that can cause copy (and presumably also sync<\/code>) operations to fail. If you want to upgrade to v1.64.0 now, you’ll have to use the --ignore-size<\/code> workaround. If you don’t want to use the workaround, it’s probably best to hold off until rclone releases v1.64.1, when the bug fix will likely be deployed\u2014I’ll come back and update this blog entry when I’ve tested it!<\/p>\n","protected":false},"excerpt":{"rendered":"
Rclone just released v1.64.0, which includes updated multithreading implementation. See the details of the release, and how it works with Backblaze B2. <\/p>\n","protected":false},"author":174,"featured_media":109828,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":"","jetpack_post_was_ever_published":false},"categories":[7,434,483],"tags":[468],"class_list":["post-109809","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-storage","category-featured-1","category-tech-lab","tag-b2cloud","entry"],"acf":[],"yoast_head":"\nBig Performance Improvements in Rclone 1.64.0, but Should You Upgrade?<\/title>\n<meta name=\"description\" content=\"Rclone just released v1.64.0, which includes updated multithreading implementation. See the details of the release, and how it works with Backblaze B2.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Big Performance Improvements in Rclone 1.64.0, but Should You Upgrade?\" \/>\n<meta property=\"og:description\" content=\"Rclone just released v1.64.0, which includes updated multithreading implementation. See the details of the release, and how it works with Backblaze B2.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/\" \/>\n<meta property=\"og:site_name\" content=\"Backblaze Blog | Cloud Storage & Cloud Backup\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/backblaze\" \/>\n<meta property=\"article:published_time\" content=\"2023-09-21T16:13:41+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-23T23:50:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2023\/09\/bb-bh-rclone-Backblaze-B2-Multithreading.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1440\" \/>\n\t<meta property=\"og:image:height\" content=\"820\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Pat Patterson\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@backblaze\" \/>\n<meta name=\"twitter:site\" content=\"@backblaze\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pat Patterson\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Big Performance Improvements in Rclone 1.64.0, but Should You Upgrade?","description":"Rclone just released v1.64.0, which includes updated multithreading implementation. See the details of the release, and how it works with Backblaze B2.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/","og_locale":"en_US","og_type":"article","og_title":"Big Performance Improvements in Rclone 1.64.0, but Should You Upgrade?","og_description":"Rclone just released v1.64.0, which includes updated multithreading implementation. See the details of the release, and how it works with Backblaze B2.","og_url":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/","og_site_name":"Backblaze Blog | Cloud Storage & Cloud Backup","article_publisher":"https:\/\/www.facebook.com\/backblaze","article_published_time":"2023-09-21T16:13:41+00:00","article_modified_time":"2024-08-23T23:50:29+00:00","og_image":[{"width":1440,"height":820,"url":"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2023\/09\/bb-bh-rclone-Backblaze-B2-Multithreading.png","type":"image\/png"}],"author":"Pat Patterson","twitter_card":"summary_large_image","twitter_creator":"@backblaze","twitter_site":"@backblaze","twitter_misc":{"Written by":"Pat Patterson","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/#article","isPartOf":{"@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/"},"author":{"name":"Pat Patterson","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/a724a8aee97b6451107442747cd101a4"},"headline":"Big Performance Improvements in Rclone 1.64.0, but Should You Upgrade?","datePublished":"2023-09-21T16:13:41+00:00","dateModified":"2024-08-23T23:50:29+00:00","mainEntityOfPage":{"@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/"},"wordCount":1025,"commentCount":0,"publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/09\/bb-bh-rclone-Backblaze-B2-Multithreading.png","keywords":["B2Cloud"],"articleSection":["Cloud Storage","Featured","Tech Lab"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/","url":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/","name":"Big Performance Improvements in Rclone 1.64.0, but Should You Upgrade?","isPartOf":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/#primaryimage"},"image":{"@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/#primaryimage"},"thumbnailUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/09\/bb-bh-rclone-Backblaze-B2-Multithreading.png","datePublished":"2023-09-21T16:13:41+00:00","dateModified":"2024-08-23T23:50:29+00:00","description":"Rclone just released v1.64.0, which includes updated multithreading implementation. See the details of the release, and how it works with Backblaze B2.","breadcrumb":{"@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/#primaryimage","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/09\/bb-bh-rclone-Backblaze-B2-Multithreading.png","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/09\/bb-bh-rclone-Backblaze-B2-Multithreading.png","width":1440,"height":820,"caption":"A decorative image showing a diagram about multithreading, as well as the Rclone and Backblaze logos."},{"@type":"BreadcrumbList","@id":"https:\/\/www.backblaze.com\/blog\/big-performance-improvements-in-rclone-1-64-0-but-should-you-upgrade\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Big Performance Improvements in Rclone 1.64.0, but Should You Upgrade?"}]},{"@type":"WebSite","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#website","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","name":"Backblaze Cloud Solutions Blog","description":"Cloud Storage & Cloud Backup","publisher":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#organization","name":"Backblaze","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.backblaze.com\/blog\/wp-content\/uploads\/2017\/12\/backblaze_icon_transparent.png?fit=512%2C512&ssl=1","width":512,"height":512,"caption":"Backblaze"},"image":{"@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/backblaze","https:\/\/x.com\/backblaze","https:\/\/www.youtube.com\/user\/Backblaze","https:\/\/en.wikipedia.org\/wiki\/Backblaze"]},{"@type":"Person","@id":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/#\/schema\/person\/a724a8aee97b6451107442747cd101a4","name":"Pat Patterson","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2022\/01\/PatPatterson1920px-150x150.png","url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2022\/01\/PatPatterson1920px-150x150.png","contentUrl":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2022\/01\/PatPatterson1920px-150x150.png","caption":"Pat Patterson"},"description":"Pat Patterson is the former chief technical evangelist at Backblaze. Over his three decades in the industry, Pat has built software and communities at Sun Microsystems, Salesforce, StreamSets, and Citrix. In his role at Backblaze, he creates and delivers content tailored to the needs of the hands-on technical professional, acts as the \u201cvoice of the developer\u201d on the Product team, and actively participates in the wider technical community. Outside the office, Pat runs far, having completed ultramarathons up to the 50 mile distance. Catch up with Pat via Bluesky or LinkedIn.","url":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/author\/pat\/"}]}},"jetpack_featured_media_url":"https:\/\/backblazeprod.wpenginepowered.com\/wp-content\/uploads\/2023\/09\/bb-bh-rclone-Backblaze-B2-Multithreading.png","_links":{"self":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/109809","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/users\/174"}],"replies":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/comments?post=109809"}],"version-history":[{"count":0,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/posts\/109809\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media\/109828"}],"wp:attachment":[{"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/media?parent=109809"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/categories?post=109809"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/backblazeprod.wpenginepowered.com\/blog\/wp-json\/wp\/v2\/tags?post=109809"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}