We are proud to introduce Backblaze Online Backup 4.0, our fastest release for backing up and recovering customer data. There are a lot of cool things in 4.0, but making sure our customers can back up and recover their data fast is at the top of the list. We have always allowed customers to back up and restore as quickly as they were able and never throttled our customers’ speeds. However, some customers have experienced less than optimal upload and download speeds and this concerned us. As we dug into the issue, we realized we had two choices, 1) fix the internet or 2) work around the problem. We chose the latter. Let’s start with how we have improved upload and download speeds and then finish up with a look at some of the other improvements in software release 4.0.
Improving Data Upload Speed with Threading
Network Latency and Data Upload
When Backblaze is installed on a PC or Mac, it regularly tests the amount of network bandwidth we can use without getting in the way. Customers have always had the ability to let us use more (or less) of their available bandwidth by using “Manual Throttle” in the Backblaze Preferences Panel. Unchecking “Automatic Throttle,” as shown below, accesses this capability.
Placing the Manual Throttle slider all the way to the right (Faster Backups) should use nearly all the bandwidth available for data uploading, but for some users this wasn’t the case. The primary culprit is network latency or more precisely, round-trip network latency. For Backblaze, this is the length of time it takes encrypted data to travel from a computer to the Backblaze data center and then to receive back an acknowledgement that the data was delivered. In general, the further away a customer is from our data center in Sacramento, California, the longer the latency. So a customer in San Francisco (87 miles from the data center) should have almost no latency, while a customer in Sydney, Australia (7,486 miles) could have hundreds of milliseconds of latency. Other factors can also impact the optimization of upload speed such as local area network traffic, wide area network traffic, and other applications on your computer using bandwidth. These are factors the customer has control over, but network latency is part of the internet infrastructure and that’s where threading comes in.
Introducing Threading
Backblaze 4.0 includes threading in our Mac and Windows software clients. Threading allows Backblaze to optimize upload and downloads speeds if there is network latency in the internet connection between the computer and Backblaze.
While threading is pretty cool, let’s take a minute to see if this is something that can improve your upload speed. We created two different sets of questions you can use to see if you can benefit from threading: one set for “regular users” and another set for “advanced users.”
Regular Users
-
- Are you doing your initial data upload to Backblaze?
- Are you over 500 miles away from the Backblaze data center in Sacramento, California?
- Do you think your data upload is taking too long?
If you answered, “Yes” to all three of these questions, you’re a candidate for using threading. If your system is working and you haven’t noticed any issues uploading data, then you don’t need to do anything, you’re all set.
Advanced Users
-
- Are you doing your initial data upload to Backblaze?
- When you “ping” backblaze.com, is the round-trip (latency) over 10 milliseconds on average?
- When you run the Backblaze online speed test on your system, is the upload speed over five times more than that shown on your last Backblaze file upload as shown in the Backblaze Preferences >> Settings >> Performance Panel?
If you answered, “Yes” to all three of these questions, you’re a candidate for using threading. If your system is working and you haven’t noticed any issues uploading data, then you don’t need to do anything, you’re all set.
Just Want to Play?
Some of you may want to try to use threading even though everything is working fine. That’s fine, but remember to document your initial Backblaze Preferences Panel settings before you make changes. Threading will be system resource intensive and if it is not beneficial to use it, you’ll want to return Backblaze back to your initial settings.
Activating Threading
If you are a candidate for using the threading option, the very first thing you’ll need to do is to be running Backblaze 4.0. You can download and install the latest version of Backblaze from here: Backblaze Updates.
Once you are running Backblaze 4.0, here’s how to activate threading.
-
- Open the Backblaze Preferences Panel.
- Click Settings >> Performance.
- Uncheck “Automatic Throttle” if it is checked.
- Using the pull down menu, select “2” as the number of threads to use.
- Click “OK” to accept the change the return to the main preferences panel.
- Your backup will restart automatically after a few minutes, but you can click “Backup Now” to restart immediately.
Over the next several of hours you can monitor your system to see if you are benefiting from activating threading. For the most part, you’ll be monitoring network traffic and CPU utilization. If appropriate, you can repeat the procedure above increasing the number of threads by “1” each time until you have optimized your upload speed. For additional information on how to set up and monitor Backblaze threading, you can review the Backblaze Knowledge Base for your Windows system or your Mac system as needed.
Threading Can Really Help
One of our 4.0 beta testers is in Taiwan. Using the Backblaze speed test, he determined he should be getting at least 10.9 Mbps on upload. The speed test just pumps data up/down the wire making it a good proxy for the real speed of the connection. The tester then checked his “Last file transfer speed” on the Backblaze Preferences >> Settings >> Performance Panel.
The tester documented his observations of his data upload speed as he progressed through different thread settings. These are below.
Using Backblaze Online Backup 4.0 with 10 threads, he was able to back up nearly 100GB of data over a 24 hour period. To quote our Beta tester:
Yep, it’s blazing now.
While using 10 threads is extreme, it demonstrates the amount of unused bandwidth that can be utilized by using the threading option when appropriate.
What Backblaze Means by Threading
Typically, threading means adding multiple threads in the given process to perform more activities—in our case, to move blocks of encrypted data from a computer over the network to our data center. We instead use multiple processes, each of which autonomously moves an encrypted block of data from a computer over the network to our data center.
We found that implementing the use of multiple processes versus multiple threads was more straight forward and allowed us to manage the data upload process more effectively. In addition, if a process fails (is killed or dies), the recovery is to simply retransmit the data via a different process. While we use the terms threads and threading, we are actually creating and using multiple processes.
How Threading Works to Improve Upload Speed
Threading is designed to optimize bandwidth utilization when network latency is present. Our implementation works by having multiple processes independently sending encrypted blocks of data over the wire. The first process sends a block of data, and while it is waiting for an acknowledgement from our data center, the void is filled by a second process that sends its block of data right after the first process is done sending. Once the second block is done sending, a third process can send a block of data, and so on. Optimally, most of the voids are now filled with data being uploaded.
Obviously, the wire can get overstuffed with too many processes and the upload speed can slow down to be even less than when you started. That’s why, if you do activate threading, you need to start with two threads and work your way up, monitoring network resources along the way.
Caveats and Things to Remember
Threading does not magically create more upload bandwidth and does not allow you to change the laws of physics. If you have a 500 kbps upload connection and terabytes of data, it will still take months/years to upload all of your data. We can only use the upstream bandwidth that’s available and changing the number of threads most likely will not help if you have a slow internet connection.
Upload speeds and download speeds for most home and some business connections are usually different. You may have a 30 Mbps download connection, but only have a 5 Mbps upload connection. If you are not sure, you can run https://www.backblaze.com/speedtest/ to check your connection speed to/from Backblaze. Backblaze uses the upload bandwidth of your connection when uploading data, so that number is most important when backing up to Backblaze.
Remember, bits and bytes are different. Your upload and download speeds are measured in bits, as in, “I have an 8 Mbps connection,” but, your data is measured in bytes, as in, “I have 8MB of data to upload.” There are eight bits in one byte, so mathematically transferring 8MB of data over an 8 Mbps connection will take eight seconds, not one.
Improving Data Recovery Speed with Threading
Our engineers just couldn’t stop with threading the upload process. They’ve also threaded the Backblaze Downloader for both the Windows and Mac platforms. If you are not familiar with the Backblaze Downloader, it is a free utility you can use to download your Backblaze recovered files instead of using your web browser.
Just like using upload threading, the Backblaze Downloader will allow you to choose the number of threads (processes) you wish to use when downloading your Backblaze restore file.
Also, like upload threading, download threading uses multiple processes to fill in the latency voids to improve overall download speeds. The graphs below depict the increase in download throughput achieved by one of the beta testers when testing the Backblaze Downloader and going from one thread (the first graph) to four threads (the second graph).
Anyone using the Backblaze Downloader can use the threading option by selecting the number of threads from the pulldown menu. We highly recommend starting with two threads to see how the system behaves. One can always increase the number of threads the next time they use the Backblaze Downloader.
Data Center Improvements to Handle Threading
To support threading, the data center/server code was upgraded so multiple threads can simultaneously back up to multiple Backblaze Pods or multiple Pods with a Backblaze Vault in parallel. Each encrypted block of data in a thread (process) that is sent to Backblaze is independent of all others and is stored in the Vault or Pod that is ready to accept it. For example, your system has four threads each sending data to Backblaze. They could all arrive at the same time and be stored at the same time, in parallel, with no issues. The upgrades removed any potential bottlenecks in accepting and storing data that is sent to us, or recovering data for our customers.
Large Files Over Slow Connections
When backing up very large files, we now allow the file to take several months to be uploaded. For example, assume you have a large 19GB video file and a slow 500 Kbps upload internet connection. Uploading this file could take several days or weeks depending on when the system is on, etc. Previously, we would have given up on sending the file after a few days and tried another time; now, we’ll just keep working on that file until we have it completely uploaded.
De-duplication and Threading
You’ll be happy to know that threading works hand-in-hand with de-duplication. Each process (thread) being sent checks to see if the same files were previously uploaded or are waiting to be uploaded across multiple processes. Duplicate files are recorded, but not transmitted. This saves network bandwidth and data storage.
Changes in Supported Platforms
Backblaze Online Backup Version 4.0 can only be installed on Mac OS 10.6 or higher. Previous versions of Backblaze will continue to run on 10.5 and will be supported for the foreseeable future.
For Windows, Backblaze Online Backup 4.0 is supported on XP (32-bit), Vista (32-bit and 64-bit), Windows 7 (32-bit and 64 bit), Windows 8 (32-bit and 64 bit), and Windows 8.1 (32-bit and 64 bit).
Backblaze Online Backup 4.0 Details:
Version: 4.0.0 for both Mac OS and Windows.
Availability: 26 March 2015
Upgrade Methods:
- Immediately when performing a “Check for Updates.”
- Immediately as a download from Backblaze Updates.
- Immediately as the default download on backblaze.com.
- Auto-update will begin in the next couple of weeks.
Cost: Free as an update for all Backblaze customers.
Questions: Please contact Backblaze Support.
