![\"A](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2023\/05\/bb-bh-CPython-3.11-vs-3.9-nogil-I-O-performance.png\")
<\/p>\n<\/p>\n
Last year, our team published a history of the Python GIL<\/a>. We tapped two contributors, Barry Warsaw<\/a>, a longtime Python core developer, and Pawel Polewicz<\/a>, a backend software developer and longtime Python user, to help us write the post. <\/p>\n Today, Pawel is back to revisit the original inspiration for the post: the experiments he did testing different versions of Python with the Backblaze B2 CLI<\/a>. <\/p>\n If you find the results of Pawel\u2019s speed tests useful, sign up to get more developer content every month in our Backblaze Developer Newsletter<\/a>. We\u2019ll let Pawel take it from here.<\/p>\n \u2014The Editors\n<\/p><\/div>\n I was setting up and testing a backup solution for one of my clients when I noticed a couple of interesting things I\u2019d like to share today. I realized by using Python 3.9-nogil, I could increase I\/O performance by 10x. I\u2019ll get into the tests themselves, but first let me tell you why I\u2019m telling this story on the Backblaze blog.\n<\/p>\n I use Backblaze B2 Cloud Storage<\/a> for disaster recovery for myself and my clients for a few reasons:<\/p>\n Naturally, I always want to make sure my clients can get their backup data out of cloud storage fast should they need to. This brings us to \u201cThe Experiment.\u201d<\/p>\n I ran a speed test to see how quickly we could get large files back from Backblaze B2 using the B2 CLI<\/a>. To my surprise, I’ve found that it depends on the Python version.<\/p>\n The chart below shows download speeds from different Python versions, 3.6 to 3.11, for both single-file and multi-file downloads.<\/p>\n The Backblaze B2 CLI is fetching data from the B2 Cloud Storage server using Python\u2019s Requests library<\/a>. It then saves it on a local storage device using Python threads\u2014one writer thread per file. In this type of workload, the newer versions of Python are much faster than the older ones\u2014developers of CPython (the standard implementation of the Python programming language) have been working hard on performance for many years. CPython 3.10 had the highest performance improvement from the official releases I\u2019ve tested. CPython 3.11 is almost twice as fast as 3.6!<\/p>\n GIL stands for global interpreter lock. You can check out the history of the GIL<\/a> in the post from last year for a deep dive, but essentially, the GIL is a lock that allows only a single operating system thread to run the central Python bytecode interpreter loop. It serves to serialize operations involving the Python bytecode interpreter\u2014that is, to run tasks in an order\u2014without which developers would need to implement fine grained locks to prevent one thread from overriding the state set by another thread. <\/p>\n Don\u2019t worry\u2014here\u2019s a diagram. <\/p>\n The GIL prevents multiple threads from mutating this state at the same time, which is a good thing as it prevents data corruption, but unfortunately it also prevents any Python code from running in other threads (regardless of whether they would mutate a shared state or not). <\/p>\n I ran one more test using the “nogil” fork of CPython 3.9. I had heard it improves performance in some cases, so I wanted to try it out to see how much faster my program would be without GIL. <\/p>\n The results of that test were added to the tests run on versions of unmodified CPython and you can see them below:<\/p>\n In this case not being limited by GIL has quite an effect! Most performance benchmarks I\u2019ve seen show how fast the CPython test suite is, but some Python programs move data around. For this type of usage, 3.9-nogil was 2.5 or 10 times faster (for single and multiple files, respectively) on the test than unmodified CPython 3.9.<\/p>\n A simple test running parallel writes on the RAID-0 array we\u2019ve set up on an AWS EC2 i3en.24xlarge instance\u2014a monster VM, with 96 virtual CPUs, 768 GiB RAM and 8 x 7500GB of NVMe SSD storage<\/a>\u2014shows that the bottleneck is not in userspace. The bottleneck is likely a combination of filesystem, raid driver, and the storage device. A single I\/O-heavy Python process outperformed one of the fastest virtual servers you can get in 2023, and enabling nogil required just one change\u2014the FROM<\/strong> line of the Dockerfile.<\/p>\n For a single file, POSIX doesn\u2019t guarantee consistency of writes if those are done from different threads (or processes)\u2014that\u2019s why the B2 Cloud Storage CLI uses a single writer thread for each file while the other threads are getting data off the network and passing it to the writer using a queue.Queue<\/a> object. Using a multiprocessing.Queue<\/a> in the same place results in degraded performance (approximately -15%).<\/p>\n The cool thing about threading is that it\u2019s easy to learn. You can take almost any synchronous code and run it in threads in a few minutes. Using something like asyncio<\/a> or multiprocessing<\/a> is not so easy. In fact, whenever I tried multiprocessing, the serialization overhead was so high that the entire program slowed down instead of speeding up. As for asyncio, it won\u2019t make Python run on 20 cores, and the cost of rewriting a program based on Requests is prohibitive. Many libraries do not support async anyway and the only way to make them work with async is to wrap them in a thread. Performance of clean async code is known to be higher than threads, but if you mix the async code with threading code, you lose this performance gain.<\/p>\n Threads might be easy in comparison to other ways of making your program concurrent, but even that\u2019s a high bar. While some of us may feel confident enough to go around limitations of Python by using asyncio with uvloop<\/a> or writing custom extensions in C, not everyone can do that. Case in point: over the last three years I\u2019ve challenged 1622 applicants to a senior Python backend developer job opening with a very basic task using Python threads. There was more than enough time, but only 30% of the candidates managed to complete it. <\/p>\n On January 9, 2023, Sam Gross<\/a> (the author of the nogil branch) submitted [PEP-703<\/a>]\u2014an official proposal to include the nogil mode in CPython. I hope that it will be accepted and that one day nogil will be merged into mainline, so that Python can exceed single core performance when commanded by lots of users of Python and not just those who are talented and lucky enough to be able to benefit from asyncio, multiprocessing, or custom extensions written in C.<\/p>\n","protected":false},"excerpt":{"rendered":" We tested different version of Python with the Backblaze B2 CLI, and the results are in. Check out which version did best. <\/p>\n","protected":false},"author":168,"featured_media":108698,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[7,483],"tags":[468],"class_list":["post-108695","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-storage","category-tech-lab","tag-b2cloud","entry"],"acf":[],"yoast_head":"\n\n
The Experiment: Speed Testing the Backblaze B2 CLI With Different Python Versions<\/h2>\n
![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2023\/05\/bb-bh-CPython-3.11-vs-3.9-nogil-I-O-performance_Infographic-2_cropped.png\")
<\/p>\nWhat\u2019s Going On Under the Hood?<\/h3>\n
Refresher: What\u2019s the GIL Again?<\/h3>\n
![\"\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2022\/05\/GIL-Serializing-Reference-Count.png\")
How Did \u201cnogil\u201d Perform?<\/h2>\n
![\"Chart](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2023\/05\/bb-bh-CPython-3.11-vs-3.9-nogil-I-O-performance_Infographic-1_cropped.png\")
<\/p>\nWhy Isn\u2019t nogil Even Faster?<\/h2>\n
Why Not Use Multiprocessing?<\/h2>\n
But Threads Can Be Hard Too!<\/h2>\n
What\u2019s Next for nogil?<\/h2>\n