![\"Load](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2016\/03\/blog-load-balance.jpg\")
<\/p>\n<\/p>\n
A few months ago we announced Backblaze B2 Cloud Storage,<\/a> our cloud storage product. Cloud storage presents different challenges versus cloud backup in the server environment. Load balancing is one such issue. Let’s take a look at the challenges that have come up, and our solutions.<\/p>\n A load balancer is a server or specialized device that distributes load among the servers that actually do work and reply to requests. In addition to allowing the work to be distributed among several servers, it also makes sure that requests only get sent to healthy servers that are prepared to handle them.<\/p>\n For Backblaze Personal Backup<\/a> and Backblaze Business Backup<\/a> products, we\u2019ve had it easy in terms of load balancing: We wrote the client, so we can just make it smart enough to ask us which server to talk to. No separate load balancer needed. The Backblaze products are also very tolerant of short outages, since the client will just upload the files slightly later.<\/p>\n For Backblaze B2, though, the clients are web browsers or programming language libraries like libcurl. They just make a single request for a file and expect an answer immediately. This means we need a load balancer both to distribute the load and allow us to take individual servers offline to update them.<\/p>\n The simplest and most flexible way to do load balancing is to have a pair of hosts, one active and one standby, that accept HTTPS connections from the client and create new connections to the server, then proxy the traffic back and forth between the two. This is usually referred to as \u201clayer 7, full proxy\u201d load balancing.<\/p>\n This doesn\u2019t generally require any special setup on the server, other than perhaps making sure it understands the x-forwarded-for header so it knows the actual client\u2019s IP address. It does have one big downside, though: The load balancer has to have enough bandwidth to handle every request and response in both directions, and enough CPU to handle TCP and SSL in both directions. Modern processors with AES-NI\u2014onboard AES encryption and decryption\u2014help a lot with this, but it can still quickly become a performance bottleneck when you\u2019re talking about transferring large files at 1 Gb\/s or higher.<\/p>\n Another option, if you want to reduce the burden on the load balancers, is layer 4 load balancing. The load balancers accept TCP connections from the client and create a new TCP session to the server, but they proxy through the HTTPS traffic inside the TCP session without decrypting or re-encrypting it. This still requires that the load balancer have enough bandwidth to handle all your traffic, but a lot less CPU compared to layer 7. Unfortunately, it also means that your servers don\u2019t really have a good way to see the original client\u2019s IP address short of hijacking the TCP options field with a proprietary extension.<\/p>\n All of this is adding a lot of work layered on top of a load balancer\u2019s basic purpose: to distribute client requests among multiple healthy back-end servers. To do this, the load balancer only needs to see the request and modify the destination at the outermost layers. No need to parse all the way to layer 7, and no need to even see the response. This is generally called Direct Server Return (DSR).<\/p>\n Especially when serving large files with SSL, DSR requires minimal amounts of bandwidth and CPU power on the load balancer. Because the source IP address is unchanged, the server can see the original client\u2019s IP without even needing an x-forwarded-for header. This does have a few tradeoffs, though: It requires a fairly complex setup not only on the load balancers, but also on the individual servers. In full-proxy modes the load balancer can intercept bad responses and retry the request on a different back-end server or display a friendlier error message to the client, but since the response bypasses the load balancer in DSR mode this isn\u2019t possible. This also makes health-checking tricky because there\u2019s no path for responses from the back-end host to the load balancer.<\/p>\n After some testing, we ended up settling on DSR. Although it\u2019s a lot more complicated to set up and maintain, it allows us to handle large amounts of traffic with minimal hardware. It also makes it easy to fulfill our goal of keeping user traffic encrypted even within our data center.<\/p>\n DSR load balancing requires two things:<\/p>\n A request packet will enter the network, and be routed to the load balancer. Once it arrives there, the load balancer leaves the source and destination IP addresses intact and instead modifies the destination MAC address to that of a server, then puts the packet back on the network. The network switch only understands MAC addresses, so it forwards the packet on to the correct server. When the packet arrives at the server\u2019s network interface, it checks to make sure the destination MAC address matches its own. It does, so accepts the packet. It then, separately, checks to see whether the destination IP address is one attached to it somehow. It is, even though the rest of the network doesn\u2019t know it, so it accepts the packet and passes it on to the application. The application then sends a response with the VIP as the source IP address and the client as the destination IP, so it\u2019s routed directly to the client without passing back through the load balancer.<\/p>\n DSR setup is very specific to each individual network setup, but we\u2019ll try to provide enough information that this can be adapted to most cases. The simplest way is probably to just pay a vendor like F5, A10, or Kemp to handle it. You\u2019ll still need the complex setup on the individual hosts, though, and the commercial options tend to be pretty pricey. We also tend to prefer open-source over black-box solutions, since they\u2019re more flexible and debuggable.<\/p>\n HAProxy and likely other applications can do DSR, but we ended up using IPVS<\/a> (formerly known as LVS). The core packet routing of IPVS is actually part of the Linux kernel, and then various user-space utilities are used for health checks and other management. For user-space management, there\u2019s a number of other good options like Keepalived<\/a>, Ldirectord, Piranha, and Google\u2019s recently-released Seesaw. We ended up choosing Keepalived because we also wanted VRRP support for failing over between load balancers, and because it\u2019s both simple and stable\/mature.<\/p>\n Good news! If your kernel is 2.6.10 or newer (and it almost certainly is), IPVS is already included. If The Keepalived load-balancing config is fairly straightforward: a virtual_server section with a real_server section inside of it for each back-end server. Most of the rest depends on your specific needs, but you\u2019ll want to set lb_kind to \u201cDR.\u201d You can use SSL_GET as a simple health checker, but we use \u201cMISC_CHECK\u201d with a custom script, which lets us stop sending new traffic to a server that\u2019s shutting down by setting its weight to 0.<\/p>\n The host config is where things get a bit more complicated. The important part is that the VIP address is assigned to an interface, but the server isn\u2019t sending ARP replies about it. There\u2019s a few ways to do this that work about equally well, but we use arptables rules defined in Once the arptables rules are in place, you\u2019ll want to add the actual address to the interface:<\/p>\n If your backend server doesn\u2019t have an actual external IP and normally talks to the outside via NAT, you will need to create a source-based route, also in the interfaces config:<\/p>\n Finally, make sure your web server (or other daemon) is listening on specifically, or on all addresses (0.0.0.0 or :::).<\/p>\n Backblaze’s firsthand account on the difference between load balancing in cloud backup and cloud storage server environments.<\/p>\n","protected":false},"author":101,"featured_media":52611,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":"","jetpack_post_was_ever_published":false},"categories":[7],"tags":[291,468,373],"class_list":["post-52441","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-storage","tag-b2","tag-b2cloud","tag-developer","entry"],"acf":[],"yoast_head":"\nOption 1: Layer 7, Full Proxy<\/h2>\n
Option 2: Layer 4, Full Proxy<\/h2>\n
Option 3: DSR<\/h2>\n
How Does It Work?<\/h3>\n
\n
\n![\"blog-network-diagram\"](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2016\/03\/blog-network-diagram.png\")
<\/a><\/p>\nHow Do I Set It Up?<\/h3>\n
Setting Up IPVS and Keepalived<\/h4>\n
\/proc\/net\/ip_vs exists<\/code>, it\u2019s already loaded. If not, modprobe ip_vs<\/code> will load the module. Most distributions will probably compile it as a kernel module, but your results with VPS providers may vary. At this point, you\u2019ll probably also want to install the ipvsadm utility so you can manually inspect and modify the IPVS config.<\/p>\n\/etc\/network\/interfaces<\/code>:<\/p>\npre-up \/sbin\/arptables -I INPUT -j DROP -d <VIPADDRESS>
\npre-up \/sbin\/arptables -I OUTPUT -j mangle -s <VIPADDRESS> \u2010\u2010mangle-ip-s <SERVERIPADDRESS>
\npre-down \/sbin\/arptables -D INPUT -j DROP -d <VIPADDRESS>
\npre-down \/sbin\/arptables -D OUTPUT -j mangle -s <VIPADDRESS> \u2010\u2010mangle-ip-s <SERVERIPADDRESS><\/code><\/p>\npost-up \/sbin\/ip addr add <VIPADDRESS>\/32 dev $LOGICAL
\npre-down \/sbin\/ip addr del <VIPADDRRESS>\/32 dev $LOGICAL<\/code><\/p>\npost-up \/sbin\/ip rule add from <VIPADDRESS> lookup 200
\npre-down \/sbin\/ip rule del from <VIPADDRESS> lookup 200
\npost-up \/sbin\/ip route add default via <VIPADDRESS> dev $LOGICAL table 200
\npre-down \/sbin\/ip route del default via <VIPADDRESS> dev $LOGICAL table 200<\/code><\/p>\nIt\u2019s Not Working!<\/h3>\n
\n
\/proc\/sys\/net\/ipv4\/conf\/*\/rp_filter<\/code> and make sure it\u2019s not set to 1 anywhere.<\/li>\ntcpdump -e -n host <\/code> on the load balancer and make sure that the requests are reaching the load balancer with a destination IP of and a destination MAC address belonging to the load balancer, then leaving again with the same source and destination IP but the MAC address of the back-end server.<\/li>\nipvsadm<\/code> on the load balancer and make sure IPVS is configured<\/li>\ntcpdump -e -n host <\/code> on the server and make sure that requests are arriving with a destination IP of , and leaving again with a source of and a destination of<\/li>\nip route get from<\/code> to make sure the host has a non-NATTED return route to the outside.<\/li>\nip neigh show <\/code>, where is the “via\u201d IP from the previous \u201cip route get\u201d command, to make sure the host knows how to reach the gateway.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"