Obama Backs Backblaze – Throttling is Bad

By | November 11th, 2014

blog-net-neutrality

President Obama is now on the record stating that Internet providers have “a legal obligation not to block or limit your access to a website.” We agree and want to thank the President for following in our footsteps by telling the world that throttling is bad. Backblaze itself does not actively throttle our customers. The President went on to say, “In plain English, I’m asking [the FCC] to recognize that for most Americans, the Internet has become an essential part of everyday communication and everyday life.” We couldn’t agree more, and wholeheartedly wish for increased bandwidth to all our users.

No Net Neutrality Bites Backblaze

Backblaze and by proxy our customers were victimized by the absence of Net Neutrality during the Comcast-Netflix War from November 2013 through February 2014. Here’s what went down. Beginning in late November 2013, a small number of our customers started reporting slow data restores – downloads from a Backblaze data center to their computer. Upon investing their claims, we examined our internal network, reconfigured our restore servers and changed the caching code on our servers to “fix” the problem.

None of the changes seemed to work as the number of support calls continued to rise.

The First Smoking Gun

We continued to investigate, but the trouble was nothing was consistent. People were using different versions of the product, different platforms, different versions of their operating systems and they were located in different parts of the country and around the world. Digging in further, we found that there was a slow down in downloaded network traffic on our Comcast/Cogent and Comcast/XO connections. They were being severely throttled. All of our other data center network connections were fine, except those two.

The Second Smoking Gun

In early January, Backblaze analyses compared the download speeds of our Oakland data center versus our Sacramento data center over comparable 24 hour periods. The Sacramento data center is serviced by the Comcast/Cogent and Comcast/XO connections. The Oakland data center uses Abovenet and other providers.

Oakland-DC-Network

Oakland Data Center

Sacramento-DC-Network

Sacramento Data Center

Observe that download network traffic from the Oakland data center is nearly flat (the thick black line). In the Sacramento data center download network traffic decreased starting at 14:00 (2:00pm Pacific Standard Time) and continued decreasing until about 22:00 (10:00pm PST), then recovered. This pattern continued each day.

Rerouting Around The Throttle

In mid-January we began rerouting download traffic from our Sacramento data center to our Oakland data center, then from Oakland to the outside using Abovenet. Customer download speeds from the Sacramento data center to the customer increased 10x versus the throttled speeds we were getting previously. The reroutes were working, but we couldn’t get a good answer as to why the Comcast/Cogent and Comcast/XO connections in the Sacramento data center were “broken”.

The Final Smoking Gun

On February 23, 2014, Comcast and Netflix reach a deal on routing Netflix network traffic. On March 1st, Backblaze removed our reroutes and returned to our original network configuration prior to the War. Downloads from our Sacramento data center were now running once again over the Comcast/Cogent and Comcast/XO network connections.

The result: All downloads now occur at normal speeds, including downloads during the 2pm to 8pm timeframe. Once the Comcast/Netflix deal was done, it took less than a week for the problem to disappear.

TL:DR

To summarize, Comcast had a problem with the amount of traffic Netflix was using. Yet, services completely unrelated to Netflix were punished, meaning consumers and businesses – innocent bystanders – unrelated to Netflix were punished, and all this occurred without notice or explanation from Comcast. That should not be the future of the Internet.

We have prepared a document titled, “Findings and Analysis of Slowdowns on the Comcast Network,” which is a chronological review used to support the assertions in this post.

 

Obama_wheeler

 

Brian Wilson
I completed my undergraduate at Oregon State University in 1990, then completed a Stanford Masters degree in 1991. Ever since then I've worked at various companies as a software engineer, in the last few years starting my own software startups called MailFrontier (started in 2002) and most recently Backblaze (started in 2007).

I have a personal web site at http://www.ski-epic.com that I started in 1999 (it was originally just for one vacation, but it kept growing) where I put up my vacation pictures and videos. Nothing professional, it's all just for fun.

In my spare time I enjoy skiing, motorcycling, and boating. I have been lucky enough to travel to a few countries, and I enjoy scouting out new places for the first time.

Follow Brian on:
Twitter: @brianwski
YouTube: brianwski
LinkedIn: brianwski
Google+: brianwski
Reddit: brianwski
Category:  Backblaze Bits
  • Per G

    I don’t get why Backblaze didn’t team up with other non-Netflix customers and filed a massive class-action lawsuit against Comcast for loss of business? – You can prove it happened and they thus breached their contract with you. I take it their contract with you does not hold an exemption for intentional throttling as part of a contract struggle with someone else?

    Sure you may need to switch providers but you can hurt Comcast badly and take them for a ton of money, possibly even kill the company. Perhaps that will teach them – and others – a lesson?

    I don’t think new laws are needed here. Just use the current system to hold companies responsible for their actions and take them apart in court if they fail to behave properly.

  • Andrew Vickers

    I also have to say, the photo included with this blog post is about as far from reality as could be. For as long as I have been dealing with ISPs (20+ years), there has never been a single time where the ISP has demanded that I pay extra money. I have never felt as though I am being held to ransom.

    On the contrary, My ISPs have regularly increased the speed/bandwidth of my connection at NO additional cost, and more recently, they have given me the choice to increase (or decrease) my level of service.

  • Henrique Vicente

    Last time the government intervened the Internet here in my country I had to pay for Internet access twice.

    One time for the last-mile provider, a second time (and based on how much I paid for the last-mile provider) to a content provider.

    If you ask for government to step in to “fix the market” (and for problems that doesn’t exists), you end up creating a bad environment… or in other words: government failures…

  • bsolestis

    I love BackBlaze. I’m a paying customer and intend to remain so for years to come. That said, if you don’t understand that Net Neutrality tramples providers’ property rights, then you don’t understand property rights…and property rights are the foundation of the free market you (Brian) and I advocate.

    Who owns the transmission wires? Who bought the miles of Ethernet and fiber cable, and the switches that tie it all together, and the ultra-secure facilities to house those servers, and pays employees to string the wires, to bury the cable, to set up the servers and to maintain all that equipment, and to take installation orders and fix problems of its customers?

    Who has the right to tell a vendor that after having does all that, they’re not allowed to throttle or promote or sell slower or faster access as they choose? No one, that’s who.

    I remain convinced that this whole Net Neutrality movement was started by Bittorrenters and gamers who were using far more bandwidth than they were paying for, and got upset when their providers realized it and started making them pay for it, or else reducing their bandwidth. “Net Neutrality” is a cute euphemism, BTW, it should really be the “Internet Providers Property Abrogration Act”.

    Either you’re for free enterprise, or you’re not — and I am. I’m for it for you, and for me, and the grocery stores and Time/Warner and everybody else. Everybody should be free to offer whatever services they like, and on whatever terms — and to accept or reject such offers. The only room for complaint is if the providers don’t give you the speed they promise or fail to fulfill the conditions of their contract.

    • Milk Manson

      Bittorrenters and gamers, huh? Think that one through, Einstein. Bittorrenters and gamers can’t organize their way out of Mom and Dad’s basement, but they orchestrated this whole net neutrality thing.

      You are the best advertising Net Neutrality ever had. Keep it up, please.

  • Steve C

    Seems to me as someone with just enough understanding of network equipment to be dangerous (but not enough to be anything of an expert) that the measurement missing from the “Findings and Analysis…” paper was a report of the TCP window size, across all BackBlaze restores, by block of IP addresses to which the restore is being done (ie the ISP used by BackBlaze’s customer). I realize it might take some work to capture such a metric, and might be difficult to do without degrading performance if TCP offload type NICs are being used, but even sampled (sflow-like) data would be sufficiently useful. Having established with primary measurements on TCP itself that a bottleneck was resulting in poor customer outcomes — which I just said Backblaze could do before its phone rang — Traceroute could then be used to try to identify the bottleneck which was causing TCP to fall back to such a low transfer rate.

    The problem with relying solely on Traceroute is that the ping itself goes to the supervisor processor of the router at each step it can see (it only sees routers and other Layer 3 boxes, not switches or transparent middleboxes like firewalls). The supervisor processor gets busy (congests) almost independent of the data path a packet would normally take through the router, and ping responses are certainly not the highest priority work the supervisor does. So it can give some very anomalous results.

    It’s not clear to me from this blog post and analysis paper that Backblaze has sufficient skill in networking to understand the significance of its traffic on the internet backbone, which is central to its business model. Do you have anyone who at least monitors the relevant IETF committees and understands issues like bufferbloat, RED, active queue management (AQM), congestion in network tunnels, and the like, and how solutions to those are thought about at a global level?

  • Paul O’Brien

    Thanks for the work you do Brian, and all the folks at Backblaze. You’ve got one heck of a product and a generous hand with the tools you develop and the results of your research.

    Even though the complainy-pants stalking this blog post might downvote this comment to the ninth circle of hell, I just want you to know that all the folks at my company rely on an Internet that works the way it’s designed. When the only bandwidth provider in town decides to hold hostage my connection to a business service provider or even an entertainment service provider I use (e.g. Backblaze, Netflix), and when I have no available recourse, yes, it’s time for the law to step in. When I notice site-specific service degradation and complain, Comcast does nothing to fix the problem. Turns out they are the problem, either over-selling their network capacity or purposefully throttling my pipe.

    Not responding to customer complaints is not good business. Overselling service is piss poor business. But extortion is not business at all. It’s a crime. It doesn’t just affect Backblaze or Netflix, it affects me and my co-workers. There’s no line to call Mr. Invisible Hand to seek recourse. There’s not even another competitor to provide the bandwidth I need. However, there is a law-making and law-enforcing authority with the power to stop a corporation from accepting my money and then using my subscription as a hostage to grab more money from the folks I need to talk to on the other end of the line. I’m already paying for that bandwidth; Backblaze and Netflix are already paying for the bandwidth they use to supply their data to the internet. I’m not paying for information from Comcast, I’m paying for a stable and predictable connection to a web of networks hosting other entities who already pay to provide the information I pay them for.

    • Brian Wallace

      “I just want you to know that all the folks at my company rely on an Internet that works the way it’s designed.”

      How exactly was the interent designed to work? Your company works nicely with the curent system. You are asking the government to use its power to keep the system exactly how it is forever so that you don’t ever have to change what you have set up. You are askig for a “referee” to create and uphold rules so that they fit your current needs.

      If that is the case you must also assume that your needs will never change, things aren’t ever going to get better then they are right now and that every one else (including that referee) has the the same agenda and needs that you do.

      • Paul O’Brien

        Thanks for the reply, Brian!

        Unfortunately an explanation of the history, design, and workings of the Internet are far beyond my available time to explain here. Wikipedia is chock-full of mostly-accurate information on the subject. MIT and Stanford offers several free online courses that would also help out.

        With all due respect, I reject your claims above on their face. I assert that I do not want to have rules so that they fit my current needs. That would be stupid, shortsighted, and selfish. Networking technology and data transfer over long distances are tools that improve over time, and I find it reasonable that the folks who provide access to the tool should be disallowed from breaking the tool to suit an end like financial gain.

        Besides, it’s not folks like me who want everything to stay the same. Isn’t it Comcast, et al. who don’t want changes to the rules because the current system suits their needs?

  • Robbll

    We do not need the government screwing up another part of our lives…they have done enough damage already. Let the free market solve the throttling problem.

  • Rhonda Koenig

    Government is NEVER the answer. Period.

  • Very glad to see others here pointing out that having government step in to require or outlaw certain behaviors according to its desires is never the answer. Only when basic rights are being violated should government get involved. Here are a couple of good pieces on why net neutrality is a bad idea:

    https://www.theobjectivestandard.com/issues/winter-2008/net-neutrality/
    http://capitalismmagazine.com/2009/09/net-neutrality-vs-internet-freedom/

    I love Backblaze and hope it will reconsider its position.

    • > having government step in to require or outlaw certain behaviors
      > according to its desires is never the answer

      Net neutrality is on the side of a free market. You seem to think it places lots of controls and rules on things like most laws, it doesn’t. It is very simple and it comes down to: all data must be treated equally.

      Look, I have libertarian tendencies, and I agree with you that MOST laws nowadays are straight up badly written and get in the way of rational policy. But in a small subset of situations a law can PRESERVE competition and do no harm. This is one of those cases.

      If you are against net neutrality, you are against the free market and you are against competition.

      Finally, I would encourage people to read this Oatmeal explanation, which is slightly condescending and snarky, but I feel does a good job of making the information approachable to people who are not network engineers and not politicians: http://theoatmeal.com/blog/net_neutrality

      • Andrew Vickers

        Net Neutrality is not a “free market” solution. Bandwidth is a commodity, not a right. It requires resources and effort to provide bandwidth to consumers, bandwidth is not inherent in human nature, like free speech or liberty. Advocating equal access to all consumers, regardless of how much they need, or how much they pay is akin to giving everyone 5 gallons of gas a week, or 2 loaves of bread a month. It is a form of rationing.

        Not all data is equal. Some data SHOULD be given priority, and there should be a way for networks to know that, for example, real-time voice communication is more sensitive to lag than downloading a software patch at 3:00am. That remote surgery should be treated differently to backing up the family photos. The most “Free Market” way to achieve this is by allowing those with the greatest need to pay for better service. Using the government to force everyone to get the same results is not “Fee Market”, it is the opposite.

        I found JLivingood’s comment (below) most interesting. Is it true that the specific issue with the Comcast/Cogent “throttling” was resolved when Cogent introduced prioritization to their network and dropped Netflix to Priority 3, while retail customers (including BackBlaze) remained at Priority 1? If so, isn’t this exactly the opposite of what would happen under the Net Neutrality rules? If true, isn’t this counter-intuitive example the epitome of Hayek: “The curious task of economics is to demonstrate to men what little they know about what they imagine they can design”?

        • Andrew Vickers

          In addition, not only did I find the Oatmeal article condescending, I also found it to be a straw-man fallacy.

          Instead of explaining how Net Neutrality works, it presents an “extreme scenario, and not a very likely one” as its argument. The article completely ignores the moderate, and extremely likely situation that is the day-to-day reality that the internet already works EXTREMELY WELL (albeit with the occasional hiccups like the one that affected “a small number” of Backblaze customers).

          By attempting to fix the “extreme” and “not very likely” scenarios, net Neutrality will break something that already works very well (almost) all the time.

      • Whether you support or oppose net neutrality, you can’t say it is an instance of the free market in operation. A free market is one in which force is absent and where buyers and sellers *alone* get to decide between themselves what is acceptable to both sides. It is not one where the government dictates that “data must be treated equally.” It is not one where a bureaucrat can decide an offering is “unfair” and ban it.

  • JLivingood

    Backblaze: I highly encourage you to read http://www.measurementlab.net/blog?date=october%202014 and note the M-Lab discovery of a prioritization change on the Cogent network, introduced the same week as the Netflix-Comcast deal. See also http://blog.streamingmedia.com/2014/11/cogent-now-admits-slowed-netflixs-traffic-creating-fast-lane-slow-lane.html.

    Backblaze noticed an improvement basically days after the Netflix-Comcast deal was announced —> prior to sufficient direct connect capacity coming online to explain a sudden improvement by 3/1/14 (in one week as you say).

    You write in your report “After Comcast spent months claiming how incredibly difficult and expensive it would be to provide sufficient bandwidth for Netflix traffic due to the need for heavy investment in infrastructure, being able to completely deploy all necessary equipment and fix all these issues in the span of one week after signing a deal seems suspect.”

    What really happened that week is that, as Cogent said a week or so ago, they introduced multiple classes of priority. Retail customers (presumably like Backblaze, representing 3% of their traffic and 53% of their revenue) were given the highest priority. Wholesale customers (presumably like Netflix) were put in a low priority class. Since you paid a higher retail rate your traffic got higher priority.

    As to why they introduced this prioritization scheme the very week of this Netflix-Comcast deal, only they would know. But obviously it has helped create some incorrect perceptions.

    Jason – Comcast Engineering

    • > it has helped create some incorrect perceptions – Comcast Engineering

      I am a Comcast customer (internet only – I cut the TV cable years ago), and very much enjoy Comcast download speeds of 100 Mbits/sec in my home. I’m also 100 percent “legal” – I pay for and watch Netflix and Vudu and Amazon streaming and iTunes and I never steal any media. But you are either clueless or being disingenuous when you say there are incorrect perceptions…

      Let’s start here: prioritization (as opposed to throttling) does not affect a system when there is enough bandwidth. Simple example: you have 3 packets, each packet takes 5 milliseconds to deliver and you don’t have any other packets arriving in the next 60 seconds. It doesn’t matter if you prioritize packet 2 higher, it gets delivered first in 5 milliseconds, then packet 1 gets delivered, then packet 3 gets delivered. All packets get delivered within 15 milliseconds.

      Ok, so for prioritization to have much of any effect means somehow a bottleneck/throttle/slowdown appeared where you keep delivering packets of type 2 causing the starvation of packets of type 3 never getting delivered. Now we’re getting down to the heart of the matter – there are two possible reasons for a slowdown and neither one is puppies and sunbeams: 1) Comcast Engineering really screwed up, like really really REALLY badly and couldn’t figure out how to deliver packets to their destinations fast enough, or 2) Comcast engineers knew how to fix it or no choke point existed but still decided to artificially throttle (possibly because told by Comcast management to throttle?)

      You seem to be implying it was the Cogent’s side causing the slowdown, but we saw severe slowdown with *BOTH* Comcast/Cogent and Comcast/XO. Did XO also change their prioritization scheme the same week Cogent did? The common factor here is Comcast. Also, even if Cogent changed something and the bandwidth soared afterwards, why did it drop so much in the weeks leading up to that moment? Why did it drop for XO?

      In my opinion, if Comcast was well run, it would ADVOCATE for net neutrality and focus on doing the very best job of: 1) deliver packets as fast as possible today, and 2) build out infrastructure on a constant ramp because more bandwidth will be required in the future. Comcast made $13.6 billion in PROFIT last year, right? And this profit IS GROWING, right? You guys should be plowing some of that profit into keeping enough infrastructure and bandwidth capacity around to deliver all the packets now and in the future. Comcast should not fear net neutrality unless they plan to run out of bandwidth.

      Finally, if it really was a prioritization change in Cogent, this points to why we need net neutrality – you are admitting and showing the problems with prioritization of one type of traffic over another (in your case one source of “retail customers” over another). What we are dealing with here is not enough bandwidth (the only other possibility is super sleazy throttling). “Prioritization” doesn’t affect throughput unless there isn’t enough bandwidth. “Throttling” affects throughput even if there is plenty of bandwidth. Which one does Comcast and Cogent have? Enough bandwidth or not enough bandwidth? If you don’t have enough bandwidth capacity, raise prices or take some of your profits and build more capacity, because we’re going to need more next year, and even more the year after that.

      • JLivingood

        > Let’s start here: prioritization (as opposed to throttling) does not affect a system when there is enough bandwidth.

        Correct. Cogent lacked sufficient bandwidth to send 1/3 of the Internet’s traffic at peak hour to many networks. No debate there. It seems fair to debate who’s “fault” that was, but that’s a different matter / different blog about how interconnect has historically worked, how it should work in the future, and what your views on that may be.

        > there are two possible reasons for a slowdown and neither one is puppies and sunbeams: 1) Comcast Engineering really screwed up, like really really REALLY badly and couldn’t figure out how to deliver packets to their destinations fast enough, or 2) Comcast engineers knew how to fix it or no choke point existed but still decided to artificially throttle (possibly because told by Comcast management to throttle?)

        There is another possible reason as well:

        3) Cogent took on the Netflix traffic and knew their existing core network & interconnects could not handle 1/3 of the Internet

        And please recall that prior to Netflix moving to their own CDN they were successfully delivered by CDNs such as Akamai at very high quality to many ISPs.

        > You seem to be implying it was the Cogent’s side causing the slowdown

        No, I am saying something different. Your paper suggests in the page 11-12 conclusion that the magical speed up around the end of February to March 1st was due to a deal between Comcast and Netflix. I’m saying that is not so – the sudden performance improvement was due to a Cogent prioritization system implemented at that time (which BTW was no disclosed until early Nov 2014 under pressure by M-Lab). A fair question may be why they waited so long to implement it rather than that week in time.

        > In my opinion, if Comcast was well run, it would ADVOCATE for net neutrality

        And we do agree with all the Open Internet rules – see http://corporate.comcast.com/comcast-voices/surprise-we-agree-with-the-presidents-principles-on-net-neutrality-reiterating-our-strong-support-for-the-open-internet

        > 2) build out infrastructure on a constant ramp because more bandwidth will be required in the future.

        And we invest huge amounts every year to do that, and keep increasing the speeds for customers. As a customer (thank you!) you very likely have 50 Mbps or 105 Mbps service today, which was likely 15 Mbps – 30 Mbps a year or so ago.

        But as you know, every company connected to this network of networks needs to invest in their own capacity. What if Backblaze only bought a 1 Mbps connection from your data center and then complained ISPs were throttling you to 1 Mbps total? If you need to transfer a peak of 10 Gbps then you invest in capacity that buys 10 Gbps (from anyone that can truly offer it). It is hard to imagine it is reasonable to say that you’ll only buy 1 Mbps and ask each ISP to pay for that 10 Gbps connection on the basis that ‘your customers are trying to use Backblaze so provide that bandwidth free’ or something like that.

        > Finally, if it really was a prioritization change in Cogent, this points to why we need net neutrality – you are admitting and showing the problems with prioritization of one type of traffic over another (in your case one source of “retail customers” over another).

        I agree that there needs to be strong Open Internet rules that apply to all ISPs, including Cogent, are good (as our blog post said). And in particular transparency is good – it shouldn’t take M-Lab to smoke out a major priority change like this – it should be openly disclosed.

        >”Prioritization” doesn’t affect throughput unless there isn’t enough bandwidth. “Throttling” affects throughput even if there is plenty of bandwidth. Which one does Comcast and Cogent have?

        Throttling is when one party puts in place a system to actively shape traffic down to a set level. This is not the case here. This is a case of insufficient capacity. Fair people can disagree on where to allocate the blame for that.

        • > Cogent lacked sufficient bandwidth to send 1/3 of the Internet’s
          > traffic at peak hour to many networks. No debate there.
          > It seems fair to debate who’s “fault” that was

          It is simply too suspicious that Netflix payed Comcast then the problem went away. If it was Cogent, why didn’t Netflix pay Cogent to make the problem go away? A really straight forward system is you pay the people you are connected to a flat rate per packet (or pay for peak bandwidth which is very common). Not pay based on the source of the traffic or the content of the traffic.

          > And we invest huge amounts every year to [increase capacity], and keep increasing the speeds for customers.

          I have seen your improvements in the last few years, but those of us technical enough know it was probably because Google Fiber has Comcast running scared. Comcast offers 105 Mbits download only while Google Fiber offers SYMMETRIC 1,000 Mbits for less money. Comcast improvements have not historically been enough and it’s late coming and Comcast has one HECK of a huge task ahead. Other urban areas in other countries have more bandwidth for cheaper rates than our (USA) urban areas which have more density and higher broadband penetration rates. (Like San Francisco and San Jose.) Yet Comcast pockets more profit each year. Comcast is obviously a company that is taking profits at the expense of not investing enough in infrastructure, all of us technical people can see it. Brian Roberts (Comcast CEO) is 55 years old, so I assume he’s just planning on milking the existing business as it dies for a couple more years before he retires – and he doesn’t care that he is destroying Comcast in the long run.

          The ability to get internet into homes is probably the single most important and most beautiful thing that has ever happened to the human race so far. The ability to get broadcast TV (NBC, CBS, etc) over the old Comcast cable is **NOT** the future, and Comcast needs to focus on internet and stop clutching onto the dying business of broadcast TV. Don’t be Kodak. Kodak tried to hold onto the film business as digital took over. They clutched that formerly lucrative product all the way into the grave.

          Comcast is desperately trying to create an uneven playing field for Netflix. Netflix represents the future – video on demand over the internet. They have PROVEN it is possible and possible on a huge scale. Broadcast TV on Comcast can’t compete on a level field, so you are cheating (throttling Netflix and making them pay you) so consumers get worse deals. Net Neutrality will prevent you from cheating. You don’t know it, but this will also preserve your life, you think you’ll make more money by cheating, but in the long run you’ll just go out of business. Net Neutrality is good for Comcast. I say let the best business win. Let the competition be fair. I’m for Net Neutrality.

          • JLivingood

            > It is simply too suspicious that Netflix payed Comcast then the problem went away.

            I agree it was odd timing. But the action was on Cogent’s part and IMHO timed to make an idealogical point. Until M-Lab found the change Cogent made on their network, everyone ascribed it to us.

            > If it was Cogent, why didn’t Netflix pay Cogent to make the problem go away?

            Great question for Netflix.

            > I have seen your improvements in the last few years, but those of us technical enough know it was probably because Google Fiber has Comcast running scared.

            I wouldn’t say running scared but any new competitor would push a company to up their game. Just as if Google entered your market, you’d push harder. But we had competition before Google – why else would we continue investing billions into the network and do 13 speed upgrades in 12 years…

            > Comcast offers 105 Mbits download only while Google Fiber offers SYMMETRIC 1,000 Mbits for less money.

            And it is not yet proven that it is economically viable for them yet at scale. That said, staging a direct response when it involved physical network infrastructure is not as instantaneous as software dev work, so I would say it is premature to say we’re not doing something (other than offer a 250 Mbps service).

            > The ability to get internet into homes is probably the single most important and most beautiful thing that has ever happened to the human race so far. The ability to get broadcast TV (NBC, CBS, etc) over the old Comcast cable is **NOT** the future, and Comcast needs to focus on internet and stop clutching onto the dying business of broadcast TV. Don’t be Kodak. Kodak tried to hold onto the film business as digital took over. They clutched that formerly lucrative product all the way into the grave.

            Since 1996 every key new business we have launched has been based on the Internet Protocol in some way (residential Internet, commercial Internet, digital voice, WiFi, home automation, etc.). I can assure you we won’t be Kodak. ;-) You may not agree, but rather than debate it wait a few years and see what happens.

            > Net Neutrality will prevent you from cheating.

            It already does. The Open Internet rules (all of them) that are being debated right now already apply to us in full as part if the NBCUniversal merger consent decree with the DoJ. The debate now seems to be whether to apply those same things to other companies, and under what legal/regulatory basis.

  • William W

    “Patriot Act” sounded like a good idea, too… How about “Affordable Care Act”? When will we learn that government uses false advertising when naming its initiatives and typically results in the exact opposite result? “Net Neutrality”… yeah, right!

  • JLivingood

    When you say throttling I think you mean congestion.

    • Most people don’t care why “Stuff slows down.”

      Throttling – “Stuff slows down” because a network engineer is evil and turns a knob and slows it down creating empty unused bandwidth capacity to be wasted.

      Congestion – “Stuff slows down” because the network engineers were incompetent and allowed the data needs to exceed bandwidth capacity on their network.

      So in summary “throttling” means a network engineer is evil, and “Congestion” means a network engineer is incompetent.

      • Andrew Vickers

        Hanlon’s razor suggests the second, not the first. But Net Neutrality rules assume the first, not the second. Net Neutrality laws will not solve the second. There will always be human errors.

      • Milk Manson

        Brian, I know you went to Oregon State, but that’s no excuse. Show me a network engineer without enough bandwidth and I’ll show you an MBA who denied the request for more.

  • FakeBillyTubbs

    Eat a d!ck Backblaze. Why don’t you just get another provider? Why do you have to try to screw everything up for the rest of us?

  • Henrique Vicente

    This is ridiculous, Backblaze. I pay you to serve me. Not to ask for government regulation to intervene in the Internet. The Internet works so well despite government trying to destroy it trough stupid laws. Do you have problems with bandwidth? Solve it using fair means with market solutions. Don’t ask the government to step in. If you think the government is the answer to everything, you better go out of business and let the government handle remote backup services. Is that what you want? Heck, no.

    • JLivingood

      They could also select higher quality ISPs to connect their datacenters to the Internet.

      • Milk Manson

        Ok, so the datacenter to internet link is the fastest and and most reliable money can buy (say backblaze charges 45/mo instead of 5/mo to cover the costs and obviously everyone here is perfectly fine with that). I’m glad that’s done.

        Now, what do we do about the internet to end user pipe? Should we “select higher quality ISPs”? We can try, though I think it’s actually just ISP, as in singular, not plural. So maybe “select the other ISP, and hope it’s higher quality” is a better way to put it.

  • lvthunder

    And when all the ISP’s switch to consumption based billing like water and power how will backblaze like that.

    • Milk Manson

      As opposed to all web sites doing basically the same thing on the backend. Want to go to amazon.com, but they can’t agree on $ terms with comcast so they’re only available on Verizon, Century Link, or bendemover networks… any of which we are free to sign up with, but not for free. Or I guess we could go for the slow and unreliable but less expensive vpn option to bridge the two networks…. sort of.

  • dohpaZ

    As far as the whole ‘regulation is evil’ argument folks are throwing about, just remember, the fact that your 9-1-1 calls work reliably, fixing of rural call completion problems, tracking phone reliability and outages during natural disasters, protecting the privacy of phone calls, and legally requiring providers to report data breaches are all rooted in Title II.

    For folks interested in details of how the FCC successfully applied Title II to a automatic voice roaming in 2007 and the insane overcomplicated mess we ended up with for data roaming by sticking to Title I/III in 2011, you might want to check out: http://www.wetmachine.com/tales-of-the-sausage-factory/the-last-time-the-fcc-classified-a-service-as-title-ii-was-2007-heres-how-it-worked/

    For a good read on just how easy it would actually be to apply Title II with Forbearance to the Internet, give this a read: http://www.wetmachine.com/tales-of-the-sausage-factory/title-ii-forbearance-is-actually-so-easy-it-makes-me-want-to-puke/

    For those interested in a good article about Net Neutrality and limiting Cable Fu#[email protected]!ery, try this one: http://www.wetmachine.com/tales-of-the-sausage-factory/of-cdns-netflix-net-neutrality-and-cable-fuery/

    • lvthunder

      So you think a law written in the 1930’s applies to the internet today?

      • dohpaZ

        And updated in 1996, and yes it can.

        It’s been working pretty well for our mobile phones and automatic voice roaming so far. Especially once appropriate bits of forbearance was applied.

        Unfortunately the other options, Title I and Title III, that the FCC has are pretty limited by the decisions of the D.C. Circuit court and the Supreme Court.

        • lvthunder

          The answer is more competition not more government red tape.

    • Andrew Vickers

      “protecting the privacy of phone calls” smirk.

    • Andrew Vickers

      Regulation is not “evil”, but it is almost always misguided. Most regulation is sponsored by people with special interests, and promoted by people with impeccable motives. Most regulation *sounds* compelling on the surface (how could anyone be opposed to “Affordable Healthcare), but is usually written by, and for the benefit of, the sponsors (http://www.forbes.com/sites/brucejapsen/2013/10/26/despite-glitches-obamacare-profit-windfall-to-insurers-well-underway/).

      “All Data should be treated Equally” sounds compelling, until you consider the implications to QoS routing, or the difference between streaming video and downloading Windows patches in the middle of the night. All data is NOT equal.

  • Andrew Vickers

    Mr. Wilson,

    I am a big fan of Backblaze. I use it for my personal computer and I have introduced it to my company to solve a long-standing backup problem. I chose this service because it meets my specific needs and the needs of my company. It is cost-effective and efficient. It is unobtrusive and automatic. It is reliable. I am so glad that the internet has developed over decades to the point where it enables such a diverse range of solutions and gives us all such a valuable tool to enrich our lives, and I am so glad that you decided to start this company to provide such a well designed product. Thank you.

    However, I feel compelled to respond to your stance on this controversial government action.

    Over the past 20 years, I have seen my personal internet connection progress from a measly 144kbps dial-up modem to a 1Mbps always-on cable connection to a 75Mbps direct fiber connection, and the cost has stayed approximately the same throughout.

    This has all happened WITHOUT government regulation and DESPITE occasional government intervention.

    The Internet, together with all the services it supplies, is a great example of the enormous advances that can be achieved by entrepreneurs like you, Mr. Wilson, when left alone to pursue your dreams.

    “Net Neutrality”, in all of its forms, including the one currently espoused by President Obama, is an attempt to insert government control into a system that CLEARLY does not need it.

    Your anecdotal example of the Comcast/Netflix dispute does not justify a policy that will have far-reaching, and negative, effects on the dynamism and progress of the Internet. Systemic solutions, such as “Net Neutrality” legislation do not solve one-off problems. As I am sure you know, Backblaze will encounter many more hurdles as it continues to grow. We cannot predict what all of these problems will be, but we CAN predict that “Net Neutrality” laws will not solve them. Backblaze will not succeed if you rely on the government to solve your problems for you.

    Please reconsider your stance on President Obama’s Net Neutrality Initiatives.

    • Well stated Andrew! We certainly do not want the government to solve our problems for us. Our main point of writing this article, was to point out that without oversight, form anywhere, whether reclassification, or more FCC involvement, our customers were getting an inferior service, noticing the change, complaining to us, but Backblaze not having any idea of why that was occurring. It wasn’t our problem, the issues were between Comcast and Cogent, we were just caught in the crossfire. And that is what we find unacceptable.

      • FakeBillyTubbs

        Yeah, that’s why you included a picture of Obama with the phrase “tear down this wall.” Because you don’t want the government to solve your problems for you.

        • That was my flair, the post coincided with the 25th anniversary of the Berlin Wall coming down, hence that reference. How about “build up a protective wall”?

          • Andrew Vickers

            or.. how about “Thanks, but we are in a better position than you to know what is right for us and for our customers, so we’ll handle our own problems” :)

      • lvthunder

        So your response to your customers should be is to tell them to yell at Comcast for their lousy service or switch ISPs to a better one.

      • Andrew Vickers

        In the specific case you cite, there WAS oversight. What is more, it was oversight by people that had a very specific knowledge of the impact of the problem and a very direct stake in resolving the problem: You (and many other customers of Comcast and Cogent) were the oversight.

        It is clear from your research that you were in a very good position to analyze and isolate the problem. You were also able to work around it while it remained a problem. You (and companies like you) have the incentive and the means to oversee your own suppliers and change their behavior when they don’t meet your needs. The government has a very different set of incentives, and the government has very little knowledge about how its actions will affect YOUR business.

        By advocating “Net Neutrality”, you are abdicating control over your suppliers to the government. An organization that is not accountable to you or your customers. An organization whose incentives come from politicians and political sponsors, not from Backblaze consumers.

        I appreciate that the experience of the late fall and winter of 2013/2014 were unpleasant, but the best possible resources were dedicated to solving the problem, and they succeeded. The government will not do a better job of looking out for your business than you.

        • JLivingood

          If someone operated service FOO from a few datacenters and FOO’s ISP supplier ABC took on more customer traffic than their network could handle (oversold) and it negatively impacted FOO and FOO’s users, why doesn’t service FOO change to supplier XYZ from supplier ABC?

          Also, was there no SLA to supplier ABC?

        • Brian Wallace

          Andrew, really on point here. There was a clear problem. It was handled by the best possible people to handle it. People who were fighting for me to get good service and the product I paid for. That is exactly who I want fighting for me.

    • > “Net Neutrality”, in all of its forms, including the one currently espoused by
      > President Obama, is an attempt to insert government control into
      > a system that CLEARLY does not need it.

      No, it is NOT inserting government control into the system. Exactly the opposite, net neutrality is on the side of a free and open market. I know this is counter intuitive, so let me explain…

      Look, I have Libertarian tendencies, and I agree with you that most legislation is bad because it attempts to control or restrict people or business’s behaviors. But every once in a while there is a profoundly “different” type of legal document that deals with “rights” and preserves competition. Take the 1st Amendment, which (among other things) prohibits the making of any law infringing on the freedom of the press. The 1st Amendment does *NOT* infringe on interesting new business plans or services. It is a different type of rule. Net Neutrality is not putting rules on the internet, it is saying the ISPs are not allowed to take away or impede the flow of packets. That FOSTERS competition, it does not destroy competition.

      I’m a HUGE believer in competition, it makes businesses and services so much better and so much cheaper. Usually this means a “free market” because there are tons and tons of businesses in any market competing. Take online backup, consumers get huge benefits because you can choose from Mozy, Carbonite, CrashPlan, Backblaze – it keeps us really really honest – if Backblaze tried to gouge customers or modify all your files or down-size your JPEGs to save space, you would switch providers.

      In this case (net neutrality), it really is defensible as a “freedom of information to be treated equally”. The ISPs will still get rich, ISPs can be extremely creative on how they run their businesses, the ISPs can price their packets any way they like. What ISPs are not allowed to do is redirect or lose traffic or charge differently based on the packet content or slow down network traffic based on packet content or slow down network traffic based on the source destination of the packet. You may think you want the future where the ISPs can control what information you see and don’t see – I want freedom. Freedom fosters competition. Freedom prevents monopolies.

      If you are against net neutrality, you are saying you are against the free market and you are against competition. Net neutrality is on the side of a free market.

      Finally, I would encourage people to read this Oatmeal explanation, which is slightly condescending and snarky, but I feel does a good job of making the information approachable to people who are not network engineers and not politicians: http://theoatmeal.com/blog/net_neutrality

      • Henrique Vicente

        Nope. Free market is about respecting contracts between adults. It’s not about forced-down equality.

        Government is the only way to create monopolies and this political agenda is there to help the large players kill the smaller ones, in spite of the consumer needs.

        http://www.americanbroadside.com/the-lie-behind-the-promise-of-net-neutrality/

      • Henrique Vicente

        If you want the state to intervene in my connection, how come can I trust you to backup my data? If you trust the government to provide quality despite everything it touches being so broken, how come I can trust you to defend me against unjust arbitrary government privacy attacks?

        • Net Neutrality means the opposite, it means the government (and ISPs) CANNOT intervene in your data. Anybody (ISPs or government) who delivers data must treat data packets equally.

          By being against net neutrality, you are saying you want the ISPs to throttle your connection to some websites so they can extort protection money. I’m confused why anybody would be in favor of that. Treat all data equally to allow innovation and startups and keep the government and ISPs out of snooping data and prioritizing it differently. That’s Net Neutrality.

          • Henrique Vicente

            By being against net neutrality what I’m doing is just respecting the rights of two adults to sign contracts where they prioritize their data exchange. Nothing but it.

          • Henrique Vicente

            Besides, the agenda promotes government intervention to “solve non-neutrality issues”, otherwise it’d be an useless agenda by definition. So don’t tell us it’s about non-intervention. Non-intervention is what we’ve achieved so far by means of an almost free market on data exchange.

          • Henrique Vicente

            What you must pursue is transparency of contracts. The parts involved shouldn’t be afraid of making it clear what’s allowed or not. Be it neutral routing or not.

            If it’s non-neutral routing let the providers express it freely and the consumer decide to use the service or not.

            We’re talking about common infrastructure here. Packet commutation over shared infra-structure, not dedicated lines / circuit-switching.

            It’s way cheaper to connect two points in the same town with a 1Gbps link than to connect two points overseas with the same bandwidth. Let the market be free and the pricing policies will reflect that in the most appropriate way.

            As a side point… With or without Internet neutrality, it’s about time datacenters get decentralized to provider a better experience to the end users. And if the Internet neutrality political agenda gets in the way making long-distance bandwidth artificially lower (over short-distance bandwidth) by these immoral / anti-ethical means this is going to take longer than we’d like and will end up creating other, unexpected, and unnecessary bottlenecks just in the name of equality (e.g., throttling down fast local connections to avoid being accused of… throttling).

          • Andrew Vickers

            False dichotomy. By being against Net Neutrality, we are not saying that we want the ISPs to throttle connections, or extort money, we are saying that the GOVERNMENT should not be interfering in the normal (and exceptionally effective) operation of the internet.

            Yes, there are isolated cases of ISPs behaving badly. But these isolated cases should be prosecuted for what they are: Incompetence, or even breach of contract. Let the affected parties seek compensation for actual damages. Don’t ask the government to punish the innocent for harm done by others.

      • Andrew Vickers

        Your argument is based on the assumption that the internet is (or should be) a public utility, that it is owned by “society” and therefore every member of society should have equal access. This assumption is false.

        Free speech and the right to assembly do not require the labor of others to support the right. The internet is a tool. It is not a human right. The internet is the product of the labor of (very many) other people.

        The Internet only exists because of massive (and constantly evolving) capital investment. Without that investment, the internet would quickly decay and die. The ISPs (who provide much of that capital investment) are also operating in a free market. A bandwidth market.

        As an advocate of free markets, you should appreciate that innovation in network connectivity/bandwidth/service is only possible if the ISPs can innovate and offer choice to their customers. Those with greater needs, or deeper pockets can pay for the premium service. Those who only want the budget service do not have to pay for features they do not want or need.

        I found JLivingood’s comment (above) to be enlightening. It turns out that Netflix is actually a “budget” customer to their ISP (because they pay wholesale rates), and as a result, Cogent now treats Netflix traffic with a lower priority than retail customer traffic. This change in traffic prioritization is the OPPOSITE of “equal treatment of all data packets”, but it is exactly what was needed to solve this problem.

        The free market in bandwidth solved the problem. “Net Neutrality” would have made the problem permanent.

        • > therefore every member of society should have equal access. This assumption is false.

          I don’t have that assumption. You can block half the people from access, I don’t care at all. Raise prices, lower prices, I don’t care. But what I’m advocating with Net Neutrality is after I buy a packet on the network, the ISP (like Comcast) cannot charge me a different amount or throttle based on where that packet originated (Netflix) or what is inside that packet (video).

          A good analogy is electricity. If I use more I pay more. But the electric company shouldn’t change the electric rate if I plug in a space heater built by the electric company vs plug in a competitor’s space heater. Electrical Neutrality preserves competition and makes the world a better place with more competition. So does Net Neutrality.

          • Andrew Vickers

            I dispute your assertion that “All data should be treated equally”. Why should all ISPs be banned from tiering their service based on content or source? QoS is one innovation that would be destroyed by this legislation. Who knows what other future network improvements will be impossible as a result of this government interference?
            As with all similar regulations, Net Neutrality is an attempt to “close the stable door after the horse has bolted”. It is a blanket rule, designed to solve a problem that happened a long time ago and has already been resolved. The next time there is a problem (and there will be a next time, even if this law is enacted), the problem will likely be of a different kind. We will then get another law to regulate that behavior, and another law, and another law, ad nauseam.

            This is how government slowly, and relentlessly, wrests freedom from its citizens. I appeal to your libertarian sensibilities to see that you are just one of many people advocating for the government to act to solve a cause dear to your heart. Every government rule is dear to someone’s heart, and every one of them is derived from well-meaning intentions. There is always a new cause that politicians are willing to work for, and in every case, someone has to pay. In this case, you are asking the government to make every ISP, and every internet user, pay for Comcast’s mistake.

            If your ISP is violating the terms of your agreement, take it up with your ISP. If your ISP is not violating the terms of your agreement, but you don’t like the terms of your agreement, then don’t do business with them. Go to a different ISP.

            You have a legitimate complaint against Comcast, but your solution to this problem is to have the government tell ALL ISPs, including Abovenet, who did nothing wrong, to behave a particular way. Your solution may even force may ISPs to violate the terms of agreements they have with other clients.

            Please reconsider your position. You are asking innocent people to pay a price for one ISPs bad behavior. By all means, seek restitution from Comcast (and use the government to do it – that is their job), but please don’t punish the rest of us.

          • Anders

            You DO realize that over half of the country doesn’t have a choice in what ISP they use, right?

          • Andrew Vickers

            This is not true. With the exception of some very rural areas, almost everyone has a choice of cable, wireless or satellite access, and many still have the choice of dial-up access.

            The wireless, satellite and dial-up choices are not as fast as the cable/fiber options, but they are still choices.

            How many times have you experienced comcast “doing something shitty” with your internet connection? If you have experienced this, was it deliberate, or was it human error? If the latter, take them to the small claims court to recover what you have lost. If the former, then legislation will not help. You can’t write laws that stop people from making mistakes.

            What you are advocating is to punish all ISPs, all the time, for the rare bad behavior of one. This is not fair. It is not justice. It is just the an easy way of getting revenge for a perceived slight. Worse, it affects far mor innocent people than “guilty” ones.

      • Henrique Vicente

        The Internet is only possible because entrepreneurs take risk all the time to make it happen. It’s not thanks to the government. It’s despite of its intention to make walls blocking access to cryptography, free flow / exchange of data and information over different territories and so on. It’s the most anarchic thing the human race has ever built and it just works. Please keep the good work and don’t support the curtail of our liberties. It’s perversion and, just like now, almost always tinted as something good / necessary evil / and so on, but it’s always wrong.

        Let the market work in peace, without the violence of government interfering with it.

        “if goods don’t cross borders, armies will,” – Frédéric Bastiat

        • > The Internet is only possible because entrepreneurs take risk all the time to make it happen. It’s not thanks to the government.

          I COMPLETELY agree, I’m advocating for net neutrality (where the government is not allowed to treat packets different from other packets). It feels like you think the government will decide when and how to throttle – that is not the case, net neutrality means exactly the opposite, it means we treat data equally and it will allow a level playing field for more private business doing interesting things. Net Neutrality also prevents the government from meddling – the government will be allowed to treat one data source as more important than another data source.

          • Andrew Vickers

            > you think the government will decide when and how to throttle

            Yes, that is exactly the point of Net Neutrality. It is the government setting rules about who can throttle and when (nobody and never). Regardless of whether there is a legitimate reason to treat data differently. Regardless of whether ISPs and clients WANT the data to be treated differently, the government gets to fix the rules now, and damn the consequences (there are always consequences).

      • Chris

        Maybe you are trying to solve a problem created by service providers that have monopolies with net neutrality instead of solving the problem by fostering more completion with internet service providers. If Comcast throttles, I can use someone else. Maybe you want to solve a government problem with a government regulation?

        • Milk Manson

          How many ISP choices do you have?

        • Net Neutrality (treating packets the same regardless of content or source) does not prevent competition between providers. Net Neutrality allows the data to flow and new businesses to thrive and rise up to compete. We (the world’s internet users) had not needed to pass any legislation until now because until recently this is how the internet worked. Only very recently Comcast blackmailed Netflix into paying Comcast MORE THAN MARKET RATE per packet in order to deliver their packets. My small business Backblaze was caught up as collateral damage in the blackmail – yet nobody at Comcast would talk with us or help us debug or route around or solve the issue. The best thing for everybody on the planet is net neutrality – it allows everybody to compete equally (especially little up and coming startups like Backblaze), it doesn’t allow anti-competitive blackmail.

          • Andrew Vickers

            It is very unfortunate (and no doubt, quite expensive) for you to be caught up in a battle between Comcast and Netflix. But the solution is not to make everyone pay for Comcast’s/Netflix’s bad behavior.

            By your own admission, this was an isolated event. Ask the government to punish the perpetrators. As the government to find Comcast/Netflix guilty of actual harm to your company, ask the government to impose a fine on Comcast and/or Netflix. This is how justice is supposed to work.

            Of course, it is much easier to ask a few politicians to sponsor a politically-charged piece of legislation, and have the taxpayer foot the bill for all the debate, then have the internet users and innocent ISPs suffer under the new regulations. But that is not justice, its just politics as usual.

          • Alter Tony

            If you don’t like your service from Comcast–you can take your business elsewhere can you not?

          • If “you” in that sentence is one end user – maybe. But if the “you” refers to Backblaze – no, Backblaze cannot route around Comcast if a customer only has Comcast. If net neutrality is in effect Backblaze can do business with the customer, but if Comcast wins and changes the way the Internet has always been run (neutral) then Comcast could refuse to route our packets, or refuse to route our packets unless we paid Comcast – thus holding us hostage. Backblaze wants Comcast to become rich – and we want them to charge more for every packet – we just do NOT want Comcast to charge different rates based on what is inside the packet, or where a packet originated.

    • Milk Manson

      The last 20 years has been great, therefore the next 20 years will be great.

      I’m going to assume this statement makes a lot more sense when delivered via fiber.

      • Andrew Vickers

        The technology for delivering bandwidths up to 10Gbps is already commercially available and being implemented in corporate data centers. So there are at least 2 more orders of magnitude of improvement coming for domestic connections.

        • Milk Manson

          Corporate data centers have their pick of providers, I don’t. And with a merger or two, guess what? Neither will you.

          • Andrew Vickers

            The only way to remove competition is to legislate against it. Monopolies only persist for as long as the government restricts competition. Monopolies get lazy and quickly succumb to entrepreneurs starting faster/cheaper/better services.

            I currently have at least 3 hard-wired and 4 over-the-air suppliers available to me, and I do not live in a major urban area, where the choices include more esoteric options. It will take more than “2 or 3” mergers to eliminate competition for me (and for most).

            I was referring to the technology that enables 10Gbps connections, NICs, switches, etc., not the backbone connections, but since you brought it up, Verizon is already offering 500Mbps premium service to the home, and Google is rolling out 1Gbps connections in specific locations, so it’s already happening.

            As long as the Federal government does succeed in forcing the ISPs to give everyone the same level of service, these innovations will continue.

          • Anders

            That, or be able to so grossly outspend the competition that there is no way anyone could compete with you? I love independent ISPs, but they are few and far between and generally only available in larger cities. Comcast and TWC together would control what, 80% of all high speed connections in this country? They have no incentive to play nice.

          • Andrew Vickers

            Grossly outspending the competition is a sure-fire way to end your monopoly. When you grossly outspend, you reduce and eliminate your profit margin, and very quickly go out of business.

            Could you define “independent ISP”? I think your are implying that comcast/time warner is not independent. From whom?

            I wonder if, perhaps, you mean “smaller” ISPs? If so, great. I too love the smaller companies, and if they can offer me a better service for the same price, or the same service at a better price, then I will use them (as, for example, Backblaze does in the backup market).

            In the case of Internet access though, the best/cheapest service that meets my needs is not provided by a smaller ISP, it is provided by Verizon. The moment one of them can beat Verizon, I will switch.

            High speed internet REQUIRES a lot of capital. The people that provide that capital do not do so because they are nice, not because they are greedy. They do it because it provides a return on investment. If you, through the government, force the ISPs to spend capital for little/yes return, they will withdraw the capital and invest it elsewhere. You cannot stop that without MORE government intervention.

            Government intervention in the free market ALWAYS has unintended consequences, because the legislation rarely accounts for what happens next. How will the people that are negatively affected react. What will they do to protect/preserve their return on investment?

          • Milk Manson

            So I can assume then you would be all for complete deregulation of, say, energy? So we can then just sit back and watch all the entrepreneurs and their disruptive tech come out of the woodwork; slinging power lines, poles, wind farms, and solar arrays all over/under/around the existing grid. Totally doable.

            I know what you were referring to, so you have me confused with someone else because I didn’t bring it up. Yes, lan speeds are increasing… mostly because the other two options are to stay the same or get slower. It’s great, but it’s also not a relevant part of any net neutrality discussion.

            Wait, esoteric options? Did you really just say that? Seriously, WTF are you talking about?

          • Andrew Vickers

            You can assume all you want, especially on unrelated subjects.

            Why are WAN (not LAN) speeds increasing instead of the other two options? Because ISPs are investing in research and development. And where is thei eu coming from for this investment? Customers, especially those willing to paya premium for the faster service. “Net Neutrality” will eliminate that sieve of funding.

            Yes, I said that. I am talking about small providors. What is the problem?

          • Milk Manson

            The problem is the word esoteric doesn’t mean the same thing as the word small. Kind of like the word sieve doesn’t mean the same thing as the word source.

            If you’re talking about a NICs, you’re talking about LANs.

            ISP’s are investing in R&D? What a joke. Where are you getting your information? Why does my Comcast “customer gateway” say SMC Networks on the front? Rhetorical questions, don’t answer.

            Neutrality is not about changing the internet, it’s about keeping it the same. Do you think Comcast and Verizon are fighting like hell against Net Neutrality for your benefit or theirs? Do you think Comcast is trying to merge with Time Warner for your benefit or theirs?

            They are not fighting for you. Your interests and theirs DO NOT MESH, you dumb egg.

  • Libscansuckmyass

    With a 36% approval rating: You might want to rethink putting that pig of president’s pic on your site too.

  • Libscansuckmyass

    I like your service. This post is stupid to the max. Why do libs think the government is the answer to everything when all it does is destroy and screw up what it touches?

    • Belvedere

      What about roads? Do you like those? What about safe air travel? Do you like that? Do you prefer having the fire department come when your house is burning – or would you rather let your neighbors donate their time and equipment to help you put out that fire? Want to save your house? Do you want the police to find your car when it gets stolen and arrest the culprit? Do you enjoy eating safe food? Because free industry was so good at it before the government stepped in.

      @Libscansuckmyass:disqus

      That pretty much sums up your contribution. Good day.

      • RightyFeep

        Roads etc are not the best analogy. A better one might be riparian rights – the problem of allocating water – “Consistent in these questions is that a riparian owner may not act to deny riparian rights to the owner of downstream properties along the waterway”

        I tend to agree with you that the government needs to step in. The problem is that Obama’s government will represent special interests, and that the Republicans will be no better – just a different set of special interests.

        In other words, under the present regulatory approaches, the problem will not be solved. “Net Neutrality” is another Obamalie – a facade to conceal the fact that all it will do is chose different winners than the free market would choose.

        Surely there’s some other solution?

        • Belvedere

          I tried replying, but it was held up by Backblaze moderators and then deleted. And I was even being cordial….
          Anyway, nutshell of my reply was, I respect your opinion and also agree with parts of it. Would just say that “free market” is a loose descriptor in the context of virtual monopolies. And it doesn’t have much to do with Obamalies because he is doing the opposite of coddling special interests in this case. IMNSHO, If you want your monopoly, then create the pipe and don’t mess with what flows through it. That’s a position I can get on board with regardless of the party sharing my opinion.

          Oh, and there might be some other solution, but let’s not all bend over in the meantime, can we? ;)

      • Brian Wallace

        I like roads just fine. I’m not sure that the U.S. government is the only organization that can figure out how to build roads. I am thankful that someone would come to my house if it were on fire and help me put it out. I’m not sure the governement is the only solution to that problem.

        Your argument assumes that becuase the government has a monopoly on something they are the only entitiy that could ever handle delivering that service. That is a constent problem when we talk about things like education, police and soon healthcare. The government takes your money and then provides these services for “free.” That makes them hard to compete with.

        I look at the playgrounds in my community that have been put up by the state. At first I think wow that’s great, a new park for my kids to go to for free. Then I wonder how much did I actually pay for that park. Then I wonder if a private company could compete, and I could pay them directly how much cooler could that park be?

        • Belvedere

          You’re putting words in my mouth. I did not say the government is the only solution. I’m arguing against the prevalent “all government is automatically bad” mentality coming from the right-wing. It’s not all bad — just ask those same people whenever they’re in charge of it.

          So far, the government is the reason we have safe air travel and safe food and cars that don’t explode every time you get rear-ended. Private industry may have been able to solve those things — and more — but it didn’t. Private industry _has_ put up a bunch of extremely nice playgrounds.

          It’s really funny that you talk about government providing services for “free”. Reminds me of that family whose house burned to the ground because they couldn’t afford to pay the fire department. Oh well, too bad for them, doesn’t affect you, right?

          Anyway, we’re talking about net neutrality here (and not Ted Cruz’s embarrassingly ignorant understanding of it) which essentially instructs monopolies to keep their hands off the data flowing through the pipe. You could call it Net Equality. Or Net Freedom. That would seem to be in everyone’s interest — but the fact that Obama gives it a nod is enough to set the right-wing to shooting itself (and everyone else) in the foot based on ideology. Again.

          • Brian Wallace

            I don’t see a right wing agenda in almost any of the comments here. What I see are people who think that agreements and contracts should be made without violence. What I see here are people making an argument that comes from principles and values. The two principals in this case are private property rights and the nonagression principle.

          • Belvedere

            That has been my observation as well.

  • Steubing

    We have to pass the net neutrality bill so that you can find out what is in it.

    • Belvedere

      Yep. Just like the Patriot Act.

  • Mark V. Fusco

    In this case, the solution was Netflix supplying severs within the Comcast network at Netflix’s cost to relieve the congestion at the gateway between Comcast’s network and the network that handles Netflix (Level 3 I think?). Users within Comcast’s network (both endpoints) never experienced an issue either.

    As Brian and Andrew both said, the government using Title II, is not the answer. It’s onerous and, frankly, ancient (in tech terms) law. Both the original 1934, and the 1996 update. Plus the FCC will cherry pick which parts of Title II to use.

    In every case of throttling and other issue, the parties involved have found a solution without government intrusion.

    The biggest issue I’ve had in using Backblaze, or anything that’s required massive usage is when I’m uploading. And that’s due to bufferbloat (http://en.wikipedia.org/wiki/Bufferbloat). Since getting a new gateway from Uverse I’ve not noticed as much of an issue, so they may have sprinkled some magic dust on these gateways to combat that (see the Mitigation portion of the Wikipedia entry).

    • JLivingood

      On Bufferbloat, agree it is an issue. We’ve worked closely with folks like Jim Gettys, Dave Taht, and others. And AQM is now baked into the upcoming DOCSIS 3.1 standard, which will be cool. Still working on how to get it into D3.0 (not to mention WiFi chipsets and the like – a bigger issue that just one network).

  • Brian Wallace

    I understand that with freedom there are some bumyp roads. It is difficult sometimes to let the market work. However, the governemnt regulating something to keep it open and free…. I have a real hard time beliveing that.

    • Andrew Vickers

      I completely agree. Yes, it is unfortunate that Backblaze suffered (briefly) because of a fight between two other companies, but the problem was resolved WITHOUT government regulation. Do you seriously think a government investigation into this problem would have resulted in a quicker, or better, resolution?

      We do not need to give the government control over internet pricing. History shows, very clearly, that the government is too slow, too partisan, and too willing to pander to special interests to have any positive impact on the Internet.

      What Backblaze has done to the backup business is remarkable. Please do not destroy everything you have achieved by helping the FCC gain control of access to the Internet. Listen to what Mark Cuban (among others) has to say about this.

      • Brian Wallace

        Really well stated. Thank you.

        • Milk Manson

          No, it’s not well stated. Tragedy of the Commons is a free market problem, the more free the market the worse the problem. The dude has it completely backwards.

      • dohpaZ

        As has been stated many times, even if the FCC reclassifies the Internet as Title II they can forebear price control. You parrot the usual Ted Cruz (aka Comcast’s paid spokeswhore) nonsense without any actual understanding of the issues involved. Polly wanna cracker?

        I’m guessing you didn’t actually read / listen to Obama’s proposal since it explicitly includes the FCC using forbearance on pricing and other aspects of Title II that wouldn’t apply to Internet access.

        For those that are unfamiliar with the term, forbearance in this context refers to the FCC’s power to abstain from enforcing specific aspects of Title II. It’s a power that the FCC has used frequently over the years when applying existing regulations to newer technologies.

        • Andrew Vickers

          I decline to respond beyond this post due to your use of Ad-hominem arguments.

          • dohpaZ

            And your lack of actual facts and knowledge about classifying services as Title I versus Title II.

          • Joseph Meyer

            Right on.

        • lvthunder

          Obama only has 2 years left. I’m not worried about what he is going to do. I’m worried about future administrations. When the federal income tax was passed it was suppose to be only the rich and not more than 3%. That lasted one administration.

          • Milk Manson

            What year did that federal income tax thing happen, anyway?

        • Joseph Meyer

          Why is it so difficult for people on the left to discuss issues without indulging in insulting rhetoric?

      • RightyFeep

        “History shows, very clearly, that the government is too slow, too partisan, and too willing to pander to special interests to have any positive impact on the Internet.”

        There it is, in a nutshell.

        The problem, though, is that History also shows, very clearly, that market based decisions result in underserved or unserved consumers, when serving those consumers is inefficient or less profitable.

        What can be done about that, short of government action?

        In the fight between Comcast and Netflix, who represented the consumer? Who, for that matter, represented Backblaze?

        Right now, the only potential consumer representative is the government. And right now, the standard liberal regulatory approach is the only model for government action.

        This is a good opportunity for the Republicans to come up with something new – something that lets the free market operate, but at the same time insures that consumers are at the table when the fights are fought and the deals are made.

        Fat chance.

        • Andrew Vickers

          “market based decisions result in underserved or unserved consumers” Correct, but I would rather risk being underserved than have the internet succumb to the tragedy of the commons.

          “In the fight between Comcast and Netflix, who represented the consumer? Who, for that matter, represented Backblaze?” Backblaze represented themselves and their customers. Netflix represented themselves and their customers. The dispute was resolved, to everyone’s satisfaction WITHOUT government regulation.

          “Right now, the only potential consumer representative is the government”. This is not true. See above.

          Customers are ALWAYS at the table. They register their satisfaction and their opinions with their wallets. One of the main reasons why our internet speeds have increased by 2 orders of magnitude in the last 15 years is the constant battle between ISPs to offer more attractive services. If the government reduces or eliminates the profitability of faster services to those who want it (and want to pay for it), then the incentive to compete will disappear.

    • Robert Stek

      Oh please! It’s all about the greed of companies like Comcast and others like them. Many European countries have twice the internet speeds at half the costs. I’d rather have a government that works for the common good of its ordinary citizens than one which passively helps monopolistic-like companies further feather the pockets of the 1%-ers.

      • Libscansuckmyass

        And what they touch gets completely messed up. Take your 1% crap and shove it.

      • lvthunder

        You want common carrier regulation then the internet is going to become like the land based phone system. When was the last time your land line phone quality got better in the last 40 years?

        • midwestspirit

          I’m sure you know that is the time frame President Reagan stopped enforcing the Sherman Anti-Trust Act. Call your Republican legislators and tell them to bring it back. Good luck with that!

      • bsolestis

        “Greed”?? Nonsense. Do YOU work for less than you’re worth? When was the last time you went to your boss and offered to take a 50% pay cut?
        People use the term “greed” to say that someone else is making more than they think he should, but somehow that standard never applies to themselves.

        • iSRS

          Let’s not forget that we should raise the minimum wage to $15. And lower costs, and improve services. All at the same time.

  • #ThanksObama

    • Libscansuckmyass

      ROFLOL. You want him to screw up the Internet like he destroyed health care? What a special little cupcake you are.

      • Backblaze’s stance is that we don’t want to be caught in the crossfire of two companies when we have nothing to do with their disagreements. Reclassification can provide protections against this scenario (which has actually happened to us and is mentioned in the post above). Even if it raises our costs, knowing that our customers are getting what they are paying us for, instead of an inferior product that is being affected by two unrelated companies fighting over something unrelated to Backblaze, we’d be OK with that. If such protections can be made under the FCC’s watch, great! Either way, we want our customers to get what they pay for.

        • lvthunder

          ” two unrelated companies” They aren’t two unrelated companies though. Your users use one of those companies to get to you.

          • My meaning was that Comcast customers in this instance were being provided a less-than-ideal service from Backblaze, even though Backblaze does not use Comcast in our data centers. All our customers saw was a decrease in performance, without any explanation, and Backblaze had to expend a lot of resources to drill down and find out exactly what was causing the slowdown for them, then expend more resources and reroute all traffic to different providers who weren’t in a disagreement with Comcast. Not a great user experience.

          • Andrew Vickers

            It is a shame that Backblaze and (“a small number of”) your customers had to go through this, but given that bad things will happen, what is the best way to solve them:
            1) Have those with the knowledge and incentive to find a solution work with each other to solve the problem in the shortest time, and with the least cost, possible.
            2) Have political appointees and federal bureaucrats, with little knowledge of the specific problem, come up with an idea that is supposed to help and mandate that idea for everyone, whether they have a problem or not. Then spread the cost (whatever it may be) among all taxpayers.