By now the problems between Verizon and Cogent are well known, at least to nerds. FiOS users, especially on the East Coast, have been complaining of increasingly poor Netflix streaming performance. The reason stems from the saturated connections between Cogent and Verizon which neither refuses to fix; see here for a good background on the dispute. Basically, Cogent is sending more data to Verizon than vice versa, and Verizon is asking Cogent to pay for the upgrades required. The reason this story has been getting any press is that it highlights the complete insanity of the commercial internet system: ISPs expect to get paid to carry data packets, even if those data packets are requested by their own customers. In this case, Cogent is simply providing Verizon with the Netflix data that Verizon’s customers have asked for.

To highlight the ludicrousness of the way the internet operates, Verizon could presumably generate traffic from Cogent for which it expects Cogent to pay by issuing requests to download data from Netflix itself. Or, as Netflix has pointed out, Netflix could resolve this situation by deciding to host its users backup data for them simply to artificially generate traffic going the other way. In fact, I’m surprised that Netflix doesn’t just program its streaming clients to repeat every bit back that they receive. That would solve this ludicrous problem, while also highlighting the stupidity of the way peering arrangements are made. At the bottom of this insanity is the fact that the companies who run networks have decided that they should get paid to carry packets like shipping companies would charge to carry packages. I would say it’s like UPS deciding to charge Amazon for shipping a package, while also deciding to charge the recipient for driving to their street. However, that’s not a perfect analogy, because if it were really like the Internet, UPS would be willing to waive the shipping if I handed them something to send back to Amazon. In fact, I struggle to find an analogy with the physical world of shipping, because there is no good analogy. Which is why it’s so incredibly stupid that network providers insist on billing arrangements that are analogous to shipping contracts.

Anyway, back to the point of this post: Comcast, which until recently had similar issues, has resolved them by getting Netflix to pay Comcast to connect directly to Netflix. There has been speculation Verizon would do the same. On the other hand, Verizon is probably not as willing to come to a reasonable solution as Comcast was, the latter trying to play nice to appease anti-trust regulators given it’s recent purchase of Time-Warner. I recently noticed an improvement in Netflix performance on FiOS, and wondered if maybe I was wrong about this. However, running a traceroute makes it clear that what happened is a third option I hadn’t considered; traffic between me and Netflix is going around Cogent and all the way to California:

3 g0-10-2-5.bstnma-lcr-21.verizon-gni.net (130.81.104.50)
4 ae1-0.bos-bb-rtr1.verizon-gni.net (130.81.151.60)
5 0.ae11.xl3.nyc1.alter.net (152.63.20.69)
6 0.xe-2-1-6.xt1.dca5.alter.net (152.63.0.113)
7 0.xe-4-1-3.xl3.iad8.alter.net (152.63.3.142)
8 tengige0-6-4-0.gw1.iad8.alter.net (152.63.35.145)
9 teliasonera-gw.customer.alter.net (152.179.50.234)
10 ash-bb4-link.telia.net (80.91.252.98)
11 las-bb1-link.telia.net (80.91.246.71)
12 netflix-ic-300871-las-bb1.c.telia.net (213.248.95.34)
13 ipv4_1.lagg0.c048.lax004.ix.nflxvideo.net (198.38.96.157)

Is it possible that rather than deal with Cogent or Netflix, Verizon has decided to just send East Coast Netflix traffic all the way to servers based in California, and to get there without using Cogent? Or, could Netflix have caused this by having clients make requests to different servers to get around the limited Cogent-Verizon ports? One thing that is clear is that we need a fundamentally different model for commercial internet if games like this are being played.