Netflix blames Verizon, Verizon blames Netflix -- let's see the data.

Netflix says Verizon is a streaming bottleneck, Verizon says the congestion is upstream -- we need data not accusations.

Netflix recently showed messages like this one when they had to adjust the bit-rate on their streams to FIOS customers:

Verizon responded, claiming that the congestion was upstream from FIOS, and threatened legal action. Netflix says they will drop the messages for now.

The public and regulators need more data and transparency, not less. For example, here are Netflix data rate histories for Google Fiber, Comcast, Verizon FIOS and Verizon DSL:

This data clearly shows the improvement in Comcast speeds when they and Netflix agreed to terms in February. (One wonders how that rapid improvement was achieved and how much investment it required).

Verizon DSL is slow because of the technology. Those Verizon customers who can get FIOS are surely better off, but one wonders why Verizon fiber is so much slower than Google fiber and Comcast cable.

If Netflix congestion is upstream from Verizon, they should present data demonstrating that.

-----
Update 6/10/2014

Google also publishes data on YouTube streaming performance for ISPs. I checked the performance of Time Warner Cable, my not-so friendly ISP, in Los Angeles, my city.

The following graph shows YouTube video consumption for time slots beginning at 6 AM and peaking at 9 PM.

Google also reports the percent of the time an ISP delivers high definition (720p), standard definition (360p) and lower quality. For me, the worst performance, HD 81% of the time and SD 19%, occurs between 9-10 PM.

This is one more source of ISP performance data -- check it out and see how your ISP rates compared to others in your location (if there are others).

-----
Update 6/13/2014

The FCC will scrutinize the pay for service deals Netflix and Comcast and Verizon. They will also look at others, including Google.

FCC Chairman, Tom Wheeler, said he already had copies of the contracts, but I hope he asks the companies to provide traffic data and also disclose the cost of the infrastructure investments they make to keep service reasonable.

-----
Update 7/7/2014

YouTube has joined Netflix in linking to data when video delivery slows down. I was watching a YouTube video that began to stutter. When it paused to rebuffer, I saw this message:

When I clicked on the find out why link, it took me to Google's Video Playback Checklist, which lists things that might go wrong and suggests fixes -- no blame on your ISP -- so far. If none of those fixes help, they also provide a link to the Google Video Quality Report (described above). As we asked -- they show us the data.

-----
Update 7/19/2014

I've asked to see congestion data and to know the cost of alleviating the bandwidth shortage noted by Netflix customers and other Internet users. Mark Taylor, VP of Content and Media at Level 3, a transit provider used by Netflix, has given us some data and some costs. This diagram shows data at one of the of the ten exchange points between Verizon and Level 3:

We see that the congestion between Level 3's customer (Netflix) and a Verizon retail customer occurs in the forwarding of packets between their routers (in the same building). Both the Level 3 and Verizon networks are uncongested (green) but the links between their routers are congested (red).

That is the data for this exchange point, but what about the cost of alleviating the problem? Taylor says it is a few thousand dollars and five minutes work to install 10gb port cards in the routers. He says they have been asking Verizon to install the cards for many months and offers to have Level 3 pay for them. He says he will throw in the short cables between the routers.

If more capacity were needed after those eight port cards were installed, new equipment would have to be purchased, but that would be a small fraction of the cost of the networks the routers connect.

Note that the above diagram is based on a Verizon post describing the situation at the Los Angeles exchange point. Furthermore, it shows under utilization on the Verizon network -- no need for data caps in that case :-).

-----
Update 7/24/2014

David Young, Verizon's Vice President for Regulatory Affairs, has posted a reply to the claim by Mark Taylor of Level 3 that Verizon is causing Internet congestion. Young says the problem is not datacenter exchange point congestion, as Taylor claims, but congestion in the Level 3 network. In fact he says that:

Level 3 insists on only using its existing settlement-free peering links even though, as Level 3 surprisingly admits in their blog, these links are experiencing significant congestion.

This is Verizon's view of the situation -- transit providers like Level 3 are the problem:

But, when I looked back at Mark Taylor's post, I saw no such admission. In fact, Taylor says "our network has plenty of available capacity" and goes on to state that:

I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS (his caps).

These are conflicting statements -- who to believe?

Considering Verizon's track record, I'd give the benefit of the doubt to Level 3. Verizon promised to install FIOS fiber in my neighborhood several years ago, but subsequently changed their mind and decided instead on a "gentleman's agreement" dividing the wireless (Verizon) and land-line (cable companies) ISP markets. Indeed Verizon has a track record of broken promises -- Bruce Kushnick has documented 300 billion dollars worth of them.

That history makes me suspicious that Level 3 is telling the truth, but it is not proof. Proof would require transparency on both sides. Level 3 and Verizon should show us the data -- let's see the traffic logs for your networks. (Or at least let the FCC staff see them).

-----
Update 10/15/2014

I've asked to see the data. It's not congestion data, but Generator Research has estimated the cost of delivering Netflix content in their report Over-the-Top Television, 2014, which includes an estimate of the cost to Netflix and Comcast of delivering Netflix content and of the fee Netflix is paying Comcast.

Their estimate of Netflix's "fast lane" fee to Comcast is $.86 per subscriber per month. That is a small portion of the inflated monopoly/oligopoly prices we pay to Comcast and other Internet service providers. It seems to me that we should be paying more attention to the lack of competition in the ISP industry than to network neutrality.

-----
Update 11/9/2014

M-Lab monitors Internet performance and has published a report on ISP Interconnection and its Impact on Consumer Internet Performance.

The report concludes that

Using Measurement Lab data, and constraining our research to the United States, we observed sustained performance degradation experienced by customers of Access ISPs AT&T, Comcast, Centurylink, Time Warner Cable, and Verizon when their traffic passed over interconnections with Transit ISPs Cogent Communications (Cogent), Level 3 Communications (Level 3), and XO Communications (XO). In a large number of cases we observed similar patterns of performance degradation whenever and wherever specific pairs of Access/Transit ISPs interconnected. From this we conclude that ISP interconnection has a substantial impact on consumer internet performance -- sometimes a severely negative impact -- and that business relationships between ISPs, and not major technical problems, are at the root of the problems we observed.

It seems that interconnection between transit ISPs and ISPs that connect customers (access ISPs) is indeed the weak link in the content delivery chain.

The above figure gives an overview of download throughput between Access ISP and Transit ISP pairs during 2013. For each of the 28 graphs, the Y axis shows the % of hours with the given download throughput, and the X axis shows the download throughput (Mbps). Red, on the left, indicates sub-broadband speeds (download throughput below 4 Mbps). What this table demonstrates is not simply a proliferation of performance issues, but that these issues cannot be laid at the feet of any one Access ISP, or any one Transit ISP: no Access ISP performs badly to all Transit ISPs, and no Transit ISP performs badly for all Access ISPs. Therefore, if the problem is not at one end, and not at the other, it must be in the middle around the interconnection between the two.

For example, my ISP, Time Warner falls below 4 Mbps (red) over 30% of the time when delivering Cogent packets, but does much better with data from other transit ISPs.