Monday, June 30, 2014

Could Google provide Internet access in Cuba?

(Cross posted on

The obstacles are political, not technical.

Eric Schmidt and other Google executives traveled to Cuba where they met with members of the Internet community and the government. Google is providing Internet access in a few US Cities and is considering others -- might they provide Internet access in Cuba?
Consider the following:

  • Cuba has very little domestic backbone infrastructure, but they could afford to extend Internet connectivity via satellite.

  • Google has a geosynchronous satellite project that could serve Cuba.
  • Of course, both governments would have to agree for Google or any other satellite ISP to connect Cubans. I believe that, if the Cuban government would agree, the US would as well.

    But, the Cuban government has feared the Internet since the time of their first IP connectivity in 1996. At that time, there was high level debate about the Internet. The hard liners, led by Raúl Castro, argued against the Internet while others argued for a "Chinese" approach of supporting Internet use while censoring content and surveilling users. (It seems Fidel Castro was ambivalent).

    The hard liners won in 1996, but what about today? Schmidt reports that a "number of the people" he spoke with said "the eventual model of Cuba would be more like China or Vietnam than of Venezuela or Mexico." If some of those were young government officials, there may be a glimmer of hope.

    Update, July 3, 2014

    It is noteworthy that Jared Cohen, Google's Director of Ideas, accompanied Schmidt on this trip -- before joining Google, he was a member of the Secretary of State's Policy Planning Staff and served as an advisor to Condoleezza Rice and later Hillary Clinton.

    Thursday, June 26, 2014

    Stockholm: 19 years of municipal broadband success

    The Stokab report should be required reading for all local government officials.

    Stockholm is one of the top Internet cities in the world -- how do they do it? Wholesale communication infrastructure in Stockholm is provided by AB Stokab, which is owned by the Stockholm City Council. Stokab leases dark fiber and space in nodes/hubs where customers can install communication equipment and interconnect networks since 1994. Stokab's goal has been to build a competition-neutral infrastructure capable of meeting future communication needs, spurring economic activity, insuring diversity and freedom of choice and minimizing disruption to the city’s streets.

    How has it worked out?

    Quite well, as you see in the following figure, taken from Stokab's report on the socio-economic cost and benefit of the project:

    Accumulated investments and socio-economic
    return in million Swedish Kronor.

    The returns shown here reflect increased property value, returns of the municipal housing companies (currently breakeven, due to large investments), value for tenants, increased employment, Stokab’s profit, saving for the municipality’s and county’s data and IT costs, and increased economic activity in the supplier industry. To drill down into the details, see the Stokab report summary or the full Stokab report.

    The Swedish Telecommunication regulator published a report calling for openness and competition at five Internet service infrastructure levels -- from physical access to land, ducts and spectrum through retail Internet service -- based on the Stockholm experience.

    A lesson for the US?

    The US needs infrastructure investment -- who will make it? The telephone and cable companies were given a chance, and they've dropped the ball. The Stockholm experience shows the role local government can play. National government's have also been important. The US Federal Government underwrote the research that gave us the Internet and governments like those of Singapore or China have worked as planners and venture capitalists. Home and building owners can also contribute to "last 100 yard" investment.

    Singapore's government acts as a planner and venture capitalist.

    Given the current US Congress, it is hard to imagine the Federal Government investing in Internet infrastructure, but FCC Chairman Tom Wheeler has praised municipal broadband efforts, wants to fight state laws prohibiting or restricting them and he is currently challenging Tennessee's anti-municipal net law.

    In spite of the fact that Wheeler came from the ISP industry, you have to like a guy who says:
    If the people, acting through their elected local governments, want to pursue competitive community broadband, they shouldn't be stopped by state laws promoted by cable and telephone companies that don't want that competition.
    The situation in the US will not change until the Internet becomes a political issue for the general public and that may be happening -- check out comedian John Oliver's piece on the Internet. Wheeler watched Oliver's piece and responded -- check that out too -- it's funny!

    ISP industry lobbyists claim that government involvement interferes with The Market, leading to waste and inefficiency, but in Stockholm, the municipal government has created a competitive market. This story is biased because we can't expect every local government to be as skillful as Stockholm's, but their example is worth considering.

    Wednesday, June 25, 2014

    Supreme Court rules against Aereo -- but people want local TV online

    The Supreme Court says Aereo is distributing copyrighted material, but there is demand for their service.

    An Aereo antenna
    The Supreme Court has ruled against Aereo, a company that stores local, over-the-air TV broadcasts then streams then on the Internet. The Court ruled that Aereo was retransmitting copyrighted material without permission.

    Aereo deployed arrays of dime-sized antennae and claimed that they were merely leasing a speciic antenna to each customer and delivering content that was received by the customer's personal antenna.

    An array of tiny antennae

    The lower court case against Aereo challenged their technology claim -- questioning whether those dime-sized antennae were electronically independent. (If they were acting as a single, integrated antenna, Aereo would have been a clear copyright violator).

    Expert witnesses on the technology were divided during the first case -- Aereo's expert said the antennae were independent and the TV station's expert disagreed.

    The Supreme Court decision did not depend on a technology argument -- they held that Aereo was not in the antenna leasing business but were sellers of subscriptions to watch copyrighted television programs.

    I am a cord cutter and was cheering for Aereo, even though I questioned their technology claim and regarded their system as a kludge designed to get around copyright law.

    What about the future of local TV? If those tiny antennae really worked independently, Aereo could sell them to end-users and let them receive local broadcasts over the air, but I suspect they either do not work as claimed or only work if they are located very close to the transmitting antenna.

    Regardless, the current Aereo business model is toast, but they have shown that there is a market for local content delivered over the Internet. If customers are willing to pay for streaming of local content (including ads), why don't the copyright holders offer the service themselves or hire Aereo to do it for them?

    Update 7/1/2014

    I said that those dime-sized antennae seemed bogus to me -- that they could not truly be independent. Pete Putnam agrees and presents a detailed technical analysis supporting his conclusion that the antenna array would have to be within a couple hundred feet of the TV transmission tower for such small antennae to work independently.

    That is technical support for the broadcaster's assertion that Aereo was essentially doing the same thing as a cable company -- capturing and retransmitting copyrighted material -- but without paying the retransmission fees charged to cable companies.

    That raises the question -- how much are those retransmission fees? I went to the Dish Network Web site and found that "Local channels for non-qualifying packages are $6.00 per month." Dish evidently finds it profitable to pay the retransmission fees and charge customers $6 per month. They also offer "Superstaions" for $2 per month each.

    So, why doesn't Aereo or someone else go into the legal business of paying the retransmission fees, keeping track of the number of viewers for the sake of advertising sales, and offer local channels over the Internet? Aereo showed there was a market demand for such service.


    I've expressed doubt over the technical feasibility of Aereo's claim that they were renting independent TV antennas to their customers, but the Supreme Court ruled that even if that technology did work, Aereo was essentially acting as a cable company. I suggested that they had demonstrated a demand for local TV over the Internet and suggested that they pay the retransmission fees and become a "cable" company. But, the Copyright Office does not agree that they are a cable company, and, even if they are, the broadcasters might be able to force them to carry arbitrary bundles of channels like they do "real" cable companies. I am not a lawyer, but it seems Aereo is finished. There is demand for local TV streamed online -- will the broadcasters allow that to happen at any price?

    Update 9/30/2014

    The Supreme Court decided that Aereo was a reseller of copyrighted material, not an antenna leaser, so ruled that they could not rebroadcast local, over the air TV signals. That shot them down ... for the time being.

    Aereo has demonstrated that there is demand for local TV over the Internet, and the FCC is now considering a proposal to treat online video services like cable and satellite TV providers. If that were the case, Aereo and others could pay the same sort of retransmission fee as Comcast, et al, and provide local scheduled programs over the Internet. Aereo and others could negotiate retransmission fees and be back in business. Aereo could also drop the technically questionable claim that they were leasing access to independent antennae.

    Monday, June 23, 2014

    Recommended podcast: Get your next eye exam on a smartphone by Andrew Bastawrous

    A smartphone with a cheap attachment can diagnose eye disease and let us restore sight to millions of blind people

    In 1973, E. F. Schumacher published a book called "Small Is Beautiful: Economics as if People Mattered." Schumacher wrote of the need for "approriate technology" -- technology that was small-scale, decentralized, labor-intensive, energy-efficient, environmentally sound, and locally controlled. His emphasis was on, but not confined to, developing nations.

    Andrew Bastawrous' low-cost smartphone ophthalmic tools are a textbook example of appropriate technology. He uses an Android phone to assess vision, view the retina, and simulate a persons vision for another to see the world as they do. Eye exams are conducted by technicians on bicycles with solar panels on their backpacks to keep the phones charged. They also have a map-based app to maintain their database of patients and village elders.

    Two quotes from the talk:
    • There are 39 million people in the world who are blind. Eighty percent of them are living in low-income countries such as Kenya, and the absolute majority do not need to be blind. They are blind from diseases that are either completely curable or preventable.
    • It makes it possible to pick up diseases of the eye and of the body that would not be possible without access to the eye, and that clip-on device can be manufactured for just a few dollars, and people can be cured of blindness, and I think it says a lot about us as a human race if we've developed cures and we don't deliver them. But now we can.

    For more information on Andrew Bastawrous and his project as well as a transcript and downloadable audio and video copies of the podcast (6:33), follow this link.

    Solar power charges the backpack and the
    technician travels on a bicycle or motor bike.

    Imaging the retina

    Tuesday, June 17, 2014

    Who will provide entry-level and continuing vocational education?

    Universities are doing an increasingly poor job of entry-level training -- will industry take over?

    My first job after college was with IBM. As soon as I was hired, I was sent to San Francisco for an eight-week training course in which I was introduced to the organization and its culture and taught to wire the control panels of unit-record machines and design unit-record systems. After that phase one training, I worked for a while, then went to a second and later a third class.

    In those days, IBM hired people regardless of their major in school, based on an aptitude test and interviews. (The most senior technical person in our office and my first mentor was an English major). IBM assumed the responsibility of training entry-level employees.

    A few decades later, universities were expected to finance entry-level training. Companies wanted new hires who were productive on day one. That worked pretty well as long as the cost of education was reasonable. (My tuition at UCLA was $76 per semester).

    But, that system has broken down -- society has cut support for universities and, to be honest, IBM did a better job of entry-level training than many of our universities. As we see below, today's students often borrow large sums to pay for their college education and many end up in dead-end jobs.

    Percent of graduates in jobs not requiring a degree
    College graduates: age 22-65 with a bachelor's degree or higher
    Recent graduates: age 22-27 with a bachelor's degree or higher
    Shaded areas designate recessions.

    Good non-college jobs: at least $45,000 a year
    low-wage jobs: $25,000 a year or less
    Shaded areas designate recessions.

    It is way too soon to call it a trend, but the Internet may be taking us back toward industry-financed entry-level job training. The most IBM-like example is AT&T's sponsorship of the development of an online masters degree in computer science at Georgia Tech. The first semester of that program has been completed and the students are satisfied and the administration is optimistic, but not declaring victory yet.

    Yesterday, AT&T (and others) announced that they would be participating in the development of tech-oriented "nanodegrees" on the Udacity platform. (Udacity also hosts the Georgia Tech MS).

    IBM may not be offering the same 3-phase training that I had as a new hire, but they are offering MOOCs at universities through their Academic Initiative and have recently agreed to partner with 28 business schools and universities on developing data science curriculum and programs.

    The Georgia Tech MS degree will cost students $7,000 and a Udacity nanodegree will cost approximately $2,400 -- about $200 per month for 12 months. AT&T is sponsoring some of their employees in the masters program and will offer internships to 100 nanodegree graduates.

    Online education has also reduced the cost of tuition reimbursement benefits for employers. Starbucks is offering tuition reimbursement for employees who complete an online bachelor’s degree at Arizona State University.

    Online education has boomed with the spread of Massive open online courses (MOOCs). MOOCs have had successes and failures, but the infusion of capital and interest has triggered a wave of innovation in technology and pedagogy. New media often mimic and substitute for old media. We first saw MOOCs as a replacement for university education, but it may be that their major impact will be on vocational training.

    Udacity has pivoted from university education to lifelong vocational training and the deal looks good enough to induce companies like AT&T and Starbucks to cover part of the cost.

    What might it mean if employer-subsidized vocational training catches on?

    Traditional universities will lose students. The majority of students see a degree as a path to a job, and universities control certification. If a $2,400 nanodegree gets one a good entry-level job, many students will skip the university.

    University education is much more common today than it was when I started at IBM. Universities, like many other organizations, typically try to grow, leading to aggressive marketing programs and lowered admission standards. While a university might have an incentive to admit and retain poorly qualified students, an employer does not. The company personnel department may replace the university admission committee as the gatekeeper to the middle class.

    That might be efficient, but tying education to employment has a downside. Consider the effect of tying medical insurance to employment -- it is an important part of an employee's benefits, but it discourages mobility, harming both the economy and the individual.

    The courses I have mentioned here are not typical MOOCs, but they are compatible with MOOCs. For example, Udacity has not spelled out the details, but a nanodegree will involve testing for certification and, no doubt, more personal interaction with instructors than today's MOOCs. However, they also intend to make the teaching material available as a MOOC. Self-study students will not get credit or personal attention, but they will have access to the same material as paying students. The material will be a fringe benefit for society and an advertisement for Udacity.

    As Steve Jobs used to say -- one more thing. I've been talking about vocational training, but I think there is also demand for curiosity-driven, non-degree, lifelong edcuation -- edutainment if you will. That may be the way those nanodegree graduates round out their education.

    Update 9/19/2014

    At a lower level, we are also seeing online trade schools. For example, Open Colleges in Australia. They claim to have over 700,000 students and offer trade school courses in these areas:

    That sounds good, but digging a bit deeper I became skeptical. The site reminded me of a TV ad for a trade school -- lots of rosy career promises, special offers and testimonials from students. I tried to find the cost of their classes or career preparation programs, but did not find a way to get the cost without starting to enroll. I did, however, learn that I could pay by the month or fortnight and would get a discount if I paid up front for the entire course.

    With my skepticism aroused, I took their multiple choice test of my language, literacy and numeracy skills and learned that I was skillful enough to be admitted to their Certificate III and IV level courses. I was not qualified for their Diploma level courses. Did I mention that I gave random answers to the questions? Just lucky I guess.

    I did not spend much time on their site and they may do a great job of training and finding employment for their students, but whether they do or do not, there may be room for MOOC-like trade schools.

    Update 6/1/2015

    Google wants more Android developers so Udacity has added an Android Development nanodegree.

    Saturday, June 14, 2014

    Google buys a satellite company -- Skybox Imaging -- why and what next?

    This acquisition is ostensibly a data play, not a connectivity play, but couldn't a constellation of low-earth orbit Skyboxes cover the globe? Teledesic 2?

    A few days ago, we looked at Google's efforts to bring connectivity to developing nations and rural areas using satellites and high altitude platforms.

    Now, Google has acquired Skybox Imaging for $500 million. Skybox uses very small satellites orbiting at an altitude of 600 kilometers, which means they move at over seven kilometers per second relative to the surface of the Earth. Their technology is sufficiently advanced to compensate for that speed and produce videos that clearly show the movement of vehicles -- check this video:

    Skybox Imaging HD Video of Las Vegas on March 25, 2014 (1080p) from Skybox Imaging on Vimeo.

    Off hand, this seems more like a data gathering move than part of a developing nation and rural connectivity strategy -- they can use these images to look at traffic (for self-driving cars?), count ships in a port or cars in parking lots, etc. For examples, check this video:

    But, they plan to launch many of these low-earth orbit satellites -- couldn't they be used for global connectivity as well? Several companies were formed to do just that in the 1990s. Bill Gates, Paul Allen and a Saudi prince backed the best-known one, Teledesic, but the technology of the time was not up to the task and the company failed.

    Shifting emphasis, you can see an earlier video of the Skybox entreprenurial team presenting their vision at the Stanford Busiiness School here.

    While this effort is not directly tied to providing connectivity, it beefs up Google's space technology and skills. Google is becoming a player in the space game along with Internet entrepreneurs Jeff Bezos, founder of Blue Origin, Elon Musk, founder of SpaceX, and Richard Brnason's Virgin Galactic. There are rumors that Google is negotiating a stake in Virgin Galactic and one can't help thinking Musk and Bezos are watching all this with interest.

    Update 6/14/2014

    More cool Skybox Imaging videos

    Update 2/25/2017

    Google has sold Skybox Imaging to Planet Labs, an Earth imaging company. My speculation about possible satellite-based Internet service was off base.

    Thursday, June 12, 2014

    Can Google connect the "other three billion" in developing nations and rural areas?

    Doing well by doing good

    One of our grand challenges is to bring Internet connectivity to rural areas and developing nations and Google may be working toward that goal. They are experimenting with terrestrial and extra-terrestrial wireless technology. I will look at terrestrial wireless in a subsequent post -- this one focuses on extra-terrestrial technologies.

    During the last couple decades, NGOs, governments and entrepreneurs have worked with four extra-terrestrial connectivity technologies:

    (See pictures at the end of this post)
    Let's look at Google's projects in this context.

    High altitude platforms (HAPs) are blimps, drones or balloons that hover or circulate in the stratosphere. They have cloudless access to solar energy and being above the weather helps with control, but their signals must travel through rain and clouds. They are the lowest flying technology, so packet latency is relatively small, but so is their "footprint" -- the area their signal covers on the ground.

    The most visible HAP Internet effort has been that of Sanswire, which has run well-publicized tests for over a decade. Sanswire has gone through bankruptcy, announced projects in Latin America that never materialized and faced complaints by suppliers and employees, but they are still working on Internet connectivity.

    Google has two HAP projects, Project Loon, using balloons and a drone project using technology from recently purchased Titan Aerospace. There have been reports of Google blimp trials, but I've not seen any details on those. Let me know if you have more information.

    Most satellites -- like the Space Station and sensing satellites -- are in low Earth orbit (LEO). LEO satellites move relative to the ground, which means that either communication windows are intermittent or many satellites -- a "constellation" -- are needed to cover the planet.

    The first LEO Internet project I know of was used for intermittent connectivity in Africa during the early 1990s. Shortly thereafter, a number of entrepreneurial LEO projects were announced. The most ambitious was Teledesic, which proposed Internet connectivity for the entire planet using a constellation of 288 satellites orbiting at 700 kilometers. Teledesic had high-profile backers like Bill Gates, Paul Allen and a Saudi prince, but the technology of the day was not up to the task and the company failed.

    Today, the best-known LEO communication system is Iridium's satellite phone service, consisting of 66 LEO satellites. (Iridium was conceived by motorola as an Internet project, but was scaled back to telephony, went bankrupt and reemerged as a phone service).

    This week, Google acquired Skybox Imaging, a company that has put a LEO satellite in a 600 kilometer orbit. The company was formed for data gathering, for example for providing real time video and images of traffic on roads, the sea and in the air, environmental monitoring, or map and earth imaging.

    As we see in the following heat-map video from Skybox, satellite imaging proliferated rapidly between 1986 and 2012:

    This sort of imagery has both economic and military value so it will provide Google both revenue and expertise in the short run. Might they be planning to parlay that into a constellation of Skybox communication satellites -- Teledesic II with modern technology -- in the long run?

    Medium Earth orbit (MEO) satellites are used for communication and navigation. Google recently announced a project with O3b Networks (other three billion). O3b currently has four satellites in 8,000 kilometer equatorial orbits and they plan to launch four more this year. They say those eight satellites will enable them to offer continuous service to all parts of the Earth within 45 degrees of the Equator.

    The project with Google is headed by two O3b executives and they speak of spending billions dollars and putting at least 180 satellites in orbit. When they speak of 180 satellites, one wonders whether they are considering a LEO constellation.

    (Update: Google's project with O3b Networks has ended, but O3b's MEO technology could still be deployed in Cuba and elsewhere).

    Today's commercial satellite Internet connectivity is provided by geostationary satellites, which are positioned above the equator and remain stationary with respect to the surface of the earth since they orbit exactly once per day. Their orbit altitude enables multi-country footprints, but latency and launch costs are high.

    Geostationary satellites have been used in rural areas and developing nations since the early days of the Internet, and the industry has remained viable as a result of technical progress in launch technology (public and private), antennas, solar power, radios and other electronics, as well as tuning of TCP/IP protocols to account for the 1/4 second latency due to the orbital altitude. (I've had surprisingly natural voice over IP conversations with people on geostationary satellite connections).

    Have those technologies progressed to the point where HAPs and lower orbit satellites are now viable as well?

    Since the beginning years of the Internet, NGOs, government agencies and entrepreneurs has been working on the Grand Challenge of connecting developing nations. They have not succeeded, but Google, with improved technology, deep pockets, a long-range viewpoint and economic motivation (ads) may be able to pull it off.

    Finally, I cannot end this post without wondering whether Jeff Bezos, founder of Blue Origin, Elon Musk, founder of SpaceX, and Richard Brnason, founder of Virgin Galactic are eyeing those other three billion people.

    Update, 6/20/2014

    There is a 99 comment discussion of this post on the Slashdot Web site.

    Update 11/15/2014

    A proposal for a low-earth-orbit satellite constellation like that attempted by Teledesic is being re-considered using modern launch and satellite technology. Entrepreneurs Elon Musk (launch technology) and Greg Wyler (satellite technology) are working on a constellation of 700 low-earth orbit satellites to provide Internet connectivity to rural areas and developing nations and Google has several related projects. Will they realize Teledesic's 1990 vision using 2020 technology?

    In an earlier post, I proposed the use of conventional geo-stationary satellites in Cuba. If this effort by Musk and Tyler were to succeed, and the Cuban government would allow it, this would provide an even better interim connectivity solution. (Modern fiber infrastructure would remain as the ultimate goal, but satellites would provide an affordable interim step).

    Update 1/27/2014

    Well, the die is cast. Google has elected to back Elon Musk's satellite Internet project in competition with Greg Wyler. It's great to see two major teams competing to be ISPs to the Earth. If one or both succeed, the upside for humanity is clear, but is there a downside? It is a bit frightening to think of a global ISP with, say, a billion customers. What sort of regulation or oversight would be needed?

    Elon Musk announcing the opening of the SpaceX
    satellite engineering and development office in Seattle

    Teledesic animation showing a satellite
    constellation that would cover the planet.

    There has been significant technical progress in connectivity
    to geostationary satellites since this picture was taken in India.

    Figure from Arthur C. Clarke's article proposing
    geostationary satellites for world wide radio coverage

    Geostationary satellites have large footprints.

    Titan Aerospace solar-poweed drone

    O3b MEOs will cover the middle of the planet/

    Does anyone know anything about this Google blimp? (Photoshopped?)

    Project Loon balloon

    Sandisk's latest HAP attempt

    Airport example from Skybox Imaging (more cool videos here)

    Update 8/6/2016

    Our balloons today are doing pretty much everything a complete system needs to do. We're in discussions with telcos around the world, and we're going to fly over places like Indonesia for real service testing this year.

    Astro Teller, head of Google X, gave a Ted Talk on the unexpected benefit of rewarding failure. Google X works on "moonshot" projects. They start with a radical idea then immediately begin trying to prove it is impossible by attacking what appears to be its weakest spot. If the weak spot cannot be overcome, they kill the project and reward the participants.

    In this talk, Teller described several projects that had failed and some that had not yet failed. Project Loon, Google's balloon-based Internet access experiment has not failed so far. In fact, it has made significant progress.

    Teller gave a status report on Project Loon, which is still very much alive, starting at 8 min 22 sec of the following full talk:

    The transcript of his Project Loon report follows, but you should check out the video for the images.
    Probably the craziest sounding project we have is Project Loon. We're trying to make balloon-powered Internet. A network of balloons in the stratosphere that beam an internet connection down to rural and remote areas of the world. This could bring online as many as four billion more people, who today have little or no internet connection.
    But you can't just take a cell tower, strap it to a balloon and stick it in the sky. The winds are too strong, it would be blown away. And the balloons are too high up to tie it to the ground.
    Here comes the crazy moment. What if, instead, we let the balloons drift and we taught them how to sail the winds to go where the needed to go? It turns out the stratosphere has winds that are going in quite different speeds and directions in thin strata. So we hoped that using smart algorithms and wind data from around the world, we could maneuver the balloons a bit, getting them to go up and down just a tiny bit in the stratosphere to grab those winds going in those different directions and speeds. The idea is to have enough balloons so as one balloon floats out of your area, there's another balloon ready to float into place, handing off the internet connection, just like your phone hands off between cell towers as you drive down the freeway.
    We get how crazy that vision sounds -- there's the name of the project to remind us of that. So since 2012, the Loon team has prioritized the work that seems the most difficult and so the most likely to kill their project.
    The first thing that they did was try to get a Wi-Fi connection from a balloon in the stratosphere down to an antenna on the ground. It worked. And I promise you there were bets that it wasn't going to. So we kept going.
    Could we get the balloon to talk directly to handsets, so that we didn't need the antenna as an intermediary receiver? Yeah.
    Could we get the balloon bandwidth high enough so it was a real Internet connection? So that people could have something more than just SMS? The early tests weren't even a megabit per second, but now we can do up to 15 megabits per second. Enough to watch a TED Talk.
    Could we get the balloons to talk to each other through the sky so that we could reach our signal deeper into rural areas? Check.
    Could we get balloons the size of a house to stay up for more than 100 days, while costing less than five percent of what traditional, long-life balloons have cost to make? Yes. In the end. But I promise you, you name it, we had to try it to get there. We made round, silvery balloons. We made giant pillow-shaped balloons. We made balloons the size of a blue whale. We busted a lot of balloons.
    Since one of the things that was most likely to kill the Loon project was whether we could guide the balloons through the sky, one of our most important experiments was putting a balloon inside a balloon.
    So there are two compartments here, one with air and then one with helium. The balloon pumps air in to make itself heavier, or lets air out to make it lighter. And these weight changes allow it to rise or fall, and that simple movement of the balloon is its steering mechanism. It floats up or down, hoping to grab winds going in the speed and direction that it wants.
    But is that good enough for it to navigate through the world? Barely at first, but better all the time.
    This particular balloon, our latest balloon, can navigate a two-mile vertical stretch of sky and can sail itself to within 500 meters of where it wants to go from 20,000 kilometers away.
    We have lots more to do in terms of fine-tuning the system and reducing costs. But last year, a balloon built inexpensively went around the world 19 times over 187 days. So we're going to keep going.
    Our balloons today are doing pretty much everything a complete system needs to do. We're in discussions with telcos around the world, and we're going to fly over places like Indonesia for real service testing this year.

    Monday, June 09, 2014

    Netflix blames Verizon, Verizon blames Netflix -- let's see the data.

    Netflix says Verizon is a streaming bottleneck, Verizon says the congestion is upstream -- we need data not accusations.

    Netflix recently showed messages like this one when they had to adjust the bit-rate on their streams to FIOS customers:

    Verizon responded, claiming that the congestion was upstream from FIOS, and threatened legal action. Netflix says they will drop the messages for now.

    The public and regulators need more data and transparency, not less. For example, here are Netflix data rate histories for Google Fiber, Comcast, Verizon FIOS and Verizon DSL:

    This data clearly shows the improvement in Comcast speeds when they and Netflix agreed to terms in February. (One wonders how that rapid improvement was achieved and how much investment it required).

    Verizon DSL is slow because of the technology. Those Verizon customers who can get FIOS are surely better off, but one wonders why Verizon fiber is so much slower than Google fiber and Comcast cable.

    If Netflix congestion is upstream from Verizon, they should present data demonstrating that.

    Update 6/10/2014

    Google also publishes data on YouTube streaming performance for ISPs. I checked the performance of Time Warner Cable, my not-so friendly ISP, in Los Angeles, my city.

    The following graph shows YouTube video consumption for time slots beginning at 6 AM and peaking at 9 PM.

    Google also reports the percent of the time an ISP delivers high definition (720p), standard definition (360p) and lower quality. For me, the worst performance, HD 81% of the time and SD 19%, occurs between 9-10 PM.

    This is one more source of ISP performance data -- check it out and see how your ISP rates compared to others in your location (if there are others).

    Update 6/13/2014

    The FCC will scrutinize the pay for service deals Netflix and Comcast and Verizon. They will also look at others, including Google.

    FCC Chairman, Tom Wheeler, said he already had copies of the contracts, but I hope he asks the companies to provide traffic data and also disclose the cost of the infrastructure investments they make to keep service reasonable.

    Update 7/7/2014

    YouTube has joined Netflix in linking to data when video delivery slows down. I was watching a YouTube video that began to stutter. When it paused to rebuffer, I saw this message:

    When I clicked on the find out why link, it took me to Google's Video Playback Checklist, which lists things that might go wrong and suggests fixes -- no blame on your ISP -- so far. If none of those fixes help, they also provide a link to the Google Video Quality Report (described above). As we asked -- they show us the data.

    Update 7/19/2014

    I've asked to see congestion data and to know the cost of alleviating the bandwidth shortage noted by Netflix customers and other Internet users. Mark Taylor, VP of Content and Media at Level 3, a transit provider used by Netflix, has given us some data and some costs. This diagram shows data at one of the of the ten exchange points between Verizon and Level 3:

    We see that the congestion between Level 3's customer (Netflix) and a Verizon retail customer occurs in the forwarding of packets between their routers (in the same building). Both the Level 3 and Verizon networks are uncongested (green) but the links between their routers are congested (red).

    That is the data for this exchange point, but what about the cost of alleviating the problem? Taylor says it is a few thousand dollars and five minutes work to install 10gb port cards in the routers. He says they have been asking Verizon to install the cards for many months and offers to have Level 3 pay for them. He says he will throw in the short cables between the routers.

    If more capacity were needed after those eight port cards were installed, new equipment would have to be purchased, but that would be a small fraction of the cost of the networks the routers connect.

    Note that the above diagram is based on a Verizon post describing the situation at the Los Angeles exchange point. Furthermore, it shows under utilization on the Verizon network -- no need for data caps in that case :-).

    Update 7/24/2014

    David Young, Verizon's Vice President for Regulatory Affairs, has posted a reply to the claim by Mark Taylor of Level 3 that Verizon is causing Internet congestion. Young says the problem is not datacenter exchange point congestion, as Taylor claims, but congestion in the Level 3 network. In fact he says that:
    Level 3 insists on only using its existing settlement-free peering links even though, as Level 3 surprisingly admits in their blog, these links are experiencing significant congestion.
    This is Verizon's view of the situation -- transit providers like Level 3 are the problem:

    But, when I looked back at Mark Taylor's post, I saw no such admission. In fact, Taylor says "our network has plenty of available capacity" and goes on to state that:
    I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS (his caps).
    These are conflicting statements -- who to believe?

    Considering Verizon's track record, I'd give the benefit of the doubt to Level 3. Verizon promised to install FIOS fiber in my neighborhood several years ago, but subsequently changed their mind and decided instead on a "gentleman's agreement" dividing the wireless (Verizon) and land-line (cable companies) ISP markets. Indeed Verizon has a track record of broken promises -- Bruce Kushnick has documented 300 billion dollars worth of them.

    That history makes me suspicious that Level 3 is telling the truth, but it is not proof. Proof would require transparency on both sides. Level 3 and Verizon should show us the data -- let's see the traffic logs for your networks. (Or at least let the FCC staff see them).

    Update 10/15/2014

    I've asked to see the data. It's not congestion data, but Generator Research has estimated the cost of delivering Netflix content in their report Over-the-Top Television, 2014, which includes an estimate of the cost to Netflix and Comcast of delivering Netflix content and of the fee Netflix is paying Comcast.

    Their estimate of Netflix's "fast lane" fee to Comcast is $.86 per subscriber per month. That is a small portion of the inflated monopoly/oligopoly prices we pay to Comcast and other Internet service providers. It seems to me that we should be paying more attention to the lack of competition in the ISP industry than to network neutrality.

    Update 11/9/2014

    M-Lab monitors Internet performance and has published a report on ISP Interconnection and its Impact on Consumer Internet Performance.

    The report concludes that
    Using Measurement Lab data, and constraining our research to the United States, we observed sustained performance degradation experienced by customers of Access ISPs AT&T, Comcast, Centurylink, Time Warner Cable, and Verizon when their traffic passed over interconnections with Transit ISPs Cogent Communications (Cogent), Level 3 Communications (Level 3), and XO Communications (XO). In a large number of cases we observed similar patterns of performance degradation whenever and wherever specific pairs of Access/Transit ISPs interconnected. From this we conclude that ISP interconnection has a substantial impact on consumer internet performance -- sometimes a severely negative impact -- and that business relationships between ISPs, and not major technical problems, are at the root of the problems we observed.

    It seems that interconnection between transit ISPs and ISPs that connect customers (access ISPs) is indeed the weak link in the content delivery chain.

    The above figure gives an overview of download throughput between Access ISP and Transit ISP pairs during 2013. For each of the 28 graphs, the Y axis shows the % of hours with the given download throughput, and the X axis shows the download throughput (Mbps). Red, on the left, indicates sub-broadband speeds (download throughput below 4 Mbps). What this table demonstrates is not simply a proliferation of performance issues, but that these issues cannot be laid at the feet of any one Access ISP, or any one Transit ISP: no Access ISP performs badly to all Transit ISPs, and no Transit ISP performs badly for all Access ISPs. Therefore, if the problem is not at one end, and not at the other, it must be in the middle around the interconnection between the two.

    For example, my ISP, Time Warner falls below 4 Mbps (red) over 30% of the time when delivering Cogent packets, but does much better with data from other transit ISPs.

    Wednesday, June 04, 2014

    A tale of two industries: package delivery and Internet service

    Why do customers like the package delivery industry and dislike information delivery service? Competition and the US Postal Service.

    Comcast CEO Brian Roberts recently complained that Netflix paid the Post Office for delivery of small packages (DVDs), but did not want to pay Internet service providers (ISPs) to deliver the same content over the Internet.

    In his view, the Post Office is in the consumer package delivery business and Comcast and other ISPs are in the information delivery business and both should be paid for their service.

    The University of Michigan publishes the American Customer Satisfaction Index in which they rate both companies and industries. Let's look at their latest ratings of the consumer package delivery and ISP industries.

    They rate 48 industries and it turns out that the package delivery industry is rated seventh overall and the ISP industry is 48th. Furthermore, merger hopefuls Time Warner Cable and Comcast are the lowest rated.

    How might we explain the differences in customer satisfaction in these two images that Mr. Roberts considers similar? The difference is that there is competition in customer package delivery between Federal Express, UPS and the Post Office. We would normally expect three companies to tacitly establish oligopoly prices, but in this case the Post Office is disruptive -- Federal Express and UPS must compete with a government agency.

    Monday, June 02, 2014

    The best video ever on network neutrality and the state of the Internet in the US

    As long as analysts, geeks and lobbyists are the only folks who care about Internet infrastructure and policy, we can be pretty sure that nothing will improve, but what if the general public becomes interested?

    Last night I watched episode five of the HBO comedy news cast "Last Week Tonight" in which host John Oliver narrated a 13 minute segment on network neutrality and the sorry state of the Internet in the United States.

    I may be a biased geek and disgruntled consumer, but the segment was right on and, more important, extremely funny.

    Is the Internet finally becoming a politically important issue? (Be sure to listen to the audience reaction at the end of the clip before you answer).

    Update 6/26/2014

    FCC Chairman Tom Wheeler responded to the video and Oliver followed up with another very funny segment.

    Update 6/28/2014

    Terry Gross interviewed John Oliver about his new HBO show -- the first 7 minutes are devoted to Oliver's Internet piece -- funny and informative.

    Consumer groups and big corporations like Google and Facebook agreeing on net neutrality is like Lex Luther agreeing with Superman.

    Comcast CEO Brian Roberts explained that there is no competition between Comcast and Time Warner Cable.

    Picking Tom Wheeler to head the FCC is like hiring a dingo as your baby sitter.

    President Obama golfing with Comcast CEO Brian Roberts

    Harvard and MIT release data on 641,139 MOOC student/course enrollments

    This illustrates the difference between a university based program and a commercial enterprise

    In the early days of the Internet, the National Science Foundation, which operated the major global backbone, NSFNET, had an acceptable-use policy (AUP) stipulating that the Net was to be used for teaching and research only, not commercial applications. The AUP led to an academic culture that encouraged and rewarded research and sharing.

    In that tradition, MIT and Harvard have published data on 16 HarvardX and MITx courses offered during the 2012-13 academic year. The release includes a 20-column, 641,139-row CSV file ready to open with a spreadsheet or other statistical analysis program along with several papers analyzing the data and describing their method of hiding the identity of the students taking the courses.

    Download the data and and analyze away -- and don't forget to share your results as a comment on the database download page.

    Comcast speed dips then recovers rapidly after the Netflix agreement is made

    What caused the sudden improvement in Comcast service and how much did it cost?

    As shown here, Netflix download speed began dropping for Comcast customers last November and began picking up in February. It rose another 2% in April.

    What happened? The cynical, but quite plausible, answer is that Comcast refused to upgrade their equipment while negotiating with Netflix on fees for faster service, resulting in poor performance and angry customers, and they made the necessary investment after a deal was concluded in February.

    Infrastructure investment is central to the network neutrality debate. If this improvement was due to investment upstream from Comcast, they should present data demonstrating that. Regardless of who made it, I would like to know how much we are talking about here -- what did it cost to increase performance so dramatically?

    The public, the FCC and the FTC need that sort of transparency to formulate policy that will incent infrastructure investment.