Affordable Starlink prices coming to low and middle income countries

SpaceX first departed from its uniform Starlink pricing policy when it offered a reduced price for throttled service in France two years ago. Since then, many new, higher-capacity satellites have been launched, enabling SpaceX to reduce prices in low and middle-income countries, like Kenya.

Starlink (≥ 100 Mbps) satellite Internet growth (source)

Kenya had 405 geostationary satellite internet subscribers when Starlink became available in July 2023. By March 2024 there were 4,808 satellite Internet users, according to data from Kenya’s Communications Authority.  

The price and offering have changed. Initially, the hardware, including shipping and handling, was KES92,100 ($715) and the monthly service was KES6,500 ($50). 

Today, the service price is the same for uncapped access today, but the hardware cost is down to KES45,500 ($353) and they offer a hardware rental plan for KES1,950 ($15)/month. 

Starlink's unlimited data plan will appeal to affluent consumers and organizations like schools and businesses. It might also appeal to people who live close enough to each other to share a single account although that must be a violation of Starlink's contract.

Starlink also has a new affordable plan with 50GB of data for KES1,300 ($10) plus KES20/GB ($0.16) for overage data. The affordable plan is a strong competitor for Airtel and Safaricom, which combined have 95.2 % of the Kenyan mobile broadband market and offer fixed connectivity in limited areas. (I wonder how much "mobile" data is consumed by users on computers connected to mobile hotspots at home).

Starlink has a speed advantage over Airtel and Safaricom and serves areas with low population density they do not reach. (As of this writing, Starlink is at full capacity in Nairobi). Despite reduced prices, Starlink's equipment cost is too high for many Kenyans, hence the hardware rental plan. 

Poa! Internet is another competitor. Poa! offers unlimited 5 Mb/s fixed connectivity for only KES1,575 ($12) /month with an installation fee of KES2,500 ($19.40). At 5 Mb/s it is slower than the others, but fast enough for most Internet applications. For example, Netflix says it is sufficient to stream 1080p HD video. (For many applications, latency is more important than speed).

Poa! Internet customers average about 200 GB/month but unlimited Starlink customers consume much more than that since they can support multiple users online simultaneously. (I have streamed four Netflix videos simultaneously on a ship off the coast of North Africa). A Starlink user in Kenya told me he had been using "multiple terabytes a month" and market tests at a large Kenyan ISP estimated the Starlink consumption would be over 2 TB/month.

Steve Song has pointed out ways Starlink contributes to inequality in Africa. Still, it is also bringing the  Internet to users and organizations that did not have access previously and putting pressure on legacy ISPs to compete. (Starlink is partially responsible for recent expansion by existing companies like Airtel and Safaricom). 

Starlink has demonstrated a willingness to adapt in Kenya as they did in France two years ago and we can look forward to ongoing improvement as new, higher capacity satellites are launched on improved rockets and the African ground infrastructure improves. We can also look forward to competition from other satellite ISPs like Amazon and Quianfan. It will be interesting to see the Kenyan Internet in ten years.

Update 9/16/2024

Suspected Kenyan Starlink gateways (source)

I've been told that Starlink latency from Nairobi to a Netflix node in Lagos is about 50ms because there is a gateway in Nigeria, but latency to a Kenyan banking site is 400ms or more because it requires transit over Africa's terrestrial infrastructure. 

Since Starlink's only African gateway is in Nigeria, my guess is that latency to Europe or even the US is somewhere between those two because undersea cables land in Nigeria.

A gateway in Nairobi would improve latency in Kenya and the East in general.

Russian Internet pioneer sentenced to two years in prison

Relcom machine room. Kremvax, the micro-VAX shown here, linked Relcom to Finland and the outside world and the modems communicated within
The Soviet Union.

Last month, Russian Internet pioneer Alexey Soldatov was sentenced to two years in a labor colony on charges of “abuse of power.”

Soldatov, the co-founder of Russia's first  Internet service provider, Relcom, was convicted for his role as a co-founder and director of the Russian Institute for Development of Public Networks (RIPN). RIPN was founded in 1992 to support research and education networking. One of its functions was the “registration of IP numbers for the customers from the blocks delegated RIPN by the Europe coordination body RIPE".

As Russia’s first local Internet registry (LIR), RIPN was allocated a block of IP addresses to distribute to customers for a fee. By early 2019, RIPN had assigned about 490,000 IPv4 addresses to more than 700 clients, mostly scientific and educational organizations.

In December 2018, the RIPN board decided to dissolve the organization since it was no longer commercially sustainable and its role as an operator of scientific and educational networks was no longer necessary. In April 2019, RIPN informed its clients that it was terminating its work as an LIR and re-registering their IP addresses in the RIPE database under a Czech company called Reliable Communications, which Alexey Soldatov and Alexey Shkittin owned. Shkittin was also a director of RIPN at the time. When the government learned of the dissolution plan, they stopped it and RIPN reclaimed the IP addresses. (Today there are over 20,000 RIPE LIRs in 76 countries).

Defenders of the aborted transfer say the addresses had to go somewhere if RIPN was being dissolved and they were neither property nor worth the 500 million rubles ($8.1 million) as charged. They also point out that the charges were initiated by Andrey Lipov, a prominent figure in the Russian Presidential Administration and the head of Roskomnadzor. the Federal Service for Supervision of Communications, Information Technology, and Mass Media.

Still, the transfer to a Czech company owned by Soldatov and Shkittin when they were RIPN directors looks fishy. Was this a case of embezzlement or political retribution by a government that wants to surveil and censor the Internet?

I don't know the facts in this case, Russian law, or, with certainty, what motivated Soldatov to attempt to transfer those IP addresses, but I do know something about the character of the people involved. 

I have first-hand knowledge of Alexey Soldatov. Relcom was launched on August 1, 1990, in the Kurchatov Institute of Atomic Research in collaboration with the DEMOS cooperative. (Soldatov is a physicist). DEMOS was a UNIX-like operating system derived from BSD Unix which Relcom developed and distributed to research and education organizations.

They used UUCP links to communicate with their Russian customers and soon connected to Europe (EUnet) through Tampere University of Technology in Finland. The ability to make regular, long international calls needed for data transfer was unheard of at that time in Russia -- The Kurchatov Institute was an important research center.

My colleague Juri Gornostaev and I used that UUCP link to organize the First East-West International Conference on Human-Computer Interaction in Moscow. After the conference, I stayed in Moscow to meet and hang out with the Relcom staff. They were friendly, smart, idealistic, and anti-communist. They were members of the international Internet community as imagined by visionaries like Vannevar Bush and J. C. R. Licklider.

Relcom sent Boris Yeltsin's speech on a tank to The West.

A few days after I returned home, a group of Russian officials attempted a coup against the relatively progressive Gorbachev government. Mass media were shut down or broadcast old movies and operas, but Relcom stayed online, relaying news within Russia and between Russia and the rest of the world. As Relcom staff member Polina Anatova said at the time, "Thank Heaven, these cretins don't consider us mass media!"

I also know the character of Vladimir Putin and the Russian government. Russia commits daily war crimes in Ukraine and political opponents who are not imprisoned tend to fall from the windows of tall buildings. Furthermore, Freedom House assessed Internet freedom in 70 nations last year and the only nations judged to have less Internet freedom than Russia were China, Myanmar, Iran, and Cuba. 

Given the uncertainties mentioned above, the government's track record, Soldatov's contributions to society, and the fact that the IP address transfer never took place, he has been under house arrest awaiting trial and is in ill health, this prison sentence is politically motivated and morally unjustified.

For more on Relcom's early days and the role it played at the time of the Soviet coup attempt, see:

• Relom, an appropriate technology network
• Before the Twitter revolution, there was the Usenet revolution
• Wide Area Collaboration
• History is written and revised by the winners -- can the Internet Archive change that?
• Soviet Coup Archive, including Usenet traffic
• The Internet as an information resource

Update 8/29/2024

MK reported that Alexey Soldatov spent his first night in pretrial detention center No. 4 in terrible conditions -- sleeping on the floor without a mattress in a cell for 40 people. The article also points out that he was sentenced to two years in prison even though the prosecutor’s office requested a more lenient sentence.

The Board of the Internet Society (ISOC) of Bulgaria has called upon Russia to release Soldatov. They cite his critical medical condition. In recent years he has had two cancer-related surgeries and has chronic cardiovascular and other health-related issues. A news article published on August 26 reported that he has a fever, problems breathing and his left lung is practically not working.

ISOC Bulgaria points out that suspended sentences are not unusual for people who have not committed crimes before and calls for Soldatov's immediate release while his conviction is being appealed. Other organizations should do the same. 

Is the "father or the Russian Internet" a criminal?

Post revised -- see https://circleid.com/posts/20240821-russian-internet-pioneer-sentenced-to-two-years-in-prison.

Geely Geespace update -- global centimeter-level positioning services

Potential Geesat applications (source)

Chinese automobile conglomerate Geely has made significant strides since I last wrote about their Geesat LEO constellation for mobile vehicle connectivity, They launched the first nine satellites in June 2022 and a second batch of eleven satellites in February 2024. The mass of the first nine satellites was 100 kg and the mass of the second eleven was 130 kg so they are not identical.

Today, they have twenty satellites orbiting at 600 km with a 50-degree inclination, and the recently launched eleven are equipped with "artificial intelligence" remote sensing capabilities, allowing them to capture clear high-resolution images with a resolution ranging from 3.2 ft to 16.4 ft. These satellites can provide data and imagery for multiple applications, including surveillance, urban planning, and infrastructure management. 

These satellites are the first of a 72-satellite first-phase constellation. The company plans to provide “global real-time data communication services” once the 72-satellite first-phase constellation is completed next year and “global centimeter-level positioning services” when the 168 second-phase satellites are in orbit at an unspecified time.

RTK-PPP (source)

Centimeter-level positioning would be impossible using only GPS (or BeiDou) satellites that orbit at around 20,000 kilometers, but Geely is planning to use RTK-PPP (Real-Time Kinematic and Precise Point Positioning) to provide centimeter-level positioning accuracy for GPS satellites.

As shown here, RTK-PPP augments location data from multiple satellites with the rover's position relative to a terrestrial base station. (The rover could be an autonomous car, truck, crop harvester, etc.).

One can imagine base stations along highways, freeways, and even in cities, but that would require investments by governments at all levels, auto producers, and other private companies. It would also need standards that are compatible with all GNSS systems. This will not happen overnight, but I am not surprised to see Geely leading the way because:

• Geely founder and Chairman Li Shufu is a risk taker in the mold of Elon Musk and Geespace was initiated about six years ago when he “floated his idea of using hundreds of proprietary mini, low-orbit satellites as a more accurate global positioning system for self-driving cars.” 
• China has built the world's biggest EV charging network, in stark contrast to US failure. This indicates a will to invest in mobility infrastructure.
• Geely is aware of and perhaps collaborating with Chinese GNSS operator BeiDou which is working toward global centimeter-accurate positioning utilizing LEO, MEO, and GEO satellites augmented by terrestrial ground stations. 
• Geely brands sold 2.79 million units in 2023 and China space expert Blaine Curcio concludes that "Within 5-10 years, Geespace could conceivably have a constellation of satellites connecting all Geely cars in a very interesting example of vertical integration (actual use cases pending)." 
• ChatGPT reports that BeiDou currently offers a Ground-Based Augmentation System (GBAS) coverage in much of China and plans to cover the entire nation, which is true, but mistakenly claimed there were GBAS instances outside of China at this time. (However, BeiDou's standard satellite navigation service is used in many nations, which must have confused ChatGPT).

A final note -- several of the references in this post are to Blaine Curcio's China Space Monitor. Check it out if you are interested in what China is doing in space. I also used but took with a grain of salt, conversations with ChanGPT: chat1 and chat2.

Two new Chinese Internet service constellations and their market

China's G60 Science and Technology Innovation Corridor, the home of two new Internet-service mega-constellations

Chinese plans for low-Earth orbit Internet service constellations began with two projects, Hongyun  (156 satellites) and Hongyan (864 satellites). These were eventually sidelined for Guowang, an ambitious, 12,992 satellite constellation that is expected to begin launching satellites this year. But, that is old news.

China's five-year plan designates satellite Internet as a strategic emerging industry and two new constellations have emerged, G60 (12,000 satellites) and Honghu-3 (10,000 satellites). 

Shanghai Spacecom Satellite Technology (SSST), aka Yuanxin Satellite, located in Shanghai's Songjiang district, was founded in 2018 and launched two test satellites in 2019. SSST recently raised 6.7 billion yuan ($943 million) from several sources for its 12,000 satellite G60 (aka Qianfan) low Earth orbit (LEO) Internet service constellation. (Also see this excellent context video). 

Their first phase constellation, called Sailspace, will consist of 1,296 satellites. Regional service will be provided by 648 satellites by the end of 2025 global service will be provided by 648 additional satellites by the end of 2027. By 2030, they hope to have 15,000 satellites in orbit and offer direct-to-mobile service. Shanghai Gesi Aerospace has begun production of the G60 satellites and expects to produce 300/year. (Also see). I've not heard any news about Guowang lately -- it feels like G60 has taken the lead. (Also see).

The satellites will orbit at 1,160 km, which is higher than all the other announced LEO satellite competitors except Telesat. While this will increase latency, collision risk, satellite lifespan, handoff frequency, and coverage footprint should improve. 

Like SSST, Shanghai Landspace Hongqing Technology Co, Ltd. (aka Hongqing Technology) is located in the Songjian District and is planning a third Internet service constellation. The May 24th ITU filing lists the satellite name as "HONGHU-3," but specifies a constellation of 10,000 satellites in 160 orbital planes. Since they have launched two satellites previously, the constellation may simply be called "Honghu." They are constructing a satellite manufacturing facility in Wuxi City near Shanghai. The Chinese launch company Landspace owns 48% so, like SpaceX, they may launch their own satellites.

It is not a coincidence that both Honghu and G60 are being developed in the G60 Science and Technology Innovation Corridor between Highway G60 and a high-speed railway line in the Yangtze River Delta (Also see). Local governments play a major role in funding and developing Chinese industry. Shanghai has published "The Shanghai Action Plan to Promote Commercial Space Development and Create a Space Information Industry Highland (2023-2025)".

Can China support three Internet service constellations?

Planned Starships (source)

These companies are commonly touted as China's answer to Starlink, but they are far behind Starlink in launch cost and rate, rocket and engine manufacturing, international licensing, marketing, etc. The gap will widen when SpaceX's Starship is in production, increasing the ability to launch full and mini v2 satellites and whatever comes after that.

However, the success of these companies will not depend solely on catching or competing with Starlink because of global politics. The Chinese companies will not compete with Starlink or any of its Western competitors for Chinese government and military business. (The Ukraine war has demonstrated both the military value of satellite Internet and the drawback of being dependent on a private company).

Starlink will not offer service in countries like China, Russia, Cuba, North Korea, or Iran. Similarly, the Chinese companies will be prohibited from operating in countries like the US and European nations which currently ban or restrict Huawei equipment. Whether motivated by a desire to encourage domestic industry, promote security, or achieve and maintain technological self-reliance, market separation is increasing, 

Belt and Road participant nations (source)

However, there are many nations where these Chinese companies will compete with Starlink and its Western competitors. The Chinese companies will have an advantage in nations participating in the Belt and Road Initiative -- home to about 70% of the world population and 40% of global GDP. The Chinese advantage is even greater in the 26 Digital Silk Road nations.

Update 6/27/2024

China is expected to launch the first 18 G60 satellites in early August. The launch vehicle was not specified, but "the Long March 6A would be the most capable option from Taiyuan in terms of capacity to low Earth orbit and payload fairing."

The Long March 6A has a carrying capacity to orbit of 4,500 kg, which could accommodate 18 250 kg satellites, around the mass of the first generation SpaceX Starlink satellites, but significantly less than the V2-mini satellites Starlink is currently launching. SpaceX is currently testing its forthcoming Starship rocket which will launch full V2 satellites. 

While China lags far behind SpaceX, G60 seems to be moving faster than Honghu and Gwowang and will launch satellites sooner than Project Kuiper or Telesat.

Update 8/5/2024

China has launched the first 18 Qianfan (aka G60) Internet service satellites:

The satellites launched were produced by Shanghai Gesi Aerospace Technology (Genesat) in Shanghai and In an interview with the Shanghai Securities News in June, an executive from GeneSat revealed plans to “explore launching configurations of 36 and 54 satellites per rocket to accelerate the pace of launches”.

What launch vehicles are they considering? Non-Chinese perhaps?

Update 8/29/2024

GSTL satellite factory

Ace China space watcher Blaine Curcio has spotted a possible dark-horse Chinese communication satellite provider. CGSTL, China’s leading remote sensing satellite manufacturer, has built and launched ~100 satellites in 2 years and been working on space-ground optical communication since 2022.

CGSTL aims to have 1,200 employees, and a manufacturing capacity of 1,000 satellites per year by the end of 2025. They plan to launch 28 remote-sensing satellites in 2025 and 100 in 2026. How will they use the 1,000 satellites per year manufacturing capacity? Will they launch their own communication satellite constellation or perhaps manufacture satellites for one of the service providers mentioned above? Something else?

Update 9/9/2024

Workers raced to protect a Long March 8 at Hainan commercial spaceport from an approaching typhoon. The Spacesail logo on the fairing shown here foreshadows another G60 launch. It also demonstrates that we now have three names for this constellation, G60, Qianfan, and Spacesail. 

@AJ_FI

 posted this photo and several others here.

How to submit a "complaint of judicial misconduct" on Judge Cannon’s handling of Trump’s classified documents case

You must submit a simple form and a brief "statement of facts" supporting your complaint.

The Form

Click here to download the editable submission form.
The form asks six questions -- complete it with the answer shown in red below.

1. Judicial Council of the Eleventh Circuit
2. Name of Judge: Aileen Cannon
3. Does this complaint concern the behavior of the judge(s) in a particular lawsuit or lawsuits? Yes
4. Court: U. S. District Court for the District of Florida
5. Case number: 23-80101-CR-Cannon
6. Are (were) you a party or lawyer in the lawsuit? Neither

The form also requires your name, address, and phone number.

The statement of facts

You should state the facts as you see them., but as an example, you can see the statement of facts I submitted here. 

You can also use ChatGPT or other "AI" programs for ideas. For example, I asked ChatGPT to "give me reasons for filing a complaint of judicial misconduct in Judge Cannon's handling of the Trump documents trial." It generated this answer.

Mail the form and statement of facts

Print the submission form and your statement and mail them in an envelope marked "COMPLAINT OF MISCONDUCT" to:

Office of the Clerk
Elbert P. Tuttle courthouse
56 Forsyth Street N.W,
Atlanta, GA 30303

(Do not put the judge's name on the envelope).

For a talk on why you should do this, see this presentation by Glen Kirshner.

Starlink service is great on (some) cruise ships.

The Seabourn Venture has six Starlink terminals with three antennas on each side of the top deck.

I recently used Starlink on a cruise along the coast of Northwest Africa and I'll summarize my experience below, but first let me explain why I put some in the title of this post. I posted the following request on the Reddit Cruise group: "What has been your experience of Starlink Internet service on Seabourn or other cruise lines? How was latency? Do video chats work smoothly? Games? etc." Here are some of the replies:

• It was amazing. Streamed video on two devices at once, I was able to log in through my VPN and work remotely across the world, and WiFi calling was seamless. I had no idea cruise Internet could be that great.
• The Internet was great, even in Antarctica.
• It was FAST.
• Zoom was fine. WhatsApp calls crystal clear too.
• Latency was a bit high for something like live multiplayer gaming, but good enough for video chats or video streaming.
• Very good mb/s. Variable latency. 
• It's usually still on the slower side of things.
• It's better than without Starlink, but still pretty terrible.
• It still Sucks.
• It's slow hot garbage.

My experience was positive. I was on the Seabourn Venture for a two-week cruise which went as far south as The Gambia in northwest Africa then followed the coast north to the Mediterranean finally stopping in Málaga, Spain.  The ship had six Starlink antennas and two geostationary (GEO) satellite antennas. The GEO satellites were used for ship navigation, weather tracking, etc. The crew on the bridge said their work was not affected by Starlink.

Seabourn offers two Internet packages, surfing and streaming. The surfing package is free for passengers and the faster streaming package is free for the crew. Passengers pay about $20 per day for the streaming package. As the name implies, streaming sites like YouTube and Netflix were blocked for surfing accounts.

According to the ship's Computer Systems Officer (CSO), streaming users always had Starlink access and surfing passengers used the GEO satellites unless there was excess Starlink capacity. He could not say what the criteria for excess capacity were, but their goal is to always keep the streaming customers happy. (Company policy prohibits crew members from playing games).

My Starlink experience was positive. I tested streaming in my cabin by watching four Netflix videos simultaneously on Android and Apple phones, an iPad, and a laptop. The screens were small and Netflix would have reduced video quality, but it worked smoothly. Video chats using WhatsApp and Zoom also worked well despite latency being consistently over 100 ms unloaded and considerably higher while up and downloading files. 

Latency on the ship was much higher than Elon Musk's stated goal of 20 ms -- what are the sources of that latency? 

First is the WiFi network configuration and your location within the ship. Demand also varies during the day and between port and sea days. The size of the ship and the provisioned Starlink capacity are also factors in determining latency.

Carnival Corporation owns Seabourn and 8 other cruise lines. I was on Carnival's smallest ship, which has six Starlink terminals and can accommodate up to 264 passengers with a crew of 120. Carnival's largest ship, the Mardi Gras, can accommodate approximately 6,500  passengers with approximately 1,745 crew members. I wonder how many Starlink terminals the Mardi Gras has and how satisfied the users are. (I got these passenger and crew counts from ChatGPT-4, but it doesn't know how many terminals the Mardi Gras has).

Since the only ground stations in Africa are in Nigeria, our traffic traversed inter-satellite links for most of the trip. Once on the ground, traffic was tunneled through the Carnival VPN to a public Internet point of presence (POP) in an English-speaking location like London, Sydney, or Miami. 

When Starlink was first installed, traffic sometimes went through Hong Kong and users got Chinese replies. The POP location can also affect copyright restrictions. When Max.com became unavailable during the cruise, the CSO explained that they had been switching back and forth between the London and Miami POPs for maintenance purposes and that Max.com was not available in the UK.

This complexity explains the varied results I reported at the start of this post and no doubt Carnival and other cruise lines monitor and tune their networks constantly. If they are doing well they should advertise performance statistics and customer-satisfaction survey results.

Starlink has begun delivering promised latency cuts

In his January 12th  SpaceX update, Elon Musk said the biggest goal for Starlink from a technical standpoint is to get the mean latency below 20 ms. He expanded by saying that given the speed of light, 8 ms is the absolute minimum latency for a satellite at 550 km. He believes they can optimize terrestrial and inter-satellite links, and minimize queueing delays and dropped packets, to recude the the rest of the time to below 10 ms. He predicted that eventually"Starlink will be more responsive than ground Intenet in most cases."

A month later, we saw early results of the latency-reduction effort. On February 12, Oleg Kutkov tested Starlink's Rev 3 and Rev 4 terminals and, as shown here, he found no latency inflation as background upload and download speeds increased simultaneously when using the Rev 3 terminal. He compared the Rev 3 and 4 terminals and found that Rev 4 upload and download speeds were about 50% faster than Rev3. Average ping times were somewhat improved for Rev 4 (88 vs 93 ms), but jitter was significantly lower (9.2 vs 111.9 ms).

You should also check out Dave Taht's take on Oleg's results. He concedes that Starlink has improved dramatically, and outlines steps they could take to further reduce latency.

Last week, SpaceX released news of progress toward the 20 ms latency goal. They have worked to reduce latency throughout the Starlink system. Since the begining of the year, they have deployed and tested 193 different satellite software builds, 75 gateway software builds, 222 Starlink software builds, and 57 WiFi software builds. 

This is the latency view of the interactive map.

For a month preceding March 7, SpaceX collected data every 15 seconds from millions of Starlink routers. In analyzing the data, they defined worst-case latency is the point at which 99% of times are shorter and peak hours as 6-9 PM local time. 

In the United States, they found that median latency was reduced by more than 30%, from 48.5ms to 33ms during peak usage hours, and worst-case peak hour latency had dropped by over 60%, from over 150ms to less than 65ms. Outside the United States median latency was reduced by up to 25% and worst-case latencies by up to 35%. The map shown here is interactive and shows availability and upload and download speeds in addition to latency.

SpaceX says it has “tuned our algorithms to prefer paths with lower latency, no matter how small the difference, and to remove any and all sources of unnecessary and non-physical latency." Dave Taht and his colleagues at  LibreQoS might disagree, but latency will improve over time regardless. 

Latency will improve as SpaceX launches more satellites with more capacity and inter-satellite laser links and the launch rate will increase when Starship becomes available. Adding ground stations will also improve latency.  (Note that the only African light-colored areas in the above latency map are within reach of the only ground stations on the continent).

I hope Musk achieves the 20 ms goal for Starlink. Doing so would not only benefit Starlink customers, it would call the attention of the FCC and terrestrial Internet service providers to the importance of latency as a performance and marketing metric.

Civilian Tech Mobilization in Ukraine

Rosie the Riveter, US World War II poster (source)

As was the case in the US during World War II, civilian volunteers are making important contributions to the Ukrainian war effort.

On February 8, 2022, the first truck load load of Starlink terminals arrived in Kyiv. A week later they were being used. By April 2022, there were 5,000 terminals in Ukraine, and 42,000 as of April 2023. (At this point, SpaceX and Ukraine have gone silent. Neither ChatGPT4, Gemini, Copilot, Perplexity, nor I could not find a current terminal count).

Whatever the number of terminals in the country, they require support. They were purchased, delivered, and set up. Users were trained and they require real-time access for troubleshooting and assistance. Broken terminals have to be repaired and some terminals have been modified. Civilian tech volunteers are doing much of this work. 

There are several Starlink support centers throughout Ukraine. For obvious reasons, they are secretive about their work, but one large one is Nebogray in Lviv. Neborgray has repaired 5,976 Starlink terminals and converted 516 for portable use mounted on vehicle roofs. In addition to the service centers, there are many individual craftsmen and small services throughout the country.

The work at Nebogry is performed by highly qualified volunteers. For example, Oleg Kutkov. is a senior engineer at Ubiquiti, and he devotes his spare time to Starlink research. He bought what may have been the first Starlink terminal in Ukraine on eBay before the war and does teardowns and research studies like this recent unboxing and review of the Version 4 Starlink terminal on his blog. Oleg is an active participant in the Starlink mailing list and the 15,700-member People's Starlink Facebook group.

The Facebook group was created by The People's Starlink project, which is involved in refurbishing, adapting, repairing, and providing technical support, as well as procuring and upgrading satellite communication terminals from SpaceX's Starlink for the needs of the Armed Forces of Ukraine and other defenders of Ukraine. 

With the help of many contributors, including Oleg, People's Starlink founder Vladimir Stepanets has written a 246-page Starlink Handbook for Military Users, which begins with a message from the author “Greetings defenders of Ukraine!”

Starlink Handbook for Military Users

The handbook is divided into eight, richly illustrated modules:

• What is Starlink?
• Starlink Terminals
• Powering Starlink terminals
• Expanding and collapsing Starlink terminals
• Terminal management and settings
• Safety of using Starlink terminals
• Diagnostics and problem-solving
• Starlink in network infrastructures

This is the second (and first public) edition of the handbook and it will continue evolving. It is currently available in Ukrainian, but Stepanets is discussing translations into several other languages and plans to publish it as a book.

Point of invincibility in Bucha, Ukraine

In response to Russian attacks on critical infrastructure, Ukraine has established thousands of Points of Invincibility, tent structures equipped with generators. The government is working to provide a Starlink terminal for each of them in addition to heat, water, lighting, and more.

The IT Army of Ukraine is an international, loosely connected organization of Ukrainian and foreign ethical hackers. They have created an online service that Ukrainian allies can use to generate denial-of-service attacks. Of course, one man's "ethical hacker" is another man's "terrorist," and Ukraine has petitioned The International Criminal Court to investigate Russian cyberattacks as war crimes. The International Committee of the Red Cross has published rules of engagement for civilian hackers involved in conflicts and the IT Army will make a best effort to follow the rules.

It was obvious from the early days of the war that two technologies -- Starlink and drones -- were going to play major roles. Model airplane hobbyists created an air reconnaissance unit within the army when fighting began in 2014 and Starlink enabled surveillance drones to relay target coordinates to artillery units. 

Today, non-technical people like Violetta Oliynyk, an artist and jeweler, are assembling drones in their spare time. She learned drone assembly by taking an online course from Prometheus, a nine-year-old education site with over 400 courses online. (The course was developed for the Victory Drones project). Social Drones UA is another volunteer drone assembly project. They vet then train and support potential assemblers with a how-to video and online support. 

Ukrainians are also assembling battery packs from batteries in discarded vapes, which is reminiscent of Americans saving and turning in excess cooking fat to be used in explosives during World War II.

Civilian volunteers and Ukrainian tech companies have pivoted to military innovation and production. Ukraine was technologically advanced before the war and has been forced to innovate and improvise. If Ukraine survives, the tech sector will thrive when peace comes. 

I've presented a few examples of civilian tech support for the Ukrainian resistance to the Russian invasion. There are many others, and if you are so inclined, the Internet makes it possible for you to contribute to them. Many project Websites have contribution links and you can also consult Reddit's list of vetted Ukrainen charities.

Update 3/8/2024

I believe Oleg is by Volodymyr Zelenskyy's ear.

Oleg Kutkov was among the "people embodying the spirit of Ukraine," chosen by Time Magazine as Person of the Year for 2022. Time wrote:

Ukraine first came back online when Elon Musk activated his low-­altitude Starlink satellite internet, as he would later do in Iran. The net was crucial to Ukrainian forces, who were issued the compact, portable Starlink antennas. But in Kyiv, self-­described “tech and space nerd” Oleg Kutkov reconstructed a Starlink dish from eBay, and after contacting SpaceX support, caught a signal. “I was the first civilian user of Starlink here in Ukraine,” says Kutkov, 34, who began a Face­book group that has grown to 8,700 people. “They read about me in the news, and they were all worrying about connectivity because the internet is really important here to get all the news, to get notifications and so on.”

 

Amazon Project Kuiper vs SpaceX Starlink

Amazon's Project Kuiper is far behind Starlink and is under time pressure, but Amazon has several things going for it.

In 2019, I wrote that Amazon would be a formidable satellite-ISP competitor. I still think so, but I didn’t expect it would be over four years until they launched the first test satellites. In the meantime, SpaceX has put over 5,000 satellites in orbit and has over two million Starlink customers.

Amazon has permission to launch 3,236 satellites. They must manufacture and launch at least half of them by July 2026 and the remainder by July 2029. Can they do it? After many delays, they have finally launched two test satellites, confirming that inter-satellite laser links (ISLLs) worked at 100 Gbps while sending traffic “in both directions from the internet over an AWS fiber-optic connection to our ground gateway station, up to our satellites, and then down to a customer terminal at our test location.” All Starlink satellites launched since September 2021 will have ISLLs, so by the time Kuiper is complete, July 2026, all or nearly all Starlink sats will have them and they will have a much larger constellation.

Amazon has not launched any production satellites and they will have to hurry to meet the 2026 and 2029 deadlines. They have signed contracts for 83 launches over a five-year period, which they say will provide capacity for “the majority” of the constellation. SpaceX was conspicuously not one of the vendors, and a shareholder lawsuit pointed out that Amazon had not considered SpaceX as a provider and nearly 45% of the overall value is for launches and engines from Blue Origin, a rocket company founded by Jeff Bezos. Subsequently, Kuiper signed a 3-launch contract with SpaceX.

(Note that Blue Origin has not yet launched their forthcoming New Glenn rocket, which was initially scheduled to fly in 2020. The New Glenn will have greater capacity than SpaceX’s current Falcon rocket but significantly less than their forthcoming Starship).

Amazon's Project Kuiper is far behind Starlink and is under time pressure, but Amazon has several things going for it:

• In his first letter to stockholders, Jeff Bezos stressed that Amazon was an infrastructure company and that has been borne out by subsequent investments in facilities and services. Amazon will bundle Kuiper access with data storage and cloud computing services.
• Kuiper will offer service-level agreements to non-consumer customers.
• Amazon will be Kuiper’s largest customer. With over 1.5 million employees staffing offices, warehouses, and other facilities, Amazon will use the Kuiper constellation internally as will their fleets of delivery trucks, planes, shipping containers, and perhaps delivery drones someday.
• Amazon is already in the space business with its satellite ground station service.
• The US, Taiwanese, and other governments and militaries will see Amazon as a more reliable supplier of critical infrastructure than Starlink given Elon Musk’s political activism and Tesla’s dependence on China.
• Some potential customers may not approve of Elon Musk’s political involvement.
• Re-usability gives Starlink a large launch-cost advantage, but if Elon Musk can afford Twitter, Jeff Bezos can afford Kuiper.
• Amazon's New Glenn rocket is designed to be reusable and eight Chinese private and state-owned entities are developing reusable rockets.
• Kuiper will be launching state-of-the-art satellites and selling state-of-the-art terminals.
• Satellite antennas are expensive, and Amazon has experience designing and manufacturing consumer devices like the Echo and Kindle. 
• Amazon has announced three Kuiper antennas.
• A user terminal is more than just an antenna and Dave Täht, Chief Science Officer at LibreQoS, has been calling attention to Starlink’s latency problem for years. It now seems Elon Musk is ready to listen. Amazon should talk with Täht.
• Amazon is already talking with enterprises, governments, schools, hospitals, and mobile operators. They have contracts with Verizon, Vodafone, and NTT and licenses to operate in more than 15 countries, including Brazil, Canada, France, Mexico, and the US.

I could go on but you get the idea -- I think Kuiper will survive despite a rocky start and will eventually offer Starlink healthy competition.

Update Feb 9, 2024

I listed Jeff Bezos’s wealth and the expected use of the Kuiper constellation among the causes of my optimism. A report on the capital expenditures by the three major cloud service companies puts Amazon’s commitment to invest $10 billion over several years in Project Kuiper in context. In 2023 alone “Amazon’s (relative) CAPEX austerity continues, as the company spent a measly $53.7 billion, a decline of 20%" and it has invested around $380 billion since 2000.

It's the latency, FCC

Section 706 of the Telecommunications Act of 1996 orders the FCC to “encourage the deployment on a reasonable and timely basis of advanced telecommunications capability to all Americans.” On October 25, The FCC issued a notice of inquiry (NOI) into how well we are doing and invited comments.

The NOI points out that COVID and the concomitant increase in the use of interactive applications has “made it clear that broadband is no longer a luxury but a necessity that will only become more important with time" and proposes “an increase from the existing fixed broadband speed benchmark of 25 Mbps download and 3 Mbps upload (25/3 Mbps) to 100/20 Mbps." They also seek comment on a long-term speed goal of 1,000/500 Mbps.

The focus is clearly on speed. They mention latency on page 12 and jitter and packet loss on page 15, but the FCC made no metrics recommendations on those metrics and requested comments.

Dave Taht, Chief Science Officer of LibreQOS and an embedded Linux developer and consultant since 1998, drafted a comment arguing that the FCC should “balance its near-term efforts on achieving Internet resilience and minimizing latency, instead of only increasing speed.” Taht invited experts to suggest edits to and sign his draft and the submitted comment has 63 signatures, many of which would be familiar to CircleID readers. 

Taht says “Calls for further bandwidth increases are analogous to calling for cars to have top speeds of 100, 500, or 1000 miles per hour” and the "only way to improve responsiveness is to robustly and reliably reduce the latency, and especially the 'latency under load'.” He points out that low latency, not speed, is critical for today’s interactive applications and high latency reduces aggregate network efficiency and increases variability in the user experience. 

Much Internet latency is caused by bufferbloat – packets working their way through queues that build up in routers and other network equipment. Taht has spent years developing tools to measure latency and reduce bufferbloat and he documents his work and that of others in his 27-page NOI comment.

How much speed does one need? 

That depends on the applications you use, which is a moving target. My first home Internet terminal was a 10-character per-second (CPS) ASR-33 Teletype with an acoustic coupler. I used it for email, FTP, Telnet, and network news and I was able to collaborate with distant colleagues. I loved it and 100 CPS would not have made a big difference because 10 CPS was about as fast as I could read and faster than I could type. My first connected computer used a 300-bps modem, and modem speeds increased to 56 Kbps driven by applications like Web, and voice over IP.

Today, Poa Internet in Kenya offers uncapped 4 Mbps service which is sufficient for downloading software, articles, books, movies, etc., shopping, making voice-over-IP calls, listening to podcasts, reading newspapers, etc., and, importantly, creating content and inventing and developing applications and services that are relevant to Africa.  

Netflix speed recommendations (source)

Streaming video is the most speed-intensive application I use today and Netflix recommends 15 Mbps for viewing UHD 4k movies. Poa Internet customers might be able to view 720p video.

Spectrum, my ISP today,  offers three plans – up to 300, 500, and 1,000 Mb/s.  I have a 300 Mbps cable connection which is more than I need. M-Lab's Internet performance test service, which measures speed and latency unloaded and while simulating background activity reported that my latency increased from 16 to 53 ms when downloading was active and 41 ms when uploading was active. Speeds were 355.3 Mbps download and 11.2 Mbps upload. Considering Netflix’s recommendation, it is unsurprising that streaming two movies on my home WiFi network while running the M-Lab test did not make much difference.

As long as I only watch one movie at a time, I suspect I would not notice much difference if Spectrum only provided me with the current FCC benchmark of 25/3 Mbps. This raises the question of opportunity cost. How much capital and operating cost could Spectrum have saved if they had only provisioned, say, a choice between 25/3 and 50/6 Mbps? Would the savings be sufficient to fill in white spaces in their national broadband map? 

Spectrum dismisses latency, writing:

Latency is typically measured in milliseconds, and generally has no significant impact on typical everyday internet usage. As latency varies based on any number of factors, most importantly the distance between a customer's internet connected device and the ultimate internet destination (as well as the number, variety, and quality of networks your packets cross), it is not possible to provide customers with a single figure that will define latency as part of a user experience.

Page load time as a function of latency (source)

If we could come up with a "single figure" to define and measure latency, ISPs would have an incentive to improve it, and the FCC could adopt benchmarks. While a single figure may be impossible, could tests isolate the latency in an ISP network and the customer premises equipment (CPE) they supply? Could we use imperfect surrogates for latency like page-load times? Could we benchmark components like the CPE an ISP provides?

While the FCC and ISP marketing are focused on speed today, attention to latency and its measurement is growing within the technical community. To learn more and get involved, check Dave's Bufferbloat.net site and LibreQOS and watch Dave's talk here. You can also give the FCC feedback by commenting on Proceeding 22-270 on the FCC Express Comments Page.

Update 1/18/2024

Elon Musk summarized SpaceX's 2023 accomplishments in a recent talk at Starbase in Texas, He covered many topics including Starlink. He stated that their biggest single technical goal for the year was to get mean latency under 20 ms. (He estimated that 10 ms was the theoretical minimum given the speed of light). Doing so will require a combination of steps including launching satellites with inter-satellite laser links, adding ground stations, and heeding the advice Dave Taht has been offering for years.

Update 7/15/2014

SpaceX Starlink has begun delivering on Elon Musks commitment to reduce latency and they are letting users know by including latency values and distribution in the Starlink app. (The 100 ms latency spike at the end of the distribution must reflect handoffs between satellites).

All ISPs should report and advertise latency as well as throughput, which is less important than latency in many applications.