History is written and revised by the winners -- can the Internet Archive change that?

Kremvax during the Soviet coup attempt

I was naively optimistic in the early days of the Internet, assuming that it would enhance democracy while providing "big data" for historians. My first taste of that came during the Soviet coup attempt of 1991 when I worked with colleagues to create an archive of the network traffic in, out and within the Soviet Union. That traffic flowed through a computer called "Kremvax," operated by RELCOM, a Russian software company.

The content of that archive was not generated by the government or the establishment media -- it was citizen journalism, the collective work of independent observers and participants stored on a server at a university. What could go wrong with that?

Mumbai terrorist attack

The advent of the Web and Wikipedia fed my optimism. For example, when terrorists attacked various locations in Mumbai, India in 2008, citizen journalists inside and outside the hotels that were under attack began posting accounts. The Wikipedia topic began with two sentences:

The 28 November 2008 Mumbai terrorist attacks were a series of attacks by terrorists in Mumbai, India. 25 are injured and 2 killed.

In less than 22 hours, 242 people had edited the page 942 times expanding it to 4,780 words organized into six major headings with five subheadings. (Today it is over 130,000 bytes, revisions continue and it is still viewed over 2,000 times per month). What could go wrong with that?

The Arab Spring

The 2011 Arab Spring was also seen as a demonstration of the power of the Internet as a democratic tool and repository of history. What could go wrong with that?

What went wrong

The problem is that the Internet turned out to be a tool of governments and terrorists as well as citizens. Furthermore, historical archives can disappear or, worse yet, be changed to reflect the view of the "winner."

Our Soviet Coup archive was set up on a server at the State University of New York, Oswego, by professor Dave Bozack. What will happen to it when he retires?

If someone tried to delete or significantly alter the Wikipedia page on the Mumbai attack, they might be thwarted by one of the volunteers who has signed up to be "page watchers" -- people who are notified whenever the page they are watching is edited. We saw a reassuring demonstration of the rapid correction of vandalism in a podcast by Jon Udell. That was cool, but does it scale? Volunteers burn out. The page on the Mumbai attacks has 358 page watchers, but only 32 have visited the page after recent edits.

Even if a Wikipedia page remains intact, links to references and supporting material will eventually break -- "link rot." If our Soviet Coup archive disappears after Dave's retirement, all the links to it will break.

By the time of the Arab Spring, we were well aware of our earlier naivete -- the Internet was already being used for terrorism and government cyberwar and the dream of providing raw data for future historians and political scientists was fading.

The Internet Archive

Soviet coup archive from Internet Archive

I was slow to understand the fragility of the Internet, but others saw it early -- most importantly, Brewster Kahle, who, in 1996, established the Internet Archive to cache Web pages and preserve them against deletion or modification. They have been at it for 20 years now and have a massive online repository of books, music, software, educational material, and, of course, Web sites, including our Soviet Coup archive. As shown here, it has been archived 50 times since October 3, 2002 and it will be online long after Dave retires -- as long as the Internet Archive is online.

Khale understands that saving static Web sites like the Soviet Coup archive only captures part of what is happening online today. Since the late 1990s, we have been able to add programs to Web sites, turning them into interactive services. As such, he has recently begun archiving virtual machine versions of interactive government services and databases.

Khale is understandably concerned by the election of Donald Trump, who has demonstrated a keen ability to exploit the Internet and a disregard for truth. As such, he is raising money to create a backup copy of the Interent Archive in Canada and working to archive US Government Web sites and services.

The Internet is inconceivably large and growing exponentially. There is no way the Internet Archive can capture all of it, but it is the leading Internet-preservation organization today. Khale and his staff will continue their work and will inspire and collaborate with other relatively specialized efforts like that of climate scientists who are working to preserve government climate-science research results, data and services.

For more on the Internet Archive check out the following PBS News Hour segment (9m 12s):

You can read the transcript here.

I'd also recommend listening to this short (5m 14s) podcast interview of Brewster Kahle. He describes the End of Term project -- a collaborative effort to record US government (.gov and .mil) Web sites and services when a new administration takes over. He describes deletions and modifications from 2008 and 2012 and feels a special urgency today for obvious reasons.

You can read a transcript of the interview here.

-----
Update 1/6/2017

The Internet Archive has launched the Trump Archive with 700+ televised speeches, interviews, debates, and other news broadcasts. Mention by a fact-checking site was the "signal" used for inclusion of a video and links to the fact-check document are included in a companion spreadsheet. I hope they use speech recognition to produce searchable transcripts as well.

Too bad we did not have Trump and Clinton archives during the campaign -- I hope we will have similar, timely archives in the future. One can even imagine similar archives for state and local campaigns if a crowd-sourcing system were developed.

-----
Update 1/7/2017

There is an annotated PowerPoint presentation on citizen journalism here. I use it in teaching an Internet literacy class and there is a note on my PowerPoint presentation style here.

-----
Update 2/5/2018

Speaker of the House Paul Ryan posted a clueless tweet bragging that the Republican tax-cut bill had saved a secretary at a public school $1.50 per week -- enough to cover her $60 Costco membership for the year. The tweet was soon deleted, but the Internet Archive had cached a Czech retweet:

Note that embarrassing news travels fast -- the day after the initial tweet, the Czech tweet had 954 retweets and 1,466 likes.

Note also that $1.50 per week will cover the secretary's $60 Costco membership with $18 to spare -- she can use that to cover the tax on her extra income.

Ryan is a more selective tweeter than Trump. For example, he did not Tweet anything when his fund-raising committee received a $500,000 campaign contribution from the Koch brothers shortly after the tax bill passed in the House.

-----
Update 7/17/2018

Yesterday, the Trump Administration shut down the National Guideline Clearinghouse, a database of medical guidelines that for nearly 20 years has been a critical resource for doctors, researchers and others in the medical community.

The shutdown was necessitated by budget cuts -- I guess they had to make a tough choice between paying for the database or Tom Price's charter flights.

The Internet Archive Wayback Machine archived the database on July 15, but that is a static copy that will not be updated.

The administration has also been removing the term "climate change" from Web sites and there have been several efforts to back up client-science data, for example this and this.

Internet censorship is a cat and mouse game, and there are many mice

When the Turkish government decided to "obliterate" Twitter, they got a court order allowing them to delete the IP address corresponding to twitter.com from the domain name servers (DNS) used by Turkish ISPs. Twitter.com became an unknown host name.

But the DNS your ISP uses is not the only one on the Internet -- there are many. If you knew the IP address of an alternative DNS, you could continue accessing Twitter.

A few years ago, Google decided to run a DNS as a public service in order to improve Internet security and speed. Word quickly got out that one could change the DNS address supplied by their ISP to Google's: 8.8.8.8 or 8.8.4.4:

The government ploy may have worked a few years ago, but people are now more computer savvy and those who wished to continue using Twitter quickly switched to Google's public DNS.

The government countered by blocking traffic to/from Twitter's IP address, but Turks could circumvent that by using proxy servers, which hide the fact that they are communicating with Twitter.com:

As the government learns the IP addresses of the various proxy servers around the Internet, they can block them, but there are many available and new ones come online when old ones are discovered.

Twitter also posted a tweet showing how to post using SMS rather than the Internet:

This cat and mouse game has been going on since the time of the 1991 Soviet coup attempt, when the network operators abandoned their office for the anonymity of their laptops at home:

At that time, there were only a handful of citizen journalists; today there are millions.
-----
Update 3/24/2014

More on the ability of the Net to route around censorship.
-----
Update 3/26/2014

Turkey will hold municipal and regional elections on March 30, and Reporters without Borders just published a post on the “Toxic climate for media a week ahead of elections.” The post links to several other posts from this month and last. While focused primarily on television and print, the posts also address electronic media. Check, for example “Blocking of Twitter -- worthy of the most repressive regimes,” which offers a tutorial on four ways to remain in contact with Twitter.
-----
Update 3/30/2014

Turkey blocked YouTube after a recording of a top-secret military meeting was posted, but the recording has been copied to Dropbox and circulated online -- you can’t put the toothpaste back in the tube once its out. Their attempt to block Twitter earlier in the week resulted in silencing regime loyalists, but not critics. Twitter now has posts mocking the prime minister’s high voice and the failed attempt to block Twitter.
-----
Update 3/30/2014

The cat fights back -- the Turkish government has blocked access to Google's public DNS -- Turks using 8.8.8.8 or 8.8.4.4 will now be using a fake domain name server.
-----
Update 4/17/2014

Turkey is now blurring (pixelating) offensive tweets. Is that for real? Do they automate the process? Is Twitter cooperating? Is April Fools Day on the 17th in Turkey?

____
Update 10/3/2014

The Turkish Constitutional Court has ruled that Turkey's telecommunication authority does not have the right to close websites on grounds of protecting national security. The court also ruled against their authority to store internet data for up to two years. (They had been granted these powers when parliament passed amendments to the Internet Law on September 10).

Before the Twitter revolutions, there was the Usenet revolution

Polina Antonova and Eugene Peskin operating Kremvax, the link between
Relcom and the West

The first example of relatively large-scale citizen journalism online was during the August 1991 coup attempt by eight high-level Soviet officials. There were two days of confrontation between demonstrators and troops, during which all Russian media except Usenet news groups were shut down by the authorities. Usenet, a precursor of today's Internet discussion forums, carried traffic into, out of and within Russia (70 cities) during the days of the coup attempt.

One of the quotes from that time foreshadows subsequent attempts to shut the Internet down. Polina Antonova of RELCOM, the Unix distributor that operated the network wrote:

They try to close all mass media, they stopped CNN an hour ago, and Soviet TV transmits opera and old movies. But, thank Heaven, they don't consider RELCOM mass media or they simply forgot about it. Now we transmit information enough to put us in prison for the rest of our life.

After two tense days, the coup failed and president Mikhail Gorbachev was restored to power. For more on the role of the network during the coup attempt, see:

• The role of the network during the coup attempt (Internet Society)
• Description of RELCOM (CACM)
• Archive of all network traffic and other material (David Bozak, SUNY)
• List of quotes from the network traffic (ACM Risks Forum)
• More quotes (EFF)

The following video clip (3:13) was part of a documentary on the Internet. (It includes a short interview of me at the time -- no white hair :-).



Usenet played a relatively significant role during the Soviet coup attempt, but there was also some Usenet traffic during the 1989 Tienanmen Square protests. Everyone is familiar with the heroic image of the unidentified man stopping a column of tanks on June 6 1989 -- the day after troops had opened fire on protesters.

But, there was some Usenet traffic as well. Olivier Crepin-Leblond has saved a description of the situation that was posted by a Sun Microsystems employee on May 23. It describes the optimistic time before the storm, beginning with:

The situation here seems getting better and better. All army members are blocked outside Beijing city. The people’s life in the city looks as normal as usual.

Here is a movie clip of the tank man at Tiananmen Square:



This clip was not on the Internet at the time, but it is today -- providing a dramatic view of a heroic act and illustrating the futility of China's attempt to erase the memory of the event.

-----
Update 8/20/2016

Slate has published a relatively long post entitled "An act of courage on the Soviet Internet" on RELCOM's role during the Soviet Coup attempt of 1991. The article describes my involvement at the time, but fails to link to the archive of the network traffic and other material, which is available to historians and others interested in the event.

I believe this was the first example of active citizen journalism involving the Internet. The Internet has given us a means of documenting historical events in detail -- "big data" for historians. We need a mechanism for discovering and preserving such archives.

-----
Update 1/12/2017

Mark Graham, who relayed news over the Sovam Teleport link during the Soviet coup attempt, just sent me an article on their activity at the time. Unfortunately, the data they transmitted and received has been lost. Our archive is still online at the State University of New York site mentioned above and, even better, it is online at the Internet Archive.

-----
Update 7/7/2018

For an excellent article on the role of RELCOM in the Soviet Coup attempt, click here.