History is written and revised by the winners -- can the Internet Archive change that?

Kremvax during the Soviet coup attempt

I was naively optimistic in the early days of the Internet, assuming that it would enhance democracy while providing "big data" for historians. My first taste of that came during the Soviet coup attempt of 1991 when I worked with colleagues to create an archive of the network traffic in, out and within the Soviet Union. That traffic flowed through a computer called "Kremvax," operated by RELCOM, a Russian software company.

The content of that archive was not generated by the government or the establishment media -- it was citizen journalism, the collective work of independent observers and participants stored on a server at a university. What could go wrong with that?

Mumbai terrorist attack

The advent of the Web and Wikipedia fed my optimism. For example, when terrorists attacked various locations in Mumbai, India in 2008, citizen journalists inside and outside the hotels that were under attack began posting accounts. The Wikipedia topic began with two sentences:

The 28 November 2008 Mumbai terrorist attacks were a series of attacks by terrorists in Mumbai, India. 25 are injured and 2 killed.

In less than 22 hours, 242 people had edited the page 942 times expanding it to 4,780 words organized into six major headings with five subheadings. (Today it is over 130,000 bytes, revisions continue and it is still viewed over 2,000 times per month). What could go wrong with that?

The Arab Spring

The 2011 Arab Spring was also seen as a demonstration of the power of the Internet as a democratic tool and repository of history. What could go wrong with that?

What went wrong

The problem is that the Internet turned out to be a tool of governments and terrorists as well as citizens. Furthermore, historical archives can disappear or, worse yet, be changed to reflect the view of the "winner."

Our Soviet Coup archive was set up on a server at the State University of New York, Oswego, by professor Dave Bozack. What will happen to it when he retires?

If someone tried to delete or significantly alter the Wikipedia page on the Mumbai attack, they might be thwarted by one of the volunteers who has signed up to be "page watchers" -- people who are notified whenever the page they are watching is edited. We saw a reassuring demonstration of the rapid correction of vandalism in a podcast by Jon Udell. That was cool, but does it scale? Volunteers burn out. The page on the Mumbai attacks has 358 page watchers, but only 32 have visited the page after recent edits.

Even if a Wikipedia page remains intact, links to references and supporting material will eventually break -- "link rot." If our Soviet Coup archive disappears after Dave's retirement, all the links to it will break.

By the time of the Arab Spring, we were well aware of our earlier naivete -- the Internet was already being used for terrorism and government cyberwar and the dream of providing raw data for future historians and political scientists was fading.

The Internet Archive

Soviet coup archive from Internet Archive

I was slow to understand the fragility of the Internet, but others saw it early -- most importantly, Brewster Kahle, who, in 1996, established the Internet Archive to cache Web pages and preserve them against deletion or modification. They have been at it for 20 years now and have a massive online repository of books, music, software, educational material, and, of course, Web sites, including our Soviet Coup archive. As shown here, it has been archived 50 times since October 3, 2002 and it will be online long after Dave retires -- as long as the Internet Archive is online.

Khale understands that saving static Web sites like the Soviet Coup archive only captures part of what is happening online today. Since the late 1990s, we have been able to add programs to Web sites, turning them into interactive services. As such, he has recently begun archiving virtual machine versions of interactive government services and databases.

Khale is understandably concerned by the election of Donald Trump, who has demonstrated a keen ability to exploit the Internet and a disregard for truth. As such, he is raising money to create a backup copy of the Interent Archive in Canada and working to archive US Government Web sites and services.

The Internet is inconceivably large and growing exponentially. There is no way the Internet Archive can capture all of it, but it is the leading Internet-preservation organization today. Khale and his staff will continue their work and will inspire and collaborate with other relatively specialized efforts like that of climate scientists who are working to preserve government climate-science research results, data and services.

For more on the Internet Archive check out the following PBS News Hour segment (9m 12s):

You can read the transcript here.

I'd also recommend listening to this short (5m 14s) podcast interview of Brewster Kahle. He describes the End of Term project -- a collaborative effort to record US government (.gov and .mil) Web sites and services when a new administration takes over. He describes deletions and modifications from 2008 and 2012 and feels a special urgency today for obvious reasons.

You can read a transcript of the interview here.

-----
Update 1/6/2017

The Internet Archive has launched the Trump Archive with 700+ televised speeches, interviews, debates, and other news broadcasts. Mention by a fact-checking site was the "signal" used for inclusion of a video and links to the fact-check document are included in a companion spreadsheet. I hope they use speech recognition to produce searchable transcripts as well.

Too bad we did not have Trump and Clinton archives during the campaign -- I hope we will have similar, timely archives in the future. One can even imagine similar archives for state and local campaigns if a crowd-sourcing system were developed.

-----
Update 1/7/2017

There is an annotated PowerPoint presentation on citizen journalism here. I use it in teaching an Internet literacy class and there is a note on my PowerPoint presentation style here.

-----
Update 2/5/2018

Speaker of the House Paul Ryan posted a clueless tweet bragging that the Republican tax-cut bill had saved a secretary at a public school $1.50 per week -- enough to cover her $60 Costco membership for the year. The tweet was soon deleted, but the Internet Archive had cached a Czech retweet:

Note that embarrassing news travels fast -- the day after the initial tweet, the Czech tweet had 954 retweets and 1,466 likes.

Note also that $1.50 per week will cover the secretary's $60 Costco membership with $18 to spare -- she can use that to cover the tax on her extra income.

Ryan is a more selective tweeter than Trump. For example, he did not Tweet anything when his fund-raising committee received a $500,000 campaign contribution from the Koch brothers shortly after the tax bill passed in the House.

-----
Update 7/17/2018

Yesterday, the Trump Administration shut down the National Guideline Clearinghouse, a database of medical guidelines that for nearly 20 years has been a critical resource for doctors, researchers and others in the medical community.

The shutdown was necessitated by budget cuts -- I guess they had to make a tough choice between paying for the database or Tom Price's charter flights.

The Internet Archive Wayback Machine archived the database on July 15, but that is a static copy that will not be updated.

The administration has also been removing the term "climate change" from Web sites and there have been several efforts to back up client-science data, for example this and this.

Backing up climate-science data

It is nearly inconceivable that Trump would order the deletion of climate-science data -- a modern book burning -- but one can imagine large budget cuts for climate-science research, making it impossible to maintain and update this sort of public data.

Climate scientists have kicked off at least two projects to create backup copies of their research and data.

One is Climate Mirror, which is part of an ad-hoc project to mirror public climate datasets before the Trump Administration takes office -- to make sure these datasets remain freely and broadly accessible.

Another is a hackathon that will be hosted on December 17th at the University of Toronto in collaboration with the Internet Archive End of Term project, which seeks to archive the federal online pages and data that are in danger of disappearing during the Trump administration. (Note that they have done the same for earlier administrations).

For example, NASA recently released data showing how temperature and rainfall patterns worldwide may change through the year 2100 because of growing concentrations of greenhouse gases in Earth’s atmosphere.

The post announcing the dataset states:

The dataset, which is available to the public, shows projected changes worldwide on a regional level in response to different scenarios of increasing carbon dioxide simulated by 21 climate models. The high-resolution data, which can be viewed on a daily timescale at the scale of individual cities and towns, will help scientists and planners conduct climate risk assessments to better understand local and global effects of hazards, such as severe drought, floods, heat waves and losses in agriculture productivity.

"NASA is in the business of taking what we’ve learned about our planet from space and creating new products that help us all safeguard our future,” said Ellen Stofan, NASA chief scientist. “With this new global dataset, people around the world have a valuable new tool to use in planning how to cope with a warming planet.

The climate-science community is obviously alarmed by Donald Trump's appointments of Ryan Zinke, who characterizes climate change as “unsettled science," as Secretary of the Interior, Rick Perry, who once could not recall the name of the department, but remembered that he did want to eliminate it, as Secretary of Energy and Scott Pruitt, who consistently opposes regulation, to head the Environmental Protection Agency.

These men are all supporters of and supported by the oil industry.

The Trump transition team also requested a list of the names of Energy Department people (contractors and employees) who have worked on climate science and the professional society memberships of lab workers.

It is nearly inconceivable that Trump would order the deletion of climate-science data -- a modern book burning -- but one can imagine large budget cuts for climate-science research, making it impossible to maintain and update this sort of public data.

-----
Update 12/29/2016

Check out this excellent, short (5:14) interview of Internet Archive founder Brewster Kahle. The interview begins with climate scientist Eric Holthaus talking about the effort to archive climate research, then Khale goes on to say more about how and why they archive government (.gov and .mil) Web sites when a new administration takes over.

He said 83% of the .pdf files on government sites were deleted between 2008 and 2012. In addition to Web pages, they will be archiving virtual machine versions of interactive government services and databases. (As noted above, those are vulnerable to defunding).

When asked for an example of the value of the archive, Khale mentioned the press release announcing George Bush's ironically famous "Mission Accomplished" speech on the deck of an aircraft carrier. As shown below, the headline reads "President bush announces combat operations in Iraq have ended" and the first sentence qualifies the headline by saying "major" combat operations have ended. Khale said that a couple of weeks later "major" was added to the title and a couple months later, the press release was deleted.

Excerpt from press release on "Mission Accomplished" speech

The Internet is potential providing raw data for historians -- it should be complete and accurate.

If you would like to see a video of the entire speech, the Internet Archive has preserved that as well:




-----
Update 12/30/2016

The following is a transcript of Bob Garfield, co-host of the podcast On The Media, interviewing Brewster Khale, founder of the Internet archive and a partner in the End-of-term Project with a lead-in question for on climate-science research Eric Holthaus of Slate Magazine.

Bob: Meanwhile a small army of volunteer archivists, scientists and advocates have been working to save the government climate change research that already exists

Eric: at NASA and NOA that takes the temperature of the planet from weather stations from satellites from ocean buoys.

Bob: Meteorologist Eric Holthaus spoke to NPR about his effort to save government climate data.

Eric: Sometimes these data sets are only stored in United States government servers so there hasn't really been an effort to catalog those in other countries because we haven't thought it was necessary before

Bob: The Internet Archive on the other hand has given a lot of thought to what gets lost in presidential transitions. Every week the archive tapes three hundred million Web pages and every four years it enlists a bunch of volunteers to make copies of government Web sites as a hedge against what the next administration may choose to delete. It's called The End-of-term web archive and for some reason this year the organizers are getting a lot more offers of help. Brewster Kahle founder of the Internet Archive says that this year his team also is backing up its data to Canada

Brewster: When the election went the way that it did, it was a bit of a surprise, so we looked through the television archive at what President-elect Trump said about freedom of the press and about the Internet and what we found was shocking. He wanted to close up parts of the Internet that there was mocking of freedom of the press. This was kind of a wake-up call and we said let's make sure we have a copy in some other location.

Bob: What are your priorities? How does it work?

Brewster: So the Internet Archive works with the Library of Congress, University of North Texas -- now a growing list of groups to try to do as best we can to record the information that's available on the Web sites and now the web services that have been made available on .gov and .mil Web sites. We found in 2008, 83 percent of the PDFs that were available back then are no longer available even by 2012. So with an 83 percent loss rate when the Obama administration came on board we're likely to see something like it maybe even more with the Trump administration.

So we're coordinating activities to go and archive web pages and we're reaching out to federal webmasters to go and see if we can keep whole services up and running. Can we take virtual machine versions of the databases that they're running and be able to run them in snapshot form so that we can keep these services going as they were in 2016 in the future?

Bob:Give me some examples of when the federal web archive has come in handy. Was there something that you and disappeared that you were super glad to have archived?

Brewster: Oh the anecdotes go on and on. Example -- there is a press release from the White House during the George W. Bush administration when he stood on an aircraft carrier and declared “mission accomplished.” And the headline of that press release was combat operations in Iraq had ceased but a couple of weeks later they changed the headline and said major combat operations had ceased with no notice that it had changed. The only reason why we know is because we had archived both versions. And then a couple of months later the press release went away completely from the web. You know what is more Orwellian is it changing a press release that's in the past or is it disappearing completely?

Bob: What are you most worried is going to disappear in a Trump administration?

Brewster: Frankly we have no idea. This upcoming administration is very aware of the power of the Internet and how it can be manipulated -- how you can go and push things out in the middle of the night and use the journalist system in ways that are really pretty blatant. So let's at least keep a record of it.

Bob: We have just experienced the interference in a political campaign by outsiders. Is this archive secure -- I mean really secure against hacking, against intrusion?

Brewster: The history of libraries is a history of loss. Libraries are burned. That's what happened in the Library of Alexandria. It'll be what happens to us -- just don't know when. So let's design for it. Let's go and make copies in other places. Let's make sure people want universal access to all knowledge, that they want education based on facts. Let's go and make sure that there is an environment that supports libraries. That's the only way that in the long term we're going to survive. And the copies that are maybe now unique at the Internet Archive will survive based on all sorts of changes whether it's earthquakes or institutional failure or law changes.

Bob: Brewster as always many thanks.

Brewster: Thank you very much.

Bob: Mr. Khale is the founder of the Internet Archive and a partner on the End-of-term Project.

Khale's interview was part of longer podcast episode called Hurry Up. They discussed other steps President Obama could take during the last weeks of his term. The suggestions included disclosing information on contributions by government contractors, surveillance and the drone program, closing Guantanamo and clemency. The episode ends with a discussion of the nature of time by science writer James Gleick.

Finally, I created the interview transcript using a nifty service called PopUpArchive. You simply upload a sound file and wait for the text version to be posted ready for download. It takes a little proofreading and editing, but it is a lot faster than manual transcription and as this Microsoft Research report shows, we can look forward to more accurate speech recognition in the future.