Showing posts with label historians. Show all posts
Showing posts with label historians. Show all posts

Thursday, December 15, 2016

Backing up climate-science data

It is nearly inconceivable that Trump would order the deletion of climate-science data -- a modern book burning -- but one can imagine large budget cuts for climate-science research, making it impossible to maintain and update this sort of public data.

Climate scientists have kicked off at least two projects to create backup copies of their research and data.

One is Climate Mirror, which is part of an ad-hoc project to mirror public climate datasets before the Trump Administration takes office -- to make sure these datasets remain freely and broadly accessible.

Another is a hackathon that will be hosted on December 17th at the University of Toronto in collaboration with the Internet Archive End of Term project, which seeks to archive the federal online pages and data that are in danger of disappearing during the Trump administration. (Note that they have done the same for earlier administrations).

For example, NASA recently released data showing how temperature and rainfall patterns worldwide may change through the year 2100 because of growing concentrations of greenhouse gases in Earth’s atmosphere.


The post announcing the dataset states:
The dataset, which is available to the public, shows projected changes worldwide on a regional level in response to different scenarios of increasing carbon dioxide simulated by 21 climate models. The high-resolution data, which can be viewed on a daily timescale at the scale of individual cities and towns, will help scientists and planners conduct climate risk assessments to better understand local and global effects of hazards, such as severe drought, floods, heat waves and losses in agriculture productivity.

"NASA is in the business of taking what we’ve learned about our planet from space and creating new products that help us all safeguard our future,” said Ellen Stofan, NASA chief scientist. “With this new global dataset, people around the world have a valuable new tool to use in planning how to cope with a warming planet.
The climate-science community is obviously alarmed by Donald Trump's appointments of Ryan Zinke, who characterizes climate change as “unsettled science," as Secretary of the Interior, Rick Perry, who once could not recall the name of the department, but remembered that he did want to eliminate it, as Secretary of Energy and Scott Pruitt, who consistently opposes regulation, to head the Environmental Protection Agency.

These men are all supporters of and supported by the oil industry.

The Trump transition team also requested a list of the names of Energy Department people (contractors and employees) who have worked on climate science and the professional society memberships of lab workers.

It is nearly inconceivable that Trump would order the deletion of climate-science data -- a modern book burning -- but one can imagine large budget cuts for climate-science research, making it impossible to maintain and update this sort of public data.

-----
Update 12/29/2016

Check out this excellent, short (5:14) interview of Internet Archive founder Brewster Kahle. The interview begins with climate scientist Eric Holthaus talking about the effort to archive climate research, then Khale goes on to say more about how and why they archive government (.gov and .mil) Web sites when a new administration takes over.

He said 83% of the .pdf files on government sites were deleted between 2008 and 2012. In addition to Web pages, they will be archiving virtual machine versions of interactive government services and databases. (As noted above, those are vulnerable to defunding).

When asked for an example of the value of the archive, Khale mentioned the press release announcing George Bush's ironically famous "Mission Accomplished" speech on the deck of an aircraft carrier. As shown below, the headline reads "President bush announces combat operations in Iraq have ended" and the first sentence qualifies the headline by saying "major" combat operations have ended. Khale said that a couple of weeks later "major" was added to the title and a couple months later, the press release was deleted.


Excerpt from press release on "Mission Accomplished" speech

The Internet is potential providing raw data for historians -- it should be complete and accurate.

If you would like to see a video of the entire speech, the Internet Archive has preserved that as well:




-----
Update 12/30/2016

The following is a transcript of Bob Garfield, co-host of the podcast On The Media, interviewing Brewster Khale, founder of the Internet archive and a partner in the End-of-term Project with a lead-in question for on climate-science research Eric Holthaus of Slate Magazine.

Bob: Meanwhile a small army of volunteer archivists, scientists and advocates have been working to save the government climate change research that already exists

Eric: at NASA and NOA that takes the temperature of the planet from weather stations from satellites from ocean buoys.

Bob: Meteorologist Eric Holthaus spoke to NPR about his effort to save government climate data.

Eric: Sometimes these data sets are only stored in United States government servers so there hasn't really been an effort to catalog those in other countries because we haven't thought it was necessary before

Bob: The Internet Archive on the other hand has given a lot of thought to what gets lost in presidential transitions. Every week the archive tapes three hundred million Web pages and every four years it enlists a bunch of volunteers to make copies of government Web sites as a hedge against what the next administration may choose to delete. It's called The End-of-term web archive and for some reason this year the organizers are getting a lot more offers of help. Brewster Kahle founder of the Internet Archive says that this year his team also is backing up its data to Canada

Brewster: When the election went the way that it did, it was a bit of a surprise, so we looked through the television archive at what President-elect Trump said about freedom of the press and about the Internet and what we found was shocking. He wanted to close up parts of the Internet that there was mocking of freedom of the press. This was kind of a wake-up call and we said let's make sure we have a copy in some other location.

Bob: What are your priorities? How does it work?

Brewster: So the Internet Archive works with the Library of Congress, University of North Texas -- now a growing list of groups to try to do as best we can to record the information that's available on the Web sites and now the web services that have been made available on .gov and .mil Web sites. We found in 2008, 83 percent of the PDFs that were available back then are no longer available even by 2012. So with an 83 percent loss rate when the Obama administration came on board we're likely to see something like it maybe even more with the Trump administration.

So we're coordinating activities to go and archive web pages and we're reaching out to federal webmasters to go and see if we can keep whole services up and running. Can we take virtual machine versions of the databases that they're running and be able to run them in snapshot form so that we can keep these services going as they were in 2016 in the future?

Bob:Give me some examples of when the federal web archive has come in handy. Was there something that you and disappeared that you were super glad to have archived?

Brewster: Oh the anecdotes go on and on. Example -- there is a press release from the White House during the George W. Bush administration when he stood on an aircraft carrier and declared “mission accomplished.” And the headline of that press release was combat operations in Iraq had ceased but a couple of weeks later they changed the headline and said major combat operations had ceased with no notice that it had changed. The only reason why we know is because we had archived both versions. And then a couple of months later the press release went away completely from the web. You know what is more Orwellian is it changing a press release that's in the past or is it disappearing completely?

Bob: What are you most worried is going to disappear in a Trump administration?

Brewster: Frankly we have no idea. This upcoming administration is very aware of the power of the Internet and how it can be manipulated -- how you can go and push things out in the middle of the night and use the journalist system in ways that are really pretty blatant. So let's at least keep a record of it.

Bob: We have just experienced the interference in a political campaign by outsiders. Is this archive secure -- I mean really secure against hacking, against intrusion?

Brewster: The history of libraries is a history of loss. Libraries are burned. That's what happened in the Library of Alexandria. It'll be what happens to us -- just don't know when. So let's design for it. Let's go and make copies in other places. Let's make sure people want universal access to all knowledge, that they want education based on facts. Let's go and make sure that there is an environment that supports libraries. That's the only way that in the long term we're going to survive. And the copies that are maybe now unique at the Internet Archive will survive based on all sorts of changes whether it's earthquakes or institutional failure or law changes.

Bob: Brewster as always many thanks.

Brewster: Thank you very much.

Bob: Mr. Khale is the founder of the Internet Archive and a partner on the End-of-term Project.

Khale's interview was part of longer podcast episode called Hurry Up. They discussed other steps President Obama could take during the last weeks of his term. The suggestions included disclosing information on contributions by government contractors, surveillance and the drone program, closing Guantanamo and clemency. The episode ends with a discussion of the nature of time by science writer James Gleick.

Finally, I created the interview transcript using a nifty service called PopUpArchive. You simply upload a sound file and wait for the text version to be posted ready for download. It takes a little proofreading and editing, but it is a lot faster than manual transcription and as this Microsoft Research report shows, we can look forward to more accurate speech recognition in the future.




Thursday, December 08, 2016

Trump's China tweets -- data for historians, political scientists, journalists and us

Trump's tweets and other posts provide us with an unprecedented stream of current information and data for political scientists, journalists and future historians.

The New York Times has published a thorough analysis of Donald Trump's recent phone conversation with the president of Taiwan. It takes a multifaceted look at the call, asking whether it was a "diplomatic gaffe or a calculated new start" in our relationship with China.

Only Trump, his advisors and perhaps some of the people he has been interviewing for Secretary of State can answer that question, but we can get clues as to Trump's thinking by looking at his Twitter stream.

A search of his Twitter stream for the word Taiwan, returned only four tweets:


The two October tweets are anti-Obama campaign statements.

The tweet announcing the call says "CALLED ME" in caps. Was that Trump crowing about his stature or was it intentionally saying he had not initiated the call? I cannot know, but I am certain that this was not a casual call -- it was planned and scheduled by both sides in advance.

The latest tweet justifies the call and serves as a message to China and Trump's constituency. (I have to reluctantly admit that I agree -- pretending that Taiwan does not exist is absurd).

Trump's tweets do not provide definitive answers, but they do give us more information about what is going on than we are used to.

"China" tweets

Since Trump is focusing on China, I searched of his Twitter stream for the word China. Twitter returned 276 tweets -- here are the earliest four:


Trump has been posting anti-China tweets for nearly six years. The first had little engagement -- one reply, 73 retweets and 26 likes -- while the latest one has had 22k replies, 39k retweets and 122k likes so far. He was already campaigning at the time of the first tweet, which refers to a site called shouldtrumprun.com. (Today that site contains only a copy of a statement by the Federal Election Commission saying he was eligible to run).

Who is the intended audience for these tweets? No doubt, the early tweets were intended for Trump supporters -- Breitbart readers and Limbaugh listeners -- but future tweets might also be for the general public and the Chinese government.

I have no doubt that both our State Department and the Chinese Foreign Service are well aware of the issues on which our nations co-depend and where we have conflicts, but discussions of such things are traditionally held in private. Whatever you think of Trump, he is providing us with a degree of transparency we are not used to in our politicians and civil servants.

Listening to a fireside chat
New media are mastered by new politicians and Trump's use of social media is reminiscent of the fireside chats President Roosevelt used to communicate with the American people and others when radio became ubiquitous.

If I were a political scientist, I would begin looking at these and Trump's other tweets and posts as research data, ripe for content analysis and fact-checking. They will also be data for historians one day. (The archive of network traffic during the 1991 Soviet coup attempt might be the earliest example of historical data online).

One thing is for sure -- I hope he keeps tweeting after becoming president.







Saturday, January 29, 2011

Before the Twitter revolutions, there was the Usenet revolution


Polina Antonova and Eugene Peskin operating Kremvax, the link between
Relcom and the West

The first example of relatively large-scale citizen journalism online was during the August 1991 coup attempt by eight high-level Soviet officials. There were two days of confrontation between demonstrators and troops, during which all Russian media except Usenet news groups were shut down by the authorities. Usenet, a precursor of today's Internet discussion forums, carried traffic into, out of and within Russia (70 cities) during the days of the coup attempt.

One of the quotes from that time foreshadows subsequent attempts to shut the Internet down. Polina Antonova of RELCOM, the Unix distributor that operated the network wrote:
They try to close all mass media, they stopped CNN an hour ago, and Soviet TV transmits opera and old movies. But, thank Heaven, they don't consider RELCOM mass media or they simply forgot about it. Now we transmit information enough to put us in prison for the rest of our life.
After two tense days, the coup failed and president Mikhail Gorbachev was restored to power. For more on the role of the network during the coup attempt, see:
The following video clip (3:13) was part of a documentary on the Internet. (It includes a short interview of me at the time -- no white hair :-).



Usenet played a relatively significant role during the Soviet coup attempt, but there was also some Usenet traffic during the 1989 Tienanmen Square protests. Everyone is familiar with the heroic image of the unidentified man stopping a column of tanks on June 6 1989 -- the day after troops had opened fire on protesters.


But, there was some Usenet traffic as well. Olivier Crepin-Leblond has saved a description of the situation that was posted by a Sun Microsystems employee on May 23. It describes the optimistic time before the storm, beginning with:
The situation here seems getting better and better. All army members are blocked outside Beijing city. The people’s life in the city looks as normal as usual.
Here is a movie clip of the tank man at Tiananmen Square:



This clip was not on the Internet at the time, but it is today -- providing a dramatic view of a heroic act and illustrating the futility of China's attempt to erase the memory of the event.

-----
Update 8/20/2016

Slate has published a relatively long post entitled "An act of courage on the Soviet Internet" on RELCOM's role during the Soviet Coup attempt of 1991. The article describes my involvement at the time, but fails to link to the archive of the network traffic and other material, which is available to historians and others interested in the event.

I believe this was the first example of active citizen journalism involving the Internet. The Internet has given us a means of documenting historical events in detail -- "big data" for historians. We need a mechanism for discovering and preserving such archives.

-----
Update 1/12/2017

Mark Graham, who relayed news over the Sovam Teleport link during the Soviet coup attempt, just sent me an article on their activity at the time. Unfortunately, the data they transmitted and received has been lost. Our archive is still online at the State University of New York site mentioned above and, even better, it is online at the Internet Archive.


-----
Update 7/7/2018

For an excellent article on the role of RELCOM in the Soviet Coup attempt, click here.