Two cool podcast interviews on bit rot, an unsolved problem

We all need to be aware of the problem of bit rot in our work.

One of my favorite podcasts, OnTheMedia, produced a program called Digital Dark Age (52:14) last year. It consists of several segments on the problems of protecting and archiving the vast amounts of data we are generating. If you don't have time to listen to the entire podcast, at least check out two segments, interviews of Vint Cerf on information preservation problems (6:28) and Nick Goldman on using DNA as a storage medium (9:02).

Let's start with the Goldman interview. He notes that applications like video entertainment and scientific research are generating immense amounts of data, which is already overwhelming today's optical and magnetic storage media. Goldman is experimenting with DNA as a very high capacity, long lived data storage medium. How might that work?

A strand of DNA is made up of strings four bases abbreviated A, T, C and G, as shown here:

The DNA strand is like a twisted ladder where
the "rungs" are either A-T or C-G bonds. (For
more, see this animation).

Biologists have developed equipment for sequencing (reading) the list of bases making up a strand of DNA and for synthesizing arbitrary strands of DNA. That makes it possible to store a copy of a binary file in a strand of synthesized DNA, for example by synthesizing strands of DNA in which binary 0s are represented by an A or C base and 1s are represented by a T or G. A DNA sequencer could then convert those A, T, C and Gs back into 1s and 0s.

Goldman thinks we will see expensive DNA storage devices in three or four years and they will be cheap enough for consumer storage in 10-15 years. DNA stored in cool dry places will last hundreds of thousands of years and "all the digital information in the whole world everything that's connected to the Internet" will fit in "the back of a minivan." If Goldman falters there are other DNA storage projects at Harvard (article, video) and Microsoft.

But even if DNA or some other storage technology gives us dense, cheap storage, there are other problems, as outlined by Vint Cerf.

Cerf talks about "bit rot." The simplest type of bit rot is media deterioration -- becoming unreadable after 20 0r 30 years. That can be overcome by creating a new copy periodically, but that is not be enough. Let me give you a personal example.

In 2004 my wife had a small hole in the atrial wall of her heart repaired. In a 30 minute, outpatient procedure a skilled surgeon installed a small device in which tiny umbrellas were clamped over the hole, held there by a spring.

A ballon determines hole diameter (left), the device in place (right)

That was pretty amazing, so I asked for a video of the procedure, which was provided on an optical CD. The CD also included a program for viewing the video, the Ecompass CD Viewer. The CD media might be somewhat deteriorated by now, but I have transferred the program and data to magnetic storage, so I can still view it on my laptop.

But, my laptop is running Windows 7. The version of Ecompass CD Viewer I have was written in the Windows XP days. It still works, but will it be compatible with Windows 10 or later? If not, I could upgrade to a current version, but, as far as Google Search knows, Ecompass CD Viewer is no longer with us -- perhaps the company went bankrupt or dropped the program.

Ecompass CD Viewer works by stringing together clips stored in a video file format called SSM. If I could find an SSM viewer, maybe I could cobble together the entire video since I have the SSM clips. That might work for these static files, but we need the functionality of the original program for something that a user can interact with, like a spreadsheet. Looking further in the future Windows will disappear along with Intel-processor machines.

Cerf does not see a complete solution to the bit rot problem, but he pointed to Project OLIVE at Carnegie Mellon University as a significant step in the right direction. OLIVE emulates old computers running old software on virtual machines. Below, you see an image from a demonstration of an emulation of a 1991, OS7 Macintosh running Hypercard. The display, mouse and keyboard of the Dell PC are interacting with the simulated Macintosh, which is running on an Internet server.

Still image from a Project OLIVE demo video.

The OLIVE demonstration is impressive, but it is a research prototype that assumes standard input/output devices and capturing the vast number of programs and hardware configurations that exist today would require a massive effort. Similarly, DNA storage is at the early proof-of-concept stage. Both feel like longshots to me, so, for now, we all need to be aware of the problem of bit rot in our work.

Update 3/9/2018

Checkout this video putting DNA storage in context.

Vint Cerf -- a concise history of packets, the ARPAnet and the Internet

Conceptual sketch of the ARPAnet by Larry Roberts

If you are interested in the history of the Internet and only have 16 minutes to spare, watch this interview of Vint Cerf, who was one of the handful of people that created the Internet.

Cerf's narrative begins with the idea of packet switched communication and runs through the creation of the ARPAnet, followed by the invention of internetworking protocols to link three disparate networks -- the ARPAnet, a mobile communication network and a satellite communication network.

This short video is like an annotated table of contents of the early history of the Internet. Cerf introduces us to Leonard Kleinrock, Paul Baran, Donald Davies, Larry Roberts, Thomas Marill, J. C. R. Licklider, Doug Engelbart, Norman Abramson, Robert Taylor, Charles Herzfeld, Steve Crocker, Jon Postel, Bob Kahn, David Reed, Danny Cohen and Bill Joy, summarizing the work of each and putting it in context.

Whether this is all you want to know about the history of the Internet or you want to use it as a jumping off place to learn more about the contributions of these people, this is a good place to start.

Cerf also conveys a sense of common purpose among those pioneers. He does not say so, but one can think of them as working together to realize Licklider's vision of a network running Engelbart's applications. Cerf makes it clear that, although they worked for several different organizations and changed jobs from time to time, these people knew each other well and collaborated closely on creating the Internet. (For example, Cerf, Postel and Crocker went to the same high school and studied under Kleinrock as graduate students at UCLA). The group also had an excellent collaboration tool -- they were the first users of the networks they built.

The ARPA/Internet project was a great example of government as a non-equity angel investor -- providing a small bit of seed funding ($124 million) to a group of smart, dedicated people, rather than setting up a department to do the work internally.

Charles Severance of IEEE Computer Magazine conducted the interview of Cerf, and it is one in a series of computing conversations.