Two cool podcast interviews on bit rot, an unsolved problem

We all need to be aware of the problem of bit rot in our work.

One of my favorite podcasts, OnTheMedia, produced a program called Digital Dark Age (52:14) last year. It consists of several segments on the problems of protecting and archiving the vast amounts of data we are generating. If you don't have time to listen to the entire podcast, at least check out two segments, interviews of Vint Cerf on information preservation problems (6:28) and Nick Goldman on using DNA as a storage medium (9:02).

Let's start with the Goldman interview. He notes that applications like video entertainment and scientific research are generating immense amounts of data, which is already overwhelming today's optical and magnetic storage media. Goldman is experimenting with DNA as a very high capacity, long lived data storage medium. How might that work?

A strand of DNA is made up of strings four bases abbreviated A, T, C and G, as shown here:

The DNA strand is like a twisted ladder where
the "rungs" are either A-T or C-G bonds. (For
more, see this animation).

Biologists have developed equipment for sequencing (reading) the list of bases making up a strand of DNA and for synthesizing arbitrary strands of DNA. That makes it possible to store a copy of a binary file in a strand of synthesized DNA, for example by synthesizing strands of DNA in which binary 0s are represented by an A or C base and 1s are represented by a T or G. A DNA sequencer could then convert those A, T, C and Gs back into 1s and 0s.

Goldman thinks we will see expensive DNA storage devices in three or four years and they will be cheap enough for consumer storage in 10-15 years. DNA stored in cool dry places will last hundreds of thousands of years and "all the digital information in the whole world everything that's connected to the Internet" will fit in "the back of a minivan." If Goldman falters there are other DNA storage projects at Harvard (article, video) and Microsoft.

But even if DNA or some other storage technology gives us dense, cheap storage, there are other problems, as outlined by Vint Cerf.

Cerf talks about "bit rot." The simplest type of bit rot is media deterioration -- becoming unreadable after 20 0r 30 years. That can be overcome by creating a new copy periodically, but that is not be enough. Let me give you a personal example.

In 2004 my wife had a small hole in the atrial wall of her heart repaired. In a 30 minute, outpatient procedure a skilled surgeon installed a small device in which tiny umbrellas were clamped over the hole, held there by a spring.

A ballon determines hole diameter (left), the device in place (right)

That was pretty amazing, so I asked for a video of the procedure, which was provided on an optical CD. The CD also included a program for viewing the video, the Ecompass CD Viewer. The CD media might be somewhat deteriorated by now, but I have transferred the program and data to magnetic storage, so I can still view it on my laptop.

But, my laptop is running Windows 7. The version of Ecompass CD Viewer I have was written in the Windows XP days. It still works, but will it be compatible with Windows 10 or later? If not, I could upgrade to a current version, but, as far as Google Search knows, Ecompass CD Viewer is no longer with us -- perhaps the company went bankrupt or dropped the program.

Ecompass CD Viewer works by stringing together clips stored in a video file format called SSM. If I could find an SSM viewer, maybe I could cobble together the entire video since I have the SSM clips. That might work for these static files, but we need the functionality of the original program for something that a user can interact with, like a spreadsheet. Looking further in the future Windows will disappear along with Intel-processor machines.

Cerf does not see a complete solution to the bit rot problem, but he pointed to Project OLIVE at Carnegie Mellon University as a significant step in the right direction. OLIVE emulates old computers running old software on virtual machines. Below, you see an image from a demonstration of an emulation of a 1991, OS7 Macintosh running Hypercard. The display, mouse and keyboard of the Dell PC are interacting with the simulated Macintosh, which is running on an Internet server.

Still image from a Project OLIVE demo video.

The OLIVE demonstration is impressive, but it is a research prototype that assumes standard input/output devices and capturing the vast number of programs and hardware configurations that exist today would require a massive effort. Similarly, DNA storage is at the early proof-of-concept stage. Both feel like longshots to me, so, for now, we all need to be aware of the problem of bit rot in our work.

Update 3/9/2018

Checkout this video putting DNA storage in context.

Amazon's Glacier storage service will disrupt online backup

The bar chart below sums it up -- Amazon's Glacier is cheap -- 1¢ per gigabyte per month. (Follow the link for pricing detail).

Amazon has published a Glacier API, and, for now, you have to be a programmer to use it. But they have committed to releasing FTP access very soon, and it will not take long for developers to release backup programs and services with graphical user interfaces to Glacier.

Google's storage pricing model combines the self-serving features of cell phone and ISP pricing

Here is Google's report of my storage utilization. It says I've not used any of my Picasa storage, but when I go to Picasaweb, I am told that I've used 84% of my one gigabyte. (I like the first result best, but I really have used most of my allocated gigabyte).

But, that reporting discrepancy is a minor bug. A more fundamental problem is that Google differentiates between my storing images and video (Picasa), email messages with attachments (Gmail) and arbitrary files (Google Drive).

That makes as much sense to me as differentiating between voice bits, data bits and text message bits on your cell phone bill. It's all bits.

(I am waiting for the water company to begin charging me different rates for my drinking, showering and car-washing water).

But, wait, there's more. Checking Google's storage pricing policy, we learn that pictures and videos you upload from Google Plus do not count against your Picasa limit and Google docs do not count against your Google Drive limit. Etc. (I said "etc.," because I got tired of reading the fine print).

That reminds me of Comcast not counting their own video material against download caps.

Google -- this seems a bit evil -- please unify your storage policy.

Intel Light Peak -- goodbye USB?

We have seen many examples of early prototypes of devices that eventually became main stream. Intel hopes to replace your USB cable with an optical link they are calling "Light Peak."

Here you see a demonstration of a laptop streaming two simultaneous HDTV programs to a television set. The black box between the laptop and display will eventually disappear, leaving a USB replacement.

Today's USB 2.0 connections have a speed of 480 Mbps, while Light Peak connections will begin at 10 Gbps in both directions. With mass production and engineering refinement, speeds would increase well beyond that. This is a common pattern -- one technology approaches its limit, and is then leapfrogged by a new technology.

This also seems to happening as electronic flash storage replaces rotating magnetic disk drives. Many portable devices already use electronic storage, and flash is beginning to be used in laptops and servers. (I've ordered my next laptop with flash storage).

If Light Peak succeeds (and it may not), it will dramatically alter the speed relationships between input/output and storage devices and memory, resulting in significant system redesign.

Can you think of other instances in which a new information technology has replaced an older one? What did we use for storage before hard drives?

New storage technologies replace old ones

In our discussion of progress in storage technology, we saw that large improvements occur when new methods of storing bits -- of differentiating ones from zeros -- are invented. Early computers used punched cards in which a one was recorded by punching a hole in a particular position on the card:

We no longer use holes in cards, but have moved to microscopic magnetized spots or pits on the surface of a rotating disk for storage. These are now being replaced by flash memory for some applications.

Researchers are always looking for new ways to differentiate between ones and zeros, and this article mentions several technologies that may take over for flash drives one day.

Do you remember computers with punched card, magnetic tape, or floppy disk storage? Does your mp3 player use a rotating disk or a flash drive for storage? Large flash drives are becoming available -- will your next laptop have a hard disk or a flash drive? What advantages would a flash drive have in a laptop? What disadvantages?

Storage progress -- every movie ever made on an iPod

We discuss the progress in electronic, communication and storage technology that underlies advances in network-based applications. Robot designer Rodney Brooks uses the Apple iPod to illustrate improving storage capacity. If the current exponential rate of growth continues, an iPod will be capable of holding every movie ever made by 2020. (That includes all the bad movies -- he figures all the worthwhile movies will fit by 2016).

Brooks made the point in a talk outlining his work on robotics. You can hear an excerpt on the growth of iPod storage here, listen to the entire talk here, or watch videos of a robot in Brooks' lab interacting with humans here.

How old will you be when your iPod has the capacity to hold every movie ever made? What other applications might it be used for? Will it be useful to you in your work and career?