Software refinement -- from research to products -- an image processing example

We have discussed the birth of image processing with Ivan Sutherland's Sketchpad program in the 1960s. As we see in this video, Sketchpad could draw, rotate, resize, copy and combine images.

Since Sutherland's research, there has been tremendous improvement in hardware technology and innumerable engineering refinements to image processing software.

Software refinements require research and the development of prototypes before they become product features. Consider, for example, the PatchMatch algorithm developed in collaboration between Princeton University and Adobe Systems. Here we see a paper describing the algorithm and some demonstration videos. The paper was presented and the videos shown at an academic engineering conference.

Successful research like the PatchMatch algorithm is subsequently incorporated into commercial products. Here you see a demonstration of tools that use PatchMatch for image retargeting, completion and reshuffling integrated into Adobe Photoshop. The tools are being demonstrated at a trade show, and it is probably safe to say they will be in the next version of Photoshop.

This is typical of engineering refinements to software. A prototype is developed in a research lab and it later becomes a product.

Researchers typically work on things that are too demanding for the technology of the day. Some of the Princeton videos are shown at 5 times actual speed because it takes time to compute the new images. By the time these features are released in Photoshop, hardware will have improved and they will execute rapidly.

Note that Sutherland's work was a pure university effort, but PatchMatch was a joint effort between a university and a company. Many significant advances have come from industrial research labs. Can you find some interesting prototypes and projects at Microsoft Research?