Using Flickr To Edit Your Photos

By Roland Piquepaille

What can you do with millions of images? Computer scientists from Carnegie Mellon University (CMU) think you can use web images to add realism to your photos. For example, the goal of one of their projects, named 'Photo Clip Art,' is to insert new objects into existing photographs by querying a vast image-based object library. Another one, called 'Scene Completion,' wants to help you to fill 'holes' in your pictures by selecting the right pictures in the Flickr database to enhance them. These two projects will be presented at the next SIGGRAPH next month in San Diego.

These two projects have been led by Alyosha Efros, assistant professor of computer science and robotics. The first one, Photo Clip Art, was developed with several graduate students, including Jean-Francois Lalonde. You can see above how the system can insert objects picked from the LabelMe database in a picture (Credit: Alyosha Efros). Here is a link to some other examples created with the system.

This research work will be presented at the Association for Computing Machinery’s Special Interest Group on Graphics and Interactive Techniques (SIGGRAPH 2007) annual conference under the name -- you guessed it? -- "Photo Clip Art." Here is a short excerpt from the abstract:" We present a system for inserting new objects into existing photographs by querying a vast image-based object library, precomputed using a publicly available Internet object database. The central goal is to shield the user from all of the arduous tasks typically involved in image compositing. The user is only asked to do two simple things: 1) pick a 3D location in the scene to place a new object; 2) select an object to insert using a hierarchical menu."

The CMU news release adds more details. "To make the resulting image appear as realistic as possible, the system analyzes the original photo to estimate the camera angle and lighting conditions, and then looks in the clip art library for an object — a car, for instance — that matches those criteria. The user need only identify the horizon in the original photo to orient the system. Using previously developed Carnegie Mellon technology for analyzing the geometric context of a photo, the system can then place the object within the scene, adjusting its size as necessary to put it in proportion to other objects of equal distance from the camera."

For even more information about this project, you can today read the full document that will be published in the next issue of ACM Transactions on Graphics (August 2007, Vol 26. No. 3) (PDF format, 10 pages, 15.2 MB).

The other project, Scene Completion, has been developed by graduate student James Hays, another member of Efros’ research team. You can see above an example of the results obtained by this system, which uses the millions of pictures found of Flickr (Credit: James Hays). "The input and the matching scenes are composited together to create the outputs. The matching scene used to produce each output is highlighted in red. Note that the algorithm can handle a large range of scenes and missing regions."

Here are some more details coming from the CMU news release. The system "draws upon millions of photos from the Flickr Web site to fill in holes in photos. Some of the holes might be from damage to a physical photograph, but more often they are created when an editor cuts out part of an image to eliminate an unsightly truck from a picturesque street scene, or removing a passerby from a group shot of friends. Photo editors often try to fill in those holes with sections derived elsewhere in the same image, but Efros said that a better match can often be found in a different photo."

Like the Photo Clip Art project, the Scene Completion will be presented at SIGGRAPH 2007. Here is a short excerpt from the abstract: "In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of image completions and we allow users to select among them."

For even more information about this project, you can today read the full document that will be published in the next issue of ACM Transactions on Graphics (August 2007, Vol 26. No. 3) (PDF format, 7 pages, 11.0 MB).

Does this mean we'll soon have access to these technologies? CMU doesn't say.

Sources: Carnegie Mellon University news release, July 10, 2007; and various websites