Image Processing Aids Art Restoration

Kevin Robinson

On Sept. 26, 1997, a magnitude 6.0 earthquake in the Umbrian and Marches provinces of central Italy brought the Cimabue and Giotto frescoes in the Basilica of St. Francis in Assisi, Italy, crashing to the floor. Think, for a moment, of the challenge to art restoration experts reassembling the tens of thousands of fragments of the 13th-century marvels. Few jigsaw puzzles could be more daunting -- or more important to get right.

After a 1997 earthquake shattered the masterpieces in the Basilica of St. Francis, researchers developed an image-analysis technique to put the rescoes back together. The system analyzes blocks of pixels on each shard and attempts to match them with the patterns in a photograph of the intact painting.

The nightmare of eyeballing where each shattered piece of painted plaster belongs in the huge paintings prompted researchers at the University of California and University of Roma Tre in Rome to find a more efficient computerized technique. The collaboration has produced an image-processing system that could help piece together the frescoes and could help Internet users find pictures more intuitively.

"For a fresco-restoration project, the analysis method should compute features that are not variant with respect to image position or orientation, since we do not know the original orientations of the fresco pieces," explained Serkan Hatipoglu, a member of the Santa Barbara team, which was led by Sanjit Mitra, a professor of electrical and computer engineering who specializes in color filtering and matching. "The orientation and frequency -- the rate of change in image brightness -- within the patterns need to be extracted. Then, for each region in the complete fresco image, the pieces with the most similar features are found and placed."

Marco Carli and Alessandro Neri of Roma Tre had developed a system that matched up to 98 percent of the fragments to a copy of a fresco that was painted and smashed for the experiment. But it was time-consuming because the technique matched individual pixels on the shards with the original image. The group turned to an analysis of texture, or blocks of pixels.

"Using texture recognition, we have a more robust way to find the locations of the pieces," Hatipoglu said. "We compare patterns instead of pixels, so changes in small regions caused by dust, smearing or fading do not affect the location matching as much."

Shards and searches

The more information the programmers have, the better able they are to match the fragments to the original image. And the process works best when they know the photographic conditions of the image, such as the camera position and lighting. Carli and Neri suggest that all artwork in earthquake-prone locations should receive a systematic photographic survey that includes this information.

The investigation is part of ongoing work on image analysis, search and retrieval, Hatipoglu said, and the results will aid research into content-sensitive image retrieval based on classification by texture from large databases.