When computers look at art

When computers look at art

Nowadays, computers are important tools for looking at art. Here's how.

bookmark
Tue 15 Oct 2024 11:37 AM

For centuries, art scholars, curators, conservators, connoisseurs and the art-loving public alike have relied on their own extensive knowledge and highly tutored eyes to understand and interpret the images found in fine-art paintings and drawings. In the past several years, however, an intriguing additional approach has arisen: computer-based image analysis of art. This new approach is not a replacement of traditional methods of analysis, but instead serves as a tool for those interpreting art, much as a microscope empowers a biologist and a telescope empowers an astronomer.  

David G. Stork
Computer analysis of Verrocchio’s Baptism of Christ (Uffizi). Computer “segmentation” identifies the locations of flesh (and hence figures), as shown in dark red and pink.

Part of the power of computer methods rests on the fact that computers can learn from an immense amount of art and related knowledge. A computer can, in principle, “see” digital images of every artwork ever created, including underdrawings and pentimenti revealed by sophisticated imaging technology, “read” every word written about every artwork in every language, have perfect knowledge of every pigment, support and much more—far more knowledge than could be possessed by any team of art scholars studying for hundreds of lifetimes. And computers never die… Their knowledge will only grow throughout the foreseeable future.

Advertisements

Computer methods applied to digital images of artworks and guided by traditional art context and scholarship expand our interpretative strategies and understanding of art in several ways, and have even resolved art debates that were resistant to mere traditional approaches applied alone.  

First, computer methods can automate traditional analyses of formal properties of paintings, such as portrait pose, landscape composition, color palette, brushstrokes, marks, perspective and more. In this way, we can now easily track, visualize and understand trends in such formal properties in tens of thousands of paintings over centuries—far more than can be done “by eye”—revealing, for example, how the dramatic sweeping “diagonal” compositions of the Baroque (Caravaggio, Titian, etc.) diffused throughout Europe, displacing the more “vertical” compositions favored in the Italian Renaissance (Giotto, Piero della Francesca, and so forth).  

Second, computer methods are more subtle and accurate than even the most tutored connoisseur on a number of properties of images. For instance, most viewers are surprisingly poor at detecting inconsistencies in lighting or even discerning the direction of lighting throughout a tableau, but computer methods are exquisitely accurate in lighting estimation. Such computer lighting analysis gives scholars a deeper understanding of praxis and artistic style, from the superb lighting consistency in the works of Johannes Vermeer and Caravaggio to the deliberate and expressive “inconsistencies” in works by René Magritte, Giorgio de Chirico and others. 

Third, computer methods are being applied to the vexing and vitally important task of art attribution and authentication. The number of fake and misattributed artworks in museum collections is surprisingly large—much larger than the general public realizes—perhaps as much as 20 per cent. (Some experts estimate that the proportion of fakes and misattributed works in the commercial art market might be twice that rate.) Hundreds of billions of dollars are at stake as well as an accurate knowledge of our cultural heritage (not to mention the trust in our institutions preserving our cultural patrimony). Computer methods show promise in authentication studies based not only on images but also provenance (documentary record of sales and ownership of works), material studies of pigments and canvases, condition of a work, broad knowledge of the prevalence of forgeries, and more.

David G. Stork
Computer “recognition” identifies the religious attributes (here, dove and crucifixion cross). Additional computer analysis thus identifies the figures as Christ and St. John the Baptist (and angels), and thus the Biblical story of the Baptism.

Fourth, computers are only very recently addressing the problem of semantics, that is, interpreting an artist’s meaning or intention expressed in a painting. We don’t fully understand a work such as Botticelli’s Birth of Venus by a mere description of its elements and composition.  Instead, we bring knowledge of Greek mythology, allegory and symbolism to our interpretation.  Up to now, computer methods have not been applied to this deeper level of understanding of an image, but first steps suggest that there is nothing in principle preventing computers from “understanding” such messages, lessons and morals expressed through art. For example, computer methods can now infer that the shaft of light onto a skull in a vanitas painting refers to religious morals about human mortality and the promise of an everlasting life after death. Much fascinating research on such problems lies ahead.

Fifth, computer methods have shown promise in reconstructing images of lost or stolen art. On February 7, 1497, the Dominican friar Girolamo Savonarola incited the Bonfire of the Vanities (falò delle vanità) in Florence, where gangs of fundamentalist Christian thugs destroyed “lascivious artworks” like paintings by Botticelli, Baccio della Porta, Lorenzo di Credi depicting women’s hats, mirrors, wigs, dolls, cupids, playing cards, dice boards and musical instruments. Enraged art lovers may appreciate the arc of justice knowing Savonarola was later excommunicated from the church, found guilty of heresy and sedition, executed by hanging and cremated.

Ongoing research is attempting to recover the images of lost or destroyed artworks, such as from that fateful day in Florence, by computationally integrating images of surviving preparatory sketches, copies, other artworks by the target artist, and even contemporaneous textual descriptions of the works before they were destroyed. Computer research may someday lead to an online “museum of lost art”, leading to a more complete understanding of the history of art.

As mentioned, computer methods have been essential for resolving some debates among art scholars. Perhaps the clearest example is David Hockney’s highly promoted theory that some artists as early as 1420 secretly used optical devices to project images onto their canvases or wood panels, traced these images and then filled in paint. Hockney was seeking to explain the emergence of a heightened realism or “optical look” in the “new art” (ars nova) of that time. Careful computer analyses of lighting, perspective and the purported optical devices have led to the overwhelming scholarly rejection of this “tracing theory,” at least for the early Renaissance. We know that some artists, such as the Baroque master Canaletto, used the camera obscura as part of their praxis. (He used such a technique for his vedute, or view paintings of Venice and London.) It should be noted that he worked more than two centuries later than the era Hockney claimed optics were used in art praxis this way.

The gentle revolution in art scholarship based on computing has a precedent. Giovanni Morelli (1816–91) was a politician, playwright and medical doctor who late in life turned his attention to the study of art. He applied his experience in careful observation of the physiognomy of his patients (used for diagnosing disease) to the authentication of artworks. As explained in his Della pittura Italiana of 1897, he found that different Renaissance masters rendered fingers, ears and hands of sitters each in a distinctive style. Armed with his new “scientific connoisseurship”, he reattributed more than 100 artworks in an impressive feat of art scholarship. By analogy, computer methods promise a new “scientific” approach to art scholarship, which in the startup parlance of our time could be called Morelli 2.0.

David G. Stork

This scholarly vision was the impetus behind the writing of the field’s first book, Pixels & paintings: Foundations of computer-assisted connoisseurship (Wiley, 2024).

There is every reason to believe that these computer methods will continue to empower art scholarship and, like Morellian analysis, which is still used today, enhance and expand our understanding of art in the decades and centuries to come.

Related articles

ART + CULTURE

Botticelli displayed in Chambord

Bringing two paintings together reflects the influences of Italian artists in the Loire Valley.

ART + CULTURE

Giunti Odeon celebrates a year of success

Everyone who visits the cultural centre on November 4 will receive a free book.

ART + CULTURE

Siena: The Rise of Painting exhibition at The Met in New York

Featuring more than 100 paintings, sculptures, metalworks and textiles.

LIGHT MODE
DARK MODE