Speakers
Description
In this talk we'll present our methodology and prototype for a meta-collection system called Meta-Acervos that uses different AI models and computer vision techniques to recombine existing archive metadata. We’ll share the challenges we faced when using pre-trained models and how we addressed questions of vocabulary and legibility in the art history field.
This work was carried out within the context of a digital humanities research group (Digital Collections and Archives) that aims to create strategies for visibility of underrepresented narratives.
The current interface focuses on combining and expanding information from 17 Brazilian museum collections available in public datasets and APIs like Wikimedia and the Brasiliana Museus project, with 4000 artworks from the XIII to XXIst century.
Our methodology makes use of open-source multimodal models like CLIP, SigLIP and OWLv2 to annotate, cluster and classify paintings and drawings according to their visual and semantic characteristics.
The augmented data and interface not only allow for new ways of searching and organizing the collections, but also enable new ways of visualizing their content, like creating composite grids with every instance of particular objects found in the artworks. Another possibility is to place cropped objects on a blank canvas, sized and positioned relative to their original location in the artworks.
This can be used to visually explore aggregate characteristics and patterns in groupings like: all the hands in religious paintings, or all the hands and faces from artworks in a specific collection, or all of the palm trees extracted from paintings created between the a given period:
Hands in religious paintings
Faces in portraits
Palm trees in the 1800s
Nevertheless, the diversity of the profiles of the collections available in Meta-Acervos leads to significant ambivalences in the use of artificial intelligence for the treatment of museum collections and, consequently, within the field of art history. On one hand, the system, based on computer vision tools, enables image-based searches that allow combinations (by color, shape, and internal visual elements such as flora and fauna) that break with the traditional canons of art-historical databases (author, date, and style). These features suggest speculative and exploratory curatorial approaches that bring forth alternative aesthetic and historical narratives. On the other hand, the results of the visualization and search filters reveal the fragility of AI models when dealing with contemporary artworks (most of which are non-figurative) and the weight of biases in the interpretation of images depicting Black and Indigenous people, thereby reinforcing stereotypes rooted in conservative historiographies.
However, visualization resources—such as the latent space and the distribution of artworks over time—are tools that reveal historiographical layers absent from online digital archives. The visualization of the latent space, for instance, makes explicit the ways in which AI models have systematized information, functioning as an invisible map of the emphases and priorities established by machine learning. In turn, the timeline allows users to understand the dynamics of acquisition and the patterns of interest that have shaped the institutions featured in Meta-Acervos.
Code and extracted metadata is available on GitHub and a public interface for navigating the results is available here.