FAME: facial recognition as a tool for metadata creation
What if metadata were so complete that you could find all the images of someone in your archive if you knew that person’s name? Doing all the necessary metadata creation and enrichment work manually would be a painstaking task, which is why we’re researching how we can use facial recognition to help.
Digital photo and video collections managed by cultural organisations often lack good-quality substantive metadata. There are various reason for this:
describing these collections manually is very time-consuming;
it often involves large collections which aren’t part of the organisation’s main collection, and so, aren’t a priority;
the nature of the content means there are additional technical requirements: videos need to be played to be able to manually describe the people shown – which is very time-consuming;
videos can be difficult or impossible to play if they have not been digitised, making it impossible to describe their content.
Limited metadata makes these digital collections difficult to find and search through online, so they are harder to look up and re-use. At the start of 2021, we received a project grant from the Minister of Culture Jan Jambon for the FAME project, in which we're developing best practices for identifying people in photos and videos using (semi-)automated facial recognition. We're also investigating how we can use existing metadata to improve facial recognition accuracy.
Metadata and linked (open) data are two of the five knowledge domains we have identified at meemoo. In 2023, we’re aiming to set up a proof-of-concept for the large-scale application of facial recognition technology in the meemoo archive system, for the benefit of metadata creation and enrichment. The pilot projects we’re running as part of FAME will help us prepare for this.
Following our previous collaborations with FOMU (Photo Museum of Antwerp) and MoMu (Fashion Museum of Antwerp) on visual recognition projects, we’ve even researched possibilities for a specific type of visual recognition, i.e. facial recognition, together with several partners.
We’re running three pilot projects in the FAME project:
pilot project for performing artists in collaboration with Flanders Arts Institute
8,300 pictures of productions and 700 (group)photographs
pilot project for sportspeople in collaboration with KOERS (the Museum of Cycle Racing)
a selection of 41,000 photographic negatives by Maurice Terryn from 1969-1978
pilot project for politicians and activists in collaboration with ADVN (Archive for National Movements) and the Flemish Parliament Archive
ADVN: 8,300 digitised pictures from the magazine WIJ and 533 digitised videos by VNOS
Flemish Parliament Archive: 5,300 pictures from the Flemish Parliament and 300 videos of their sessions from 1992-1994 and 2001-2021
Our technical partner for each of these projects is IDLab from Ghent University.
In order to prepare for the FAME project, Liesbeth Bogaert worked with us as a trainee in the summer of 2020. At the time, she was a student of Information Engineering Technology at Ghent University, had just completed her third bachelor’s degree, and was supervised by meemoo colleagues Nastasia and Miel, and Prof. Steven Verstockt from Ghent University, during her work placement at meemoo. She worked with a set of around 9,100 digitised photos and 19,500 born-digital photos from Flanders Arts Institute.
Metadata-driven facial recognition
We’re using metadata-driven facial recognition in this project, and – in an attempt to improve and enrich the facial recognition results – researching how we can use our content partners’ metadata and linked open data sources to train the facial recognition algorithms and post-process the software output. In the process, we make use of reference sets.
The names of the folders provided by Flanders Arts Institute give us the following information for many of the photos:
name of performing arts organisation
We can use this information to look up further details in Wikidata and the Flanders Arts Institute database. This means we can find out which actors, dancers and performers have worked on certain productions and could appear on these photos, and use this information to limit any mistakes. Furthermore, we can then also identify each recognised actor, dancer or performer with a unique code from Wikidata and the Flanders Arts Institute database.
Alongside the many benefits of using metadata, it brings some challenges. Metadata spring from different sources which means they exist in many different shapes. Writing errors, abbreviations and inconsistencies occur regularly, the metadata is not structured or not even present. Several manual actions must be taken to solve these issues.
Technology such as facial recognition should be treated with caution. We’re taking this into account in the project and making sure we heed specialist legal advice, among other things. We also have a legal obligation to process the personal data of the people depicted responsibly.