FAME: facial recognition as a tool for metadata creation
What if metadata were so complete that you could find all the images of someone in your archive simply by knowing that person’s name? Doing all the necessary metadata creation and enrichment work manually would be a painstaking task, which is why we investigated in this project how new technologies such as facial recognition could help.
Digital photo and video collections managed by cultural organisations often lack good-quality content metadata. There are various reason for this:
describing these collections manually is very time-consuming;
it often involves large collections which aren’t part of the organisation’s main collection, and so aren’t a priority;
the nature of the content means there are additional challenges: you need to actually watch videos to be able to describe the people appearing in it – which is very time-consuming;
videos can be difficult or impossible to play if they haven’t been digitised yet, making it impossible to describe their content.
Limited metadata makes these digital collections difficult to find and search through, so they are harder to consult and re-use. As part of the FAME project, for which we received a project grant from the Flemish Minister for Culture, Jan Jambon, at the start of 2021, we therefore developed best practices for using (semi-)automated facial recognition to identify people in photos and videos. We also investigated how to use existing metadata to improve facial recognition accuracy.
Metadata and linked (open) data are two of the five knowledge domains we have identified at meemoo. The FAME pilot projects served as preparation for larger-scale applications – such as the GIVE metadata project – for facial recognition technology aimed at helping with metadata creation and enrichment in the meemoo archive system.
Following on from our previous collaborative projects on visual recognition with FOMU (Photo Museum of Antwerp) and MoMu (Fashion Museum of Antwerp), we researched possibilities for a specific type of visual recognition – i.e. facial recognition – together with several partners.
We ran three pilot projects as part of the FAME project:
pilot project for performing artists in collaboration with Kunstenpunt (Flanders Arts Institute)
This involved us taking 8,300 photos of productions and 700 (group) portraits alongside several videos;
pilot project for sports people in collaboration with KOERS (Museum of Cycle Racing)
We used a selection of 115,000 photo negatives from Maurice Terryn for the period 1967-2006 for this;
pilot project for politicians and activists in collaboration with ADVN (Archive for National Movements) and the Flemish Parliament Archive
We based this project on 8,300 digitised photos from the ADVN magazine ‘WIJ’ together with some digitised videos from VNOS;
We also used 5,300 photos from the Flemish Parliament and several videos of Flemish Parliament sessions from the periods 1992-1994 and 2001-2021 from the Flemish Parliament’s archive.
IDLab from Ghent University was our technical partner in each of these projects.
To prepare for the FAME project, Liesbeth Bogaert worked on an internship with us in the summer of 2020. She was a student of Industrial Computer Science Engineering at Ghent University, had just completed the third year of her bachelor’s degree, and was supervised during her work placement at meemoo by meemoo colleagues Nastasia and Miel along with Prof. Steven Verstockt from Ghent University. She worked with a set of around 9,100 digitised photos and 19,500 born-digital photos from Flanders Arts Institute.
Metadata-driven facial recognition
We used metadata-driven facial recognition in this project. Then, in an attempt to further improve and enrich our facial recognition results, we also used reference sets to investigate how we could use our content partners’ metadata and linked open data sources to train the facial recognition algorithms and post-process the software output.
How did we approach this?
By collecting metadata sources for the relevant partners:
Filenames for Kunstenpunt (Flanders Arts Institute);
For the Flemish Parliament’s archives, we’re using the metadata from photos they’ve created themselves. We’ve also collected descriptions of videos from the meemoo archive system and the Flemish Parliament’s open API. We can use this data to retrieve information about Flemish MPs and parliamentary meetings;
For ADVN (Archive for National Movements), we used descriptions of photos and the (limited) metadata from videos from the meemoo archive system;
For KOERS (Museum of Cycle Racing), we were able to use a batch inventory with limited metadata.
We also used Wikidata for all partners.
But not everything is always rosy: alongside the many benefits of metadata, there are also some challenges to overcome. For example, metadata can have many different sources and therefore come in many different forms. We also encountered lots of spelling errors, abbreviations and inconsistencies, or metadata that was unstructured or not available at all, and had to perform time-consuming manual interventions to resolve this.
We created reference sets. Reference photos are portraits of people we can identify based on the metadata sources collected. We were able to identify the people on the photos because we know who’s been a Flemish MP or involved in a production. We used this as our basis for collecting the portrait photos, which was partly done manually and partly automated using APIs from open data sources such as Wikimedia Commons. Institutions that manage collections, such as Amsab (Institute for Social History), KADOC (Documentation and Research Centre on Religion, Culture and Society from KU Leuven) and Liberas (Liberal Archive) supplied us with their own reference sets. Note: a lack of metadata meant it wasn’t always possible to create a full reference set for all collections.
IDLab at Ghent University developed a face recognition pipeline and validation tool, a combination of software that makes it possible to identify people on photos. All the photos were processed through the pipeline, and the relevant partners validated the results. We then used a similarity score assigned by the software to verify whether the identifications were correct. This score indicates the level of similarity between one face and the other faces in the same cluster. If this is score was above 50%, there was a 94% chance that the name provided was correct. We therefore considered this 50% threshold to be the value for automatically assuming identifications to be correct in case partners did not have time for manual validation.
Frequently occurring faces that were not correctly recognised by the facial recognition workflow were grouped together in a cluster. These faces were then identified manually by the partners before the photos could be added to the reference set.
FAME study days: ethical aspects and more
Technologies such as facial recognition need to be treated with caution. We took this into account in our project, and made sure we heeded specialist legal advice, among other things. We also have a legal obligation to process the personal data of the people depicted responsibly. In January 2022, we organised a first FAME study day where we delved deeper into the legal and ethical aspects of facial recognition technology.
The second FAME study day took place a month later, when we looked in more detail at scaling up the infrastructure for automatic metadata creation.
The third and final FAME study day took place in March 2022 and focused on the impact of facial recognition technology on collection registrations.
The results from the FAME project are very positive. Find more details here soon! In short, we concluded that semi-automated facial recognition can have a positive impact on metadata creation, registration and enrichment. Including newly created metadata in the management systems will increase the findability and searchability of the content. There are also some other points for attention, e.g. integrating the metadata, creating good reference sets and (manual) validation all still take up a lot of time. Want to learn more about our findings?