FAME: facial recognition as a tool for metadata creation

Challenge

Digital photo and video collections managed by cultural organisations often lack good-quality content metadata. There are various reason for this:

• describing these collections manually is very time-consuming;

• it often involves large collections which aren’t part of the organisation’s main collection, and so aren’t a priority;

• the nature of the content means there are additional challenges: you need to actually watch videos to be able to describe the people appearing in it – which is very time-consuming;

• videos can be difficult or impossible to play if they haven’t been digitised yet, making it impossible to describe their content.

Limited metadata makes these digital collections difficult to find and search through, so they are harder to consult and re-use. As part of the FAME project, for which we received a project grant from the Flemish Minister for Culture, Jan Jambon, at the start of 2021, we therefore developed best practices for using (semi-)automated facial recognition to identify people in photos and videos. We also investigated how to use existing metadata to improve facial recognition accuracy.

Our role

Metadata and linked (open) data are two of the five knowledge domains we have identified at meemoo. The FAME pilot projects served as preparation for larger-scale applications – such as the GIVE metadata project – for facial recognition technology aimed at helping with metadata creation and enrichment in the meemoo archive system.

Following on from our previous collaborative projects on visual recognition with FOMU (Photo Museum of Antwerp) and MoMu (Fashion Museum of Antwerp), we researched possibilities for a specific type of visual recognition – i.e. facial recognition – together with several partners.

We ran three pilot projects as part of the FAME project:

• pilot project for performing artists in collaboration with Kunstenpunt (Flanders Arts Institute)

  • This involved us taking 8,300 photos of productions and 700 (group) portraits alongside several videos;

• pilot project for sports people in collaboration with KOERS (Museum of Cycle Racing)

  • We used a selection of 115,000 photo negatives from Maurice Terryn for the period 1967-2006 for this;

• pilot project for politicians and activists in collaboration with ADVN (Archive for National Movements) and the Flemish Parliament Archive

  • We based this project on 8,300 digitised photos from the ADVN magazine ‘WIJ’ together with some digitised videos from VNOS;

  • We also used 5,300 photos from the Flemish Parliament and several videos of Flemish Parliament sessions from the periods 1992-1994 and 2001-2021 from the Flemish Parliament’s archive.

IDLab from Ghent University was our technical partner in each of these projects.

Approach

To prepare for the FAME project, Liesbeth Bogaert worked on an internship with us in the summer of 2020. She was a student of Industrial Computer Science Engineering at Ghent University, had just completed the third year of her bachelor’s degree, and was supervised during her work placement at meemoo by meemoo colleagues Nastasia and Miel along with Prof. Steven Verstockt from Ghent University. She worked with a set of around 9,100 digitised photos and 19,500 born-digital photos from Flanders Arts Institute.

Metadata-driven facial recognition

We used metadata-driven facial recognition in this project. Then, in an attempt to further improve and enrich our facial recognition results, we also used reference sets to investigate how we could use our content partners’ metadata and linked open data sources to train the facial recognition algorithms and post-process the software output.

How did we approach this?

1. By collecting metadata sources for the relevant partners:

   • Filenames for Kunstenpunt (Flanders Arts Institute);

   • For the Flemish Parliament’s archives, we’re using the metadata from photos they’ve created themselves. We’ve also collected descriptions of videos from the meemoo archive system and the Flemish Parliament’s open API. We can use this data to retrieve information about Flemish MPs and parliamentary meetings;

   • For ADVN (Archive for National Movements), we used descriptions of photos and the (limited) metadata from videos from the meemoo archive system;

   • For KOERS (Museum of Cycle Racing), we were able to use a batch inventory with limited metadata.

We also used Wikidata for all partners.

2. We created reference sets. Reference photos are portraits of people we can identify based on the metadata sources collected. We were able to identify the people on the photos because we know who’s been a Flemish MP or involved in a production. We used this as our basis for collecting the portrait photos, which was partly done manually and partly automated using APIs from open data sources such as Wikimedia Commons. Institutions that manage collections, such as Amsab (Institute for Social History), KADOC (Documentation and Research Centre on Religion, Culture and Society from KU Leuven) and Liberas (Liberal Archive) supplied us with their own reference sets. Note: a lack of metadata meant it wasn’t always possible to create a full reference set for all collections.
   
   

3. IDLab at Ghent University developed a face recognition pipeline and validation tool, a combination of software that makes it possible to identify people on photos. All the photos were processed through the pipeline, and the relevant partners validated the results. We then used a similarity score assigned by the software to verify whether the identifications were correct. This score indicates the level of similarity between one face and the other faces in the same cluster. If this is score was above 50%, there was a 94% chance that the name provided was correct. We therefore considered this 50% threshold to be the value for automatically assuming identifications to be correct in case partners did not have time for manual validation.

Frequently occurring faces that were not correctly recognised by the facial recognition workflow were grouped together in a cluster. These faces were then identified manually by the partners before the photos could be added to the reference set.