A team from the Computer Vision Center (CVC) and the University of Barce­lona has evaluated the accuracy and bias in gender and skin colour of automatic face recog­nition algorithms tested with real world data. Although the top solutions exceed the 99.9 % of accuracy, researchers have detected some groups that show higher false positive or false negative rates.

Face recognition has been routinely used by both private and govern­mental organi­zations worldwide. Automatic face recognition can be used for legi­timate and bene­ficial purposes (e.g. to improve security) but at the same time its power and ubiquity increases a potential negative impact that unfair methods can have on society (e.g. discri­mination against ethnic minorities). A necessary, albeit not suffi­cient, condition for a legi­timate deployment of face recognition algorithms is the equal accuracy for all demo­graphic groups.

With this purpose in mind, researchers from the Human Pose Recovery and Behavior Analysis Group at the Computer Vision Center (CVC), University of Barcelona (UB), led by Sergio Escalera, organized a challenge within the European Con­ference of Computer Vision (ECCV) 2020. The results evaluated the accuracy of the submitted algorithms by the partici­pants on the face verification task in the presence of other con­founding attributes. The challenge was a success, since “it attracted 151 parti­cipants, who made more than 1,800 submissions in total, exceeding our expec­tations regarding the number of parti­cipants and submissions”, explained Sergio Escalera, also member of the Institute of Mathematics of the UB.

The parti­cipants used a not balanced image dataset, which simulates a real world scenario where AI based models are supposed to be trained and evaluated on imbalanced data – considerably more white males than dark females. In total, they worked with 152,917 images from 6,139 iden­tities. The images were annotated for two protected attributes: gender and skin colour; and five legi­timate attri­butes: age group (0-34, 35-64, 65+), head pose (frontal, other), image source (still image, video frame), wearing glasses and a bounding box size.

The obtained results were very promising. Top winning solu­tions exceeded 99.9 % of accuracy while achieving very low scores in the proposed bias metrics, “which can be considered a step toward the development of fairer face recog­nition methods”, expounded Julio C. S. Jacques Jr., researcher at the CVC and at the Open University of Catalonia. The analysis of top 10 teams showed higher false positive rates for females with dark skin tone and for samples where both indi­viduals wear glasses. In contrast, there were higher false negative rates for males with light skin tone and for samples where both individuals were aged 35 and below. Also, it was found that in the dataset indi­viduals younger than 35 wear glasses less often than older individuals, resulting in a combination of effects of these attributes.

“This was not a surprise, since the adopted dataset was not balanced regarding the different demo­graphic attri­butes. However, it shows that overall accuracy is not enough when the goal is to build fair face recognition methods, and that future works on the topic must take into account accuracy and bias miti­gation together”, concluded Julio C. S. Jacques Jr.. (Source: U. Barcelona)

Reference: T. Sixta et al.: FairFace Challenge at ECCV 2020: Analyzing Bias in Face Recognition, Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science 12540, Springer (2021); DOI: 10.1007/978-3-030-65414-6_32

Link: Computer Vision Center CVC, University of Barcelona, Barcelona, Spain