Improving Set-Based Face Recognition



Journal Title

Journal ISSN

Volume Title



A face image set is a group of face images from the same person. In set-based face recognition systems, face image sets are employed either in the gallery, probe, or both for comparisons. Compared with a single face image, an image set provides more information; hence, better performance is expected. However, it also brings a lot of challenges and remains an open problem in real-life scenarios. First, there are large variations within an image set (e.g., poses, expressions, and occlusions). Second, the number of images varies for different image sets. Third, there may be outliers in a set due to misdetection or mistracking. Fourth, the computation and storage costs are very high, especially for large-scale image sets. The goal of this dissertation is to design effective and efficient algorithms in template generation and matching that can represent identity information and take advantage of the within-set variations. The first contribution is a set-based prototype and metric learning algorithm (SPML) that generates compact templates and robust similarity measurements for set-to-set matching. The second contribution is a confidence-driven network (CDN) to quantify the confidence level of images in a set and enhance the point-to-set matching. The third contribution is a confidence prediction network (CPN) that can serve as an add-on module to enhance the performance of a sample-based face recognition system for set-based face recognition tasks. The fourth contribution is an attention-based recursive binary embedding (ARBE) algorithm to extract binary templates for face image sets. The proposed algorithms achieved significant improvements when compared with previous advances.



Image set classification, Face recognition, Performance prediction, Attention mechanism


Portions of this document appear in: Leng, Mengjun, Panagiotis Moutafis, and Ioannis A. Kakadiaris. "Joint prototype and metric learning for image set classification: Application to video face identification." Image and Vision Computing 58 (2017): 204-213. And in: Leng, Mengjun, and Ioannis A. Kakadiaris. "Confidence-Driven Network for Point-to-Set Matching." In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3414-3420. IEEE, 2018. And in: Leng, Mengjun, and Ioannis A. Kakadiaris. "Recursive Binary Template Embedding for Face Image Sets." In 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1-8. IEEE, 2018.