University of Toronto logo

Multimedia Lab

The Edward S. Rogers Dept. of Electrical and Computer Engineering


Haiping Lu
Mohammad Shahin Mahanta

Prof. K. N. Plataniotis
Prof. A. N. Venetsanopoulos

Jie Wang
Juwei Lu

Learning for Biometric Signal Recognition

Last Update: June 11, 2009

Related News:

  • June 10, 2009: Three papers co-authored by Multimedia Laboratory researchers have been named "core research papers" in the area of  "MACHINE LEARNING AND FACE RECOGNITION," by Thomson Citation Index. Read more...

» More news...

Biometrics is an important component in security-related applications such as access control, forensic investigation, and identity fraud protection. We focused on the very important problem of feature extraction through subspace learning in biometric systems.

Sub-Project Quick Links

For thousands of years, humans have used visually perceived body characteristics such as face and gait to recognize each other. This remarkable ability of human visual system has inspired researchers to build automated systems to recognize individuals from digitally captured facial images and gait sequences. Face and gait recognition belong to the field of biometrics, a very active area of research in the computer vision and pattern recognition society, mainly motivated from security-related applications. Face and gait are two typical physiological and behavioral biometrics, respectively. Compared with other biometric traits, face and gait have the unique property that they facilitate human recognition at a distance, which is extremely important in surveillance applications. Moreover, their unintrusive nature leads to high collectability and acceptability, making them very promising technologies for wide deployments. The collectability refers to the ease of acquisition for measurement and the acceptability indicates the extent to which people are willing to accept the use of a particular biometric identifier in their daily lives.

Face/Gait Recognition system diagram

The research on biometrics in our lab focuses on face recognition and gait recognition, where individuals are recognized by their faces and the way they walk, respectively. The figure above depicts a typical face or gait recognition system. By observing a subject in the view, a digital camera captures a digital raw facial image or a digital raw gait video. This image or video is then pre-processed (e.g., filtered to remove noise) to extract a facial image or a gait sequence for feature extraction. In the feature extraction, face or gait features are extracted from the input image or image sequence, and these features are passed to the recognition module, where classifiers are employed to match them with the stored features in the face or gait database and a person is recognized with his/her ID as the output. A review of face and gait recognition is available in the following book chapter:

Haiping Lu, Jie Wang and K.N. Plataniotis, "A Review on Face and Gait Recognition: System, Data and Algorithms", to appear in Advanced Signal Processing Handbook, Second Edition, S. Stergiopoulos, Editor, CRC Press, Boca Raton, Florida.

The research works carried out in our lab have mainly investigated the important problem of feature extraction. There are two general approaches: the model-based approach and the appearance-based approach. We describe our works according to the following different categories.

A. Appearance-Based Face Recognition

Appearance-based face recognition approach processes 2-D facial image as 2-D holistic patterns. The whole face region is the raw input to a recognition system and each face image is commonly represented by a high-dimensional vector consisting of the pixel intensity values in the image, i.e., a point in a high-dimensional vector space.  Thus, face recognition is transformed to a multivariate statistical pattern recognition problem. Although the embedding is high-dimensional, the natural constraints of the face data indicate that the face vectors lie in a lower-dimensional subspace (manifold). The popular subspace learning is such a method to identify, represent, and parameterize this subspace with some optimality criteria.

A.1. Extensions of LDA

The classical LDA has been extended in order to address the small sample size problem in face recognition. In particular, direct LDA is combined with fractional-step LDA and extensive regularization studies have been done.


  1. Juwei Lu, K.N. Plataniotis, A.N. Venetsanopoulos, “Regularization Studies of Linear Discriminant Analysis in Small Sample Size Scenarios with Application to Face Recognition”, Pattern Recognition Letter, vol. 26, issue 2, pp. 181-191, 2005.
  2. Juwei Lu, K.N. Plataniotis, and A.N. Venetsanopoulos, "Regularized Discriminant Analysis For the Small Sample Size Problem in Face Recognition", Pattern Recognition Letter, Vol. 24, Issue 16, Page: 3079-3087, December 2003.
  3. Juwei Lu, K.N. Plataniotis, and A.N. Venetsanopoulos, "Face Recognition Using LDA Based Algorithms", IEEE Transactions on Neural Networks, Vol. 14, No. 1, Page: 195-200, January 2003.

A.2. Kernel-Based Discriminant Learning

The complexity of face pattern distributions poses significant difficulties on linear learning algorithms. Thus, nonlinear, kernel-based algorithms have been developed to handle complexly distributed data. Originating from the well-known support vector machine, the so-called “kernel machine” technique is considered an important tool in the design of nonlinear feature extraction techniques. The premise behind the kernel machine technique is to find a nonlinear mapping from the original input space to a higher dimensional kernel feature space by using a nonlinear function so that patterns are better separated in the kernel space.


  1. Jie Wang, Haiping Lu, K.N. Plataniotis and Juwei Lu, "Gaussian Kernel Optimization for Pattern Classification", Pattern Recognition, to appear.
  2. Jie Wang, K. N. Plataniotis, Juwei Lu and A. N. Venetsanopoulos, “Kernel Quadratic Discriminant Analysis for Small Sample Size Problem”, Pattern Recognition, Vol. 39, Issue 5, pp.1528-1538, 2008.
  3. Juwei Lu, K.N. Plataniotis and A.N. Venetsanopoulos, “Kernel Discriminant Learning with Application to Face Recognition”, in “Support Vector Machines: Theory and Applications”, Lipo WANG, Editors, Springer-Verlag, ISBN: 3-540-24388-7, 2005.
  4. Juwei Lu, K. N. Plataniotis, A. N. Venetsanopoulos and Jie Wang, “An Efficient Kernel Discriminant Analysis Method”, Pattern Recognition, Vol. 38, Issue 10, pp. 1788-1790, 2005.
  5. Juwei Lu, K.N. Plataniotis, and A.N. Venetsanopoulos, "Face Recognition Using Kernel Direct Discriminant Analysis Algorithms" IEEE Transactions on Neural Networks, Vol. 14, No. 1, Page: 117-126, January 2003.

A.3. One-Training-Sample Scenario

In the so-called one-training-sample scenario, there is only one face image per subject available for training. This is a very challenging problem with practical importance. Works have been done to investigate techniques that can handle this scenario effectively.


  1. Jie Wang, K. N. Plataniotis, Juwei Lu and A. N. Venetsanopoulos, “On Solving the  Face Recognition Problem with One Training Sample per Subject”, Pattern Recognition, Vol. 39, Issue 9, pp.1746-1762, 2006.
  2. Jie Wang, K. N. Plataniotis and A. N. Venetsanopoulos, “Selecting Discriminant Eigenfaces for Face Recognition”, Pattern Recognition Letters, Vol.26, Issue 10, pp. 1470-1482, 2005.

A.4. LDA-Style Boosting

Boosting is a very effective general learning technique to improve generalization performance. However, the requirement of weak learners limits its wider use. We have proposed a novel scheme to boosting LDA-style learners for better generalization performance, shown on face recognition tasks.


  1. Juwei Lu, K.N. Plataniotis, A.N. Venetsanopoulos, and Stan Z. Li, “Ensmeble-based Discriminant Learning with Boosting for Face Recognition”, IEEE Trans. on Neural Networks, Vol. 17, No. 1, pp. 166-178, January 2006.

B. Appearance-Based Gait Recognition

Similar to appearance-based face recognition, appearance-based gait recognition approach considers gait as a holistic pattern and uses a full-body representation of a human subject as silhouettes or contours.

Gait video sequences are naturally three-dimensional objects, formally named tensor objects, and they are very difficult to deal with using traditional vector-based learning algorithms. In order to deal with these tensor objects effectively and efficiently, we have developed a framework of multilinear subspace learning so that computation and memory demand are significantly reduced, natural structure and correlation in the original data are preserved, and  more compact and useful features can be obtained.


  1. Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos, "A Taxonomy of Emerging Multilinear Discriminant Analysis Solutions for Biometric Signal Recognition", to appear in Biometrics: Theory, Methods, and Applications, N. Boulgouris, K.N. Plataniotis, and E. Micheli-Tzanakou, Eds., IEEE/Wiley Press.
  2. Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos, "Uncorrelated Multilinear Discriminant Analysis with Regularization and Aggregation for Tensor Object Recognition", IEEE Trans. on Neural Networks, to appear.
  3. Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos, "Uncorrelated Multilinear Principal Component Analysis through Successive Variance Maximization", in Proceedings of the 25th International Conference on Machine Learning (ICML 2008) , Helsinki, Finland, pp. 616-623, July 2008.
  4. Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos, "MPCA: Multilinear Principal Component Analysis of Tensor Objects", IEEE Trans. on Neural Networks, Vol. 19, No. 1, Page: 18-39, Jan. 2008.

C. Model-Based Gait Recognition

Model-based gait recognition approach considers a human subject as an articulated object represented by various body poses. We have proposed a full-body layered deformable model (LDM), inspired by the manually labeled body-part-level silhouettes. The LDM has a layered structure to model self-occlusion between body parts and it is deformable so simple limb deformation is taken into consideration. In addition, it also models shoulder swing. The LDM parameters can be recovered from automatically extracted silhouettes and then used for recognition.


  1. Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos, "A Full-Body Layered Deformable Model for Automatic Model-Based Gait Recognition", EURASIP Journal on Advances in Signal Processing, vol. 2008, Article ID 261317, 13 pages, 2008. doi:10.1155/2008/261317.
  2. Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos, "Coarse-to-Fine Pedestrian Localization and Silhouette Extraction for the Gait Challenge Data Sets", in Proceedings of the IEEE International Conference on Multimedia & Expo (ICME 2006), Toronto, Canada, July 2006.
  3. Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos, "A Layered Deformable Model for Gait Analysis", in Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FGR 2006), Southampton, UK, April 2006.