Viewpoint Invariant Gesture Recognition And 3D Hand Pose Estimation Using RGB-D

Abstract

The broad application domain of the work presented in this thesis is pattern classifi-cation with a focus on gesture recognition and 3D hand pose estimation. One of the main contributions of the proposed thesis is a novel method for 3D hand pose estimation using RGB-D. Hand pose estimation is formulated as a database retrieval problem. The proposed method investigates and introduces new similarity measures for similarity search in a database of RGB-D hand images. At the same time, towards making 3D hand pose estimation methods more automatic, a novel hand segmentation method is introduced which also relies on depth data. Experimental results demonstrate that the use of depth data increases the discrimination power of the proposed method.On the topic of gesture recognition, a novel method is proposed that combines a well known similarity measure, namely the Dynamic Time Warping (DTW), with a new hand tracking method which is based on depth frames captured by Microsoft's Kinect RGB-Depth sensor. When DTW is combined with the near perfect hand tracker gesture recognition accuracy remains high even in very challenging datasets, as demonstrated by experimental results. Another main contribution of the current thesis is an extension of the proposed gesture recognition system in order to handle cases where the user is not standing fronto-parallel with respect to the camera. Our method can recognize gestures capturedunder various camera viewpoints.At the same time our depth hand tracker is evaluated against one popular open source user skeleton tracker by examining its performance on random signs from a dataset ofAmerican Sign Language (ASL) signs. This evaluation can serve as a benchmark for the assessment of more advanced detection and tracking methods that utilize RGB-D data.The proposed structured motion dataset of (ASL) signs has been captured in both RGB and depth format using a Microsoft Kinect sensor and it will enable researchers to explore body part (i.e., hands) detection and tracking methods, as well as gesture recognition algorithms.