3D SKELETON CONSTRUCTION FROM MULTIPLE CAMERA VIEWS FOR QUANTIFYING GAIT PARAMETERS

Abstract

Research has shown that human gait characteristics permit inference with respect to different personal and health characteristics and can thus be used as a diagnostic tool. To do this automatically it is important to be able to extract them from senor information. This thesis work is aimed at doing this from multiple camera views and for this focuses on construction of a 3D body skeleton from multiple viewpoint video (MVV) and then quantifying a number of gait characteristics such as swing time, step time, cadence, stride length, single support, or double support. The method introduced here uses a marker-less approach over marker-based approach for skeleton extraction because it significantly reduces preparation time as well as the equipment cost compared to marker-based techniques. The drawback with the marker-less approach is that the processing time is longer because a body model needs to be created. The solution proposed in this thesis attempts to reduce the processing time by using transfer learning utilizing pre-trained deep learning models and then solving inverse projection problem to get the 3D body skeleton. The extracted visual and 3D skeleton data, along with IMU data from different parts of body is then analyzed to quantify gait characteristics. In particular, three Neural Networks were developed and trained to quantify gait characteristics of human body, one each for sensors data, 2D visual body data and 3D body skeleton data. These Neural Networks were trained to classify whether the human body is in a Single support (left leg), Single support (right leg) or Double support phase. The performance of the networks were evaluated against ground truth and the performance based on the different sensor sets were compared. Among the three Neural Networks the classification accuracy using IMU data was better than using both 2D and 3D skeleton data. Among the visually derived models, the accuracy using 3D skeleton data was better than for 2D visual data. In addition, gait characteristics like Gait Velocity, Step time, Stride length, Stride time, Swing time were extracted from the 3D body skeleton and the performance was validated for multiple individuals.