Combining 3D Human Pose Estimation and IMU Sensors for Human Identification and Tracking in Multi-Person Environments

Human pose estimation (HPE) based on deep neural networks (DNN) aims to predict the poses of human body in videos without needing markers. One of the main limitations in its applicability is consistently identifying and tracking the keypoints of an individual in multi-person scenarios. Despite various solutions based on image analysis being attempted, challenges such as model accuracy, occlusions, or individuals exiting the camera’s field of view often result in the loss of the association between humans and their keypoints across video frames. In this article, we propose a human identification and tracking methodology in multi-person environments based on data fusion between HPE software and wearable IMU sensors. We demonstrate how to align the data generated by these two sensor categories (camera-based HPE and IMUs) and assess the alignment between each skeleton of keypoints and IMU pair using a scoring system. Additionally, we illustrate how to combine different metrics, such as orientation, acceleration, and velocity, to address alignment problems caused by inaccuracies in sensor data.