Fusion of Infrared and 3D Heatmap Volume Representation of Skeleton Data for Human Activity Recognition

Abstract

Human Activity Recognition (HAR) is a fundamental component of intelligent monitoring systems in healthcare, rehabilitation, and assisted living, where accurate recognition of fine-grained medical actions is often challenged by subtle motion variations, illumination changes, and background clutter. This research investigates a multimodal HAR framework that integrates infrared (IR) video cues with skeleton- based motion representations to improve recognition reliability for medical-condition activities. A curated subset of the NTU RGB+D dataset is constructed using nine medical action classes (A41–A49: sneeze/cough, staggering, falling down, headache, chest pain, back pain, neck pain, nausea/vomiting, and fan self), comprising 1134 paired IR–skeleton samples with a defined training and validation protocol. The skeleton stream adopts PoseConv3D-style 3D heatmap-volume representations and is trained under two encodings (joint-based and limb-based). The IR stream is trained using an R(2+1)D-18 spatio-temporal backbone, and an ablation study demonstrates that full- frame IR training exhibits severe overfitting, whereas skeleton-guided subject-centric cropping improves generalization and reduces computational redundancy. Multimodal integration is performed through score-level fusion on a paired overlap validation subset, yielding improved recognition accuracy compared to single-modality baselines. Experimental results show strong skeleton-only performance and further gains through IR integration, with the best multimodal setting achieving 96.79% Top-1 and 100% Top-5 accuracy on the paired validation subset. The findings confirm that combining robust volumetric pose cues with appropriately processed infrared features improves discrimination among visually and kinematically confusable medical actions, providing an effective direction for multimodal HAR in healthcare-oriented environments.