Enhancing Multimodal Sentiment Analysis for Urdu Using Cross Modal Transfer Learning

Abstract

Urdu, which is spoken by more than 230 million people around the world, is also a case in point, since it is severely underrepresented in the field of natural language processing research. Most of the existing systems used for Urdu sentiment analysis rely solely on the text, overlooks the rich emotional indications in speech e.g., tone, emphasis and prosody. This gap and the inadequacy of datasets or complex code-switching patterns between Urdu and English combined has restricted the accuracy of sentiment analysis in this low-resource language. This research involved creation of multimodal sentiment analysis framework for Urdu by combining text and audio using cross modal transfer learning. We built a dataset of 10,847 aligned text-audio pairs obtained from Facebook, Twitter and from original recordings and annotated in the positive, negative and neutral categories with high overlap between annotators (Cohen's kappa = 0.78). The framework uses multilingual BERT (mBERT) for the encoding of the text and convolutional neural networks for audio feature extraction. An attention based fusion mechanism is a dynamic mechanism that computes weight for modality contribution dependent upon the signal quality. Cross-modality transfer learning - through contrastive loss and knowledge distillation - allows conducting knowledge transfer across modalities. The proposed framework was able to achieve 84.7% accuracy and 83.2% macro F1-score on the held out test set. And it stood a good 21.8% relative improvement from text only mBERT (79.2%) and also 24.0% improvement over traditional SVM baselines (68.3%). Ablation studies validated that cross-lingual transfer had a positive effect of 8.4 percentage points while multimodal integration had a positive effect of 5.5 points and attention-based fusion +3.4 points over early fusion. The model is quite good at processing code-switched content (83.1% accuracy for instances mixing Urdu and English) and low demographic bias with respect to age, gender and regional variations. These findings show that multimodal strategies with transfer learning can significantly reduce the gap in performance for high-resource and low-resource languages. The dataset and methodology provide zero foundation for imitating Urdu NLP held in future to be engaged within research and practical applications in social media monitoring and customer feedback.