Developing a Gold Standard Benchmark Corpus and Methods for Multi-label Emotion Classification for Urdu Language

Abstract

Emotion Classification (EC) involves detecting the emotional tone conveyed by text, while Multi-Label Emotion Classification (MLEC) aims to assign relevant emotion labels that accurately reflect the author's state of mind. This research focuses on MLEC in Urdu text. In previous studies, only one work has been done for the problem of MLEC, including the development of a dataset based on Urdu Tweets using six emotions [5]. However, the problem of MLEC has not been explored for 12 emotions and diverse genres. To fulfill this research gap, this research study has developed a large benchmark corpus of 16,011 Urdu reviews, manually annotated with 12 emotions and gathered from 9 different genres. Furthermore, this research study has Developed, Applied, Evaluated, and Compared Classical Machine Learning, Deep Learning and Transfer Learning based methods. Whereas in Transfer Learning methods, we have implied 6 distinct sentence transformers (xlm-r-100langs-bert-base-nli-stsb-mean-tokens, xlm-r-100langs-bert-base nli-mean-tokens, gtr-t5-large, all-mpnet-base-v2, distiluse-base-multilingual-cased-v2, and LaBSE), Also the Combined Feature-based approach is applied which averages the features of all the columns of the sentence transformer model and then the Machine Learning models are applied. The proposed transformers-based approach outperforms with an F1 score of 0.4562 using the ‘xlm-r-100langs-bert-base-nli-stsb-mean-tokens’ sentence transformer