Lexical Sample Word Sense Disambiguation for Urdu Language – Data and Methods

Abstract

In all natural-languages, there exists a range of words that have multiple meanings. Word Sense Disambiguation (WSD) is the process of automatically determine the correct sense of a word in a given context. Thus, this lexical ambiguity is observed as one of the significant and critical issues in natural language processing (NLP). To build, evaluate and compare WSD methods the standard assessment resources are required. To develop benchmark WSD corpora and methods for a broad variety of languages, a range of initiatives are needed. But despite the fact, that there are more than 325 million speakers of Urdu and a larger volume of the digital text of Urdu is available online, the standard is deficient, high-quality big corpus for the WSD task. To overcome that gap, the current work aims to create a novel and large size benchmark corpus for the task of Urdu Lexical Sample WSD with the application of classical machine learning and deep learning approaches on this large corpus. The work comprises manually crafting the 500 most frequent polysemic words with 127 General Urdu terms, 136 terms from News, 72 terms from Religion, 21 terms from Science, 52 terms from Literature, 29 terms from Medical and 63 terms from Finance from UrMono Corpus (a benchmark corpus of Urdu with 95.4 million Urdu tokens) and Urdu Lughat (a sense inventory and standard dictionary of Urdu Language). After the creation of a corpus, an inspection will be conducted on the ambiguous terms for the verification of the WSD system through four deep learning techniques (1) Multi-Layer Perceptron (MLP), (2) Simple Recurrent Neural Network (RNN), (3) Long-Shot-Term-Memory (LSTM), and (4) Bidirectional Long Short Term Memory (Bi-LSTM). Five classical machine learning algorithms will also be tested on the same dataset. A comparison between of results of classical machine learning and deep learning techniques reflects that machine learning models outperform the deep learning techniques.