Developing an Urdu Lemmatize

Abstract

Lemmatization is a process of obtaining root form of a given word. Lemmatizer is an important part of Natural Language Processing (NLP) toolkit and is essential for many NLP systems e.g. Information Retrieval (IR), plagiarism and text reuse detection, Information Extraction (IE), Machine Translation (MT), Word Sense Disambiguation (WSD) etc. Urdu is a widely spoken language in the world, but very less work has been done on developing basic NLP tools for this language, one of them is Urdu lemmatizer. Since Urdu is a morphologically rich language and has words with many inflectional and derivational forms, development of an efficient lemmatizer is a challenging task and it will be useful for many Urdu NLP applications. This research work aims to develop an efficient lemmatizer for Urdu language. For this purpose, we explored two main approaches: (1) dictionary lookup approach and (2) rule based approach. Note that we also tried combinations of these two approaches. For the first approach, we build a large dictionary of 131,539 entries. In this dictionary, 3,026 are exceptional words, 22,181 are multi-word expression, and 18,267 are Named Entities (NEs) for person and 79,418 are Named Entities (NEs) for locations. Remaining 8,647 are variants of most frequent 5,000 Urdu language words. For the second approach, we extracted a total of 215 rules (67 are prefix, 128 are postfixes) from raw Urdu text. We also developed a benchmark test dataset of 1,000 Urdu words to evaluate our proposed approaches for Urdu lemmatization task. The proposed lemmatizer was evaluated using six different approaches: (1) Dictionary Lookup Approach (DLA), (2) Rule Based Approach (RBA), (3) Dictionary (DIC) + Exceptional Word List (EWL) Approach, (4) Rule Based + Exceptional Word List Approach, (5) Rule Based + Final Dictionary Approach and (6) Final Dictionary + Rule Based Approach. Evaluation was carried out using Accuracy measure. Results showed that Dictionary + Rule Based Approach gave highest Accuracy of 82.9%. We also developed a prototype system, which takes an Urdu word as input and outputs its lemma