M.Phil / MS

Now showing 1 - 2 of 2

Developing a Urdu Lemmatizer Using Dictionary Look-up Approach
(Library Information Services, COMSATS University Islamabad, Lahore Campus, 2020) Saima Shaukat; FA18-RCS-009; LHR TP 5987; Dr. Rao Muhammad Adeel Nawab
Lemmatization is the process of finding the root form of a word. Lemmatization is used as preprocessing step for most of the NLP applications. Lemmatization is im portant part of NLP applications which involve the use of root form instead of using derivational forms of words. Text mining, information retrieval systems, document clustering, search engines and text categorizations are few of the applications where lemmatization process is used. Urdu is a widely spoken language in the world, but very less work has been done on developing basic NLP tools for this language, one of them is Urdu lemmatizer. Since Urdu is a morphologically rich language and has words with many inflectional and derivational forms, development of an efficient lemmatizer is a challenging task and it will be useful for many Urdu NLP applications. These challenges help us to choose the customized and more appropriate approach for Urdu lemmatization which is dictionary-based approach. The main aim of this research work is to develop an Urdu lemmatizer. To develop the efficient Urdu lemmatizer, we used dictionary look-up approach. Majority of exist ing lemmatizers are of in English and other European languages. Most of the existing lemmatizers are rule based and also the information about parts of speech tags is ig nored. Development of large dictionaries for Urdu Language is also missing in existing work. To fulfill this gap, the main goals of the study are: (1) Develop a large bench mark corpus for the Urdu language and (2) Explore the relationship between parts of speech tag and lemmatizer (3) Developed the standard approach for Urdu lemmatizer. The first major contribution of this study is the development of a large benchmark Urdu dictionary. The proposed corpus was developed in the following steps: (1) In the first step we collected data from two different sources: urmono corpus and Wikipedia dump (2) Preprocess and tokenize the collected data (3) extract the frequency of each word using python script (4) Selection of most frequent words (5) Assign parts of speech ix tags to selected words (6) manually annotate data (assign lemma to each word) (7) De velop a benchmark corpus and save in a standard csv format. Second major contribution of this research work is exploring the relationship be tween POS tag and lemma of a word. Train the POS tagger on collected Wikipedia dump and then used in the development of Urdu lemmatizer because user can enter input with or without POS tags. If user will enter input without POS tag, tagger will assign most frequent used tag to the word before generating lemma of a word. As a third major contribution of this research work, proposed a dictionary-based approach for Urdu lemmatizer. As the Urdu is morphological rich language; it also includes many Persian, Turkish and Arabic words. So, for the efficient lemmatizer, dictionary-based approach is more suitable. The evaluation was carried out using two techniques (1) With-POS-DLA (2) Without-POS-DLA. We achieved accuracy of 66.79% and 76.44% respectively. The best result is achieved by using Without-POS DLA technique.
Author Region Identification for Urdu Language
(Library Information Services, COMSATS University Islamabad, Lahore Campus, 2020) Rehman Akram Chughtai; FA18-RCS-007; LHR TP 6407; Dr. Rao Muhammad Adeel Nawab
Author region identification is a subtask of author profiling which intends to identify the region of the author, based on the written text. Identification of author region may improve content recommendation, security, reduction in cyber-crimes due to its different applications in fake profile detection, content recommendation, sales and marketing, and forensic linguistics. Lot of work has been done on several author profiling tasks in English, Arabic and other European languages but no significant work found on author region identification task using Urdu language. To fill this gap, we proposed a large benchmark corpus and methods to predict author’s region using Urdu language articles. The proposed corpus consists of 12,100 Urdu articles written by authors from different regions. These articles are categorized into three different regions i.e. Pakistan, India and others. There are 5000 Urdu articles for Pakistan region, 5000 Urdu articles for India region and 2100 articles for Others category. For each article, article web links, author name and domain of each article is mentioned for future research purposes. To demonstrate how our proposed corpus can be used for the development and evaluation of author region identification for Urdu, I applied different classical machine learning and deep learning approaches. For classical machine learning, I used word n-grams technique for feature extractions and then applied classifiers i.e. Logistic Regression, SVM, Naïve Bayes, Random Forest, Adaboost and Gradient boosting for multi classification task. For deep learning algorithms, I applied CNN, RNN, LSTM, Bi-LSTM, and GRU on proposed corpus. Evaluation was carried out using accuracy, weighted precision, weighted recall and weighted F1-measure as evaluation measures. Experiments show that the best results with accuracy of 85 % is achieved by classical machine learning approach using SVM classifier on 3000 features with word n-gram [n ranges from 1 to 3], while for deep learning approach CNN with 1 layer outclass other classifiers with accuracy of 96 %. Deep learning CNN method performed best among all classical machine learning and deep learning methods for author region identification task on proposed Urdu articles corpus.

M.Phil / MS

Browse

Filters

Settings

Sort By

Results per page

Search Results