Image Caption Generation for Urdu Language – Data and Methods

Abstract

Image Captioning (IC) generates a suitable caption for the given query image. The focus of IC is understanding the contextual meaning of visual contents in an image and then describing them in semantically and syntactically correct description in the natural language. The most salient part of an image is described in a sentence to increase the visual understanding of people. It has many applications like navigation guidance for blind persons, image-based optimized search engines, visual context understanding, early babyhood education, image description in an e-commerce site, and image indexing. Previously most of the work done on Image Captioning was in English and other high resource languages like Chinese, Arabic, and Hindi, etc. To the best of our knowledge, the Urdu language has not been explored previously for Urdu Image Captioning (UIC). The primary reason for this is the unavailability of benchmark corpora for UIC. So, the foremost aim of this research is to develop a benchmark Image Captioning corpus for the Urdu language (known as UIC-21 corpus). The proposed corpus is based on the Fliker-8k English corpus. It is developed by translating the English image caption into Urdu image captions using a semi-automatic translation approach. In this approach, automated translation tools are used to automatically translate the image captions in Urdu language and then a manual inspection and correction approach is used to improve the quality of the captions. To demonstrate how UIC systems can be developed and evaluated on our proposed UIC-21 corpus, we applied deep learning techniques (CNN for images and RNN, LSTM, and GRU for Urdu textual data) and state-of-the-art transfer learning techniques (InceptionV3 with ImageNet weights and VGG16 with ImageNet weights for images, and pre-trained word2vec urduvec and W2V models for textual data) for UIC systems, which are further evaluated using BELU-1, BLEU-2, BLEU-3, and BLEU-4. Their experimental results show that the transfer learning InceptionV3-GRU with the pretrained word embedding method achieved the highest BLEU-1 score of 0.60 on our proposed UIC-21 corpus