Enhancing Automated Question Tagging: Integrating Large Language Models with Traditional Retrieval Methods

Abstract

When people ask questions on platforms like Stack Overflow or Quora, the system needs to automatically assign relevant tags to help others find and answer those questions quickly. This is called automatic question tagging and is quite essential since it directly influences the efficiency with which the user can receive help. The majority of existing tagging systems just examine the words of the question itself and the tags accessible in the database. They are deprived of expansive real-world information that can guide them in making superior tagging decisions. Large language models such as GPT-4 are very good at context, real-world understanding, and this may be invaluable in this case. The issue is, these models cannot be confident in selecting tags out of a list of predefined tags of a particular site that they may propose tags that do not exist in the system at all. To address this problem, I created EAQTILLM (Enhancing Automated Question Tagging Integrating Large Language Model). My method is in two steps: I will first run a tagging model that will give me a shortlist of possible tags based on the database of the website. I then run the question through a large language model with prompts that are designed to explore the question in more depth and choose the most suitable tags using that shortlist. By doing this we get the advantage of both worlds the ability of the LLM to understand and the restriction of just being able to pick out of valid and existing tags. I applied EAQTILLM (Test) to two real-life datasets, achieving a precision of 0.4952, recall of 0.6667, and accuracy of 0.9842, which demonstrated remarkable performance. The algorithm was always ahead of current state-of-the-art algorithms, demonstrating that the combination of the traditional ones and large language models is a good direction to enhance automatic question tagging in online communiti