Evaluating Knowledge Preferences in Question Answering via Bloom’s Taxonomy

Abstract

Generative Pre-trained Language Models (PLMs) can struggle with challenges such as hallucinations, particularly in domain-specific usage, where largely specific knowledge is required to correctly answer a user's prompt. A proposed solution is the inclusion of external information (i.e., not part of the PLMs trained data set). We investigate the following questions: How important is the externally provided information? How essential is the knowledge encoded within the PLM itself? And to what extent do these two sources provide complementary information? The method we apply is an automated pipeline for synthetic Question Answer (QA) generation and evaluation, where the source of the QAs is a textbook on database design. To generate QAs with increasing complexity, we make use of Bloom's taxonomy, a system classifying the cognitive dimensions of human learning processes. We find that external information improves response quality (5%), yet overlap exists between the parametric knowledge of a PLM and the external information (75%).