Effective Context Selection in LLM-Based Leaderboard Generation

An Empirical Study

authored by: Salomon Kabongo, Jennifer D’Souza, Sören Auer
Abstract: This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.
Organisation(s): L3S Research Centre
External Organisation(s): German National Library of Science and Technology (TIB)
Type: Conference contribution
Pages: 150-160
No. of pages: 11
Publication date: 20.09.2024
Publication status: Published
Peer reviewed: Yes
ASJC Scopus subject areas: Theoretical Computer Science, General Computer Science
Electronic version(s): https://doi.org/10.1007/978-3-031-70242-6_15 (Access: Closed)

BibTeX

@inproceedings{2ade8ea55580423cb8c7f6de5411d9fb,
title = "Effective Context Selection in LLM-Based Leaderboard Generation: An Empirical Study",
abstract = "This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.",
author = "Salomon Kabongo and Jennifer D{\textquoteright}Souza and S{\"o}ren Auer",
note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 29th International Conference on Natural Language and Information Systems, NLDB 2024 ; Conference date: 25-06-2024 Through 27-06-2024",
year = "2024",
month = sep,
day = "20",
doi = "10.1007/978-3-031-70242-6_15",
language = "English",
isbn = "9783031702419",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Science and Business Media Deutschland GmbH",
pages = "150--160",
editor = "Amon Rapp and {Di Caro}, Luigi and Farid Meziane and Vijayan Sugumaran",
booktitle = "Natural Language Processing and Information Systems",
address = "Germany",
}