Effective Context Selection in LLM-Based Leaderboard Generation

An Empirical Study

verfasst von: Salomon Kabongo, Jennifer D’Souza, Sören Auer
Abstract: This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.
Organisationseinheit(en): Forschungszentrum L3S
Externe Organisation(en): Technische Informationsbibliothek (TIB) Leibniz-Informationszentrum Technik und Naturwissenschaften und Universitätsbibliothek
Typ: Aufsatz in Konferenzband
Seiten: 150-160
Anzahl der Seiten: 11
Publikationsdatum: 20.09.2024
Publikationsstatus: Veröffentlicht
Peer-reviewed: Ja
ASJC Scopus Sachgebiete: Theoretische Informatik, Allgemeine Computerwissenschaft
Elektronische Version(en): https://doi.org/10.1007/978-3-031-70242-6_15 (Zugang: Geschlossen)

BibTeX

@inproceedings{2ade8ea55580423cb8c7f6de5411d9fb,
title = "Effective Context Selection in LLM-Based Leaderboard Generation: An Empirical Study",
abstract = "This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.",
author = "Salomon Kabongo and Jennifer D{\textquoteright}Souza and S{\"o}ren Auer",
note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 29th International Conference on Natural Language and Information Systems, NLDB 2024 ; Conference date: 25-06-2024 Through 27-06-2024",
year = "2024",
month = sep,
day = "20",
doi = "10.1007/978-3-031-70242-6_15",
language = "English",
isbn = "9783031702419",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Science and Business Media Deutschland GmbH",
pages = "150--160",
editor = "Amon Rapp and {Di Caro}, Luigi and Farid Meziane and Vijayan Sugumaran",
booktitle = "Natural Language Processing and Information Systems",
address = "Germany",
}