Probing BERT for Ranking Abilities

verfasst von

Jonas Wallat, Fabian Beringer, Abhijit Anand, Avishek Anand

Abstract

Contextual models like BERT are highly effective in numerous text-ranking tasks. However, it is still unclear as to whether contextual models understand well-established notions of relevance that are central to IR. In this paper, we use probing, a recent approach used to analyze language models, to investigate the ranking abilities of BERT-based rankers. Most of the probing literature has focussed on linguistic and knowledge-aware capabilities of models or axiomatic analysis of ranking models. In this paper, we fill an important gap in the information retrieval literature by conducting a layer-wise probing analysis using four probes based on lexical matching, semantic similarity as well as linguistic properties like coreference resolution and named entity recognition. Our experiments show an interesting trend that BERT-rankers better encode ranking abilities at intermediate layers. Based on our observations, we train a ranking model by augmenting the ranking data with the probe data to show initial yet consistent performance improvements (The code is available at github.com/yolomeus/probing-search/ ).

Organisationseinheit(en)

Forschungszentrum L3S

Externe Organisation(en)

Delft University of Technology

Typ

Aufsatz in Konferenzband

Seiten

255-273

Anzahl der Seiten

Publikationsdatum

17.03.2023

Publikationsstatus

Veröffentlicht

Peer-reviewed

ASJC Scopus Sachgebiete

Theoretische Informatik, Allgemeine Computerwissenschaft

Elektronische Version(en)

https://doi.org/10.1007/978-3-031-28238-6_17 (Zugang: Geschlossen)

BibTeX

@inproceedings{320363a182464e93a5b50745cbfd3298,
title = "Probing BERT for Ranking Abilities",
abstract = "Contextual models like BERT are highly effective in numerous text-ranking tasks. However, it is still unclear as to whether contextual models understand well-established notions of relevance that are central to IR. In this paper, we use probing, a recent approach used to analyze language models, to investigate the ranking abilities of BERT-based rankers. Most of the probing literature has focussed on linguistic and knowledge-aware capabilities of models or axiomatic analysis of ranking models. In this paper, we fill an important gap in the information retrieval literature by conducting a layer-wise probing analysis using four probes based on lexical matching, semantic similarity as well as linguistic properties like coreference resolution and named entity recognition. Our experiments show an interesting trend that BERT-rankers better encode ranking abilities at intermediate layers. Based on our observations, we train a ranking model by augmenting the ranking data with the probe data to show initial yet consistent performance improvements (The code is available at https://github.com/yolomeus/probing-search/ ).",
author = "Jonas Wallat and Fabian Beringer and Abhijit Anand and Avishek Anand",
note = "Funding Information: Acknowledgements. This research was (partially) funded by the Federal Ministry of Education and Research (BMBF), Germany under the project LeibnizKILabor with grant No. 01DD20003.; 45th European Conference on Information Retrieval, ECIR 2023 ; Conference date: 02-04-2023 Through 06-04-2023",
year = "2023",
month = mar,
day = "17",
doi = "10.1007/978-3-031-28238-6_17",
language = "English",
isbn = "9783031282379",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer",
pages = "255--273",
editor = "Jaap Kamps and Lorraine Goeuriot and Fabio Crestani and Maria Maistro and Hideo Joho and Brian Davis and Cathal Gurrin and Annalina Caputo and Udo Kruschwitz",
booktitle = "Advances in Information Retrieval",
}