Transfer learning for molecular property predictions from small datasets

authored by: Thorren Kirschbaum, Annika Bande
Abstract: Machine learning has emerged as a new tool in chemistry to bypass expensive experiments or quantum-chemical calculations, for example, in high-throughput screening applications. However, many machine learning studies rely on small datasets, making it difficult to efficiently implement powerful deep learning architectures such as message passing neural networks. In this study, we benchmark common machine learning models for the prediction of molecular properties on two small datasets, for which the best results are obtained with the message passing neural network PaiNN as well as SOAP molecular descriptors concatenated to a set of simple molecular descriptors tailored to gradient boosting with regression trees. To further improve the predictive capabilities of PaiNN, we present a transfer learning strategy that uses large datasets to pre-train the respective models and allows us to obtain more accurate models after fine-tuning on the original datasets. The pre-training labels are obtained from computationally cheap ab initio or semi-empirical models, and both datasets are normalized to mean zero and standard deviation one to align the labels’ distributions. This study covers two small chemistry datasets, the Harvard Organic Photovoltaics dataset (HOPV, HOMO-LUMO-gaps), for which excellent results are obtained, and the FreeSolv dataset (solvation energies), where this method is less successful, probably due to a complex underlying learning task and the dissimilar methods used to obtain pre-training and fine-tuning labels. Finally, we find that for the HOPV dataset, the final training results do not improve monotonically with the size of the pre-training dataset, but pre-training with fewer data points can lead to more biased pre-trained models and higher accuracy after fine-tuning.
Organisation(s): Institute of Inorganic Chemistry
External Organisation(s): Helmholtz-Zentrum Berlin für Materialien und Energie (HZB)
Type: Article
Journal: AIP Advances
Volume: 14
No. of pages: 9
ISSN: 2158-3226
Publication date: 10.2024
Publication status: Published
Peer reviewed: Yes
ASJC Scopus subject areas: General Physics and Astronomy
Electronic version(s): https://doi.org/10.48550/arXiv.2404.13393 (Access: Open)
https://doi.org/10.1063/5.0214754 (Access: Open)

BibTeX

@article{1221151c67ac44be989f65b361f14611,
title = "Transfer learning for molecular property predictions from small datasets",
abstract = "Machine learning has emerged as a new tool in chemistry to bypass expensive experiments or quantum-chemical calculations, for example, in high-throughput screening applications. However, many machine learning studies rely on small datasets, making it difficult to efficiently implement powerful deep learning architectures such as message passing neural networks. In this study, we benchmark common machine learning models for the prediction of molecular properties on two small datasets, for which the best results are obtained with the message passing neural network PaiNN as well as SOAP molecular descriptors concatenated to a set of simple molecular descriptors tailored to gradient boosting with regression trees. To further improve the predictive capabilities of PaiNN, we present a transfer learning strategy that uses large datasets to pre-train the respective models and allows us to obtain more accurate models after fine-tuning on the original datasets. The pre-training labels are obtained from computationally cheap ab initio or semi-empirical models, and both datasets are normalized to mean zero and standard deviation one to align the labels{\textquoteright} distributions. This study covers two small chemistry datasets, the Harvard Organic Photovoltaics dataset (HOPV, HOMO-LUMO-gaps), for which excellent results are obtained, and the FreeSolv dataset (solvation energies), where this method is less successful, probably due to a complex underlying learning task and the dissimilar methods used to obtain pre-training and fine-tuning labels. Finally, we find that for the HOPV dataset, the final training results do not improve monotonically with the size of the pre-training dataset, but pre-training with fewer data points can lead to more biased pre-trained models and higher accuracy after fine-tuning.",
author = "Thorren Kirschbaum and Annika Bande",
note = "Publisher Copyright: {\textcopyright} 2024 Author(s).",
year = "2024",
month = oct,
doi = "10.48550/arXiv.2404.13393",
language = "English",
volume = "14",
journal = "AIP Advances",
issn = "2158-3226",
publisher = "American Institute of Physics",
number = "10",
}