In recent times, machine learning models have gained traction for assessing the risks posed by chemical substances. Nevertheless, their opaque nature often leads them to be labeled as ‘black boxes,’ which creates doubt among toxicologists and regulatory bodies. To build trust in these models, researchers from the University of Vienna have suggested pinpointing the specific areas of chemical space where the models show weaknesses. They created a novel software tool named ‘MolCompass’ to aid in this effort, and the findings from this research have recently been published in the Journal of Cheminformatics.
Traditionally, new pharmaceuticals and cosmetics have undergone testing on animals. These tests not only incur high costs but also raise ethical issues and often fail to accurately predict human responses. Recently, the European Union backed the RISK-HUNT3R project to create advanced non-animal risk assessment techniques. The University of Vienna is part of this project consortium. Utilizing computational methods enables the assessment of the toxicological and environmental risks associated with new chemical entities entirely through computerized processes, eliminating the need to create the actual chemical substances. However, a critical question persists: How reliable are these computer models?
Focusing on accurate predictions
To tackle this challenge, Sergey Sosnin, a senior researcher in the Pharmacoinformatics Research Group at the University of Vienna, concentrated on binary classification. In this scenario, a machine learning model generates a probability score ranging from 0% to 100%, reflecting whether a chemical compound is deemed active or inactive (for instance, toxic or non-toxic, bioaccumulative or not, or capable of binding to a human protein or not). This probability indicates the model’s confidence in its predictions. Ideally, the model should exhibit confidence only in correct forecasts. If the model shows uncertainty, yielding a confidence score near 51%, these predictions should be ignored in favor of alternative approaches. However, an intricate issue arises when the model confidently declares wrong predictions.
“This presents a genuine nightmare for computational toxicologists,” explains Sergey Sosnin. “If a model asserts that a compound is non-toxic with 99% assurance, but it is, in fact, toxic, there is no way to identify the error.” The key to addressing this lies in recognizing regions of ‘chemical space’—the spectrum of potential organic compound classes—where the model possesses ‘blind spots’ ahead of time and steering clear of them. Unfortunately, this requires researchers to meticulously review the predicted outcomes for thousands of chemical compounds one at a time—a laborious and error-prone endeavor.
Conquering this major obstacle
“To aid researchers in this task,” Sosnin elaborates, “we developed interactive graphical tools that map chemical compounds on a 2D plane, similar to geographical maps. Utilizing colors, we indicate the compounds that were inaccurately predicted with high confidence, allowing users to spot them easily as clusters of red dots. The map is interactive, permitting users to delve into the chemical space and examine areas of concern.”
This methodology was validated using a model for estrogen receptor binding. After visually analyzing the chemical space, it was evident that the model performs well for substances such as steroids and polychlorinated biphenyls but fails entirely for small non-cyclic compounds, indicating that it should be avoided for those.
The software developed from this project is publicly accessible on GitHub. Sergey Sosnin hopes that MolCompass will guide chemists and toxicologists toward a clearer understanding of the limitations inherent in computational models. This research marks a significant stride toward a future where animal testing is rendered unnecessary and the only workspace for toxicologists is a computer desk.
