📝 SummarXiv

Abstract

Chemical language models (CLMs) are increasingly used for molecular modeling, yet their reliability is undermined by inconsistencies in the SMILES notation. Even canonical SMILES differ across toolkits, and stereochemical annotations are frequently incomplete, but the consequences for model behavior remain unclear. Here we systematically assess these effects through a literature survey, dataset analyses, and controlled modeling experiments. Nearly half of recent CLM studies omit canonicalization details, while public benchmarks contain redundant encodings and missing stereochemistry. We show that such inconsistencies destabilize latent representations and impair structural reconstruction, whereas property prediction tasks appear deceptively robust because feature selection filters unstable features. Most critically, we uncover confounding artifacts in benchmark datasets, where notational variants spuriously correlate with class labels and inflate predictive performance. These findings expose a reproducibility gap in the field and highlight the need for rigorous preprocessing standards and transparent reporting to ensure that CLMs capture genuine chemical principles.