📝 SummarXiv

Abstract

As AI capabilities continue to grow exponentially on economically relevant human expert tasks, with task completion horizons doubling every 7 months according to the Model Evaluation and Threat Research (METR), we are interested in how this applies to the task of mathematics research. To explore this, we evaluated the capability of four frontier large language models (LLMs), ChatGPT 5, Claude 4.1 Opus, Gemini 2.5 Pro, and Grok 4, at the task of creating a mini-paper on reservoir computing. All models produced engaging papers with some apparent understanding of various techniques, but were sometimes lead to mistakes by surface level understanding of key ideas. That said, the capabilities on LLMs on this task was likely as good or greater than that predicted by METR.