Last week, we organized a one-day group retreat to reflect on our projects, how we work together, and where we see the field heading. What quickly became clear is the variety of interests within the group: from data extraction and large language models to vision, spectroscopy, polymers, and beyond; while our topics differ, they all share one common goal. We want to work on problems that matter.
Across all discussions, there was a shared conviction that AI in the sciences should do more than optimize benchmarks or produce incremental gains—it should meaningfully support and accelerate research. Whether through better data, more realistic models, or new ways of integrating information, we are all trying to push toward that goal in our unique ways.
(Illustration by Daniela Cappuccio.)
Pick: Grand Challenges for Predictive Modeling in Small Molecule Drug Discovery
I read this paper because I think it’s important to be clear about what the big problems in a field actually are. Writing them down well is harder than it sounds, and I’ve been trying to do this myself in different ways. So I was looking forward to seeing what this group came up with. I’d also be curious to see a paper behind the paper: how they gathered input, how they prioritized, what got cut.
The authors organize challenges across four domains: Chemistry, Structure, Energy, and Pharmacology. Each follows a template: problem definition, relevance, current state, inputs/outputs, and metrics. What struck me is the difference in detail across challenges. Some have sharp metrics and near-formal definitions. Protein structure prediction benefits from decades of CASP benchmarking, and it shows. Other challenges are framed in a much fuzzier way. There’s also an inconsistency in how cleanly the challenges separate the question/challenge from the solution. Some sections define what you’d need to predict and how you’d score it, leaving the method open. Others already suggest an answer, like pointing to a missing benchmark dataset. Perhaps this highlights how difficult it is to really formulate “grand challenges” from a vantage point that is not too tainted by one’s own background.
Some of the challenges connect to what Nicola Marzari has written about predictive accuracy and complexity. The frontier isn’t only about better accuracy on clean systems. It’s about the real fuzziness of realistic systems, and that matters across domains. pKa prediction is a good example. It sounds basic, just the acid-base behavior of a molecule, but state-of-the-art methods still carry errors around 0.7 to 1.0 pKa units. One unit means a tenfold difference in equilibrium constant. The reason it’s so difficult to get right is that it’s already an ensemble problem. Predicting pKa means modeling a delicate equilibrium between multiple microstates of a molecule in solution: different tautomers, different protonation sites, intramolecular hydrogen bonds, all in a solvent environment. It’s not a single-structure problem. Many of the other pharmacology challenges (solubility, permeability, metabolic stability, bioavailability) share this same structure: predict what happens to a molecule in a messy, heterogeneous environment where multiple interacting processes play out across an ensemble of states. A more abstract framing might reveal shared bottlenecks that the challenge-by-challenge enumeration obscures.
I think this points to a deeper missing challenge, one about language and representation. Our modeling infrastructure is built around single entities: one molecule, one protein conformation, one polymer chain. We define datasets that way. We train models that way. But reality lives in ensembles. A protein samples a distribution of conformations. A drug may aggregate or precipitate in concentration-dependent ways. In polymer science, we know this well: you never have one chain, you have a distribution of lengths, architectures, and even different end groups. When we do address ensembles, it’s usually an afterthought (e.g. by taking ensemble averages of predictions for the ensemble members). We predict a property for a single structure, then bolt on averaging at the end. The representations, the loss functions, the data formats, none of them are natively ensemble-aware. Even though many of the actual observables we care about are ensemble properties (and we often pretend that they can be cleanly mapped to one SMILES). Making ensemble-level prediction a first-class problem, across protein dynamics, solubility, pharmacokinetics, and materials design, might itself be a grand challenge. The paper touches on it in several places but never names it as a unifying theme.
The paper covers one part of the drug making pipeline (and clearly calls this out). But I wanted it to zoom out. I have a chronic disease and was surprised at how few studies exist. This turns out to be a common pattern. Even with a clear therapeutic hypothesis, the clinical endpoints for chronic diseases are often delayed and rare. Osteoporosis is the well-known case: we have therapeutic ideas, but the definitive endpoint, someone falling and breaking a bone, is rare enough that trials become prohibitively expensive. Computation might speed up everything before the clinic. But if the endpoint bottleneck remains, patients still wait. The grand challenges framework measures computational progress, but the elephant in the room for many diseases is that the expensive bottleneck sits downstream, limiting access to therapeutics for people who would benefit from them.
Perhaps the ultimate challenge is to figure something out on this frontier.
Picks: Isoreactivity in chemistry and Chemical reactivity from the perspective of enthalpic coupling
Recently, Yudin and coworkers published two new papers that examine reactivity from a different angle. The authors argue that no branch of science can exist without terms that capture similarities among its objects of inquiry.
They define the concept of isoreactivity (in chemistry jargon: iso defines similarity between two structures, think iso-topes, iso-mer). While not previously formalised, and not fully fitting existing analogy frameworks (e.g. isoelectronic, isolobal), it seems that the general idea identifies with the way that chemical intuition works.
The best example from literature was for me to look at perhaps the most known reaction for the AI4Science community: and that is the Suzuki-Miyaura coupling and the way its discovery came to be. In 1972, Kumada and Corriu independently found out that Grignard reagents (R-MgX, X=halogen) could be cross-coupled with aryl and vinyl halides using nickel catalysis. Three years later, Palladium was introduced as an alternative by Murahashi. Negishi then systematically explored whether other organometallic modules could replace magnesium in this catalytic cycle, testing aluminum, zinc, zirconium, and boron among others. Organozinc proved to be the most effective, but in the case of neutral, tricoordinate organoboranes, the reaction failed or resulted in very low yields of the desired product. It was not until 1979 that Miyaura, Yamada, and Suzuki discovered that the addition of a base (e.g. NaOH) activated the organoborane toward transmetalation, turning a failed reaction into what would become one of the most widely used cross-coupling methods in organic chemistry.
The latter paper (Nat. Synth. 2026) looks at reaction mechanisms from the perspective of enthalpic coupling. An example is alkene epoxidation via Bartlett’s butterfly mechanism: the reaction works because the O–O bond in a peracid (45 kcal/mol) is weak enough to cleave in the key step. Swapping to a hydroxamic acid, where the bond being broken is now N–O (48 kcal/mol), introduces a modest increase of just ~3 kcal/mol — yet no Bartlett-type epoxidation with hydroxamic acids is known. This sensitivity to small enthalpic changes in the transition state resonates with the Negishi case: when boron was substituted for magnesium in the cross-coupling cycle, the transmetalation step failed — not because the idea was wrong, but because the enthalpic balance of that specific elementary step could no longer sustain a forward path without external activation (in Suzuki’s case, a base).
Carefully screening such examples with synthetic or ML-based approaches, could enable the discovery of new reactions, or in the case of the organoborane, just get another look. We could use the isoreactivity paradigm to generate candidate substitutions, and then use enthalpic coupling — the energy bookkeeping of bond-making and bond-breaking within an elementary step — to evaluate whether the new pathway remains viable.
Pick: Beyond Language Modeling: An Exploration of Multimodal Pretraining
Large language models have been the main talking point for a few years now. While some claim we will reach AGI with LLMs any day now, others raise over a billion dollars to show that world models are the true AGI. While AGI might be a new cold fusion and we might never reach it, I believe that world models might offer us the next step in scientific discovery. World models come with the idea that language is just an abstract way for us to understand the world, and real models have to be multi-modal, combining vision and language.
Authors of “Beyond Language Modeling: An Exploration of Multimodal Pretraining” conduct a set of experiments by pretraining models via the Transfusion framework from scratch on a mixture of text, images, videos, image-text pairs, and action-conditioned sequences. Transfusion combines the next token prediction for text and diffusion modeling for images, creating a unified multimodal system. They demonstrate that vision does not compete with text, showing that text + video mixing can achieve lower perplexity, highlighting the compatibility of the two modalities. While image-text pairs are essential for visual understanding and generation, they often degrade perplexity a bit, which the authors attribute to the distribution gap between images and text captions. Additionally, they show that adding text to a consistent vision token budget helps improve the generation at a steady rate without vision having a significantly negative effect on language. Furthermore, they explore action-based navigation, both as text navigation input and as WASD controls. But more importantly, they show that a mixture of experts can replace modality-specific FFNs by dynamic decomposition of FNN layers without human-induced bias; they showcase that as we go forward in layers, the experts become less and less text-focused and contain more vision and multimodal experts.
One paragraph is not enough to cover the whole of this work, but it is hopefully enough to entice you into reading it. More importantly, how do the world models connect to our work, and why are they important for scientifically intelligent machine learning models? Research is naturally multi-modal; it is communicated via papers (text and images), data is collected from a diverse set of modalities (captured by all types of sensors, optical, thermal, radiation intensity, and more), and this data is represented in a human-readable format, images, signal curves, graphs, text, etc. To make models that are truly scientific in nature, we have to move away from just using text; we have to learn to absorb all of this information from the real world. We need models that can understand data that is measured and be able to communicate it in a human-readable format. In my mind, not only are LLMs not enough, but world modes might be only the first step in making true scientific intelligence…
Pick: Intelligence Per Watt: Measuring Intelligence Efficiency Of Local Ai
One key lesson from Richard Sutton still holds true: scale drives progress. For most AI companies, especially in LLMs, the path forward seems clear. They sign massive compute contracts and scale everything. The Qwen director put it plainly: scaling is their future direction. Training keeps improving, but inference is becoming the real bottleneck, especially because of the low latency that it needs (arxiv.org). Recent papers show it will only get tighter. This applies to both cloud usage and on-device inference.
Because of this, this month I found the article “Intelligence per watt” really interesting.
In this work, Saad-Falcon et al. present an evaluation of how good local LLMs are. For this, they present a new metric that they call “intelligence per watt” (IPW). Using this metric, they also compared local models with the same models served from datacenters (more powerful chips). The scale of the analysis is quite big, over 1M queries, for more than 20 models, and using different chips. The findings are really interesting, as while local models’ accuracy is really good, the IPW is lower for the ones that are served locally. One thing I missed: a look at very small models (~1B or less). Several good options exist at that scale.
I expect this type of evaluation to grow more important. Local inference will gain ground over API-based models as small open models get better.
It is also great to see metrics that measure energy and power use. We need more of this. LLMs consume enormous amounts of energy. As a community, we should not normalize consumption without reason for accuracy. This matters even more now. Small reasoning models can run indefinitely and still fail at complex tasks. Normalized consumption metrics will push us toward genuinely energy-efficient models.
My pick this month is a paper by Dehghan-Toranposhti et al. on robust learning from literature data for predicting conjugated polymer solution conformation (https://doi.org/10.1063/5.0303721), which investigates how polymer structure, formulation, and processing jointly determine solution-state aggregation and how reliably these relationships can be learned from heterogeneous literature data.
The authors manually curate a dataset of 256 literature-derived data points linking polymer properties, solvent interactions, and processing conditions to the radius of gyration (Rg).
Models achieve good performance under IID conditions (R² ≈ 0.9), but generalization breaks down under OOD settings, especially for new chemistries.
Polymer chain length is the dominant factor, while structure alone adds little; processing and solvent effects are essential for accurate predictions.
SHAP analysis reveals physically meaningful trends (e.g., concentration and solvent quality driving aggregation), while environmental and thermal history appear to have little predictive value, likely due to inconsistent reporting.
What I particularly like about this work is that it moves beyond a purely structure-centric view and explicitly demonstrates that polymer behaviour emerges from the interplay of molecular structure, processing conditions, and formulation. This is much closer to how polymers are actually handled in reality. I also appreciate the effort to interpret the model outputs in chemically meaningful terms via SHAP analysis and even explaining trends chemically. The authors are also quite transparent about the limitations of their data, for instance in pointing out that environmental and thermal history appear uninformative—likely not because they are physically irrelevant, but because they are inconsistently reported in the literature. This observation strongly resonates with my own experience. Finally, their discussion of standardized data collection and the potential role of self-driving laboratories suggests a possible way to address these issues, even if it is not a complete solution.
At the same time, there are several limitations worth discussing. The dataset is relatively small (256 data points), which raises questions about robustness. In addition, a large fraction of the initially extracted data is discarded during curation. While this reflects the challenges of inconsistent reporting, it also raises the question of efficiency and suggests that the main bottleneck lies in data usability rather than availability.
It also seems plausible that more automated extraction approaches, for example using LLMs or vision-language models, could help scale up data collection.
Pick: Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
I selected this paper because the authors discuss an interesting effect in large language models called Artificial Hivemind, where all the LLMs output similar responses. For example, when they ask to write a metaphor involving time. ALL the models that they used output similar responses, like time is a river or time is a weaver, even when they keep different temperatures for those models. This is surprising because it’s an open question, and there is no correct answer for it. Models have plenty of options and potential to choose different answers; however, models used the river and the weaver as a choice. This artificial hivemind effect remained constant in two different settings or environments. In both intra-model repetition and inter-model homogeneity. Which means within the same model itself and across other models respectively.
To evaluate this behavior of models, the authors created an infinite chat dataset categorized into a taxonomy of six main skills. Within each skill, they also have various sub-skills. What is interesting for authors is that pluralistic alignment in llms. Since there is no single ground truth, can models generate multiple answers for the same open-ended questions?
For intra-model repetition evaluation, the authors considered 100 samples from the Infinity Chat dataset. For each of the questions, they have sampled or generated 50 responses by keeping T=1 and T=2. They reported an average score similarly across all models, and this shows that the responses are very similar.
Another interesting part is that the authors wanted to calibrate the LLM-generated responses to match human preferences for open-ended questions, which has not been studied before. The annotators annotated their preferences for each of these questions, and they could either agree or disagree with one another. Because there is no single correct answer, it makes sense that the model struggles to align with human judgments on topics where the human annotators cannot reach an agreement.
This means that the models are providing safer responses or choices for generating outputs. The models are trained on examples with higher human agreement.
From this paper, what I feel is that LLMs are not very creative enough in generating different responses for open-ended questions that have no ground truth.
Pick: GraphT5: Unified Molecular Graph-Language Modeling via Multi-Modal Cross-Token Attention
I have recently been exploring graph-text multimodal approaches. The idea of unifying these two modalities appears relatively intuitive, and straightforward implementations have already been proposed (For example, in 2023 (arXiv:2306.13089), but there are of course earlier efforts also addressed similar questions in the molecular domain (e.g., 10.1609/aaai.v35i1.16131; arXiv:2110.09681v1).
Much of the recent work in this area focuses on dual-encoder, contrastive multimodal learning frameworks. In contrast, GraphT5 adopts an encoder–decoder architecture with cross-attention between node representations and text tokens. Conceptually, the approach is relatively simple, and the authors report performance gains over prior methods, including text-only models such as MolT5 (arXiv:2204.11817) and multimodal approaches like MolCA (Q-Former style; arXiv:2310.12798).
My main critique concerns the choice of downstream evaluation tasks. The model is primarily evaluated on IUPAC name generation and structure description. While these are reasonable baselines, particularly for comparison with SMILES-based models - they do not fully test the model’s ability to reason over graph structure. A more informative evaluation would involve tasks that explicitly require understanding of the molecular graph, such as determining connectivity between specific nodes.
In this regard, the recently accepted ICLR 2026 “Entropy-Guided Dynamic Tokens for Graph-LLM Alignment in Molecular Understanding” (arXiv:2602.02742) provides a more compelling direction. This work, also inspired by the Q-Former paradigm, introduces an entropy-based mechanism derived from next-token prediction over SMILES sequences to guide subgraph partitioning within a GNN encoder. This is an interesting trick, and the reported performance on more structurally grounded tasks appears promising.
Pick: Measuring all the noises of LLM Evals
This paper provides a systematic and principled analysis of uncertainty in LLM evaluation. The authors introduce a unified variance decomposition that separates evaluation noise into two main components: prediction noise, which arises from the stochastic nature of LLM generation, and data noise, which comes from variation in task difficulty across examples. Based on this framework, they show that prediction noise is often the dominant source of uncertainty in modern LLM evaluations.
A key methodological contribution of the paper is the use of paired comparison, where model differences are computed on a per-example basis before aggregation. This approach effectively cancels out data noise, thereby improving statistical efficiency. In addition, the authors propose an all-pairs paired analysis to estimate noise across model pairs, and empirically demonstrate that the variance of model differences on a fixed benchmark is remarkably stable and can be approximated by a simple form (e.g., p(1−p)). This finding implies that the uncertainty of evaluation results is largely determined by the benchmark itself, enabling practitioners to estimate confidence intervals and statistical significance without extensive resampling. Overall, the paper enables more reliable model comparison under a controlled evaluation protocol by quantifying and accounting for evaluation noise. They proposed to run multiple evaluations to reduce prediction noise and arise dataset sampling to reduce data noise. Increasing the dataset size can reduce variance, allowing us to determine whether the performance differences are statistically meaningful rather than just noise.
I think this paper aims to reduce bias in current benchmark evaluations by introducing a more statistically grounded framework than the current ones, which just use average accuracy. In particular, by effectively removing data noise and benchmark-related uncertainty through paired comparison, it establishes a cleaner and more controlled evaluation setting. This makes experimental results more reliable and, importantly, easier to interpret when analyzing other factors.
However, the framework assumes a fixed evaluation protocol—especially with respect to the prompt—and does not explicitly account for variability introduced by different prompt designs. But by minimizing data-related uncertainty, the paper creates a clearer setting in which prompt-induced effects can be more effectively isolated and studied. In this sense, it provides important groundwork that motivates and enables subsequent research on prompt sensitivity, such as BrittleBench.
I picked this paper because it works on a problem that overlaps with my project: predicting reaction kinetics from literature-extracted data. Their domain is peroxymonosulfate-based advanced oxidation processes (AOP) for water treatment — a different AOP than ozonation which I focus on in my project, but the pipeline structure is similar: automated literature acquisition, LLM-based data extraction, database construction, and training a regression model to predict rate constants. The pipeline follows a standard template by now: keyword-based corpus collection (~5100 papers from Elsevier, ACS, and Springer), PDF-to-Markdown conversion with MinerU, screening and extraction using DeepSeek-V3, and manual review to arrive at a final database of 3276 records from ~3030 papers.
The extraction step includes some choices worth noting: the screening returns a confidence score and reasoning alongside each classification, which helps with debugging. Qualitative descriptors like XRD peak intensities are mapped into categorical levels, and vague quantitative language from the papers (”about”, “approximately”) is assigned defined variation ranges. For the modeling, they train XGBoost to predict observed rate constants (k_obs) and removal efficiencies (η) in stages. A baseline with original features gives R²(k) = 0.30 on the holdout set. Feature engineering — 17 features derived from AOP chemical principles — brings this to 0.40, and Bayesian hyperparameter optimization with Optuna pushes the final performance to R²(k) = 0.675 and R²(η) = 0.835. They exclude reaction time to prevent data leakage, since k, η, and t are linked by k = −ln(1−η)/t. For feature selection they combine a Pearson correlation threshold (|r| > 0.03) with Mutual Information (MI) to capture non-linear relationships. Pearson’s r only measures linear association, so a feature can appear unimportant by that metric while still carrying information about the target in a non-linear way. MI quantifies any statistical dependence regardless of its shape. The stoichiometric ratio illustrates this: it has an MI of 0.788 but a Pearson r of only 0.044, meaning it is a strong predictor that a purely linear screening would have discarded. Extraction accuracy on 100 expert-annotated documents reaches 93% accuracy for screening and 84.6% for data extraction, with a ~270-fold speedup over manual processing.
While automated extraction pipelines paired with ML models for kinetic prediction are still uncommon in AOP research, this paper is a useful example to compare against when building a similar pipeline. It is interesting to see where others make different design choices, particularly around imputation strategies on missing data, engineered features from domain knowledge, and the gap between automated extraction accuracy and clean training data. The effort to manually annotate 100 papers for validation is worth acknowledging — this is time-consuming work that many studies skip, and it provides a concrete baseline for evaluating pipeline performance. One methodological gap is the absence of cross-validation: all reported performance metrics rest on a single random 70/20/10 split, with hyperparameters tuned on the fixed validation set and final R² values reported on a 328-sample holdout set. Cross-validation would give a more robust estimate of model performance and some indication of its variance across different data partitions. The paper also acknowledges survivor bias in literature-sourced data, which may push model predictions toward optimistic outcomes — a concern that applies to any ML pipeline trained on published data, including my own. On the presentation side, moving some of the figures from the results section to the SI could have improved readability — the core findings would come across more clearly with fewer but more focused figures.
About this newsletter: The paragraphs about the papers are written by group members individually and reflect their own opinions. The newsletter has been reviewed by other group members for factual correctness and edited by Gordan Prastalo. The editorial is written by Gordan Prastalo.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.