More compute and larger models are not always the most interesting way forward. Sometimes the bigger gain comes from changing what a model sees, what it predicts, or even how the problem itself is defined.
That idea appears across very different settings: representing atoms and bonds in complementary graphs, planning synthesis through higher-level chemical abstractions, predicting molecular properties without relying on the most expensive calculations, or accelerating language models by deciding which generated tokens are actually worth validating.
Even seemingly fixed categories are worth questioning. A solvent can turn out to be a reagent, and data conventions can determine what a machine-learning model is capable of learning. Across chemistry and AI, the recurring lesson is that better performance often starts not with doing more of the same, but with choosing a better representation of the problem.
Pick: Chemical Shift Prediction Beyond the Electronic Structure Limit
This paper from Keir Adams et al. (who in Mar 2025 left Connor Coley’s group to join DE Shaw) comes from a collaboration between many different institutions: including well-known groups at MIT (Connor Coley, Tess Smit), experimental groups at Harvard (Richard Liu - who I had the pleasure to meet in the CAS Future Leaders cohort) and Colorado State (Robert Paton). But it also includes industrial collaborators from Rowan and Merck - with Eugene Kwan being the last author (and also hosting the code and data on his private repo, where he also benefited from Claude in preparing the repositories).
So what did this team of celebrated and award-decorated collaborators build? They promise with the title “Chemical Shift Prediction Beyond the Electronic Structure Limit”. The paper opens with an analysis of the performance of current simulation techniques. In their first figure, they argue that 38% of 1H shifts are currently uninterpretable with DFT (this number makes sense, even though I could not fully find how they performed this analysis; also not sure how it is linked to the molecule they have in the inset). They dig deeper and find (1) a very high correlation between existing DFT techniques, and most of the differences between different techniques can be removed with linear scaling (they find Pearson correlations from 0.99 to 0.9999). They also plot a Pareto frontier between accuracy and computational cost that one might see from existing techniques (one of their main arguments will be that they shift this Pareto frontier to much lower cost compared to the current best technique wp04). The authors proceed to outline that also implicit solvation (e.g., PCM) is unphysical, which probably makes a lot of sense to every chemist. In the end, especially the outer hydrogen atoms still notice what is happening around them, and just a dielectric constant does not seem to do this full justice.
After this long buildup, they show that explicit solvation helps. They get it from shielding computed on the solute after an MD in solvent (using classical techniques) and the solute in an empty box. They add to this rotovibrational effects from quasiclassical simulations (that Eugene Kwan developed).
This is computationally expensive (and they also mention this in the text - but I could not find detailed numbers on the computational scaling behavior of the different techniques, including their new one). To address this, they use machine learning:
They use MagNET (a model based on an Equiformer architecture) to predict shifts trained on new large (and open) datasets they assembled. (They seem to also have created experimental datasets, but I couldn’t find them - there are some Mestra Nova files on HuggingFace, but it was not easy for me to understand how they have been produced).
Since this only replaced the shielding calculation, they still have the expensive geometry optimization cost. To address this, they then trained a model to directly consume geometries produced with an MLIP
Given that much of their buildup was about solvent correction, they then also trained MagNET models for implicit and explicit shift computation.
The discussion section gives some honest discussion about the limitations. One important one is that at some point the solvent correction errors on the solvents (due to, for example, conformational flexibility) seem to matter a lot. I really would have loved more of this discussion (perhaps by increasing the narrative in the supporting information).
The authors added a lot of new methods for shift prediction on the table where there are already many techniques. I am sure that - especially for a platform like Rowan - automated routing to the right technique will be an important frontier.
Pick: DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
As models continue to scale, inference speed must keep pace to remain practical. Mixture-of-Experts (MoE) architectures were already designed to be more efficient and faster than dense models, yet there remains substantial room for further acceleration. Multi-token decoding offers one promising avenue: rather than generating tokens one by one, the model produces several in parallel, drastically reducing inference time. However, this approach demands careful design to avoid compromising the quality or coherence of the generated output. Speculative decoding addresses this challenge by generating multiple candidate tokens that are subsequently validated through a forward pass of the model. This enables faster inference without sacrificing accuracy, as the parallel generation is always checked for validity and coherence.
Recently, speculative decoding has seen significant research interest. To my knowledge, the current state-of-the-art was the DFlash method, which employs a 5-layer diffusion model to generate multiple tokens in parallel from textual anchors - such as the first token of a sentence. These generated tokens are then validated in parallel via a forward pass of the main model.
The paper I selected presents DSpark, which builds upon DFlash by introducing a system that discards incorrect predictions early, rather than always relying on the expensive forward pass of the large language model. This is achieved through two key components: a confidence-prediction head that estimates the reliability of each generated token, and a hardware-aware scheduler that decides whether to proceed with validation based on current system load. This approach enables more efficient resource utilization by skipping unnecessary forward passes for tokens likely to be wrong, while still maintaining the validity and coherence of the final output. Additionally, DSpark introduces a lightweight sequential head to enhance inter-token coherence within the drafted block. The results demonstrate clear inference speed improvements over both DFlash and Eagle3.
I particularly like Figure 7 in the article, which illustrates how the hardware-aware scheduler boosts inference speed in response to device load conditions. It’s a really nice visual because it clearly shows how the method improves for such production-heavy conditions. More broadly, I find speculative decoding to be a powerful technique, and the fact that the current SOTA leverages a diffusion model is especially intriguing. It offers a fresh perspective on diffusion models as faster-than-autoregressive tools for text generation. I’m drawn to this emerging role of “diffusion language models” as facilitators for autoregressive LLMs - a dynamic that echoes a point made in “Position: There are futures that benchmark-driven AI cannot see”: some ideas find value in applications far beyond their original intent. This is a prime example of how diffusion models can be repurposed to accelerate autoregressive systems - as in DFlash, the diffusion block serves as the primary predictive component - and I believe it opens up a very exciting direction for future research - even as efforts in block diffusion models also remain remarkably promising.
The Current trend of building increasingly large single-cell foundation models assumes that more pre-training will lead to better performance. This paper provides an important counterpoint to that assumption. Across five model architectures, 400 pre-trained models, and 6400 evaluations, the authors show that performance on cell-type classification, batch integration, and perturbation-response prediction often saturates after only 1-10% of a 22.2 million-cell dataset. Increasing diversity through cell-type balance, geometric sketching, or the addition of perturbation data also does not consistently improve performance. In my view, the key message is that dataset size and nominal diversity do not necessarily represent the amount of useful information available to a model. This idea also connects to our work on ChemFuse, where we align different chemical modalities to help models learn from otherwise separate datasets.
Another important finding is that simple approaches such as principal component analysis (PCA) and single-cell variational inference (scVI) often match or outperform more complex transformer-based models such as Geneformer. SCimilarity performs comparatively well, likely because its contrastive objective is closely aligned with learning cell-type representations. This suggests that the relationship between pre-training objective and the downstream task may be more important than the architectural complexity. The perturbation-prediction results show a simple no-change baseline outperforms the fine-tuned models for most of the tested drugs. This indicates that current foundational models may capture general cell identity but struggle to represent context-dependent biological responses.
However, the paper should not be interpreted as showing that large-scale pretraining is always unnecessary. Its conclusions are limited to the evaluated models, datasets, and downstream tasks. Nevertheless, it convincingly demonstrates that simply adding more cells is unlikely to guarantee improved performance.
Pick: Enhancing molecular property prediction of transformer models with dual graph representation
Molecular representation learning methods mainly focus on atoms, often ignoring bonds, or they use them just to represent graph connections (often ignoring the bond types completely). Dual Graph Transformer (DGT) tries to solve this problem by building an atom graph and a bond graph. In the atom graph, atoms are nodes and bonds are edges; in the bond graph, bonds become nodes and neighboring bonds are connected through their shared atoms. The information between these two representations is exchanged, and they are processed in parallel. DGT directly incorporates molecular topology, structure, and stereochemistry into self-attention through pairwise encodings of shortest-path and random-walk positions, ring membership, optional 3D descriptors such as bond lengths, interatomic distances, and bond–bond angles, as well as chirality and E/Z isomerism.
DGT is ranked as the best-performing model on the aggregated results in all evaluation domains (physiology, biophysics, physical chemistry, and quantum mechanics) across 10 datasets and 58 tasks when compared against D-MPNN, GROVER, MolCLR, GraphMVP, MoleBERT, and Uni-Mol. Ablation studies indicate that both graph views contribute to performance, while ring and relative-position encodings provide additional improvements. Additionally, they showcase that accurate 3D information substantially reduces errors for HOMO and LUMO prediction.
I find the idea of DGT very interesting, and intuitively it makes sense that adding more information helps. However, like many previous molecular representation studies, the evaluation is constrained by the limitations of MoleculeNet. Several datasets are small, noisy, and based on relatively narrow chemical distributions, while the reported scaffold split does not fully capture the difficulty of generalising to genuinely novel chemical spaces. Aggregating performance across tasks can also obscure cases in which the model does not outperform every baseline individually. The main manuscript reports that DGT performs best at the domain level, but it does not provide a clear count of how many of the 58 individual tasks it wins, making it difficult to assess the consistency of the improvement.
I’m not an organic chemist by training, but I read chemistry papers when something catches my interest. This one did, because it belongs to a genre I like: finding something interesting in something seemingly boring. The authors took dichloromethane and asked what happens if you treat it as a reagent instead of a solvent. It turns out DCM will attack a carboxylate and then hand the acyl group off to an amine, forming amide bonds without any of the usual coupling reagents.
It doesn’t beat HATU on most axes. It needs 80 °C for twelve hours, its atom economy is 53%, and the mechanism releases a stoichiometric equivalent of formaldehyde, which the paper’s own conclusion admits complicates the green chemistry. The appeal is elsewhere. Every step of this was reachable with 1950s glassware, and it has not been tried before. DCM was only used as a solvent for the longest time, without inquiry about its other possible functions.
That’s the part I think about sometimes with respect to using ML to identify such candidates. The failure here seems a data-model failure. In every reaction database — USPTO, Reaxys, the sets everyone trains on — there is a field called reagent and a field called solvent. A retrosynthesis model reading this paper’s reaction sees benzoic acid plus benzylamine plus carbonate, in DMSO, in DCM. Ask that model to predict the product, and it cannot propose the amide, because the atoms that form the bond were parsed out.
Picks: Higher-Level Strategies for Computer-Aided Retrosynthesis
Retrosynthesis is a process of tracing a complex target molecule to its building blocks. This analysis is very useful as it can potentially avoid long, wasteful reaction chains by spotting shortcuts. We can build strategic choices, flag reactive groups, and protect them early. The infamous Computer-Aided Synthesis Planning (CASP) algorithm by E. J. Corey tries to automate this process and recursively propose precursors of a target. The major limitations with CASP include high computational cost, and since it breaks down the solutions into mathematical and logical rules, breaking down every possible rule is yet another limitation, and the feasibility of synthesis in laboratories is always questionable. There have been tremendous advancements in modern search algorithms and various data-driven approaches with the release of datasets like USPTO and ORD, but these still do not fully align with how a chemist approaches this problem.
Jihye Roh et al. take a very interesting approach where they try to simplify the problem while trying not to compromise on the search space. From a chemist’s perspective, the search for building blocks begins by breaking a molecule into hypothetical fragments with certain chemical logic, resulting in synthon(s). This allows one to focus on the retrosynthetic strategy rather than specific conditions. A synthon typically carries an imaginary positive or a negative charge. Next, these are assigned synthetic equivalents to make them real chemicals and run experiments. The authors of the paper try to mimic this approach, and we see later that this comes to an advantage in identifying strategic patterns. They introduce a higher-level retrosynthetic planning algorithm that also emphasises multi-step pathway predictions, unlike others. In this framework, they systematically create synthon-like structures by avoiding protecting groups, leaving groups, and functional groups and tagging the linked atom with its electronegativity. They curated a dataset with roughly 780,000 abstracted higher-level reactions, and from these, they generalized over 32,000 reaction templates that would later be prediction targets. They trained a feed-forward neural network that takes the molecular fingerprints of the target molecule to predict scores of which template rule the molecule would break into most effectively. By taking the top-scoring templates, another graph-transformation algorithm uses these templates to cleave the molecule into abstract precursors. Finally, Monte Carlo Tree Search (MCTS) combined with substructure match aids in identifying the real, commercially viable building blocks, and if not found, the model is asked for next precursors with this abstracted precursor as an input.
With top-10 accuracy over 91.9% for generalized abstraction for a single-step model, the authors show that the model successfully learns the broad synthetic strategies. The higher-level algorithm proposes shorter routes compared to conventional algorithms, enabling more computational efficiency. Conventional algorithms fail to find synthesis routes for complex targets such as pendolmycin and pinolidoxin, while their algorithm successfully navigates through the search space to propose valid synthetic routes. Another example that the authors show is that the high-level algorithm found a viable synthesis route for narlaprevir (used for treatment of chronic Hepatitis C) in merely 25 iterations while the conventional algorithm required 331 iterations!
I enjoyed reading this paper because it successfully adapts the intuitive way chemists think about synthon-based strategy into a computational framework and felt that I could draw direct inspiration for my own project to model electron flow narratives of reaction mechanisms using LLMs. Although my project and this work are very different, the fundamental challenge of abstracting core chemical logic from highly variable data is incredibly similar, and their approach to solving it is brilliant
Picks: Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Emergent abilities in LLMs continue to be a vaguely understood topic. While there is interest in understanding this, in my mind at least a thorough study of this problem deserves an elaborate and complex experimental setup computationally not tractable to many. The authors attempt to look into this problem based on an analysis of Pythia checkpoints and two controlled experiments; the authors were able to reproduce findings from earlier analysis, which makes this study interesting.
The paper argues that emergent capabilities in language models appear abruptly and at random points in training. It says emergence happens when the right attention patterns are formed, and it forms abruptly. Detailed findings include that larger models find them earlier, long-context tasks do not emerge early, and depending on the task, the number of heads and the width of attention heads might be crucial. They also claim that architectural choices can speed up this search.
There are two parts to their experimental setup. First, on real language models, they use the Pythia suite (14M–410M parameters, ten random seeds per size, 154 saved checkpoints per run); they probe some skills that arise incidentally from ordinary language modeling (string copying, in-context repetition, pattern completion, and indirect object identification ). Each is defined by a single prompt where emergence means the model’s greedy prediction matches the correct token. Second, in a controlled setting, they train small transformers from scratch on two synthetic tasks with known ground-truth attention patterns: a sparse linear map and cellular automata (predict a local-rule update, requiring attention to a 3-cell window, with context length scaled via state size and trajectory length).
On Pythia, capabilities emerge at wildly different training steps depending on the seed, sometimes never; larger models emerge earlier and more often. The correct-token probability spikes abruptly at emergence; this coincides with specific heads snapping from diffuse attention into interpretable patterns (previous-token, copy, name-mover), and patching just those attention maps into the earlier checkpoint recovers most of the capability. On the synthetic tasks, training shows long loss plateaus ending in sharp jumps, each jump corresponding to one or two heads learning a single row of the target pattern; medium sparsity and long contexts make tasks unlearnable, while biasing attention toward the ground truth removes the plateau almost entirely. More attention heads consistently help, head dimension matters only up to a task-dependent minimum, and MLP-Mixer learns the linear map an order of magnitude faster than a transformer but underperforms on cellular automata.
The inference chain followed in the paper is that 1. emergence coincides with attention-pattern changes, 2. patching attention alone elicits the capability early, and 3. therefore attention learning is the bottleneck. The synthetic tasks then serve as the mechanism study: since the ground-truth pattern is known, the attention-bias intervention isolates pattern search as the hard part, and the sparsity/context-length sweeps identify what makes the search hard. Each head is interpreted as an independent candidate searcher, so more heads raise the chance of finding the pattern, which they offer as the explanation for why bigger models emerge earlier and more reliably. The main caveats are that causal-head selection is post-hoc on the same metric, and emergence is defined on single samples. I would assume that things can change when you train on multiple tasks together; there could be the same patterns that are jointly useful or training dynamics negating earlier formed patterns. Formalizing emergence as a search problem, and not as a capacity problem, still needs to be studied imo.
Picks: Can AI Help Chemists To Solve NMR?
I chose this paper because one of my next projects will be about benchmarking NMR prediction systems, and this preprint does exactly such a comparison.
The question is simple: if a chemist pastes a ¹H and a ¹³C peak list into a language model, does it give back the correct structure? The authors built a benchmark of 105 molecules from the literature. Every structure was checked by hand against the original publication. The set covers 21 functional-group classes, with one molecule per complexity level in each class. The only input is the two peak lists as text, without molecular formula, starting materials, or other spectra. Six general LLMs and four specialized NMR solvers were compared.
The specialized models are clearly better. BLIND and NMRPeak both solve 49 % at Top-1, and BLIND reaches 69 % at Top-10. The best LLM (Gemini 3.1 Pro) solves 20 %, and all six LLMs together only 24 %. More interesting than the ranking is how the models fail. Both families often delete groups that give no signal in the proton spectrum (nitriles, halogens, nitro, diazo) and propose smaller molecules than the true one. The LLMs also return an invalid SMILES or no answer at all in a considerable number of cases.
What I like is that the paper is written from the view of a working chemist, and that it reports failure modes and not only accuracy. The supporting information is very detailed: the prompts, the model snapshots, the token budgets, and even the manually recovered answers are documented, and data and code are public. The authors also discuss openly that they cannot fully exclude data leakage.
Two things would be useful in addition. First, every model was run once per molecule, and no confidence intervals are given. With 105 molecules, one molecule is already about one percent, so it is difficult to judge which of the smaller differences are meaningful. Second, all models were used with their provider-default reasoning settings, so the thinking budgets were not the same for all of them. Claude Opus 4.8 answered without extended thinking, while DeepSeek V4 Pro ran into the token limit in 68 of 105 cases. The authors describe this clearly and justify it with the cost for a normal user, which is a fair argument. A follow-up with the same reasoning budget for every model would nevertheless be interesting, because it would show how much of the difference comes from the model and how much from the setting.
For my own project, I take three points: report uncertainty, separate “wrong answer” from “no answer”, and report the results by difficulty.
Picks: Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs
With my current work being focused on fine-tuning a small language model to produce JSON-formatted output using human-readable prompts, I was looking into more recent research on SFT. While the paper is already a bit older, it covers an empirical grid-search study on fine-tuning smaller language models in the 3B-7B parameter range. Overall, the authors test four models on various datasets, spanning general instruction, knowledge, skills, and more. The models are trained using sequential phased (data divided into different phases, trained sequentially) and stacked (training on full data at once) training, and compared based on training strategies and hyperparameters, such as batch sizes, learning rates, warmup steps, and learning-rate schedules. This is especially interesting for me, since the results can help me to make more educated decisions on the most optimal settings for my SFT task.
A few key observations are presented as:
combining larger batch sizes with lower learning rates improves performance on benchmarks like MMLU, MTBench, and Open LLM Leaderboard v2.
Early-stage training dynamics, such as lower gradient norms and higher loss values can indicate better model performance, allowing you to terminate runs early, saving computational costs.
Omitting warmup steps and using a constant learning rate does not affect performance.
Stacked training and phased training have similar performance, but stacked training is more sample efficient.
For batch size impact, the authors experimented on 128, 3840, and 7680 samples, which is way above the batch sizes I could test out using our servers. The results show that larger batch sizes lead to better performance, but also come with higher computational cost. A main reason is the reduced statistical error in gradient estimation. Also, in terms of processed samples, models trained with smaller batch sizes achieved higher performance faster, although they reach their plateau earlier compared to models trained with larger batch sizes.
For the learning rate, the authors observed that cosine decay does not necessarily improve performance when fine-tuning smaller language models with larger batch sizes. In such cases, a constant learning rate is more beneficial to ensure consistent progress throughout training, potentially simplifying training processes.
Between the Granite and Mistral 7B models, the optimal learning rate for Granite is 20 times higher than for the Mistral model. This shows that optimal hyperparameters are more dependent on the model family, rather than the size. This means that the numeric results are not necessarily transferable to my models. Furthermore, with our server size, possible batch sizes will not reach beyond 100 samples. As for myself, I don’t believe that I will be actively changing any settings for my fine-tuning process yet. Still, I have learned a lot about the effect of hyperparameters on post-training methods and have gained a better understanding of model training.
Pick: Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretraining
Instead of pixel-level or atom-level reconstruction (like generative models trying to guess masked atom types or coordinates), C-FREE predicts the representation of a target part of the molecule directly inside an abstract latent space. Just like I-JEPA masks patches of an image and uses the surrounding visual context to predict the masked patches, C-FREE treats local ego-nets (subgraphs) as "patches" or "tokens." It uses a context encoder to read the surrounding molecular environment and a target encoder to process a specific target ego-net, training a predictor network to bridge them. While previous graph adaptations of JEPA (such as GraphJEPA) exist, they rely on heavy preprocessing, like METIS graph clustering or complex hierarchical/positional encodings. C-FREE streamlines this by using fixed-radius ego-nets. Ego-nets basically pick a central atom, expand outward by k hops along the molecular bonds, and make it a subgraph.
Kim Review: In this contribution, we discuss two directions that we believe could be the most relevant ones for LLMs in chemistry: serving as “soft inductive bias” (i.e., having a well-grounded preference for certain solutions) and being able to encode and use “fuzzy” information that is hard to represent in structured form (e.g., preference for “feasible” reactions).
Paper: https://kimreview.org/commentaries/10-25950-6c2e5798/
Openclatura: We report a tool that solves a long-standing problem: converting a SMILES into an IUPAC name. The code is open source and deterministic in Python. The correctness can be checked via self-audit or round-trip with OPSIN. The immediate community feedback was unexpected and very motivating. Ironically, the hardest structural problem we faced was figuring out what to name the tool itself.
Paper: https://chemrxiv.org/doi/abs/10.26434/chemrxiv.15006114/v1
Roadmap on Digitalising Materials Science: In this article (led by Kevin Rossi) we review with many other contributors the “Roadmap on Digitalising Materials Science”. Our sections focus on “Institutional reforms to foster innovation in AI and materials research” (e.g. how we think European institutions should transform, e.g. to more flexibility) and “AI agents in Materials Science” (where we claim there is much need for a “science of evals”).
Paper: https://iopscience.iop.org/article/10.1088/2515-7639/ae9010
Clever Materials Hans: I was quite intrigued by an article by Leash Biosciences, where they showed that they can predict molecular activity by predicting author names. The idea there is that groups focus on certain molecular targets and scaffolds. And if you can recognize what kind of group might have made a compound, you may have a very good guess on what a molecule might be good for. I figured it would be interesting to see if this is also the case in materials science. Thus, I ran the experiments on some datasets I had at hand and found that also in materials science it is very easy to fool ourselves. I think this really highlights that we sometimes miss the scientific spirit of testing all the possible ways in which we could have fooled ourselves.
Paper: https://pubs.rsc.org/FD/article-lookup/doi/10.1039/D6FD00024J
MatBind: This paper is a product of the SOL-AI project. In this paper, we train a model inspired by the ImageBind architecture in which different materials science modalities are aligned in a joint embedding space. This architecture can then be used for retrieval tasks across modalities that were not paired in training.
ChemFuse: In chemical research, we have to deal with different flavours of data, that is, multimodal datasets. Traditionally, we model each dataset independently, so we do not utilize the collective information when the same structure is reported in another modality. As a result, we lack sufficient training data. To address this, we propose ChemFuse, a general approach inspired by the ImageBind training strategy, in which we align different modalities using contrastive learning in a shared latent space. After alignment, we use these encoders to obtain embeddings for the multimodal datasets, which lets us effectively utilize all of the available data to train ML models for downstream tasks. We demonstrate this across polymers, molecules, and MOFs, where the models consistently improve on downstream tasks compared to those trained on isolated single-modality datasets. In addition, we show that on a proxy task, models trained on the combined datasets produce stable rankings that recover the ground-truth ordering of the important features, whereas models trained on isolated datasets give unstable and misleading rankings.
Paper: https://chemrxiv.org/doi/abs/10.26434/chemrxiv.15006406/v1
About this newsletter: The paragraphs about the papers are written by group members individually and reflect their own opinions. The newsletter has been reviewed by other group members for factual correctness and edited by Gordan Prastalo and Kevin Jablonka. The editorial is written by Gordan Prastalo.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.