Introducing “What is LamaLab reading?”

Keeping up with the current pace of published papers seems close to impossible — especially if you work in a cross-domain field.

As a group working in such a cross-domain field, we found journal clubs suboptimal. Presentations take too long to prepare and cover too wide a range of topics for everyone to follow in detail. In addition, some on the team have joked that those working on LLMs would need a daily journal club.

So we asked ourselves: can we do something better? Can we still push ourselves to read papers carefully while respecting our own and each other’s time? We think we found an answer. Each month, we collect short, opinionated paper reports from around the lab and compile them into a single document. We decided to publish these as part of a “What Is LamaLab Reading?” newsletter series.

Here are our picks, in no particular order.

Pick: Accelerating Scientific Research with Gemini: Case Studies and Common Techniques

I am unsure how well the current LLMs and Agents live up to the hype and claims of their coding, engineering, and scientific prowess. While I use them myself, in my experience, they do not perform as promised, at least for the use cases where I need help. I always asked myself if I am the one doing something wrong and misusing them, and Google’s recent paper came just in time to answer that. The 150-page report provides a large set of case studies, techniques, and learnings from using LLMs to accelerate research. While the report spans the techniques from vibe coding to writing mathematical proofs, I feel like most of the workflows follow a similar pattern. While the models often make mistakes and are unable to do everything on their own, it seems that by mastering the iterative approach, providing crude overviews, hypotheses, or proofs, letting the LLM expand on them, and pruning the responses, one can come to a seemingly convincing solution. Distilling this report myself a bit, I would say that they show two types of use cases of LLMs that seem the most interesting. LLMs show themselves quite capable of generating many hypotheses, giving you ideas of possible experiments and what one could show with them. While, from personal experience, I have found that sometimes it does not generate a single hypothesis/experiment that I end up using, it gives a good overview of techniques, and it is much quicker than searching for these on the web. In the report, they show the second theme, the LLMs seem to be able to capture mistakes and open questions in one’s work. Although I have not incorporated this so much in my research, I do let LLMs judge my manuscript at the end, and I found it to work similarly to the previous example. They would provide a mixed quality of responses, and as long as you are able to identify what is most relevant, you could improve your work.

At the end, what is echoed, from both the report and my personal experience, is that they are as powerful as you are competent. If you have a good overview of the project and its related subfield, the ability of LLM to give you experiment ideas, write standard boilerplate code and proofs, provide some checks for writing, story and experiments makes it a powerful tool for conducting research; it can help you iterate quicker and focus more on the actual important tasks in your research project by removing some of the busy work.

Pick: Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference

This paper claims to be the first systematic investigation into how numerical precision affects reproducibility in LLM inference. It was particularly interesting to me, since I have recently started working on benchmarking projects.

The main finding is that inference can be affected by many factors, such as batching, different operator implementations, operator hyperparameters (e.g., block size in MatMul), collective operations in parallel settings, and parallelization strategies. As a result, factors such as GPU count, batch size, and GPU architecture can influence the output. The authors attribute these discrepancies to floating-point arithmetic on GPUs. Floating-point arithmetic is non-associative, meaning that (a + b) + c ≠ a + (b + c) due to finite precision and rounding errors. Although these errors are individually small, they tend to accumulate across the network. This can lead to significantly different results even when using greedy decoding and other deterministic settings. The fundamental cause of nondeterministic outputs is often the small gap between competing logits, which makes token selection highly sensitive to minute numerical fluctuations.

In my opinion, investigating the accuracy of benchmarking results with respect to experimental setup is extremely important but largely overlooked. This paper demonstrates that for small models, the standard deviation on some benchmarks can be as high as 9%, which is substantial. Unfortunately, the paper does not re-evaluate all major benchmarks across a wide range of models, but instead highlights the importance of accounting for numerical precision effects when interpreting results.

Pick: MathArena: Evaluating LLMs on Uncontaminated Math Competitions

This month, a paper that I found interesting was MathArena, which proposes a new benchmark for math, similar to SWE-Bench or LIVEBench, that gets automatically updated over time. The key points of the work are:

• The authors propose using the annual math competition problems as sources for the benchmark dataset. Previous benchmarks (AIME2024 and 2025, for example) already base their corpus on this, but they remain static over time, resulting in data contamination.

• In MathArena, an automated workflow is proposed that combines LLM calls and human reviewers, resulting in a corpus of 160 tasks that they plan to update as future competitions take place.

• A massive evaluation (50 models) is conducted, revealing that frontier models saturate benchmarks that suffer from data contamination, but in other, more recent competitions, they struggle, especially when problems are proof-based.

Some stuff that I find particularly interesting:

• I believe that the self-updating benchmarks are a great thing for the field, as they push companies to train models into new edge cases. Regarding this, I believe SWE-Bench helped a lot in improving the coding abilities of the models.

  • However, I am not sure if MathArena is such a benchmark. While the workflow is well-defined, it includes a lot of human intervention (e.g., review of the parsing of the problems, proof-based evaluation), which makes it difficult to scale.

• I do not know why they call it Arena, as it is not an Arena. There is no face-to-face comparison of the models (resulting in a leaderboard based on ELO scores), as the scoring is completely based on the final answer or proof of correctness.

• The data contamination (Figure 4) analysis that they do is difficult to assess, as those results could be due to the continuous improvement of the models.

The work is mainly based on Math competition problems. People have raised concerns that such problems do not measure complete math understanding but measure speed or other attributes (something related to one of the papers below). I would love to hear what the authors think about these opinions of well-reputed researchers.

Pick: Ranking over Regression for Bayesian Optimization and Molecule Selection

I selected this article because it addresses the real-world problem where surrogate models would fail when you have sparse data along with noisy chemical landscapes. What is really interesting to me is the empirical proof that the model’s accuracy is the R² score, which has near-zero correlation; this makes sense when you don’t have a lot of data. Instead, the authors showed that the ranking ability of the model, measured by the Kendall tau correlation, drives finding the top-performing molecule.

To me, this shifts the machine learning objective. Instead of predicting an exact value in a regression task, the goal becomes giving priority to better candidates. The authors leverage deep learning models trained with pairwise margin ranking loss to learn the relative ranking of the molecules.

The advantage I see in using rank-based Bayesian optimization (RBO) is that it also becomes effective in handling “activity cliffs,” meaning if there is a structural change in molecules that would have a huge variation in their properties. While normal regression models try to fit the entire data irrespective of outliers and eventually fail in “activity cliff” regions, the ranking model simply learn the relative loss of fitness.

The downsides of this approach are that training a pairwise ranking model requires comparing combinations of data, which directly increases training time. Furthermore, RBO might be less suitable for multi-objective optimization due to Pareto-optimal tradeoffs. Finally, if the chemical dataset happens to have a highly smooth functional space, traditional Gaussian Process regression models still perform better.

BO package not related to this paper, but useful https://github.com/emdgroup/baybe. The package is from Merck.

Pick: Measuring What AI Systems Might Do: Towards A Measurement Science in AI

We have been thinking a lot about measurement, which is essential to science — without it, there is no science. Measurement itself can be a rigorous discipline (e.g., metrology), and many fields have developed dedicated subfields for measuring things correctly (e.g., analytical chemistry).

There are typically many ways to measure something, but in the machine learning community, this is often reduced to “benchmarks.” These benchmarks are interesting because they tend to be what measurement scientists would call “pragmatic” measurements — defined by the measurement process itself. IQ, for example, does not exist as an independent physical property in the world; it is simply what an IQ test measures. “Representational” measurements like mass, by contrast, correspond to properties that exist in the world independently of how we measure them. In practice, this means it is often unclear what a benchmark actually measures. Benchmark developers rarely articulate this explicitly, and construct validity — whether a measurement measures what it is supposed to measure — is often left undefined. We discussed this in our review, “Lessons from the Trenches on Evaluating Machine Learning Systems in Materials Science.” It is also something others, such as NIST, discussed.

Voudouris et al. offer a new perspective on improving how we measure machine learning systems. They propose that many things we care about — capabilities — are “dispositional” properties: properties that dispose a system to behave in a certain way. Fragility, for instance, is a property that disposes an object to break under force. This already hints at a causal structure: we assume that force causes fracture, and so we measure a system’s behavior by systematically varying force.

The authors suggest applying this same logic to machine learning evaluations. We should evaluate

the probability of some behavior v of a system (pi) under a change in context (theta). One appealing feature of this framework is that it demands transparency and systematicity — we must clearly define the system (pi) we are targeting and articulate how changes in some property (e.g., task complexity measured by the number of arithmetic operations) cause changes in behavior.

I find the article valuable because it gave me new language for things we had independently started doing. Some readers might take it as an argument that benchmarks are useless — I don’t think that’s right; they can be quite useful in the right context — but it is certainly worth highlighting that many measurements currently used in machine learning look very different from measurements in the natural sciences.

Pick: Energy Landscape and Stability in Random Heteropolymers: Somewhere Between Protein Folding and Plastic Miscibility

My pick this month is a paper by Jin et al. on energy landscapes and structural stability in random heteropolymers (RHPs), which investigates how synthetic, protein-mimetic polymers collapse and how their behaviour sits conceptually between protein folding and classical polymer physics.

Summary:

• The authors study a methyl methacrylate–based random heteropolymer (Tetrapolymer) using ~14.6 μs of atomistic molecular dynamics simulations combined with umbrella sampling. They propose a hierarchically rugged free energy landscape with multiple metastable minima separated by substantial energy barriers (~8 kT).

• Unlike proteins, which typically fold into one dominant native structure, a single RHP sequence collapses into an ensemble of distinct, stable globular conformations, with no interconversion observed on the microsecond timescale.

• By analyzing local segments, they show that identical local sequences can adopt different local conformations, indicating that RHPs do not obey a deterministic sequence → structure relationship.

• Using XGBoost and SHAP analysis, they identify key stabilizing interactions. Structural stability is linked not only to specific residue–residue contacts (protein-like behaviour), but also to position-nonspecific backbone–backbone and backbone–side group contacts, which resemble design rules from polymer miscibility (Flory–Huggins χ).

• The polymers exhibit chemically heterogeneous microenvironments, supporting their potential as ensemble-based protein mimetics rather than single-structure mimics.

Why I like the paper:

• I like that the paper clearly shows how polymer morphology is highly complex and governed by many small, coupled parameters, rather than by a simple sequence-to-structure rule. This is especially relevant when thinking about inverse design of copolymers targeting specific properties.

• On the downside, the study focuses on a single chemistry and a single representative sequence, so the generality of the conclusions is argued but not systematically demonstrated.

For me, this work further raises the fundamental question of how inverse design in polymer systems can be approached in practice. If identical local monomer sequences can lead to different structural outcomes even under controlled simulation conditions, coupling this with the inherent sequence distributions and synthetic variability of real polymerizations makes predictive inverse design substantially more complex.

Pick: Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

Summary: A paper from October, 2025 by Zhang et al. The authors introduce verbalised sampling, a prompting approach meant to prevent mode collapse in LLMs, originating from post-training alignment. There is plenty of evidence, also from psychology, that information more aligned with existing mental models tends to be less critiqued.

The authors provide a model-agnostic method that does not require logit access. Their method can be used in three ways: standard (just ask for probabilities), chain-of-thought verbalised sampling, and multi-turn. The typical prompt: “give me N (e.g., 5) of M (e.g., jokes about coffee)” underperforms the verbalised probability approach on both quality and semantic diversity.

Key insights for scientific applications: Genuine scientific insights have been a challenge for frontier models, perhaps originating from that same post-training alignment, which causes mode collapse and, as a result, unidimensional thinking when the prompting technique is trivial. Maybe this approach can facilitate the model to “recall” more from its pretrained weights. The authors provide non-trivial evidence of diversity improvements using the random number generation as an example.

Weaknesses: One key weakness is perhaps the ability of this method to work well only in cases where multiple solutions are acceptable. Moreover, the model does not compare to more recent prompting/sampling strategies, such as for example the diversity-risk generation parameter optimization.

An additional shout-out for DeepSeek-OCR-2, which introduced a cool modelling approach for vision.

Pick: SMILES all around: structure to SMILES conversion for transition metal complexes

Assigning SMILES to transition metal complexes (TMCs) is not a trivial task. Here, the authors extend the existing xyz2mol pipeline for organic molecules to mononuclear TMCs. This extension is still limited to closed-shell complexes with a total charge of +1.0, 0, or -1.0. This is a severe constraint, in my opinion.

This pipeline takes atomic coordinates of the complex from CSD and produces a SMILES from the 3D structure. They try to do it in 3 different ways: 1) Using distance heuristics and cheap extended Hückel calculation to estimate the ligand charges and bond orders, 2) DFT + NBO (natural bonding orbitals analysis), and 3) cleaning and sanitizing the deposited SMILES from CSD. They also standardize all metal-ligand bonds as “dative”.

Results on the tmQMg dataset (60,799 complexes):

• Hückel: 59,878 valid SMILES

• DFT: 46190 valid SMILES

• CSD: 53,320 valid SMILES

In ~41K complexes, all three methods succeed, but only ~70% of the generated SMILES are identical across all three. This agreement increases if the ligands are first disconnected from the metals and their resonance structures are considered, meaning that a significant portion of the disagreement is directly due to metal coordination. Ambiguity in assigning ligand charges, as well as complicated haptic ligands, contributes to the issue.

The authors then use the generated SMILES to train ML models (RF, FFNN, GCNN) for property predictions: dipole moment, polarizability, and HOMO-LUMO gap, using fingerprints and rdkit-derived graph features. The results are ok as a baseline. It is not surprising that dipole moments, which need 3D geometric information, cannot be reliably predicted from SMILES.

Overall, I think the goal here is very important, and it is still a step forward in extending text-based representations to inorganic chemistry, but I don’t find the actual results impressive!

Pick: C-MOP: Integrating Momentum and Boundary-Aware Clustering for Enhanced Prompt Evolution

This paper proposes an automatic framework for optimizing discrete prompts. Existing methods typically rely on iterative rephrasing and validation feedback, but they often suffer from unstable updates and noisy evaluation signals, especially when easy samples dominate the validation set. To address this, the paper introduces Boundary-Aware Contrastive Sampling (BACS), which clusters validation examples and gives more weight to boundary clusters where prompts differ most in performance. It also proposes Momentum-Guided Semantic Clustering (MGSC), which accumulates historical semantic update directions to stabilize the optimization and reduce oscillation across iterations.

What I think:

• What I found interesting is that this paper proposes a novel way to explore a “larger” prompt space. It selects the update direction from multiple candidate directions, weighted by semantic clusters and historical momentum.

• However, the optimization is still restricted to a single branch, since only one prompt is retained after each iteration. Although multiple candidate directions are sampled, the overall trajectory remains single-path. The difference is that the selected direction may be more reliable compared to some other methods.

• The stopping criterion of this framework is based on a predefined hyperparameter n, which specifies the number of iteration loops. The paper does not provide a clear guideline on how to choose n. In that case, can we really claim that model A (0.90) performs better than model B (0.87) after 5 iterations?

We might sometimes be perceived as a group that is “hyping” LLMs. These picks underscore that this is not the case — we cover a broad range of interests.

Perhaps they also show what makes our team strong: our colleagues think in ways we do not. They made us look at papers we otherwise would not have. This increases the diversity of our thinking and, hopefully, leads to better research.

If you are interested in joining us, reach out.

• Clever Materials: When we use “meaningful” descriptors, the model will learn something meaningful.” This is something that we easily assume, but often is not the case. We show that we can predict bibliographic descriptors (authors, publication year, journal) from the chemical descriptors and use those bibliographic fingerprints to predict materials properties. In some cases, this is not worse than directly using the descriptors. That means that with only performance metrics, we cannot say if the model is learning something meaningful or just exploiting some spurious shortcut pattern (e.g., temporal trends or “styles” of different research groups).

• PERLA: Machine learning is often portrayed as one of the main techniques to accelerate materials discovery. However, ironically, in fast-paced fields, the data the models are trained on is quickly outdated. For example, there might be new discoveries in a field, like the introduction of a new materials class, that are not represented in existing databases. Models based on the old databases, hence, cannot make meaningful predictions (machine learning practitioners call this covariate shift). To address this, we introduce living databases: Databases that self-update by monitoring the literature and extracting the data with LLM. With a large team - amongst others from FAIRmat and HZB - we applied this concept to create a living database of perovskite solar cells.

• General-purpose models review: ML models with general purpose have become widely adopted in recent years. In this review, we attempt to describe their utility for the chemical sciences and present several different data collection and training paradigms. Moreover, in the context of each step of the scientific process, we describe its limitations (more fundamental) and open challenges (what can we already attempt to solve?).

• ACE-Mol: Deep learning models promise to produce powerful molecular representations. While they are more generalizable than their fingerprint counterpart, they still remain static. We try to tackle this with ACE-Mol, our model adds an extra dimension to the molecular embeddings, by providing adaptability. ACE-Mol’s embeddings change based on the downstream application; they adapt to the task at hand.

• WhatCounts: An algorithm, by definition, should be invariant to the content that is being applied to. In WhatCounts, we show and measure that the accuracy of LLM-based counting functions greatly depends on the semantic content of what is being counted. With careful ablations, we discard potential factors, such as token count or aggregation-identification issues. We further show that such sensitivity to the semantic content varies unpredictably across equivalent training regimes and datasets, and it is not reduced between different reasoning efforts. And even more surprising is that LLM-based agents continue to suffer from such instability across different semantics, despite being able to run Python tools.

About this newsletter: The paragraphs about the papers are written by group members individually and reflect their own opinions. The newsletter has been reviewed by other group members for factual correctness. The editorial is written by Gordan Prastalo and Kevin Jablonka.