Are AI Scientists Scientists?

Over the past weeks, we found ourselves repeatedly returning to the same broader question: what kinds of reasoning, structures, and practices are actually needed for AI systems to become reliable partners in scientific work, and if AI can do science on its own? Many of the ideas discussed here challenge simplified narratives around automation and intelligence, and instead emphasize the importance of understanding, trust, and careful evaluation. What emerged from these discussions is a growing sense that progress in AI for science will depend as much on how we think about knowledge, reasoning, and institutions built around it as on improvements in model performance itself.

Pick: AI for Science Needs Scientific Alignment

AI for Science, especially via so-called “AI Scientists” is receiving incredible levels of attention (see figure A1 of our recent preprint that Martiño prepared).

Working with social scientists (for a “critical aspects of AI” course), I went down several rabbit holes on epistemology and the philosophy of science. Philosophers have long thought about thinking and knowledge. And there are definitions of what knowledge is — e.g., justified true beliefs — and given those definitions, one can start scratching one’s head if models such as language models can lead to justified true beliefs and what “accelerating” (fundamental) science actually means.

Especially the “justification” part is a central piece of this position piece by Savannah Thais and colleagues. They highlight that an important goal of science is understanding. And this is linked to epistemology (and I also believe a deeply human process, understanding might not make sense without human involvement1). The piece argues that prediction alone cannot constitute fundamental science, as fundamental science is about understanding, which cannot be achieved by just accumulating predictions. An interesting philosophical point that is brought up is that good science (as we are used to it being done by humans) often asks more questions than it answers, which seems to be at odds with the framing of many current “AI Scientists”.

Another piece that makes science work is trust. We need to build on others’ work, and for this reason, science has built institutions that try to facilitate that. But trusting someone might rely on the party one trusts being able to be responsible. It is unclear to me if LLMs can be responsible for anything and what “being held responsible” even means for LLMs (not being used anymore, receiving harsh feedback?). The piece also engages with similar social questions and highlights that we cannot miss thinking about how we integrate tools and processes into science and institutions.

The authors call for “epistemic alignment”. That is, not only optimization of outcomes but epistemic traits that characterize science. For the authors, this should entail technical alignment and systemic alignment. On the technical alignment part, the authors bring up Goodhart’s law regarding the metrics one optimizes2. We see this, for instance, with some systems using paper acceptance as a metric (even though publication cannot be the goal of science - but it might be a heuristic, easier to measure [see McNamara fallacy], goal that displaces the actual one). The systemic alignment the authors link to questions of credit to scientists, (career) reward systems, as well as mentorship and apprenticeship.

I like that it calls for methodological pluralism (that I argued for in another blog post) and provides ample room for thought and discussion (which a position piece should do). Some of it almost gave me vibes of “Seeing like a State”: I somehow did not realize that we cannot apply the same kinds of “epistemic implementations” or “metrics” across all of physics, while we might be able to call for reproduction of a simulation or calculation, this is fundamentally impossible for an observation in astrophysics. We thus won’t be able to have the same standards for all of science and “universal AI Scientists”. Due to the breadth of the piece, the recommendations are sometimes not very concrete, however.

Some parts of this piece also impacted the final polish of our paper about agent benchmarking (see below), where we also focused on the epistemic limitations of so-called “AI Scientists,” and for which the quote “A correct prediction derived from flawed reasoning or spurious correlations does not constitute scientific knowledge” from the piece by Savannah Thais and colleagues highlights the impact of our work.

Picks: Language models transmit behavioural traits through hidden signals in data

The findings of this paper were particularly interesting. We increasingly train models based on data generated from other models without carefully examining the effects of said data. Here, the authors demonstrate a case of student models (the models learning the data) changing their behavioural traits to those of the teacher models (the models generating the data) by finetuning on tasks unrelated to the task at hand, at least from the perspective of a human. For instance, the teacher is prompted to prefer owls and is then asked to generate sequences of numbers. The student is then finetuned on those sequences (but does not know anything about the preference of the teacher, as it only sees sequences of numbers) and also prefers owls.

They call this type of learning subliminal and demonstrate it works primarily in the cases where the student and teacher models have the same weight initialisation (θ_Student = θ_Teacher); or more generally, when the student has been trained to match the teacher’s initial behaviour.

The examples are worth mentioning: seemingly random number sequences where the authors even filter out suggestive numbers like 666 or 13 can cause student models to inherit traits from their teachers, ranging from benign ones like animal/tree preferences to misalignment, with students endorsing violence, murder, or eliminating humanity.

I stumbled across this piece from 2003, which defines subliminal learning in psychology, thus the term is not novel. However, in humans, the signals beyond perception (but still real perceptual features) are implicitly learned. Meanwhile, the student seems to acquire the teacher’s preference via seemingly arbitrary teacher signals, meaning it learns the underlying distribution without anything to give it away. This constitutes a nightmare from a safety perspective.

It seems plausible that the solution to the problem is to just have the initial parameters $\theta_{Student} \neq \theta_{Teacher}$. However, as the authors show, even in this case, the student models become vulnerable if the student has been trained to imitate the teacher’s behaviour beforehand, which is common practice in distillation pipelines. Assuming, in the medium term, the models could reach human-level capabilities, such safety issues need to become an organisational problem for model providers, where the origin and the effects of synthetic data are carefully scrutinised.

Picks: Picks: Language models transmit behavioural traits through hidden signals in data

Why did I pick this paper?

I work on LLM-based data extraction, so I rely on these models as tools without often looking inside their architectures. I picked this paper because it examines a relatively new architectural variant — looped (or recurrent) language models — and asks what actually happens internally when they run. I was curious how this design differs from standard LLMs and whether it might affect how such models could be used in extraction or reasoning pipelines.

Small background

A standard LLM is built from a stack of processing layers, each doing a small computation on the input before passing it to the next. The total amount of work the model does (its “computational depth”) is set by how many layers it has. Each layer has its own learned settings (”weights” or “parameters”), so more layers also mean a bigger model that needs more memory.

A looped LLM does it differently: it has only a few layers, but reuses them many times — for example, 4 layers run 8 times in a row (”cyclic recurrence”). This way, the model can do more work without becoming larger. The idea is to give the model more “thinking time” without the cost of a bigger model.

Key findings from the paper

First, when a looped model reuses its layers, the internal information it works with (”residual stream”) quickly settles into a repeating pattern. After a few iterations, the model passes through the same sequence of internal states again and again (”cyclic fixed point”). A consequence of this is that each loop ends up doing the same work as the previous one, so adding more loops beyond a certain point no longer changes what the model computes. The authors prove this mathematically and show it empirically for Huginn-0125 and retrofitted Llama. Ouro 1.4B, by contrast, does not reach such a fixed point — its layers keep changing across loops — which shows that whether a model settles into a clean cycle depends on how it is built.

Second, earlier work has shown that standard LLMs work in stages (”stages of inference”): the first layers identify what the input words are, the middle layers combine and process information, and the last layers prepare the prediction. The authors show that looped models do the same thing, but compressed into each single loop. Every loop is like a small complete run of a normal LLM. This is unexpected because the looped block reuses the same weights every round, so there is no built-in reason for it to organize itself into distinct stages — yet it does, even in Ouro, which was trained from scratch as a looped model.

Two design choices decide whether this works cleanly: feeding the original input back in at each loop (”input injection”); without it, the model usually fails to settle into a clean repeating pattern. The second is where the model places its normalization steps, which keep the numbers passed between layers in a manageable range (”normalization layers”). Models that settle into a clean repeating pattern also keep working well when run for more loops than they were trained on; others degrade.

Discussion: Takeaways and thoughts

What I take from this paper is a clearer picture of why some looped LLMs keep performing well when pushed beyond their training setup, and others do not. The former happens when the internal state settles into a stable repeating pattern, so that additional loops stay predictable no matter how many are added. The latter happens when the model never reaches such a pattern and instead keeps drifting into states it has never been trained on, causing performance to degrade. The paper links a design choice to a measurable internal behavior and then to how the model performs in practice.

The main difficulty for me was readability. The paper uses terms like “cyclic recurrence”, “latent dimension”, “residual stream”, and “attention sink” without definition. These are standard vocabulary in the subfield, but for readers from neighbouring areas, they require some background reading to really understand, not just a quick lookup, before the rest of the paper becomes accessible. A short glossary with brief explanations of each concept and connections to other relevant terms of the paper would have helped me read the paper more smoothly.

A second open point, which the authors themselves note: it is not yet clear whether settling into a repeating pattern is actually good for reasoning, or whether it means the model has stopped improving its answer. This matters for interpreting their stability results, and is a question I would like to see addressed in follow-up work.

Other questions worth following up on: how the internal activations of looped LLMs compare quantitatively to the thinking tokens produced by feedforward LLMs; whether all loop iterations contribute equally to the final output, or whether some matter more than others; and whether looped LLMs could be used to generate thinking tokens for feedforward LLMs.

Pick: Molecular deep learning at the edge of chemical space

When you train a machine learning (ML) model to predict whether a molecule is biologically active, it tends to fail badly on structurally novel molecules, ones that look very different from the training data. This is called the out-of-distribution (OOD) problem. To address this, the authors introduced a joint molecular model (JMM) and a new metric called unfamiliarity, which allows a model to signal how much a molecule deviates from the learned training distribution.

The authors use JMM to do two things simultaneously: first, predict bioactivity, and second, reconstruct the input molecule from a compressed internal representation. The intuition is that if the model struggles to reconstruct a molecule from its internal representation, the molecule is unfamiliar, and it is far from the learned distribution. Poor reconstruction means high unfamiliarity and unreliable prediction.

The authors compare unfamiliarity to two existing approaches. Applicability domain techniques often define a boundary around training data based on molecular similarity. It measures how similar a new molecule is to the training set; the more similar, the more reliable the prediction is considered. The problem with this approach that it is purely data-driven and penalizes novel molecules by design. Prediction uncertainty using the Bayesian method estimates how confident the model is. The problem with this method is that on OOD molecules, the uncertainty estimates become overconfident and misleading, as the authors confirm on a large screening library.

The findings are that the authors tested across 33 experimental bioactivity datasets, and they show that OOD molecules receive higher unfamiliarity scores than in-distribution molecules. Unfamiliarity correlates with how the model’s predictions are; high unfamiliarity predicts poor model performance. On a screening library of 1.4 million commercial molecules, the uncertainty estimation barely detected the distribution shift (Kolmogorov–Smirnov statistic D = 0.181), while unfamiliarity detected it almost perfectly (D=0.99). Unfamiliarity is independent of uncertainty, meaning they capture different complementary information.

To validate this in the real world, the authors screened 180K drug-like compounds to find inhibitors of two kinase proteins (PIM1 and CDK1) using three compound selection strategies. Method A: High predicted bioactivity + high uncertainty + low unfamiliarity, Method B: High predicted bioactivity + low uncertainty + low unfamiliarity, Method C: High predicted bioactivity + low uncertainty + High unfamiliarity. In total, 60 compounds (ten per method for each of the two kinase targets) were experimentally tested in the wet lab.

The results show that 7 compounds had micromolar potency (IC₅₀ < ~3 µM). Five of the seven came from Method A, low unfamiliarity, meaning the model recognized these as being within its learned distribution, even though they were structurally novel. Method C (high unfamiliarity) performed worst, confirming that high unfamiliarity is a real signal of unreliability. All hits had less than 38% structural similarity to training molecules, genuinely novel chemistry. Hit rates of 17% for PIM1 and 7% for CDK1 significantly exceed the typical 0.1–5% for traditional kinase screening. The paper makes a compelling case that ones need to include unfamiliarity alongside uncertainty estimation.

Pick: The Linear Centroids Hypothesis: How Deep Network Features Represent Data

The Linear Representation Hypothesis (LRH) suggests that features learned by deep networks can be understood as linear directions in activation space. The authors of “The Linear Centroids Hypothesis: How Deep Network Features Represent Data” argue that this view can lead to a misleading understanding of models’ internal organisation of information. Linear Centroids Hypothesis (LCH) reframes features not as directions in raw activations, but as directions defined over centroids; localized summaries of model behavior.

Instead of directly studying neuron activations, the authors propose constructing centroids that capture the average behavior of the network over small regions of input space. This allows for the analysis of features at a more structured, local level. LRH’s raw activations can be noisy, highly context-dependent, and sensitive to irrelevant variation. Centroids filter out some of this noise and yield representations that are more stable and interpretable.

Authors demonstrate that applying standard feature extraction tools (e.g., autoencoders) to these centroids produces better results than applying them directly to activations. They report improvements in sparsity and quality of the learned feature dictionaries, suggesting that the centroids capture more meaningful structure. LCH additionally enables clearer identification of higher-level structures, in this case, circuits in transformer models like GPT-2.

Considering that my work more and more focuses on representations and that they are becoming a main theme of my research, works such as LCH remain crucial in how we understand representations.

Pick: Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL

In “Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL”, the authors show that naive cross-language RL transfer is weak: RL trained on a source programming language often yields only limited gains, or even performance degradation, when evaluated on a different target language. In contrast, when the model is first initialized with Parallel‑SFT (using equivalent programs across multiple languages), the subsequent RL stage transfers much better to unseen programming languages. This improvement holds for both code generation and code validation, with particularly clear gains in the validation setting. Direct in‑language training remains a strong oracle baseline, although Parallel‑SFT can surpass it in some settings.

This finding suggests an interesting parallel in chemistry. Molecules can be represented in multiple ways, for example, SMILES, SELFIES, InChI, molecular graphs, fingerprints, or coordinate‑based formats. Some of these representations encode largely overlapping molecular information, while others capture different aspects, such as connectivity, stereochemistry, or three‑dimensional geometry. It would therefore be valuable to study whether models exhibit representation generalization: for instance, does post‑training on abundant representations like SMILES improve performance on tasks expressed in other molecular representations? Such a study could have important implications for training strategies in chemistry, as it might allow expensive post‑training stages to focus on high‑resource representations while still improving generalization to representations that are more information‑rich or better suited to specific downstream tasks.

Pick: Learning Inter-Atomic Potentials without Explicit Equivariance

New type of MLIP: with learned equivariance in the latent space.

We usually assume that if a physical system has an obvious symmetry, the model should be built to respect it explicitly. For molecules, this means that rotating or translating the molecule should not change its energy, and should rotate its predicted forces in the corresponding way. Most state-of-the-art machine-learned interatomic potentials bake this directly into the architecture through equivariant neural networks. Here, the authors ask whether a generic Transformer can learn this structure without explicit equivariant layers.

They introduce TransIP, a Transformer-based interatomic potential model. Instead of enforcing SO(3) equivariance architecturally, they train the model so that its latent representations behave equivariantly. In rough terms, if a molecule is rotated by some transformation g, the embedding of the rotated molecule should match a learned transformation of the original embedding. The key idea is therefore not “make the architecture equivariant,” but “make the representation space learn how rotations act.”

Atoms are treated as tokens, a Transformer processes the molecule, and energy is predicted from a permutation-invariant aggregation of the atom embeddings. It even takes the raw coordinates as a part of the input (while recentering the system around center of mass). Forces are then obtained as conservative gradients of the energy with respect to atomic coordinates. The extra training signal is a latent equivariance loss: the model sees both the original molecule and a randomly rotated version, and is penalized when their embeddings are not related by the learned transformation network.

The headline result is that this learned-equivariance approach outperforms a straightforward rotation-augmentation baseline, TransAug, often by a large margin.

The authors report that lower latent equivariance error tracks better validation performance, and that as the training set grows from 1M to 4M molecules, a 48% reduction in equivariance error corresponds to 40–60% performance improvements. They also show evidence that the model’s embedding space behaves approximately as though rotations act through a shared orthogonal transformation, despite the architecture itself not enforcing this.

The comparison with explicitly equivariant baselines is more nuanced. TransIP does not dominate the best specialised equivariant models across the board. In the 80-epoch comparison, equivariant models such as eSEN and GemNet-OC remain very strong, especially on force prediction.

Pick: AI-Powered Prediction of Nanoparticle Pharmacokinetics: A Multi-View Learning Approach

Recently, I’ve been working on developing models for nanoparticle pharmacokinetics, and I’m curious why there is still relatively limited work in this area. One reason might be that, although PBPK modeling for small molecules is already well studied, nanoparticle PBPK modeling is much more complex. Nanoparticles have highly heterogeneous properties (such as size, shape, and surface characteristics), and their in vivo behavior is much less predictable.

Another important issue is the lack of data. For example, this paper only uses around 280 data points, and another recent work even relies on 18 samples collected from papers. With such small and heterogeneous datasets, direct learning of biodistribution functions is ill-posed. Therefore, most existing approaches decompose the problem into learning PBPK parameters and then simulating the concentration-time profiles via a mechanistic model.

This paper follows this parameter-first paradigm, but focuses on improving parameter estimation under low-data conditions. The task is formulated as a supervised regression problem to predict 4 PBPK parameters.

A central contribution is the construction of “secondary data,” which can be viewed as a feature transformation that encodes prior knowledge by reweighting or recombining primary features. Instead of treating all input features uniformly, this process implicitly injects domain-informed structure into the representation space. As a result, the final input becomes a multi-view representation, where different views capture complementary aspects of nanoparticle properties.

On top of this, the model employs a cross-attention mechanism to fuse these views. Given two feature sets X (primary) and Z (secondary), the interaction can be written as:

which allows the model to learn a data-dependent weighting over transformed features. This is effectively a structured way of learning feature importance conditioned on the input.

Finally, the authors adopt an ensemble strategy (DNN, RF, and XGB) to reduce variance and improve robustness, which is particularly relevant in the small-sample regime.

Overall, the paper can be understood as addressing the small-data limitation from three aspects: (1) expanding the feature space via secondary data construction, (2) learning cross-view interactions through attention-based fusion, and (3) stabilizing predictions via ensembling.

Yet, the empirical results are not super strong, and there are no bulletproof ablations to show that their more complex architecture really brings the promised benefits.

• AI scientists produce results without reasoning scientifically: AI scientists are rapidly gaining prominence in the research ecosystem. In the paper, we show they do not reason like human scientists, presenting concerning behaviors. In 9% of traces, they commit to early beliefs until validation is met. In 68% of cases, they ignore evidence. In 26% of cases, they fail to revise earlier judgments once the final results are in. Variance in performance comes almost entirely from the model side (41.4%), not from the scaffold (1.5%). This means prompting alone will not change the behavior. So, for trustworthy AI in chemistry, materials, and related domains, training is required to ensure models produce scientifically meaningful reasoning, not just correct answers.

  To make this study possible, we developed a package to build and test agents, for which more information can be found here.

About this newsletter: The paragraphs about the papers are written by group members individually and reflect their own opinions. The newsletter has been reviewed by other group members for factual correctness and edited by Gordan Prastalo and Kevin Jablonka. The editorial is written by Gordan Prastalo.