This is the website and blog of David Strohmaier.
If you model the acquisition of words that don’t exist in any language, what distribution do you assume for these words? For example, how often should they be used in false sentences vs. true sentences? This admittedly abstract issue motivated the paper “Transformers Learning Contrafactives: The Importance of Data Distributions”, 1 which I am presenting this Saturday (11. 07. 2026) at the 3rd…
I’m offering a philosophy of AI masterclass, “Concepts in Machines”, at the University of Zürich later this month (April 23–25, 2026). In this post, I briefly sketch my intentions for this course and collect extra materials not included in the syllabus . Sketch of Course Topic The course explores the role of concepts in neural models. What are the candidates for conceptual representations in LLMs?…
This post collects a set of notes on the Symbol Grounding Problem in the context of LLMs. In this context, the question is whether LLM systems have an appropriate connection to the world. The appropriate connections are presumed to establish that the representations and output of the models are meaningful. You can find a list of papers on the topic of symbol grounding on my page . My notes assume…
I’m presenting a poster at the upcoming CoNLL workshop in Vienna collocated with the main ACL conference . The poster summarises our paper on the Cambridge Dictionary Look-Up dataset and our preliminary results on it. If you are around, come check it out at either of the two poster sessions! You can find the paper online on OpenReview . The public portions of the dataset are available on the ELiT…
I’m currently reading Melissa Bowerman’s (2018) “Ten Lectures on Language, Cognition, and Language Acquisition”, and right at the start of the first lecture, she provides a fascinating remark on assumptions historically made about language learning. When we look into her remark and consider the larger consequences, we arrive at a warning about being too sure about one’s assumptions. According to…
I will give two talks about Philosophy and AI in the next few weeks (February 2025). Workshop on Contrafactives When: February 6-7, 2025 Where: University of Düsseldorf The workshop “How we do (not) talk about mistaken beliefs” will bring together researchers interested in the phenomenon of contrafactive predicates: ascription verbs that denote attitudes by which people get things wrong. According…
Being sufficiently arrogant to provide a list of all time favourite books on my website, it behooves me to pay some thought to the books included on it. Thus, I have recently revisited Plato’s Politeia , better known as The Republic . To fit more of the engagement into my holidays, I listened to an audiobook production of Politeia , while reading Julia Annas’ (1981) An Introduction to Plato’s…
2024 was my year of training Transformer Language Models (LMs). The hours I spent on building, training and debugging my models! I’ve been building and using NLP models using LMs as the basis for a while, but this year I spent much more time on training them from scratch, trying many variations and learning the hard way how brittle these systems are and how difficult they are to comprehend. A…
To my surprise, the recent 2024 EMNLP conference included multiple papers with a philosophical angle. One of them attracted my attention in particular: “Pragmatic Norms Are All You Need” by Reto Gubelmann, who seeks to address the Symbol Grounding Problem (SGP). As a first approximation, the SGP can be understood as the problem of endowing computational representations with a connection to objects…
I have two upcoming events to share: NLP4CALL Workshop I will present my paper “Semantic Error Prediction: Estimating Word Production Complexity” (co-authored with Paula Buttery) at this year’s NLP4CALL workshop. The presentation is this Friday 25th of October 2024. The paper is already available online . It argues that lexical semantic complexity in production hat its own distinct patterns and…
What’s not to like? Bob asked that question in many of our reading group sessions, usually to conclude one of his discussions of Hegel’s philosophy. Amazingly, he was able to make you think Hegel made sense. That was truly an achievement, even though one was rarely able to recapture that sense later on one’s own. With Bob out of the room, the spark seemed to vanish together with his slightly…
Last week, I had the pleasure to present a poster at the third conference on Experiments in Linguistic Meaning (ELM3). My poster presented the latest collaborative work with Simon Wimmer on the topic of contrafactives. Click here to get the poster as a PDF. 1 A paper fill follow. Walking interested conference participants through the poster, I liked to start as follows: We are investigating…
Introduction Due to their lack of success in resolving problems, philosophers like to think that they are failing productively. For example, a philosopher might suggest that while the problem hasn’t gone away, one sees it more clearly after the debate. Such insight is a consolation prize awarded to the readers for the failure of the authors. This blog post discusses one unsuccessful debate, a…
Daniel Dennett has passed away. While my own connection to Dennett was limited, I want to share a few of memories, because these moments spent with and around Dennett impressed me greatly. I met Dennett during my visit at Tufts University in 2017, when I was in the middle of my PhD in philosophy. I don’t believe I had been aware of Dennett’s presence at Tufts when I initially planned the trip, but…
Although I am acutely aware of the shortcomings of the current generation of AI models (transformer-models in particular), most of the criticisms of AI I stumble upon on the internet have become repetitive and lacking insight. I am not interested in picking on anyone here, I’m interested in reading more interesting criticisms. Therefore, I will provide a list of proposals for better AI criticism.…
Transformer models do not learn compositionality. That is, they do not acquire the ability to construct hierarchical structures from smaller units by repeatedly applying the same rules. 1 I speculated about this a while ago, in this post . More importantly, research has shown that while transformer models perform better on compositionality tasks than previous model types, they still cannot…
Now available, an open-access introduction to preference change ! When I got into the topic of preference change, a few years back by now, such an introduction was sorely lacking. I hope many readers find it of value. Much research on preference change remains to be done and I hope the readers can help with that! Michael Messerli and I have co-authored the book, which has been published by…
Prolog is not the tool of choice for most of NLP nowadays, but this didn’t always use to be the case. The unreasonable effectiveness of neural networks for most practical NLP tasks has led to this shift, since implementing neural networks in Prolog is rather awkward. But some ideas and theories from previous decades are still interesting, for theoretical exploration if not practical application,…
Compositionality is one of the long-standing challenges to neural NLP. I’m myself a bit sceptical that transformers really offer the kind of compositional processing found in human language processing . But even formulating the challenge can be a challenge. In it’s formulation by Partee (1995), the principle of compositionality states: The meaning of a whole is a function of the meanings of the…
Large transformer-based language models (LLMs) are performing well on a variety of tasks. This is a reason to reconsider our understanding of language and how humans process it. Especially those sceptical of neural network approaches face a challenge, such as Noam Chomsky and his followers, have come under pressure. They need to justify their scepticism about the abilities of neural networks in…
A while ago, I wrote up a number of papers (see this post ), all of which suggested that transformer models have partially converged with human language cognition. Using various correlational measures and predictions the literature leads towards the conclusion that transformers and human language processing resemble each other. The rate of publishing in this field being what it is, new papers have…
I recently had the pleasure to attend the Groningen Cognitive Modeling Spring School . This spring school is an annual event, but I’ve only recently heard of it and applied soon after. I’ve been interested in the interpretation of neural network models as cognitive models for a while, and so it was time for deeper engagement with the dedicated cognitive modelling research. The spring school had…
A common criticism of Transformer models, such as ChatGPT , BERT , and Bard , is that they only learn surface statistics. According to this criticism, the predictions by transformers are superficial, because they do not represent the underlying state. In the case of language, the models would only capture general co-occurrence, on which transformer LLMS are typically trained, but neither the…
Transformer LLMs, such as ChatGPT , BERT , and Bard , have sufficiently impressed the public that some have described them as AI minds. But is this ascription of a mind justifiable? 1 Betteridge’s law of headlines states: Any headline that ends in a question mark can be answered by the word no . I believe this law fails for the present blog post. The correct answer to whether Transformer LLMs are…
Warning: Speculative Content. Expect that parts of it will be proven wrong. The meaning of natural language sentences is compositional. The meaning of an expression \( \mathbf{E} \) syntactically derived from the sub-expressions \( \mathbf{E}_1, \mathbf{E}_2, \dots \) is a function of the semantic value of the sub-expressions Writing \( |\mathbf{E}| \) for the semantic value of the expression \(…
Once in a while, a publication resets the literature. There is a clear before and after them, as researchers cite the new publications, while neglecting the earlier literature upon which they built. The word2vec papers by Mikolov et al., which have been published about a decade ago in 2013, are an instance of this. 1 As can happen with such papers, the original motivations became overshadowed by…
I regularly use Zotero for managing my bibliographies. I then usually typeset my writings in LaTex, citing references with BibLaTeX. For exporting BibLaTeX .bib files, I recommend the “Better BibTeX” extension . But sometimes I prefer to attach citations to my clipboard rather than have to export an entire .bib file. For BibTeX, there exists a bibliography style that allows Zotero users to do so.…
Another year has passed and I want to take the opportunity to reflect on my research pursuits at a higher-level of abstraction. Hence, I will jot down notes on the lessons I cannot but take myself to have learned and sketch an intention of how to live as a researcher. A Conclusion I’ve Drawn, Rightly or Wrongly As a human being, it is hard to avoid drawing lessons from one’s life, even though one…
Introduction Neural networks with Transformer-architecture remain the state of the art in natural language processing (NLP). For many tasks the first approach is to throw some version of the BERT model ( Devlin et al. 2019 ) at it – a practice I’ve participated in (Yuan et al. 2021a , 2021b ). The success of the Transformer-architecture has raised the question how such models compare to language…
Disclaimer I am currently funded by money from Cambridge University Press and Assessment, which also stewards various English dictionaries. All opinions in this post are distinctly mine. The Unmasking of English Dictionaries (CUP, 2018) is on a mission to change lexicography forever and on the way it tries to insult as many lexicographers as possible. Its author, the linguist R. M. W. Dixon, has a…
Will machine learning (ML) solve natural language understanding (NLU)? A recent essay in The Gradient by Walid Saba argues that it won’t. I lack the confidence for either affirming or denying that ML will lead to NLU, especially without much further explanation of what we understand ML and NLU to be, but I am confident that Saba’s arguments are not of the knock-down kind. A part of me would like…
A debate about the state of analytic philosophy has been developing in the philosophical blogosphere over the last few months, started by Liam Bright’s pessimistic assessment of the state . In this original post, Bright described analytic philosophy as a “degenerate research programme” . No longer was there a shared paradigm, and instead philosophers either took a politically applied turn or just…
The Sudoku scene has undoubtedly been one of the pandemic winners. Thanks to the Youtube channel “Cracking the Cryptic” , its viral video on the “Miracle Sudoku” , and the many entertaining videos that followed, Sudoku puzzles with extended rule-sets have received widespread attention. That is a prime opportunity for Prolog aficionados like myself to show off the power of the language. Many Sudoku…
Lexical semantics investigates the meaning of words, but one might distinguish multiple levels of meaning for a single word. For example, the word “semantics” might have a very broad and a very narrow meaning at the same time, depending on how much contextual information we take into account. The relative dependence on contextual information created lexical levels of meaning. In this post, I will…
My recent blog post arguing why one should lean Prolog in 2021 made its way to the front page of Hacker News (HN), where it started a discussion with more than 100 comments. I’m glad to see that some saw value in my post and I want to respond to a few comments from this discussion. Given the number of comments, I won’t write an exhaustive response but instead focus on a few themes I care about and…
?- learn(prolog). Why should one learn Prolog in 2021? I should better have an answer to this question, because I will soon offer supervisions for a Prolog course. While I’m a personal admirer of this unusual programming language, students might rightfully demand a justification that goes beyond my preferences. Prolog certainly isn’t the most glamorous programming language to learn in 2021.…
What has 2020 brought? In this post I want to offer a selective reflection on my research career and its developments in 2020. I will take a perspective that is at the same time deeply personal and highly abstract. From my personal heights, I’ll gesture at the turns I’ve taken this year, note a few outcomes, and point towards my future commitments. Reorientations and Pivots My 2020 was…
Can we simulate basic logic operations, i.e. the operations of first-order predicate logic, using tensors? In his 2013 paper “Towards a Formal Distributional Semantics: Simulating Logical Calculi with Tensors”, Edward Grefenstette made some suggestions for such simulation. The paper’s motivation was to take a step towards combining distributional with formal semantics. I’ve explored this paper in…
I’ve recently been reading up on distributional representations, that is representation of meaning that are based on count vectors. They were the exciting technology before neural networks and the embeddings networks create changed the field of NLP. Nowadays we do not count token occurrences, but let Word2Vec or BERT models create representations. While they have decidedly fallen out of favour,…
In this blog post I share some preliminary musings on conceptual grain – how fine-grained concepts such as DOG and MAMMAL are – which have arisen from my work in NLP and specifically word sense disambiguation. The upshot is that we can develop multiple metrics of conceptual grain and that we have to address the question of what we want these metrics to do for us. The classic task of word sense…
The Canadian Journal of Philosophy has published my paper on what I call “Social-Computation-Supporting Kinds”. This paper is a first attempt to re-describe the role of computation in social ontology. I argue – in move I would self-servingly love to call “bold” – that there is a kind of social kinds which is distinguished by supporting social computations, that is groups implementing computational…
I am presenting at the 2020 Social Ontology Conference and because it is virtual, you can all watch it online. The conference website provides videos of all talks. In my talk , I discuss the notion of social-computation-supporting kinds. A longer paper exploring the idea will hopefully be published soon. The main advantage of the video is the excellent pixel art. There will also be a Q&A session,…
I’ve recently posted a short list of social ontology highlights from 2019 , but Kirk Ludwig sent me a much more extensive list. I present it here in rearranged form. Publications Tony Lawson: The Nature of Social Reality: Issues in Social Ontology (Economics as Social Theory) Angela Condello, Maurizio Ferraris, John Rogers Searle: Money, Social Ontology and Law Trish Reay, Tammar B. Zilber, Ann…
Social ontology, by which I mean a subfield of contemporary analytic philosophy, is a comparatively small enterprise so far. That makes gathering a best-of-2019 list difficult. There just aren’t that many great papers coming out each year, or other notable events. Here are five highlights I could find. Feel free to send me an email and suggest other contributions to the field. I might update this…
Over the years there have been a number of visualisations of Wittgenstein’s Tractatus Logico-Philosophicus. Most of them have made use of the tree structure Wittgenstein imposed on his text. With today’s web-technologies, these representations of the text can be excellent. In this post, however, I present a map of the Tractatus unlike any of these previous experiments. The picture shows me playing…
For the fourth year in a row, I will present a paper at a social ontology conference this Summer. After the last one in Boston, I thought it would be time to do something more ambitious. While my previous papers went well enough and led to two publications, they made relatively narrow arguments. This year in Tampere my claims will be much bolder. I do not want to give too much away, but I will…
In another life I read a lot of Hegel, now a mere side-interest of mine. Despite the assurances of my former supervisor Bob Stern to the contrary, Georg Wilhelm Friedrich Hegel’s work is infamously opaque. Making sense of his Phenomenology of Spirit poses a considerable challenge, and those who claim to understand him often end up with rather different readings. In my current life, I am finishing…
Two of my publications are finally out. Both of them are related to my PhD research into social ontology and the both investigate groups. The first one discusses group membership and argues that reducing it to mereological parthood plus further conditions is a viable option. The paper has an unusual history. Originally, I wrote another paper that argued the opposite conclusion, that is I tried to…