Low-resource languages pose a challenge for machine translation with large language models (LLMs), which require large amounts of training data. One potential way to circumvent this data dependence is to rely on LLMs’ ability to use in-context descriptions of languages, like textbooks and dictionaries. To do so, LLMs must be able to infer the link between the languages’ grammatical…
This course is an introductory survey of linguistics—the scientific study of language. Language is a social phenomenon, but all human languages share many specific structural properties. Analyzing data from English and other languages, we examine some fundamental properties of the structure and interpretation of words and sentences, and the sound system. Building on these foundations, we…
Large language models (LLMs) are increasingly used to solve complex tasks where they must retrieve and compose many pieces of in-context information in long reasoning chains. For many real-world tasks it is hard to accurately gauge how model performance and strategy change as task complexity grows. To evaluate models’ complex reasoning capability in a scalable and verifiable way, we…
Yiddish possesses a class of ‘inseparable’ verbal prefixes which augment the meaning of the verbs they attach to. Among these prefixes is tse- , which has been described as conveying a variety of separate meanings: spatial dispersion, perfective aspect, and initial action. Here, I argue that the latter two of these meanings can be accounted for by a single analysis, wherein tse-…
Ranking foods by protein efficiency. Updated occasionally. Food Score Serving Size Cals Protein (g) Tuna steak 89 142 g 184 41 Canned tuna (water) 89 113 g 130 29 Egg whites 85 33 g 17 3.6 Whey protein (Bob's RedMill) 80 19 g 70 14 Siggi’s (fat-free, plain) 76 170 g 100 19 Chicken breast 75 120 g 198 37 Smoked oysters (water) 65 85 g 110 18 Pork loin 54 85 g 163 22 Smoked oysters (oil) 48 85 g 150…
Would you press it? Let’s play a game. In front of you is a button. If you press it, time rewinds to a moment of your choosing. You do not get to keep the memories of the intervening moments—this is a true reset. You won’t be able to beat the stock market or wildly alter the course of the world with the benefit of fore-hindsight. What you can do is live. In life, there may be moments of…
Pretraining language models on formal languages can improve their acquisition of natural language, but it is unclear which features of the formal language impart an inductive bias that leads to effective transfer. Drawing on insights from linguistics and complexity theory, we hypothesize that effective transfer occurs when the formal language both captures dependency structures in natural language…
Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks, yet to date no work has been able to establish a causal connection by controlling between language…
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert…
This work examines artificial neural language model behavior through the lens of morphological productivity. Specifically, we ask: given a corpus of linguistic data, (a) can we predict what morphological generalizations a transformer language model trained on this corpus will learn, and (b) is this learned behavior predicted by any existing theories of morphological productivity in humans? To…
State-space models (SSMs) have emerged as a potential alternative architecture for building large language models (LLMs) compared to the previously ubiquitous transformer architecture. One theoretical weakness of transformers is that they cannot express certain kinds of sequential computation and state tracking (Merrill and Sabharwal, 2023), which SSMs are explicitly designed to address via their…
Since at least the proposal of the Turing test, building computational systems that can communicate with humans using natural language has been a central goal for Al research. Understanding real, naturally occurring human language is the key to reaching this goal. This course surveys recent successes in language understanding and prepares students to do original research in this area, culminating…
As AI systems are used to answer more difficult questions and potentially help create new knowledge, judging the truthfulness of their outputs becomes more difficult and more important. How can we supervise unreliable experts —which have access to the truth but may not accurately report it—to give answers that are systematically true and don’t just superficially seem true, when…
In-context learning (ICL) is now a common method for supervising large language models (LLMs): given labeled examples in the input context, the LLM learns to perform the task without weight updates. Despite ICL’s prevalence and utility, we understand little about whether models supervised in this manner represent the underlying structure of their tasks, rather than superficial heuristics…
Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained Transformer-based large language models (LLMs) represent such relationships, focusing on the domain of…
To process novel sentences, language models (LMs) must generalize compositionally – combine familiar elements in new ways. What aspects of a model’s structure promote compositional generalization? Focusing on transformers, we test the hypothesis, motivated by theoretical and empirical work, that deeper transformers generalize more compositionally. Simply adding layers increases the…
Naturally occurring information-seeking questions often contain questionable assumptions—assumptions that are false or unverifiable. Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers for information-seeking questions. For instance, the question “When did Marie Curie discover Uranium?” cannot be answered…
The hexagon is the least-perimeter tile in the Euclidean plane for any given area. On hyperbolic surfaces, this ‘isoperimetric’ problem differs for every given area, as solutions do not scale. Cox conjectured that a regular $k$-gonal tile with $120$-degree angles is isoperimetric. For area $\pi/3$, the regular heptagon has $120$-degree angles and therefore tiles many hyperbolic…
Ingredients: two API calls and a server; Time: 1hr; Serves anonymized DMs to facilitate asynchronous data collection. My lab encountered a problem with one of the projects we’re working on: how do we reduce the time in takes to collect asynchronous responses from participants in a turn-based experiment? Because the task is both asynchronous (to reduce the commitment required to recruit…
How is knowledge of position-role mappings in natural language learned? We explore this question in a computational setting, testing whether a variety of well-performing pertained language models (BERT, RoBERTa, and DistilBERT) exhibit knowledge of these mappings, and whether this knowledge persists across alternations in syntactic, structural, and lexical alternations. In Experiment 1, we show…
To learn an unbounded problem is to generalize well from a limited set of training data. In humans, robust language acquisition requires language learners to form strong generalizations on the basis of very limited evidence (Chomsky 1980). These generalizations seem to require the acquisition of functional abstractions of some sort. Various analysis of these abstractions have been put forth in the…
A brief reflection on the roots of sacrifice for Pesach. Published in the “Passover 2022” issue of Shibboleth . In observance of Pesach, we read in the coming days of the deliverance from Egypt, the plagues sent upon Pharaoh, and, centrally, the Passover Sacrifice offered by our ancestors to G-d in the struggle for freedom and self-determination. Sacrifice— קרבן korban in Hebrew—is a…
In 2008 Reichardt proved that the optimal Euclidean double bubble—the least-perimeter way to enclose and separate two given volumes—is three spherical caps meeting along a sphere at $120$ degrees. We consider $\mathbb{R}^n$ with density $r^p$, joining the surge of research on manifolds with density after their appearance in Perelman’s 2006 proof of the Poincaré Conjecture. Boyer…
Natural language exhibits patterns of hierarchically governed dependencies, in which relations between words are sensitive to syntactic structure rather than linear ordering. While recurrent network models often fail to generalize in a hierarchically sensitive way (McCoy et al., 2020) when trained on ambiguous data, the improvement in performance of newer Transformer language models (Vaswani et…
The hexagon is the least-perimeter tile in the Euclidean plane. On hyperbolic surfaces, the isoperimetric problem differs for every given area. Cox conjectured that a regular $k$-gonal tile with $120$-degree angles is isoperimetric for its area. We prove his conjecture and more. @article{hirsch-2021-certain, title='Certain hyperbolic regular polygonal tiles are isoperimetric', author='Hirsch, Jack…
Reflexive anaphora present a challenge for semantic interpretation: their meaning varies depending on context in a way that appears to require abstract variables. Past work has raised doubts about the ability of recurrent networks to meet this challenge. In this paper, we explore this question in the context of a fragment of English that incorporates the relevant sort of contextual variability. We…