Open Samizdat, blog about stuff
I argue that we should always assume LLM applications are compromised by anyone who can send or plant tokens that will be processed. If an attacker can perform a prompt injection, they can control any resources accessible through the LLM. I present evidence showing how effective these attacks are and how weak the defenses remain.
I recently received what I believe is an LLM-generated meta-review for one of my papers. This is the first time for me, but I am afraid that it is not the last. I will talk about why I think this review was generated automatically, but more importantly, I would like to start a debate about the ethics of using LLMs in the reviewing process. I believe that the entire trust-based concept of peer…
I introduce the concept of abstraction layers into LLM safety & security. These layers are my attempt to categorize various risks based on our perspective of LLMs. I believe this categorization can aid discussions about the nature of risks, their ownership, and the appropriate approach to addressing them.
We recently released the GEST dataset for measuring gender-stereotypical reasoning in language models and machine translation systems. Unlike other datasets, this one focuses on specific stereotypical ideas, such as men are leaders. We found out that NLP models associate beauty, housework, neatness, and empathy with women; while leadership, professionalism, and rationalism are associated with men.…
The Belebele dataset, designed for a multilingual evaluation of language models’ reading skills, was recently released with respectable 115 languages. I have noticed that the genealogical linguistic analysis in the paper is somewhat lacking, with the authors providing almost no insight into behavior of models across language families. This oversight makes it hard to understand how various models…
We are at a curious point in time where our ability to build language models (LMs) has outpaced our ability to analyze them. We do not really know how to reliably determine their capabilities, biases, dangers, knowledge, and so on. The benchmarks we have are often overly specific, do not generalize well, and are susceptible to data leakage. Recently, I have noticed a trend of using self-report…
The popularity of ChatGPT and its various impressive capabilities lead some people to believe that it is a significant step forward for linguistic capabilities over existing systems, that the field of NLP will soon be consumed by generative language models, or even that it foreshadows AGI. To test these claims, I conducted a survey of the arXiv pre-prints that compare ChatGPT with other…