A blog about reverse engineering, mathematics, politics, economics and more ...
After a number of Twitter discussions, and repeating myself a lot in these discussions, it is time to write a short note on the economics of advancing LLM capabilities through RL, about principles of propaganda and coining new words, and about my stubborn refusal to use the term "distillation" except in a specific narrow sense. How do models advance when human-curated data has run out? It's been a…
I have seen a lot of public discussion around supply-chain attacks on the Python ecosystem, prompt injection risks when using coding agents, and general worries about the security implications of "vibe coding" for the development machine. In some of these discussions I find myself puzzled as to what problem is being solved - and it took me a while to realize that my failure to understand lies in…
I recently embarked on a small toy project/experiment: How well can I equip Claude Code to automatically analyze and triage crashes in a C++ code base? For the experimentation, I worked on a small number of crashes in the ffmpeg bug tracker. The initial results were very discouraging, Claude hallucinated all sorts of implausible root causes and tended to write typical "AI slop" -- things that…
In a series of previous blogposts [ 1 , 2 , 3 , 4 ] I ran some experiments drawing the boundaries of the polytopes generated by a fully-connected leaky ReLU network while it was getting trained on reproducing an input image. As I tried to scale the experiments to larger networks, I noticed a dramatic slowdown in the code, caused by the calculation of a hash of the activation pattern happening on…
In many discussions where questions of "alignment" or "AI safety" crop up, I am baffled by seriously intelligent people imbuing almost magical human-like powers to something that - in my mind - is just MatMul with interspersed nonlinearities. In one of these discussions, somebody correctly called me out on the simplistic nature of this argument - "a brain is just some proteins and currents". I…
After the previous blog posts here , here , and here , a friend of mine pointed me to some literature to read, and I will do so now :-). The papers on my reading list are: 1. https://proceedings.mlr.press/v80/balestriero18b.html - Randall Balestrieros paper on DNNs as splines. 2. https://arxiv.org/abs/1906.00904 - ReLU networks have surprisingly few activation patterns (2019) 3.…
What is this? After my first post on the topic, 9 months elapsed before I posted again, and now I am posting within days of the last post? Anyhow, after my last post I could not resist and started running some experiments trying to see whether I could induce "overfitting" in the neural networks I had been training - trying to get a heavily overparametrized neural network to just "memorize" the…
In this post, I will explain my current thinking about neural networks. In a previous post I explained the intuition behind my "origami view of NNs" (also called the "polytope lens" in some circles). In this post, I will go a little bit into the mathematical details of this. The standard textbook explanation of a layer of a neural network looks something like this: \[ \sigma( \overline{W}x + b )\]…
Welcome to one of my political posts. This blog post should rightfully be titled "the German debt brake is stupid, and if you support it, so are you (at least in the domain of economics)". Given that a nontrivial number of Germans agree with the debt brake, and given that there is a limit on the sensible number of characters in the title, I chose a shorter title - for brevity and to reduce…
Hey all, I have written about the various drags on the European tech industry in the past , and recently been involved in discussions on both X and BlueSky about what Europe needs. In this post, I will not make a wishlist of what concrete policy reforms I want, but rather start "product centric" -- e.g. what "user experience" would I want as a founder? Once it is clear what experience you want as…
The last few years have seen a wave of hysteria about LLMs becoming conscious and then suddenly attempting to kill humanity. This hysteria, often expressed in scientific-sounding pseudo-bayesian language typical of the „lesswrong“ forums, has seeped into the media and from there into politics, where it has influenced legislation. This hysteria arises from the claim that there is an existential…
While I have been a sceptic of using ML and AI in adversarial (security) scenarios forever, I also quite like the fact that AI/ML has become important, if only to make me feel like my Math MSc (and abortive Math PhD) were not a waste of time. I am a big proponent of "bottom-up" mathematics: Playing with a large number of examples to inform conjectures to be dealt with later. I tend to run through…
Hey all, == tl;dr == Today is my last day at Elastic. I'll take an extended break and focus on rest, family, health, writing, a bit of startup mentoring/investing, and some research - at least for a while. I'm thankful for my great colleagues and my leadership at Elastic - y'all are stellar, even if I was often grumbly about some technical or architectural issues. I'll also miss the ex-optimyze…
This post is an adaption of a Twitter thread where I listed the various factors that in my experience led to a divergence of the trajectories of the US tech industry around Silicon Valley (SV) and the tech industry in Europe. Not all of these factors are current (some of the cultural ones are less pronounced today than they used to be), and some of them could be relatively easily fixable. I'll add…
This blog post is a review of the book "This Is How They Tell Me the World Ends" by Nicole Perlroth. The book tries to shed light on the "zero day market" and how the US government interacts in this market, as well on various aspects of nation-to-nation "cyberwarfare". I was excited to see this book come out given that there are relatively few hats in this field I have not worn. I have worked in…
Preface: When I joined Google in 2011, I quoted a quip of a friend of mine: "There are roughly one and a half computers in the world, and Google has one of them." The world has changed quite a bit since 2011, and there may possibly be half a dozen computers in the world now. That said, for the following text to make sense, when I say "the computer", I mean a very large assembly of individual…
Twitter is great, and Twitter is terrible. While it enables getting in contact and starting loose discussions with a great number of people, and while it has certainly helped me broaden my perspectives and understanding of many topics, it also has a lot of downsides. Most importantly, Twitter discussions, due to their immediacy of feedback and the fact that everybody is busy, often end up in…
It has become painfully evident over the last decade or so that social media has a somewhat corrosive effect on "truth" and "discussion". There are a variety of reasons for this - many unidentified - but a few factors are: For every opinion, no matter how bizarre, it has become easy to find a community with similar beliefs. The discoverability of almost all information coupled with the shortening…
Hey everybody, During my years doing vulnerability research and my time in Project Zero, I frequently encountered proposals for new security mitigations. Some of these were great, some of these - were not so great. The reality is that most mitigations or "hardening" features will impose a tax on someone, somewhere, and quite possibly a heavy one. Many security folks do not have a lot of experience…
In a world of changing technology, there are few constants - but if there is one constant in security, it is the rhythmic flare-up of discussions about disclosure on the social-media-du-jour (mailing lists in the past, now mostly Twitter and Facebook). Many people in the industry have wrestled with, and contributed to, the discussions, norms, and modes of operation - I would particularly like to…
First off, an apology to the reader: I normally spend a bit of effort to make my blog posts readable / polished, but I am under quite a few time constraints at the moment, so the following will be held to lesser standards of writing than usual. A discussion arose on Twitter after I tweeted that the use of the term "Turing-complete" in academic exploit papers is wrong. During that discussion, it…
During my long sabbatical in 2015-2016 I had plenty of time to think about random things and come up with strange ideas. Most of these ideas are more funny than practical - their primary use is boring people that are reckless enough to have drinks with me. This blog post describes one of these ideas. With the recent renewed interest in privacy and overreach of smart phone apps, it seems like a…
After a long hiatus on this blog, a new post! Well, not really - but a whitepaper was published today titled " The Malicious Use of Artificial Intelligence ", and I decided I should cut/paste/publish two notes that apply to the paper from an email I wrote a while ago. Perhaps they are useful to someone: 1) On the ill-definedness of AI : AI is a diffuse and ill-defined term. Pretty much *anything*…
I am a huge fan of Wikipedia -- I sometimes browse Wikipedia like other people watch TV, skipping from topic to topic and - on average - being impressed by the quality of the articles. One thing I have noticed in recent years, though, is that the base-democratic principles of Wikipedia open it up to manipulation and whitewashing - Wikipedia's guidelines are strict, and a person can get a lot of…
This post grew out of a friend on Facebook asking (I paraphrase) "why do you spend your time on security instead of using your brainpower for something more lasting ?". I tried to answer, and ended up writing a very long reply. Another friend then encouraged me to re-post my reply to a wider audience. The below is a slightly edited and expanded version. It is much less polished than my usual blog…