Recent content on Philosophical Hacker
Mythos’ system card contains the following graph to support its argument that Mythos performs better on SWE-bench: Anthropic and others are worried LLMs are memorizing SWE-bench, so they asked an LLM to estimate the probability that a solution is memorized. Next, they calculated the pass rate if they only included solutions an LLM judged to be memorized with less than 5% confidence, 10%…
Bertrand Russell, the 20th century philosohper and mathematician, once said: The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt. LLMs are introducing something like this dynamic into programming. The programming version of the statement is: The fundamental cause of the trouble is that in the modern world the less…
Libraries are under-used. Why? Briefly: Writing code is more fun than reading documentation. We tend to understimate the complexity of problems we don’t understand well, so we undervalue libraries that solve these poorly understood problems. Perverse incentives: libraries compete with big internal engineering projects that look good in a promo packet. LLMs make this problem worse. Why? Less…
Our original idea for our coding challenge was to ask candidates to build a tic-tac-toe game in React with a few curve balls thrown in around making the solution more general for larger game boards and play modes. We scrapped that idea when we discovered ChatGPT could trivially do this. Here are some things we did to “LLM-proof” our new challenge. But first, why the scare quotes around…
Founders are often told to price based on the value they are providing to their customers. For example, if you’re saving your customer 1 million dollars, charge a 10th of that. Here’s Kevin Hale at YC advocating for this approach: In startups, and almost pretty consistently across all businesses, everyone will tell you, you should strive for value-based pricing. It allows you to charge…
Parsing a Sqlite database file is a nice way to brush up on data structures, bit manipulation, and recursion. I know this because I recently implemented the read_schema_tables function below such that the following test passes: import sqlite3 def test_read_table_names (db_file): con = sqlite3 . connect(db_file) assert list(con . execute( 'SELECT * FROM sqlite_schema;' )) ==…
We’re thinking about building a product for developers that enables them to build applications that operate on encrypted data via homomorphic encryption. We think developers have seen enough data leaks to want a product like this, but we’re worried we’re wrong. Even if we’re right, we’re worried about finding specific devs who can be early adopters. Where do these devs live? What languages do they…
I’ve been an programmer for a decade, and I can’t believe how much wasteful code we write. Even more unbelievably, many of us “justify” our waste with vague appeals to “clean code” or “best practices.” I used to do this all the time. These vague appeals — and the religious fervor that often accompanies them — betray a common lack of serious thinking…
When we were raising money for ATLAS, I often told investors that my cofounder and I were probably the most skeptical GenAI founders they would meet. Sam Altman’s recent hyperbolic claim that Open AI has “scientific certainty that GPT-5 will be better than GPT-4” at Stanford University fuels this skepticism: I’m impressed by OpenAI, we use their models, and I’m sure…