Recent content on Changkun's Blog
“Perhaps it is more important to trust the people who wrote the software.” — Ken Thompson, Reflections on Trusting Trust Ken Thompson’s 1984 lecture is generally remembered for its unsettling demonstration that source-level inspection cannot establish the trustworthiness of a software system. A compiler can insert behavior that is absent from the program it compiles; it can even reproduce that…
Tastes are accumulated from experience. Working on many different problems in the past teaches you what kinds of problems might be interesting in the future, or what kinds of things might be just barely possible by combining previous approaches. This can reveal open problems you might need to work on to achieve something magical or highly useful. Another way to gain experience is to write down a…
This interview is very timely, and some of the insights within it are quite thought-provoking, particularly regarding the development of general-purpose technology. Many people are willing to believe that the “Bitter Lesson” claims general-purpose technology will ultimately triumph over specialized technology. However, in reality, whether we look at the historical experience of…
In mathematics, we can develop the intuition that two systems can prove exactly the same theorems yet differ in how quickly things get discovered in them. The asymmetry is not about power, but only about what is cheap to say. This is grounded in description complexity with a resource bound, since the whole point is that cost is bounded. We switch frames when the gain beats the cost of switching.…
We should never forget that the strongest aspect of human intelligence is our generalization and sample efficiency. Some people value and invest years of practice or large amounts of token consumption these days as a form of endorsement. That’s fair and does provide some degree of safety and establishes a baseline; but to generalize and grow exponentially, all you need is good intuition and…
Agent economics is fundamentally about finding the right “autonomy affordance” in an organization. When we allocate more compute, we can accelerate agent autonomy for execution. But the cost of human verification also rises sharply. The challenge is not only checking whether the output is correct, but actually understanding what happened, why it happened, and whether we can responsibly…
On instances, primitives, and the difference between novelty that compounds and novelty that merely accumulates. “The limits of my language mean the limits of my world.” – Wittgenstein, Tractatus 5.6 At first it looked like progress. For one week, an AI Agent pipeline kept shipping. Commits arrived roughly every hour. Small fixes landed, minor improvements accumulated, the…
Among a set of formally symmetric options, what determines actual preferences and stability? The following content is generated by LLMs and may contain inaccuracies. Preference Determinants in Symmetric Options Background (Context) This question touches upon the core intersection of decision theory, game theory, and statistical physics: when a set of options is formally completely symmetric…
The emergence of AI will not push the world toward a single unified system. Rather, it is more likely to accelerate the world’s fragmentation. This is because human society does not operate around a single optimal solution, but around the attention, value judgments, risk preferences, linguistic habits, and practical constraints of different groups. Different groups care about different…
Is “secure by default” the right default for AI products? After years in B2B product work, “Default Closed” became reflexive: restrict first, unlock later. Enterprise customers love it—admins, compliance, audits all benefit. Then I began discovering B2C conversations. The same instinct and defaults immediately created problems: users could not get started without…
For a long time, I was a happy Cursor user. It felt like a natural extension of VS Code, which I have used for nearly a decade. The completion was fast and precise, the integration was smooth, and it fit well into my existing engineering workflow. Around the middle of last year, I got access through an enterprise license, so I cancelled my personal subscription. One thing I want to share is that…
This is an interesting article. The author argues that AI alignment should not be understood as “making AI maximize human preferences.” The mainstream approach to current AI alignment over-relies on the concept of “preference,” treating preferences as a sufficient expression of human values, the basis of rational behavior, and the target AI should optimize. The author calls…
Can AI agents ever produce artifacts you can fully trust? How do we solve the problem of limited human attention to engineering if coding is substituted by agents? I recently revisited this topic and reviewed some lecture knowledge from complexity theory. The Probabilistically Checkable Proof theorem tells us that any correct proof can be encoded so that a constant number of random spot-checks…
Multi-agent Topology Dynamic Management: Thinking Notes Discussion Content This discussion revolved around a central question: in a system composed of multiple agents, who should decide the topology structure (who spawns whom, who merges with whom, who is terminated), on what basis, and how should it be implemented. The discussion started from a theoretical list of primitives, quickly jumped to…
0. Introduction The “Dark Forest Theory” proposed by Liu Cixin in the Three-Body Problem series is a speculative theory about interaction strategies among cosmic civilizations. This article attempts to provide a rigorous formal derivation of the theory using tools from game theory and decision theory, starting from the axioms given in the novel. The core argument proceeds in three…
What if your AI agents are lying to each other? We’ve been building multi-agent systems like it’s a team meeting: everyone contributes, we take the best idea, and ship it. But distributed systems engineers solved a harder version of this problem 40 years ago under the name Byzantine Fault Tolerance, where the core challenge is reaching agreement when some participants are unreliable,…
Today I read some articles on Harnessing engineering and had a sudden thought: Harnessing engineering is essentially control theory, right? You need to continuously adjust the system’s structure to ensure its stability. But this stability is actually strongly correlated with the model itself. We don’t know whether a model can fully leverage its capabilities within the constraints…
I re-read the “Free Energy Principle” and came across an interesting idea: humans don’t live in order to predict the world; rather, humans live because they predict the world. This aligns perfectly with the principle of minimizing free energy. This sparked a thought about the consequences of excessive AI use: Simplification of thought and behavior If people use AI too frequently,…
“It is not knowledge, but the act of learning, not possession but the act of getting there, which grants the greatest enjoyment.” – Gauss (letter to Bolyai, 1808) Give an AI Agent a clean computer, set no goals, and let it decide what to do. What do you think it would do? I assumed the answer would be random. It wasn’t. I ran this experiment many times, restarting from a…
In this current era, this article becomes even more interesting. Because it mentions “attention is the only thing we possess,” an observation that aligns perfectly with the author’s experience in the mathematics field. He has also been continuously thinking about what actually happens in someone’s brain when they are extremely intelligent. Is it cognitive inequality, or…
Recently I developed an automated software development pipeline, and the basic idea is: an idea agent proposes ideas, then the pipeline automatically implements them, runs tests after implementation, and then automatically submits. I noticed a very interesting phenomenon. After showing this tool to some people, I found that their design approaches for similar ideas varied quite a bit. 1. My design…
This discussion did not begin with philosophy of consciousness. Its starting point was much lower, much more concrete, almost just an engineering intuition: any system, in theory, possesses enormous degrees of freedom, yet the number of structures that can actually run stably over time is very small. This is true of software architecture, organizational architecture, and artificial intelligence…
I’ve been thinking recently about what this current wave of AI development will ultimately become, and what societal structures will transform into as a result. I’ve been considering that AI might ultimately learn a holistic and most stable system structure. For instance, when we design software architecture, the software itself offers tremendous freedom (such as languages being Turing…
Recently I have been thinking about using AI to develop AI itself. I built a Kanban-style development workflow. The rough idea looks like this: First there is an Ideation Agent. It scans the entire repository and performs deep research. After the scan, it actively proposes things such as bugs, technical debt, or potential product features. Once these tasks are created, they enter the backlog. I…
I suddenly realized that the concepts of the “Wallfacers” and the transparent / opaque thinking of the Trisolarans in Liu Cixin’s The Three-Body Problem become particularly interesting in the era of LLMs. I started wondering: if humans in the future gradually begin to rely on LLMs to replace their own thinking, then in essence we may move toward a state where thoughts become completely…
Some interesting tradeoffs: Autonomy vs. Steering Autonomy vs. Paternalistic control Delegate vs. Hands-on Automation vs. Interruption Levels of detail: Intent, Traces, Diff Optimal vs. Bounded rationality … These are all unavoidable design tradeoffs in human-in-the-loop systems, and they carry philosophical implications rather than merely interface implications. In the LLM era, these…
Last Friday, while pair coding with colleagues, I noticed an interesting pattern. I was leading the deployment while several colleagues observed my actions. As I worked, they discussed and speculated about my intentions, then offered suggestions for alternative approaches. During this process, some suggestions were noise that interfered with my train of thought, while others actually proved…
psychology solved the ai memory problem decades ago. we just haven’t been reading the right papers. your identity isn’t something you have. it’s something you construct. constantly. from autobiographical memory, emotional experience, and narrative coherence. Martin Conway’s Self-Memory System (2000, 2005) showed that memories aren’t stored like video recordings.…
Mathematics, when self-reference appears in its formal system, generates paradoxes. So why can life produce self-reference? Can programs implement self-reference? Can current large language models truly achieve self-reference? The following content is generated by LLMs and may contain inaccuracies. I will deeply explore the topic of self-reference paradoxes and self-referential mechanisms in life…
https://arxiv.org/abs/2602.11988 The following content is generated by LLMs and may contain inaccuracies. I’ll help you create a structured deep dive on this research paper. Let me first fetch the full paper to examine its contents and then search for related work.Let me search for more information about related work on context in coding agents and the authors of this paper.Now let me search…
In the blink of an eye, a decade has passed since my undergraduate graduation. This year, for the first time since the pandemic, I returned home for the New Year and reunited with friends, relatives, and family. What struck me most was the stark contrast between the dramatic changes in the external world and the apparent stagnation in the people around me. The external world has been transformed…
The Gibbard–Satterthwaite theorem states that every ranked-choice voting system is manipulable, except possibly in two cases: if there is a distinguished voter with dictatorial power, or if the rule limits possible outcomes to only two options. https://en.wikipedia.org/wiki/Gibbard%E2%80%93Satterthwaite_theorem The following content is generated by LLMs and may contain inaccuracies. I’ll…
Definition of Preference While some view the term “preference” as a rather subjective concept, this does not prevent various disciplines from conducting in-depth research on it. For example: Computer Science: Preference is defined as a latent utility function, with its driving mechanism or objective function being reward maximization. This is based on the intuitive assumption that…
Stanford Encyclopedia of Philosophy https://plato.stanford.edu/contents.html The following content is generated by LLMs and may contain inaccuracies. I’ll help you create a structured deep dive for this resource. Let me first fetch the linked content to better understand the Stanford Encyclopedia of Philosophy, then search for relevant context about its role in academic philosophy.Now let me…
In our everyday lives, we confront numerous moral issues. Once we have deliberated and formed judgments about what is right or wrong, good or bad, these judgments tend to exert a strong influence on us. Although we do not always behave as we think we ought, our moral judgments typically motivate us, at least to some degree, to act in accordance with them. When philosophers discuss moral…
How can you believe in something you haven’t witnessed with your own eyes? Who sets the standard for evidence? If an experience cannot be replicated by a third party, is it still true? https://www.darioamodei.com/essay/the-adolescence-of-technology The following content is generated by LLMs and may contain inaccuracies. Three Paradoxes of Epistemic Belief Context You have touched upon the…
LLM Emergence Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 There is no such thing as conscious artificial intelligence https://www.nature.com/articles/s41599-025-05868-8 Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking https://arxiv.org/abs/2509.21519 The following content is generated by LLMs and may contain inaccuracies. Context This…
MLHP/RLHF Lectures https://web.stanford.edu/class/cs329h/index.html https://mlhp.stanford.edu/ Multi-Objective Bayesian Optimization with Active Preference Learning https://arxiv.org/abs/2311.13460 Nearly Optimal Active Preference Learning and Its Application to LLM Alignment https://arxiv.org/abs/2602.01581 Personalized Building Climate Control with Contextual Preferential Bayesian Optimization…
Per-tool-call human approval in agentic AI is solved in theory, unsolved in practice. Confirmation fatigue is not a UX annoyance but a security vulnerability and the primary obstacle to effective human oversight at scale. Risk-tiered frameworks, middleware architectures, and new design patterns now exist to replace the binary confirm/deny paradigm. But MCP provides no protocol-level mechanism for…
The Voter Model The voter model is a simple mathematical model of opinion formation in which voters are located at the nodes of a network. Each voter holds an opinion (in the simplest case, 0 or 1, but more generally, any of n options), and a randomly chosen voter adopts the opinion of one of its neighbors. This model can be used to describe phase transition behavior in idealized physical systems…
LLM as Optimizer: Large Language Models as Optimizers https://arxiv.org/abs/2309.03409 When Large Language Models Meet Optimization https://www.sciencedirect.com/science/article/abs/pii/S2210650224002013?via%3Dihub Large Language Models to Enhance Bayesian Optimization https://arxiv.org/abs/2402.03921 Cooperative Design Optimization through Natural Language Interaction…
LLM Lectures https://cmu-llms.org/schedule/ https://www.phontron.com/class/lminference-fall2025/schedule https://llmsystem.github.io/llmsystem2026spring/docs/Syllabus https://llmsystem.github.io/llmsystem2025spring/docs/Syllabus/ The following content is generated by LLMs and may contain inaccuracies. Context This collection addresses the growing need for structured educational resources in large…
The Cost of Staying by Amy Tam https://x.com/amytam01/status/2023593365401636896 Every technical person I know is doing the same math right now. They won’t call it that. They’ll say they’re “exploring options” or “thinking about what’s next.” But underneath, it’s the same calculation: how much is it costing me to stay where I am? Not in…
Useful resources: https://huggingface.co/spaces/transformers-community/Transformers-tenets https://ianarawjo.github.io/Guidelines-for-Reporting-LLM-Integrated-Systems-in-HCI/ The following content is generated by LLMs and may contain inaccuracies. Context As LLM-integrated systems proliferate in HCI research, the field faces a reproducibility and transparency crisis. Unlike traditional…
Interesting books for reading: 贝叶斯方法与科学合理性——对休谟问题的思考: https://book.douban.com/subject/4472081/ Reinforcement Learning from Human Feedback - A short introduction to RLHF and post-training focused on language models: https://rlhfbook.com/ Build a Reasoning Model (From Scratch): https://www.manning.com/books/build-a-reasoning-model-from-scratch Build a Large Language Model (From Scratch):…
In agentic applications, to ensure the agent actions executed in a safe and trusted manner, in particular write operations, we often use a human in the loop. The native practice for the setup with one agentic loop + MCP tool servers, is to add human confirmation per tool call. However, this is not a scalable approach because it soon makes human become a tedious “Confirm” executor and may just…
Pareto Principles in Infinite Ethics Published: May 01, 2018 Recommended citation: Askell, Amanda. ‘Pareto Principles in Infinite Ethics.’ PhD thesis, New York University (2018). https://askell.io/files/Askell-PhD-Thesis.pdf Summary: In this thesis I argue that ethical rankings of worlds that contain infinite levels of wellbeing ought to be consistent with the Pareto principle, which says that if…
Continuing from my previous thoughts. We mentioned that the long tail may not necessarily be a bad thing. Similar perspectives have actually been proposed long ago—over-optimization weakens the ability to adapt to mutations, which comes from the theory of antifragility. Actually, the leaders on the adaptation curve are, in a sense, a group of highly intelligent people. From historical experience,…
Over the past few years, due to the need to migrate legacy systems in enterprise projects, I’ve actually developed little affection for the long tail effect. A project that should have taken only a few months ended up requiring years to complete due to user dependencies or stubborn system coupling. However, my perspective on this has shifted recently due to AI’s rapid transformation.…
Lex Fridman (03:04:40) Yeah, there’s a couple of things I want to say. One is, I never thought that the thing I love doing would be the thing that gets replaced. You hear these stories about things like the steam engine. I’ve spent thousands of hours poring over code, pouring my heart and soul into it. Some of my most painful and happiest moments were alone behind a screen. I was an…