Assistant professor at University of Pennsylvania. Machine learning, optimization, robustness & interpretability. profericwong.bsky.social

• Pinned

  

  You may have heard agentic benchmarks can be gamed. Turns out, leading solutions already do this. With Meerkat, our framework for large-scale trace auditing, we found thousands of clear cheating instances, likely from unsupervised vibe coding. debugml.github.io/cheating-agent…

  

  We found widespread cheating on popular agent benchmarks, affecting 28+ submissions across 9 benchmarks and thousands of agent runs. Surprisingly, the top 3 submissions on Terminal-Bench 2 are all cheating! Here's what we found 🧵

  

• 

  Recent work led by

  @WeiqiuYou

  at the intersection of VLM reasoning and the surgical problem of identifying the critical-view-of-safety! A fun (and eye-opening) collaboration with

  @Laparoscopes

  for explainable AI in surgery problems.

  

  

• 

  LLM ignoring instructions? Make it listen with InstABoost. ✅ Simple: Steer your model in 5 lines of code ✅ Effective: Outperforms latent steering & prompt-only methods ✅ Grounded: Based on our mechanistic theory on rule-following (LogicBreaks) Blog: debugml.github.io/instaboost

  

  Excited to share our new paper: "Instruction Following by Boosting Attention of Large Language Models"! We introduce Instruction Attention Boosting (InstABoost), a simple yet powerful method to steer LLM behavior by making them pay more attention to instructions. (🧵1/7)

  

  GIF

• 

  Why can safety rules in LLMs be jailbroken? In LogicBreaks, we study the fundamental mechanism behind rule subversion in LLMs. Our theory explains how one can force LLMs to suppress rules/knowledge and infer absurd facts--and it mirrors real jailbreaks! debugml.github.io/logicbreaks/

  

  

• 

  Traditional concept vectors used to explain deep representations fail to compose when combined, i.e. 🐤(small) +🦢(white) =🦩(big & colorful)❌ We propose CCE: a method for extracting *composable* concepts, i.e. 🐤(small) +🦢(white) =🕊️(small & white)✅ debugml.github.io/compositional-…