Zheng Zhang

Amazon Lab126 • 12K followers

Recently the Muon optimizer has shown great promise in LLM pre-training. By performing gradient matrix orthogonalization, Muon can effectively avoid/migitate gradient norm explosions or vanishing. In collaboraiton with Prof. Sijia Liu, recently my students Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, Dongyang Li and Yupeng Su developed and implemented Teon, a tensorized version of Muon, to improve LLM pre-training efficiency & accuracy. The key idea is to stack multiple gradient matrices together and perform mode-i orthogonalization of the resulting higher-order tensor, which can effectively explore the correlation among the gradient matrices. When tested on GPT-2 and LlaMA models, Teon shows consistent improvement over the conventional Muon baselines. They also provided theory to explain why "tensorization" can offer benefits, and how to maximize the benefit during the tensorization process. Paper link: https://lnkd.in/gpq-kxmW

Vishnu Pendyala

San José State University • 5K followers

In this research funded by a grant that I received and published in a Q1 journal (Impact Factor 6.5, CiteScore 13.6, h5 index: 148, h5 median: 211), my SJSU College of Information, Data and Society student, Neha Bais Thakur and I systematically evaluate and interpret 15 open-source models (Code LLaMa, Granite Code, DeepSeek-Coder-V2, Yi-Coder, etc.) on code translation and generation, uncovering insights into reliability and prompt sensitivity. The study provides interpretability insights into the LLM-driven code translation process, which is the first of its kind on coding LLMs for translation tasks. We demonstrate that LLMs can not only translate code but also comprehend, explain, and correct their own errors when provided with compiler feedback. This led to an average improvement of 10% in the CCR metric and 8% in the Pass@1 metric. Key contributions: + Comprehensive performance benchmarking with CodeBLEU, chrF, METEOR, and Pass@k. + Interpretability analysis using Feature Ablation and Shapley Value Sampling, uncovering prompt sensitivities. + Insights into reliability, error correction, and practical deployment of open-source LLMs for software engineering. We also showed that line-to-line interpretability heatmaps can be a valuable tool for guiding LLM output towards a desired outcome. We hope this study helps researchers and practitioners better understand the capabilities and limitations of code-focused LLMs. https://lnkd.in/gw3YW9eJ #AI #LLMs #CodeGeneration #XAI #SoftwareEngineering

Thien-Minh Nguyen

The University of Queensland • 2K followers

Imho, Isaac Lab’s Manager-Based approach is very anti-pedagogical. As a student with zero knowledge of RL, before I could even start modifying most existing locomotion examples in Isaac Lab, I had to dig through layer after layer of class dependencies — in the Unitree Go1 walking example alone, there are five levels of inheritance. As a result, all the environment parameters are scattered across the library. And all of this just to understand the “EnvCfg” of the actual Env class… Every other important components of an RL problem like reward, observation, termination, command,... will also need multiple class definitions... I asked myself, does it really have to be this complicated? The answer is NO. After several months of mucking around, I ended up writing my own RL package with just a single file defining all the essential classes. In the end, there are really only three classes that matter: EnvCfg, EnvCfgPlay, and Env.