RSSAmplifier

Blog

main

Recent content on main

main-horse.github.io ↗RSS feed ↗89 posts

Latest posts

Muon Optimizer Guide - Quick Start and Key Details

Asymptotic Estimation of AdamW's Weight RMS (Part 2)

Asymptotic Estimate for the Maximum of n Normal Random Variables

5. Dual Gradient Descent

Low-Precision Attention May Suffer from Biased Rounding Errors

1. Three Characteristics of a Good Model

Fast Estimation of the Spectral Norm of Random Matrices

A Very Concise VQ Training Scheme

Why Add Short Conv to Linear Attention?

Asymptotic Estimation of AdamW's Weight RMS

Rethinking Learning Rate and Batch Size (IV) - EMA

Rethinking Learning Rate and Batch Size (III) - Muon

Rethinking the Relationship between Learning Rate and Batch Size (II) - Mean Field

H-Net - Scaling Laws (Byte)

H-Net - Parameterization (S≥1)

H-Net - Intuitions

H-Net - Engineering (1gpu)

Why is Adam's Update RMS 0.2?

H-Net - Parameterization (Baseline)

Rethinking the Relationship Between Learning Rate and Batch Size (Part 1) - Current Status

4. Muon + Spectral Sphere

3. Muon + Stiefel

2. Muon + Orthogonal

1. SGD + Hypersphere

H-Net - Inference

Advancing Muon Further on its Scale-up Journey

Newton-Schulz Iteration for the msign Operator (Part II)

A minor point about precision

Newton-Schulz Iteration for the msign Operator (Part 1)

19. The Second Type of Rotary Positional Embedding

Things I learned digging into 5090 perf

Higher-Order muP - Simpler Yet More Profound Spectral Condition Scaling

Hyperparameter Scaling Laws Across Model Scales

Why Did We Choose to Try Muon?

Various approaches to parallelizing Muon

Why is the default norm length for gradient clipping 1?

Reflections on Novel Weight Decay from Spectral Norm Gradient

Testing the 4090 48GB

Visualizing 6D Mesh Parallelism

A Fundamental Leap from Vectors to Matrices

1 - Initialization and Generation of ncclUniqueId

2 - Establishment of Bootstrap Network Connection

3 - Intra-machine Topology Analysis

4 - Graph Building Process

5 - Path Calculation

6 - Channel Search

7 - Inter-machine Channel Connection

8 - Establishment of Data Communication Link Transport

9 - Single-machine ncclSend and ncclRecv Process

10 - Inter-machine ncclSend and ncclRecv Process