Escaping Flatland: Weight-Space Regularization and Hyperbolic Geometry in Mechanistic Interpretability

A formal mathematical critique of Euclidean Sparse Autoencoders (SAEs). By transitioning from flat linear algebra to the negative curvature of Riemannian manifolds, we demonstrate how Hyperbolic Sparse Autoencoders (HypSAEs) utilize Weight-Space Regularization to extract hierarchical ontologies without optimization collapse.