Hyperbolic Sparse Autoencoders: Empirical Validation of Poincaré Manifold Geometry on LLM Activations

An empirical scale-up validation of Hyperbolic Sparse Autoencoders (HyperSAE) trained on Google Gemma-2-2B Layer 13 residual stream activations across 20M tokens of FineWeb-Edu. Featuring 4 interactive charts, a 6-model comparative sweep across three L1 sparsity parameters (0.0005, 0.001, 0.005), qualitative activation case studies, code listings, and downstream evaluation on MMLU-Pro (12,032…