When Eigenvalues Collide
Accelerating convergence of eigenvalue models by Moreau regularization of the mid-eigenvalue.
Alex Shtoff
Blog on optimization, machine learning, and software development.
Latest posts
Cheaper eigenvalue training and inference
Cheaper eigenvalue training and inference with symmetric tridiagonal matrices: preserve useful expressiveness, use fast SciPy-backed PyTorch autograd, and avoid dense eigensolvers.
Interpreting eigenvalue models
Interpreting eigenvalue-based ML models: read the k-th eigenvalue as a two-player game using Courant principle, a sequential orthogonality process via Courant-Fischer theorem, and a difference-of-convex function using Ky Fan variational principle.
I feel the need for Eigen-Speed
PyTorch eigenvalues on CUDA can be unexpectedly slow due to device synchronization. This post shows how to call CuPy via DLPack for fast GPU eigvalsh/eigh while keeping gradients for training.
Robustness, interpretability, and scaling of eigenvalue models
Robustness, interpretability, and scaling of eigenvalue models: stability bounds from Weyl's inequality, operator-norm feature importance, and regularization experiments for tabular data.
Behold the power of the spectrum!
Eigenvalues as neurons: represent nonlinear models as the k-th eigenvalue of a learned symmetric matrix pencil. Explore monotonicity/convexity properties and train simple spectral models.
Paying attention to feature distribution alignment
Orthogonal polynomial features are only uncorrelated when the feature distribution matches the basis weight. Use CDF/quantile transforms to align distributions and get more informative Legendre features.
Off with the polynomial’s tail!
Legendre polynomial feature regression on California Housing shows double descent; a simple tail-pruning of high-degree coefficients yields smaller, competitive models. Implemented in scikit-learn.
Let the polynomial monster free
Overparameterized polynomial regression can exhibit double descent: past the interpolation threshold, some bases memorize and still generalize. Experiments compare power, Legendre, Chebyshev, and Fourier features.
Shape restricted function models via polyhedral cones
Enforce richer shape constraints (convexity/concavity and combinations with monotonicity) by constraining coefficient vectors to polyhedral cones. Implement a PyTorch cone layer and fit concave functions.