Neural Networks from Maximizing Rate Reduction

While we have witnessed much empirical evidence of the success of deep learning, much of it is due to trial and error and not guided by underlying mathematical principles. I attended Yi Ma's keynote on "Pursuing the Nature of Intelligence" at ICLR this year, which took on a statistical lens towards urging the community to view model training as learning to do compression. I was especially struck…