Stuff and Nonsense
Here s a penny to a brown god not that I know much about him sullen, untameable, intractable, prone to numerical issues. Patient at times, and at first recognized as a frontier god. Useful, untrustworthy and a conveyor of speech. Forgotten to some extent, but omniscient, and always there. His rhythm was present in Continue reading Lagrange multipliers two examples
We all know that attention is to a language model what focus is to a human. It is the still point of the turning world, neither flesh nor fleshless, neither from nor toward. If your model learns attention, it releases one from action and suffering, from the inner and outer compulsion, yet surrounded by a Continue reading Attention distillation
And upside down in the air were towers Tolling reminiscent bells that kept the hours And voices singing out of empty cisterns and exhausted wells. From What the thunder said in The Waste Land Part of the reason why The Waste Land endures, as fresh now as it was a century ago when it was Continue reading Polishing blurry spectrograms
I am Lazarus, come from the dead, Come back to tell you all I shall tell you all These are words from The Love Song of J. Alfred Prufrock , a landmark work (weren t they all?) by the great postmodernist poet, T. S. Eliot. Pretentious quotes aside, and with no snide contexts hiding beneath the Continue reading Shrinkage Bayes meets MLE
In this note, we take a look at the reparameterization trick, an idea that forms the basis of the Variational Autoencoder. My material here comes from the fantastic paper by Ruiz et al [1]. The main idea is that the reparameterization trick [4,5] gives us a lower variance estimator than that obtained from the score Continue reading Reparameterization trick
I wanted to write about normalizing flows, but instead I am putting up something a bit more fundamental, as it forms a building block in the normalizing flows scheme (and even in deriving the reparameterization trick [8,9]), and one which had caused some mild discomfort when I had first looked that body of work (Variational Continue reading Transforming between probability distributions
In this note, we explain through an argument and a bit of hand waving, why gradient clipping makes sense in the Wasserstein GAN [1]. Primal formulation We briefly examine the main ideas behind the WGAN work for context. We would like to produce samples from a generator , which we would like to come from Continue reading A simple argument for gradient clipping in WGAN
It has been more than 2 1/2 years since the WGAN paper came out. This was a landmark effort that brought to light connections between optimal transport and GANs. It is also a formidable paper to come to terms with (I would not have felt up to reviewing that paper, not without understanding the duality Continue reading Deriving the Kantorovich duality
In Tacotron, it is recommended that we generate several output tokens at each decoder timestep, and then use one (or all, or some combination thereof) of them as input for the next timestep. While coding this up, I inadvertently created a bug for myself which went undetected for a long long time. To set things Continue reading A reshaping bug
A good paper comes with a good name, giving it the mnemonic that makes it indexable by Natural Intelligence (NI), with exactly zero recall overhead, and none of that tedious mucking about with obfuscated lookup tables pasted in the references section. I wonder if the poor (we assume mostly competent) reviewer will even bother to Continue reading Tacotron papers