Abstract:Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution $p(z)$ and decoding distribution $p(x|z)$, we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.
| Comments: | Added CIFAR10 experiments; ICLR 2017 |
| Subjects: | Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:1611.02731 [cs.LG] |
| (or arXiv:1611.02731v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.1611.02731 arXiv-issued DOI via DataCite |
Submission history
From: Xi Chen [view email]
[v1]
Tue, 8 Nov 2016 21:43:34 UTC (240 KB)
[v2]
Sat, 4 Mar 2017 06:19:22 UTC (2,203 KB)