This past weekend I had a lot of fun participating in the Smart Software Hackathon by the Purdue Advanced Neural Projects group.
A while ago, I stumbled upon this GIF on Twitter and I thought it was the coolest thing ever.
StyleGAN2 interps pic.twitter.com/pccZpxHrCo
This is a StyleGAN2 interpretation, however, there are a lot of interesting things that can be done with autoencoders.
I am currently taking a neural image processing class at Purdue, and we recently covered generative adversarial networks (GANs). I found generative models really interesting and wanted to learn more about autoencoders.
Autoencoders have a bottleneck architecture and are made up of an encoder and decoder.
The goal of the encoder is to reduce the dimensionality of the input and compress it into a latent space representation. The goal of the decoder is to reconstruct the input from the latent vector (encoder’s output).
For example, the encoder would compress this 5x5 matrix into a 3x1 latent vector, and the decoder would try to reconstruct the 5x5 input.
At first, I wanted to learn more about how autoencoders worked, so I created a basic implementation for the MNIST dataset.
After training for 30 epochs, here is a bilinear interpolation of the latent space. You can see in the top left, the zeros are very round. As you move through the interpolation, the digits gradually morph into other classes.
The following visualization is a representation of the digits in latent space colored according to each digit. It can be seen that a 0 digit has a vastly different shape than a 1.
On Kaggle, I found the dataset “Face Mask Lite” which has over 23GB of images of faces with and without face masks.
One downside about this dataset is that the faces are overlaid with a clipart face mask instead of real face masks. For a short-term hackathon, this seemed reasonable to train with. However, it would be cool to continue this project with a more robust dataset.
In this model, the input gets reduced down to a latent vector of shape (16, 16, 64). From this latent vector, we attempt to reconstruct the input with the decoder network. I was able to train on a small subset of the dataset (instead of download the full 23GB) for this hackathon.
Model: “model_1” _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= img (InputLayer) [(None, 256, 256, 3)] 0 _________________________________________________________________ conv2d (Conv2D) (None, 256, 256, 16) 448 _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 128, 128, 16) 0 _________________________________________________________________ conv2d_1 (Conv2D) (None, 63, 63, 32) 4640 _________________________________________________________________ batch_normalization (BatchNo (None, 63, 63, 32) 128 _________________________________________________________________ leaky_re_lu (LeakyReLU) (None, 63, 63, 32) 0 _________________________________________________________________ conv2d_2 (Conv2D) (None, 32, 32, 64) 18496 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 16, 16, 64) 0 _________________________________________________________________ conv2d_3 (Conv2D) (None, 16, 16, 64) 36928 _________________________________________________________________ batch_normalization_1 (Batch (None, 16, 16, 64) 256 _________________________________________________________________ leaky_re_lu_1 (LeakyReLU) (None, 16, 16, 64) 0 _________________________________________________________________ conv2d_4 (Conv2D) (None, 16, 16, 128) 73856 _________________________________________________________________ batch_normalization_2 (Batch (None, 16, 16, 128) 512 _________________________________________________________________ leaky_re_lu_2 (LeakyReLU) (None, 16, 16, 128) 0 _________________________________________________________________ conv2d_5 (Conv2D) (None, 16, 16, 128) 147584 _________________________________________________________________ conv2d_6 (Conv2D) (None, 16, 16, 256) 295168 _________________________________________________________________ batch_normalization_3 (Batch (None, 16, 16, 256) 1024 _________________________________________________________________ leaky_re_lu_3 (LeakyReLU) (None, 16, 16, 256) 0 _________________________________________________________________ conv2d_7 (Conv2D) (None, 16, 16, 512) 1180160 _________________________________________________________________ conv2d_8 (Conv2D) (None, 16, 16, 512) 2359808 _________________________________________________________________ up_sampling2d (UpSampling2D) (None, 32, 32, 512) 0 _________________________________________________________________ conv2d_9 (Conv2D) (None, 32, 32, 256) 1179904 _________________________________________________________________ dropout (Dropout) (None, 32, 32, 256) 0 _________________________________________________________________ leaky_re_lu_4 (LeakyReLU) (None, 32, 32, 256) 0 _________________________________________________________________ conv2d_10 (Conv2D) (None, 32, 32, 128) 295040 _________________________________________________________________ conv2d_11 (Conv2D) (None, 32, 32, 128) 147584 _________________________________________________________________ dropout_1 (Dropout) (None, 32, 32, 128) 0 _________________________________________________________________ leaky_re_lu_5 (LeakyReLU) (None, 32, 32, 128) 0
We tried running the autoencoder on faces with real face masks, however, the results were not as accurate as the clipart face masks. We even tried using a hacky approach of overlaying a clipart face mask on top of a real face mask, however, that didn’t really work. Although a dataset with real face masks would introduce a lot of variabilities, the results would be very satisfying to see if there were enough training samples.
Source code for Mask Autoencoder: https://colab.research.google.com/drive/1_kNQl1yPUqOWRGQVxO5MK4dq_pEbkxOW
Floating in latent space was originally published in Nikhil's Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.