Transfer Learning

If you’ve been following my blog, you know that I have an interest to make efficient neural networks. But every so often, I’m lost on how to modify model architecture to improve accuracy. How many layers should the neural net have? What should be the size and depth of the hidden layers?

Sometimes, I run into a different set of problems. For example, how do I make an accurate neural net if my dataset is too small? Or, how do I train a deep network if I don’t have GPUs?

This is where transfer learning comes in.

Transfer Learning — the improvement of learning in a new task through the transfer of knowledge from a related task that has already been learned. — Handbook of Research on Machine Learning Applications.

With transfer learning, you can take the knowledge that a neural network has learned from one task, and apply that knowledge to a similar, but different task. This approach works very well on Convolutional Neural Networks (CNNs) because they are feature-based networks. The first layers of a CNN detect simple features that can be recognized easy (ex. lines, edges, corners). The last layers of a CNN detect more abstract features in the image (ex. features that differentiate a cat from a dog).

Transfer learning works by pretraining your CNN on a dataset, freezing the feature layers of the CNN, and then training again on a similar dataset so that classification layers are fine-tuned for new dataset.

“Talk is cheap. Show me the code.” — Linus Torvalds

In the following implementation of transfer learning, I will be making a CNN to classify whether an image is a sunflower or a tulip.

First, I will pretrain the model on a dataset of flowers that contain daisies, dandelions, and roses. Then, I will freeze the feature layers and fine-tune my classification layers by training on my actual dataset of sunflowers and tulips.

Dataset: http://download.tensorflow.org/example_images/flower_photos.tgz.

<a href="https://medium.com/media/735875228401422eb565aeb5e411d32a/href">https://medium.com/media/735875228401422eb565aeb5e411d32a/href</a>

Pretraining on daisy, dandelion, and rose images:

Freezing feature layers and training on sunflower and tulip images:

Notice how the accuracy of the neural net starts at 84% and each epoch only takes 11 seconds to train.

For the full source code, check out the project on Github:

nikhiljay/transfer-learning

So, where am I going with all of this?

In a realistic setting, image datasets are not going to be very large, so transfer learning is very important. You can pretrain a CNN on a very large dataset such as ImageNet, and then fine-tune the higher-level layers of the CNN by training on your own dataset. Since ImageNet contains 14 million images with 20,000 categories, pretraining on ImageNet would be a nightmare (with or without GPUs). However, people have publicly released their pretrained CNNs, so that we can use them. Some examples of models for image classification with weights trained on ImageNet are: Xception, VGG16, VGG19, ResNet50, InceptionV3, InceptionResNetV2, MobileNet, DenseNet, and NASNet.

Keras has a cool way of implementing these models. For example to use the VGG19 dataset, all you need to do is:

model = applications.VGG19(weights = "imagenet", include_top=False, input_shape = (img_width, img_height, 3))

In other words, there is no need to write a CNN from scratch. Just use transfer learning.

One of the reasons why I got engrossed in machine learning is because of the similarities between animal biology and neural nets. For example, CNNs work by using multiple convolutional and fully connected layers. Well, that’s exactly how the human eye works. There are 126 million rods and cones, 6 million bipolar cells, and 2.5 million ganglion cells in the retina of the human eye (each layer has fewer cells than the previous layer).

Similar to CNNs, transfer learning is very human-like. For instance, our brain can categorize certain shaped objects as flowers and analyze the more abstract features of the object to recognize which species the flower is a part of. I think if we want to create better neural networks, we need to thoroughly understand the human sensory system.

Transfer Learning was originally published in Nikhil's Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.