ML paper notes

2017-09 #

LEARNING FINE-GRAINED IMAGE SIMILARITY WITH DEEP RANKING #

• describes efficient sampling technique based on reservoir sampling for building triplets; requires an relevance function
• multi scale CNN

DEEP METRIC LEARNING USING TRIPLET NETWORK #

• learns a semantic embedding; results show better discrimination vs siamese network (contrastive loss function)
• MSE softmax shows improved performance rather than simple binary softmax (see paper for def)
• feed a triplet of x, x1, x2 where x1 is same class as x and x2 is different

DISTILLING THE KNOWLEDGE IN A NEURAL NETWORK #

• explores compression technique of ensemble model into a single model
• Distillation
  • softmax qi = exp(zi/T)/Sigma(j)(exp(zj/T) where z are logits and T is temperature
  • T is usually 1
  • increasing T creates softer probability distribution
  • knowledge is tranferred via training smaller/compressed model by targeting over softer target (ie temperature T > 1) from more cumbersome model
  • small model trained with higher T as well but in prediction mode uses T = 1
  • tranfer training can be improved by using datasets with true label
• demonstrate distillation with minist dataset - tranfer works well even when smaller model trained by omitting certain numbers
• discusses using soft distribution target technique for training specialists on very large datasets
• Google internal JFT data of 100M images

Questions #

• teacher - student model, relation to curriculum learning?