Distillation Is a Geometry, Not an Answer Key

This page did not load. You can still read it on the original site — the toolbar below keeps your place in the directory.

Knowledge distillation has a standing puzzle: Hinton's student recognized 98.6% of the digit 3s in the test set after training on a transfer set with every 3 deleted. An answer key cannot do that, so what actually crosses the wire? This post gives dark knowledge a data type, a class-similarity kernel, and runs the experiment that isolates it: a student trained on nothing but pairwise relations, no…

Read on tahabouhsine.com ↗