Tensor Cores

I had always thought that the difference between high end CPUs vs GPUs were between 3x to 20x. It turns out that I was totally wrong. Modern GPUs can have up to 100x better performance than CPUs on Machine Learning workloads. This is because of Tensor Cores . Tensor Cores are parts of the GPUs that specialize in Fused-Multiply Add operations (a*b + c) . These operations are common in Machine…