Twelve Attempts at an FP4 Kernel

I recently participated in GPU Mode's NVFP4 Kernel Hackathon. The goal was to write CUDA kernels for Blackwell's new 4-bit floating point format (NVFP4) and get as close as possible to the hardware's speed of light. The competition ran on B200 GPUs, and there were four tasks in total: matrix-vector multiplication (GEMV), matrix-matrix multiplication (GEMM), gated GEMM (with SiLU activation), and…