Stochastic Weight Averaging Drop-in replacements for SGD (or other optimizers); Finds flat surfaces of the loss surfaces The idea is that there is no gradient information inside flat regions for SGD to explore, so SGD will explore around these regions.