Why is it better to subtract a baseline in REINFORCE?

Suppose we have a softmax policy over five actions. In the current state, an oracle tells us that the true Q-values are -1, -2, -3, -4, and -5. The first action is the best, even though it is negative, so we should increase the probability of it. We sample an action from our policy. It&rsquo;s the first! What a stroke of luck! Or is it? Unfortunately, our algorithm is REINFORCE. It decreases the…