Why Do Votes Go Down?

Votes are only supposed to go up as they get added together.