How to actually delete Rigid Bodies in Blender

I've been simulating ball physics in Blender - and it has been a little bit of a struggle. Yesterday while testing out different behavior of balls I realized that if I deleted some of the balls - Blender didn't actually remove their colliders. I couldn't see the balls themselves but I noticed that the deleted balls collided with the visible balls. I tried to restart Blender and clear the cache,…