Rigidbodies experience huge changes in velocity if they are colliding when I change Time.timeScale

I've tried to isolate my issue as much as possible, so I have a simple scene with with only one script that changes from regular speed to slow motion and back when you press space:
In slow motion, collisions cause...

Rigidbodies experience huge changes in velocity if they are colliding when I change Time.timeScale