So, I decided to do something to fix that: a piece of wood screwed to the wooden support of the rudder will prevent it from swinging.
It is not visible in the photo but inside there is a plastic bit that pushes against the rudder to keep it from swinging. The construction is fragile enough to be what breaks if we hit the bottom.