While it is true that F= ma, what does the damage in a collision is the transfer of Momentum from one body to another. Imagine that you lay your open hand on a bench then slowly place a 1 kg weight on it. Because if the slow speed of the hammer as it contacts your hand, there is virtually no pain. You just feel an increase in pressure at the point of contact. Now drop the hammer from 30 cms. The hammer will have a higher velocity and therefore greater momentum. When it reaches your hand, that greater momentum is going to create more pressure (Pressure = Force per unit area) as the momentum of the hammer is transferred to your hand, and thus more pain.
Also involved is the rate of change of momentum, which is called Impulse. It's the magnitude of the Impulse on your body that causes pain. If the hammer is lowered slowly, it takes longer for the transfer of momentum to take place. Therefore the Impulse is less (Impulse = Change in Momentum divided by time) than if the time taken for the hammer to transfer its momentum is less.