You have the velocity of the impact through the normal don't you?
And even I think you can get the velocity for each body before the collissioned;
And you have the mass of every body, so you can use (I think is the better) momentum for computing damage.
You besides can add to the computing algorith the elasticity of each body (I think this also is important on damage collision):
In this way you have no problem about the mass of the object, his velocity and even his energy restitution.
Greate video, I like the city "real life" sensation, I don't know wich effect you used but it gives a realistic sensations. Very nice.