What causes space debris to be so damaging?

I was reading an article about how destructive some of the debris in space can be. When it impacts some other object, if Force=mass times acceleration, then no matter what happens the force it applies is limited by the product of it's mass and the acceleration. So...is it the impulse, i.e. how short the time interval is, what makes a speck of paint catastrophic?