The main reason that it is an issue George is that even more than nuclear the exact same mechanism to ship processed ore downhill in the gravity well to Earth can be converted to function as high mass/velocity rail guns capable of inflicting serious harm to the planet. Accidents might be possible but intentional adverse use is the greatest threat as the impacts would yield nuclear equivalent force without the associated nuclear radiation.

The irony is also that the same easiest method of changing orbital Delta Vee of known NEO's might use smaller asteroids crashed (with lots of English ;) into larger ones (or comets) (just like billiards) to bring the larger objects closer to Earth for a closer more cost efficient capture.

This is one of the reasons however that I advocate using the Moon's gravity well for primary capture targeting rather than the Earth's directly.

We are going to need to build roughly the same infrastructure to protect the planet from the existential risk of an asteroid collision as we would need to mine and profit from asteroids and comets.

The profits will be in many forms, life support materials harvested in space that aren't flown up into orbit at excessive cost, real revenues from manufactured goods and refined ore, as well as viable habitats creating the necessary stepping stone oasis' to cross and develop the Solar System and beyond.