If I was going to be afraid of an invisible killer, I would be far more concerned with gamma-ray bursts. It covers a much larger area than that a black hole and could crisp our planet like some kind of sci-fi intergalactic superweapon laser beam. And some might consider getting turned to ash in mere moments a mercy compared to the potential of a near "miss" that would give half the planet instant cancer and completely fuck our weather patterns beyond any comprehension.

Still not super likely, but I would think far more likely than a direct hit by a black hole.