Ghosts can be hit only by spells or by weapons made of specific materials (silver, witchwood). That said, it is a bug that they spawn around energy crystals, they should only spawn in their dedicated events.