Years ago someone begged to the noble of the land to give refuge from the stormy night. The noble not only refused but personally sent the beggar away.
Then the beggar revealed herself as a powerful enchantress that has been made aware of the noble's wicked ways. The enchantress called upon her magic and when it fell upon them it manifested in the form of a powerful curse that affected noble, land and servant alike. So much misery and suffering came from that single moment.
So during another night that was just as stormy, dark and cold... well, better not make the same mistake again.
The lightning is practically the only source of light and during a brief flash they can see a castle in the distance. They definitely don't remember a castle of that size anywhere nearby, but their lantern can barely shine light to let their horse proceed ahead, it's not even a question that they are lost.