I guess one could use the name of the population (Norse? Danes?) or the specific deity or something associated with it.

E.g. we refer to Greek or Roman temples rather than Pagan ones. We also say Apollo's Temple, or Solomon's temple. I guess "pagan" is a fallback for lack of more precise names.

(I didn't notice it in the title at all until GP's comment, I just realized now this is an interesting oddity)