The SDXL model was trained on different aspect ratios, but each one of those aspect ratios has a precise resolution, so anything other than those resolutions would be considered out of distribution, so maybe when you were inspecting the wide landscape images you use an out-of-distribution resolution while with the portrait ones you didn't as much.