Because in most cases you already use your card interchangeably across a wide variety of (hopefully sealed and certified) terminal devices.

Meanwhile your password is very specific to one website, and never entering it elsewhere is key to phising prevention.

(my "security domain" comment was probably worded a bit poorly with the reference to your wallet, the relevant point is that most people consider card terminals interchangeable.)