> People always use causality to argue against FTL, but then readily acknowledge that wormholes or Albicurie might allow it

Wormholes and Alcubierre drives do lead to causality violations. That's why most physicists view them as not physically possible. They're just mathematical solutions to the Einstein Field Equations that can't be physically realized. There are plenty of other such solutions; nobody has ever claimed that every mathematical solution of the equations has to correspond to something that's physically possible.