Am I missing something?

Yes

You would typically remove the default any to any ACL rule

This part doesn’t happen.

Defaults are rarely changed.