If an IdP is SAML compliant, you only need to write code once. Of course, if you also want to authz based on IdP roles, you will need to have a mapper (from your app perms to IdP roles), but this is not because of differences in IdP