That was my thought too, the candidate spec for this[0] seems to have taken that into consideration by requiring the scripts to be served with an `Access-Control-Allow-Origin: <origin>` header.

Since the server needs to grant you full cross-origin read permissions to even start the hash check, it's not likely that an attacker could use this to infer more about cross-origin resources than they already can.

[0]: http://w3c.github.io/webappsec/specs/subresourceintegrity/