If we let judges infer non-enumerated rights, they could do anything.

You are conflating rights with entitlements. The nature of non-enumerated rights is clear: Everything that is outside the enumerated powers of government, until the people decide to add to those powers through the amendment process.

We are safe from judges doing "anything."