I haven't used colemak, but I suspect this isn't really a problem if you're going to do a full-transition and not use qwerty at all.

But if you have to switch between two layouts regularly then you definitely want something completely-distinct rather than something close-but-not-quite.

It seems like colemak is in the awkward position where you wouldn't even consider it unless you're already comfortable with qwerty, but you're also in a position to stop using qwerty entirely and switch all your keyboard layouts.