binaries compiled with that particular compiler are de-facto under GPLv3, and you have to abide by its terms (eg. if a customer using these binaries requests the source code)

You'll be violating the terms that allow you to use the compiler if your source code isn't GPL licensed. That doesn't mean your source code is automatically converted to GPL. It just means you don't have a right to use the compiler or to distribute the binaries you've produced with it.