I'd contend that the "signal when permitted" minigame makes Jeopardy! more difficult, too, because the part of your brain that arrives at the answer must yield to the part of your brain that tenses for a stimulus. I definitely have to switch "modes" between pulling up deeply buried knowledge and rapid reaction "stance", and I noticed this when I played a well-made copy of the show's rules complete with the signaling system.

There's another side, too, which is if folks could signal early, reading fast would become an even stronger optimization. It already is, but at least everybody gets an equal chance the way it is (even if they got the answer fairly quickly from reading the clue while ignoring Trebek).