That is not my experience (doing this sort of testing on compilers). In my experience, the shrinking is usually very fast. When it takes a non-negligible amount of time, debugging the failure would have been hopeless on the non-shrunk input.

What takes the time in my case was simply getting a failure to occur at all. It might take days on a mature compiler before a failure occurs, if then. This would be millions of attempts.