If there are no media with your secret key, it can't be stolen.

Of course, if your computer is cracked into, or your adversaries are using rubber-hose cryptanalysis, all bets are off. But this scenario is usually less probable than having your physical wallet or keychain stolen.