This seems to be a common experience. Felix Dennis mentioned the same thing in his book - earlier on in his career, he'd just 'flip out' and go somewhere remote for a couple weeks, without even telling people ahead of time (more quirky and unplanned than your vacation!). But the same thing happened; he found that rather than adding tremendous value every second he was there, the company did much better when he got out of their way.

It's probably a good thing to do.