I suggest that they visit all the time. The first thing a truly intelligent species would do is discard the physical form since it's inefficient and it decays over time. Think of the movie 'Lucy' where, at the end, she discards the body and just disappears. This could be the answer to the Fermi paradox since a civilisation would get to the point where this is possible and would basically disappear from the universe. Once the 'being' is part of the fabric of the universe then they could do anything and could 'exist' right in front of our noses.