I have a top end Pixel phone running stock Android and encounter latency all the time. All first start time is usually a couple of seconds. Switching back to an open app is fast in many cases but some still have to do a refresh (which I suspect involves communication to servers, although that's still not much of an excuse).