It's a fair concern - we recommend using just Google Tag Manager (synchronous and lightweight) code to fire all your other tags asynchronously. So in most cases, where you call more than one remote code snippet, it could speed it up.