CtrlP (and other similar plugin)s doesn't have knowledge of your filesystem before you point them to a directory and it has scanned its content. Obviously, this operation can take some time on large/deep directories which will vary between implementations.

Luckily CtrlP gives you a bunch of ways to accelerate the whole thing. See `:h ctrlp`.