I guess that the kernel cannot know that the application (in your case scp) will not try to touch the data ever again.

The current default behavior, which you call crazy, does however favor programs which do actually need the data that was just read (e.g., databases).