This may be related to overall linux interactivity scheduling. There has been a running debate on the Linux kernel mailing list about this for years. The Linux scheduler is more efficient for server-workloads and compiles, but it is not as good for interactivity.

Very recent kernels have a new scheduler which supports a new scheduling algorithm which people are anicdotally reporting better interactivity with. Perhaps this will help the situation.