It's interesting idea but a lot of mail "contact" is either rare but still important, or one off thing that you read and/or remove/spam.

Funnily enough in old school mail client you could trivally make directory per contact via sieve filter but mainstream mail clients don't really want to give users much power.