You could eliminate a lot of complexity by first checking if the email itself is signed. If it is from someone you know then you are done. If it is not from someone you know (i.e. anonymous) you would then kick in some sort of external reputation system ... or not even bother, just add a bunch of spam points.

I think that sometimes we forget that we are almost always sending and receiving anonymous emails and how weird that is.