I haven't looked at this in detail, but at a glance, you need an "ask me later" option. There are times when I need to consider before deciding but don't or can't interrupt what I'm currently doing.

If a user can revisit a declined invitation and accept it, later, make this clearer. Still, I'd rather not have to remember to go and fish those out of the declined "list".