I think there's sometimes a bug with the X. I totally believe you're not misclicking.

The X button has always been weird because it's not visible until you hover, it has a weird fade in, it's way too tiny, and it seems like sometimes the notification registers a click rather than the X. It's bad all around, and I don't understand why they still haven't improved it.