So like... a couple people a year? As far as annoyances go, not exactly a big one.

This is a safety issue more than it is an annoyance issue. Someone may get violent if they think they have evidence that you've stolen their phone and are denying it.