The difference is always "laughing at you" vs "with you." It's not that certain groups are "fair game" but that among equals poking fun has a default interpretation of "busting a friend's chops." Anyone can absolutely cross the line when making fun of white people.

If you're non-white and making fun of white people it can also be light mockery of your oppressors depending on the context. It has "court jester making fun of the king" energy.

Why it's almost always not kosher for white people to make fun of non-whites is because the default interpretation of jokes among non-equals is bullying and mockery. It can be done but it has to be done with extreme care.

The topical example of jokes among more complicated non-equals going extremely poorly is Chappelle's treatment of the trans community. It's not that trans folks are "off limits", you can find plenty of comedians and people on social media that make tasteful jokes at the expense of trans folks, but Chappelle's jokes just came off mean and weren't things that any trans person would react like, "haha that's so true!"