"Not to be rude" is one of those statements that does the opposite of what people who use it seem to think it does. It basically announces that that's exactly their intention and they think it gives them a get-out-of-jail-free card for saying it. It's always followed by something totally unnecessary, it's never followed by something useful or insightful that the other person might inadvertently take the wrong way.

These types of phrases are common and obvious enough that I'm surprised there's not a word for them. See also:

"I hate to tell you, but" (they don't hate to tell you, in fact the delight they get from telling you is why they're telling you)

"NOW, this may NOT be a poe-lit-tic-cull-lee core-rect thing to say, BUT" (in practice, for some reason it's very important to linger on every single syllable of "politically correct" when playing this card, maybe even roll it out so slowly that there are extra syllables, because apparently this is still 1995 and political correctness is some hot button issue that exists in their world that they're always inadvertently stumbling into) (also they're about to say something totally shitty and stupid, guaranteed, and political correctness has nothing to do with it)

"I'm not a racist, but" (they're about to say something so totally racist, and if they weren't racists themselves they'd know not to prefix anything with this phrase- it's up there with "but I have black friends" as being an obvious and transparent tell)

"I'm just keeping it real" (they really want to say the quiet part out loud and think this gives them license to do so)

"I'm a straight shooter and don't pull punches, and I hope everyone treats me the same way" (only ever said by the thinnest skinned people you'll ever meet)

"I'm going to be brutally honest" (in practice, the people who preemptively declare themselves to be brutally honest are always in it for 100% of the brutality and 0% of the honesty.)