Bugger is not common, but also not uncommon American slang. It has rather lost it's original meaning. I have heard little old American ladies use it, probably having heard it in a movie, with the intention of a meaning derived from to bug (bother). I don't think any slang dictionary has ever caught up with this usage.

I've used bugger since the first of the Austin Powers movies. My friends and I used it until one of them married a South African girl and she made us stop being as it was incredibly impolite and rude apparently. I started again last year after watching Top Gear and then recently got yelled at by a new accounting manager who moved into the office across from my cubicle. Once again, I was being crude but in all honesty, for me, the word is no different than saying frag, frell, crude bunnies, etc.