I have this issue myself as well, I can't say no. I recently got a promotion and it was very public for some reason.

Now people keep asking me things, trying to push their work on me. Everytime I do that I feel like I am assigning a bit more responsiblity on myself