Basically, avoid literal (insecure direct object) references to resources where possible so you have fewer areas where a server can goof authorization checks. Preempt the possibility of a server expecting any direct reference and structure your API to only load resources from a backend, not the user's input.

This goes hand in hand with abstracting all authorization checks to a single gateway/middleware layer that each call inherits, rather than a spot check per call or a group of checks for different groups of calls.

(This is in addition to what 'lvh and 'tptacek have said already.)