The Court's power lies in its perceived integrity and being above the fray of election-driven politics. If it appears partisan, it risks being marginalized.

Remember, the Court gave itself the powers it has in Marbury vs. Madison, and has at times had folks like Jackson go "let them enforce it". If public opinion turns against the Court, politicians can as well.