"Decision-making algorithms are politics played out at a distance, generating a troubling amount of emotional remove."
This is absolutely key. Adding distance[1] between the point where a decision is made and where the consequences of that decision are realized make it harder for any feedback from those consequences to affect the person making the decision. This makes the decisions worse (from lack of information) and the implementation worse (error must be much larger before the feedback from that error reaches the decision maker).
You see this effect in many areas. An obvious example is the law enforcement mentioned the article (or military), where "just following orders" to the modern variant of "just following an algorithm" end up causing problems.
A more interesting example might be the existence of the derivatives market and the invention of increasingly-exotic financial instruments. A bank giving someone a loan has some fairly well-known possible behaviors, and is (probably) close enough to allow feedback between the parties for things like capitalism to work (if you don't like the bank's behavior, you let them know that isn't acceptable by refinancing at a different bank). On the other hand, bad decisions bundled up and hidden in collateralized debt obligations sheltered these bad decisions until the problem blew up and introduced the world to the phrase "too big to fail".
A very interesting discussion of this problem - focused on how this kind of distance relates to human honesty (and rationalization) - is this RSA Animate featuring Dan Ariely: https://www.youtube.com/watch?v=XBmJay_qdNc
Comments
"Decision-making algorithms are politics played out at a distance, generating a troubling amount of emotional remove."
This is absolutely key. Adding distance[1] between the point where a decision is made and where the consequences of that decision are realized make it harder for any feedback from those consequences to affect the person making the decision. This makes the decisions worse (from lack of information) and the implementation worse (error must be much larger before the feedback from that error reaches the decision maker).
You see this effect in many areas. An obvious example is the law enforcement mentioned the article (or military), where "just following orders" to the modern variant of "just following an algorithm" end up causing problems.
A more interesting example might be the existence of the derivatives market and the invention of increasingly-exotic financial instruments. A bank giving someone a loan has some fairly well-known possible behaviors, and is (probably) close enough to allow feedback between the parties for things like capitalism to work (if you don't like the bank's behavior, you let them know that isn't acceptable by refinancing at a different bank). On the other hand, bad decisions bundled up and hidden in collateralized debt obligations sheltered these bad decisions until the problem blew up and introduced the world to the phrase "too big to fail".
A very interesting discussion of this problem - focused on how this kind of distance relates to human honesty (and rationalization) - is this RSA Animate featuring Dan Ariely: https://www.youtube.com/watch?v=XBmJay_qdNc
[1] measured in either number-of-hops or time