I think the main point of the article is that the algorithms are applied without deference to real world conditions or impact.
This particular example appears to be a serious violation with the Americans with Disabilities Act, which I have noticed often falls by the wayside when developing many systems like this. In this case the humans deferred to the machines the decisions, most likely because the administrative regulations say so and leave no way to get around it "manually".
I have myself encountered these sort of issues, at one point many years ago I stopped filing for unemployment because I had found a job, weeks later a computer decided that my job search records needed to be audited for the period I was not filing, and if that didn't fly, they would want the last 6 months of UI back. I filed a blank form with a letter saying I was employed and who to talk to to verify that. This was based on advice of the employment department. There was no process for this, so a blank job search form was entered and nothing else done, I then was subject to collection actions, appealed and an administrative law judge interviewed me, and determined the collection was in error. A year later the employment department stopped many of the practices.
We should not blindly defer to machines, the more we do, the less we know how to, or have power to, correct situations when they go out of hand.
I think the main point of the article is that the algorithms are applied without deference to real world conditions or impact.
I'm not seeing evidence that algorithms are worse than people in this regard. How many horror stories are there of people citing rules or policy "without deference to real world conditions," or just simply being flat-out wrong? It's almost a cliche when talking about government bureaucracies.
I'm less concerned about algorithms strictly applying policy (that's what they do after all) and more interested in whether overall results are better than what humans do. Most people have biases about one thing or another (race, gender, age, physical appearances) and it's very difficult to eliminate those as they may be operating below the level of conscious thought. Also government bureaucracies aren't exactly known for hiring the best and brightest. It would seem to me likely that algorithms should eliminate those issues, at least.
Comments
I think the main point of the article is that the algorithms are applied without deference to real world conditions or impact.
This particular example appears to be a serious violation with the Americans with Disabilities Act, which I have noticed often falls by the wayside when developing many systems like this. In this case the humans deferred to the machines the decisions, most likely because the administrative regulations say so and leave no way to get around it "manually".
I have myself encountered these sort of issues, at one point many years ago I stopped filing for unemployment because I had found a job, weeks later a computer decided that my job search records needed to be audited for the period I was not filing, and if that didn't fly, they would want the last 6 months of UI back. I filed a blank form with a letter saying I was employed and who to talk to to verify that. This was based on advice of the employment department. There was no process for this, so a blank job search form was entered and nothing else done, I then was subject to collection actions, appealed and an administrative law judge interviewed me, and determined the collection was in error. A year later the employment department stopped many of the practices.
We should not blindly defer to machines, the more we do, the less we know how to, or have power to, correct situations when they go out of hand.
I think the main point of the article is that the algorithms are applied without deference to real world conditions or impact.
I'm not seeing evidence that algorithms are worse than people in this regard. How many horror stories are there of people citing rules or policy "without deference to real world conditions," or just simply being flat-out wrong? It's almost a cliche when talking about government bureaucracies.
I'm less concerned about algorithms strictly applying policy (that's what they do after all) and more interested in whether overall results are better than what humans do. Most people have biases about one thing or another (race, gender, age, physical appearances) and it's very difficult to eliminate those as they may be operating below the level of conscious thought. Also government bureaucracies aren't exactly known for hiring the best and brightest. It would seem to me likely that algorithms should eliminate those issues, at least.
Edit: posted before I saw jqm's reply