Employees at the center referred her to the online system. Uncomfortable with the technology, she asked for help with the online forms and was refused.
Seems like it was humans that failed her. I'm not sure what algorithms have to do with this.
I think the main point of the article is that the algorithms are applied without deference to real world conditions or impact.
This particular example appears to be a serious violation with the Americans with Disabilities Act, which I have noticed often falls by the wayside when developing many systems like this. In this case the humans deferred to the machines the decisions, most likely because the administrative regulations say so and leave no way to get around it "manually".
I have myself encountered these sort of issues, at one point many years ago I stopped filing for unemployment because I had found a job, weeks later a computer decided that my job search records needed to be audited for the period I was not filing, and if that didn't fly, they would want the last 6 months of UI back. I filed a blank form with a letter saying I was employed and who to talk to to verify that. This was based on advice of the employment department. There was no process for this, so a blank job search form was entered and nothing else done, I then was subject to collection actions, appealed and an administrative law judge interviewed me, and determined the collection was in error. A year later the employment department stopped many of the practices.
We should not blindly defer to machines, the more we do, the less we know how to, or have power to, correct situations when they go out of hand.
I think the main point of the article is that the algorithms are applied without deference to real world conditions or impact.
I'm not seeing evidence that algorithms are worse than people in this regard. How many horror stories are there of people citing rules or policy "without deference to real world conditions," or just simply being flat-out wrong? It's almost a cliche when talking about government bureaucracies.
I'm less concerned about algorithms strictly applying policy (that's what they do after all) and more interested in whether overall results are better than what humans do. Most people have biases about one thing or another (race, gender, age, physical appearances) and it's very difficult to eliminate those as they may be operating below the level of conscious thought. Also government bureaucracies aren't exactly known for hiring the best and brightest. It would seem to me likely that algorithms should eliminate those issues, at least.
I agree.
It seems like it's a design problem, for her at least. When I read this I couldn't help but think of AKO (Army Knowledge Online), which, ostensibly, is supposed to be the Army "one source" for soldier information. It's basically a website made in like 2001 or something, before mobile internet was a thing. No one building/managing the site is paying attention to how people actually interact with it, otherwise they'd notice that the links people most often click on are at the least accessible point. And that it's just a hideous website, to start with.
It seems to be a recurring theme here that when citizens interact with government technology they are met with a UI that doesn't make sense or that doesn't work for them. I think that's sad. I really want to change that...not sure how yet, I guess.
Edit: I'm for sure not trying to downplay the global implications of this post. Especially since I've seen how drone decisions actually play out. Basically, we all need to pay attention. That's all I'm saying.
That's because most government agencies don't know how to run a software project. From experience, they don't understand the requirements, don't pay attention to conflicting requirements, and often times have no choice in who they hire as a contractor (must submit 3 bids, take the lowest cost.) So what you end up with are a bunch of under paid, under qualified and under staffed people building things.
Does anyone have a link to the lady's obituary? According to the story, without government money, she was surely going to die. And, if she didn't die, then the hyperbole about the "life sustaining nutrition" was just bullshit. It goes to make me wonder just how necessary government handouts really are.
Also, she has emphysema and COPD -- what are the odds that was caused by smoking? How much money does smoking cost? $5 per day? How many years did this woman smoke? Has she quit smoking? Did she die from starvation? My point is that her being deaf isn't a money-deserving disability. There are thousands of deaf people who aren't on welfare. Her health problems would likely have been caused by years of smoking. Why should my tax dollars go to support someone who caused many of her own problems? If I bet my life savings on blackjack, should that make me eligible for government assistance? If I am an alcoholic, should I get a liver transplant?
So, unless she died from starvation, the algorithm worked. She shouldn't be getting my money just because she has had a lifetime of bad decisions. As far as the bipolar, I could point to dozens of programs in almost every state that would provide her with free care for that.
As far as the other gent, the prospective gang member, do we have any evidence of harm caused by that algorithm? Was he wrongly arrested, detained or otherwise victimized due to that algorithm? His history does seem almost textbook in terms of risk factors to commit a violent crime. Unless his rights were infringed upon, I'm not sure I understand the issue.
Comments
Seems like it was humans that failed her. I'm not sure what algorithms have to do with this.
I think the main point of the article is that the algorithms are applied without deference to real world conditions or impact.
This particular example appears to be a serious violation with the Americans with Disabilities Act, which I have noticed often falls by the wayside when developing many systems like this. In this case the humans deferred to the machines the decisions, most likely because the administrative regulations say so and leave no way to get around it "manually".
I have myself encountered these sort of issues, at one point many years ago I stopped filing for unemployment because I had found a job, weeks later a computer decided that my job search records needed to be audited for the period I was not filing, and if that didn't fly, they would want the last 6 months of UI back. I filed a blank form with a letter saying I was employed and who to talk to to verify that. This was based on advice of the employment department. There was no process for this, so a blank job search form was entered and nothing else done, I then was subject to collection actions, appealed and an administrative law judge interviewed me, and determined the collection was in error. A year later the employment department stopped many of the practices.
We should not blindly defer to machines, the more we do, the less we know how to, or have power to, correct situations when they go out of hand.
I think the main point of the article is that the algorithms are applied without deference to real world conditions or impact.
I'm not seeing evidence that algorithms are worse than people in this regard. How many horror stories are there of people citing rules or policy "without deference to real world conditions," or just simply being flat-out wrong? It's almost a cliche when talking about government bureaucracies.
I'm less concerned about algorithms strictly applying policy (that's what they do after all) and more interested in whether overall results are better than what humans do. Most people have biases about one thing or another (race, gender, age, physical appearances) and it's very difficult to eliminate those as they may be operating below the level of conscious thought. Also government bureaucracies aren't exactly known for hiring the best and brightest. It would seem to me likely that algorithms should eliminate those issues, at least.
Edit: posted before I saw jqm's reply
I agree. It seems like it's a design problem, for her at least. When I read this I couldn't help but think of AKO (Army Knowledge Online), which, ostensibly, is supposed to be the Army "one source" for soldier information. It's basically a website made in like 2001 or something, before mobile internet was a thing. No one building/managing the site is paying attention to how people actually interact with it, otherwise they'd notice that the links people most often click on are at the least accessible point. And that it's just a hideous website, to start with.
It seems to be a recurring theme here that when citizens interact with government technology they are met with a UI that doesn't make sense or that doesn't work for them. I think that's sad. I really want to change that...not sure how yet, I guess.
Edit: I'm for sure not trying to downplay the global implications of this post. Especially since I've seen how drone decisions actually play out. Basically, we all need to pay attention. That's all I'm saying.
That's because most government agencies don't know how to run a software project. From experience, they don't understand the requirements, don't pay attention to conflicting requirements, and often times have no choice in who they hire as a contractor (must submit 3 bids, take the lowest cost.) So what you end up with are a bunch of under paid, under qualified and under staffed people building things.
Does anyone have a link to the lady's obituary? According to the story, without government money, she was surely going to die. And, if she didn't die, then the hyperbole about the "life sustaining nutrition" was just bullshit. It goes to make me wonder just how necessary government handouts really are.
Also, she has emphysema and COPD -- what are the odds that was caused by smoking? How much money does smoking cost? $5 per day? How many years did this woman smoke? Has she quit smoking? Did she die from starvation? My point is that her being deaf isn't a money-deserving disability. There are thousands of deaf people who aren't on welfare. Her health problems would likely have been caused by years of smoking. Why should my tax dollars go to support someone who caused many of her own problems? If I bet my life savings on blackjack, should that make me eligible for government assistance? If I am an alcoholic, should I get a liver transplant?
So, unless she died from starvation, the algorithm worked. She shouldn't be getting my money just because she has had a lifetime of bad decisions. As far as the bipolar, I could point to dozens of programs in almost every state that would provide her with free care for that.
As far as the other gent, the prospective gang member, do we have any evidence of harm caused by that algorithm? Was he wrongly arrested, detained or otherwise victimized due to that algorithm? His history does seem almost textbook in terms of risk factors to commit a violent crime. Unless his rights were infringed upon, I'm not sure I understand the issue.