Sounds like a load of crap to me. Seriously, 500 million lines of code? I bet not even mission-critical code written to power NASA spacecraft and satellites are anywhere near that length.
I propose they give the project to the HN community, divide up the $400 million figure amongst the participants and give us a weekend and I bet we'll have more to show than that of the so-called developers the government hired.
I am not underestimating the complexity of such a site, but seriously, it's not like it hasn't been done elsewhere to a degree. if I can do Internet banking, the a site for providing healthcare can be built a lot easier. They're both of similar scale and security requirements I would imagine.
This site seems like the greatest scam ever perpetuated. I bet the contractors are laughing themselves to the bank literally as we speak. Even though it's bad publicity, money is a higher priority than credibility for some businesses out there it would seem. Is it any wonder people don't trust the government? They can't even hire competent people to build them a website. This seems to be a recurring theme for local councils and governments elsewhere as well, so it's not just the US this is happening.
People need to get mad about this. That's $400 million dollars that could have been invested elsewhere like medical research, charities, food programs, rehabilitation for the homeless... A $400 million mistake has been made, and another $400 million will probably be spent before it's even ready.
I guess I also suspect that the code for satellites and space shuttles can be kept reasonably compact. I mean, these systems are (ideally) not designed to be everything for everybody, unlike Linux or Windows or KDE or Visual Studio or Firefox or what have you.
It's best to keep the code in everything to the minimum!
Optimizing for readability is usually most important.
Readability is usually more important than reusability, in fact, although readability very often enables reusability.
But we're talking about mission-critical embedded code, which is a special case in many respects. Still, making it as readable as possible makes bugs easier to find.
Thanks for saying this... it needs to be said far more often.
I wish there were a book about how to READ software. (If anyone knows of one, please speak up.)
One tip I picked up on my own over time is when you're looking at an unfamiliar project, don't read the code like a book, read it like a program. In other words, try to find main() and trace how things work from the bottom up, rather than trying to understand it based on how the authors have broken up the code into directories. I don't know yet whether this is truly good advice.
I prefer the term "maintainability" to "readability", and I completely agree with you there.
Reuse is an emergent property of code, though. It's very tough to engineer for reuse upfront and have that effort be successful. It's best to keep your code simple and put your effort into making it well-tested. Then, over time spot the commonalities in everyone's code, and factor out the common bits to become reusable code if everyone agrees it's worth it to do so.
I definitely agree with that. I was thinking of minimal code as a way to achieve the ends of readability and maintainability. Excessive minimalism goes against both of those goals, when you get to compactness for the sake of compactness, perl code golf or micro optimization style.
Yeah, I remember reading somewhere that they have extremely strict standards at NASA, to the point that multiple reviews are mandatory on even single line changes for mission critical systems.
Yeah, that's the best example I could come up with at the time of writing the comment. It's been a long day today, and I don't mind the nit-pick, you are right.
Yes, we've burned at least $400 mil. Last week, everyone insisted it only cost $55 million, and the $768 million figure was misreported. I can only speculate what the final cost will be after victory is declared over these bugs.
It is too late to get mad about any new/rising costs related to the ACA. It is here to stay and might as well be Medicare 2. "Law of the land" as the president is fond of saying.
Here's a fascinating visio diagram that shows just how much government has grown under the ACA. Anything red indicates new or expanded government:
Seems like the ACA created very painful integrations between federal systems and insurers, asking them to do many new things. Not a big surprise they're having problems. Some federal IT systems are archaic and integrations/change are costly and fraught with danger. The insider stories I've read about the Soc Security mainframe-based system are frightening!
"It is too late to get mad about any new/rising costs related to the ACA. It is here to stay and might as well be Medicare 2. "Law of the land" as the president is fond of saying."
Reading this article, which at the highest level tells us the people in charge, the White House and HSS's CMMS (all government workers, the latter the integrator of the project) are still discussing how to fix this mess, hoping to have a plan by Thursday, combined with e.g. CMMS delivering changes to the contractors through the week before launch, and per this article "In the last 10 months alone, government documents show, officials modified hardware and software requirements for the exchange seven times.", suggests to me it could easily be years before the exchange works, and possibly never. We've seen this before (FAA, FBI, IRS, etc.).
All depends on CMMS (non-specialist civil servants and perhaps political appointees) learning big project management "on the job" or the vast majority of them getting replaced by an integrator who has a clue, either of which will take months.
That's just what's required to get sane high level management of the project.
In the meanwhile, can the individual parts done by different contractors get some serious progress? Well, first it depends on CMMS freezing requirements, and doing a good enough job of deciding what requires a total rewrite. Per this article:
"Quality Software Services Inc., or Q.S.S.I., a unit of the UnitedHealth Group, developed the identity management system, another major component that allowed consumers to register and establish accounts.
The identity management system from Q.S.S.I., which also taps into government databases to retrieve users’ personal information, was a particular source of trouble when the exchange opened. Change orders show that on Oct. 4 — after millions of people had been trapped in technological loops trying merely to log in — the government asked CGI to help it devise a new identity management system to replace the one provided by Q.S.S.I. But specialists said that approach was abandoned as too risky. Ultimately it was decided to fix the current identity system."
And we've heard from a HN member who was involved in a big university project that Oracle's identity software is fairly gnarly to get working. Which I have absolutely no trouble believing, because the underlying problem is gnarly.
But one wonders just how much panic was behind CMMS's idea of punting the existing one 3 days after launch. And you know the managers of both contractors had to spend a lot of time working with CMMS on just this detail ... working with people who are simply not (very) experienced in software development and management.
(Ask for my worst experience working for a government for the level of cluelessness that's likely involved here.)
So, getting to my point, you may say "It is here to stay", but "You Can't Fight Mother Nature" or the hard reality that computers do exactly what you tell them to do, not what you want them to do. How many months or years do you think people will stand for being fined by the IRS for not enrolling when they can't? How many will die because they've lost their existing coverage due to the ACA changes but can't sign up for anything, or anything they can afford without the subsidies which only the exchanges can (now) deliver? What happens if this not entirely nonfunctional system develops an adverse selection death spiral?
That's where only the very dedicated, the ones who really need a lot of healthcare, jump through the hoops, whereas the healthy 20 somethings and beyond, who are asked to do a 30 step (!) application on an unreliable system, punt, pay the fine but not the much larger sums needed to subsidize the 50-64 age bracket. Which raises the prices of the policies offered, which further decreases enrollment....
It does seem that dismal failure is the most likely outcome.
I just read between the lines when I hear "law of the land" and I think there's NO way the administration is going to back down here. It is going to be interesting to see how things go down. Perhaps they'll just hire human agents to take the place of the broken site?
Can human agents handle the 7 million applicants needed to make it work (at least early on)? And they'll still need computer support, I presume you're only talking about taking the place of the front end web site.
Trying to do 7 million identity checks manually? Well, maybe something can be kludged to speed that up. On the other hand subsidy calculations had better be done by computer, and that's not working reliably right now; it could probably be Tiger Teamed into something that works and at scale, a simple calculator (heck, I bet it could be done in Javascript).
But again I get to the biggest problem of management. Starting at the very top, the CEO of this organization (Obama, per his Rose Garden event earlier today), they're in (public) denial of how bad the problem. In part they can't help it, they aren't experienced IT people, they aren't like us who can look at less than 1,000 words of clues and know "it's fucked".
Really salvaging anything out of this needs decisiveness. E.g. to get your kludge working for 2014---and again I note the critical detail of subsidies, especially since the only plan that can now be offered is gold-plated (e.g. coverage of your children through age 26) and inherently very expensive even before you get to community rating (everyone under 50 is to pay a bundle to cover the 50-64 set)---someone would have to decide to start it now. "Wastefully", in parallel with trying to get the site to work. As it is, millions are losing their old-fashioned high deductible major medical polices as of Jan 1st, things start getting really bad for a lot of people really soon.
If this nightmare drags out to 2016 (which today we can't discount, and that's even counting the website or its substitutes vaguely working), do you really expect it to be the law of the land by 2018?
Comments
Sounds like a load of crap to me. Seriously, 500 million lines of code? I bet not even mission-critical code written to power NASA spacecraft and satellites are anywhere near that length.
I propose they give the project to the HN community, divide up the $400 million figure amongst the participants and give us a weekend and I bet we'll have more to show than that of the so-called developers the government hired.
I am not underestimating the complexity of such a site, but seriously, it's not like it hasn't been done elsewhere to a degree. if I can do Internet banking, the a site for providing healthcare can be built a lot easier. They're both of similar scale and security requirements I would imagine.
This site seems like the greatest scam ever perpetuated. I bet the contractors are laughing themselves to the bank literally as we speak. Even though it's bad publicity, money is a higher priority than credibility for some businesses out there it would seem. Is it any wonder people don't trust the government? They can't even hire competent people to build them a website. This seems to be a recurring theme for local councils and governments elsewhere as well, so it's not just the US this is happening.
People need to get mad about this. That's $400 million dollars that could have been invested elsewhere like medical research, charities, food programs, rehabilitation for the homeless... A $400 million mistake has been made, and another $400 million will probably be spent before it's even ready.
What a waste of money.
While I completely agree with you in general, a nit pick:
That doesn't seem like a good example. You want to keep the code in those kinds of things to a minimum, and that's probably not too hard.
It's best to keep the code in everything to the minimum!
True!
I guess I also suspect that the code for satellites and space shuttles can be kept reasonably compact. I mean, these systems are (ideally) not designed to be everything for everybody, unlike Linux or Windows or KDE or Visual Studio or Firefox or what have you.
Optimizing for readability is usually most important.
Readability is usually more important than reusability, in fact, although readability very often enables reusability.
But we're talking about mission-critical embedded code, which is a special case in many respects. Still, making it as readable as possible makes bugs easier to find.
Thanks for saying this... it needs to be said far more often.
I wish there were a book about how to READ software. (If anyone knows of one, please speak up.)
One tip I picked up on my own over time is when you're looking at an unfamiliar project, don't read the code like a book, read it like a program. In other words, try to find main() and trace how things work from the bottom up, rather than trying to understand it based on how the authors have broken up the code into directories. I don't know yet whether this is truly good advice.
I prefer the term "maintainability" to "readability", and I completely agree with you there.
Reuse is an emergent property of code, though. It's very tough to engineer for reuse upfront and have that effort be successful. It's best to keep your code simple and put your effort into making it well-tested. Then, over time spot the commonalities in everyone's code, and factor out the common bits to become reusable code if everyone agrees it's worth it to do so.
I definitely agree with that. I was thinking of minimal code as a way to achieve the ends of readability and maintainability. Excessive minimalism goes against both of those goals, when you get to compactness for the sake of compactness, perl code golf or micro optimization style.
Yeah, I remember reading somewhere that they have extremely strict standards at NASA, to the point that multiple reviews are mandatory on even single line changes for mission critical systems.
Here's an example: http://spinroot.com/gerard/pdf/P10.pdf
And another: http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
Dunno how closely these are followed in day-to-day operations, but they clearly put some thought into it.
Yeah, that's the best example I could come up with at the time of writing the comment. It's been a long day today, and I don't mind the nit-pick, you are right.
Yes, we've burned at least $400 mil. Last week, everyone insisted it only cost $55 million, and the $768 million figure was misreported. I can only speculate what the final cost will be after victory is declared over these bugs.
It is too late to get mad about any new/rising costs related to the ACA. It is here to stay and might as well be Medicare 2. "Law of the land" as the president is fond of saying.
Here's a fascinating visio diagram that shows just how much government has grown under the ACA. Anything red indicates new or expanded government:
http://www.jec.senate.gov/republicans/public/index.cfm?a=Fil...
Seems like the ACA created very painful integrations between federal systems and insurers, asking them to do many new things. Not a big surprise they're having problems. Some federal IT systems are archaic and integrations/change are costly and fraught with danger. The insider stories I've read about the Soc Security mainframe-based system are frightening!
Here's the frightening story I recalled on Social Security from 2011. http://www.informationweek.com/government/enterprise-archite...
Woe be to the phalanx of ACA consultants that has to integrate with those systems... :/
"It is too late to get mad about any new/rising costs related to the ACA. It is here to stay and might as well be Medicare 2. "Law of the land" as the president is fond of saying."
Reading this article, which at the highest level tells us the people in charge, the White House and HSS's CMMS (all government workers, the latter the integrator of the project) are still discussing how to fix this mess, hoping to have a plan by Thursday, combined with e.g. CMMS delivering changes to the contractors through the week before launch, and per this article "In the last 10 months alone, government documents show, officials modified hardware and software requirements for the exchange seven times.", suggests to me it could easily be years before the exchange works, and possibly never. We've seen this before (FAA, FBI, IRS, etc.).
All depends on CMMS (non-specialist civil servants and perhaps political appointees) learning big project management "on the job" or the vast majority of them getting replaced by an integrator who has a clue, either of which will take months.
That's just what's required to get sane high level management of the project.
In the meanwhile, can the individual parts done by different contractors get some serious progress? Well, first it depends on CMMS freezing requirements, and doing a good enough job of deciding what requires a total rewrite. Per this article:
"Quality Software Services Inc., or Q.S.S.I., a unit of the UnitedHealth Group, developed the identity management system, another major component that allowed consumers to register and establish accounts.
The identity management system from Q.S.S.I., which also taps into government databases to retrieve users’ personal information, was a particular source of trouble when the exchange opened. Change orders show that on Oct. 4 — after millions of people had been trapped in technological loops trying merely to log in — the government asked CGI to help it devise a new identity management system to replace the one provided by Q.S.S.I. But specialists said that approach was abandoned as too risky. Ultimately it was decided to fix the current identity system."
And we've heard from a HN member who was involved in a big university project that Oracle's identity software is fairly gnarly to get working. Which I have absolutely no trouble believing, because the underlying problem is gnarly.
But one wonders just how much panic was behind CMMS's idea of punting the existing one 3 days after launch. And you know the managers of both contractors had to spend a lot of time working with CMMS on just this detail ... working with people who are simply not (very) experienced in software development and management.
(Ask for my worst experience working for a government for the level of cluelessness that's likely involved here.)
So, getting to my point, you may say "It is here to stay", but "You Can't Fight Mother Nature" or the hard reality that computers do exactly what you tell them to do, not what you want them to do. How many months or years do you think people will stand for being fined by the IRS for not enrolling when they can't? How many will die because they've lost their existing coverage due to the ACA changes but can't sign up for anything, or anything they can afford without the subsidies which only the exchanges can (now) deliver? What happens if this not entirely nonfunctional system develops an adverse selection death spiral?
That's where only the very dedicated, the ones who really need a lot of healthcare, jump through the hoops, whereas the healthy 20 somethings and beyond, who are asked to do a 30 step (!) application on an unreliable system, punt, pay the fine but not the much larger sums needed to subsidize the 50-64 age bracket. Which raises the prices of the policies offered, which further decreases enrollment....
It does seem that dismal failure is the most likely outcome.
I just read between the lines when I hear "law of the land" and I think there's NO way the administration is going to back down here. It is going to be interesting to see how things go down. Perhaps they'll just hire human agents to take the place of the broken site?
Can human agents handle the 7 million applicants needed to make it work (at least early on)? And they'll still need computer support, I presume you're only talking about taking the place of the front end web site.
Trying to do 7 million identity checks manually? Well, maybe something can be kludged to speed that up. On the other hand subsidy calculations had better be done by computer, and that's not working reliably right now; it could probably be Tiger Teamed into something that works and at scale, a simple calculator (heck, I bet it could be done in Javascript).
But again I get to the biggest problem of management. Starting at the very top, the CEO of this organization (Obama, per his Rose Garden event earlier today), they're in (public) denial of how bad the problem. In part they can't help it, they aren't experienced IT people, they aren't like us who can look at less than 1,000 words of clues and know "it's fucked".
Really salvaging anything out of this needs decisiveness. E.g. to get your kludge working for 2014---and again I note the critical detail of subsidies, especially since the only plan that can now be offered is gold-plated (e.g. coverage of your children through age 26) and inherently very expensive even before you get to community rating (everyone under 50 is to pay a bundle to cover the 50-64 set)---someone would have to decide to start it now. "Wastefully", in parallel with trying to get the site to work. As it is, millions are losing their old-fashioned high deductible major medical polices as of Jan 1st, things start getting really bad for a lot of people really soon.
If this nightmare drags out to 2016 (which today we can't discount, and that's even counting the website or its substitutes vaguely working), do you really expect it to be the law of the land by 2018?