This doesn't surprise me AT ALL. You guys wouldn't believe some of the stuff I've seen out there. Work in the industrial automation field is largely done by individuals/companies called System Integrators. Integrators are cowboys and most of the industry is an unregulated wild-west. There is a pervasive "git-er-done" attitude; nothing else matters, security included.
(I'm a developer at one of the smaller SCADA software companies.)
Indeed, while our politicians bloviate about 'terrorism' and how safe they're keeping us, this is the reality. Notable that this occurred in Texas, where the legislature would be as likely consult Jesus about the incident as a security expert. I honestly have to wonder whether our society will be going down in flames soon due to the odd combination of reliance upon and willful, resentful ignorance of knowledge that is being promoted in public culture.
Just sit in on a discussion of the business case of a new technology. "This is where the future is, we need to be where consumers/customers are, everything must be digital." So they spend $3m on a new "social" or "cloud" platform, then say "oh yeah, we don't have the budget for security monitoring or personnel to cover this platform. Roll it out anyway."
Amen to that. I do some work on the network side of control systems—and default passwords (or no passwords) are rampant, and often recommended under manufacturer guidelines. The last part there is the more worrisome.
Business phone systems are the same. Most manufactures only allow a 4 char password. Now phone systems are being hack left and right; "hackers" (more like organized crime) selling carrier service and the company gets the bill.
It doesn't surprise me at all either. SCADA systems invariably seem to be ancient museum pieces pressed into service out of necessity and kept in service out of inertia.
The kind of shit I've seen in SCADA comms rooms blows my mind -- bare, homebrew breadboards screwed into 19" racks; SparcStations caked with dirt and grime; passwd files containing active accounts for people who are now dead.
The only thing I find surprising about any of this is that it doesn't happen every single day.
Well, just so you know, it's not limited to your industry. The "You guys wouldn't believe some of the stuff I've seen out there" is almost instulting given what I have seen. I guess it's just anybody out there who's been in contact the real world who can say that.
The big industrial automation and SCADA companies all provide some level of application engineering/system integration services. But, it's only practical to do the integration work in-house for large (millions of dollars in revenue annually or tens of millions of dollars in revenue one time) customers. Managing thousands of local teams of integrators all over the world to take on smaller jobs would be prohibitively expensive.
For smaller, "one-off" jobs, the integration work is done by distributors or by 3rd party integrators. Distributors will often do the integration for free and cover the integration cost and their profit from the discount the manufacturer gives them from list price. Most integrators do fixed-price bids for work, and may also make money from equipment markups. In both cases, there is a lot of incentive to do the minimum possible, especially since the projects tend to be poorly specified.
This is all made worse by the fact that the customers tend to be technically unsophisticated. That makes it hard for them to effectively manage projects, and hard for them to make informed judgements when selecting suppliers. Suppliers are usually picked based on personal relationships with the sales team (manufacturer's or distributor's) and the in-house engineer's familiarity with a given supplier.
Finally, the whole industrial automation industry isn't terribly glamorous. The typical problems being solved on any given job have been solved thousands of times before. The technology is often old and clunky (the most common language is called ladder logic... look it up, it's good for a laugh). Being successful requires a mixture of software, electrical engineering, mechanical engineering, and sales skills. Since most distributors and integrators live almost hand-to-mouth, sales skills tend to be emphasized, even among the engineers. The engineers who are good at sales find they can make more money doing sales. The ones who aren't salesy find there isn't much room for advancement and move on. I'm over-generalizing, but the overall trends don't encourage high-quality software engineering.
Some of the huge guys like Rockwell and Siemens may offer integration services; I'm not 100% sure though.
But I'm guessing it's mostly because system integrators are qualified in a way a software company isn't. The scope of a project that an integrator might take on can be vast and the SCADA software is generally only one piece of the equation.
Comments
This doesn't surprise me AT ALL. You guys wouldn't believe some of the stuff I've seen out there. Work in the industrial automation field is largely done by individuals/companies called System Integrators. Integrators are cowboys and most of the industry is an unregulated wild-west. There is a pervasive "git-er-done" attitude; nothing else matters, security included.
(I'm a developer at one of the smaller SCADA software companies.)
Indeed, while our politicians bloviate about 'terrorism' and how safe they're keeping us, this is the reality. Notable that this occurred in Texas, where the legislature would be as likely consult Jesus about the incident as a security expert. I honestly have to wonder whether our society will be going down in flames soon due to the odd combination of reliance upon and willful, resentful ignorance of knowledge that is being promoted in public culture.
Just sit in on a discussion of the business case of a new technology. "This is where the future is, we need to be where consumers/customers are, everything must be digital." So they spend $3m on a new "social" or "cloud" platform, then say "oh yeah, we don't have the budget for security monitoring or personnel to cover this platform. Roll it out anyway."
Amen to that. I do some work on the network side of control systems—and default passwords (or no passwords) are rampant, and often recommended under manufacturer guidelines. The last part there is the more worrisome.
Business phone systems are the same. Most manufactures only allow a 4 char password. Now phone systems are being hack left and right; "hackers" (more like organized crime) selling carrier service and the company gets the bill.
It doesn't surprise me at all either. SCADA systems invariably seem to be ancient museum pieces pressed into service out of necessity and kept in service out of inertia.
The kind of shit I've seen in SCADA comms rooms blows my mind -- bare, homebrew breadboards screwed into 19" racks; SparcStations caked with dirt and grime; passwd files containing active accounts for people who are now dead.
The only thing I find surprising about any of this is that it doesn't happen every single day.
Well, just so you know, it's not limited to your industry. The "You guys wouldn't believe some of the stuff I've seen out there" is almost instulting given what I have seen. I guess it's just anybody out there who's been in contact the real world who can say that.
But yeah, the real world kind of stinks.
What's stopping the SCADA software companies from taking on the integration work themselves?
The big industrial automation and SCADA companies all provide some level of application engineering/system integration services. But, it's only practical to do the integration work in-house for large (millions of dollars in revenue annually or tens of millions of dollars in revenue one time) customers. Managing thousands of local teams of integrators all over the world to take on smaller jobs would be prohibitively expensive.
For smaller, "one-off" jobs, the integration work is done by distributors or by 3rd party integrators. Distributors will often do the integration for free and cover the integration cost and their profit from the discount the manufacturer gives them from list price. Most integrators do fixed-price bids for work, and may also make money from equipment markups. In both cases, there is a lot of incentive to do the minimum possible, especially since the projects tend to be poorly specified.
This is all made worse by the fact that the customers tend to be technically unsophisticated. That makes it hard for them to effectively manage projects, and hard for them to make informed judgements when selecting suppliers. Suppliers are usually picked based on personal relationships with the sales team (manufacturer's or distributor's) and the in-house engineer's familiarity with a given supplier.
Finally, the whole industrial automation industry isn't terribly glamorous. The typical problems being solved on any given job have been solved thousands of times before. The technology is often old and clunky (the most common language is called ladder logic... look it up, it's good for a laugh). Being successful requires a mixture of software, electrical engineering, mechanical engineering, and sales skills. Since most distributors and integrators live almost hand-to-mouth, sales skills tend to be emphasized, even among the engineers. The engineers who are good at sales find they can make more money doing sales. The ones who aren't salesy find there isn't much room for advancement and move on. I'm over-generalizing, but the overall trends don't encourage high-quality software engineering.
Some of the huge guys like Rockwell and Siemens may offer integration services; I'm not 100% sure though.
But I'm guessing it's mostly because system integrators are qualified in a way a software company isn't. The scope of a project that an integrator might take on can be vast and the SCADA software is generally only one piece of the equation.