The classic thought experiment is the paper clip optimizer. Quoting Nick Bostrom:
Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans. https://www.huffpost.com/entry/artificial-intelligence-oxfor...
Comments
The classic thought experiment is the paper clip optimizer. Quoting Nick Bostrom:
If you substitute "shareholder value", this is basically how corporations already make decisions with effects on human life.
Sure, I once write that companies are artificial intelligence and it still feels true: https://beza1e1.tuxen.de/companies_are_ai.html
Cstross certainly did say that in 2017.
https://www.antipope.org/charlie/blog-static/2018/01/dude-yo...
The fossil fuel industry managed to invent this unhappy process long before machine learning was a thing.