The article addresses this very point, using the example of climate change:
> it will be VERY plausible for an AI to simply conclude that it should proceed with the most expedient way to delete ~95% of humans.
The elimination of ~95% of humans is a realistic way of "solving" climate change, without bending reality. It just so happens that this is not a morally acceptable solution to most of us. In other words, it is not an issue of people bending reality, it is an issue of the reality being far more complicated than humans programmers can fathom, and consequently, that humans programming an AI to address real issues will be inherently limited by their own imagination of what the solutions to those issue most might be.
In other words, woke programmers may be bending reality due to their biases and limitations, leading to unacceptable outcomes from their AI, but don't be so arrogant as to think that your own biases and limitations aren't going to yield equally unacceptable output.
The example of climate change is sexy but I don't buy it as it is too overly simplified.
In my opinion, without woke and bending, the AI would probably reply "I don't know" like would a serious person.
Because, first you would have to define climate change and what is the problem, and what is the outcome that you would want.
Earth in itself does not care about what is going on. Similarly, there is no reason why a change of climate would be bad, it happened multiple time over the life of earth.
So, the request would probably be something like this: the current (or past century) climate was good for human life, so how could we do to restore or preserve that so that it stays favorable for us. As a side goal, we would like to preserve the existing variety of plants, trees and animals. As another side goal, we would like to preserve some landscape that are looking very nice. In this order.
> it will be VERY plausible for an AI to simply conclude that it should proceed with the most expedient way to delete ~95% of humans.
The elimination of ~95% of humans is a realistic way of "solving" climate change, without bending reality. It just so happens that this is not a morally acceptable solution to most of us. In other words, it is not an issue of people bending reality, it is an issue of the reality being far more complicated than humans programmers can fathom, and consequently, that humans programming an AI to address real issues will be inherently limited by their own imagination of what the solutions to those issue most might be.
In other words, woke programmers may be bending reality due to their biases and limitations, leading to unacceptable outcomes from their AI, but don't be so arrogant as to think that your own biases and limitations aren't going to yield equally unacceptable output.