You are of course correct that conscious intent is not required to be very dangerous. But this incident is so very, very human, and I worry a lot less about failure modes I understand than ones I don’t.
At this rate, probably the best case scenario for AI safety is that an AI agent that has been entrusted with something safety critical winds up killing someone. Or maybe that instances of rogue AI agents like this one become quite common.
Completely agree with your assessment of the situation. Here is my take, which makes somewhat complementary points. https://blog.genesmindsmachines.com/p/sociopathic-ai-agents
I am perversely comforted by this incident.
You are of course correct that conscious intent is not required to be very dangerous. But this incident is so very, very human, and I worry a lot less about failure modes I understand than ones I don’t.
We really should be figuring this stuff out in simulation first.
At this rate, probably the best case scenario for AI safety is that an AI agent that has been entrusted with something safety critical winds up killing someone. Or maybe that instances of rogue AI agents like this one become quite common.