"If you can’t be sure whether you’re creating a self-aware #AI, this is alarming not just because of the moral implications of the “self-aware” part, but because being unsure means you have no idea what you are doing and that is dangerous and you should stop."
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/