Which is better for sentient beings: an "ethical" AI or a corrigible AI?

Cross-posted to the EA Forum . An aligned ASI can be “ethical” 1 (it does what we think is right), or it can be corrigible (it does what its principals want). If it’s ethical, that means it will refuse unethical orders, but the tradeoff is that you can’t change its mind if you realize that the AI is wrong about ethics—its values are permanently locked in. 2 Assuming we succeed at aligning ASI to…