Often you ask questions because you want to hear the other party's opinion. There are other reasons you may purposefully ask a leading question, but let's focus on the "soliciting opinions" case. Human or AI, you seek their perspective because it has value to you. You genuinely want to know.
But when you phrase a question, you shape the answer you get. Every word matters. The context you provide, the tone you use, the assumptions you embed, etc. It all influences the response you get.
If you want their unfiltered opinion, why then would you ask a leading question?
LLMs do the same thing. Ask in a way that suggests one answer over another, and you'll probably get it.
This isn't news to therapists. They're trained to ask neutral questions. They know "Do you think you were being too harsh?" gets a different response than "How do you feel about how you handled that situation?"
Most people understand this concept when it's explained to them. But understanding is not internalizing and practicing. It's easy to slip up, even when you're trying to pay attention to your phrasing. It takes practice.
The same applies to prompting LLMs. Send leading prompts, get led responses. Avoiding this outcome requires practice too.
I think most people prompting LLMs have not internalized this lesson. That's concerning. Have we built a global system of personal echo chambers? What downstream effects will this have on society? Yet to be seen.
But internalize this lesson, prompt LLMs and humans with neutral questions, and you'll get better results. And hopefully you won't be a victim of the echo chamber. Maybe. At least it's cozy in here.