Feedback is the Root of All Good

I heard the folktale of The Emperor's New Clothes, in various iterations, throughout elementary school. The story, which is attributed to Hans Christian Andersen, goes as follows:

"Long ago, there was a rich emperor obsessed with opulence and luxury. One day, a man decided to take advantage of his weakness and pretended to be a skilled weaver. When he met the emperor, he offered to sew him a one-of-a-kind robe that would only be visible to the intelligent. Fools would see nothing.

The man pretends to work on the garment for months, slaving away on intricate details, all the while showing the emperor and his ministers the progress. They all see nothing but pretend they see a magnificent robe to avoid being called a fool.

Months later, the conman declares the robe done and mimes dressing the emperor. A great parade is arranged to show off the emperor's new clothes. All along the way, the townspeople pretend to see the robe to avoid appearing stupid, all the while being deeply uncomfortable. Suddenly, a child yells out 'Why is the Emperor naked?' All of the people realize they have been tricked, and begin laughing."

The folktale is so irresistibly interpretable because of its simplicity, and scholars have ascribed many moral lessons to it. Horace Scudder praised the story in the Atlantic way back in 1875 for its endorsement of childlike truthfulness. Sigmund Freud himself saw the invisible clothes as an allegory for the way the mind attempts to disguise repressed desires. Jack Zipes saw the work as a Marxist critique of a gullible ruling class obsessed with sycophancy and consumption. More recently, Hollis Robbins compared the emperor's courtiers to men in the workplace who pretend not to see sexual harassment.

So yeah, the story is alluringly adaptable to any moral maxim. And I couldn't resist.

In the story, it is obvious that everyone has given the situation some thought, and arrived at the same confusion. The dissonance is palpable--- there is tension between the general improbability of a partially invisible garment and the idea that everyone around you has bought into the idea. The townsfolk who heard about this magical garment may have had the following train of thoughts:

1) That's impossible... there is no such thing as a magical garment.

2) Wait, I don't see anything!

3) But everyone around me is nodding... could it be real?

4) Oh god, everyone else sees it. They will think I am a fool if I admit to not seeing it.

5) Even if it is not real, I will say I see it.

On first glance, this passes muster. It’s obviously wrong, though. Firstly, magical garments don’t exist, and extraordinary claims require extraordinary proof. However, as many medieval peasants may have believed in magic, I cannot fault the townsfolk for their lack of skepticism.

A second error comes from assuming that the ministers’ claims accurately reflect reality, without considering the social pressures influencing their behavior. In ordinary situations, if a group of people tells Bob that it rained yesterday, he can reasonably conclude that it did. But in this story, each minister has a strong incentive to claim they see the garment, even if they do not, in order to avoid appearing foolish. As a result, the careful observer cannot treat the unanimous agreement of the ministers as reliable evidence in the same way they could for a verifiable event like rainfall.

Still, I can hardly fault the medieval villagers for their lack of meta-logic.

The child-- whether through rashness or cleverness-- stops after the second thought on the reasoning chain and cuts to the heart of the truth.1 The townspeople think for longer and are worse off than they would have been had they blurted out their initial thoughts.

(Now, there are some caveats here. In a medieval setting, the child indeed would have looked foolish if the garment had been real; in this case their observation would be incorrect. Perhaps the child got lucky. But when I heard the story, it seemed as though the child felt the tension and discomfort and wondered why no one was pointing it out. )

There's a cautionary lesson here: the process of refining something does not necessarily make it better, despite our frequent assumptions to the contrary.

My advisor recently introduced a new writing exercise during our weekly lab meeting. He writes a short prompt that formulates an assertion about artificial intelligence or the future of computation. Several randomly chosen students must take the opposite position as the prompt and rebut it in a short response of their own. During the meeting, their responses are displayed anonymously, and everyone who did not rebut that week rates each rebuttal. This is followed by a discussion of their strengths and weaknesses.

I have written several rebuttals, and the first few times I spent a decent chunk of time revising and redrafting what I wrote. These responses were both ranked poorly. The last two times I wrote, I forced myself to spend under five minutes on the response and edited them only for minor grammatical edits--- and both were rated quite well.

Sure, in my specific case there was a learning curve to adjusting to the preferences of the students in my group, and the process of redrafting can be noisy and stochastic, but it felt counterintuitive that my best responses were, like the child from The Emperor's New Clothes, my initial thoughts.

People assume the more time they spend refining a project, the better it will be. This broadly holds true-- think of carving a sculpture or painting on canvas. As you add the nose or paint the sky blue, the canvas becomes more representative of reality. This is a statistical truth, not a guarantee; it is possible to make a mistake and extend the sky too far down, or use the wrong shade, or to leave room for one too few teeth to carve. Still, in the early stages of a project, additional effort usually contributes to steady improvement.

I'm not cherry-picking examples, either. When you write an essay, adding more details makes it better. When coding during software engineering interviews, writing more complete solutions has generally worked to my advantage. (To be clear, I am not yet referring to practice—the process by which people improve their skills—but to the improvement of a specific piece of work. For example: completing a piano recital fits into this, but not rehearsing for it.)

However, this is only true up to a point. Once you add the sky, the clouds, the trees, and fill everything in, each decision of what to paint next has the potential to ruin the painting. Make the moon too big? Now it looks like an oversized disco ball. Those birds don't belong there, either. Did you try to add a funny joke to your essay's introduction? It didn't land, and now all your readers are confused.

I posit there are two regimes of refinement. In the first, you fill in details, and in the second, you attempt to improve them. You are almost guaranteed to make your work monotonically better in the first phase, because you're adding a necessary component. An apartment building looks weird without windows, so adding them--- no matter the quality of said windows--- improves your painting's appearance. Once you try to transform a blurry blob of gray into a gothic building, though, all bets are off.

Allen Ginsberg, a celebrated American poet of the postwar era, rejected the conception of a poem as a "patiently constructed artifact," opting instead for the governing principle of "first thought, best thought." In championing this principle, Ginsberg overturned a precedent set by the even better known poet, William Wordsworth, a century and a half before. Wordsworth had famously rejected the older notion of poetry as a spontaneous emotional overflow, famously redefining it as "emotion recollected in tranquility."

I am not qualified to settle this debate, but I am interested in defining the point between the certain-improvement regime and the risky-improvement regime. Intuitively, it has something to do with the concept of filling in-- filling in the text of an essay or filling in the regions of a painting— but I want to define it more rigorously. The two regimes are not the same for everyone; Bob Ross would likely be able to continue detailing a painting long after the canvas was filled with paint, but I could not say the same for myself. It's an abstract notion of filling in that is person-dependent.

Furthermore, in the case of formulating a rebuttal, it is not at all clear how to delineate the point at which you stop filling in and start stylizing. Is it just your initial thoughts, or when you have had one logical chain from beginning to end? Do you just think for a prescribed amount of time, and as you get better at thinking you can just think for longer? These don't seem like clean definitions.

When you think about something, you generally build an intuition very quickly. This is how humans evolved, and it may or not be justified, but you'll formulate an opinion quite quickly. This is one of those features of human psychology that people complain a lot about, but imagine how difficult it would be if we had to start from scratch every time we thought about something new. It would be exhausting.

On more complex issues, your first thoughts might not be opinions on the entire underlying question but only small parts of it. For instance, consider the following position: "Neuro-symbolic approaches are the best path to artificial general intelligence." When this was given as a writing prompt, I had to first think about my definition of neuro-symbolic--- an ill-defined term which means something different to everyone-- before even considering whether I agreed or disagreed with the assertion.

Okay, so you've formed an opinion. Now begins the process of refining and rethinking your assumptions and line of reasoning. As you do this, two processes are occurring: you are changing and your argument is changing. You become informed through research and introspection, and you use that knowledge and familiarity to improve your argument.

Unfortunately, this is not good. Chances are, your audience is not made up of people who have researched the issue as thoroughly as you have. Thus, at the beginning, you probably shared your audience’s assumptions about the world. However, as you learned more, you gave credence to certain arguments more than others. Perhaps one of your initial assumptions was inaccurate. Alternatively, you were on shaky ground about Conjecture A, and then read a series of complex proofs, and suddenly Conjecture A is obvious beyond belief. Now you don't need to explain it nearly as clearly.

It doesn't matter that your argument is more refined because that new argument is answering a different question now, and it is not the one being asked by your audience. Your "first thought" is not necessarily your "best thought" but it is certainly your most common thought. That’s often worth more.

On April 15th, 2013, the brothers Tamerlan and Dzhokhar Tsarnaev detonated two homemade bombs at the end of the Boston Marathon race, killing three and injuring hundreds more. The first images of the brothers were released by the FBI on April 18th, and they killed an MIT policeman in a carjacking a day later in which Tamerlan was killed. Dzhokhar was taken into custody a day later.

Meanwhile, Reddit went crazy. On April 16th, the “Find Boston Bombers” subreddit was created by would-be civilian investigators. These redditors— who had no connection to law enforcement— quickly accused a missing student named Sunil Tripathi of perpetrating the bombings. When the FBI released photos of the suspects, some redditors saw resemblance between him and the released photos and used this as further evidence of his guilt. A woman claiming to be his classmate claimed she thought Tripathi looked like the man in the FBI photo.

People quickly started contacting the Tripathi family to demand answers through social media and phone calls. After the carjacking, a post on Twitter went viral claiming the two suspects were Mike Mulugeta and Sunil Tripathi.

The Tzarnaev brothers were soon apprehended. Days later, on April 23rd, a body was found in the Seekonk River. It was Sunil Tripathi.

Redditors issued an apology over the witchhunt, but the damage had been done. Sunil had been missing for a month—his family had endured weeks of uncertainty and grief, only to face the added torment of watching the internet accuse their son of being a cold-blooded killer.

Now, the Reddit Bureau of Investigation, as they like to call themselves, has managed to successfully solve cases. They identified "Grateful Doe," a young man who died in a car crash with Grateful Dead tickets in his pocket, as Jason Callahan. A different redditor helped identify Walter Scott's shooter in 2015 after stabilizing bystander footage.

During the Boston Marathon bombing, though, Reddit’s investigation failed spectacularly. In a stunning display of groupthink, each person took everyone else's confidence as proof for the veracity of the accusation against Sunil. Once it was understood to be a true statement that just needed more substantiation, amateur investigators were only too happy to believe a tweet that said Sunil was the man in the photographs without any verification.

The investigators were well-meaning, but they had taken crumbs and reconstructed an alternate reality that had no connection to our own. Much of their analysis was based on shaky evidence, but everyone seemed to accept it. They couldn't possibly be wrong, could they?

Confirmation bias is a bitch.

Finding Sunil and marking him for further investigation made sense, as additional information can give context and provide additional leads. The usage of circular reasoning to mentally indict him is what destroyed the investigation's legitimacy. Once Sunil became the target of an investigation, the rhetoric changed completely and it became a game of finding the evidence needed to indict him, not the evidence needed to find the real killer.2

Returning to the painting metaphor, finding Sunil is the first regime and is akin to painting in the sky. Assuming his guilt is equivalent to enlarging the moon to obnoxious proportions.

The commonality between all these situations and the part of the process that corresponds with filling in the painting lies in a grounded representation of the real world. Before a person does any action or process, they have a mental model of that process--- or at least the next few steps in the process--- internally. In the case of a painting, this is a mental image of the scene. For an investigation, it might be a causal model obtained through accumulation of evidence, complete with the likelihood of different possibilities having occurred.

We can thus reduce this process we have referred to as refinement into two phases: modeling and translation. Modeling is creating a workable, accurate representation of some external phenomenon— such as the real world, another person, or a mathematical truth— internally such that it is useful for a specific purpose. Translation is reconstructing that mental model and communicating externally. To give another example, writing involves a modelling phase, where ideas are synthesized into streamlined and coherent narrative threads, and a translation phase, where the author puts pen to paper and weaves a coordinated whole. During modeling, the actor understands; during translation, the audience understands.

These are distinct skills, though they are linked and may be improved in tandem. Sculptors and painters practice memorizing a scene and then capturing it in full fidelity without looking at it. As most artists don't possess a photographic memory, this capability has less to do with memorizing every detail than it does capturing the attributes that are most relevant to recreating the scene. This is not a general skill— as an artist gets better, perhaps they can use their background knowledge to recreate a nose or ear instead of memorizing each curve. A novice artist might need to model every ear he or she sees.

Translation tends to get all the credit, and for good reasons. There are scenes I can remember very well that I will never be able to paint because I don't know how to get them onto a canvas. I cannot paint.

Modeling is a subtler skill of equal import. A skilled painter would never overscale the moon because, through years of painting, he or she has an intuitive grasp of what the moon looks like when composed of brush strokes.

During the first phase of improvement, the improver possesses an internal model and translates it into a tangible or communicable object. Inevitably, though, the practitioner reaches the limits of their mental model and has to expand it. This is person dependent— a novice sculptor might only think about the eyes, nose, and mouth in broad strokes, but when he or she has carved out places for these facial features he or she will be at a loss for where to continue. A seasoned artisan might be thinking about every capillary in each eye and how the muscles interact, and only reaches the limits of their model for very unusual faces. At this point, they too must update their model.

Put simply, modeling is about understanding internally, while translation is about expressing that understanding.

Of course, this is a simplified view of improvement. Most skills are not atomic, and each sub-skill might possess a translation and modeling component. This complicates the translation-modeling dichotomy because some modeling skills may be hindered by translation skills required to build the mental model. (For example, imagine a musician who excels at understanding melodies and is an adept violinist, but cannot sight read well. They might fail to build a mental model because they cannot translate from sheet music to an internal melody, not because they can’t understand music. Perhaps if they heard the piece played, they could easily mimic it.)

Furthermore, some physical skills– if they possess an analogous modeling component at all– are modelled for us in an inaccessible part of the motor cortex, far away from conscious thought. Walking is second nature to most people, but it is a complex multi-step action that is coordinated by the cerebellum and motor cortex.

Even if we accept that there is a domain of skills where this separation is valid, is it useful? Knowledge of improvement intrinsically seems useless, especially where translation is the bottleneck. A bad writer will find it very difficult to communicate their top-notch ideas, and improving the idea does very little to help them.

I would argue that the distinction does matter, and it matters most in the skills that a person is most invested in.

Earlier I emphasized that this is not merely a matter of practice. So far, I have not addressed whether progress in “translation” or in “modeling” plays a greater role in developing underlying skill. The answer is almost certainly area-dependent. Yet for individuals already operating at a high level, improvements in modeling are generally the most critical.

Phillip E. Tetlock showed in his book Superforecasting: The Art and Science of Prediction that experts from disparate areas perform poorly when asked to make predictions in their respective domains, such as about geopolitical events or technological developments. Over decades of data collection, his study revealed that these experts’ forecasts were barely more accurate than random chance—roughly equivalent to flipping a coin. The results of the study also showed that experts were more accurate in making forecasts outside their areas of expertise, and well-known experts underperformed their more obscure peers.

Tetlock dubbed the experts who predicted the future successfully “superforecasters” and identified characteristics they all shared. One of them, unsurprisingly, was practice. However, not all practice is created equal. Remember, well-known experts with longer careers no doubt have more practice than early stage professionals. The key component of good practice was feedback:

“Effective practice also needs to be accompanied by clear and timely feedback. My research collaborator Don Moore points out that police officers spend a lot of time figuring out who is telling the truth and who is lying, but research has found they aren’t nearly as good at it as they think they are and they tend not to get better with experience. That’s because experience isn’t enough. It must be accompanied by clear feedback.” (Gardner and Tetlock, Superforecasting: The Art and Science of Prediction)

This is analogous to being a good modeler. We say that a model of the Titanic is a good model if it resembles the actual Titanic, and so too this is the case with the models we have inside our heads. The designer improves a model by comparing it to its referent. That’s what feedback is! Police officers rarely get to the bottom of who is lying, even if the person is convicted or freed. They are not able to verify if the signals they use to distinguish truth and falsehood are accurate or not.

Translation often comes with immediate feedback. Jerry Seinfeld says comedians are “graded” every seven seconds—if the delivery fails, the audience doesn’t laugh. What’s harder to discern is whether the joke itself is weak, or whether it simply needs a different setup or delivery to succeed. Improving your mental model involves aggregating data from multiple circumstances and coming to the conclusion it’s not a translation issue.

Recognizing that modeling is part of every skill is valuable because it highlights where and how to seek feedback. Determining whether a model is truly coherent can be extremely difficult—after all, most pundits perform worse at forecasting political events than ordinary observers. Like the emperor’s subjects, they can reinforce one another’s confidence rather than testing their assumptions. Perhaps their accuracy would improve if they were forced to revisit their predictions a year after making them.

Becoming aware of your own mental model pays off both immediately and over time. In the short term, it allows you to make your assumptions explicit—whether about the size of the moon or the future of AI—and test them against reality. In the long term, it enables you to refine how you represent the world, by comparing your expectations with outcomes and learning when your reasoning holds up and when it does not.

Improvement is still really, really hard. Plato, with Socrates as his mouthpiece, considered virtue to be knowledge. In his view, wrongdoing stems from ignorance, and he thought that no one willingly does evil--- had they known it was an evil act, they would have self-corrected. History has disproved this myth pretty thoroughly.

Nevertheless, if knowledge itself is not sufficient, it is at least a reasonable first step.