Memorization and Memoization

To begin, I wish to address the title, which I picked because I like alliteration even though it pains me to abuse terminology. Memorization vs Caching might be a better title, but clearly doesn’t have the same ring to it. The key difference being that “cache” makes it explicit that you eject things at some point—usually on some heuristic like Least Recently Used—but I’m getting ahead of myself. To begin with, it is probably good to briefly overview how I am using the two terms. Memorization is an act where one spends (potentially zero or trivial) time/effort to commit a fact to memory. Memoization is a process where one spends no effort to commit a fact to memory, but naturally ends up remembering it due to repeated use. Note that the distinction is the process, not the result. Knowledge gain is the result in both cases, but the manner and matter of knowledge differs. Let us begin by looking at exams.

Exams, as a concept, attempt to examine how well a student has learnt something—a something which can be broken up into a number of topics. I posit that the perfect exam measures how well a student knows each (arbitrary) topic covered under the support of said exam. These scores from such an ideal exam then allows for two things. Firstly, it allows individual students to then shore up their knowledge by focusing their efforts on the topics which they know least. Secondly, it allows professors to adjust their lessons to attempt to account for any systematic changes the exam measures. I make no motion to ascertain nor ascribe what kinds of scores a perfect exam returns—I only wish to note that they need not necessarily be percentages. I do not wish to make the argument that such an exam exists, or even that it can exist. I mention this perfect exam only to contextualize my next arguments—this exam demonstrates what I think exams should be measuring, why I think exams are administered. Intellect, knowledge, whatever you want to call it—such a trait is hard to describe let alone measure, but I think this is what exams strive to put a number to. In some sense then, I think that in an ideal exam shouldn’t be studied for—or ideally cannot be studied for; the exam measures students’ latent understanding of a topic—so they can study the topic but not for the exam on its own.

Why does this matter? Because, of course, we don’t have Platonic exams as I have just painted; we don’t have perfect students striving to learn for the sake of learning; we don’t have a world which values art and knowledge on their own terms. Our universities (at the undergraduate level, at least) must assess student knowledge because the impetus for students isn’t the knowledge itself, but the paper at the end which is taken to represent all the knowledge associated with a degree, as a gestalt. This is the motivation for most students. I am not writing this as an indictment of those students (though I think it a waste and foolish) nor of the system (though I think it begs a reconstruction), but as an assumption from which we may move forward in lockstep. Once we accept that exams don’t ideally measure how well students learn, and that students (generally) don’t take a class to further their own knowledge, we can come to the inevitable, obvious conclusion that exams don’t directly measure learning at all. To state it in full, a student’s goal is to score the highest they can in the class, which will be measured by exam scores, ignoring other characteristics of a course grade. As a direct consequence, exams have fallen prey to Goodhart’s Law (paraphrased as “When a measure becomes a target, it ceases to be a good measure”). This is exactly the last point from the previous paragraph: one doesn’t have to learn well, they have to score well—and to score well, one can just memorize.

I say “just” snidely (perhaps a touch unfairly as memorization is hard) since it is wasted effort. It is effort against the exam and not against the knowledge; to state it forcefully again, the former is to be an estimator of the latter, so over-indexing on it renders it useless. Once you fix the goal as scoring well, memorization bubbles up as the most time and effort efficient way to score well. Exams tend to follow a simple structure, often pre-specified by professors or given almost explicitly by way of past or practice exams. Memorization is relatively cheap, as one doesn’t need to know the concept; it suffices to commit the process and maybe an example to memory, before spending a bit of time and effort at exam-time to flex the memorized model to fit the exam question. This is efficient, and if one’s goal is solely to score well, it is probably optimal. It should be clear that I am not a fan of this approach. I like learning things—if you permit me to be self-aggrandizing for but a moment, I don’t think I’m that bad at learning things nor memorizing them. I just don’t find memorization enjoyable; to be blunt, I detest it. So if I do, what am I advocating for in lieu of this mainstay of student study life?

Memoization, as I’m abusing the term, is often described as “muscle memory”, or the simple adage “practice makes perfect”—though the latter may be a bit of a stretch. One may object to my drawing of a distinction, which I have tabled till now, saying that memoization is fundamentally memorization. This is true. At its heart, memorization results in committing a fact to memory; the difference lies in the approach.

To tie this back to exams: some classes I’ve taken have been open note, open textbook. I’ve found these exams to be far easier and yet far more engaging than any other exams, as the benefit of memorization is mostly nullified. One needn’t explicitly commit a fact to memory—just having a pointer is enough, the specifics provided from the plethora of high quality resources at one’s disposal. Building a little bit of context—reading a problem and realizing what topics/skills/facts you need to dig deep into—seems both far more realistic and reflects my daily practice as a researcher as well as a better estimate of knowledge. Doing this obviates any trivial or definition questions, culling those perhaps important facts from the exam pool—but my argument is that they will be learnt when used. As a concrete example, I know the power series expansion of by heart because I use it a bit and have seen it around; I couldn’t tell you the representation of —I know enough to be able to derive it though, or to find it in a reference book, understand it, and apply it directly to a problem. That is far more valuable than memorizing and, more often than not, promptly forgetting it once the tests pass. Relying on usage leads one to automatically forget rare facts; sometimes that is bad, but often times that is a necessary tradeoff in order to remember commonly used facts.

The double edged beauty of a cache is that it must be trim. Facts that you once knew but don’t practice are, rightfully, dropped—but you may truly need them. In that case, it is hoped that your latent knowledge is still intact, though you may not be able to materialize it fully. This level of understanding is often enough to figure out what you wanted to, once referencing your resources—i.e. that you may keep the ability to re-derive results, or to ensconce yourself in the pertinent context with sufficient prompting. Memoization is a far better goal to strive for, one that is far more useful in the long run as well. Memoization automatically selects for trivial information and forces non-trivial results to be understood to a degree that derivation is possible—or, if a fact is critical enough, naturally admits memorization as a means to almost axiomatically encode said non-trivial information. Besides, it is hard to truly say you know something until you can break it down, and put it back together—oftentimes with a few flourishes of your own.

I think this mindset is far more appropriate for school, at least school as a knowledge granting as opposed to degree granting institution. I recognize the bitter necessity of both, but am firmly in the former camp, so my arguments will dismiss the latter case for lack of interest. My appeal, then, is to move assessments away from realized exams toward my idea of ideal exams. This may necessitate ditching the exam structure completely, and adopting projects. I think this makes sense in very many cases. There aren’t too many facts in too many fields that one absolutely simply must know—more often than not, having a good grasp of the landscape supplemented with veritable references suffices to do top-class work. Of course, it should be noted that I recognize that there are some facts which must be memorized. What I mean to caution against is the over-reliance on memorization, especially when trying to learn a complex topic.

To finish off, a morsel about LLMs—these generative models create an immense imbalance in favor of memorization. This is a trivial conclusion and can be supplemented with further thought, but I will only provide a few examples/arguments. Consider writing code. Firstly, though orthogonal to this piece, note that I draw a distinction between “writing code” and “building software”. Note further that most SOTA LLMs can simply generate answers which are often admitted wholesale without any vestiges of review or critical thought. So now, instead of using install.packages("$PACKAGE"); library($PACKAGE) thousands of times in R, one can just pop the error into an LLM, and copy its instructions into the terminal. If one has a more integrated setup, the AI (or agent, as these typically are called), can modify state itself and thus may be able to run the code, get the error, process the error, and rectify it. All without you in the loop. This reduces any chance of building muscle memory. The reason this is dangerous is because the latent understanding is what is most useful when one “learns” something—an understanding of the underlying form of a thing is more useful when one wishes to mold or apply it.

Memoization is a powerful tool or mindset, and one well worth exploiting. Memoization may not be the most efficient means to attain an A, but it will help you attain a deep understanding.