A Deranged Mathematician Does Quantum Mechanics

For those of you who don’t know, the name “The Deranged Mathematician” came from an old Quora post in which I laid out a mathematically rigorous treatment of basic quantum mechanics (i.e., at the level of the Schrödinger equation, before any of the relativistic corrections). This was something that started as my undergraduate thesis—my advisor suggested that I write an understandable introduction to quantum mechanics for the mathematician trying to learn the subject.

I have made various adjustments/improvements to this since then—one of the biggest came from my realization that defining observables in terms of self-adjoint operators is in some sense entirely backwards. (You’ll see what I mean.) It seems to me that now that I have laid out some of the fundamentals of measure theory and functional analysis, it is time to return to this once again.

(If you haven’t yet read those last three posts, I recommend doing so!)

There is one major thing that I feel that I have not adequately addressed in the previous times that I have done an introduction to quantum like this: why does quantum mechanics use complex numbers? I hope that this time, it will actually be believable.

Without further ado, let’s get into the meat of things.

Our goal is to find a mathematically rigorous description of quantum mechanics. This means everything is assumed to be non-relativistic, and we do not treat waves at all—we are only looking to describe particles that are moving relatively slowly, and we are not describing particle interactions. This is definitely a theory of mechanics—that is, we describe how things move when acted upon by forces.

What I hope to convince you of is that while the various mathematical components of quantum mechanics—Hilbert spaces, self-adjoint operators, the Schrödinger equation—might look odd at first glance, they actually fall out as the neatest possible description of what we know to be true experimentally. In a certain sense, it was the only thing that anyone could have reasonably come up with.

At the same time, I should note that the description that I am giving is certainly not the way that quantum mechanics was first developed, nor is it how it is taught today. There are many reasons for this, including the fact that the approach I will be giving depends rather crucially on knowing some slightly advanced mathematics—measure theory and functional analysis, to be precise. That said, it is my opinion that this is the most natural description that one can give.

I should also explain what I mean by a mathematical model of quantum mechanics. This will encompass a description of three things:

1. The state space—the space of all possible configurations of the system. For a single particle in classical mechanics, that might be the collection of all the possible positions of that particle, as well as all of the possible velocities of the particle. In quantum mechanics, we shall see that it is something quite different.

2. Observables—all of the various measurements about the state that we can make. This might be momentum, position, energy, or something more exotic like spin. Unlike the classical case, we shall also have to describe the effect that taking an observation has on the system.

3. Time-evolution—absent any measurements, how does the system change over time? We know that this is described by the Schrödinger equation, but I will give a derivation of sorts to show why the Schrödinger equation is the right thing.

But first, let’s discuss how quantum mechanics differs from classical mechanics.

Quantum mechanics has two closely intertwined properties that separate it from Newtonian mechanics. Experimentally, we observe the following phenomena.

1. Wave-function collapse: when we take a measurement of the system, the state of the system changes in correspondence to whatever it is we measured. Unlike Newtonian mechanics, we might model a particle in quantum mechanics as having energy A or energy B or energy C—when we make a measurement of the energy, the particle will suddenly become one of these.

2. Superposition: states can be some mixture of other states. In the above example, we had a particle that was in a superposition of having energies A, B, and C, where we might have measured it as being in any one of these states when we actually did the measurement. Of course, once we did the measurement, then the state would no longer be in this same superposition—it would collapse to something consistent with whatever measurement we obtained.

Before we launch into any mathematics, let’s look at a physical example so that we understand these two properties. Specifically, we’ll consider the Stern-Gerlach experiment.

Classical electromagnetism tells you that a spinning, electrically charged object will act like a magnet, and so it will get deflected in a magnetic field. This is true even if this object is part of a larger, electrically neutral system. (So you might have one, negatively charged, spinning piece counterbalanced by a second, positively charged piece.) So here is the experiment that Stern cooked up, and then Gerlach carried out.

Take a beam of silver atoms, pass it through an inhomogeneous magnetic field, and then measure where these atoms collide on a metallic plate. (See the figure above—the initial beam is labeled 2, and the magnetic field is labeled 3.) Classically, you might expect that either there is no deflection at all (if silver atoms and their constituents don’t spin), which would lead to a single point of collision, or that there would be a continuous line of collisions due to the silver atoms spinning in random directions. (This is the classical prediction, labeled 4.)

Instead, Gerlach discovered that the beam split into two! (The actual measurement is labeled 5.) How to explain this?

In principle, you could try to modestly extend classical theory as follows: we might posit that there are actually two kinds of silver atoms which behave in different ways in the presence of a magnetic field, and this is why it splits into two. (There are many issues with this attempted explanation, but let’s run with it for now.) This explanation is seemingly reinforced if we put in a second magnetic field, oriented in the same way as the first: the beam no longer splits—all particles get deflected in the same direction.

But let’s extend this even a little more, and add a third magnetic field in between these two, but rotated by 90 degrees.

Something very strange happens! First, the split beam will split again when it goes through the rotated magnetic field. This can still be explained by supposing that there are actually four types of different silver atoms, and so we have sorted in one direction, but not the other after the first pass. What cannot be explained this way is that once we pass one of the output beams through the third magnetic field, it once again splits in half!

With the crumbling of the classical picture, we are forced to turn to quantum mechanics. The new explanation is actually vaguely similar to our suggestion that silver atoms come in two distinct types: we suppose that there are two possible states, called spin up and spin down. The initial stream contains particles that are in a superposition of those two states, and then when we pass through the first magnetic field, we get wave-function collapse so that all particles are either spin up or spin down. Thus, if we measure again, we’ll get the deflection as the first time. But if we turn the magnetic field 90 degrees, then we are now measuring something different than we were initially—spin up can itself be conceived of as a superposition of two different states (which we might call spin left and spin right here), and passing through the rotated magnetic field causes them to collapse into one of those two. But now, since they are no longer purely spin up or spin down, wave-function collapse again produces two beams when we pass through the last field.

If this feels a little fuzzy, don’t worry: this was very difficult to understand even for some of the greatest minds of the 20th century. You are in good company. And, if you are anything like me, it will start to make a lot more sense once we actually start writing down some math.

So, let’s begin.

If we want to model this bizarre phenomenon, what should our state space look like?

Well, to start, since we have to model superposition, the state space should probably be a vector space. This is so that given two states ψ₁, ψ₂, we can talk about a linear combination of them, c₁ψ₁+c₂ψ₂, which we think about as a superposition of the two states. For instance, in the Stern-Gerlach experiment, we might define four states:

1. |z₊〉 - spin up

2. |z_-〉 - spin down

3. |x₊〉 - spin right

4. |x_-〉 - spin left.

However, this raises a question: what should the scalars be?

The naive answer is that c₁, c₂ should be real numbers—indeed, one might guess that they should be non-negative real numbers, positing that c_i might be the probability that we measure something consistent with ψ_i when we do an experiment.

But that doesn’t fit with the Stern-Gerlach experiment. From our discussion, it must be that |z₊〉=c₁|x₊〉+c₂|x_-〉; indeed, since the beam splits neatly in two after going through the second magnetic field, it would have to be that |z₊〉=(|x₊〉+|x_-〉)/2. But then, what is |z_-〉? We could have taken that beam through the second magnetic field, and it would also split neatly in two. Does that mean that |z_-〉=(|x₊〉+|x_-〉)/2? But then |z₊〉=|z_-〉, which is nonsense!

So, as a minimum, we need to include negative numbers: this allows us to write |z₊〉=c|x₊〉+c|x_-〉 and |z_-〉=c|x₊〉-c|x_-〉 for some appropriate constant c, which yields |x₊〉=(|z₊〉+|z_-〉)/(2c) and |x_-〉=(|z₊〉-|z_-〉)/(2c). This seems to fit with observation.

Except… we are forgetting about something very important. In 3D space, we have three orthogonal directions in which to measure: we are forgetting about y! If we, for example, rotate the apparatus that we used in the Stern-Gerlach experiment, we will get the same results as before, but where we were previously measuring deflection in z and x, we might now be measuring deflection in z and y. All of this is to say that there are actually six states that we should consider: we forgot about |y₊〉 and |y_-〉.

The same thing should be true as before: |y₊〉 and |y_-〉 are superpositions of |z₊〉 and |z_-〉 (or |x₊〉 and |x_-〉), in equal magnitude. But this now presents a problem—after all, we have already determined that |x_±〉 should be ±c|z₊〉±c|z_-〉. From what we have just said, |y_±〉=±c|z₊〉±c|z_-〉 as well. But |y_±〉 should also be linearly independent from |x_±〉 (because it is equally true that |y_±〉 should be ±c|x₊〉±c|x_-〉) and that isn’t possible!

Real numbers just plain don’t give us enough degrees of freedom to resolve this issue. But there is a solution: we can make the scalars be complex numbers. Then, for example, we could write

and all of the relations that we suggested will hold just fine. It’s not entirely clear at this juncture what information the coefficients are capturing, but we will return to this point later: for now, we content ourselves by saying that the state space should be a complex vector space.

However, just saying that the state space should be a complex vector space isn’t enough—there is actually more structure that we are ignoring. Specifically, we now introduce a notion of orthogonality. We shall write ⟨ψ₁,ψ₂⟩=0 if and only if there is some collection of observables A₁, A₂,…, A_n such that if I measure A₁, A₂,… A_n for both ψ₁ and ψ₂ with sufficient precision, I will be able to distinguish them—we shall call ψ₁ and ψ₂ incompatible. Loosely speaking, I can’t possibly get the same measurements of all of A₁, A₂,… A_n for both ψ₁ and ψ₂.

In the Stern-Gerlach experiment, |z₊〉 and |z_-〉 are incompatible: if I measure deflection in the z direction, I will get different results. In contrast, |z₊〉 and |x₊〉 are not: it is entirely possible that I will get the same deflection for both, depending on which way we collapse.

If I have a notion of orthogonality and I choose a basis for the complex vector space, then if I also select lengths for each of the basis elements, this determines a unique (positive definite) inner product on the vector space. (If you don’t remember what an inner product is, look at my post on Hilbert spaces.) This is advantageous—inner product spaces are very structured, and there are a great many tools available for studying them. However, now we have to figure out how to decide on what lengths we should assign to whatever basis we select—what is the physical interpretation of that?

Thankfully, we are saved by the fact that I have been lying slightly to you, and it is time to address the elephant in the room: what is the meaning of ψ+ψ? We noted previously that we were thinking about ψ₁+ψ₂ as being a ‘mixture’ of two states; however, surely if we put a state in superposition with itself it shouldn’t change? That doesn’t fit with our intuitive notion of what superposition is. And indeed, this is correct—we consider ψ and 2ψ as corresponding to the same physical state. Indeed, more generally, for any non-zero complex number c, we consider ψ and cψ as corresponding to the same physical state.

What this means is that we can rescale all of the basis vectors however we like without changing the underlying physical states, and so it actually doesn’t matter at all what lengths we choose for the basis vectors—without loss of generality, we can just choose all of them to have length 1!1

This suggests an interpretation for the coefficients in the linear combination c₁|z₊〉+c₂|z_-〉, because

where we used the fact that |z₊〉 and |z_-〉 are incompatible (hence their inner product is 0) and that the inner product of a state with itself should be 1. Since |c₁|²+|c₂|²=1, we might posit that |c₁|² is the probability of collapsing into |z₊〉, and |c₂|² is the probability of collapsing into |z_-〉—these two probabilities add up to 1, as expected. More generally, if we have an orthonormal basis ψ₁, ψ₂,… for our space and ψ=c₁ψ₁+c₂ψ₂+… has length 1, then we will interpret |c_i|² as the probability that when we measure our observables for ψ, we will get a result consistent with ψ_i.

In any case, we have turned our state space into an inner product space. (Technically, it is a projective inner product space, since scaling does not change the underlying state, but we shall follow the usual physics convention of ignoring this detail, since we can always rescale as necessary after we have done relevant calculations.) Any inner product ⟨⋅,⋅⟩ on a complex vector space V induces a metric on V defined by

However, since we have turned V into a metric space, this means we can talk about convergence. In particular, we might ask whether the space V is complete or not.

Working with complete metric spaces is much, much easier than working with non-complete metric spaces. Furthermore, any non-complete metric space can be realized as a subspace of a complete metric space. On the other hand, many state spaces that physicists work with are not actually complete.2

We therefore make a compromise: we assume that the state space is indeed complete, but for any sort of practical applications we restrict our calculations to actually physically realizable sets. This isn’t as terrible as it perhaps sounds—for any non-physically realizable set, there will necessarily be a physically realizable set that is an arbitrarily good approximation. Taking into account that experiments can only determine things up to tolerance anyway, this actually makes no practical difference whatsoever.

We have shown that our state space is a complete (projective) complex vector space with a positive definite inner product. This is called a (projective) Hilbert space, and so correspondingly we shall henceforth denote the state space by H.

To recap: the state space H is a (projective) Hilbert space, where the inner product ⟨⋅,⋅⟩ is characterized by the fact that ⟨ψ₁,ψ₂⟩=0 if and only if ψ₁, ψ₂ are incompatible. Furthermore, if ψ₁, ψ₂,… are incompatible, and ψ₁, ψ₂,… and ψ=c₁ψ₁+c₂ψ₂+… all have norm 1, then |c_i|² is the probability that when we measure our observables for ψ, we will get a result consistent with ψ_i.

We move on to talking about observables. In the classical realm, we would model these as real-valued functions on our state space. This doesn’t work in the quantum world, since we might get a different measurement every time we measure some observable A for a fixed state ψ. Furthermore, as we discussed in the beginning, measurement of an observable changes the state ψ.

How exactly does it change? Well, if we measure A to be in some set S, then the components of ψ that are inconsistent with that measurement vanish to zero, leaving just the components that are consistent with that measurement.

As an example, I might have a state that is in a superposition—it has a 50% chance of being measured to be in the interval [−1,0], and a 50% chance of being measured to be in the interval [1,2] (which we might think of as being two bins slightly separated from one another). We do the measurement, and find that the position is in the range [1,2]. The components of the state vector corresponding to it being in [−1,0] instantaneously vanish, and we are left only with the components consistent with it being in [1,2].

How do we model this mathematically? Fix some observable A. For every (measurable) subset S of R, we have two important subspaces: H_S, the subspace consisting of all elements that are consistent with measuring A in S, and H⊥H_S, the subspace consisting of everything orthogonal to H_S—that is, it consists of all states which have 0% probability of having A measured in S.

When we do wavefunction collapse relative to S, what we want is to project onto H_S—that is, we want to keep any components in H_S untouched, but send all components in H⊥H_S to 0. In other words, we’re killing all components of the state that are incompatible with measuring A in S.

What we are describing is a projection operator—a linear transformation H→H characterized by the fact acts like the identity operator on H_S and like the zero operator on H⊥H_S. We call this projection operator P^A(S).

A little bit of thought shows that this identification S↦P^A(S) should satisfy some basic properties:

1. P^A(∅)=0, P^A(R)=I—the first is because no state is consistent with not getting any measurement, and the second is because every state is consistent with measuring A to be in R.

2. Let S₁, S₂, S₃,… be disjoint subsets of R. Then measuring A to be in ⋃_iS_i should be the same as measuring A to be in S₁ or S₂ or S₃…, which should correspond to projecting onto the sum of the subspaces corresponding to S₁, S₂, S₃,… Or, more simply,

   \(P^A\left(\bigcup_i S_i\right)=\sum_i P^A(S_i).\)

Those who have studied measure theory will notice that this definition is very, very similar to the definition of a measure—the key difference is that rather than returning a positive real number, P^A returns (orthogonal) projections on H. Correspondingly, we call P^A a projection-valued measure.

The beautiful thing about this is that—with some small adjustments—you can use this to build a theory of integration exactly like how we use classical measures to define integrals. (For those who need a refresher, see my article on measure theory that I linked to above.) That is, we can define an integral

and it behaves in much the same way as a standard integral, except that rather than returning a real number, it returns a linear operator on (a dense subset of) H. As usual, the defining property of this integral is that

Using this, you can prove various properties that we are used to seeing, such as

However, this integral also has an unusual property that is less familiar:

The proof of this comes from the fact that P^A(S₁∩S₂)=P^A(S₁)P^A(S₂) (which makes intuitive sense considering our discussion about consistent measurements, but I leave the formal proofs of both of these statements to the reader).

Another curious property of this integral is that it transforms complex conjugation into taking the adjoint. Recall (or learn for the first time—nobody is judging) that if A is a linear operator, then its adjoint A† is the unique linear operator such that ⟨ψ, Aϕ⟩=⟨A†ψ, ϕ⟩ for every ψ, ϕ in H.

(Strictly speaking, this is only the right definition if A is defined on the entirety of H, which most linear operators in quantum mechanics are not—more on this in a second.)

Given this, we have for any function f: ℝ→ℂ,

Curious as this all is, what is the relevance of this integral to observables? Using it, we define a linear operator on H, which in some sense is the simplest interesting operator that we could define this way. Abusing notation, we denote this operator by A as well.

This operator has many interesting properties. To start with, it is self-adjoint—that is to say, if it is defined on all of H, then A=A†. If A happens to only be defined on a dense subset of H (which is possible), then a slightly more involved definition is required.3 However, even in that case the right intuitive idea is that A=A†, which is easy to check from the fact that the integral transforms conjugation into taking the adjoint.

Furthermore, the spectrum σ(A) of A is the set of possible values for a measurement of A. (The spectrum of a linear operator is sort of like the set of its eigenvalues, but really it is a substantial generalization. There are many self-adjoint operators without any eigenvectors, but their spectrum is never empty.) This follows from the fact that σ(A) is the largest set such that

but we already know that this is the same as asking that P^A(R\σ(A))=0.

More importantly, for any state ψ in H, ⟨ψ, Aψ⟩ gives the expected value of the observable A! To demonstrate this, we decompose σ(A)=⋃_iS_i, with the sets S_i chosen such that the diameter4 of S_i is small—say, no more than 2ε. Then there exists some real number λ_i such that every element in S_i is within ε of it. We then consider the corresponding orthogonal sum

where H_S is the largest subspace of H with the property that for any state in the subspace, if we measure A for that subspace, that measurement has to be in S. Due to the way that we defined the inner product, we know that all of these subspaces are indeed orthogonal and (the closure of) their orthogonal sum must be the whole space H.

Now, choose any state ψ in the domain of A. We note that

Observe that due to how we defined them, the states P^A(S_i)ψ are all incompatible. They are not, however, norm 1. So, let’s define ψ_i to be the normalization of P^A(S_i)ψ, so that we can write P^A(S_i)ψ=c_iψ_i for some complex number c_i. Thus,

and so we know that |c_i|² is the probability that when we measure our observable A for ψ, the result will be consistent with ψ_i—that is, it will be in S_i!

Let’s write this in a more convenient form. Observe that

Ah ha! So, ⟨ψ, P^A(S_i)ψ⟩ measures the component of ψ consistent with measuring A in S_i, and this is precisely the probability of measuring A in S_i for the state ψ.

This means that the expected value of A for the state ψ should be

(In the first line, we multiply the probability of landing in S_i by the value that we get if we do so—this is approximately λ_i by construction.) This approximation becomes better and better if we decompose σ(A) more and more finely, and therefore in the limit we get exact equality.

We can in fact show that by calculating with A², A³, A⁴ and so on, we can also obtain the higher moments of the distribution of A for the state ψ—to see this, note that due to the properties of the integral that we described earlier,

and so we can do the same argument as we did for 𝔼(Aψ).

This should point us toward the right conclusion, which is that this operator A actually encodes all of the information about the distribution of the observable A. Indeed, that this must be true is a consequence of the Spectral Theorem, which says that5 for any self-adjoint operator A there is a unique projection-valued measure P^A such that

Ergo, we can consider projection-valued measures or the self-adjoint operator—they actually capture the same information. The operator A is usually easier to work with, and so we just identify it with the observable A.

To recap: observables on the state space H are self-adjoint operators A, or equivalently projection-valued measures P^A with the property that

1. ⟨ψ,P^A(S)ψ⟩ is the probability of measuring A in the set S for the state ψ, and

2. P^A([a,b])ψ is the state that ψ collapses to if we measure A to be in the range [a,b].

It remains to determine how time-evolution works. We define a family of operators A(t) such that A(t)ψ gives the state in which ψ will be after time t.

Now, some observations.

1. A(0)=I—if I don’t time-evolve at all, the state shouldn’t change.

2. A(s+t)=A(s)A(t)—this is because time-evolving by t and then time-evolving by s should be the same as time-evolving by s+t. (It follows that A(-t) is the inverse of A(t), since A(t)A(-t)=A(0)=I.)

3. A(t) had better be linear. This is because we expect that a superposition of states c₁ψ+c₂ϕ time evolves like the constituent states—i.e., A(t)(c₁ψ+c₂ϕ)=c₁A(t)ψ+c₂A(t)ϕ.

4. A(t) had better be unitary—that is, ⟨A(t)ψ, A(t)ϕ⟩=⟨ψ,ϕ⟩ for every pair of states ψ, ϕ. This is because of what physicists call “conservation of probability” (really, just the fact that we should maintain our states as normalized), and also the fact that if two states ψ and ϕ are incompatible (i.e., ⟨ψ, ϕ⟩=0) then their time-evolved states should also be incompatible.

5. A(t) should be continuous in t, in the sense that we should have lim_Δt→0(A(t+Δt)ψ−A(t)ψ)=0. This is because if we measure a state, and then measure it again a very small time amount later, we don’t expect it to change much.

This is what is called a strongly-continuous, 1-parameter unitary group, and there is a very beautiful theorem about such things (called Stone’s theorem): for any 1-parameter unitary group A(t), there exists a self-adjoint operator H such that A(t)=e^−itH. Here, if P^H is the projection-valued measure associated to H by the spectral theorem, then

Now, we don’t know anything a priori about this operator H, but from the fact that it is a self-adjoint operator, we might guess that it corresponds to some observable quantity of our system. Knowing the significance of energy in time-evolution in the classical case, we might guess that H is the operator corresponding to the energy of the system—and experiment confirms this, although it would be slightly more accurate to write A(t)=e^−itH/ℏ in that case, which we shall do henceforth.6

So, now, we can define ψ(t)=A(t)ψ, which we know is the state that ψ becomes after some time t. We have ψ(t)=e^−itH/ℏψ, and to get out the Schrödinger equation, all we have to do is to take a derivative:

Rearranging, we get the classic equation iℏψ’(t)=Hψ(t).

To recap: states ψ(t) satisfy the Schrödinger equation iℏψ’(t)=Hψ(t), where H is the observable corresponding to energy, and ℏ is an experimentally determined parameter.

So there you have it. In my opinion, although this formulation requires a lot more familiarity with mathematics, it has two clear advantages over the usual treatment given in physics courses:

1. We never have to talk about things that clearly don’t exist (like Dirac delta functions)—projection-valued measures fairly directly model collapse and what we actually do in experiments, and self-adjoint operators come along as a nice computational convenience.

2. We can directly motivate each piece of the construction. It may have taken a genius like von Neumann to come up with using Hilbert spaces as tools in quantum mechanics, but it shouldn’t take a genius to appreciate why that is the right choice!

That said, if you do want to work with Dirac delta functions (admittedly, there are cases where it is convenient), then we can insert this into our paradigm—it just requires some extra work. Specifically, we need to introduce something called a rigged Hilbert space. But, again, that is a problem for another time.