Row and Column Attention Are Kernel Smoothing

The language around tabular transformers makes row attention and column attention sound like new modeling primitives. Algebraically they are not. They are two applications of an idea every statistician already owns: define a similarity, exponentiate it, normalize it and take a weighted average. That is Nadaraya-Watson.

The clean way to see this is to not start with attention. Start with the table.

We have a dataset

where each row xᵢ = (xᵢ₁, ..., xᵢₚ) is a sample point and each column is a feature. For prediction the object we care about is the row: a borrower, a patient, a transaction, a period. The supervised problem is xᵢ ↦ yᵢ, and for a new point x⋆ the operative question is which training rows are close to x⋆.

That is already the kernel question.

Take the dot product as the similarity between two rows,

Under constant norm this is monotone in distance, since ‖x⋆ − xᵢ‖² = ‖x⋆‖² + ‖xᵢ‖² − 2 x⋆ᵀxᵢ. Larger dot product, smaller distance. Exponentiate to get a strictly positive kernel,

We rarely want similarity in the raw coordinates. Map the points into query and key representations first,

so the kernel becomes

Nothing exotic has happened. We took the table, chose a similarity between rows and wrote it as a kernel in a learned feature space.

Given a kernel, the natural nonparametric rule is Nadaraya-Watson smoothing:

Substitute the exponential dot-product kernel and the weights are

This is softmax. The exponential dot product builds the kernel, the denominator is the Nadaraya-Watson normalization and the output is a weighted average. So:

Softmax attention is Nadaraya-Watson smoothing with an exponential dot-product kernel. When the objects are rows, row attention is kernel smoothing over sample points.

One point of care, because it is where the analogy is usually stated too loosely. A generic attention layer does not average labels. It averages learned value vectors vⱼ = W_V xⱼ, and a head maps the smoothed output to a prediction afterward. The estimator has three learned slots — the key map that selects, the value map that is averaged and the query that probes — not two.

The literal “average the labels” reading is the special case that matters most for tables. In in-context tabular models such as TabPFN and TabICL, the training labels enter through the value slot, so the row smoother really does compute ŷ(x⋆) = Σᵢ αᵢ(x⋆) yᵢ. That is Nadaraya-Watson regression on the support set, with a kernel learned across many synthetic tasks rather than chosen by the modeler.

Is it a Mercer kernel? Only when the maps are tied. With W_Q = W_K = W,

which is a Gram kernel, symmetric and positive semidefinite. With W_Q ≠ W_K the score is bilinear,

and M is generally not symmetric, so the kernel is neither symmetric nor PSD. It is not a reproducing kernel. It is still a valid smoother — exp(sᵢⱼ) > 0 and the row normalizes to one — so the Nadaraya-Watson interpretation survives even though the RKHS one does not. Writing M = S + A as symmetric plus antisymmetric parts, the symmetric part is the metric similarity and the antisymmetric part is a directional term that makes i attend to j differently than j to i. Tied maps set A = 0.

Where is the bandwidth? The 1/√d temperature is it. Classical Nadaraya-Watson picks a scalar bandwidth h; attention fixes the global scale to 1/√d and lets the singular values of M do anisotropic, direction-dependent scaling. So attention does not select a bandwidth, it learns an anisotropic metric.

A twenty-line check that the kernel view and the softmax code are the same object:

import numpy as np
rng = np.random.default_rng(0)
n, p, d = 200, 20, 16
X   = rng.standard_normal((n, p))
W_Q = rng.standard_normal((p, d)) / np.sqrt(p)
W_K = rng.standard_normal((p, d)) / np.sqrt(p)
Q = X @ W_Q                       # queries  n x d
K = X @ W_K                       # keys     n x d
S    = Q @ K.T / np.sqrt(d)       # scores, 1/sqrt(d) is the bandwidth
Kmat = np.exp(S)                  # exponential dot-product kernel, strictly positive
A    = Kmat / Kmat.sum(1, keepdims=True)   # Nadaraya-Watson weights
def softmax(Z):                   # standard stabilized softmax
    Z = Z - Z.max(1, keepdims=True)
    E = np.exp(Z)
    return E / E.sum(1, keepdims=True)
assert np.allclose(A, softmax(S)) # attention weights are the normalized kernel

The max-subtraction in the stabilized softmax shifts every score in a row by a constant, which cancels in the ratio. Same kernel, same weights.

For prediction the object is the row, but the distance between two rows depends on how the features inside a row are represented. Debt-to-income means one thing at high income and another at low. Utilization means one thing at high credit score and another at low. Before comparing two rows it helps to build a better representation of each,

and apply the row kernel to the representations, K(r⋆, rᵢ), rather than to raw rows. Column attention is one way to build g.

Fix a row. The objects are now the feature entries xᵢ₁, ..., xᵢₚ. A scalar dot product between two feature values is not a similarity, so each entry is first embedded into a token,

with a per-feature embedding, in the style of FT-Transformer. Then the within-row kernel between features a and b is

normalized over b, and each feature token is updated by smoothing over the others,

Pool the smoothed tokens, through a CLS token or mean pooling, into the row representation rᵢ. This is again a kernel smoother, now over features within a row instead of over rows.

Column attention is kernel smoothing over features. Row attention is kernel smoothing over rows.

The column pass builds each row’s representation, the row pass compares rows in that representation and averages values. The effective predictive kernel is the composition,

which is what makes it adaptive: the geometry depends on the table.

The form similarity → exponentiate → normalize → weighted average is classical kernel smoothing. The dot product is a Gram similarity, the exponential dot product is a kernel, the softmax is the Nadaraya-Watson normalization and the output is a weighted average. As a mathematical primitive, attention is not new.

What is new is that the kernel is learned and composed. Classical kernel regression fixes the geometry, for example K(x⋆, xᵢ) = exp(−‖x⋆ − xᵢ‖² / 2σ²), with the modeler choosing σ. A tabular transformer learns how to represent feature values, how to compose them into a row and how to compare rows, and amortizes that choice across tasks.

That reframes the empirical question. It is not whether attention differs from kernels — it does not. It is whether the learned kernel Kθ(x⋆, xᵢ; X) defines a better geometry for the problem than the alternatives a statistician would already reach for: an RBF kernel, random forest proximity, a gradient-boosted tree kernel or a supervised embedding. That is the right comparison and it is an empirical one.

For tabular prediction, start with the table. Rows are sample points, columns are coordinates, a kernel defines similarity. Column attention applies a kernel over features and learns what a point means. Row attention applies a kernel over rows and decides which points are close. Prediction is Nadaraya-Watson smoothing of the values, which in the in-context setting are the labels themselves.

From a kernel perspective there is nothing to mystify. Attention is kernel smoothing on learned objects, run once down the columns to build the geometry and once across the rows to predict in it.