Associative memoryas a fuzzy perspective on
architecture design
Presented at Perplexity · November 2026
Based on You could have invented transformers from memory and Test-time regression (JMLR)
Associative memoryas a fuzzy perspective on
Why are transformer architectures so arbitrary?
- We describe each part by what it computes: projections, nonlinearities, where connections sit
- We rarely describe what it's for
- So design choices feel contrived:
- ReLU, GELU, SiLU, SwiGLU, ...
- short convolutions on keys and values
- attention sinks, gated attention
- linear attention, DeltaNet, state-space models, ...
- Is there a language for what each part is trying to do?
Associative memory is an old idea
- Kohonen, Hopfield, and others studied it in the 1970s and 80s
- Hopfield shared the 2024 Nobel Prize in Physics for it
- A Hopfield network is a toy model of how a brain might store memories: give it a corrupted pattern, and it recalls the stored one
- Most of the field has forgotten this, or never learned it
The simplest memory store adds outer products
- A memory is a key-value pair: a key $k_i \in \mathbb{R}^{d_k}$ to look it up by, and a value $v_i \in \mathbb{R}^{d_v}$ to retrieve
- Store $M$ memories by adding their outer products: $$W = \sum_{i=1}^{M} v_i k_i^\top$$
Recall weights each value by its key's similarity
- Recall by multiplying with a query $x$: $$f(x) = Wx = \sum_{i=1}^{M} v_i \, (k_i^\top x)$$
- If the keys are orthonormal, $k_i^\top k_j$ is 1 when $i = j$ and 0 otherwise, so querying with a stored key returns its value: $$f(k_2) = v_1 \underbrace{(k_1^\top k_2)}_{0} + v_2 \underbrace{(k_2^\top k_2)}_{1} + \cdots + v_M \underbrace{(k_M^\top k_2)}_{0} = v_2$$
Recall picks up interference
- Query with a stored unit-length key $k_j$: $$f(k_j) = \underbrace{v_j}_{\text{signal}} + \underbrace{\sum_{i \neq j} v_i \, (k_i^\top k_j)}_{\text{interference}}$$
- Orthonormal keys give no interference, but $\mathbb{R}^{d_k}$ only has $d_k$ of them
- Random keys have $k_i^\top k_j \approx \pm 1/\sqrt{d_k}$, so $$\frac{\|\text{interference}\|}{\|\text{signal}\|} \sim \sqrt{\frac{M}{d_k}}$$
- Heuristic: interference overwhelms the signal and recall fails when $M \sim d_k$
How can we do better?
- Make the memory larger by increasing $d_k$ (the dimension of your linear map)
- Store things more carefully
- Suppress interference
Store carefully: the optimal linear memory
- Choose $W$ to fit every memory at once, a linear regression: $$\min_W \sum_{i=1}^{M} \|W k_i - v_i\|^2$$
- For $M > d_k$, the solution whitens the keys before comparing them with the query: $$W = V^\top K \, (K^\top K)^{-1}$$
- No interference up to $M = d_k$, and capacity doubles to $M \approx 2 d_k$
Recall selectively: the MLP
- Pass each similarity through a nonlinearity $\sigma$: $$f(x) = \sum_{i=1}^{M} v_i \, \sigma(k_i^\top x) = V^\top \sigma(Kx)$$
- This is a two-layer MLP: the rows of the first weight matrix are keys, the columns of the second are values
- $\sigma = \mathrm{ReLU}$ chops off keys with negative similarity, so the less relevant memories add no interference
- For random keys, about half the similarities are negative, so ReLU drops about half the interference terms
Suppress harder: square the ReLU
- $\sigma(s) = \mathrm{ReLU}(s)^2$ shrinks small similarities far more than large ones: 0.1 becomes 0.01, while 1 stays 1
- This is a GLU MLP whose gate shares weights with the input projection: $$V^\top \mathrm{ReLU}(Kx)^2 = V^\top \big(\mathrm{ReLU}(Kx) \odot Kx\big)$$
- SwiGLU gives the gate its own weights $G$ and uses Swish in place of ReLU: $V^\top \big(\mathrm{Swish}(Gx) \odot Kx\big)$
Capacity: linear maps vs MLPs
- Capacity is where each line crosses interference = signal
- Linear: $M \approx d_k = 64$
- ReLU MLP: $M \approx 2 d_k = 128$
- Squared ReLU (GLU) and SwiGLU MLPs: $M \approx 2 d_k^2 / 3 \approx 2{,}700$, about 40 times the linear store's
Every power at once: the exponential
- Raise the power: $\mathrm{ReLU}(s)^p$ keeps a match of 1 at 1, but shrinks a typical similarity $1/\sqrt{d_k}$ to $d_k^{-p/2}$
- Capacity grows to about $d_k^p$: about 35,000 for $p = 3$
- Push it to the limit and use every power at once: $$e^{\beta s} = \sum_{p=0}^{\infty} \frac{(\beta s)^p}{p!}$$
- Capacity grows exponentially: $M \approx e^{2\beta(1 - \beta/d_k)}$, about 1.2 million at $\beta = \sqrt{d_k} = 8$
Every layer's capacity
- How many random memories each store holds before interference matches the signal
| Memory store | $f(x)$ | Capacity | At $d_k = 64$ |
|---|---|---|---|
| Outer product | $V^\top K x$ | $d_k$ | 64 |
| Optimal linear | $V^\top K (K^\top K)^{-1} x$ | $2 d_k$ | 128 |
| ReLU MLP | $V^\top \mathrm{ReLU}(Kx)$ | $2 d_k$ | 128 |
| Squared ReLU (GLU MLP) | $V^\top \mathrm{ReLU}(Kx)^2$ | $2 d_k^2 / 3$ | about 2,700 |
| SwiGLU MLP | $V^\top \big(\mathrm{Swish}(Kx) \odot Kx\big)$ | about $2 d_k^2 / 3$ | about 2,900 |
| Cubed ReLU | $V^\top \mathrm{ReLU}(Kx)^3$ | $2 d_k^3 / 15$ | about 35,000 |
| Exponential, $\beta = \sqrt{d_k}$ | $V^\top \exp(\beta K x)$ | $e^{2\sqrt{d_k} - 2}$ | about 1.2 million |
Slow weights and fast weights
- An MLP's keys and values are parameters: training writes them, slowly, and they stay fixed while the model runs
- Attention's keys and values come from the tokens in context: the forward pass writes them, one memory per token
- So attention's memory store grows with the context
- Slow weights change only during training; fast weights change with every token (Schmidhuber 1992)
Memorizing is regression
- The optimal linear memory fit every memory at once: $$W = \arg\min_W \sum_{i=1}^{M} \|v_i - W k_i\|^2$$
- Generalize: weight each memory by $\gamma_i^{(t)}$, and fit any function class $\mathcal{M}$: $$m_t = \arg\min_{m \in \mathcal{M}} \tfrac12 \sum_{i=1}^{t} \gamma_i^{(t)} \|v_i - m(k_i)\|^2, \qquad y_t = m_t(q_t)$$
- Design fast-weight layers, attention included, by having the layer solve this regression inside its forward pass
Test-time regression: three knobs
$$m_t = \underbrace{\arg\min_{m \in \mathcal{M}}}_{\color{#B3202A}{3}} \; \tfrac12 \sum_{i=1}^{t} \underbrace{\gamma_i^{(t)}}_{\color{#B3202A}{1}} \, \big\|v_i - \underbrace{m}_{\color{#B3202A}{2}}(k_i)\big\|^2$$
- Regression weights $\gamma_i^{(t)}$: which memories to trust (uniform, decaying, by similarity)
- Function class $\mathcal{M}$: linear, MLP, non-parametric
- Optimizer: one gradient step, one SGD step per token, exact solve
One objective, many layers
| Layer | 1. Weights | 2. Function class | 3. Optimizer | Memory update |
|---|---|---|---|---|
| Linear attention | uniform | linear | one gradient step | $M_t = M_{t-1} + v_t k_t^\top$ |
| Mamba-2, GLA, RetNet | decaying | linear | one gradient step | $M_t = \gamma_t M_{t-1} + v_t k_t^\top$ |
| Mesa-layer | uniform or decaying | linear | exact solve | $M_t = V_t^\top K_t (K_t^\top K_t)^{-1}$ |
| DeltaNet | uniform | linear | SGD, one step per token | $M_t = M_{t-1}(I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top$ |
| Gated DeltaNet | uniform | linear | SGD with weight decay | $M_t = \alpha_t M_{t-1}(I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top$ |
| TTT-MLP | uniform | MLP | mini-batch SGD | |
| Softmax attention | by similarity to $q_t$ | non-parametric | exact solve | keeps every $(k_i, v_i)$ |
How it all fits together
ReLU
SwiGLU
attention sinks
RoPE
Mamba
GELU
short convolutions
gated attention
QK-norm
linear attention
squared ReLU
GQA
DeltaNet
sliding window
RetNet
SiLU
mixture of experts
Gated DeltaNet
GLA
MLA
RWKV
softmax attention
log-n scaling
Titans
TTT
memory layers