How Much Catalog Fits in a Semantic ID

There are two live bets about how a recommender should name its items, and they point in opposite directions.

The first bet is to keep the naming scheme dumb and make the model enormous. Meta's HSTU does this. Every item keeps a sparse ID with its own embedding row, and the work goes into the sequence architecture and the scaling. The paper is candid about what that costs: "With a 10b vocabulary, 512d embeddings, and Adam optimizer, storing embeddings and optimizer states in fp32 already requires 60TB memory." That is the item vocabulary alone, before any of the parameters that do the actual modeling. In exchange you get a scaling law that holds across three orders of magnitude, and a 1.5 trillion parameter model in production.

The second bet is to make the naming scheme carry meaning so the model can be ordinary. This is the semantic ID line: TIGER, then PLUM, then OneRec. An item stops being a row in a table and becomes a short sequence of discrete codes derived from what the item is. Recommendation becomes next-token prediction over a vocabulary where the tokens mean something, and the embedding table stops growing with the catalog. Under PLUM's codebook schedule the entire item vocabulary is 3,840 rows, about 7.9 MB at 512 dimensions, whether the catalog has ten thousand items or ten million.

Seven megabytes against sixty terabytes is a big enough gap to be suspicious of. The compression has to come from somewhere, and it does. This post is about where, because the answer turns out to be measurable, and it gives you a way to size a codebook before you train anything.

What a semantic ID actually is

Start with a content embedding z for an item, produced by running its title, description, and images through frozen encoders. Residual quantization turns that continuous vector into L discrete codes by rounding it repeatedly, each time against what the previous rounding missed.

Set r₀ = z. At each level l you pick the nearest entry in that level's codebook, emit its index, and subtract it off:

The reconstruction is just the sum of the codes you picked, ẑ = Σ_l e^(l)_cl, and the semantic ID is the tuple (c₁, …, c_L). Because level 1 is chosen before level 2 exists, items that agree on c₁ are genuinely in the same coarse neighborhood. The hierarchy is a consequence of the algorithm, not something imposed afterward.

Training the codebooks is where the subtlety is. The RQ-VAE objective has a reconstruction term and a quantization term, and the quantization term is split in two by where the gradient is allowed to flow:

sg[·] is the stop-gradient. The two terms measure the same distance.

Both terms measure the distance between a residual and the code that was picked for it. They differ only in which side is held fixed. In the first, the residual is frozen and the gradient moves the codebook vector toward it, so this is the term that trains the codebook. In the second, the codebook vector is frozen and the gradient moves the encoder, so this is the term that pressures the encoder to produce vectors that are already close to codes it can reach. β sets how hard you lean on the encoder to commit. Eugene Yan found β = 0.5 gave the lowest validation loss on his catalog, against the 0.25 used in the original semantic ID papers, which is a reminder that this is a dataset-dependent knob and not a constant.

The capacity gap

Now the arithmetic that motivated this post.

Three levels of 256 codes gives 256³ = 16,777,216 distinct semantic IDs. Yan trained exactly that configuration on 66,000 Amazon video game products and reported 89% unique IDs, meaning roughly one item in nine landed on a code some other item already had. He resolved it by appending a fourth, sequential disambiguation token.

Sixty-six thousand items into sixteen million slots collides 11% of the time. That should feel wrong, and it is worth seeing how wrong. If the codes were spread evenly over the space, the probability that a given item is unique is the probability that none of the other N − 1 items chose its code:

With N = 66,000 and M = 16.8 million, that predicts 99.6% unique, or about 260 collisions in total. The observed number is roughly 7,000. The uniform model is off by a factor of twenty-five, which means the codes are nowhere near evenly spread.

The useful move is to stop treating that as a failure of the model and start treating it as a measurement. Define the effective capacity of a codebook as the reciprocal of the probability that two randomly drawn items collide:

the inverse of the Simpson index over the code distribution p

If every code were used equally often this equals the nominal capacity K^L. Any concentration in the code distribution drives it down. And because the same quantity governs the collision rate, you can read it straight off the observed uniqueness by inverting the expression above:

Substituting N = 66,000 and u = 0.89:

Spread evenly across three levels, 19.1 bits is 6.4 bits per level, which is about 83 usable codes out of every 256. Two thirds of each codebook is, for addressing purposes, not there. This is despite codebook usage histograms that look broadly uniform on inspection, which is the standard health check. A histogram that looks flat to the eye can still be concentrated enough to cost you a factor of thirty in capacity, because collisions are driven by Σp² and that sum is dominated by the heavy codes.

The intuition is that content embeddings are not uniformly distributed over the sphere. Catalogs are lumpy. A video game catalog has an enormous number of near-identical PlayStation sports titles and very few Sega Saturn cables, and residual quantization faithfully reproduces that lumpiness in the code distribution. The quantizer is not malfunctioning. It is compressing a distribution that has far less entropy than its support suggests.

A rule for sizing a codebook

Inverting the relation once more gives a design rule. To hold uniqueness at u on a catalog of N items you need M_eff ≥ N / ln(1/u), which for the thresholds anyone actually cares about is:

The 99% row is the one to remember: you need roughly a hundred times more effective addresses than you have items, and if your quantizer realizes 3.4% of nominal you need roughly three thousand times more nominal ones. Sizing a codebook to the catalog is the mistake. Sizing it to a hundred times the catalog, divided by whatever efficiency you actually measure, is the rule.

Worked through for a one million item catalog at 99% uniqueness: you need about 99.5 million effective addresses, so about 2.9 billion nominal ones. Four levels of 256 gives 4.29 billion, which clears it, and at the same 3.4% efficiency predicts 99.3% unique. Three levels of 256 would have given you 16.8 million nominal, 566 thousand effective, and a catalog where five out of six items share a code with something else. The difference between those two designs is one line of config, and you cannot see it without doing the arithmetic.

Where the missing bits went

Averaging the loss across levels was a convenience. It is not what happens. The entropy of an item's full code decomposes by the chain rule:

Each level can contribute at most log K_l bits, and it contributes exactly that only if its codes are used uniformly given everything above it. But residual quantization removes the largest structure first. By level four the residual is close to noise, its conditional entropy is small, and a wide codebook there is mostly empty rows. Uniform codebooks spend the same capacity at every level regardless of how much entropy is left to encode, which is precisely the concentration that shows up as a collapsed M_eff.

This is the argument for multi-resolution codebooks, and PLUM's schedule follows it directly: codebook cardinality of 2048/2^(level−1), so 2048, then 1024, then 512, then 256. Allocate capacity where the conditional entropy actually is, and stop paying for addresses at depth where nothing is left to distinguish.

There is a second, blunter lever, and OneRec pulls it. Rather than tapering capacity to match the entropy, force the code distribution to be uniform by construction. OneRec quantizes with a balanced residual K-means in which each cluster holds exactly w = |𝒱| / K items, explicitly to avoid what the authors call the hourglass phenomenon of unbalanced code usage.

In the terms of this post, balanced assignment sets p_c = 1/K exactly, which drives Σp² to its minimum and makes M_eff equal to the nominal K^L. It recovers the entire missing factor of thirty. It is not free: an item near a cluster boundary gets assigned to a centroid that is not its nearest, because the balance constraint has to be satisfied. You buy addressing capacity by spending geometric fidelity. Which trade is right depends on whether your failure mode is collisions or blurred neighborhoods, and that is an empirical question about your catalog rather than a matter of taste.

PLUM pairs this with progressive masking: during training the codes are truncated at a random depth, using a binary mask m_l = 𝟙_{(l < r)} with r drawn uniformly from [1, L]. The model is therefore forced to reconstruct from a prefix as often as from the full code, which means the level-1 code has to be independently meaningful rather than meaningful only in combination with the levels below it.

That property is not cosmetic. It is what makes constrained beam search work. Decoding a recommendation means walking the prefix tree and pruning branches, and pruning on a prefix is only sound if the prefix names a coherent region. If c₁ is meaningless alone, every pruning decision at the first step is close to arbitrary, and the beam has thrown away the right answer before it has seen enough tokens to know. Hierarchy in the codes and correctness in the decoder are the same property viewed from two ends.

Behavior as an objective, not an input

Everything so far derives codes from content. But the thing you want to predict is behavior, and the obvious move is to concatenate a collaborative filtering embedding onto the content vector before quantizing.

That move fails for a reason worth stating precisely: CF embeddings drift with popularity. If an item's code is a function of its interaction counts, the code changes as the item trends. An identifier that means something different this month is not an identifier, and every downstream model that memorized last month's codes is now wrong.

PLUM's answer is to keep the code a function of stable content and push behavior in through the loss instead. A contrastive term over items that co-occur in user sequences pulls their quantizer representations together and pushes non-co-occurring items apart:

The collaborative structure ends up in the geometry of the code space without the code itself depending on interaction counts. Items bought together land near each other; an item that goes viral keeps its ID. UTGRec reaches a similar place from the transfer direction, using co-occurrence alignment so one tokenizer generalizes across catalogs rather than being refit per tenant.

What this means for the bet

Return to the two bets. HSTU's 60 terabytes buys an addressing scheme with no collisions at all, because every item has its own row by construction. Semantic IDs buy a vocabulary that does not grow with the catalog, transfers across domains, and gives brand new items a sensible code the moment they exist, because the code is a function of content rather than of interaction history. Cold start stops being a special case.

What the arithmetic here says is that the second bet has a specific, quantifiable cost, and it is not the one people usually name. The cost is not that the codes are approximate. It is that the addressable space collapses to a few percent of what the configuration advertises, and if you size the codebook off the nominal number you will discover this after training rather than before. Yan's fourth disambiguation level is the standard patch, and it works, but it is a patch applied to a capacity shortfall that was predictable from N, the target uniqueness, and one measured efficiency number.

The honest summary is that both bets are still open. HSTU demonstrates power-law scaling to GPT-3 scale on sparse IDs. PLUM reports substantial retrieval gains over a heavily optimized production model built on large embedding tables. They cannot both be the last word, and the comparison nobody has run cleanly is the one that holds data, compute, and evaluation fixed while varying only how items are named.

Where we are

We are building the tokenizer end of this: fitting codebooks over a catalog, folding collaborative signal in through the objective rather than the input, and handing back codes you can feed to a model you already own. The product spec is public and is deliberately a spec rather than an announcement. The endpoints are not live.

Two questions we have not resolved. Whether one tokenizer should transfer across tenants or be fit per catalog, which is the cold-start quality of day one traded against the ceiling at maturity. And what to do about catalogs where a large fraction of items carry no real text or images, since the entire approach assumes content exists to encode. If you have a catalog with either property we would rather hear about it than guess.

Acknowledgement

This post is downstream of Eugene Yan's write-up on training an LLM-recommender hybrid with semantic IDs. He built the thing, ran the sweeps, and published the collision figure that the entire analysis here is anchored to. Reporting 89% uniqueness rather than rounding it away is what made the derivation possible at all, and the shape of this piece owes a good deal to his. The arithmetic built on top of his number, and any errors in it, are ours.