Skip to main content

Semantic IDs

The endpoints below describe what we are building. Reach out to shape the surface or join the closed beta.

Every generative recommender needs a vocabulary before it can be trained. Not a catalog, not an index, a vocabulary: a set of discrete tokens that stand for items and carry meaning in their own structure. Building that vocabulary well is the first bottleneck any team hits, and it caps everything downstream before a single training step runs.

We build the vocabulary. It is the part of the stack we think is most underbuilt and the part we are betting on.

What a semantic ID is​

Classical recommenders give every item a random integer ID and learn an embedding row for it. That works, and it has three costs that get worse with scale: the embedding table grows with the catalog, a new item is meaningless until it has interactions, and nothing learned about one catalog transfers to another.

A semantic ID replaces the random integer with a short sequence of discrete codes derived from what the item actually is.

item_8f3a2b  →  <sid_0_1487><sid_1_302><sid_2_91>
                 coarse       finer      finest

Recommendation then becomes next-token prediction over a vocabulary where the tokens mean something. The model is no longer retrieving from an index. It is generating an identifier, and every prefix of that identifier is a real region of item space.

Why this is the bottleneck​

Most of the interesting variance in generative recommendation sits in how you build the codes, not in the model that consumes them.

A tokenizer that collapses distinct items into the same code caps your ceiling before training starts, and no amount of model capacity recovers it. One that encodes content but ignores how items are actually consumed produces codes that are semantically tidy and commercially useless. One that leans too hard on behavior produces codes that move as items trend, which means the identifier is not an identifier.

Getting this right is difficult, it is upstream of everything else, and it is not the part most teams want to spend two years on. As generative recommenders become the default architecture, every team building one will need a vocabulary, and most of them should not be building their own.

What makes ours grounded​

Grounded means the code reflects both what an item is and how it is actually consumed, while staying stable enough to function as an identifier. Those pull against each other, and holding both is the hard part.

Multi-modal contentText and images are encoded per modality and fused before quantization, so the code reflects the item rather than a description of it.
Behavior as objective, not inputReal consumption sequences shape the quantizer during training, so items consumed together land near each other. Behavior never enters as an input feature, so a code does not drift when an item trends.
Hierarchical by constructionEach level is trained to be meaningful on its own, not only in combination with the levels below it. This is what makes prefix-constrained decoding prune a coherent region instead of an arbitrary one.
Uniquely addressableItems that quantize identically are disambiguated, so a semantic ID always resolves to exactly one item.

The result is a vocabulary where a prefix is a real neighborhood, a full code is a stable identity, and both properties survive contact with a live catalog.

Continuously updated​

This is the part that tends to decide whether the approach survives production.

A catalog is not static. Items arrive, categories emerge, and the distribution the tokenizer was fit on stops describing reality. The naive answer is to refit, which produces a new code space, which invalidates every stored ID and forces a full reindex. Teams that plan for a quarterly refit generally discover they cannot afford one.

Ours updates in place. New regions are absorbed into the existing codebooks without moving the codes already assigned, so:

  • Existing semantic IDs stay valid. No reindex, no cutover, no dual-write window.
  • A model trained against the vocabulary stays trained. The tokens it learned still mean what they meant.
  • New items and new regions become representable continuously rather than at refit boundaries.

A full refit into a fresh code space remains available and is occasionally the right call after a genuine change in what you sell. It stops being the routine cost of running the system.

Cold start​

A semantic ID is a function of content, so an item that went live sixty seconds ago with zero interactions still gets a code in the right neighborhood, and the model can recommend it immediately.

Pass a brand-new item to POST /semantic-ids and it is assigned against the existing codebooks. The response flags it:

{ "item_id": "sku_new", "codes": [1487, 88, 405, 3], "cold_start": true }

The API​

Four endpoints. Fit a tokenizer, check on it, assign IDs, generate.

curl https://api.jeantechnologies.com/v1/tokenizers \
  -H "Authorization: Bearer $JEAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "catalog-v1",
    "catalog": { "uri": "s3://acme/catalog.jsonl" },
    "codebook": { "levels": 4, "resolution": "multi", "base_size": 2048 },
    "collaborative_signal": {
      "mode": "co_occurrence_contrastive",
      "interactions": { "uri": "s3://acme/sequences.jsonl" }
    }
  }'

A generation response returns real items, because beam search is constrained to prefixes that exist in the codebook:

{
  "recommendations": [
    {
      "item_id": "sku_402",
      "codes": [1487, 302, 91, 12],
      "tokens": "<sid_0_1487><sid_1_302><sid_2_91><sid_3_12>",
      "score": 0.31
    }
  ],
  "beams_explored": 40,
  "beams_pruned_invalid": 0
}

Full reference: Fit tokenizer, Get tokenizer, Assign semantic IDs, Generate.

Sizing the codebook​

resolution: "multi" sizes each level as base_size / 2^(level-1), which matches how residual entropy actually falls off with depth. With the defaults:

LevelCodebook sizeEncodes
12048Coarse semantic region
21024Sub-region
3512Fine distinction
4256Residual detail

Total addressable space is the product of the level sizes, so the defaults cover roughly 275 billion distinct IDs. Address space is almost never the binding constraint. Whether level 1 has enough genuinely distinct regions in your catalog to fill 2048 codes usually is. See Fit tokenizer for how to check that after a fit.

What we are still deciding​

Writing this page is partly how we are working out the product. Open questions, and we would rather hear from you than guess:

Should the tokenizer be per-tenant or universal?

A tokenizer fit on your catalog alone is likely better on your catalog. One fit across many catalogs transfers, which is what makes day one useful before you have your own signal. The tradeoff is cold start quality now versus ceiling at maturity, and we suspect the answer differs by catalog size.

Do you want the codes, or the recommendations?

Some teams want POST /generate and nothing else. Others want to export semantic IDs and feed them into a ranker they already own. These imply fairly different products.

How much of the catalog do you actually have content for?

The whole approach assumes items carry real text or images. Catalogs where half the items are a bare SKU string change the design.

Talk to the team: jonathan@jeantechnologies.com