Fit tokenizer

POST /tokenizers

The one call that matters. Everything downstream inherits the quality of this fit.

Item content is encoded per modality and residually quantized into levels codebooks. Level 1 places the item in a coarse semantic region; each subsequent level refines within it. See Semantic IDs for the reasoning behind the defaults.

curl

curl https://api.jeantechnologies.com/v1/tokenizers \
  -H "Authorization: Bearer $JEAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "catalog-v1",
    "catalog": { "uri": "s3://acme/catalog.jsonl" },
    "modalities": ["text", "image"],
    "codebook": { "levels": 4, "resolution": "multi", "base_size": 2048 },
    "collaborative_signal": {
      "mode": "co_occurrence_contrastive",
      "interactions": { "uri": "s3://acme/sequences.jsonl" },
      "weight": 0.3
    },
    "progressive_masking": true
  }'

Python

job = requests.post(
    "https://api.jeantechnologies.com/v1/tokenizers",
    headers={"Authorization": f"Bearer {os.environ['JEAN_API_KEY']}"},
    json={
        "name": "catalog-v1",
        "catalog": {"uri": "s3://acme/catalog.jsonl"},
        "codebook": {"levels": 4, "resolution": "multi", "base_size": 2048},
        "collaborative_signal": {
            "mode": "co_occurrence_contrastive",
            "interactions": {"uri": "s3://acme/sequences.jsonl"},
        },
    },
).json()

TypeScript

const res = await fetch("https://api.jeantechnologies.com/v1/tokenizers", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.JEAN_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "catalog-v1",
    catalog: { uri: "s3://acme/catalog.jsonl" },
    codebook: { levels: 4, resolution: "multi", base_size: 2048 },
    collaborative_signal: {
      mode: "co_occurrence_contrastive",
      interactions: { uri: "s3://acme/sequences.jsonl" },
    },
  }),
});
const tokenizer = await res.json();

Body

Field	Type	Default	Notes
name	string	required
catalog	object	required	{ uri, format }. uri is an s3:// or gs:// path, or an https URL we have read access to. format is jsonl or parquet.
modalities	string[]	["text","image"]	Each modality gets its own frozen encoder; outputs are concatenated before quantization.
codebook	object	see below	Residual quantization layout.
collaborative_signal	object	see below	How behavioral signal shapes the quantizer.
progressive_masking	boolean	true	Randomly truncate to the first r levels during training so each level is meaningful on its own. This is what makes prefix-constrained beam search prune a coherent region.

codebook

Field	Type	Default	Notes
levels	integer	4	1 to 8. Every semantic ID is this many codes long.
resolution	multi | uniform	multi	multi halves cardinality at each level.
base_size	integer	2048	Cardinality of the level-1 codebook.

collaborative_signal

Field	Type	Default	Notes
mode	co_occurrence_contrastive | none	co_occurrence_contrastive	Applies co-occurrence as a contrastive loss on the quantizer while the code stays a function of content.
interactions	object		{ uri, format }, same shape as catalog.
weight	number	0.3	Weight of the contrastive term relative to content reconstruction.

Input format

catalog is JSONL or Parquet, one item per record:

{"item_id": "sku_771", "title": "Merino crew socks", "description": "Mid-weight, charcoal.", "image_url": "https://acme.com/771.jpg", "attributes": {"brand": "Acme"}}

collaborative_signal.interactions is one user sequence per record, oldest first:

{"user_id": "u_18", "items": ["sku_310", "sku_884", "sku_771"], "timestamps": [1735689600, 1736294400, 1736899200]}

Choosing a codebook

resolution: "multi" sizes each level as base_size / 2^(level-1). With the defaults:

Level	Size	Encodes
1	2048	Coarse semantic region
2	1024	Sub-region
3	512	Fine distinction
4	256	Residual detail

Total addressable space is the product of the level sizes, so the defaults cover roughly 275 billion distinct IDs. The binding constraint is almost never address space. It is whether level 1 has enough distinct semantic regions in your catalog to fill 2048 codes.

tip

Rule of thumb: start with base_size near the number of genuinely distinct categories in your catalog, rounded up to a power of two. Check codebook_utilization on the fitted tokenizer and adjust. Utilization below about 0.5 at level 1 means you oversized it.

warning

resolution: "uniform" is available for parity with older RQ-VAE setups but is not recommended. By level 4 there is little residual entropy left to encode, and a full-width codebook there mostly sits unused.

Response

Fitting is asynchronous. The call returns 202 immediately:

{
  "tokenizer_id": "tok_9k2m",
  "name": "catalog-v1",
  "status": "queued",
  "codebook": { "levels": 4, "resolution": "multi", "base_size": 2048 },
  "created_at": "2026-08-31T18:04:11Z",
  "ready_at": null
}

Poll Get tokenizer until status is ready.