Get tokenizer

GET /tokenizers/{tokenizer_id}

Poll this after fitting. It is also where you find out whether the fit was any good.

curl https://api.jeantechnologies.com/v1/tokenizers/tok_9k2m \
  -H "Authorization: Bearer $JEAN_API_KEY"

{
  "tokenizer_id": "tok_9k2m",
  "name": "catalog-v1",
  "status": "ready",
  "codebook": { "levels": 4, "resolution": "multi", "base_size": 2048 },
  "items_tokenized": 1840221,
  "codebook_utilization": [0.97, 0.91, 0.78, 0.61],
  "collision_rate": 0.004,
  "created_at": "2026-08-31T18:04:11Z",
  "ready_at": "2026-08-31T19:22:47Z"
}

status is one of queued, fitting, ready, or failed.

Reading the diagnostics

codebook_utilization

Fraction of each level's codebook in use, ordered by level. Healthy fits are high at level 1 and taper with depth.

Low utilization at level 1 means the codebook is wider than your catalog is diverse. Reduce base_size and refit.

Low utilization at the deepest level is normal and not worth chasing. There is little residual entropy left down there by design.

collision_rate

Fraction of items that quantized to identical codes and needed a disambiguating suffix. Suffixed items are still uniquely addressable, but they are effectively invisible to the semantics of the code, so a high rate erodes the whole premise.

Under 0.01 is fine. Above roughly 0.05, add a level or widen base_size.

status: failed

Most common causes are an unreadable catalog.uri, item records missing both title and description, or an interactions file whose item_id values do not join to the catalog. The error body names which.

note

Fit time scales with catalog size. Roughly 30 to 90 minutes for a few million items, longer when modalities includes image.