libfreemkv/docs/vocab.md

src/labels/vocab.rs — shared label vocabulary

Labels come from BD-J authoring tool files (bluray_project.bin, playlists.xml, menu_base.prop, .class string pools, etc.) — NOT from BD spec fields. This module is the central, regression-tested source of truth for:

• Codec brand name aliases (MLP → TrueHD).
• English / multi-word language name → ISO 639-2 code.
• English text → LabelPurpose (Commentary / Descriptive / etc.).
• English text → LabelQualifier (SDH / Forced / Descriptive Service).

Not for BD spec STN codec IDs; those decode in mpls.rs separately.

Rules of engagement

1. Only map values we are 100% certain about — published codec names, well-known ISO 639-2 mappings, vendor-documented purpose keywords.
2. Unknown codes / unrecognized phrases pass through raw or return None. We never guess.
3. Matching is case-insensitive and word-boundary-aware where relevant (so "Commenter" doesn't match "commentary"). Anchoring on whole tokens is the responsibility of this module — callers pass raw text, we handle it.

lang() — free-form language name → ISO 639-2 + variant

Handles both bare English names ("English", "Spanish") and the multi-word vendor variants seen in the corpus ("Brazilian Portuguese", "Castilian Spanish", "Canadian French"). Match is case-insensitive.

Compound phrases are scanned BEFORE bare names, so "Brazilian Portuguese" returns LangInfo { code: "por", variant: "Brazilian" } rather than being consumed by the bare "Portuguese" entry. Within COMPOUND_LANGS the scan is positional (first contains hit wins), so that table MUST be maintained longest-first — a longer phrase must precede any shorter phrase it contains (e.g. "latin american spanish" before "latin spanish").

Bare-name matches return variant: "". Returns None for unrecognized input — callers decide whether to fall back to MPLS spec codes, pass through raw, or drop the stream. Never guesses.

Why the variant: returning only the ISO code would silently drop regional dialect info — "Brazilian Portuguese 5.1" would become language="por", variant="" and the UI would display plain "Portuguese" even though the disc explicitly labeled the stream Brazilian. Returning the variant lets callers populate StreamLabel::variant with the dialect.

iso639_1_to_iso639_2() — the whole ISO 639-1 set

Covers the WHOLE of ISO 639-1, unlike menu_lang, whose table only spans the languages that show up in Blu-ray menu-graphic filenames. Callers that convert a spec field — a DVD IFO attribute block, say — need the whole set: narrowing it to the menu vocabulary would fold every other language onto one value and make a disc's tracks indistinguishable from each other.

ISO_639_1_TO_2 pairs every ISO 639-1 two-letter code with its ISO 639-2/T (terminological) code — the variant the rest of this crate uses (deu not ger, fra not fre, zho not chi, plus ces, nld, ell, ron, slk, isl, eus, hrv) — so lang and menu_lang both normalize to it and the three tables cannot disagree (iso639_1_agrees_with_menu_lang pins that). For the 165 codes where 639-2/B and /T are identical this distinction does not arise; it only matters for the 20-odd languages with a distinct bibliographic code.

ISO_639_1_DEPRECATED covers the three two-letter codes ISO 639-1 has since withdrawn, mapped to their replacements. DVD-Video froze its language list on the 1988 edition, so discs authored to the spec carry these spellings and no other table sees them: iw Hebrew (now he), in Indonesian (now id), ji Yiddish (now yi).

purpose() / qualifier() — English text → enum

purpose() recognized keywords (case-insensitive; single-word keywords are word-boundary matched, multi-word phrases are substring matched):

• "commentary", "director's commentary" → Commentary
• "descriptive", "description", "audio description", "described" → Descriptive
• "score", "music only" → Score
• "ime" (alternate music for closing themes etc.) → Ime
• anything else → Normal

Word-boundary matching means "Commentary track" matches but "Commenter Pro audio" does not. Multi-word phrases like "audio description" and "music only" are matched as plain substrings.

qualifier() recognized keywords (case-insensitive, word-boundary matched):

• "sdh", "captions" → Sdh
• "forced", "forced narrative" → Forced
• "rnib", "descriptive service" → DescriptiveService
• anything else → None

SDH (Subtitles for the Deaf and Hard of hearing) wins over Forced when both keywords are present, because an SDH track is its own stream regardless of whether the player flags it as "forced".

has_word() — word-boundary primitive

The load-bearing primitive for lang / purpose / qualifier: bare-token matchers MUST use it, otherwise we match "english" inside "englishman" and "sdh" inside "lambdash". The existing parsers used .contains() and got lucky on the corpus; vocab guarantees the boundary.

Char-aware, not byte-level: an accented/CJK char adjacent to the match is alphanumeric and so NOT a boundary, preventing false positives like "sdh" inside "cafésch". Needles are ASCII, so byte offsets align with char bounds.

Test rationale (mutation-testing notes)

• purpose_descriptive_service_substring_without_word_boundary: the multi-word-compound fast path in purpose() ORs two independent phrase checks ("audio description" / "descriptive service"). Each phrase, when it appears as a word-bounded match, is independently caught by the has_word fallback further down — so the OR only matters when a phrase appears as a substring inside a larger word (no boundary), which .contains() still catches but has_word() would reject. Mutation: replace || with && at the compound check → since "audio description" is absent in the test input, the AND fails, the fast path doesn't fire, and the fallback has_word("descriptive") also fails (no word boundary before "descriptive" in "nondescriptive"), so purpose() wrongly returns Normal instead of Descriptive.

• menu_lang_covers_every_table_entry: menu_lang() maps every authoring-filename token in its table (ISO-639-2/B and /T spellings, plus ISO-639-1) to the canonical /T code used by the rest of the pipeline. Exhaustive per-arm check: deleting any single match arm makes that arm's tokens return None instead of the documented code.

• iso639_1_table_is_complete_and_well_formed: structural invariants of ISO_639_1_TO_2 — it must hold the complete ISO 639-1 set (184 codes), every key a distinct pair of lowercase letters and every value three lowercase letters. A typo'd or duplicated row fails here rather than silently mislabelling a track.

• iso639_1_agrees_with_menu_lang: the two tables must not disagree. Every two-letter token menu_lang accepts has to yield the same ISO 639-2/T code through iso639_1_to_iso639_2, so a DVD-sourced language and a Blu-ray menu-label language for the same tongue never produce different Language elements.