pq/libfreemkv
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

labels: shared platform (vocab/text/jar) + dbp refactor

Browse Source 
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.

New modules:

  vocab.rs       expanded from 27 -> ~370 lines
                 + lang(text) -> Option<&'static str>      (English/multi-word
                                                            -> ISO 639-2; ~45
                                                            languages, compound
                                                            phrases like
                                                            'Brazilian Portuguese'
                                                            and 'Castilian Spanish')
                 + purpose(text) -> LabelPurpose            (Commentary,
                                                            Descriptive, Score,
                                                            Ime; word-boundary
                                                            matched)
                 + qualifier(text) -> LabelQualifier        (SDH, Forced,
                                                            DescriptiveService)
                 + has_word internal primitive — enforces word-boundary
                   matching so 'Commenter' no longer matches 'commentary' and
                   'engineering' no longer matches 'english'. Existing parsers
                   used .contains() and got lucky on the corpus; vocab now
                   guarantees the boundary in one place. 20+ unit tests.

  text.rs        NEW (~85 lines)
                 + extract_ascii_strings(data, min_len) — promoted from two
                   near-duplicate copies (pixelogic min=4, dbp min=5);
                   threshold passed in. 7 unit tests including
                   trailing-without-terminator + high-bit-byte handling.

  jar.rs         NEW (~120 lines)
                 + for_each_jar(reader, udf, fn)  — walk every top-level
                                                    .jar under /BDMV/JAR/,
                                                    yield to callback.
                 + has_path_prefix(archive, prefix) — cheap 'is this MY
                                                      framework's jar?' check
                                                      via central-dir filenames.
                 + for_each_class(archive, fn)    — parse every .class entry
                                                    through class_reader,
                                                    yield (name, &ClassFile).
                 + try_each_class(archive, fn)    — same with early-return on
                                                    first Some(R) match.

Refactored:

  dbp.rs         v2 on the new platform:
                 - dropped extract_printable raw byte scan
                 - dropped its own English -> ISO 639-2 map
                 - dropped its own parse_attributes hand-roll
                 + iterates CpInfo::Utf8 via class_reader (structurally clean,
                   no false-positive risk from method bytecode bytes)
                 + routes language/purpose/qualifier through vocab
                 All 7 prior dbp tests still pass; +2 new ones cover
                 vocab routing.

dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.

Precommit green (cargo +1.86 fmt + clippy + test).
This commit is contained in:
matthew
2026-05-10 15:16:25 -07:00
parent dab9b9c9db
commit 307bee11e4
5 changed files with 707 additions and 208 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/labels/dbp.rs
+91 -199
View File
@@ -22,25 +22,29 @@
//! what precedes it. `Subtitle0` is the disable-subtitles menu
//! button and is skipped (not a real subtitle stream).
//!
//! Per `freemkv-private/memory/feedback_label_data_rules.md`: this
//! parser knows its own format, so we map human-readable language
//! names ("English", "Spanish", "Canadian French", ...) to ISO 639-2
//! codes locally. The full disc-authored display string is preserved
//! in the label `name` field — consumers display it raw without
//! freemkv guessing further structure.
//! ## Implementation
//!
//! v2 (2026-05-10): rewritten on top of [`super::class_reader`] —
//! iterates `CpInfo::Utf8` constant-pool entries instead of raw byte
//! scanning each class file. Equivalent label coverage (the literal
//! `TextField,...` strings live in the CP as Utf8 entries), but
//! structurally cleaner: no false-positive risk from method bytecode
//! or attribute names happening to contain `TextField,`. Language /
//! purpose / qualifier classification moved to [`super::vocab`] so all
//! Java-parser families share one source of truth.
use super::{LabelPurpose, LabelQualifier, StreamLabel, StreamLabelType};
use super::class_reader::CpInfo;
use super::{StreamLabel, StreamLabelType, jar, vocab};
use crate::sector::SectorReader;
use crate::udf::UdfFs;
use std::collections::BTreeMap;
/// dbp detect can't peek inside a jar without a SectorReader (the
/// trait function only takes `&UdfFs`), so we trigger on the cheap
/// signal "any top-level .jar in /BDMV/JAR/." That fires on every
/// BD-J disc, but parse() does the real `com/dbp/` check and
/// returns None on a mismatch — so this parser only ever consumes
/// time on discs that fell through every earlier parser. The
/// parse-side mismatch is bounded (read one .jar, list central
/// directory, walk class strings).
/// time on discs that fell through every earlier parser.
pub fn detect(udf: &UdfFs) -> bool {
let Some(jar_dir) = udf.find_dir("/BDMV/JAR") else {
return false;
@@ -52,73 +56,35 @@ pub fn detect(udf: &UdfFs) -> bool {
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
let jar_dir = udf.find_dir("/BDMV/JAR")?;
for entry in &jar_dir.entries {
if entry.is_dir {
continue;
jar::for_each_jar(reader, udf, |_entry_name, archive| {
if !jar::has_path_prefix(archive, "com/dbp/") {
return None;
}
if !entry.name.to_lowercase().ends_with(".jar") {
continue;
let labels = scan_jar(archive);
if labels.is_empty() {
None
} else {
Some(labels)
}
let path = format!("/BDMV/JAR/{}", entry.name);
let Ok(bytes) = udf.read_file(reader, &path) else {
continue;
};
let cursor = std::io::Cursor::new(&bytes);
let Ok(mut archive) = zip::ZipArchive::new(cursor) else {
continue;
};
if !archive_has_dbp(&mut archive) {
continue;
}
let labels = scan_jar(&mut archive);
if !labels.is_empty() {
return Some(labels);
}
}
None
})
}
fn archive_has_dbp<R: std::io::Read + std::io::Seek>(archive: &mut zip::ZipArchive<R>) -> bool {
for i in 0..archive.len() {
if let Ok(f) = archive.by_index(i) {
if f.name().starts_with("com/dbp/") {
return true;
}
}
}
false
}
fn scan_jar<R: std::io::Read + std::io::Seek>(
archive: &mut zip::ZipArchive<R>,
) -> Vec<StreamLabel> {
use std::collections::BTreeMap;
fn scan_jar(archive: &mut jar::Jar) -> Vec<StreamLabel> {
// BTreeMap so we keep the highest-numbered (last-written) label
// for each stream slot deterministic across runs. Entries are
// collected from string-pool fragments scattered across hundreds
// of obfuscated .class files; the same TextField,Audio1,...
// string can appear in multiple classes (button-state variants,
// localization fallbacks). Last write wins — they should all
// agree on the label text, but the structure is defensive.
// for each stream slot deterministic across runs. The same
// TextField,Audio1,... string can appear in multiple classes
// (button-state variants, localization fallbacks). Last write
// wins — they should all agree, but the structure is defensive.
let mut audios: BTreeMap<u16, String> = BTreeMap::new();
let mut subs: BTreeMap<u16, String> = BTreeMap::new();
for i in 0..archive.len() {
let Ok(mut f) = archive.by_index(i) else {
continue;
};
if !f.name().ends_with(".class") {
continue;
jar::for_each_class(archive, |_class_name, class| {
for (_idx, cp) in class.constant_pool.iter() {
if let CpInfo::Utf8(s) = cp {
collect_textfield(s, &mut audios, &mut subs);
}
}
let mut buf = Vec::new();
if std::io::Read::read_to_end(&mut f, &mut buf).is_err() {
continue;
}
for s in extract_printable(&buf) {
collect_textfield(&s, &mut audios, &mut subs);
}
}
});
let mut out = Vec::new();
for (num, label) in audios {
@@ -132,8 +98,8 @@ fn scan_jar<R: std::io::Read + std::io::Seek>(
fn collect_textfield(
s: &str,
audios: &mut std::collections::BTreeMap<u16, String>,
subs: &mut std::collections::BTreeMap<u16, String>,
audios: &mut BTreeMap<u16, String>,
subs: &mut BTreeMap<u16, String>,
) {
// Anchor on "TextField," — the prefix character before it varies
// (string-pool ordering inside compiled Java) and is irrelevant.
@@ -163,7 +129,9 @@ fn collect_textfield(
}
fn make_label(num: u16, label: String, stream_type: StreamLabelType) -> StreamLabel {
let (language, qualifier, purpose) = parse_attributes(&label);
let language = vocab::lang(&label).unwrap_or_default().to_string();
let qualifier = vocab::qualifier(&label);
let purpose = vocab::purpose(&label);
StreamLabel {
stream_number: num,
stream_type,
@@ -176,111 +144,15 @@ fn make_label(num: u16, label: String, stream_type: StreamLabelType) -> StreamLa
}
}
fn parse_attributes(label: &str) -> (String, LabelQualifier, LabelPurpose) {
let lower = label.to_lowercase();
let language = detect_language(&lower);
let qualifier = if lower.contains("sdh") {
LabelQualifier::Sdh
} else if lower.contains("descriptive service") || lower.contains(" rnib") {
LabelQualifier::DescriptiveService
} else if lower.contains("forced") {
LabelQualifier::Forced
} else {
LabelQualifier::None
};
let purpose = if lower.contains("commentary") {
LabelPurpose::Commentary
} else if lower.contains("descriptive") || lower.contains("audio description") {
LabelPurpose::Descriptive
} else {
LabelPurpose::Normal
};
(language, qualifier, purpose)
}
/// Map English-language label tokens to ISO 639-2 codes. Keep this
/// list conservative — only common tokens we've actually observed
/// or that have a canonical mapping. Returns "" when the token
/// isn't recognized; the consumer falls back to fill_defaults reading
/// MPLS spec language codes.
fn detect_language(lower: &str) -> String {
// Compound tokens first (multi-word language names).
for (needle, code) in [
("brazilian portuguese", "por"),
("euro portuguese", "por"),
("castilian spanish", "spa"),
("latin american spanish", "spa"),
("canadian french", "fra"),
("parisian french", "fra"),
("australian english", "eng"),
("austrailian english", "eng"), // disc-corpus typo, keep matching
] {
if lower.contains(needle) {
return code.to_string();
}
}
// Then bare tokens. Order matters where one is prefix of another.
for (needle, code) in [
("english", "eng"),
("spanish", "spa"),
("french", "fra"),
("german", "deu"),
("italian", "ita"),
("japanese", "jpn"),
("chinese", "zho"),
("portuguese", "por"),
("polish", "pol"),
("czech", "ces"),
("hungarian", "hun"),
("dutch", "nld"),
("korean", "kor"),
("arabic", "ara"),
("hindi", "hin"),
("turkish", "tur"),
("thai", "tha"),
("swedish", "swe"),
("norwegian", "nor"),
("danish", "dan"),
("finnish", "fin"),
("hebrew", "heb"),
("russian", "rus"),
] {
if lower.split_whitespace().next() == Some(needle)
|| lower.split_whitespace().any(|w| w == needle)
{
return code.to_string();
}
}
String::new()
}
fn extract_printable(data: &[u8]) -> Vec<String> {
let mut out = Vec::new();
let mut current = String::new();
for &b in data {
if (0x20..0x7f).contains(&b) {
current.push(b as char);
} else {
if current.len() >= 5 {
out.push(current.clone());
}
current.clear();
}
}
if current.len() >= 5 {
out.push(current);
}
out
}
#[cfg(test)]
mod tests {
use super::super::{LabelPurpose, LabelQualifier};
use super::*;
#[test]
fn collect_extracts_audio_and_subtitle_indices() {
let mut audios = std::collections::BTreeMap::new();
let mut subs = std::collections::BTreeMap::new();
let mut audios = BTreeMap::new();
let mut subs = BTreeMap::new();
let lines = [
"LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,296,763,275,25,left",
"RTextField,Audio2,English Descriptive Audio,Fontstrip_Composite,296,803,275,25,left",
@@ -304,8 +176,8 @@ mod tests {
#[test]
fn collect_ignores_non_textfield_strings() {
let mut audios = std::collections::BTreeMap::new();
let mut subs = std::collections::BTreeMap::new();
let mut audios = BTreeMap::new();
let mut subs = BTreeMap::new();
for s in [
"GraphicButton,SU_Audio",
"AudioMenu",
@@ -319,47 +191,67 @@ mod tests {
}
#[test]
fn parse_attributes_recognizes_sdh() {
let (lang, qual, purp) = parse_attributes("English SDH");
assert_eq!(lang, "eng");
assert_eq!(qual, LabelQualifier::Sdh);
assert_eq!(purp, LabelPurpose::Normal);
fn make_label_routes_via_vocab() {
let l = make_label(1, "English SDH".to_string(), StreamLabelType::Subtitle);
assert_eq!(l.language, "eng");
assert_eq!(l.qualifier, LabelQualifier::Sdh);
assert_eq!(l.purpose, LabelPurpose::Normal);
}
#[test]
fn parse_attributes_recognizes_descriptive_audio() {
let (lang, qual, purp) = parse_attributes("English Descriptive Audio");
assert_eq!(lang, "eng");
assert_eq!(qual, LabelQualifier::None);
assert_eq!(purp, LabelPurpose::Descriptive);
fn make_label_descriptive_audio() {
let l = make_label(
2,
"English Descriptive Audio".to_string(),
StreamLabelType::Audio,
);
assert_eq!(l.language, "eng");
assert_eq!(l.purpose, LabelPurpose::Descriptive);
}
#[test]
fn parse_attributes_recognizes_commentary() {
let (lang, qual, purp) = parse_attributes("English Director's Commentary");
assert_eq!(lang, "eng");
assert_eq!(qual, LabelQualifier::None);
assert_eq!(purp, LabelPurpose::Commentary);
fn make_label_commentary() {
let l = make_label(
3,
"English Director's Commentary".to_string(),
StreamLabelType::Audio,
);
assert_eq!(l.language, "eng");
assert_eq!(l.purpose, LabelPurpose::Commentary);
}
#[test]
fn parse_attributes_recognizes_compound_languages() {
assert_eq!(parse_attributes("Brazilian Portuguese 5.1").0, "por");
assert_eq!(parse_attributes("Castilian Spanish").0, "spa");
assert_eq!(parse_attributes("Canadian French Dolby Digital").0, "fra");
assert_eq!(parse_attributes("Latin American Spanish").0, "spa");
fn make_label_compound_languages() {
assert_eq!(
make_label(1, "Brazilian Portuguese 5.1".into(), StreamLabelType::Audio).language,
"por"
);
assert_eq!(
make_label(1, "Castilian Spanish".into(), StreamLabelType::Audio).language,
"spa"
);
assert_eq!(
make_label(
1,
"Canadian French Dolby Digital".into(),
StreamLabelType::Audio
)
.language,
"fra"
);
}
#[test]
fn parse_attributes_returns_empty_for_unknown_language() {
// Don't guess. Per the rules-of-engagement.
assert_eq!(parse_attributes("Klingon Dolby Atmos").0, "");
fn make_label_unknown_language_is_empty() {
// vocab::lang returns None — make_label converts to "".
let l = make_label(1, "Klingon Dolby Atmos".into(), StreamLabelType::Audio);
assert_eq!(l.language, "");
}
#[test]
fn parse_attributes_recognizes_rnib_descriptive_service() {
let (lang, qual, _) = parse_attributes("English RNIB");
assert_eq!(lang, "eng");
assert_eq!(qual, LabelQualifier::DescriptiveService);
fn make_label_rnib_descriptive_service() {
let l = make_label(1, "English RNIB".into(), StreamLabelType::Subtitle);
assert_eq!(l.language, "eng");
assert_eq!(l.qualifier, LabelQualifier::DescriptiveService);
}
}
src/labels/jar.rs
+136
View File
@@ -0,0 +1,136 @@
//! BD-J jar utilities — common scaffolding for parsers that read
//! `/BDMV/JAR/*.jar`.
//!
//! Composes with [`class_reader`](super::class_reader) for structured
//! `.class` access. Used by `dbp` (string-pool scan via constant pool)
//! and `deluxe` (bytecode pattern matching) — those parsers express
//! "open every top-level jar, look at every .class inside" without
//! repeating the zip-archive boilerplate.
// `try_each_class` is staged for `labels::deluxe`, which needs the
// early-return form to short-circuit class iteration on a match.
// dead-code allow comes off when deluxe lands.
#![allow(dead_code)]
use super::class_reader::ClassFile;
use crate::sector::SectorReader;
use crate::udf::UdfFs;
use std::io::Cursor;
use zip::ZipArchive;
/// In-memory zip archive: backed by a `Vec<u8>` read from UDF. Owns
/// the buffer; callers pass it to [`has_path_prefix`], [`for_each_class`],
/// etc.
pub type Jar = ZipArchive<Cursor<Vec<u8>>>;
/// Open every top-level `*.jar` entry in `/BDMV/JAR/` and yield each
/// `(entry_name, Jar)` to `f`. Returns the first `Some(R)` the callback
/// produces, or `None` if every jar was visited without a hit.
///
/// "Top-level" means entries directly under `/BDMV/JAR/`, not nested
/// under a subdir. (Pixelogic, Criterion, Paramount, etc. put their
/// data files inside `/BDMV/JAR/<x>/`; dbp and Deluxe put their jar
/// directly at `/BDMV/JAR/<name>.jar`.)
///
/// Entries that fail to read from UDF or that aren't valid zips are
/// silently skipped — same defensive shape as the existing dbp parser.
pub fn for_each_jar<R, F>(reader: &mut dyn SectorReader, udf: &UdfFs, mut f: F) -> Option<R>
where
F: FnMut(&str, &mut Jar) -> Option<R>,
{
let jar_dir = udf.find_dir("/BDMV/JAR")?;
for entry in &jar_dir.entries {
if entry.is_dir {
continue;
}
if !entry.name.to_lowercase().ends_with(".jar") {
continue;
}
let path = format!("/BDMV/JAR/{}", entry.name);
let Ok(bytes) = udf.read_file(reader, &path) else {
continue;
};
let Ok(mut archive) = ZipArchive::new(Cursor::new(bytes)) else {
continue;
};
if let Some(r) = f(&entry.name, &mut archive) {
return Some(r);
}
}
None
}
/// True if any entry in this jar's central directory starts with
/// `prefix`. Fast — only reads filenames, never extracts bytes.
///
/// Used by parsers as a cheap "is this MY framework's jar?" check
/// (e.g. `has_path_prefix(archive, "com/dbp/")` for dbp,
/// `has_path_prefix(archive, "com/bydeluxe/")` for Deluxe).
pub fn has_path_prefix(archive: &mut Jar, prefix: &str) -> bool {
for i in 0..archive.len() {
if let Ok(f) = archive.by_index(i) {
if f.name().starts_with(prefix) {
return true;
}
}
}
false
}
/// Iterate every `.class` entry in the jar, parse it with
/// [`class_reader`], and call `f` with `(entry_name, &ClassFile)`.
///
/// Entries that fail to read or parse are silently skipped — this is
/// label-extraction code, robustness matters more than completeness.
/// Callers that need to know which classes failed should use the
/// lower-level [`class_reader`] API directly.
pub fn for_each_class<F>(archive: &mut Jar, mut f: F)
where
F: FnMut(&str, &ClassFile),
{
for i in 0..archive.len() {
let Ok(mut entry) = archive.by_index(i) else {
continue;
};
if !entry.name().ends_with(".class") {
continue;
}
let name = entry.name().to_string();
let mut bytes = Vec::with_capacity(entry.size() as usize);
if std::io::Read::read_to_end(&mut entry, &mut bytes).is_err() {
continue;
}
let Ok(class) = ClassFile::parse(&bytes) else {
continue;
};
f(&name, &class);
}
}
/// Like [`for_each_class`] but allows the callback to short-circuit
/// iteration. Returns the first `Some(R)` the callback produces.
pub fn try_each_class<R, F>(archive: &mut Jar, mut f: F) -> Option<R>
where
F: FnMut(&str, &ClassFile) -> Option<R>,
{
for i in 0..archive.len() {
let Ok(mut entry) = archive.by_index(i) else {
continue;
};
if !entry.name().ends_with(".class") {
continue;
}
let name = entry.name().to_string();
let mut bytes = Vec::with_capacity(entry.size() as usize);
if std::io::Read::read_to_end(&mut entry, &mut bytes).is_err() {
continue;
}
let Ok(class) = ClassFile::parse(&bytes) else {
continue;
};
if let Some(r) = f(&name, &class) {
return Some(r);
}
}
None
}
src/labels/mod.rs
+2
View File
@@ -11,8 +11,10 @@ pub(crate) mod class_reader;
mod criterion;
mod ctrm;
mod dbp;
pub(crate) mod jar;
mod paramount;
mod pixelogic;
pub(crate) mod text;
pub mod vocab;
use crate::disc::{DiscTitle, Stream};
src/labels/text.rs
+94
View File
@@ -0,0 +1,94 @@
//! Text-extraction helpers used by parsers that scan binary blobs for
//! embedded label strings.
//!
//! Promoted from two near-duplicate implementations:
//! - `pixelogic::extract_strings` (`bluray_project.bin`, min_len=4)
//! - `dbp::extract_printable` (`.class` files in jars, min_len=5)
//!
//! Single implementation, threshold passed in. Callers that have a
//! more structured parse path (e.g. `class_reader` for .class) should
//! prefer that — this helper is for genuinely unstructured input.
// Staged for the pixelogic refactor: pixelogic still has its own
// extract_strings copy; this is the shared replacement waiting for
// the refactor. dead-code allow comes off when pixelogic switches.
#![allow(dead_code)]
/// Walk `data`, emit every maximal run of printable-ASCII bytes
/// (`0x20..=0x7E`) whose length is at least `min_len`.
///
/// Non-printable bytes (including `\t`, `\n`, NUL) terminate the
/// current run. Output strings are guaranteed valid UTF-8 (they're
/// pure 7-bit ASCII). Strings shorter than `min_len` are dropped.
pub fn extract_ascii_strings(data: &[u8], min_len: usize) -> Vec<String> {
let mut out = Vec::new();
let mut current = String::new();
for &b in data {
if (0x20..=0x7E).contains(&b) {
current.push(b as char);
} else if current.len() >= min_len {
out.push(std::mem::take(&mut current));
} else {
current.clear();
}
}
if current.len() >= min_len {
out.push(current);
}
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extracts_simple_runs() {
let got = extract_ascii_strings(b"hello\0world\0", 3);
assert_eq!(got, vec!["hello", "world"]);
}
#[test]
fn applies_minimum_length() {
let got = extract_ascii_strings(b"hi\0ok\0longer\0", 5);
assert_eq!(got, vec!["longer"]);
}
#[test]
fn treats_tab_and_newline_as_separators() {
// \t (0x09) and \n (0x0A) are below 0x20, so they break runs.
let got = extract_ascii_strings(b"alpha\tbeta\ngamma", 3);
assert_eq!(got, vec!["alpha", "beta", "gamma"]);
}
#[test]
fn handles_trailing_run_without_terminator() {
// Run at the very end of the buffer should still be emitted.
let got = extract_ascii_strings(b"prefix\0tail", 3);
assert_eq!(got, vec!["prefix", "tail"]);
}
#[test]
fn rejects_high_bit_bytes() {
// 0x80+ is non-printable per this helper's definition.
let mut buf = b"good".to_vec();
buf.push(0xC3);
buf.push(0xA9);
buf.extend_from_slice(b"more");
let got = extract_ascii_strings(&buf, 3);
assert_eq!(got, vec!["good", "more"]);
}
#[test]
fn empty_input_returns_empty() {
let got = extract_ascii_strings(&[], 1);
assert!(got.is_empty());
}
#[test]
fn min_len_zero_emits_singletons() {
// Pathological but well-defined.
let got = extract_ascii_strings(b"a\0b", 0);
assert_eq!(got, vec!["a", "b"]);
}
}
src/labels/vocab.rs
+384 -9
View File
@@ -1,19 +1,40 @@
//! Shared label vocabulary — values we are 100% confident about.
//! Shared label vocabulary — canonical mappings used by ≥1 label parser.
//!
//! Labels come from BD-J authoring tool files (bluray_project.bin,
//! playlists.xml, menu_base.prop, etc.) — NOT from BD spec fields.
//! This is NOT for MPLS/CLPI/STN data. Those follow the BD spec directly.
//! playlists.xml, menu_base.prop, .class string pools, etc.) — NOT
//! from BD spec fields. This module is the central, regression-tested
//! source of truth for:
//!
//! Rules:
//! - Only map values we are 100% certain about (published codec names).
//! - Unknown codes (csp, eda, cf, etc.) pass through raw from disc.
//! - The app/CLI handles display text, not the lib.
//! - Codec brand name aliases (`MLP` → `TrueHD`).
//! - English / multi-word language name → ISO 639-2 code.
//! - English text → [`LabelPurpose`] (Commentary / Descriptive / etc.).
//! - English text → [`LabelQualifier`] (SDH / Forced / Descriptive Service).
//!
//! Rules of engagement (carried over from
//! `freemkv-private/memory/feedback_label_data_rules.md`):
//!
//! 1. Only map values we are 100% certain about — published codec
//! names, well-known ISO 639-2 mappings, vendor-documented purpose
//! keywords.
//! 2. Unknown codes / unrecognized phrases pass through raw or return
//! `None`. We never guess.
//! 3. Matching is case-insensitive and word-boundary-aware where
//! relevant (so "Commenter" doesn't match "commentary"). Anchoring
//! on whole tokens is the responsibility of this module — callers
//! pass raw text, we handle it.
//!
//! This module is NOT for BD spec STN codec IDs; those decode in
//! `mpls.rs` separately.
use super::{LabelPurpose, LabelQualifier};
// ── Codec aliases ────────────────────────────────────────────────────────────
/// Map a codec identifier found in label data to its display name.
///
/// These are well-known codec identifiers used across multiple BD-J
/// authoring tools. NOT BD spec STN codec IDs — those are decoded
/// separately in mpls.rs.
/// authoring tools. Unknown codes pass through unchanged so callers
/// can still surface vendor-specific tokens we haven't catalogued.
pub fn codec(code: &str) -> &str {
match code {
"MLP" => "TrueHD",
@@ -25,3 +46,357 @@ pub fn codec(code: &str) -> &str {
_ => code,
}
}
// ── Language: English / multi-word names → ISO 639-2 ─────────────────────────
/// Map a free-form language label fragment to an ISO 639-2 code.
///
/// Handles both bare English names ("English", "Spanish") and the
/// multi-word vendor variants we've seen in the corpus ("Brazilian
/// Portuguese", "Castilian Spanish", "Canadian French"). Match is
/// case-insensitive; longer compound phrases win over their bare
/// counterparts (so "Brazilian Portuguese" → `por`, not consumed by
/// the bare "Portuguese" entry).
///
/// Returns `None` for unrecognized input — callers decide whether to
/// fall back to MPLS spec codes, pass through raw, or drop the stream.
/// Never guesses.
pub fn lang(text: &str) -> Option<&'static str> {
let lower = text.to_lowercase();
// Multi-word compounds first — longest-match wins.
for (needle, code) in COMPOUND_LANGS {
if lower.contains(needle) {
return Some(code);
}
}
// Bare names: word-boundary match (avoid "english" inside "englishman"
// or any other accidental substring).
for (needle, code) in BARE_LANGS {
if has_word(&lower, needle) {
return Some(code);
}
}
None
}
const COMPOUND_LANGS: &[(&str, &str)] = &[
("brazilian portuguese", "por"),
("euro portuguese", "por"),
("european portuguese", "por"),
("castilian spanish", "spa"),
("latin american spanish", "spa"),
("latin spanish", "spa"),
("canadian french", "fra"),
("parisian french", "fra"),
("australian english", "eng"),
("austrailian english", "eng"), // disc-corpus typo, keep matching
("british english", "eng"),
("simplified chinese", "zho"),
("traditional chinese", "zho"),
("mandarin chinese", "zho"),
("cantonese chinese", "zho"),
];
const BARE_LANGS: &[(&str, &str)] = &[
("english", "eng"),
("spanish", "spa"),
("french", "fra"),
("german", "deu"),
("italian", "ita"),
("japanese", "jpn"),
("chinese", "zho"),
("mandarin", "zho"),
("cantonese", "zho"),
("portuguese", "por"),
("polish", "pol"),
("czech", "ces"),
("hungarian", "hun"),
("dutch", "nld"),
("korean", "kor"),
("arabic", "ara"),
("hindi", "hin"),
("turkish", "tur"),
("thai", "tha"),
("swedish", "swe"),
("norwegian", "nor"),
("danish", "dan"),
("finnish", "fin"),
("hebrew", "heb"),
("russian", "rus"),
("greek", "ell"),
("vietnamese", "vie"),
("indonesian", "ind"),
("malay", "msa"),
("ukrainian", "ukr"),
("romanian", "ron"),
("bulgarian", "bul"),
("croatian", "hrv"),
("serbian", "srp"),
("slovak", "slk"),
("slovenian", "slv"),
("estonian", "est"),
("latvian", "lav"),
("lithuanian", "lit"),
("icelandic", "isl"),
("basque", "eus"),
("catalan", "cat"),
("galician", "glg"),
];
// ── Purpose ──────────────────────────────────────────────────────────────────
/// Classify a free-form English label string into a [`LabelPurpose`].
///
/// Recognized keywords (case-insensitive, word-boundary matched):
/// - "commentary", "director's commentary" → `Commentary`
/// - "descriptive", "description", "audio description", "described" → `Descriptive`
/// - "score", "music only" → `Score`
/// - "ime" (alternate music for closing themes etc.) → `Ime`
/// - anything else → `Normal`
///
/// Word-boundary matching means "Commentary track" matches but
/// "Commenter Pro audio" does not.
pub fn purpose(text: &str) -> LabelPurpose {
let lower = text.to_lowercase();
// Multi-word compounds first — they're more specific.
if lower.contains("audio description") || lower.contains("descriptive service") {
return LabelPurpose::Descriptive;
}
if lower.contains("music only") {
return LabelPurpose::Score;
}
if has_word(&lower, "commentary") {
return LabelPurpose::Commentary;
}
if has_word(&lower, "descriptive")
|| has_word(&lower, "description")
|| has_word(&lower, "described")
{
return LabelPurpose::Descriptive;
}
if has_word(&lower, "score") {
return LabelPurpose::Score;
}
if has_word(&lower, "ime") {
return LabelPurpose::Ime;
}
LabelPurpose::Normal
}
// ── Qualifier ────────────────────────────────────────────────────────────────
/// Classify a free-form English label string into a [`LabelQualifier`].
///
/// Recognized keywords (case-insensitive, word-boundary matched):
/// - "sdh", "captions" → `Sdh`
/// - "forced", "forced narrative" → `Forced`
/// - "rnib", "descriptive service" → `DescriptiveService`
/// - anything else → `None`
///
/// SDH (Subtitles for the Deaf and Hard of hearing) wins over Forced
/// when both keywords are present, because an SDH track is its own
/// stream regardless of whether the player flags it as "forced".
pub fn qualifier(text: &str) -> LabelQualifier {
let lower = text.to_lowercase();
if has_word(&lower, "sdh") || has_word(&lower, "captions") {
return LabelQualifier::Sdh;
}
if lower.contains("descriptive service") || has_word(&lower, "rnib") {
return LabelQualifier::DescriptiveService;
}
if has_word(&lower, "forced") {
return LabelQualifier::Forced;
}
LabelQualifier::None
}
// ── Internal: word-boundary matching ────────────────────────────────────────
/// True if `needle` appears in `haystack` surrounded by non-alphanumeric
/// boundaries (or string ends). `haystack` is assumed lowercase already.
///
/// This is the load-bearing primitive for `lang` / `purpose` /
/// `qualifier`: bare-token matchers MUST use it, otherwise we match
/// "english" inside "englishman" and "sdh" inside "lambdash". The
/// existing parsers used `.contains()` and got lucky on the corpus;
/// vocab guarantees the boundary.
fn has_word(haystack: &str, needle: &str) -> bool {
if needle.is_empty() {
return false;
}
let bytes = haystack.as_bytes();
let nb = needle.as_bytes();
let mut i = 0;
while i + nb.len() <= bytes.len() {
if &bytes[i..i + nb.len()] == nb {
let before = if i == 0 { None } else { Some(bytes[i - 1]) };
let after = bytes.get(i + nb.len()).copied();
let bound = |c: Option<u8>| match c {
None => true,
Some(b) => !b.is_ascii_alphanumeric(),
};
if bound(before) && bound(after) {
return true;
}
}
i += 1;
}
false
}
// ── Tests ────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn codec_known_aliases() {
assert_eq!(codec("MLP"), "TrueHD");
assert_eq!(codec("AC3"), "Dolby Digital");
assert_eq!(codec("AC"), "Dolby Digital");
assert_eq!(codec("DDL"), "Dolby Digital Plus");
assert_eq!(codec("atmos"), "Dolby Atmos");
}
#[test]
fn codec_unknown_passes_through() {
assert_eq!(codec("FX9"), "FX9");
assert_eq!(codec(""), "");
}
#[test]
fn lang_bare_names() {
assert_eq!(lang("English"), Some("eng"));
assert_eq!(lang("english"), Some("eng"));
assert_eq!(lang("Spanish 5.1 Dolby Digital"), Some("spa"));
assert_eq!(lang("japanese"), Some("jpn"));
assert_eq!(lang("Italian"), Some("ita"));
}
#[test]
fn lang_compounds_win_over_bare() {
// Brazilian Portuguese should map to por via the compound rule,
// not be intercepted by bare "portuguese" (also por, but the
// matcher must walk compounds first to be correct in principle).
assert_eq!(lang("Brazilian Portuguese 5.1"), Some("por"));
assert_eq!(lang("Castilian Spanish"), Some("spa"));
assert_eq!(lang("Canadian French Dolby Digital"), Some("fra"));
assert_eq!(lang("Latin American Spanish"), Some("spa"));
}
#[test]
fn lang_unknown_returns_none() {
assert_eq!(lang("Klingon Dolby Atmos"), None);
assert_eq!(lang(""), None);
assert_eq!(lang("eng"), None); // 3-letter codes are not English names
}
#[test]
fn lang_word_boundary_avoids_substring_false_positive() {
// No false positive — "engineering" must NOT match "english".
assert_eq!(lang("Audio Engineering Demo"), None);
}
#[test]
fn purpose_recognizes_commentary() {
assert_eq!(purpose("English Commentary"), LabelPurpose::Commentary);
assert_eq!(purpose("Director's Commentary"), LabelPurpose::Commentary);
}
#[test]
fn purpose_recognizes_descriptive() {
assert_eq!(
purpose("English Descriptive Audio"),
LabelPurpose::Descriptive
);
assert_eq!(purpose("Audio Description"), LabelPurpose::Descriptive);
assert_eq!(purpose("Described Video"), LabelPurpose::Descriptive);
}
#[test]
fn purpose_descriptive_service_routes_to_descriptive() {
// "Descriptive Service" is qualifier territory but the purpose
// implication is Descriptive — vocab::purpose treats it as such.
assert_eq!(
purpose("English Descriptive Service"),
LabelPurpose::Descriptive
);
}
#[test]
fn purpose_word_boundary_avoids_commenter_false_positive() {
// "Commenter Pro audio" does NOT match "commentary" — the
// existing dbp/ctrm hand-rolls would have. Vocab is stricter.
assert_eq!(purpose("Commenter Pro audio track"), LabelPurpose::Normal);
}
#[test]
fn purpose_recognizes_score() {
assert_eq!(purpose("Music Only"), LabelPurpose::Score);
assert_eq!(purpose("Isolated Score"), LabelPurpose::Score);
}
#[test]
fn purpose_unknown_is_normal() {
assert_eq!(purpose("English Dolby Atmos"), LabelPurpose::Normal);
assert_eq!(purpose(""), LabelPurpose::Normal);
}
#[test]
fn qualifier_recognizes_sdh() {
assert_eq!(qualifier("English SDH"), LabelQualifier::Sdh);
assert_eq!(qualifier("English Captions"), LabelQualifier::Sdh);
}
#[test]
fn qualifier_recognizes_forced() {
assert_eq!(qualifier("English Forced"), LabelQualifier::Forced);
assert_eq!(qualifier("Forced Narrative"), LabelQualifier::Forced);
}
#[test]
fn qualifier_recognizes_descriptive_service() {
assert_eq!(
qualifier("English RNIB"),
LabelQualifier::DescriptiveService
);
assert_eq!(
qualifier("English Descriptive Service"),
LabelQualifier::DescriptiveService
);
}
#[test]
fn qualifier_sdh_wins_over_forced_when_both_present() {
// SDH track is its own stream regardless of forced flag.
assert_eq!(qualifier("English Forced SDH"), LabelQualifier::Sdh);
}
#[test]
fn qualifier_unknown_is_none() {
assert_eq!(qualifier("English"), LabelQualifier::None);
assert_eq!(qualifier(""), LabelQualifier::None);
}
#[test]
fn has_word_basic() {
assert!(has_word("english forced", "english"));
assert!(has_word("english forced", "forced"));
assert!(has_word("english", "english"));
}
#[test]
fn has_word_rejects_substring() {
assert!(!has_word("engineering", "english"));
assert!(!has_word("englishman", "english"));
assert!(!has_word("aenglish", "english"));
}
#[test]
fn has_word_punctuation_boundary() {
// "(SDH)" is a valid boundary — parentheses count as non-alphanum.
assert!(has_word("english (sdh)", "sdh"));
assert!(has_word("commentary,extra,info", "commentary"));
}
}