pq/libfreemkv
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

labels: shared platform (vocab/text/jar) + dbp refactor

Browse Source 
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.

New modules:

  vocab.rs       expanded from 27 -> ~370 lines
                 + lang(text) -> Option<&'static str>      (English/multi-word
                                                            -> ISO 639-2; ~45
                                                            languages, compound
                                                            phrases like
                                                            'Brazilian Portuguese'
                                                            and 'Castilian Spanish')
                 + purpose(text) -> LabelPurpose            (Commentary,
                                                            Descriptive, Score,
                                                            Ime; word-boundary
                                                            matched)
                 + qualifier(text) -> LabelQualifier        (SDH, Forced,
                                                            DescriptiveService)
                 + has_word internal primitive — enforces word-boundary
                   matching so 'Commenter' no longer matches 'commentary' and
                   'engineering' no longer matches 'english'. Existing parsers
                   used .contains() and got lucky on the corpus; vocab now
                   guarantees the boundary in one place. 20+ unit tests.

  text.rs        NEW (~85 lines)
                 + extract_ascii_strings(data, min_len) — promoted from two
                   near-duplicate copies (pixelogic min=4, dbp min=5);
                   threshold passed in. 7 unit tests including
                   trailing-without-terminator + high-bit-byte handling.

  jar.rs         NEW (~120 lines)
                 + for_each_jar(reader, udf, fn)  — walk every top-level
                                                    .jar under /BDMV/JAR/,
                                                    yield to callback.
                 + has_path_prefix(archive, prefix) — cheap 'is this MY
                                                      framework's jar?' check
                                                      via central-dir filenames.
                 + for_each_class(archive, fn)    — parse every .class entry
                                                    through class_reader,
                                                    yield (name, &ClassFile).
                 + try_each_class(archive, fn)    — same with early-return on
                                                    first Some(R) match.

Refactored:

  dbp.rs         v2 on the new platform:
                 - dropped extract_printable raw byte scan
                 - dropped its own English -> ISO 639-2 map
                 - dropped its own parse_attributes hand-roll
                 + iterates CpInfo::Utf8 via class_reader (structurally clean,
                   no false-positive risk from method bytecode bytes)
                 + routes language/purpose/qualifier through vocab
                 All 7 prior dbp tests still pass; +2 new ones cover
                 vocab routing.

dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.

Precommit green (cargo +1.86 fmt + clippy + test).
This commit is contained in:
MattJackson
2026-05-10 15:16:25 -07:00
parent 99236ffd55
commit 4ec75a03f2
5 changed files with 707 additions and 208 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/labels/dbp.rs
+91 -199
View File
@@ -22,25 +22,29 @@
//! what precedes it. `Subtitle0` is the disable-subtitles menu
//! button and is skipped (not a real subtitle stream).
//!
//! Per `(internal)/memory/feedback_label_data_rules.md`: this
//! parser knows its own format, so we map human-readable language
//! names ("English", "Spanish", "Canadian French", ...) to ISO 639-2
//! codes locally. The full disc-authored display string is preserved
//! in the label `name` field — consumers display it raw without
//! freemkv guessing further structure.
//! ## Implementation
//!
//! v2 (2026-05-10): rewritten on top of [`super::class_reader`] —
//! iterates `CpInfo::Utf8` constant-pool entries instead of raw byte
//! scanning each class file. Equivalent label coverage (the literal
//! `TextField,...` strings live in the CP as Utf8 entries), but
//! structurally cleaner: no false-positive risk from method bytecode
//! or attribute names happening to contain `TextField,`. Language /
//! purpose / qualifier classification moved to [`super::vocab`] so all
//! Java-parser families share one source of truth.
use super::{LabelPurpose, LabelQualifier, StreamLabel, StreamLabelType};
use super::class_reader::CpInfo;
use super::{StreamLabel, StreamLabelType, jar, vocab};
use crate::sector::SectorReader;
use crate::udf::UdfFs;
use std::collections::BTreeMap;
/// dbp detect can't peek inside a jar without a SectorReader (the
/// trait function only takes `&UdfFs`), so we trigger on the cheap
/// signal "any top-level .jar in /BDMV/JAR/." That fires on every
/// BD-J disc, but parse() does the real `com/dbp/` check and
/// returns None on a mismatch — so this parser only ever consumes
/// time on discs that fell through every earlier parser. The
/// parse-side mismatch is bounded (read one .jar, list central
/// directory, walk class strings).
/// time on discs that fell through every earlier parser.
pub fn detect(udf: &UdfFs) -> bool {
let Some(jar_dir) = udf.find_dir("/BDMV/JAR") else {
return false;
@@ -52,73 +56,35 @@ pub fn detect(udf: &UdfFs) -> bool {
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
let jar_dir = udf.find_dir("/BDMV/JAR")?;
for entry in &jar_dir.entries {
if entry.is_dir {
continue;
jar::for_each_jar(reader, udf, |_entry_name, archive| {
if !jar::has_path_prefix(archive, "com/dbp/") {
return None;
}
if !entry.name.to_lowercase().ends_with(".jar") {
continue;
let labels = scan_jar(archive);
if labels.is_empty() {
None
} else {
Some(labels)
}
let path = format!("/BDMV/JAR/{}", entry.name);
let Ok(bytes) = udf.read_file(reader, &path) else {
continue;
};
let cursor = std::io::Cursor::new(&bytes);
let Ok(mut archive) = zip::ZipArchive::new(cursor) else {
continue;
};
if !archive_has_dbp(&mut archive) {
continue;
}
let labels = scan_jar(&mut archive);
if !labels.is_empty() {
return Some(labels);
}
}
None
})
}
fn archive_has_dbp<R: std::io::Read + std::io::Seek>(archive: &mut zip::ZipArchive<R>) -> bool {
for i in 0..archive.len() {
if let Ok(f) = archive.by_index(i) {
if f.name().starts_with("com/dbp/") {
return true;
}
}
}
false
}
fn scan_jar<R: std::io::Read + std::io::Seek>(
archive: &mut zip::ZipArchive<R>,
) -> Vec<StreamLabel> {
use std::collections::BTreeMap;
fn scan_jar(archive: &mut jar::Jar) -> Vec<StreamLabel> {
// BTreeMap so we keep the highest-numbered (last-written) label
// for each stream slot deterministic across runs. Entries are
// collected from string-pool fragments scattered across hundreds
// of obfuscated .class files; the same TextField,Audio1,...
// string can appear in multiple classes (button-state variants,
// localization fallbacks). Last write wins — they should all
// agree on the label text, but the structure is defensive.
// for each stream slot deterministic across runs. The same
// TextField,Audio1,... string can appear in multiple classes
// (button-state variants, localization fallbacks). Last write
// wins — they should all agree, but the structure is defensive.
let mut audios: BTreeMap<u16, String> = BTreeMap::new();
let mut subs: BTreeMap<u16, String> = BTreeMap::new();
for i in 0..archive.len() {
let Ok(mut f) = archive.by_index(i) else {
continue;
};
if !f.name().ends_with(".class") {
continue;
jar::for_each_class(archive, |_class_name, class| {
for (_idx, cp) in class.constant_pool.iter() {
if let CpInfo::Utf8(s) = cp {
collect_textfield(s, &mut audios, &mut subs);
}
}
let mut buf = Vec::new();
if std::io::Read::read_to_end(&mut f, &mut buf).is_err() {
continue;
}
for s in extract_printable(&buf) {
collect_textfield(&s, &mut audios, &mut subs);
}
}
});
let mut out = Vec::new();
for (num, label) in audios {
@@ -132,8 +98,8 @@ fn scan_jar<R: std::io::Read + std::io::Seek>(
fn collect_textfield(
s: &str,
audios: &mut std::collections::BTreeMap<u16, String>,
subs: &mut std::collections::BTreeMap<u16, String>,
audios: &mut BTreeMap<u16, String>,
subs: &mut BTreeMap<u16, String>,
) {
// Anchor on "TextField," — the prefix character before it varies
// (string-pool ordering inside compiled Java) and is irrelevant.
@@ -163,7 +129,9 @@ fn collect_textfield(
}
fn make_label(num: u16, label: String, stream_type: StreamLabelType) -> StreamLabel {
let (language, qualifier, purpose) = parse_attributes(&label);
let language = vocab::lang(&label).unwrap_or_default().to_string();
let qualifier = vocab::qualifier(&label);
let purpose = vocab::purpose(&label);
StreamLabel {
stream_number: num,
stream_type,
@@ -176,111 +144,15 @@ fn make_label(num: u16, label: String, stream_type: StreamLabelType) -> StreamLa
}
}
fn parse_attributes(label: &str) -> (String, LabelQualifier, LabelPurpose) {
let lower = label.to_lowercase();
let language = detect_language(&lower);
let qualifier = if lower.contains("sdh") {
LabelQualifier::Sdh
} else if lower.contains("descriptive service") || lower.contains(" rnib") {
LabelQualifier::DescriptiveService
} else if lower.contains("forced") {
LabelQualifier::Forced
} else {
LabelQualifier::None
};
let purpose = if lower.contains("commentary") {
LabelPurpose::Commentary
} else if lower.contains("descriptive") || lower.contains("audio description") {
LabelPurpose::Descriptive
} else {
LabelPurpose::Normal
};
(language, qualifier, purpose)
}
/// Map English-language label tokens to ISO 639-2 codes. Keep this
/// list conservative — only common tokens we've actually observed
/// or that have a canonical mapping. Returns "" when the token
/// isn't recognized; the consumer falls back to fill_defaults reading
/// MPLS spec language codes.
fn detect_language(lower: &str) -> String {
// Compound tokens first (multi-word language names).
for (needle, code) in [
("brazilian portuguese", "por"),
("euro portuguese", "por"),
("castilian spanish", "spa"),
("latin american spanish", "spa"),
("canadian french", "fra"),
("parisian french", "fra"),
("australian english", "eng"),
("austrailian english", "eng"), // disc-corpus typo, keep matching
] {
if lower.contains(needle) {
return code.to_string();
}
}
// Then bare tokens. Order matters where one is prefix of another.
for (needle, code) in [
("english", "eng"),
("spanish", "spa"),
("french", "fra"),
("german", "deu"),
("italian", "ita"),
("japanese", "jpn"),
("chinese", "zho"),
("portuguese", "por"),
("polish", "pol"),
("czech", "ces"),
("hungarian", "hun"),
("dutch", "nld"),
("korean", "kor"),
("arabic", "ara"),
("hindi", "hin"),
("turkish", "tur"),
("thai", "tha"),
("swedish", "swe"),
("norwegian", "nor"),
("danish", "dan"),
("finnish", "fin"),
("hebrew", "heb"),
("russian", "rus"),
] {
if lower.split_whitespace().next() == Some(needle)
|| lower.split_whitespace().any(|w| w == needle)
{
return code.to_string();
}
}
String::new()
}
fn extract_printable(data: &[u8]) -> Vec<String> {
let mut out = Vec::new();
let mut current = String::new();
for &b in data {
if (0x20..0x7f).contains(&b) {
current.push(b as char);
} else {
if current.len() >= 5 {
out.push(current.clone());
}
current.clear();
}
}
if current.len() >= 5 {
out.push(current);
}
out
}
#[cfg(test)]
mod tests {
use super::super::{LabelPurpose, LabelQualifier};
use super::*;
#[test]
fn collect_extracts_audio_and_subtitle_indices() {
let mut audios = std::collections::BTreeMap::new();
let mut subs = std::collections::BTreeMap::new();
let mut audios = BTreeMap::new();
let mut subs = BTreeMap::new();
let lines = [
"LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,296,763,275,25,left",
"RTextField,Audio2,English Descriptive Audio,Fontstrip_Composite,296,803,275,25,left",
@@ -304,8 +176,8 @@ mod tests {
#[test]
fn collect_ignores_non_textfield_strings() {
let mut audios = std::collections::BTreeMap::new();
let mut subs = std::collections::BTreeMap::new();
let mut audios = BTreeMap::new();
let mut subs = BTreeMap::new();
for s in [
"GraphicButton,SU_Audio",
"AudioMenu",
@@ -319,47 +191,67 @@ mod tests {
}
#[test]
fn parse_attributes_recognizes_sdh() {
let (lang, qual, purp) = parse_attributes("English SDH");
assert_eq!(lang, "eng");
assert_eq!(qual, LabelQualifier::Sdh);
assert_eq!(purp, LabelPurpose::Normal);
fn make_label_routes_via_vocab() {
let l = make_label(1, "English SDH".to_string(), StreamLabelType::Subtitle);
assert_eq!(l.language, "eng");
assert_eq!(l.qualifier, LabelQualifier::Sdh);
assert_eq!(l.purpose, LabelPurpose::Normal);
}
#[test]
fn parse_attributes_recognizes_descriptive_audio() {
let (lang, qual, purp) = parse_attributes("English Descriptive Audio");
assert_eq!(lang, "eng");
assert_eq!(qual, LabelQualifier::None);
assert_eq!(purp, LabelPurpose::Descriptive);
fn make_label_descriptive_audio() {
let l = make_label(
2,
"English Descriptive Audio".to_string(),
StreamLabelType::Audio,
);
assert_eq!(l.language, "eng");
assert_eq!(l.purpose, LabelPurpose::Descriptive);
}
#[test]
fn parse_attributes_recognizes_commentary() {
let (lang, qual, purp) = parse_attributes("English Director's Commentary");
assert_eq!(lang, "eng");
assert_eq!(qual, LabelQualifier::None);
assert_eq!(purp, LabelPurpose::Commentary);
fn make_label_commentary() {
let l = make_label(
3,
"English Director's Commentary".to_string(),
StreamLabelType::Audio,
);
assert_eq!(l.language, "eng");
assert_eq!(l.purpose, LabelPurpose::Commentary);
}
#[test]
fn parse_attributes_recognizes_compound_languages() {
assert_eq!(parse_attributes("Brazilian Portuguese 5.1").0, "por");
assert_eq!(parse_attributes("Castilian Spanish").0, "spa");
assert_eq!(parse_attributes("Canadian French Dolby Digital").0, "fra");
assert_eq!(parse_attributes("Latin American Spanish").0, "spa");
fn make_label_compound_languages() {
assert_eq!(
make_label(1, "Brazilian Portuguese 5.1".into(), StreamLabelType::Audio).language,
"por"
);
assert_eq!(
make_label(1, "Castilian Spanish".into(), StreamLabelType::Audio).language,
"spa"
);
assert_eq!(
make_label(
1,
"Canadian French Dolby Digital".into(),
StreamLabelType::Audio
)
.language,
"fra"
);
}
#[test]
fn parse_attributes_returns_empty_for_unknown_language() {
// Don't guess. Per the rules-of-engagement.
assert_eq!(parse_attributes("Klingon Dolby Atmos").0, "");
fn make_label_unknown_language_is_empty() {
// vocab::lang returns None — make_label converts to "".
let l = make_label(1, "Klingon Dolby Atmos".into(), StreamLabelType::Audio);
assert_eq!(l.language, "");
}
#[test]
fn parse_attributes_recognizes_rnib_descriptive_service() {
let (lang, qual, _) = parse_attributes("English RNIB");
assert_eq!(lang, "eng");
assert_eq!(qual, LabelQualifier::DescriptiveService);
fn make_label_rnib_descriptive_service() {
let l = make_label(1, "English RNIB".into(), StreamLabelType::Subtitle);
assert_eq!(l.language, "eng");
assert_eq!(l.qualifier, LabelQualifier::DescriptiveService);
}
}