237794a6ea86917668d7a6b7642203502ab03b16
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matthew Jackson
|
4959b48386 |
Bind borrowed stream labels by the stream they name, not by a slot
A vendor label's `stream_number` is a slot in the one stream table its
config blob describes. The labels merged in from the playlists — to
cover streams the vendor named nothing for — carried a different number
entirely: a dense counter over every distinct stream found while
scanning the whole disc in directory order, related to no playlist's
slot numbering at all. Two coordinate systems, one field name. The
merge matched them by equality and the binder then counted streams
against the result.
Measured over the 44-image corpus: 22 discs merge such labels; of the
566 places one lands on a stream, 443 (78%) are a stream it does not
describe — the label states the PID it read itself from and it is a
different one. 142 of those are stopped by the language check
|
||
|
Matthew Jackson
|
9fda690d00 |
Number vendor label streams by STN slot, not by parsed entry
Sweep of every parser in src/labels/ for the numbering bug fixed in
pixelogic: a blob lists one entry per STN slot, but the parser advances
its per-kind counter only for entries it can use, so every entry it
skips shifts all later labels onto the wrong stream.
Three parsers were affected; the rest key each label off a number the
blob states outright and are immune.
* paramount — `aud` / `sub` are the STN-ordered stream lists, and
`forced_sub` / `*_com1_idx` index those same cells. A cell with an
empty language was skipped without consuming its slot, so every
label behind it bound one stream early while the vendor's own
positional indices still pointed at the raw cell. `stream_number`
is now the cell's 1-based position. The forced flag is the payload
here, so the shift lands `forced` on a full-dialogue track.
* mpls_universal — its counters must agree with the stream list
`disc::bluray` builds from the same STN entries, since that list is
what `apply_labels` counts against. They disagreed twice: a
`coding_type == 0` padding entry was counted here and dropped
there, and a PG coding_type in an audio STN slot (a layout
`mpls::parse_stream_entry` has a dedicated arm for) was counted as
audio here and built as a subtitle there. Both rules now live in
one `label_type_for`.
* deluxe — a binding construction whose Language `getstatic` did not
resolve was skipped outright. It is still an STN slot; it just has
nothing to label. It now advances the counter, with the list it
belongs to taken from its CodingType argument or, failing that,
from what its binding type's resolved siblings showed.
Two paramount tests asserted the renumbering as if it were the spec
(`empty_middle_slot_does_not_inflate_stream_number`,
`audio_stream_numbering_skips_empty_slots`) and are rewritten. Immunity
pins added for ctrm, dbp and criterion so the property cannot rot.
Also scrubs two commercial disc titles from the menu-graphic filename
examples in png_filenames and vocab.
|
||
|
Matthew Jackson
|
5360f8d309 |
test: salvage the orphaned labels/disc triage, and extract build_labels
Thirteen agents triaging src/labels and src/disc died on a saturated
machine, leaving 5,836 insertions across 28 files uncommitted in a
worktree. Recovered by 3-way apply onto twelve commits of drift; zero
conflicts. The diff was archived to freemkv-private first, because a
worktree is not a backup and this one had already nearly been lost.
One production change, and it is the right one: mpls_universal::parse
read every playlist off the disc AND converted the entries to labels in
a single function, so the conversion — stream-type mapping, dedup key,
the dense global counters — could only be reached through a synthetic
UDF image. Extracted to build_labels(&[Playlist]), which unit tests can
drive from already-parsed values. Behaviour-preserving: same iteration
order, same skip-on-error.
Two collisions resolved by hand:
A second mod pass_progress_tests, written independently against the
same survivors as the one committed in
|
||
|
Matthew Jackson
|
18f8b285c4 |
Bound the BD-J label parsers, and stop a crafted disc hanging the scan
Ten defects in code no previous round had ever scoped. `src/labels/` identifies a disc's studio by parsing jar archives and JVM class files off untrusted media, so every byte here is attacker-controllable — and 813 of its lines were executed by no test at all. The worst is a non-terminating loop. A fallback stream-number scan advanced with `saturating_add`, and the comment says why: a crafted XML "must not overflow (panic in debug, wrap-to-0 in release)". Once the counter pins at u16::MAX and that number is taken, the loop cannot exit. So a fix for an overflow panic produced an unbounded hang, which is strictly worse — a panic is observable and catchable, and catch_unwind cannot interrupt a live loop. Reachable from about 8 MB of XML. Where the same overflow appears in the deluxe decoder the fix is checked_add and stop, NOT saturation — twice wrong there, because saturating would peg every stream past the ceiling at one number and apply_labels binds on (type, number), silently mislabelling tracks. A correctness bug wearing the costume of success. Round 7 capped the ldc-string retention per class; nothing capped the aggregate, so a 64 MiB jar held that budget for every class at once. Same defect one level up, which is the shape that keeps recurring in this directory. Four other amplifications are bounded the same way, each with a stated headroom and a paired test proving real media passes untouched — the tightest is 5x on a label length, the loosest 2000x on the stream numbering space, against BD's 32-per-type STN_table limit. Two are not caps at all: a quadratic membership scan became a set, and an attacker-derived length added to a cursor without saturation now cannot wrap. Nothing is excluded by either. A `#[cfg(test)]` hand-copy of a shipping parser was the ninth bad test this audit has found, and the first proven by mutation rather than inspection: deleting the guard from the REAL function left all 26 tests green, including the one named for that guard. Pointed at the real function, the same mutation fails. Separately, all three failure arms of the bounded fsync returned Ok(()) on both macOS and Linux, so sync_all reported success for a durability barrier that never ran. Only macOS was in scope; the Linux twin is fixed here too, because a platform disagreeing with its sibling about whether a failed sync is an error is the class that already produced an over-length SCSI CDB macOS rejected and the other two truncated. Note the behaviour change: a mux whose final sync times out on a wedged mount now fails rather than exiting 0. Three of the caps are proven by wall-clock deadline rather than an operation count, with 18-80x margin on the passing side. On a heavily oversubscribed machine those could flake. |
||
|
Matthew Jackson
|
5f8dc392c0 |
Sweep the pinned toolchain to Rust 1.97
The Windows UI needs current winsafe, whose real minimum is 1.89 (its manifest under-declares 1.87 while it uses NonNull::from_ref). Rather than stop at the minimum, this goes to current stable and fixes what that costs. The counter-intuitive result: 1.97 is CHEAPER than 1.89. libfreemkv had 54 clippy errors at 1.89 and 6 at 1.97, because clippy tightened the noisy collapsible_if lint in between. Stopping at the minimum would have been the most expensive choice available. Roughly 47 lints across the eight repos, the large majority auto-fixed: libfreemkv 6, freemkv-engine 14, bdemu 8, freemkv-keysources 7, autorip 6, freemkv-unlock 3, freemkv-i18n 3. The hand-fixed ones are a descending sort to sort_by_key(Reverse), four manual checked-division sites, a loop counter replaced by enumerate, and a loop whose first let-else became a while-let. Worth recording for whoever bumps next: clippy is MSRV-AWARE. Those 54 lints only appear once the crate DECLARES 1.89 or later, because let-chains become available. A bare `cargo +1.89 clippy` against a manifest still pinned at 1.87 reports clean and is meaningless — gate with the real precommit script, which is also the only thing that covers build scripts. The pin still sits below the Mac default, so it keeps doing its job: catching lint drift locally before CI sees it. |
||
|
Matthew Jackson
|
45c12fc5ce |
labels: reader-backed detection, Criterion fix, menu-language fallback
- DetectFn now takes a SectorSource so a parser can inspect a jar's
central directory in detect() instead of firing on "any BD-J jar".
dbp/deluxe do the real com/<vendor>/ prefix check up front, so each
claims only its own discs (foundational for scaling the registry).
- criterion: treat a stream-map value of 0 as unmapped and synthesize a
real 1-based number, so a 0 can't shadow or collide with a genuine
stream 1 (with regression tests).
- png_filenames: new Low-confidence, last-resort parser reading menu
language from {title}_UHD01_{LANG}_Composite artwork; sits below the
MPLS floor so a real framework parser always wins.
- vocab: add menu_lang() for 639-2/B to 639-2/T menu-token normalization.
|
||
|
Matthew Jackson
|
061f68594a |
0.31.0: hardening and correctness pass across mux, codec, AACS/CSS, UDF/MPLS/CLPI, recovery, drive/SCSI, labels, and I/O
Library-wide review-and-fix pass: tightened AACS keydb/handshake/variant handling and trailing-partial-unit policy, corrected MPLS mark offset and added UDF allocation bounds, hardened the mux/codec framing and M2TS paths, guarded SCSI READ CAPACITY short transfers and unified error mapping, added overflow guards on untrusted disc input, and made prefetch shutdown deterministic. Release profile now builds with thin LTO + single codegen unit. |
||
|
MattJackson
|
1565da610a | chore: scrub non-shippable references from tests/comments | ||
|
MattJackson
|
f1926c38dc |
v0.20.1: delete SectorReader, extract Disc::patch, doc/stub cleanup
WO-2 (delete SectorReader trait): - The 0.18 trait split into SectorSource (read-only) and SectorSink (write-only) is final; the legacy SectorReader alias was a bridge. - Renames every internal &mut dyn SectorReader (~25 sites) to &mut dyn SectorSource. The trait method capacity() becomes capacity_sectors() with a default of 0 (preserves SectorReader's default-0 behavior). - Deletes the SectorReader trait, its blanket-to-Source bridge, and the FileSectorReader type alias. Adds explicit forwarding impls for Box<dyn SectorSource> and &mut dyn SectorSource so generic decorators like DecryptingSectorSource<S: SectorSource> compose. WO-3a (extract Disc::patch): - Moves Disc::patch (1230 lines) and bytes_bad_in_title from disc/mod.rs into disc/patch.rs as a split inherent impl. Zero behavior change — pure mechanical relocation. disc/mod.rs drops from 3,945 to 2,714 LOC. WO-6 (partial): - Deletes src/labels/png_filenames.rs — was a 72-LOC stub with detect() returning false, never wired into the PARSERS registry. project docs doc drift fixes (audited 2026-05-13): - JUMP_BASE_SECTORS: 256→1024 (64 MB base for UHD, not 8 MB) - PASSN_DAMAGE_THRESHOLD_PCT: 12→6 - PASSN_SKIP_SECTORS_BASE: 64→32 - MAX_RANGE_SECS=180: replaced by proportional range_sectors × 25, capped at RANGE_BUDGET_CAP_SECS=1800. |
||
|
MattJackson
|
7c6b0f82ab |
labels: per-parser confidence + highest-confidence-wins registry
Replaces 'first-match-wins by array order' with 'highest-confidence-
wins, array order tiebreaker'. Removes the arbitrariness when more
than one parser can claim a disc (e.g. one with both
bluray_project.bin and playlists.xml).
New types in labels::mod:
pub enum Confidence { Medium, High }
pub struct ParseResult { labels: Vec<StreamLabel>, confidence }
ParseResult::high(labels) / ::medium(labels) constructors
Parser signature change: every parse() now returns
Option<ParseResult> instead of Option<Vec<StreamLabel>>. Updated all
six parsers in lockstep:
paramount: High (fully structured XML)
criterion: High (fully structured XML)
pixelogic: High by default, Medium when an unknown token component
is encountered (the skip-unknown path now propagates the
coverage gap to the caller instead of silently degrading)
ctrm: High (structured key-value)
dbp: High (anchor scan with vocab routing)
deluxe: still returns None pending Phase D — signature aligned
Registry behavior:
extract() iterates all detect-positive parsers, picks highest
Confidence with non-empty labels. Equal confidence falls to array
order (deterministic). Same selection logic in analyze().
LabelAnalysis grew a confidence: Option<Confidence> field so the
diagnostic surface (freemkv-tools labels-analyze) exposes which
confidence tier the selected parser claimed. labels-analyze JSON
and labels-corpus-check structural diff both gained the field.
Precommit (cargo +1.86 fmt + clippy + test) green.
|
||
|
MattJackson
|
4226a53e73 |
labels: fresh-eyes audit — capture variant, dedupe detect, lock registry
Three targeted fixes from a second-pass audit of the labels module. 1. vocab::lang now returns Option<LangInfo> with both code AND a human-readable variant string. Pre-fix: 'Brazilian Portuguese 5.1' became language=por, variant='', dropping the dialect info the disc had explicitly authored. Post-fix: language=por, variant='Brazilian' — matches the convention pixelogic / ctrm / criterion already use for their region variants. dbp now populates StreamLabel::variant from this. Compound table grew a 3-tuple (needle, code, variant); bare matches still return variant=''. 2. dbp and deluxe had duplicated detect() boilerplate (any top-level .jar in /BDMV/JAR/). Both now call jar::has_any_top_level_jar. The trait-level detect contract — see super::PARSERS — can't peek inside a jar without a SectorReader, so loose-detect-plus-real- check-in-parse is the unavoidable pattern for jar-content parsers. Consolidating in jar.rs at least makes the duplication visible. 3. mod.rs comment about parser ordering said 'dbp last'; deluxe is actually now last. Updated to explain the dbp-before-deluxe order is by cost (cp-iteration cheaper than bytecode walking when Phase D lands). Plus a registry-level lock test in mod.rs::registry_tests — asserts the PARSERS array order is exactly [paramount, criterion, pixelogic, ctrm, dbp, deluxe]. This was previously implicit; if someone reorders the array (which changes which parser wins on overlapping signals), unit tests would have stayed green. Now they fail with an explanatory message about why the order matters. Audit findings deferred to follow-ups (each its own commit + design discussion): - Stronger detect contract — current loose-detect-real-check pattern is forced by SectorReader-not-in-detect-signature; could be fixed by changing the trait to take an Option<&mut dyn SectorReader> or similar. - Per-parser confidence scoring — registry currently first-match-wins. A high-confidence parser ought to beat a low-confidence one regardless of array order. - class_reader fuzzing — handles malformed input via Result but no adversarial corpus yet. Precommit (cargo +1.86 fmt + clippy + test) green. |
||
|
MattJackson
|
4ec75a03f2 |
labels: shared platform (vocab/text/jar) + dbp refactor
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.
New modules:
vocab.rs expanded from 27 -> ~370 lines
+ lang(text) -> Option<&'static str> (English/multi-word
-> ISO 639-2; ~45
languages, compound
phrases like
'Brazilian Portuguese'
and 'Castilian Spanish')
+ purpose(text) -> LabelPurpose (Commentary,
Descriptive, Score,
Ime; word-boundary
matched)
+ qualifier(text) -> LabelQualifier (SDH, Forced,
DescriptiveService)
+ has_word internal primitive — enforces word-boundary
matching so 'Commenter' no longer matches 'commentary' and
'engineering' no longer matches 'english'. Existing parsers
used .contains() and got lucky on the corpus; vocab now
guarantees the boundary in one place. 20+ unit tests.
text.rs NEW (~85 lines)
+ extract_ascii_strings(data, min_len) — promoted from two
near-duplicate copies (pixelogic min=4, dbp min=5);
threshold passed in. 7 unit tests including
trailing-without-terminator + high-bit-byte handling.
jar.rs NEW (~120 lines)
+ for_each_jar(reader, udf, fn) — walk every top-level
.jar under /BDMV/JAR/,
yield to callback.
+ has_path_prefix(archive, prefix) — cheap 'is this MY
framework's jar?' check
via central-dir filenames.
+ for_each_class(archive, fn) — parse every .class entry
through class_reader,
yield (name, &ClassFile).
+ try_each_class(archive, fn) — same with early-return on
first Some(R) match.
Refactored:
dbp.rs v2 on the new platform:
- dropped extract_printable raw byte scan
- dropped its own English -> ISO 639-2 map
- dropped its own parse_attributes hand-roll
+ iterates CpInfo::Utf8 via class_reader (structurally clean,
no false-positive risk from method bytecode bytes)
+ routes language/purpose/qualifier through vocab
All 7 prior dbp tests still pass; +2 new ones cover
vocab routing.
dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.
Precommit green (cargo +1.86 fmt + clippy + test).
|
||
|
MattJackson
|
0081955686 |
labels: add dbp parser (Magnolia Pictures BD-J framework)
5th BD-J authoring framework recognized. Discriminator: any top-
level .jar in /BDMV/JAR/ that contains com/dbp/ package paths.
Identified during the 2026-05-10 corpus session via string-mining
disc-07's BD-J jar — perm files reference bd-live.magpictures.com
(Magnolia / Magnet Releasing).
Stream labels live as plain ASCII strings inside compiled .class
files in the form
LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,...
HTextField,Subtitle1,English SDH,Fontstrip_Composite,...
ATextField,Subtitle0,None,Fontstrip_Composite,...
— a quirk of the menu-rendering layer encoding TextField positions
and content as constant strings the Java compiler retained in the
class string pool. The leading single-letter prefix is string-pool
ordering noise; parser anchors on `TextField,`. Subtitle0 is the
disable-subtitles button and is skipped.
The parser:
- reads each top-level .jar via udf.read_file
- opens it with the existing zip dependency
- confirms com/dbp/ presence in the central directory
- walks .class entries, extracts printable strings, matches the
`TextField,(Audio|Subtitle)<N>,<label>,...` pattern
- maps human-readable language names ("English", "Castilian
Spanish", "Brazilian Portuguese", "Canadian French", ...) to
ISO 639-2 codes via a parser-local table (per the rules-of-
engagement memo, each parser knows its own format)
- preserves the full disc-authored label string in `name` so
consumers display it raw without the lib guessing further
structure
- detects SDH / Forced qualifiers and Commentary / Descriptive
purposes from substring matches; everything else falls through
to fill_defaults using BD-spec MPLS data
Verified live on the corpus: disc-07 (Civil War UHD) now matches
parser=dbp with 3 audio + 2 subtitle labels, exactly the count
visible in the disc's authored TextField definitions and what BD
spec MPLS reports.
Limitation: dbp's detect() returns true for ANY top-level .jar in
/BDMV/JAR/ (every BD-J disc has one), since the discriminator
trait function takes only `&UdfFs` and can't read jar contents.
parse() does the real com/dbp/ check — a non-dbp disc gets
parsed-as-dbp, archive_has_dbp returns false, parse() returns
None, and we fall through. Diagnostic noise: parsers_detected
includes "dbp" on non-dbp BD-J discs. Real fix is refactoring the
DetectFn signature to take a SectorReader; deferred.
7 unit tests cover the TextField extraction, language detection
(simple + compound: "Brazilian Portuguese", "Castilian Spanish",
"Canadian French", "Latin American Spanish", "Australian English"
plus the disc-corpus typo "Austrailian English"), SDH/Forced/RNIB
qualifier detection, and Commentary/Descriptive purpose detection.
Don't-guess discipline preserved: unknown languages return ""
(consumer falls back to MPLS spec data via fill_defaults).
|