Sweep of every parser in src/labels/ for the numbering bug fixed in
pixelogic: a blob lists one entry per STN slot, but the parser advances
its per-kind counter only for entries it can use, so every entry it
skips shifts all later labels onto the wrong stream.
Three parsers were affected; the rest key each label off a number the
blob states outright and are immune.
* paramount — `aud` / `sub` are the STN-ordered stream lists, and
`forced_sub` / `*_com1_idx` index those same cells. A cell with an
empty language was skipped without consuming its slot, so every
label behind it bound one stream early while the vendor's own
positional indices still pointed at the raw cell. `stream_number`
is now the cell's 1-based position. The forced flag is the payload
here, so the shift lands `forced` on a full-dialogue track.
* mpls_universal — its counters must agree with the stream list
`disc::bluray` builds from the same STN entries, since that list is
what `apply_labels` counts against. They disagreed twice: a
`coding_type == 0` padding entry was counted here and dropped
there, and a PG coding_type in an audio STN slot (a layout
`mpls::parse_stream_entry` has a dedicated arm for) was counted as
audio here and built as a subtitle there. Both rules now live in
one `label_type_for`.
* deluxe — a binding construction whose Language `getstatic` did not
resolve was skipped outright. It is still an STN slot; it just has
nothing to label. It now advances the counter, with the list it
belongs to taken from its CodingType argument or, failing that,
from what its binding type's resolved siblings showed.
Two paramount tests asserted the renumbering as if it were the spec
(`empty_middle_slot_does_not_inflate_stream_number`,
`audio_stream_numbering_skips_empty_slots`) and are rewritten. Immunity
pins added for ctrm, dbp and criterion so the property cannot rot.
Also scrubs two commercial disc titles from the menu-graphic filename
examples in png_filenames and vocab.
Two comments identified the disc the STN-numbering bug was found on by its
vendor project name. The reproduction does not need it: what matters is the
SHAPE of the token list — placeholder slots, region-only tokens, an
uncatalogued component — not which release happened to exhibit it. Both now
describe the shape.
The remaining `SEG_MainFeature` references stay. That is a vendor section name
the parser matches on at pixelogic.rs:88, not a disc identifier — it is the
format's vocabulary, like `FPL_` or the `eng_MLP_` stream tokens beside it, and
removing it would break the parser.
Also drops the last prohibited citation from the changelog: an mp4:// bullet
said "no ffmpeg". The website changelog page is REGENERATED from this file at
release time, so a scrub of the site alone would have been reverted by the next
release.
A pixelogic feature section lists one entry per STN slot. Only the
entries that parse were advancing the counters, so every surviving
label was renumbered 1..N and applied to the wrong stream.
Three kinds of entry were being skipped:
* `PG Stream N` placeholders — the subtitle twin of `Audio Stream N`,
which was already counted. The old comment claimed the corpus showed
subtitle tokens align without counting them; on a disc that has both
placeholders and later editorial tokens they do not.
* region-only tokens (`fra_CF_`, `spa_LS_`) — REGIONS sets `variant`
but neither `is_audio` nor `is_subtitle`, so the token is dropped.
* tokens whose distinguishing component is uncatalogued (`jpn_DUB_`).
On UHD_Crime101_WW_150728 the PG list is 18 slots: five placeholders,
four region-only tokens, and `jpn_DUB_`, with the seven `*_TXT_FOR_`
forced-narrative tokens at STN 11-16 and 18. Counting only the ten
that parse put `eng_SDH_` on STN 1 and the seven forced markers on STN
2-8 — the disc's FULL subtitle tracks. `labels::apply_labels` turns a
Forced qualifier into `SubtitleStream::forced`, which the muxer writes
as Matroska `FlagForced`, so the output offered an "English (forced)"
track carrying 1731 display sets of complete English dialogue while the
real 19-set forced track went unflagged.
Unclassifiable entries carry no stream type, so they advance the list
currently being enumerated; sections run video -> audio -> PG and
`domain` follows the last entry whose type was known.
The remaining triage items after tonight's HIGH fixes: 1,290 lines, almost
all tests. Covers disc/mod.rs's DVD scan path (with real minimal VMG/VTS IFO
fixtures rather than mocks), drive/mod.rs, labels/class_reader.rs and
labels/mod.rs — the two biggest untriaged survivor clusters in the crate —
plus hevc.rs and ps.rs.
One production change, and it is an extraction rather than a behaviour
change: MacScsiTransport::open mapped the shim's negative failure sentinels
to typed errors inline, where nothing could reach it without a real IOKit
FFI call. It is now map_shim_open_error, so the mapping can be pinned. It
matters because collapsing -5 into the DeviceNotFound catch-all turns
"another process holds the drive" into "no such drive", and an operator
chasing the wrong problem is worse than a blunt error.
Gate green on the pinned toolchain including the secrets scanner.
mod.rs had two independent implementations of "pick the winning label
parser". select_result owns the rule — highest confidence wins, and on a tie
the earlier entry in PARSERS wins — and carries a regression test for a past
bug where analyze() picked the LAST equal-confidence parser instead of the
first. extract(), the path that actually ships, re-derived the same rule with
its own inline scan and had no test of its own.
The tie-break is load-bearing rather than incidental: array order encodes a
trust ordering, with the hand-vetted parsers registered ahead of the ones
that detect on any BD-J disc. A later parser winning a tie means the disc
gets labels from a less trusted source, silently.
extract now collects its candidates and calls select_result. Confirmed by
flipping the tie-break to prefer the later entry: the shared test fails,
where before the fix that mutation was invisible to the whole suite.
This is the seventh instance tonight of one policy implemented twice with
only one copy hardened, and the second in this file after the chapter
mark_type filter.
Two cases of the same shape: one policy implemented twice, with only one
copy hardened.
Disc::read_capacity decoded buf[0..4] from READ CAPACITY (10) without
checking that the transport actually delivered four bytes, even though its
comment claims to mirror decode_read_capacity — which has exactly that
check, and documents why. A drive answering GOOD with an empty data phase
leaves the buffer zeroed, so last_lba decodes to 0 and the probe reports a
one-sector disc instead of an error. It now calls the shared decoder rather
than re-deriving it.
collect_chapter_summary filtered chapters on mark_type <= 1, counting the
reserved type 0. PlaylistMark's own doc says filters must test == 1, and
disc/bluray.rs did; the labels path did not, inflating the public
chapter_count and letting a playlist whose only marks are reserved pass the
chapter_count == 0 skip. Both sites now share PlaylistMark::is_chapter_mark
so the copies cannot drift again.
Both fixes were confirmed red before green.
Every other repo's CI now runs clippy with --all-targets. libfreemkv,
the crate the other seven build against and the one held up as the
reference workflow, was the last one still linting the library only — so
its ~3,000 tests, by far the largest body of test code in the project,
had never been linted at all. Turning the flag on surfaced 74 findings.
Most were mechanical and applied with clippy --fix. The rest, by hand:
- Four discarded Results in decrypt.rs. css::descramble_region returns a
Result and four CSS tests threw it away, so a descramble that FAILED
would have surfaced as a confusing buffer-comparison mismatch instead
of the actual error. They expect() now.
- A dead `kp` field on the PlantedWalk fixture. The test deliberately
asserts Kp as the explicit AES-G3(dk, 1) relation from [C] §3.2.4
rather than against a stored value — its doc comment says so — which
makes the field not just unused but a trap: the obvious "fix" of
asserting against it would quietly weaken the test to comparing the
fixture with itself. Removed.
- Two hand-rolled ICB counters in the HD-DVD fixtures, a needless mut,
three vec!s that only ever needed arrays, a filter_map whose every arm
was Some, and a Vec::new()+push chain.
- Doc list indentation in mkv.rs and mp4/read.rs, which was mis-rendering
in the generated docs.
- A five-[u8; 16]-tuple return type named FourLevelParts.
Three lints are allowed at the specific sites, with reasons, because
they are wrong for this domain: the underscores in the bitstream-header
literals mark BITFIELD boundaries, not digit groups, so regrouping them
uniformly would satisfy the lint by destroying the only thing they
encode; and in three table-validation loops the loop variable is the
domain value under test (a DTS SFREQ code, an AMODE value, a palette
entry number), which is what the assertion messages name.
Resolution::pixels() returned (0, 0) for Unknown, and the MP4 sink wrote
it verbatim into tkhd (ISO/IEC 14496-12 8.3.2) and VisualSampleEntry
(12.1.3). Both fields are MANDATORY there, so unlike Matroska — which
omits the optional PixelWidth/PixelHeight elements — MP4 has nothing to
leave out. The result was a structurally complete file that passes every
container check, declares a 0x0 video track, cannot be rendered, and is
written with no error anywhere.
WHY IT WAS POSSIBLE, which is the part worth keeping:
pixels() previously fabricated 1920x1080 for Unknown. That was wrong but
playable, so this sink never needed a guard and the absence of one was
invisible. Changing the sentinel to (0, 0) moved the defect instead of
removing it — a zero PAIR still reads as a usable value, so the sink
stored it and serialised it.
The accessor's doc comment then ENUMERATED the callers it believed were
safe: "the Matroska sink omits the optional elements, the VobSub writer
omits its size: line, and no caller divides by either dimension." Two of
those three are true. MP4 was not on the list because MP4 has no guard
at all, and a prose list cannot enforce itself. mkv.rs's own comment
even states the principle — "the check belongs in the one accessor
rather than in each caller that remembered to write it" — and
labels/mod.rs still carried its own duplicate Unknown test long after
the accessor took that job over.
So: pixels() now returns Option. Not because Option is tidier, but
because every caller genuinely needs a DIFFERENT answer and the compiler
is the only thing that reliably makes them choose one. Matroska and the
metadata sinks take unwrap_or((0, 0)) with the reason stated at each
site; the VobSub path degrades to a palette-only .idx; MP4 fails with
E_MP4_UNKNOWN_RESOLUTION (9055).
Six call sites, not the five my first grep showed — I piped it through
`head` and acted on a truncated list. The compiler caught the sixth.
That is the same mistake as trusting a lens that reported silence.
identify_master_enums picks, for each fingerprint, the best candidate
class out of CandidatePool. Its tie-break only prefers an exact ldc
count over an inexact one, so two candidates that are BOTH inexact but
both within LDC_COUNT_TOLERANCE are decided purely by iteration order —
and the pool was a HashMap.
Rust seeds HashMap per instance, so this is not merely unstable across
runs: the new test resolves the SAME jar to both Alpha and Beta within a
single process, across 16 iterations. The same disc could emit different
commentary/SDH/descriptive labels on consecutive rips of unchanged
input, with nothing in the output saying the choice was arbitrary.
BTreeMap fixes it by construction rather than by a sort someone can
forget to keep. The pool is capped at MAX_CANDIDATE_CLASSES, so the
ordering cost is irrelevant.
The test runs the whole identification sixteen times and asserts one
distinct winner. A single run cannot distinguish deterministic from
lucky, and the seed does not change within a process — so repetition is
what makes this a test rather than a hope.
Found by the round-9 labels pass, which was dispatched specifically
because every one of the ten lenses had reported leaving deluxe.rs
unread. 1,159 new lines that nobody had opened.
Thirteen agents triaging src/labels and src/disc died on a saturated
machine, leaving 5,836 insertions across 28 files uncommitted in a
worktree. Recovered by 3-way apply onto twelve commits of drift; zero
conflicts. The diff was archived to freemkv-private first, because a
worktree is not a backup and this one had already nearly been lost.
One production change, and it is the right one: mpls_universal::parse
read every playlist off the disc AND converted the entries to labels in
a single function, so the conversion — stream-type mapping, dedup key,
the dense global counters — could only be reached through a synthetic
UDF image. Extracted to build_labels(&[Playlist]), which unit tests can
drive from already-parsed values. Behaviour-preserving: same iteration
order, same skip-on-error.
Two collisions resolved by hand:
A second mod pass_progress_tests, written independently against the
same survivors as the one committed in c610285. Kept mine — it covers
the distinct-counters case and the Progress blanket impl, which theirs
does not — but theirs had three clamp tests mine lacked: good_pct,
bad_pct and pending_pct also clamp an overshoot, and I had only tested
that for work_pct. Merged those in as one test and proved each of the
three clamps load-bearing by removing them individually.
An unused_parens warning in a new fixture.
Method note, recorded because it cost real time: git apply --3way
STAGES its result, so `git diff` reads empty and the tree looks
untouched. I nearly concluded the patch had silently failed. Worse, the
first attempt piped through `head -20`, so `echo exit=$?` reported
head's status rather than git's — the same mistake this audit has
already documented once. Check the real exit status, and check
--cached, not just the working tree.
Ten defects in code no previous round had ever scoped. `src/labels/`
identifies a disc's studio by parsing jar archives and JVM class files off
untrusted media, so every byte here is attacker-controllable — and 813 of
its lines were executed by no test at all.
The worst is a non-terminating loop. A fallback stream-number scan
advanced with `saturating_add`, and the comment says why: a crafted XML
"must not overflow (panic in debug, wrap-to-0 in release)". Once the
counter pins at u16::MAX and that number is taken, the loop cannot exit.
So a fix for an overflow panic produced an unbounded hang, which is
strictly worse — a panic is observable and catchable, and catch_unwind
cannot interrupt a live loop. Reachable from about 8 MB of XML.
Where the same overflow appears in the deluxe decoder the fix is
checked_add and stop, NOT saturation — twice wrong there, because
saturating would peg every stream past the ceiling at one number and
apply_labels binds on (type, number), silently mislabelling tracks. A
correctness bug wearing the costume of success.
Round 7 capped the ldc-string retention per class; nothing capped the
aggregate, so a 64 MiB jar held that budget for every class at once. Same
defect one level up, which is the shape that keeps recurring in this
directory. Four other amplifications are bounded the same way, each with a
stated headroom and a paired test proving real media passes untouched —
the tightest is 5x on a label length, the loosest 2000x on the stream
numbering space, against BD's 32-per-type STN_table limit.
Two are not caps at all: a quadratic membership scan became a set, and an
attacker-derived length added to a cursor without saturation now cannot
wrap. Nothing is excluded by either.
A `#[cfg(test)]` hand-copy of a shipping parser was the ninth bad test
this audit has found, and the first proven by mutation rather than
inspection: deleting the guard from the REAL function left all 26 tests
green, including the one named for that guard. Pointed at the real
function, the same mutation fails.
Separately, all three failure arms of the bounded fsync returned Ok(()) on
both macOS and Linux, so sync_all reported success for a durability
barrier that never ran. Only macOS was in scope; the Linux twin is fixed
here too, because a platform disagreeing with its sibling about whether a
failed sync is an error is the class that already produced an over-length
SCSI CDB macOS rejected and the other two truncated. Note the behaviour
change: a mux whose final sync times out on a wedged mount now fails
rather than exiting 0.
Three of the caps are proven by wall-clock deadline rather than an
operation count, with 18-80x margin on the passing side. On a heavily
oversubscribed machine those could flake.
The Program Stream demuxer appended every fed byte and enforced its 4 MiB
cap only inside a branch reached once a start code had been found. Input
containing no start code anywhere therefore hit no cap at all, and since a
whole title is fed through this demuxer, a zero-filled or ciphertext VOB
extent buffered the entire title — up to ~90 GB. When the buffer holds no
start code, only a two-byte `00 00` prefix can begin a PS unit on the next
feed, so that is kept and the rest dropped. The bound is exact rather than
a heuristic: a start code can straddle a feed boundary by at most its
first two bytes, so no real byte is discarded, and a test feeding
`FF FF 00 00` then `01 E0 ...` pins that.
The existing test named for this case fed a real start code first, so the
cap it exercised was the in-PES one. Renamed to say what it covers.
The BD-J label path had a different shape to anything found so far: the
cap is on the COMPRESSED size of a disc file while the allocation scales
with the decompressed size. A `.class` gated only by a path prefix
inflates to the 64 MiB ceiling, yielding ~33M retained strings from `ldc`
operands or ~67M pushes onto a symbolic stack whose depth was unbounded
despite the Code attribute's own `max_stack` being parsed and then
ignored. Bounded both, the stack by `max_stack` itself (JVMS §4.7.3).
The VMG TT_SRPT title count is an untrusted u16 with no de-duplication,
so ~800 KB of crafted IFO re-parsed one PGC 65535 times. Capped at 99, the
DVD-Video maximum, so no conformant disc is clipped.
Every cap carries stated headroom against real media, and each has a test
locking that real media still passes.
I rewrote three of the new assertions before landing them. They compared
the result against the very constant under test — `total <= MAX_TT_SRPT_TITLES`
— which passes vacuously the moment someone raises the constant, the most
likely future regression and the seventh instance of this tautology shape
in this audit. They now assert literals derived from the spec.
The TT_SRPT fixture also had to change: with 65535 identical entries the
de-duplication collapsed them on its own and the cap was never what
bounded the result, so the test passed with the cap removed entirely.
Distinct entries defeat dedup and leave the cap as the only guard;
de-duplication now has its own fixture. Verified by raising each of the
three constants and confirming all three tests fail.
The Windows UI needs current winsafe, whose real minimum is 1.89 (its manifest
under-declares 1.87 while it uses NonNull::from_ref). Rather than stop at the
minimum, this goes to current stable and fixes what that costs.
The counter-intuitive result: 1.97 is CHEAPER than 1.89. libfreemkv had 54
clippy errors at 1.89 and 6 at 1.97, because clippy tightened the noisy
collapsible_if lint in between. Stopping at the minimum would have been the most
expensive choice available.
Roughly 47 lints across the eight repos, the large majority auto-fixed:
libfreemkv 6, freemkv-engine 14, bdemu 8, freemkv-keysources 7, autorip 6,
freemkv-unlock 3, freemkv-i18n 3. The hand-fixed ones are a descending sort to
sort_by_key(Reverse), four manual checked-division sites, a loop counter replaced
by enumerate, and a loop whose first let-else became a while-let.
Worth recording for whoever bumps next: clippy is MSRV-AWARE. Those 54 lints only
appear once the crate DECLARES 1.89 or later, because let-chains become
available. A bare `cargo +1.89 clippy` against a manifest still pinned at 1.87
reports clean and is meaningless — gate with the real precommit script, which is
also the only thing that covers build scripts.
The pin still sits below the Mac default, so it keeps doing its job: catching
lint drift locally before CI sees it.
- DetectFn now takes a SectorSource so a parser can inspect a jar's
central directory in detect() instead of firing on "any BD-J jar".
dbp/deluxe do the real com/<vendor>/ prefix check up front, so each
claims only its own discs (foundational for scaling the registry).
- criterion: treat a stream-map value of 0 as unmapped and synthesize a
real 1-based number, so a 0 can't shadow or collide with a genuine
stream 1 (with regression tests).
- png_filenames: new Low-confidence, last-resort parser reading menu
language from {title}_UHD01_{LANG}_Composite artwork; sits below the
MPLS floor so a real framework parser always wins.
- vocab: add menu_lang() for 639-2/B to 639-2/T menu-token normalization.
Carry per-picture truth and byte-exact source provenance THROUGH the stream so
the muxer (and the upcoming video index) read MEASURED facts instead of
assuming them. Honest data in, honest data out.
- codec/coding.rs: codec-agnostic PictureInfo (CodingType / FieldOrder + the
accessors field_order/coding_type/nb_fields/progressive/keyframe). Each codec
folds its raw signals in; consumers use only accessors, never branch on codec.
- mpeg2: builds PictureInfo from the picture coding extension and carries it +
SourcePos (source_marks, parallel to pts_marks) on every emitted frame.
- pes / codec::Frame: additive `coding` + `source`, forwarded through the
highway; None for audio/subtitle and the network/stdio deserialize hop.
- mkvstream: DEFER muxer construction until the first coded picture, set the
video track's FieldOrder from the MEASURED value, THEN write the header —
right the first time, no guess, no seek-back. An interlaced track that arrives
with no measured order is LOGGED loudly and left UNDETERMINED, never faked.
- mkv: MkvTrack::video no longer guesses TFF (a bitstream property the scan
cannot know is UNDETERMINED at build). Removed VideoStream::top_field_first
(the dead scan-time guess) crate-wide.
- Tests: parser population (every PictureInfo facet + per-PES source carry) and
mux-stream consumption (measured -> correct; missing -> UNDETERMINED, not
faked). Two obsolete tests updated only after confirming (their own comments)
they existed to enforce the deleted hardcoded-TFF.
- FieldOrder now derives from the bitstream's measured top_field_first
(Some(true)→TFF, Some(false)→BFF) instead of hardcoding TFF for all
interlaced content; falls back to TFF when unmeasured. Adds
VideoStream::top_field_first; DVD/BD scan sets None with a precise
TODO(spec) for parser→title plumbing.
- CICP (matrix/transfer/primaries/range) now prefers measured CICP from
the bitstream (VideoStream::measured_cicp) over the coarse ColorSpace
enum, so the container stops assuming a colour space the stream may
contradict. Enum remains the fallback.
- VobSub S_VOBSUB CodecPrivate now emits a `size: WxH` line ahead of the
palette per the .idx format so players place/scale subs correctly.
- SeekHead: when zero cues are written, the CUES Seek entry is Voided
instead of leaving a dangling pointer to the Cues offset (now Tags/EOF).
- AC-3 Channels back-patch offset is captured from the writer instead of
the hardcoded chan_elem_pos+2 (decoupled from the VINT width choice).
- Hoisted inline CICP codes and the dvcC fourcc to named constants citing
ITU-T H.273 / RFC 9559; fixed the stale FieldOrder comment.
- DefaultDuration vs pulldown: precise TODO(spec) left (needs the same
parser→title channel as top_field_first).
Tests: BFF-from-measured-flag, measured-CICP-overrides-enum, VobSub size:
line present/omitted, zero-cue SeekHead Void. precommit (1.86) green.
Interlacing is detected upstream (PAL DVD -> R576i) but was dropped in
two places: the video label hardcoded a 'p' suffix, and the muxer never
wrote any scan-type flag, so MediaInfo inferred progressive and reported
576p for a 576i source.
- Add Resolution::is_interlaced() for the R*i variants.
- generate_video_label now branches i/p for the heights that can be
interlaced (1080, 576, 480) instead of always emitting 'p'.
- MkvTrack carries interlaced + field_order; the video serializer emits
FlagInterlaced (0x9A; 1=interlaced, 2=progressive) and, for interlaced
content, FieldOrder (0x9D) - bottom-field-first for PAL 576i,
top-field-first otherwise. Adds the EBML constants.
Fix three DVD video-attribute bugs surfaced by a PAL disc detected as
NTSC:
- PAL/NTSC: parse video_format from VTS_V_ATR bits 5-4, not bits 1-0
(the old mask read permitted_df, so PAL 576i/25fps was mis-detected
as NTSC 480i/29.97). Named consts replace the magic bit positions.
- Anamorphic aspect: write MKV DisplayWidth/Height from the disc's
display_aspect (16:9 720x576 -> 1024x576) instead of square pixels,
so 16:9 DVDs no longer render as 4:3.
- Colour: stamp SD colorimetry (PAL=BT.470BG, NTSC=SMPTE-170M) instead
of BT.709 (HD).
Adds VideoStream.display_aspect (threaded through every muxer) plus
TvSystem/DvdAspect/ColorSpace plumbing, with regression tests. Removes
the deprecated Disc mux set_halt bridge (use with_halt).
Test-hardening release, no runtime changes. Adds spec-grounded unit tests
across the silent-corruption surfaces — UDF/MPLS/CLPI/IFO parsing, BD/DVD
title + extent assembly, AACS/CSS key handling, TS/PS demux + codec parsers,
MKV/EBML container output, the mux pipeline, sector prefetch + decrypt
decorator, drive/SCSI sense decoding, label extraction, and core I/O. Each
test is grounded in the format spec or real on-disc behavior and verified to
fail under a targeted source mutation. No behavior changed.
Add a precommit fixture proving Disc::read_aacs_inputs reads a Long-AD,
multi-extent /AACS/Unit_Key_RO.inf in full — the exact input the online
key-request path depends on (no disc/deploy needed). Make
read_aacs_inputs_from_reader pub(crate) for the test.
Reword internal comments/doc examples to generic descriptions instead of
specific media titles.
Library-wide review-and-fix pass: tightened AACS keydb/handshake/variant
handling and trailing-partial-unit policy, corrected MPLS mark offset and
added UDF allocation bounds, hardened the mux/codec framing and M2TS paths,
guarded SCSI READ CAPACITY short transfers and unified error mapping, added
overflow guards on untrusted disc input, and made prefetch shutdown
deterministic. Release profile now builds with thin LTO + single codegen unit.
correct_truehd_channels() probes the first DECRYPTED access units of the
chosen title (TrueHD PIDs, bounded 8 MiB read of the first extent) and
sets AudioStream.channels from the MLP major sync — fixing the MPLS
audio_format understatement (5.1 declared on a 7.1/Atmos TrueHD track).
Regenerates the basic codec label for the corrected count; richer
editorial labels are left untouched. Wired in resolve.rs input() for
iso:// after decrypt_with (the m2ts is only decryptable post-key), using
a fresh reader so the mux reader is undisturbed. generate_audio_label
made pub(crate).
Two validated audio-correctness fixes (proven on real discs Wicked/
Paddington/Dune/Fight Club via head-captures):
labels: apply_labels now derives the codec/channel descriptor from the
stream's OWN codec/channels unless the parser's codec_hint is BOTH
consistent with it AND richer (e.g. "Dolby Atmos" on a TrueHD stream).
A mis-bound hint ("AC-3 2.0" on a TrueHD track) is rejected and the
stream's own codec used — killing the cross-labeled shuffle (Wicked) and
the compat-core mislabel (Paddington), while keeping rich hints and
normalizing plain ones to uniform marketing names. (codec_hint_consistent
+ codec_hint_adds_detail, 5 tests.)
disc: canonical_title_order gains an audio-richness tiebreak
(lossless > channels > track-count) for titles that tie on
duration+clips — so a movie authored as a full-audio playlist plus a
stereo-only twin (Fight Club 00800 vs 00004) picks the full-audio one
instead of falling to array order.
Pixelogic discs that authored a SEG_MainFeature menu/preview segment
containing a stray commentary token made the parser anchor on that
segment instead of the real FPL_MainFeature playlist. Combined with
'Audio Stream N' placeholder tokens not advancing the audio counter, a
lone eng_ACOM_ commentary at STN slot 4 collapsed onto slot 1 — tagging
the main feature track as Commentary (seen on Wicked: For Good).
Fix:
- Anchor the feature section on the FPL_ playlist when present; only
fall back to SEG_MainFeature on discs with no FPL_ section.
- Count 'Audio Stream N' placeholders so editorial audio tokens get
their true STN ordinal. Subtitle (PG Stream N) numbering is left
unchanged — counting it regressed several corpus discs.
Extract the section/numbering loop into assign_labels with unit tests.
Verified: 11/11 labels-corpus discs still PASS; Wicked commentary now
lands on the AC-3 2.0 stereo track instead of the main TrueHD.
WO-2 (delete SectorReader trait):
- The 0.18 trait split into SectorSource (read-only) and SectorSink
(write-only) is final; the legacy SectorReader alias was a bridge.
- Renames every internal &mut dyn SectorReader (~25 sites) to
&mut dyn SectorSource. The trait method capacity() becomes
capacity_sectors() with a default of 0 (preserves SectorReader's
default-0 behavior).
- Deletes the SectorReader trait, its blanket-to-Source bridge, and
the FileSectorReader type alias. Adds explicit forwarding impls
for Box<dyn SectorSource> and &mut dyn SectorSource so generic
decorators like DecryptingSectorSource<S: SectorSource> compose.
WO-3a (extract Disc::patch):
- Moves Disc::patch (1230 lines) and bytes_bad_in_title from
disc/mod.rs into disc/patch.rs as a split inherent impl. Zero
behavior change — pure mechanical relocation. disc/mod.rs drops
from 3,945 to 2,714 LOC.
WO-6 (partial):
- Deletes src/labels/png_filenames.rs — was a 72-LOC stub with
detect() returning false, never wired into the PARSERS registry.
project docs doc drift fixes (audited 2026-05-13):
- JUMP_BASE_SECTORS: 256→1024 (64 MB base for UHD, not 8 MB)
- PASSN_DAMAGE_THRESHOLD_PCT: 12→6
- PASSN_SKIP_SECTORS_BASE: 64→32
- MAX_RANGE_SECS=180: replaced by proportional range_sectors × 25,
capped at RANGE_BUDGET_CAP_SECS=1800.
Disc-04 (Top Gun: Maverick) re-test 2026-05-11 surfaced a real-world
bdmt_eng.xml where <di:description> contained no prose, only nested
<di:thumbnail href="…"/> elements. The previous parser surfaced
the raw XML fragment as the description string ("<di:thumbnail
href=\"tgm_meta_sm.jpg\" />\\r\\n <di:thumbnail
href=\"tgm_meta_lg.jpg\" />"). Worse than no description.
Fix: filter description candidates that begin with `<` after
trimming. Real prose never starts with an angle bracket; XML-only
content always does. Net: title extraction unaffected (it uses its
own element-priority path); description field drops when it would
otherwise carry XML noise.
Two new bdmt tests, 12 of 12 passing.
Three layered sources of stream labels now, in precedence order:
1. **Framework parser** (paramount/criterion/pixelogic/ctrm/dbp/deluxe)
— editorial labels with purpose/qualifier ("English Atmos",
"Director's Commentary", "English SDH"). High or Medium confidence.
2. **MPLS gap-fill** (`fill_gaps_from_mpls`) — every stream the
playlist references gets at least a basic lang+codec label, even
when the framework parser missed it.
3. **CLPI orphan append** (`append_clpi_orphans`) — streams in
/BDMV/CLIPINF/*.clpi ProgramInfo that NO MPLS playlist references.
Empirical (2026-05-11): ~5% of streams across the 11-disc corpus,
most dramatic on disc-02 (HDMV-only) at 40% CLPI-only.
Orphan numbering: each appended orphan gets
`stream_number = max(existing per type) + N` so playlist-reachable
streams keep their original positions and orphans sort cleanly at
the tail.
Orphan dedup: (stream_type, language, codec_hint) tuple — fuzzier
than PID matching (PIDs aren't carried on StreamLabel) but it's the
only signal available downstream of the gap-fill. False positives
(genuine orphan that happens to share lang+codec with an existing
entry) silently drop, which is the conservative failure mode — the
user-facing display would just see a confusing duplicate otherwise.
`mpls_universal::language_display_name` and `::codec_name` promoted
from private fn to pub(crate) so this module can build orphan labels
with consistent naming.
Tests: 2 new in gap_fill_tests — synthetic-input verification of the
dedup tuple logic and the stream_number assignment. 6/6 tests in the
gap-fill module now passing.
Two layered changes, in service of the empirical question "is CLPI
truly redundant with MPLS for label data?":
1. **clpi.rs ProgramInfo parser**. The existing CLPI parser only
walked the EP map (for sector-range lookups). Added a parser for
the ProgramInfo section's per-stream stream_coding_info table:
pid, coding_type, audio_format/rate, video_format/rate, ISO 639-2
language. Spec layout per libbluray clpi_parse.c. Best-effort —
malformed program_info leaves `streams: vec![]`, EP map keeps
working. `ClipInfo` gains a `streams: Vec<ClpiStream>` field.
2. **labels/clpi_audit.rs**. Diagnostic that walks both
`/BDMV/CLIPINF/*.clpi` (via the new program_info parser) and
`/BDMV/PLAYLIST/*.mpls`, builds a (PID → fields) merged view, and
classifies each row:
- `Match`: both sources agree (same coding_type + language)
- `ClpiOnly`: PID in CLPI but no MPLS playlist references it
(orphan stream on disc — reachable via low-level access, not via menu)
- `MplsOnly`: PID in MPLS but no CLPI lists it (would indicate a
parser bug; verified empirically that this NEVER happens)
- `Divergent`: same PID, different coding_type or language between
sources (playlist re-tagged or attribute encoding mismatch)
Surfaced via `labels-analyze` as `clpi_vs_mpls_audit: {matches,
clpi_only, mpls_only, divergent, total_pids}`. Doesn't affect the
label output — pure diagnostic.
Empirical findings on the 11-disc corpus (excl. disc-04 truncated):
- 226 matches / 0 mpls_only / 8 clpi_only / 5 divergent across 239 PIDs
- 6 of 10 non-truncated discs have CLPI-only streams (orphans)
- disc-02 (HDMV-only) is the most dramatic: 40% of its 5 streams are
CLPI-only — MPLS sees 3, CLPI sees 5
- Conclusion: CLPI is NOT truly redundant. ~5% of streams disc-wide
are CLPI-exclusive. Future work: layer CLPI as a tertiary source
below MPLS in the labels pipeline (orphan streams marked with even
lower confidence than MPLS).
LabelAnalysis gains `chapter_summary: Vec<ChapterSummary>` — one row
per .mpls file in /BDMV/PLAYLIST/, with chapter count (PlaylistMark
entries with mark_type ≤ 1) and approximate playlist duration in
seconds. Sorted by playlist filename.
Sourced from the existing crate::mpls parser (no new format work).
Useful for identifying the main feature playlist at a glance — it's
the one with the longest duration. Verified on disc-11 (Dune Pt 2):
00800.mpls correctly identified as 2h 45m 49s with 18 chapters
amid 30+ shorter playlists.
Doesn't touch the per-title `disc::DiscTitle::chapters` field which
disc::bluray.rs already populates from the same marks during disc
init — this is purely the diagnostic surface for labels-analyze.
When a framework parser (paramount, criterion, pixelogic, ctrm, dbp,
deluxe) is chosen but its label list covers only a subset of the
stream slots MPLS knows about, merge MPLS-derived entries for the
uncovered (stream_type, stream_number) slots. Framework labels keep
their richer fields (purpose=Commentary, codec_hint with "Atmos",
qualifier=Sdh); MPLS only fills slots the framework left unnamed.
Implementation:
- `fn fill_gaps_from_mpls` walks the MPLS label list, pushing any
entry whose (type, number) tuple isn't already in the framework
output. Stable sort by (type, number) groups audios before
subtitles in the merged result.
- Called from both `extract()` and `analyze()`. Skipped when the
chosen parser is itself `mpls_universal` (no gaps possible).
- `LabelAnalysis::gap_fill_added` field reports how many slots got
filled — useful diagnostic from `labels-analyze`.
- `StreamLabelType` gains `Eq + Hash` so the dedup HashSet works.
Tested via 4 new unit tests (155 of 155 labels tests passing, was
151). End-to-end on partial-yield corpus discs:
- disc-05 (Oppenheimer): pixelogic 4/5 already covered, gap_fill_added=0
- disc-11 (Dune Pt 2): pixelogic 8/11 already covered, gap_fill_added=0
(Real-world gap-fill activations are rare in the current corpus because
pixelogic already incorporates MPLS-equivalent data when matching;
the merge is defensive for less-thorough frameworks.)
Per-playlist stream_number counters were resetting between MPLS
files, so a disc with 2 MPLS files each listing the same 8 audio
streams produced labels with stream_number 1..8 then 1..8 again
(dedup kept whichever PID was different, leaving the numbering
visibly broken — multiple "audio1: eng" rows).
Move the counters outside the per-file loop and increment only
when an entry survives dedup. Surviving entries now get dense
1..N numbering across the whole disc per stream_type.
Verified on corpus disc-02 (HDMV-only): was `audio1, audio1, audio1`
for the 3 distinct audio codecs (TrueHD/AC-3/DTS-HD MA), now
`audio1, audio2, audio3`. Same fix applies to disc-01 (12 audio
streams across multiple MPLS) and disc-09 (14 audio streams).
Three new modules in the labels platform, all layered so framework-
specific parsers (paramount, criterion, pixelogic, ctrm, dbp, deluxe)
always take precedence over the universal layer.
**mpls_universal.rs** (~600 LOC, 9 tests): consumes the already-parsed
`crate::mpls::Playlist::streams` and emits StreamLabel entries with
language + codec_hint per stream. Returns `Confidence::Low` (new
variant) so framework parsers' Medium/High always win the registry's
max-by-confidence tiebreaker; MPLS only gets picked when no framework
matched. Closes the "no BD-J disc" case (HDMV-only navigation) that
previously produced zero labels — language and base codec are
spec-mandated in MPLS STN tables on every Blu-ray ever made.
**bdmt.rs** (~350 LOC, 10 tests): reads `/BDMV/META/DL/bdmt_<lang>.xml`
files into a new `DiscMetadata` struct (localized title names per
ISO 639-2 code, descriptions, optional box-set position). Runs
independently of the parser registry — disc-level metadata, not
per-stream, so the registry's confidence selection doesn't apply.
Surfaced on a new `LabelAnalysis::disc_metadata` field.
**png_filenames.rs** (noop stub): pattern documentation + dead-code
detect/parse for future reactivation. Deferred because MPLS already
delivers per-stream lang/codec/type on every disc; PNG filename
language tokens only add studio variant disambiguation (FRC vs FRP,
LAS vs CSP) — niche enough to not justify the implementation cost
right now.
Wiring changes in `mod.rs`:
- New `Confidence::Low` variant (PartialOrd places it below Medium/High)
- New `ParseResult::low()` constructor
- `mpls_universal` appended last to `PARSERS` registry
- `LabelAnalysis::disc_metadata: Option<DiscMetadata>` field
- `analyze()` runs `bdmt::parse` independently and surfaces result
- `pub use bdmt::DiscMetadata` re-export so the labels-analyze tool
in freemkv-tools can construct the JSON payload
Total: 151 of 151 labels tests passing (was 132 — added 19 new).
Replaces the speculative arg-position heuristic with type-presence
detection driven by real disc bytecode. Ground truth captured in
(internal)/research/deluxe-poc/data/ via POC v0.3 binding-
bytecode dumps against disc-01 (Disney) and disc-09 (Warner).
What changed:
1. StackVal::CodingType(String) — new variant. getstatic against
org/bluray/ti/CodingType (the BD-J spec codec enum) now pushes
this, carrying the field name (e.g. DOLBY_LOSSLESS_AUDIO). The
pre-fix code was treating codecs as a Deluxe-internal enum
subclass walk (Phase B), which is the wrong model — codecs are
standard BD-J API references.
2. coding_type_to_codec_hint(field) — new function. Maps
org.bluray.ti.CodingType field names to human-readable codec
strings (DOLBY_LOSSLESS_AUDIO -> "Dolby TrueHD", DOLBY_AC3_AUDIO
-> "Dolby Digital", etc.). Unknown field names pass through
verbatim so future codec values still surface something.
3. find_binding_classes — multi-class variant. Some Deluxe discs
split per-stream tables across two binding classes (audio +
subtitle). Returns top-K candidates by getstatic count, filtered
to >=40% of the top count and capped at 4. Replaces the old
single-class find_binding_class (which was unused after this
change).
4. interpret_streams — rewritten. Args identified by TYPE not
position:
- First EnumRef{kind:"Language"} -> language
- First EnumRef{kind:"Purpose"} -> purpose
- First CodingType(name) -> codec_hint
- First Int(n) -> stream index hint (traced
only; per-type sequential stream_number still wins because BD
spec stream-numbering is anchored on MPLS)
- Construction has CodingType -> Audio stream; otherwise Subtitle
- No Language -> skip (not a stream construction)
This handles BOTH the Disney 5-arg pattern (I, Lbe, Llp, I,
LCodingType) and the Warner 4-arg pattern (I, Law, Lgp,
LCodingType) automatically — same code path because args are
identified by type rather than constructor-signature shape.
5. parse() now walks all binding-class candidates and unions
their constructions before calling interpret_streams. Logs each
candidate at INFO with getstatic_count for diagnosis.
Tests:
- 2 new tests verify the CodingType -> codec_hint mapping for
known + unknown field names.
- Existing interpret_streams tests updated to use the new
signature (dropped CodecTable arg).
- Audio-emission test rewritten to use CodingType arg instead of
the old binding_type substring-match approach.
Confidence is still Medium for now (single-corpus verification);
ready to promote to High once tested against a third Deluxe disc.
Precommit (cargo +1.86 fmt + clippy + test) green.
Closes the final two audit items from this session.
labels::apply_labels: factored out of apply() so the matching logic
is unit-testable without needing a SectorReader / UdfFs. 11 new
tests in apply_tests cover:
- codec_hint + variant flow through to AudioStream.label
- purpose set on audio with no label English text
- name fallback only when purpose=Normal (CLI owns purpose i18n)
- subtitle SDH qualifier set; forced flag flipped on Forced
- per-type 1-based indexing (audio #2 maps to 2nd audio stream,
not 2nd stream overall)
- labels for nonexistent streams are no-ops
- empty labels list leaves streams untouched
- fill_defaults generates audio + video labels; preserves existing
class_reader: robustness smoke tests. ClassFile::parse must NEVER
panic on adversarial input — only return Err. 9 new tests:
- empty input
- short magic (0..4 bytes)
- wrong magic
- truncated after magic
- bad CP tag
- truncated UTF-8 in CP
- 200 random byte buffers (deterministic xorshift)
- 100 magic + random tail (most adversarial — magic check passes,
everything else garbage)
- instructions iter on random code (200 buffers)
- instruction_size on every opcode 0..255 with varied tail buffers
- modified_utf8 on random byte buffers (500)
The xorshift PRNG keeps the tests deterministic (no rand dep) and
reproducible — failures will be the same buffer every time. This is
the lightweight alternative to a cargo-fuzz setup; if/when we adopt
cargo-fuzz, these tests stay as regression cases.
All 451 tests passing on cargo +1.86 fmt + clippy + test.
Replaces two near-duplicate hand-rolled XML scrapers in paramount.rs
and criterion.rs with a single labels::xml module that's robust to:
- Case-insensitive tag / attribute names ('<Playlist>' matches the
same as '<playlist>'; 'Name=...' matches 'name=...').
- XML namespace prefixes (matches '<ns:tag>' for tag='tag').
- Arbitrary whitespace inside open tags and around '=' separators
('<tag name = "X">' works).
- Both quote styles for attribute values (" and ').
- Self-closing tag forms ('<tag/>' and '<tag />').
- '>' chars inside quoted attribute values (no premature end-of-tag).
Three functions:
xml::attr(element, name) -> Option<String>
Extract attribute value from an open-tag fragment.
xml::text(xml, tag) -> Option<String>
Trimmed text content of first <tag>...</tag>.
xml::find_element(xml, tag, from) -> Option<(start, end)>
Locate next <tag>...</tag> for iteration; handles self-closing.
22 unit tests cover the robustness properties: case-insensitivity,
namespace stripping, whitespace tolerance, quote styles, self-close
forms, no-substring-false-positive (looking for 'lang' must NOT
match 'lang_id' or 'language'), '>' inside quoted attrs, iteration
across repeated elements.
paramount.rs: drops local extract_attr; find_feature_playlist now
walks xml::find_element('playlist', ...) so it works regardless of
case and self-closing style. Pre-refactor: required exactly
'<playlist ' (single space, exact case) and '/>' for self-close.
criterion.rs: drops local extract_tag; parse_stream_infos and
parse_playback_config iterate via xml::find_element. Same case-
sensitivity + namespace gains. The 'COMMENTARY' / 'SDH' / 'DS'
content-value match is now case-insensitive too (previously a disc
authored with 'commentary' would have been miscategorized as Normal).
Pre-refactor known failure modes (none observed yet, but trivial
to trip on a future disc): vendor switches whitespace around '=',
uses single quotes, capitalizes a tag, prefixes a namespace. All
now handled.
Out of scope by design: XML entity decoding (&, <), CDATA
sections, comments, processing instructions. None observed in BD-J
authored label data. If a future disc trips them, the entity
decoder is a localized addition.
Precommit (cargo +1.86 fmt + clippy + test) green.
Completes the Deluxe parser pipeline. Phase A (master enums) was
already shipping; this commit lands Phases B/C/D so the parser now
emits per-stream StreamLabel records on Deluxe-authored discs.
Phase B (decode_codec_enum): walks the codec enum's subclass
references (one .class per codec ordinal) and extracts the codec
name string from each subclass's constant pool. Heuristic: pick the
first Utf8 entry that's uppercase + underscored + >=4 chars, or one
of the known codec roots (ATMOS/DOLBY/DTS/TRUEHD/MLP/AC3/EAC3/PCM)
when no underscored candidate is found. CodecTable maps ordinal ->
codec string; empty string for ordinals where extraction failed
(logged via tracing, not fatal).
Phase C (find_binding_class): identifies the class that builds the
per-stream label table by counting getstatic operations targeting
any of the master enum classes from Phase A. Class with the highest
count >= 4 wins. Threshold is empirical (real binding classes have
50+ matches; floor of 4 admits small discs while rejecting incidental
single-reference classes).
Phase D (decode_binding + BindingDecoder): symbolic stack machine
that walks the binding class's <clinit> bytecode. Handles:
- constant pushes: iconst_<n>/bipush/sipush/ldc(Integer)
- new <X>: pushes uninit-object marker
- dup: stack copy
- getstatic <Y.Z>: pushes EnumRef when Y is in MasterEnumTable,
else Unknown
- invokespecial X.<init>(...)V: pops args per descriptor; when the
receiver is NewObj(X), emits a Construction { binding_type: X,
args: [...] }
- invokevirtual/invokestatic/invokeinterface: pop args per
descriptor, push return placeholder unless void
- pop/pop2/aastore/putstatic/putfield: standard stack effects
- branches/returns: clear stack (conservative resync — binding
<clinit> is straight-line in practice)
parse_method_arg_count: JVMS field-descriptor parser, handles
primitives, references (L...;), arrays ([...).
interpret_streams: converts Constructions to StreamLabels using
the master enum table + CodecTable. Each construction with a
Language ref becomes a stream. Audio when codec_hint resolves via
binding_type substring match against CodecTable; subtitle otherwise.
Purpose ordinal -> LabelPurpose via the verified Deluxe Purpose enum
order (Normal/Commentary/PiP/Trivia/Descriptive/Score/NoForced/
NoForcedDescriptive). Stream index = sequential per type. Language
goes through vocab::lang for ISO code + variant.
deluxe::parse now returns Some(ParseResult::medium(labels)) when
all four phases produce labels. Medium confidence — the bytecode
mechanism is rigorously tested but the signal-to-StreamLabel
mapping (which arg is which, audio vs subtitle classification) is
heuristic until corpus binding-class bytecode confirms the exact
pattern.
Test coverage: 13 new unit tests in deluxe.rs
parse_method_arg_count: 3 tests (basic types, references, malformed)
BindingDecoder: 4 tests (simple construction, with int pushes,
skips unmatched invokespecial, resolves master-enum ordinal)
interpret_streams: 4 tests (subtitle on no codec, audio on codec
match, purpose routing, skips no-language)
MasterEnumTable: 3 tests (resolve, value, class_name_set)
extract_codec_name: 1 test (uppercase+underscore matching)
class_reader.rs gained a #[cfg(test)] ConstantPool::from_entries
test-only constructor so Phase D tests can build synthetic CP
fixtures without writing raw .class bytes.
Precommit (cargo +1.86 fmt + clippy + test) green.
Replaces 'first-match-wins by array order' with 'highest-confidence-
wins, array order tiebreaker'. Removes the arbitrariness when more
than one parser can claim a disc (e.g. one with both
bluray_project.bin and playlists.xml).
New types in labels::mod:
pub enum Confidence { Medium, High }
pub struct ParseResult { labels: Vec<StreamLabel>, confidence }
ParseResult::high(labels) / ::medium(labels) constructors
Parser signature change: every parse() now returns
Option<ParseResult> instead of Option<Vec<StreamLabel>>. Updated all
six parsers in lockstep:
paramount: High (fully structured XML)
criterion: High (fully structured XML)
pixelogic: High by default, Medium when an unknown token component
is encountered (the skip-unknown path now propagates the
coverage gap to the caller instead of silently degrading)
ctrm: High (structured key-value)
dbp: High (anchor scan with vocab routing)
deluxe: still returns None pending Phase D — signature aligned
Registry behavior:
extract() iterates all detect-positive parsers, picks highest
Confidence with non-empty labels. Equal confidence falls to array
order (deterministic). Same selection logic in analyze().
LabelAnalysis grew a confidence: Option<Confidence> field so the
diagnostic surface (freemkv-tools labels-analyze) exposes which
confidence tier the selected parser claimed. labels-analyze JSON
and labels-corpus-check structural diff both gained the field.
Precommit (cargo +1.86 fmt + clippy + test) green.
Three targeted fixes from a second-pass audit of the labels module.
1. vocab::lang now returns Option<LangInfo> with both code AND a
human-readable variant string. Pre-fix: 'Brazilian Portuguese 5.1'
became language=por, variant='', dropping the dialect info the
disc had explicitly authored. Post-fix: language=por,
variant='Brazilian' — matches the convention pixelogic / ctrm /
criterion already use for their region variants. dbp now
populates StreamLabel::variant from this. Compound table grew a
3-tuple (needle, code, variant); bare matches still return
variant=''.
2. dbp and deluxe had duplicated detect() boilerplate (any top-level
.jar in /BDMV/JAR/). Both now call jar::has_any_top_level_jar.
The trait-level detect contract — see super::PARSERS — can't peek
inside a jar without a SectorReader, so loose-detect-plus-real-
check-in-parse is the unavoidable pattern for jar-content parsers.
Consolidating in jar.rs at least makes the duplication visible.
3. mod.rs comment about parser ordering said 'dbp last'; deluxe is
actually now last. Updated to explain the dbp-before-deluxe order
is by cost (cp-iteration cheaper than bytecode walking when Phase
D lands).
Plus a registry-level lock test in mod.rs::registry_tests — asserts
the PARSERS array order is exactly [paramount, criterion, pixelogic,
ctrm, dbp, deluxe]. This was previously implicit; if someone reorders
the array (which changes which parser wins on overlapping signals),
unit tests would have stayed green. Now they fail with an explanatory
message about why the order matters.
Audit findings deferred to follow-ups (each its own commit + design
discussion):
- Stronger detect contract — current loose-detect-real-check pattern
is forced by SectorReader-not-in-detect-signature; could be fixed
by changing the trait to take an Option<&mut dyn SectorReader> or
similar.
- Per-parser confidence scoring — registry currently first-match-wins.
A high-confidence parser ought to beat a low-confidence one
regardless of array order.
- class_reader fuzzing — handles malformed input via Result but no
adversarial corpus yet.
Precommit (cargo +1.86 fmt + clippy + test) green.
Closes the platform unification: every label parser now routes
purpose/qualifier/codec classification through one source of truth
(vocab.rs) instead of N hand-rolls, and every binary-blob byte
scanner goes through one helper (text::extract_ascii_strings).
pixelogic.rs:
- Drop local extract_strings (~20 lines) — use text::extract_ascii_strings.
- HARDENING: replace with skip-unknown-component +
trace log. Pre-refactor behavior: any single uncatalogued token part
(e.g. a future codec ID, new framework variant) silently dropped the
entire stream record. New behavior: skip just the unknown part,
surface what we know about the stream.
- 8 new unit tests cover basic audio/subtitle paths, commentary,
descriptive, region variant, the new skip-unknown-component
regression, and the non-audio/non-subtitle early-out.
ctrm.rs:
- Replace with
vocab::purpose(&name). Now word-boundary matched — 'Commenter Pro
Track' no longer false-matches Commentary.
- Replace with vocab::qualifier(&name).
Same word-boundary tightening, plus picks up Forced and
DescriptiveService for free.
- Preserved structural commentary signal via
as a fallback when name is silent (e.g. 'audio_commentary_1.name=Track 2').
- 6 new unit tests including the 'Commenter' false-positive regression
and the SDH-only-on-subtitles boundary.
text.rs:
- Drop module-level dead_code allow now that pixelogic uses
extract_ascii_strings.
Net: all 5 framework parsers now on the unified platform. Future work
(deluxe Phase D, paramount/criterion XML hardening) builds on the
same scaffolding.
Precommit green.
Closes the 'detected but no parser' gap on Deluxe-authored BD-J discs
(com/bydeluxe/ package signature; ~20% of UHD discs in our corpus per
the 2026-05-10 11-disc capture session).
What ships in this commit:
- detect() — registers the parser in the chain (loose pre-check at the
/BDMV/JAR/ level; real signal in parse via has_path_prefix on
'com/bydeluxe/').
- Phase A master enum identification — walks every .class's <clinit>
ldc sequence and matches against framework-stable fingerprints for:
Language (70 ldcs starting English/French/Spanish/Dutch)
Purpose (8 ldcs starting Normal/Commentary/PiP/Trivia)
VideoFormat (7 ldcs starting HD/HDR10 Plus/HD Dolby)
Region (22 ldcs starting USA_D1/LIC1/LIC2/LIC3 — Disney-only)
Studio (6 ldcs starting Disney/Marvel/Pixar — Disney-only)
All identifications verified out-of-tree on corpus disc-01 (Disney,
The Amateur) and disc-09 (Warner, Dune Part 1) via the standalone
POC.
- Phase B structural skeleton (find_codec_enum) — identifies the
codec enum class by structural signature (>=20 'new' ops, 0 ldcs),
returns the ordered subclass list. Codec string extraction from
subclasses is dead-coded pending the follow-up commit.
What does NOT ship yet:
- Phase D (per-stream binding-class decoder). parse() returns None
intentionally — the master enums alone don't yield StreamLabels
without the streamTable.put(...) bytecode walker. analyze() will
show 'deluxe' in parsers_detected with the enum identification in
tracing logs, so the analyzer reports honestly: 'detected, can't
emit labels yet' rather than silent failure.
Why ship A without D: A is proven on real corpus discs; D's design
needs ground-truth binding bytecode from at least 2 corpus discs
side-by-side to verify the stack-machine pattern. The Phase A
infrastructure (master enum identification + ordinal->name table)
is what D will consume — landing it now unblocks D's design without
holding back the parser registration.
5 unit tests cover the fingerprint matcher logic + a roster lock that
forces explicit consideration when adding/removing fingerprints.
Precommit green.
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.
New modules:
vocab.rs expanded from 27 -> ~370 lines
+ lang(text) -> Option<&'static str> (English/multi-word
-> ISO 639-2; ~45
languages, compound
phrases like
'Brazilian Portuguese'
and 'Castilian Spanish')
+ purpose(text) -> LabelPurpose (Commentary,
Descriptive, Score,
Ime; word-boundary
matched)
+ qualifier(text) -> LabelQualifier (SDH, Forced,
DescriptiveService)
+ has_word internal primitive — enforces word-boundary
matching so 'Commenter' no longer matches 'commentary' and
'engineering' no longer matches 'english'. Existing parsers
used .contains() and got lucky on the corpus; vocab now
guarantees the boundary in one place. 20+ unit tests.
text.rs NEW (~85 lines)
+ extract_ascii_strings(data, min_len) — promoted from two
near-duplicate copies (pixelogic min=4, dbp min=5);
threshold passed in. 7 unit tests including
trailing-without-terminator + high-bit-byte handling.
jar.rs NEW (~120 lines)
+ for_each_jar(reader, udf, fn) — walk every top-level
.jar under /BDMV/JAR/,
yield to callback.
+ has_path_prefix(archive, prefix) — cheap 'is this MY
framework's jar?' check
via central-dir filenames.
+ for_each_class(archive, fn) — parse every .class entry
through class_reader,
yield (name, &ClassFile).
+ try_each_class(archive, fn) — same with early-return on
first Some(R) match.
Refactored:
dbp.rs v2 on the new platform:
- dropped extract_printable raw byte scan
- dropped its own English -> ISO 639-2 map
- dropped its own parse_attributes hand-roll
+ iterates CpInfo::Utf8 via class_reader (structurally clean,
no false-positive risk from method bytecode bytes)
+ routes language/purpose/qualifier through vocab
All 7 prior dbp tests still pass; +2 new ones cover
vocab routing.
dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.
Precommit green (cargo +1.86 fmt + clippy + test).
Foundation for label parsers that need structured access to .class
files inside /BDMV/JAR/<x>.jar. Replaces noak (~3KLOC dep) with a
~1000-line std-only reader.
Public API:
- ClassFile::parse(&[u8]) -> Result<ClassFile>
- ConstantPool::{get, utf8, class_name, string, integer, member_ref, iter}
- Member::code(&pool) -> Option<CodeAttribute>
- CodeAttribute::instructions() -> Instructions iterator
- Instruction::{name, operand_u8, operand_u16, cp_index}
- Opcode constants (LDC, AASTORE, NEW, GETSTATIC, INVOKESPECIAL, ...)
Spec coverage:
- Constant pool: all 17 tag types incl. Long/Double 2-slot quirk
- Modified UTF-8 incl. 0xC0 0x80 -> U+0000 special case
- Bytecode iteration with full opcode size table
- Variable-length tableswitch / lookupswitch / wide
12 unit tests cover the opcode table edge cases (padded switch tables,
wide-iinc 6-byte form), modified-UTF-8 decoder, and iterator
stop-on-truncated behaviour.
Module is currently #![allow(dead_code)] — the public API is staged
for labels::deluxe (Phases A-E bytecode walker) and a labels::dbp
refactor onto the constant-pool iterator. Tests exercise the API
in isolation. The allow comes off as those callers land.
Also fixes two pre-existing clippy lints that 1.86's stricter checks
flagged after I touched the labels module:
- src/mux/disc.rs: while-let-loop in test fixture
- tests/pass_n_size_aware_skip.rs: type_complexity in helper signature
Precommit (cargo +1.86 fmt + clippy + test) green.
5th BD-J authoring framework recognized. Discriminator: any top-
level .jar in /BDMV/JAR/ that contains com/dbp/ package paths.
Identified during the 2026-05-10 corpus session via string-mining
disc-07's BD-J jar — perm files reference bd-live.magpictures.com
(Magnolia / Magnet Releasing).
Stream labels live as plain ASCII strings inside compiled .class
files in the form
LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,...
HTextField,Subtitle1,English SDH,Fontstrip_Composite,...
ATextField,Subtitle0,None,Fontstrip_Composite,...
— a quirk of the menu-rendering layer encoding TextField positions
and content as constant strings the Java compiler retained in the
class string pool. The leading single-letter prefix is string-pool
ordering noise; parser anchors on `TextField,`. Subtitle0 is the
disable-subtitles button and is skipped.
The parser:
- reads each top-level .jar via udf.read_file
- opens it with the existing zip dependency
- confirms com/dbp/ presence in the central directory
- walks .class entries, extracts printable strings, matches the
`TextField,(Audio|Subtitle)<N>,<label>,...` pattern
- maps human-readable language names ("English", "Castilian
Spanish", "Brazilian Portuguese", "Canadian French", ...) to
ISO 639-2 codes via a parser-local table (per the rules-of-
engagement memo, each parser knows its own format)
- preserves the full disc-authored label string in `name` so
consumers display it raw without the lib guessing further
structure
- detects SDH / Forced qualifiers and Commentary / Descriptive
purposes from substring matches; everything else falls through
to fill_defaults using BD-spec MPLS data
Verified live on the corpus: disc-07 (Civil War UHD) now matches
parser=dbp with 3 audio + 2 subtitle labels, exactly the count
visible in the disc's authored TextField definitions and what BD
spec MPLS reports.
Limitation: dbp's detect() returns true for ANY top-level .jar in
/BDMV/JAR/ (every BD-J disc has one), since the discriminator
trait function takes only `&UdfFs` and can't read jar contents.
parse() does the real com/dbp/ check — a non-dbp disc gets
parsed-as-dbp, archive_has_dbp returns false, parse() returns
None, and we fall through. Diagnostic noise: parsers_detected
includes "dbp" on non-dbp BD-J discs. Real fix is refactoring the
DetectFn signature to take a SectorReader; deferred.
7 unit tests cover the TextField extraction, language detection
(simple + compound: "Brazilian Portuguese", "Castilian Spanish",
"Canadian French", "Latin American Spanish", "Australian English"
plus the disc-corpus typo "Austrailian English"), SDH/Forced/RNIB
qualifier detection, and Commentary/Descriptive purpose detection.
Don't-guess discipline preserved: unknown languages return ""
(consumer falls back to MPLS spec data via fill_defaults).
Adds `parsers_detected: Vec<&'static str>` to `LabelAnalysis`. Records
every parser whose discriminator matched, regardless of whether its
parse() then returned Some/None.
Why: when `parser=None` we currently can't tell apart:
(a) no parser recognized this disc — missing parser, candidate for a
new module
(b) a parser recognized it but parse() returned None / empty —
capture truncated, or genuine empty authoring data, or a parser
bug
Surfaced concretely on the 11-disc capture session 2026-05-10:
disc-04 had `bluray_project.bin` in jar_inventory (pixelogic detect()
returned true) but parse() returned None because file content was
past the 1 GB capture window. Old API: parser=None — looked like
"missing parser." New API: detected=[pixelogic], parser=None — clearly
"capture problem, not a parser gap."
The tracing log line on the no-parser-emitted-labels path now
distinguishes the two cases too.
No behavior change to production rip path. extract() is unchanged;
only the diagnostic analyze() returns the richer result.
Matthew Jackson