Adds the observability we need to debug wedge incidents from logs
alone — without needing to enable verbose TRACE-level SCSI tracing.
Goal stated by user: "when error occurs we can debug and code
correctly."
Pre-fix the WARN log on each read error showed only sense codes
and consecutive_failures. Missing: timing context (was the failed
read fast or slow?), gap to previous events (cumulative vs.
immediate failure?), and family transitions (did the drive just
flip into wedge mode, or has it been there?).
New fields on ReadCtx (no caller signature change):
last_success_at: Option<Instant>
last_error_at: Option<Instant>
last_error_family: Option<SenseFamily>
total_errors: u64
total_reads_ok: u64
zones_entered: u64
jumps_taken: u64
in_damage_zone: bool
New SenseFamily enum (NotReady / Medium / Hardware / IllegalRequest
/ Other) with is_wedge_family predicate.
handle_read_error WARN log now carries:
consecutive_failures
consecutive_outer_failures
ms_since_last_error NEW gap between this and previous error
ms_since_last_success NEW gap to last good read
total_errors NEW aggregate this pass
total_reads_ok NEW
wedge_count
sense_family NEW typed category, easier to filter
sense_key / asc / ascq (existing)
NEW WARN log "wedge_transition" fires once when the sense family
changes from non-wedge to wedge (Medium to Hardware/IllegalRequest).
That's the moment the drive's firmware flipped into fast-fail
mode. Single timestamped event in the log so post-mortems can
pinpoint the transition without scanning thousands of TRACE lines.
Worked example: if the next wedge incident shows
read_error ms_since_last_success=18234 ms_since_last_error=null
read_error ms_since_last_success=28000 ms_since_last_error=10000
read_error ms_since_last_success=43000 ms_since_last_error=68
(drive returned <100ms = wedge symptom)
wedge_transition errors_in_zone=5 ms_since_last_success=43000
we can immediately tell cumulative damage, 5 errors over 43 s,
drive went into fast-fail mode at the 5th. If instead we see
read_error ms_since_last_success=200 ms_since_last_error=null sense_family=Hardware
wedge_transition errors_in_zone=1
the wedge was triggered by ONE read at a physically-bricked LBA
(immediate fast-fail, no warm-up).
These two patterns demand different tuning responses (longer
pause vs. larger initial jump), and now we can distinguish them
from a single WARN log line each instead of needing TRACE
verbose for the whole rip.
Plus jumps_taken / zones_entered counters that feed an end-of-pass
INFO summary (PassSummary). Caller invokes pass_summary at sweep
end and logs structured stats: "Pass 1 saw N errors / M ok reads
/ K zones / J jumps". Single-line post-mortem for any rip.
No caller signature change (timing is internal to the handler;
end-of-pass summary is a new method callers opt into). Precommit
green; 433+ tests pass. Staged for the 0.18.10 release once we
have user-validation data on 0.18.9's avoidance tuning.
Complements the wedge-skip backstop (fbdb50c) with proactive
avoidance so we don't HIT the wedge in the first place. User's
take after seeing the Dune Pt 2 rip wedge at 48%: 'we shouldn't be
wedging.'
Empirical observations from the 23:09:12-23:09:55 wedge timeline:
5 read errors over 43 s, ~8 s apart (drive's own ECC recovery
takes 5-10 s per failure). Not 'hammering' in any usual sense,
but cumulative firmware-state buildup over 5 in-cluster errors
was enough to tip the BU40N into wedge mode at the 5th error.
Damage cluster spanned ~140 MB (LBAs 19.898M-19.965M). Current
damage-jump base of 256 sectors × batch=32 = 16 MB first jump,
doubling to 32 MB, 64 MB... Each jump landed BACK INSIDE the
140 MB cluster, exposing the drive to MORE in-cluster errors.
Two avoidance levers:
1. Inter-error pause on Pass 1 (PASS_1_FAIL_PAUSE_SECS = 5 s):
pre-fix Pass 1 ran pause_secs=0 on all errors to 'zoom past'
damage zones. Successful reads still zoom at zero pause — the
pause applies only to FAILED reads, giving the drive's firmware
cool-down between cluster exposures. Cost: ~5 s per scattered
failure (~30-60 s total on a damage cluster); trivial vs.
crashing the rip at 48%.
2. Larger damage-jump base (JUMP_BASE_SECTORS = 1024, up from
256): first jump at batch=32 now covers 64 MB instead of 16 MB,
second jump 128 MB instead of 32 MB. Two jumps clear 192 MB —
well past most single-cluster damage patterns. Smaller jumps
were landing inside the cluster and adding to the wedge counter.
Plus a halt-aware sleep helper (sleep_secs_or_halt) so the new
inter-error pause doesn't degrade halt response time. Halt poll
granularity 100 ms — halt fires within ~100 ms regardless of
remaining pause time. Updated three sleep call sites in disc/mod.rs
(SkipBlock pause, JumpAhead post-pause, Retry pause).
The wedge-SKIP backstop (fbdb50c) stays — combined with this
avoidance work, the flow becomes:
damage cluster encountered →
pause 5 s, mark NonTrimmed →
second failure →
pause 5 s, mark NonTrimmed →
...
threshold hit →
damage-jump 64 MB (clears 95% of clusters) →
if jump lands in another cluster: 128 MB next jump →
only if drive STILL wedges after all this:
wedge-skip kicks in (1 GB jump + 30 s cooldown × 16 budget)
Tests:
pass_1_pauses_briefly_on_skip_for_wedge_avoidance — locks the
new 5 s pause behavior in place (replaces the old pause=0 test).
integration test threshold bumped from 5 s to 60 s with comment
explaining the new bound is 'not infinite' rather than
'milliseconds-fast'.
All 433+ tests green on cargo +1.86 fmt + clippy + test.
Precommit green.
Closes the final two audit items from this session.
labels::apply_labels: factored out of apply() so the matching logic
is unit-testable without needing a SectorReader / UdfFs. 11 new
tests in apply_tests cover:
- codec_hint + variant flow through to AudioStream.label
- purpose set on audio with no label English text
- name fallback only when purpose=Normal (CLI owns purpose i18n)
- subtitle SDH qualifier set; forced flag flipped on Forced
- per-type 1-based indexing (audio #2 maps to 2nd audio stream,
not 2nd stream overall)
- labels for nonexistent streams are no-ops
- empty labels list leaves streams untouched
- fill_defaults generates audio + video labels; preserves existing
class_reader: robustness smoke tests. ClassFile::parse must NEVER
panic on adversarial input — only return Err. 9 new tests:
- empty input
- short magic (0..4 bytes)
- wrong magic
- truncated after magic
- bad CP tag
- truncated UTF-8 in CP
- 200 random byte buffers (deterministic xorshift)
- 100 magic + random tail (most adversarial — magic check passes,
everything else garbage)
- instructions iter on random code (200 buffers)
- instruction_size on every opcode 0..255 with varied tail buffers
- modified_utf8 on random byte buffers (500)
The xorshift PRNG keeps the tests deterministic (no rand dep) and
reproducible — failures will be the same buffer every time. This is
the lightweight alternative to a cargo-fuzz setup; if/when we adopt
cargo-fuzz, these tests stay as regression cases.
All 451 tests passing on cargo +1.86 fmt + clippy + test.
Pre-fix: when the drive returned HARDWARE_ERROR or ILLEGAL_REQUEST
during Pass 1 sweep, libfreemkv immediately returned ReadAction::
AbortPass. Autorip surfaced this as a fatal error and stopped the
rip at whatever progress percentage Pass 1 had reached — typically
40-50%. On a disc with one physical-damage cluster, the user would
see Pass 1 die at ~48% with the cryptic message 'E6000: <lba>
0x02/0x04/0x3e' and have no rip output to work with.
Root cause analysis: BU40N firmware transitions into a fast-fail
state when it hits cumulative read failures in a small LBA range —
returns HARDWARE_ERROR for every subsequent read near that LBA, even
sectors that aren't physically damaged. Per project docs 'Bad-sector
handling' rule #2, 'Recovery requires eject+reload OR significant
cool-down.' Aborting on first wedge throws away the rest of the
disc; the right response is to SKIP the wedged region (mark as
NonTrimmed for Pass N), pause for drive cooldown, and continue.
Fix: in handle_read_error, the HARDWARE_ERROR / ILLEGAL_REQUEST arm
now branches on bisect_on_marginal:
Pass 1 (bisect_on_marginal=false): JumpAhead with WEDGE_JUMP_SECTORS
(1 GB at 2048 bytes/sector) and WEDGE_PAUSE_SECS (30 s cooldown).
Tracks wedge_count in ReadCtx; resets on any successful read.
Truly aborts only after WEDGE_ABORT_THRESHOLD (16) consecutive
wedges with no good read in between — generous enough to clear
most physical-damage clusters, bounded enough to not loop forever
on a permanently bricked drive.
Pass N (bisect_on_marginal=true): unchanged AbortPass. Pass N's
job is single-sector recovery; if the drive won't talk near a
specific LBA, skipping doesn't help. Pass N exits and lets the
outer layer decide retry/eject/surface.
5 unit tests cover the new policy:
pass_1_hardware_error_jumps_ahead_not_aborts — JumpAhead emitted
with correct sectors+pause, wedge_count incremented.
pass_1_hardware_error_aborts_after_threshold — AbortPass kicks in
on the WEDGE_ABORT_THRESHOLD-th consecutive wedge.
pass_1_good_read_resets_wedge_count — on_success clears
wedge_count; subsequent wedge gets fresh skip budget.
pass_n_hardware_error_still_aborts — Pass N's AbortPass behavior
intact.
pass_1_illegal_request_also_routes_to_wedge_skip — both wedge
sense families get the skip treatment.
Impact: on the Dune Pt 2 disc that consistently wedged at 48%
(physical damage at LBA ~19.9M), Pass 1 will now jump ahead 1 GB
on the wedge, give the drive 30 s cooldown, and continue scanning
the rest of the disc. The damaged region becomes Pass N's job to
revisit. Worst case if the drive stays wedged: 16 GB of NonTrimmed
disc area before honest AbortPass.
Precommit (cargo +1.86 fmt + clippy + test) green; 430 passing.
Replaces two near-duplicate hand-rolled XML scrapers in paramount.rs
and criterion.rs with a single labels::xml module that's robust to:
- Case-insensitive tag / attribute names ('<Playlist>' matches the
same as '<playlist>'; 'Name=...' matches 'name=...').
- XML namespace prefixes (matches '<ns:tag>' for tag='tag').
- Arbitrary whitespace inside open tags and around '=' separators
('<tag name = "X">' works).
- Both quote styles for attribute values (" and ').
- Self-closing tag forms ('<tag/>' and '<tag />').
- '>' chars inside quoted attribute values (no premature end-of-tag).
Three functions:
xml::attr(element, name) -> Option<String>
Extract attribute value from an open-tag fragment.
xml::text(xml, tag) -> Option<String>
Trimmed text content of first <tag>...</tag>.
xml::find_element(xml, tag, from) -> Option<(start, end)>
Locate next <tag>...</tag> for iteration; handles self-closing.
22 unit tests cover the robustness properties: case-insensitivity,
namespace stripping, whitespace tolerance, quote styles, self-close
forms, no-substring-false-positive (looking for 'lang' must NOT
match 'lang_id' or 'language'), '>' inside quoted attrs, iteration
across repeated elements.
paramount.rs: drops local extract_attr; find_feature_playlist now
walks xml::find_element('playlist', ...) so it works regardless of
case and self-closing style. Pre-refactor: required exactly
'<playlist ' (single space, exact case) and '/>' for self-close.
criterion.rs: drops local extract_tag; parse_stream_infos and
parse_playback_config iterate via xml::find_element. Same case-
sensitivity + namespace gains. The 'COMMENTARY' / 'SDH' / 'DS'
content-value match is now case-insensitive too (previously a disc
authored with 'commentary' would have been miscategorized as Normal).
Pre-refactor known failure modes (none observed yet, but trivial
to trip on a future disc): vendor switches whitespace around '=',
uses single quotes, capitalizes a tag, prefixes a namespace. All
now handled.
Out of scope by design: XML entity decoding (&, <), CDATA
sections, comments, processing instructions. None observed in BD-J
authored label data. If a future disc trips them, the entity
decoder is a localized addition.
Precommit (cargo +1.86 fmt + clippy + test) green.
Completes the Deluxe parser pipeline. Phase A (master enums) was
already shipping; this commit lands Phases B/C/D so the parser now
emits per-stream StreamLabel records on Deluxe-authored discs.
Phase B (decode_codec_enum): walks the codec enum's subclass
references (one .class per codec ordinal) and extracts the codec
name string from each subclass's constant pool. Heuristic: pick the
first Utf8 entry that's uppercase + underscored + >=4 chars, or one
of the known codec roots (ATMOS/DOLBY/DTS/TRUEHD/MLP/AC3/EAC3/PCM)
when no underscored candidate is found. CodecTable maps ordinal ->
codec string; empty string for ordinals where extraction failed
(logged via tracing, not fatal).
Phase C (find_binding_class): identifies the class that builds the
per-stream label table by counting getstatic operations targeting
any of the master enum classes from Phase A. Class with the highest
count >= 4 wins. Threshold is empirical (real binding classes have
50+ matches; floor of 4 admits small discs while rejecting incidental
single-reference classes).
Phase D (decode_binding + BindingDecoder): symbolic stack machine
that walks the binding class's <clinit> bytecode. Handles:
- constant pushes: iconst_<n>/bipush/sipush/ldc(Integer)
- new <X>: pushes uninit-object marker
- dup: stack copy
- getstatic <Y.Z>: pushes EnumRef when Y is in MasterEnumTable,
else Unknown
- invokespecial X.<init>(...)V: pops args per descriptor; when the
receiver is NewObj(X), emits a Construction { binding_type: X,
args: [...] }
- invokevirtual/invokestatic/invokeinterface: pop args per
descriptor, push return placeholder unless void
- pop/pop2/aastore/putstatic/putfield: standard stack effects
- branches/returns: clear stack (conservative resync — binding
<clinit> is straight-line in practice)
parse_method_arg_count: JVMS field-descriptor parser, handles
primitives, references (L...;), arrays ([...).
interpret_streams: converts Constructions to StreamLabels using
the master enum table + CodecTable. Each construction with a
Language ref becomes a stream. Audio when codec_hint resolves via
binding_type substring match against CodecTable; subtitle otherwise.
Purpose ordinal -> LabelPurpose via the verified Deluxe Purpose enum
order (Normal/Commentary/PiP/Trivia/Descriptive/Score/NoForced/
NoForcedDescriptive). Stream index = sequential per type. Language
goes through vocab::lang for ISO code + variant.
deluxe::parse now returns Some(ParseResult::medium(labels)) when
all four phases produce labels. Medium confidence — the bytecode
mechanism is rigorously tested but the signal-to-StreamLabel
mapping (which arg is which, audio vs subtitle classification) is
heuristic until corpus binding-class bytecode confirms the exact
pattern.
Test coverage: 13 new unit tests in deluxe.rs
parse_method_arg_count: 3 tests (basic types, references, malformed)
BindingDecoder: 4 tests (simple construction, with int pushes,
skips unmatched invokespecial, resolves master-enum ordinal)
interpret_streams: 4 tests (subtitle on no codec, audio on codec
match, purpose routing, skips no-language)
MasterEnumTable: 3 tests (resolve, value, class_name_set)
extract_codec_name: 1 test (uppercase+underscore matching)
class_reader.rs gained a #[cfg(test)] ConstantPool::from_entries
test-only constructor so Phase D tests can build synthetic CP
fixtures without writing raw .class bytes.
Precommit (cargo +1.86 fmt + clippy + test) green.
Replaces 'first-match-wins by array order' with 'highest-confidence-
wins, array order tiebreaker'. Removes the arbitrariness when more
than one parser can claim a disc (e.g. one with both
bluray_project.bin and playlists.xml).
New types in labels::mod:
pub enum Confidence { Medium, High }
pub struct ParseResult { labels: Vec<StreamLabel>, confidence }
ParseResult::high(labels) / ::medium(labels) constructors
Parser signature change: every parse() now returns
Option<ParseResult> instead of Option<Vec<StreamLabel>>. Updated all
six parsers in lockstep:
paramount: High (fully structured XML)
criterion: High (fully structured XML)
pixelogic: High by default, Medium when an unknown token component
is encountered (the skip-unknown path now propagates the
coverage gap to the caller instead of silently degrading)
ctrm: High (structured key-value)
dbp: High (anchor scan with vocab routing)
deluxe: still returns None pending Phase D — signature aligned
Registry behavior:
extract() iterates all detect-positive parsers, picks highest
Confidence with non-empty labels. Equal confidence falls to array
order (deterministic). Same selection logic in analyze().
LabelAnalysis grew a confidence: Option<Confidence> field so the
diagnostic surface (freemkv-tools labels-analyze) exposes which
confidence tier the selected parser claimed. labels-analyze JSON
and labels-corpus-check structural diff both gained the field.
Precommit (cargo +1.86 fmt + clippy + test) green.
Three targeted fixes from a second-pass audit of the labels module.
1. vocab::lang now returns Option<LangInfo> with both code AND a
human-readable variant string. Pre-fix: 'Brazilian Portuguese 5.1'
became language=por, variant='', dropping the dialect info the
disc had explicitly authored. Post-fix: language=por,
variant='Brazilian' — matches the convention pixelogic / ctrm /
criterion already use for their region variants. dbp now
populates StreamLabel::variant from this. Compound table grew a
3-tuple (needle, code, variant); bare matches still return
variant=''.
2. dbp and deluxe had duplicated detect() boilerplate (any top-level
.jar in /BDMV/JAR/). Both now call jar::has_any_top_level_jar.
The trait-level detect contract — see super::PARSERS — can't peek
inside a jar without a SectorReader, so loose-detect-plus-real-
check-in-parse is the unavoidable pattern for jar-content parsers.
Consolidating in jar.rs at least makes the duplication visible.
3. mod.rs comment about parser ordering said 'dbp last'; deluxe is
actually now last. Updated to explain the dbp-before-deluxe order
is by cost (cp-iteration cheaper than bytecode walking when Phase
D lands).
Plus a registry-level lock test in mod.rs::registry_tests — asserts
the PARSERS array order is exactly [paramount, criterion, pixelogic,
ctrm, dbp, deluxe]. This was previously implicit; if someone reorders
the array (which changes which parser wins on overlapping signals),
unit tests would have stayed green. Now they fail with an explanatory
message about why the order matters.
Audit findings deferred to follow-ups (each its own commit + design
discussion):
- Stronger detect contract — current loose-detect-real-check pattern
is forced by SectorReader-not-in-detect-signature; could be fixed
by changing the trait to take an Option<&mut dyn SectorReader> or
similar.
- Per-parser confidence scoring — registry currently first-match-wins.
A high-confidence parser ought to beat a low-confidence one
regardless of array order.
- class_reader fuzzing — handles malformed input via Result but no
adversarial corpus yet.
Precommit (cargo +1.86 fmt + clippy + test) green.
Closes the platform unification: every label parser now routes
purpose/qualifier/codec classification through one source of truth
(vocab.rs) instead of N hand-rolls, and every binary-blob byte
scanner goes through one helper (text::extract_ascii_strings).
pixelogic.rs:
- Drop local extract_strings (~20 lines) — use text::extract_ascii_strings.
- HARDENING: replace with skip-unknown-component +
trace log. Pre-refactor behavior: any single uncatalogued token part
(e.g. a future codec ID, new framework variant) silently dropped the
entire stream record. New behavior: skip just the unknown part,
surface what we know about the stream.
- 8 new unit tests cover basic audio/subtitle paths, commentary,
descriptive, region variant, the new skip-unknown-component
regression, and the non-audio/non-subtitle early-out.
ctrm.rs:
- Replace with
vocab::purpose(&name). Now word-boundary matched — 'Commenter Pro
Track' no longer false-matches Commentary.
- Replace with vocab::qualifier(&name).
Same word-boundary tightening, plus picks up Forced and
DescriptiveService for free.
- Preserved structural commentary signal via
as a fallback when name is silent (e.g. 'audio_commentary_1.name=Track 2').
- 6 new unit tests including the 'Commenter' false-positive regression
and the SDH-only-on-subtitles boundary.
text.rs:
- Drop module-level dead_code allow now that pixelogic uses
extract_ascii_strings.
Net: all 5 framework parsers now on the unified platform. Future work
(deluxe Phase D, paramount/criterion XML hardening) builds on the
same scaffolding.
Precommit green.
Closes the 'detected but no parser' gap on Deluxe-authored BD-J discs
(com/bydeluxe/ package signature; ~20% of UHD discs in our corpus per
the 2026-05-10 11-disc capture session).
What ships in this commit:
- detect() — registers the parser in the chain (loose pre-check at the
/BDMV/JAR/ level; real signal in parse via has_path_prefix on
'com/bydeluxe/').
- Phase A master enum identification — walks every .class's <clinit>
ldc sequence and matches against framework-stable fingerprints for:
Language (70 ldcs starting English/French/Spanish/Dutch)
Purpose (8 ldcs starting Normal/Commentary/PiP/Trivia)
VideoFormat (7 ldcs starting HD/HDR10 Plus/HD Dolby)
Region (22 ldcs starting USA_D1/LIC1/LIC2/LIC3 — Disney-only)
Studio (6 ldcs starting Disney/Marvel/Pixar — Disney-only)
All identifications verified out-of-tree on corpus disc-01 (Disney,
The Amateur) and disc-09 (Warner, Dune Part 1) via the standalone
POC.
- Phase B structural skeleton (find_codec_enum) — identifies the
codec enum class by structural signature (>=20 'new' ops, 0 ldcs),
returns the ordered subclass list. Codec string extraction from
subclasses is dead-coded pending the follow-up commit.
What does NOT ship yet:
- Phase D (per-stream binding-class decoder). parse() returns None
intentionally — the master enums alone don't yield StreamLabels
without the streamTable.put(...) bytecode walker. analyze() will
show 'deluxe' in parsers_detected with the enum identification in
tracing logs, so the analyzer reports honestly: 'detected, can't
emit labels yet' rather than silent failure.
Why ship A without D: A is proven on real corpus discs; D's design
needs ground-truth binding bytecode from at least 2 corpus discs
side-by-side to verify the stack-machine pattern. The Phase A
infrastructure (master enum identification + ordinal->name table)
is what D will consume — landing it now unblocks D's design without
holding back the parser registration.
5 unit tests cover the fingerprint matcher logic + a roster lock that
forces explicit consideration when adding/removing fingerprints.
Precommit green.
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.
New modules:
vocab.rs expanded from 27 -> ~370 lines
+ lang(text) -> Option<&'static str> (English/multi-word
-> ISO 639-2; ~45
languages, compound
phrases like
'Brazilian Portuguese'
and 'Castilian Spanish')
+ purpose(text) -> LabelPurpose (Commentary,
Descriptive, Score,
Ime; word-boundary
matched)
+ qualifier(text) -> LabelQualifier (SDH, Forced,
DescriptiveService)
+ has_word internal primitive — enforces word-boundary
matching so 'Commenter' no longer matches 'commentary' and
'engineering' no longer matches 'english'. Existing parsers
used .contains() and got lucky on the corpus; vocab now
guarantees the boundary in one place. 20+ unit tests.
text.rs NEW (~85 lines)
+ extract_ascii_strings(data, min_len) — promoted from two
near-duplicate copies (pixelogic min=4, dbp min=5);
threshold passed in. 7 unit tests including
trailing-without-terminator + high-bit-byte handling.
jar.rs NEW (~120 lines)
+ for_each_jar(reader, udf, fn) — walk every top-level
.jar under /BDMV/JAR/,
yield to callback.
+ has_path_prefix(archive, prefix) — cheap 'is this MY
framework's jar?' check
via central-dir filenames.
+ for_each_class(archive, fn) — parse every .class entry
through class_reader,
yield (name, &ClassFile).
+ try_each_class(archive, fn) — same with early-return on
first Some(R) match.
Refactored:
dbp.rs v2 on the new platform:
- dropped extract_printable raw byte scan
- dropped its own English -> ISO 639-2 map
- dropped its own parse_attributes hand-roll
+ iterates CpInfo::Utf8 via class_reader (structurally clean,
no false-positive risk from method bytecode bytes)
+ routes language/purpose/qualifier through vocab
All 7 prior dbp tests still pass; +2 new ones cover
vocab routing.
dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.
Precommit green (cargo +1.86 fmt + clippy + test).
Foundation for label parsers that need structured access to .class
files inside /BDMV/JAR/<x>.jar. Replaces noak (~3KLOC dep) with a
~1000-line std-only reader.
Public API:
- ClassFile::parse(&[u8]) -> Result<ClassFile>
- ConstantPool::{get, utf8, class_name, string, integer, member_ref, iter}
- Member::code(&pool) -> Option<CodeAttribute>
- CodeAttribute::instructions() -> Instructions iterator
- Instruction::{name, operand_u8, operand_u16, cp_index}
- Opcode constants (LDC, AASTORE, NEW, GETSTATIC, INVOKESPECIAL, ...)
Spec coverage:
- Constant pool: all 17 tag types incl. Long/Double 2-slot quirk
- Modified UTF-8 incl. 0xC0 0x80 -> U+0000 special case
- Bytecode iteration with full opcode size table
- Variable-length tableswitch / lookupswitch / wide
12 unit tests cover the opcode table edge cases (padded switch tables,
wide-iinc 6-byte form), modified-UTF-8 decoder, and iterator
stop-on-truncated behaviour.
Module is currently #![allow(dead_code)] — the public API is staged
for labels::deluxe (Phases A-E bytecode walker) and a labels::dbp
refactor onto the constant-pool iterator. Tests exercise the API
in isolation. The allow comes off as those callers land.
Also fixes two pre-existing clippy lints that 1.86's stricter checks
flagged after I touched the labels module:
- src/mux/disc.rs: while-let-loop in test fixture
- tests/pass_n_size_aware_skip.rs: type_complexity in helper signature
Precommit (cargo +1.86 fmt + clippy + test) green.
5th BD-J authoring framework recognized. Discriminator: any top-
level .jar in /BDMV/JAR/ that contains com/dbp/ package paths.
Identified during the 2026-05-10 corpus session via string-mining
disc-07's BD-J jar — perm files reference bd-live.magpictures.com
(Magnolia / Magnet Releasing).
Stream labels live as plain ASCII strings inside compiled .class
files in the form
LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,...
HTextField,Subtitle1,English SDH,Fontstrip_Composite,...
ATextField,Subtitle0,None,Fontstrip_Composite,...
— a quirk of the menu-rendering layer encoding TextField positions
and content as constant strings the Java compiler retained in the
class string pool. The leading single-letter prefix is string-pool
ordering noise; parser anchors on `TextField,`. Subtitle0 is the
disable-subtitles button and is skipped.
The parser:
- reads each top-level .jar via udf.read_file
- opens it with the existing zip dependency
- confirms com/dbp/ presence in the central directory
- walks .class entries, extracts printable strings, matches the
`TextField,(Audio|Subtitle)<N>,<label>,...` pattern
- maps human-readable language names ("English", "Castilian
Spanish", "Brazilian Portuguese", "Canadian French", ...) to
ISO 639-2 codes via a parser-local table (per the rules-of-
engagement memo, each parser knows its own format)
- preserves the full disc-authored label string in `name` so
consumers display it raw without the lib guessing further
structure
- detects SDH / Forced qualifiers and Commentary / Descriptive
purposes from substring matches; everything else falls through
to fill_defaults using BD-spec MPLS data
Verified live on the corpus: disc-07 (Civil War UHD) now matches
parser=dbp with 3 audio + 2 subtitle labels, exactly the count
visible in the disc's authored TextField definitions and what BD
spec MPLS reports.
Limitation: dbp's detect() returns true for ANY top-level .jar in
/BDMV/JAR/ (every BD-J disc has one), since the discriminator
trait function takes only `&UdfFs` and can't read jar contents.
parse() does the real com/dbp/ check — a non-dbp disc gets
parsed-as-dbp, archive_has_dbp returns false, parse() returns
None, and we fall through. Diagnostic noise: parsers_detected
includes "dbp" on non-dbp BD-J discs. Real fix is refactoring the
DetectFn signature to take a SectorReader; deferred.
7 unit tests cover the TextField extraction, language detection
(simple + compound: "Brazilian Portuguese", "Castilian Spanish",
"Canadian French", "Latin American Spanish", "Australian English"
plus the disc-corpus typo "Austrailian English"), SDH/Forced/RNIB
qualifier detection, and Commentary/Descriptive purpose detection.
Don't-guess discipline preserved: unknown languages return ""
(consumer falls back to MPLS spec data via fill_defaults).
Pass 1 sweep was grinding through damage zones because the marginal-
media handler returned `Bisect` for every failed 32-sector batch —
forcing 32 single-sector reads per bad block at ~5s each on a real
BU40N-vs-Dune-Pt-2 trace. AND the JumpAhead trigger required a 16-
block damage window to fill before firing, so entry into a
contiguous damage zone took ~40 minutes of grinding before the
first jump fired. Architecturally wrong: Pass 1's job is "fast and
accurate, get the most data in the shortest time." Bisection +
recovery is Pass N's purpose-built role.
ReadCtx now carries two new fields:
- `consecutive_outer_failures: u64` — outer-batch failures since
last outer success. Bisect inner failures don't count.
- `bisect_on_marginal: bool` — whether to return Bisect on a
marginal-media batch failure.
- `fast_jump_threshold: u64` — outer-failures count that triggers
JumpAhead before the damage window has filled.
`for_sweep` (Pass 1) sets `bisect_on_marginal=false`,
`fast_jump_threshold=4`, and zeroes the post-failure pause. Failed
batches become SkipBlock → whole block NonTrimmed → advance, no
sleep. After 4 consecutive outer failures: JumpAhead with the
existing escalating multiplier.
`for_patch` (Pass N) sets `bisect_on_marginal=true`,
`fast_jump_threshold=u64::MAX`, keeps the original cooldown pauses.
Pass N's whole reason to exist is to grind on bad ranges with
proper recovery semantics — single-sector reads, 60s recovery
timeout, retry budget, escalating skip — and that's unchanged.
`on_success` resets `consecutive_outer_failures` only when not
bisecting, so a good single-sector read inside Pass N's bisect
doesn't pretend we've escaped the damaged batch.
Tests:
- `pass_n_marginal_with_batch_gt_1_bisects` — Pass N still bisects.
- `pass_1_marginal_skips_instead_of_bisecting` — Pass 1 doesn't.
- `pass_1_jumps_after_4_consecutive_outer_failures` — fast-entry.
- `pass_n_does_not_fast_jump` — fast-entry is Pass-1-only.
- `outer_success_resets_consecutive_outer_failures` — counter reset.
- `bisect_inner_success_does_not_reset_outer_counter` — semantics.
- `pass_1_does_not_pause_on_skip` — explicit zero-pause contract.
- `long_failure_streak_extends_pause_on_pass_n` — Pass N still
extends pauses on long failure streaks (renamed from the old
sweep-based test).
Integration test `test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed`
updated: it used to assert Pass 1 recovers all sectors via bisect
(bytes_good=total). New contract: Pass 1 marks NonTrimmed; Pass N
recovers. Test now asserts Pass-1-only outcome (bytes_pending=total,
complete=false) consistent with the redesign.
Real-world impact on the user's BU40N + Dune Pt 2 trace from this
session: a damage zone that was on track to take ~40 minutes of
Pass-1 grinding will now jump in ~20 seconds. Pass N still has the
full 7-pass recovery budget to revisit those NonTrimmed ranges.
Adds `parsers_detected: Vec<&'static str>` to `LabelAnalysis`. Records
every parser whose discriminator matched, regardless of whether its
parse() then returned Some/None.
Why: when `parser=None` we currently can't tell apart:
(a) no parser recognized this disc — missing parser, candidate for a
new module
(b) a parser recognized it but parse() returned None / empty —
capture truncated, or genuine empty authoring data, or a parser
bug
Surfaced concretely on the 11-disc capture session 2026-05-10:
disc-04 had `bluray_project.bin` in jar_inventory (pixelogic detect()
returned true) but parse() returned None because file content was
past the 1 GB capture window. Old API: parser=None — looked like
"missing parser." New API: detected=[pixelogic], parser=None — clearly
"capture problem, not a parser gap."
The tracing log line on the no-parser-emitted-labels path now
distinguishes the two cases too.
No behavior change to production rip path. extract() is unchanged;
only the diagnostic analyze() returns the richer result.
Promotes `mod labels` to `pub mod labels` and adds `analyze()` plus
`LabelAnalysis` (both `#[doc(hidden)]`) so an out-of-tree diagnostic
binary (freemkv-tools labels-analyze) can introspect which BD-J parser
matched a given disc, what JAR files the discriminators saw, and what
labels came out — without going through the production `apply()` path
that mutates DiscTitles.
Also adds `tracing::info!(parser = name, "label parser matched")` /
"no label parser matched" inside `extract()`. Dev-only signal: users
get the same seamless behavior; developers can finally tell whether a
disc hit a real parser or fell through to the codec-name fallback in
fill_defaults().
The new `jar_inventory()` helper deduplicates and sorts filenames
under any `/BDMV/JAR/<x>/` subdirectory — same plumbing the existing
`jar_file_exists()` discriminators use, just enumerated rather than
predicate-tested. Used by `analyze()` to surface unrecognized
parser-source files when no parser matches, which is the input to
"do we need a new parser?" triage.
No behavior change to the production label path. `apply()` and
`extract()` remain functionally identical; the new public surface
exists alongside.
libfreemkv is a library — Cargo.lock is gitignored (standard for
libs). --locked refuses to create a lockfile on a fresh runner,
so it always fails CI. --locked stays in the binary crates
(freemkv, autorip, bdemu) which DO track Cargo.lock and benefit
from the dependency-race hard-fail behaviour.
The 0.18.4 commit landed with rustfmt diffs in the new
canonical_order tests because my local validation script piped
'cargo fmt --check' to 'tail -1', masking the diff output and
reporting green when fmt was actually unhappy. CI's lint job
caught it immediately. No code change — pure formatting.
Disc::titles previously sorted purely by duration_secs descending,
which puts a play-all virtual playlist at index 0 on UHDs that ship
one. Such playlists reference the same source clips multiple times
for seamless alternate-angle / alternate-ending playback and report
inflated duration AND inflated size_bytes that exceeds the disc's
physical capacity.
Concrete observed case (The Amateur 2025 4K UHD, 58.5 GB BD-100):
Title 1 — 00020.mpls — 4h13m — 92.4 GB — 253 clips ← impossible
Title 2 — 00800.mpls — 2h02m — 57.2 GB — 1 clip ← the movie
92.4 GB > 58.5 GB capacity is proof of clip double-counting. With
the duration-only sort, freemkv -t 1 / disc.titles.first() / autorip's
main-feature picker all selected the 4-hour composite instead of the
2-hour movie.
New canonical_title_order:
1. Real titles (size_bytes <= capacity_bytes) before virtual
composites — capacity gate is hard physical truth.
2. Among real titles, fewer clips first (1-clip wins as the
canonical main feature; multi-clip is either chapter-stitched
or composite).
3. Tiebreak on longer duration first.
Behaviour:
- Non-branching discs: unchanged. The longest 1-clip title is
already the movie.
- Branching UHDs: virtual composite drops to the back, the real
movie surfaces at index 0.
Comparator exposed as Disc::canonical_title_order for downstream
consumers that need the same logic on custom title sets.
Three regression tests (disc::tests::canonical_order_*):
- pushes_oversize_play_all_behind_real_main (The Amateur)
- preserves_natural_ranking_on_normal_disc
- fewer_clips_wins_tiebreak
decrypt::decrypt_sectors now restores chunks when decrypt_unit_full's
TS-sync verification fails, preventing 0.18.1's silent corruption of
MPLS/CLPI navigation files when DecryptingSectorSource decorates the
sweep reader. Fixes E6009 NoStreams on info iso:// for AACS-encrypted
UHDs ripped without --raw.
Disc::sweep progress takes max(snapshot.bytes_good, bytes_done) so
the user-visible counter never regresses below what the producer has
already sent.
The library's public-facing docs were sitting on the 0.17 trait
surface — Disc::copy, pes::Stream, SectorReader, etc. — even though
all in-tree callers migrated in 0.18 rounds 1-3. With 0.18.1 about
to ship, a user copy-pasting the README sample from crates.io would
have hit a compile error.
This commit is purely doc-side:
- README.md: Quick Start rewritten onto Disc::sweep + Disc::patch
with caller-orchestrated multipass; Streams table footnote and
Architecture row reference FrameSource / FrameSink.
- CHANGELOG.md: 0.18.1 entry describing the redesign — primitives,
trait splits, deprecations (kept alive through 0.18.x, deletion
target 0.18.2), throughput numbers.
- docs/{rip-recovery,api-design,architecture,disc-to-rip,
drive-access,udf}.md: every Disc::copy / pes::Stream /
SectorReader reference updated to the 0.18 trait surface.
- FEATURES.md: deleted (8+ versions stale; capabilities live in
README.md and CHANGELOG.md now, matching the workspace-top
FEATURES.md removal in 84acd65).
- examples/iso_dump.rs: verified compiles against 0.18.1.
No code changes.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Round 3 step 1: lift the visibility on the two flat rip-phase verbs
so consumers (autorip + freemkv CLI) can call them directly instead
of going through Disc::copy's multipass dispatcher. Also lift their
option/outcome types and re-export at crate root.
- fn sweep -> pub fn sweep (with rustdoc explaining its role)
- fn patch -> pub fn patch (ditto)
- pub(crate) struct SweepOptions -> pub struct SweepOptions
- pub(crate) struct PatchOpts -> pub struct PatchOptions (renamed
for consistency — both are 'Options')
- pub(crate) struct PatchOutcome -> pub struct PatchOutcome
- libfreemkv::{SweepOptions, PatchOptions, PatchOutcome} re-exports
at crate root.
Disc::copy still exists and still calls Disc::sweep / Disc::patch
through the now-private sweep_internal / patch_internal wrappers.
Migration of the two autorip callers + the freemkv CLI's
disc_to_iso to direct sweep/patch is a follow-up; once those land
Disc::copy + CopyOptions + CopyResult delete in the same commit.
See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.
Single contributor: MattJackson.
Adds a Halt field to DiscStream, populated via the new
`with_halt(halt)` builder. The internal recovery / fill_extents
loops check `halt.is_cancelled()` directly. The existing
`set_halt(Arc<AtomicBool>)` method stays through the deprecation
window for callers (autorip mux) that haven't migrated; marked
#[deprecated] with a pointer to the constructor-time path.
Both signals are unified inside DiscStream: either Halt or the
legacy Arc<AtomicBool> triggers cancellation, so callers can mix
during the deprecation window without breaking stop behaviour.
See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.
Single contributor: MattJackson.
existing call sites — sweep producer and DiscStream demux
Round 1 shipped the DecryptingSectorSource decorator
(libfreemkv/src/sector/decrypting.rs) but the existing decrypt
sites kept calling crate::decrypt::decrypt_sectors inline. This
commit migrates both:
- Disc::sweep (disc/mod.rs): producer wraps the input reader
in DecryptingSectorSource::new(reader, keys) before the read loop.
The inline decrypt_sectors call goes away — read_sectors yields
plaintext directly.
- DiscStream (mux/disc.rs): constructor wraps the underlying
Box<dyn SectorReader> in DecryptingSectorSource so the internal
fill_extents / read path sees plaintext bytes. The DecryptKeys
field stays on DiscStream for metadata-side use; it just no
longer drives decryption.
Disc::patch carried the same inline decrypt step at three call
sites (main read, backtrack read, non-NOT_READY retry read). All
three migrated onto the same wrapping for a single audit surface.
Two small support changes carry the migration without touching
the round-1 decorator shape:
- sector/mod.rs gains specific SectorSource impls for
&mut dyn SectorReader and Box<dyn SectorReader>, mirroring
std's Read forwarding pattern. Generic blankets would conflict
with the existing SectorReader → SectorSource blanket under the
orphan rule (downstream could impl SectorReader for &mut U), so
the impls are scoped to the dyn-trait shape we actually consume.
- sector/decrypting.rs gains DecryptingSectorSource::set_keys so
DiscStream::set_raw() can flip the wrapped reader to a
DecryptKeys::None pass-through without rebuilding the decorator
(which would require moving the inner Box out from behind &mut self).
After this commit, grep `decrypt_sectors` in src/ shows the
function definition, its single use inside DecryptingSectorSource,
plus comments only. One audit surface for AACS / CSS / passthrough
correctness.
Behaviour-preserving: same plaintext bytes flow through; the only
difference is which type owns the decrypt step.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Auto-fmt nit on the multi-line map.flush() expression that landed
when the sweep + patch + DiscStream-FrameSource branches were merged
together. No semantic change.
Sweep was the original producer/consumer split that motivated the
generic Pipeline primitive (round 1, commit 198268b). Now that
Pipeline + Sink exist, sweep stops shipping its own bespoke
threading.
- New SweepSink: Sink<WorkItem> impl in src/disc/sweep.rs. Owns
WritebackFile + Mapfile + ProgressSnapshot back-channel. apply()
carries the file-write + mapfile.record per WorkItem; close()
drains writeback, fsyncs, flushes mapfile.
- Disc::sweep: constructs SweepSink, calls Pipeline::spawn_named
(so the consumer thread keeps showing up as
freemkv-sweep-consumer), sends WorkItems, calls pipe.finish().
The producer-side ReadCtx state machine, decrypt, set_speed,
halt — all unchanged.
- Pipeline gains spawn_named(name, depth, sink) so callers can
preserve identifiable thread names without the primitive baking
one in. Also adds Pipeline::try_send for the throttled
StatsRequest path that must not block the producer.
- Deleted src/disc/sweep_pipeline.rs entirely. WorkItem,
ProgressSnapshot, ConsumerSummary moved into disc/sweep.rs as
module-private types. WorkItem::Finish dropped — dropping the
channel is the end-of-stream signal Pipeline already uses.
Behaviour-preserving: the sweep algorithm, mapfile invariants,
back-pressure via channel depth (DEFAULT_PIPELINE_DEPTH = 4) all
match the 0.17.13 implementation. New synthetic regression test
(sweep_pipeline_full_good_100_batches) exercises ~100 batches of
clean reads end-to-end through the new Pipeline path and verifies
bytes_good and ISO file size.
See (internal)/memory/0_18_redesign.md.
Patch was strictly serial (per-sector recovery: read → seek+write
→ mapfile.record → next). Lifting the write+record onto a consumer
thread lets the drive issue the next per-sector retry while the
previous block's recovered bytes are being committed — small but
real win on damaged discs with many bad sectors, and uniform with
sweep's threading model.
- New PatchSink: Sink<PatchItem> impl in src/disc/patch.rs. Owns
WritebackFile + Mapfile. apply() seeks+writes recovered bytes
and records mapfile state per item; close() runs sync_all and
mapfile.flush.
- Channel depth: WRITE_THROUGH_DEPTH (1). Patch wants minimum
buffering — back-pressure should kick in immediately so the
drive's per-sector retry budget isn't ahead of the writer.
- Disc::patch: keeps every existing recovery decision on the
producer (reverse walk, damage-window skip, NOT_READY pauses,
bridge-degradation handling, wedge exit, range watchdog).
WritebackFile ownership moves to the sink.
Behaviour-preserving: per-sector single-shot read budget unchanged
(BU40N+Initio bridge wedge concern still respected); recovery
algorithm bit-identical.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Round 2 #1 landed `Pipeline` / `Sink` / `Flow` / `DEFAULT_PIPELINE_DEPTH`
in `crate::io::pipeline` but only re-exported them through
`crate::io` (which is `pub(crate)`), so no out-of-tree consumer could
reach them. autorip's round 2 #2 (lifting the mux loop onto Pipeline +
MuxSink) is the first such consumer; surface the primitives at the
crate root for ergonomic access.
No behaviour change — the items themselves are unchanged from round
2 #1; this is just `pub use` plumbing.
Single contributor: MattJackson.
Mirror of the FrameSink concrete migrations slice (f52e4c5) on the
read side. DiscStream is the only meaningful source impl in tree;
all the mux/* impls are sinks.
The round-1 blanket impl<T: Stream + Send> FrameSource for T
already covers DiscStream if it's Send. This slice:
- Audits DiscStream's interior types for Send (its Box<dyn
SectorReader> already requires Send via the trait's super-bound;
verify nothing else interior breaks Send).
- Adds a synthetic-input test that constructs Box<dyn FrameSource>
over a DiscStream, reads frames through the trait object, and
exercises info() / headers_ready() / codec_private().
- (Conditional) Adds a direct FrameSource impl on DiscStream only
if call-site ergonomics demand it; otherwise relies on the
blanket.
No caller migrated. mux::resolve::input still returns
Box<dyn Stream>; autorip / CLI consumers still call Stream::read.
Per-caller migration is a later slice.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Two clippy issues surfaced when round 1 polish + round 2 FrameSink
migrations both landed on libfreemkv main:
- src/halt.rs: clippy::new_without_default fires when a public new()
exists without Default. The polish pass dropped the derive thinking
it was redundant — clippy disagrees, so add a manual impl that
forwards to new(). Doc-comment notes why both exist.
- src/disc/read_error.rs:372: pre-existing
assert_eq!(.., true) trips clippy::bool_assert_comparison. Pre-0.18
precommits passed because that lint sat outside the gate; the
round-2 commits brought enough new clippy surface that it now
shows up. Trivial cleanup: assert!(...) instead of assert_eq!.
Single contributor: MattJackson.
Per-impl migration of MkvStream / M2tsStream / NetworkStream /
NullStream / StdioStream from the deprecated pes::Stream trait
to the typed pes::FrameSink trait. Both impls coexist during
the 0.18 deprecation window — the existing Stream impls are
unchanged.
The FrameSink::finish signature differs (Box<Self> vs &mut self),
which is why this couldn't be a blanket impl. Each migration
re-borrows the box and delegates to the underlying Stream::finish
body.
FrameSink: Send forced two struct fields (M2tsStream's boxed
Write/Read, MkvStream's boxed WriteSeek/Read) to gain `+ Send`
bounds — minimum surface needed to make the Send-bounded trait
impl-able. mux::resolve::output's local Box<dyn WriteSeek>
construction picks up the same `+ Send`. tests/streams.rs's
shared `stream.write/.finish/.info/.read` calls were
disambiguated to `PesStream::*` to resolve the now-multiple
candidates from coexisting trait impls.
Caller migration (mux::resolve::output return type, autorip,
CLI) is a later slice. This commit only adds new impls; nothing
removed.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
The SectorSource/Sink agent and the WritebackFile-rename agent both
branched from main concurrently; the sector branch wrote against the
0.17 Writer name and only the rename branch knew about WritebackFile.
This integration commit reconciles the two: FileSectorSink::create /
::open / the inner-field type all use WritebackFile directly, and the
module-level + struct-level docs are corrected.
MattJackson