Compare commits

...
115 Commits
Author SHA1 Message Date
MattJackson d7243a6044 v0.20.2: delete dead retry block + wedge-family cooldown in Disc::patch
WO-3c: Delete dead non-NOT_READY retry block (~100 LOC). The block
declared retry_count = 0 inside the per-iteration Err arm, so the
'MAX_NON_NOT_READY_RETRIES=3' budget actually fired exactly once
(1s pause + 1 retry) before falling through to NonTrimmed. The
'exponential backoff: 2s, 4s, 8s' comment was wrong by construction.
Cross-pass NonTrimmed retry (each pass gives the same sectors another
shot) already covers the recovery case, and gives the drive minutes
between attempts instead of 1-8 seconds — empirically what stochastic
recovery on the BU40N actually needs.

WO-4 (targeted slice): Add wedge-family cooldown on HARDWARE_ERROR /
ILLEGAL_REQUEST senses. These are what the BU40N's firmware fast-fail
state returns; every subsequent read in that state comes back in
<100ms. Pre-fix patch hammered the drive: mark NonTrimmed, sleep 1s,
advance, hit next wedge, mark, sleep 1s — exactly the rapid-retry
cadence the firmware is sensitive to. Now a wedge-family sense triggers
WEDGE_FAMILY_COOLDOWN_SECS=30 cooldown (matches read_error.rs's
ZONE_ENTRY_COOLDOWN_SECS), and WEDGE_ABORT_THRESHOLD=16 consecutive
wedges aborts the pass for autorip eject+reload. Any non-wedge read
clears the counter.

Also drops the duplicate NonTrimmed dispatch (Mapfile::record is
idempotent so it wasn't a correctness bug, but it doubled per-failure
consumer work).
2026-05-13 11:53:57 -07:00
MattJackson f1926c38dc v0.20.1: delete SectorReader, extract Disc::patch, doc/stub cleanup
WO-2 (delete SectorReader trait):
- The 0.18 trait split into SectorSource (read-only) and SectorSink
  (write-only) is final; the legacy SectorReader alias was a bridge.
- Renames every internal &mut dyn SectorReader (~25 sites) to
  &mut dyn SectorSource. The trait method capacity() becomes
  capacity_sectors() with a default of 0 (preserves SectorReader's
  default-0 behavior).
- Deletes the SectorReader trait, its blanket-to-Source bridge, and
  the FileSectorReader type alias. Adds explicit forwarding impls
  for Box<dyn SectorSource> and &mut dyn SectorSource so generic
  decorators like DecryptingSectorSource<S: SectorSource> compose.

WO-3a (extract Disc::patch):
- Moves Disc::patch (1230 lines) and bytes_bad_in_title from
  disc/mod.rs into disc/patch.rs as a split inherent impl. Zero
  behavior change — pure mechanical relocation. disc/mod.rs drops
  from 3,945 to 2,714 LOC.

WO-6 (partial):
- Deletes src/labels/png_filenames.rs — was a 72-LOC stub with
  detect() returning false, never wired into the PARSERS registry.

project docs doc drift fixes (audited 2026-05-13):
- JUMP_BASE_SECTORS: 256→1024 (64 MB base for UHD, not 8 MB)
- PASSN_DAMAGE_THRESHOLD_PCT: 12→6
- PASSN_SKIP_SECTORS_BASE: 64→32
- MAX_RANGE_SECS=180: replaced by proportional range_sectors × 25,
  capped at RANGE_BUDGET_CAP_SECS=1800.
2026-05-13 11:36:55 -07:00
MattJackson 4709a73c80 v0.20.0: delete FrameSource/FrameSink, keep single Stream trait
The 0.18 trait split into FrameSource (read-only) and FrameSink
(write-only) was an over-engineered API. Consumers don't think
"frame source backed by MKV" — they think "open MKV for reading".
The split paid a real API-complexity cost (two trait names, two
re-exports, dual impls per bidirectional type, deprecation bridge)
for one marginal property: compile-time direction-safety at the
trait-object boundary. The runtime error path on a wrong-direction
call (StreamReadOnly / StreamWriteOnly) is unambiguous and rare in
practice.

Deletions:
- pes::Stream is no longer #[deprecated]
- pes::FrameSource trait + its blanket-from-Stream bridge
- pes::FrameSink trait + the trampoline impls on every concrete type
- The compile-time-direction-safety test scaffolding
- Crate-root FrameSource / FrameSink re-exports

Additions:
- Stream is now Send-bounded (Stream: Send supertrait). Every
  concrete impl was already Send-compliant — Box<dyn Read + Send>
  and Box<dyn Write + Send> were already in place on the trait
  objects MkvStream / M2tsStream / etc hold internally. Promoting
  Send into the trait makes Box<dyn Stream> Send too, which lets
  autorip drop its SendStream unsafe newtype.

The public API is now: one Stream trait, one concrete type per
format, two constructors (open/create or input/output). Bidirectional
types route through internal Mode { Read | Write } discriminants.

Net: -347 lines libfreemkv, -38 lines autorip, -5 lines freemkv.
2026-05-13 08:42:14 -07:00
MattJackson 55bd1ee868 v0.19.1: repair Cargo.toml after botched 0.19.0 bump
v0.19.0 was tagged with a search-and-replace gone wrong:
rust-version, serde, and zip all had their version strings
replaced with "0.19.0". Edition 2024 rejected rust-version
0.19.0 (< 1.85), failing every CI build. No artifacts shipped
to crates.io.

Repair:
- rust-version: 0.19.0 → 1.86 (CI pin)
- serde: 0.19.0 → 1
- zip: 0.19.0 → 2

Also drops an unused start_lba binding in mux/disc.rs that
clippy 1.86 catches.
2026-05-12 21:29:07 -07:00
MattJackson 5285e8b6ec v0.19.0: bump version 2026-05-12 19:24:06 -07:00
MattJackson c5af61e46b v0.18.25: bump version 2026-05-12 16:27:28 -07:00
MattJackson ed73c1ab68 v0.18.25: add mux debug logging for reader type and stall detection 2026-05-12 16:27:16 -07:00
MattJackson 5ce2df0888 v0.18.24: bump version 2026-05-12 15:46:47 -07:00
MattJackson ea591a36a5 v0.18.23: bump version 2026-05-12 10:59:45 -07:00
MattJackson 368e10486e v0.18.22: bump version 2026-05-12 09:33:00 -07:00
MattJackson a99c4f8487 v0.18.21: format code 2026-05-12 09:18:52 -07:00
MattJackson 4df470e572 v0.18.21: fix borrow checker, make io public 2026-05-12 09:01:52 -07:00
MattJackson a0b13941fe v0.18.21: fix borrow checker error in debug logging 2026-05-12 08:49:33 -07:00
MattJackson 275c9da009 v0.18.21: verbose debug logging for pipeline stalls (fixed) 2026-05-12 08:37:42 -07:00
MattJackson da183722fa v0.18.20: bump version 2026-05-11 22:33:15 -07:00
MattJackson 4a5424eabe v0.18.20: bump version 2026-05-11 22:28:57 -07:00
MattJackson 293d89c586 v0.18.20: separate read/write pipeline depths 2026-05-11 22:18:56 -07:00
MattJackson 4840f5134d libfreemkv: increase DEFAULT_PIPELINE_DEPTH to 32 for smoother mux speed 2026-05-11 22:01:27 -07:00
MattJackson 11c8211605 v0.18.18: bump version 2026-05-11 20:16:30 -07:00
MattJackson 1038beaf06 v0.18.17: bump version 2026-05-11 19:58:36 -07:00
MattJackson f1df57196c libfreemkv: add Clone derive to MuxAtomics 2026-05-11 19:54:50 -07:00
MattJackson 8ba9dc1c0b v0.18.16: bump version 2026-05-11 15:57:03 -07:00
MattJackson 49b131f39a add debug logging for MKB processing 2026-05-11 15:53:11 -07:00
MattJackson 8238ec4ce7 optical vs block batch sizing 2026-05-11 15:35:45 -07:00
MattJackson 4fd5d27df8 v0.18.15: bump version 2026-05-11 11:30:19 -07:00
MattJackson 1fbe272832 disc/scan: surface AACS resolution error on Disc.aacs_error
scan_with() collapsed every failure path from resolve_encryption() into
None via .ok(), so callers couldn't tell the difference between "no
KEYDB found", "KEYDB failed to parse", "disc hash not in KEYDB and
fallback derivation failed", "AACS files unreadable on disc", and a
handshake that rejected every host cert. autorip's UI was stuck
printing "no decryption keys found (check KEYDB)" for all of them,
which is a particularly bad message when the user has actually loaded
a KEYDB and the real failure is something else.

Changes:
- New pub field Disc.aacs_error: Option<Error>. Populated by scan_with
  whenever encrypted && aacs.is_none(). Sentinel KeydbLoad path
  "<no keydb in search paths>" distinguishes the no-keydb case from
  a real load failure without adding a new Error variant (which would
  be a breaking change for downstream exhaustive matches).
- tracing::warn in scan_with at scan_aacs_resolve_failed and
  scan_aacs_no_keydb, with error_code and keydb path for grepping.
- tracing in do_handshake: keydb load failure, host-cert exhaustion
  (with cert count and last error code), VID read failure post-auth,
  and a debug-level success log. Lets us see whether handshake even
  got off the ground for a given disc.

Test fixtures updated to set aacs_error: None.
2026-05-11 11:29:48 -07:00
MattJackson 6bd635725e v0.18.14: bump version 2026-05-11 08:27:10 -07:00
MattJackson 09b77b4dea disc/patch: relabel "possible wedge" heuristic log to avoid confusion
The 'All probes failed — possible wedge condition' log fired during patch
probing whenever 10+ consecutive failures hit AND a probe sweep at the
local zone returned 0 successes. This was distinct from the read_error.rs
'wedge_transition' log that fires when the SCSI sense family ACTUALLY
flips into Hardware/IllegalRequest fast-fail mode.

Two logs both saying 'wedge' caused operator confusion during the
2026-05-11 Dune Pt 2 wedge investigation — was the drive wedged, or was
it just a zone of fully-bad sectors? They mean different things.

Relabel to 'patch_zone_fully_bad' with explicit pointer to read_error.rs
for the canonical wedge detection. Same triggering condition; just clearer
wording in the log stream.
2026-05-10 22:20:44 -07:00
MattJackson 688058b3e8 labels/bdmt: drop description fields that are just XML child elements
Disc-04 (Top Gun: Maverick) re-test 2026-05-11 surfaced a real-world
bdmt_eng.xml where <di:description> contained no prose, only nested
<di:thumbnail href="…"/> elements. The previous parser surfaced
the raw XML fragment as the description string ("<di:thumbnail
href=\"tgm_meta_sm.jpg\" />\\r\\n      <di:thumbnail
href=\"tgm_meta_lg.jpg\" />"). Worse than no description.

Fix: filter description candidates that begin with `<` after
trimming. Real prose never starts with an angle bracket; XML-only
content always does. Net: title extraction unaffected (it uses its
own element-priority path); description field drops when it would
otherwise carry XML noise.

Two new bdmt tests, 12 of 12 passing.
2026-05-10 22:12:59 -07:00
MattJackson c96bac7977 labels: append CLPI orphan streams after gap-fill
Three layered sources of stream labels now, in precedence order:
1. **Framework parser** (paramount/criterion/pixelogic/ctrm/dbp/deluxe)
   — editorial labels with purpose/qualifier ("English Atmos",
   "Director's Commentary", "English SDH"). High or Medium confidence.
2. **MPLS gap-fill** (`fill_gaps_from_mpls`) — every stream the
   playlist references gets at least a basic lang+codec label, even
   when the framework parser missed it.
3. **CLPI orphan append** (`append_clpi_orphans`) — streams in
   /BDMV/CLIPINF/*.clpi ProgramInfo that NO MPLS playlist references.
   Empirical (2026-05-11): ~5% of streams across the 11-disc corpus,
   most dramatic on disc-02 (HDMV-only) at 40% CLPI-only.

Orphan numbering: each appended orphan gets
`stream_number = max(existing per type) + N` so playlist-reachable
streams keep their original positions and orphans sort cleanly at
the tail.

Orphan dedup: (stream_type, language, codec_hint) tuple — fuzzier
than PID matching (PIDs aren't carried on StreamLabel) but it's the
only signal available downstream of the gap-fill. False positives
(genuine orphan that happens to share lang+codec with an existing
entry) silently drop, which is the conservative failure mode — the
user-facing display would just see a confusing duplicate otherwise.

`mpls_universal::language_display_name` and `::codec_name` promoted
from private fn to pub(crate) so this module can build orphan labels
with consistent naming.

Tests: 2 new in gap_fill_tests — synthetic-input verification of the
dedup tuple logic and the stream_number assignment. 6/6 tests in the
gap-fill module now passing.
2026-05-10 22:08:14 -07:00
MattJackson c32acff3e4 disc/read_error: rustfmt fixup for wedge-prevention commit 2026-05-10 22:00:08 -07:00
MattJackson b58e2d9873 disc/read_error: wedge PREVENTION — jump on first error + 30s cooldown
Rewrites the Pass 1 wedge handling from "slow skip after the drive
has already wedged" to "prevent the wedge transition in the first
place." Driven by 2026-05-11 empirical data: the BU40N transitioned
into IllegalRequest fast-fail mode at exactly 7 medium errors in
6.5 seconds (~1 read/sec retry cadence). Once there, only physical
eject + reload clears it — 30s pauses + 1 GB jumps do not.

The fix is the user's mental model from that session:

  "We can detect bad reads, failed reads, and asking to read again
   fast after causes a wedge. We need to prevent the wedge in the
   first place."

Two changes to the centralized error handler:

1. **`for_sweep().fast_jump_threshold = 1`** (was 4). Pass 1 now
   JumpAheads on the FIRST outer-batch failure, not the 4th. The
   drive never gets back-to-back retries at the same LBA in Pass 1
   — every error → jump 64 MB forward + long cooldown. Pass N keeps
   `fast_jump_threshold = u64::MAX` because retries on already-known-
   bad LBAs are its whole job.

2. **`ZONE_ENTRY_COOLDOWN_SECS = 30`**. The FIRST error after a
   clean run (when `consecutive_outer_failures == 1` and we're not
   bisecting) uses this long pause instead of the standard 5 s
   FAIL_PAUSE_SECS. Gives the BU40N's firmware / bridge internal
   retry counters 30 s of breathing room before the next read,
   preventing the "7 errors in 6.5 s" cascade. Subsequent errors
   in the same zone use the standard 5 s pause (we've already
   jumped past the initial damage; further errors mean we landed
   in another bad cluster).

Pass N exempt from the zone-entry cooldown — `bisect_on_marginal=
true` skips the long-pause arm. Pass N's per-sector retries on
known-bad LBAs would multiply uselessly with 30 s/error.

Test updates: 4 tests' expected behavior changed under the new
policy. Renamed `pass_1_marginal_skips_instead_of_bisecting` →
`pass_1_marginal_jumps_immediately_not_bisecting`. Renamed
`pass_1_jumps_after_4_consecutive_outer_failures` →
`pass_1_jumps_immediately_on_first_outer_failure`. Updated
`both_passes_pause_on_failed_read_for_wedge_avoidance` (now
`pass_1_zone_entry_uses_long_cooldown` + `pass_n_pauses_uniformly_on_failed_read`).

Cost analysis:
- Clean disc (no errors): unchanged. 0% overhead.
- Lightly damaged (1-2 zones): +30 s per zone = ~1 min total. Fine.
- Heavily damaged (10+ zones): +5+ min total. The trade for never
  wedging the drive and getting a usable Pass N afterwards.

Expected behavior on the next damaged-disc rip:
- Pass 1 hits damage at LBA X → jumps 64 MB forward immediately,
  pauses 30 s
- Drive's firmware never accumulates the retry pressure that triggers
  IllegalRequest fast-fail
- bytes_maybe accumulates faster (we skip more), but Pass N picks up
  the slack with proper per-sector recovery — and Pass N can actually
  RUN because the drive isn't wedged
2026-05-10 21:56:27 -07:00
MattJackson a9e802c1c2 clpi+labels: extract program_info stream table + CLPI vs MPLS audit
Two layered changes, in service of the empirical question "is CLPI
truly redundant with MPLS for label data?":

1. **clpi.rs ProgramInfo parser**. The existing CLPI parser only
   walked the EP map (for sector-range lookups). Added a parser for
   the ProgramInfo section's per-stream stream_coding_info table:
   pid, coding_type, audio_format/rate, video_format/rate, ISO 639-2
   language. Spec layout per libbluray clpi_parse.c. Best-effort —
   malformed program_info leaves `streams: vec![]`, EP map keeps
   working. `ClipInfo` gains a `streams: Vec<ClpiStream>` field.

2. **labels/clpi_audit.rs**. Diagnostic that walks both
   `/BDMV/CLIPINF/*.clpi` (via the new program_info parser) and
   `/BDMV/PLAYLIST/*.mpls`, builds a (PID → fields) merged view, and
   classifies each row:
   - `Match`: both sources agree (same coding_type + language)
   - `ClpiOnly`: PID in CLPI but no MPLS playlist references it
     (orphan stream on disc — reachable via low-level access, not via menu)
   - `MplsOnly`: PID in MPLS but no CLPI lists it (would indicate a
     parser bug; verified empirically that this NEVER happens)
   - `Divergent`: same PID, different coding_type or language between
     sources (playlist re-tagged or attribute encoding mismatch)
   Surfaced via `labels-analyze` as `clpi_vs_mpls_audit: {matches,
   clpi_only, mpls_only, divergent, total_pids}`. Doesn't affect the
   label output — pure diagnostic.

Empirical findings on the 11-disc corpus (excl. disc-04 truncated):
- 226 matches / 0 mpls_only / 8 clpi_only / 5 divergent across 239 PIDs
- 6 of 10 non-truncated discs have CLPI-only streams (orphans)
- disc-02 (HDMV-only) is the most dramatic: 40% of its 5 streams are
  CLPI-only — MPLS sees 3, CLPI sees 5
- Conclusion: CLPI is NOT truly redundant. ~5% of streams disc-wide
  are CLPI-exclusive. Future work: layer CLPI as a tertiary source
  below MPLS in the labels pipeline (orphan streams marked with even
  lower confidence than MPLS).
2026-05-10 21:50:39 -07:00
MattJackson a876ce846b labels: surface MPLS chapter summary in LabelAnalysis
LabelAnalysis gains `chapter_summary: Vec<ChapterSummary>` — one row
per .mpls file in /BDMV/PLAYLIST/, with chapter count (PlaylistMark
entries with mark_type ≤ 1) and approximate playlist duration in
seconds. Sorted by playlist filename.

Sourced from the existing crate::mpls parser (no new format work).
Useful for identifying the main feature playlist at a glance — it's
the one with the longest duration. Verified on disc-11 (Dune Pt 2):
00800.mpls correctly identified as 2h 45m 49s with 18 chapters
amid 30+ shorter playlists.

Doesn't touch the per-title `disc::DiscTitle::chapters` field which
disc::bluray.rs already populates from the same marks during disc
init — this is purely the diagnostic surface for labels-analyze.
2026-05-10 21:32:54 -07:00
MattJackson 058fd8396f labels: gap-fill MPLS streams when framework parser under-yields
When a framework parser (paramount, criterion, pixelogic, ctrm, dbp,
deluxe) is chosen but its label list covers only a subset of the
stream slots MPLS knows about, merge MPLS-derived entries for the
uncovered (stream_type, stream_number) slots. Framework labels keep
their richer fields (purpose=Commentary, codec_hint with "Atmos",
qualifier=Sdh); MPLS only fills slots the framework left unnamed.

Implementation:
- `fn fill_gaps_from_mpls` walks the MPLS label list, pushing any
  entry whose (type, number) tuple isn't already in the framework
  output. Stable sort by (type, number) groups audios before
  subtitles in the merged result.
- Called from both `extract()` and `analyze()`. Skipped when the
  chosen parser is itself `mpls_universal` (no gaps possible).
- `LabelAnalysis::gap_fill_added` field reports how many slots got
  filled — useful diagnostic from `labels-analyze`.
- `StreamLabelType` gains `Eq + Hash` so the dedup HashSet works.

Tested via 4 new unit tests (155 of 155 labels tests passing, was
151). End-to-end on partial-yield corpus discs:
- disc-05 (Oppenheimer): pixelogic 4/5 already covered, gap_fill_added=0
- disc-11 (Dune Pt 2):   pixelogic 8/11 already covered, gap_fill_added=0

(Real-world gap-fill activations are rare in the current corpus because
pixelogic already incorporates MPLS-equivalent data when matching;
the merge is defensive for less-thorough frameworks.)
2026-05-10 21:28:01 -07:00
MattJackson 5ee28c08b9 labels/mpls_universal: dense stream numbering across playlists
Per-playlist stream_number counters were resetting between MPLS
files, so a disc with 2 MPLS files each listing the same 8 audio
streams produced labels with stream_number 1..8 then 1..8 again
(dedup kept whichever PID was different, leaving the numbering
visibly broken — multiple "audio1: eng" rows).

Move the counters outside the per-file loop and increment only
when an entry survives dedup. Surviving entries now get dense
1..N numbering across the whole disc per stream_type.

Verified on corpus disc-02 (HDMV-only): was `audio1, audio1, audio1`
for the 3 distinct audio codecs (TrueHD/AC-3/DTS-HD MA), now
`audio1, audio2, audio3`. Same fix applies to disc-01 (12 audio
streams across multiple MPLS) and disc-09 (14 audio streams).
2026-05-10 20:58:33 -07:00
MattJackson 764230b1eb labels: universal MPLS fallback + bdmt disc metadata + png stub
Three new modules in the labels platform, all layered so framework-
specific parsers (paramount, criterion, pixelogic, ctrm, dbp, deluxe)
always take precedence over the universal layer.

**mpls_universal.rs** (~600 LOC, 9 tests): consumes the already-parsed
`crate::mpls::Playlist::streams` and emits StreamLabel entries with
language + codec_hint per stream. Returns `Confidence::Low` (new
variant) so framework parsers' Medium/High always win the registry's
max-by-confidence tiebreaker; MPLS only gets picked when no framework
matched. Closes the "no BD-J disc" case (HDMV-only navigation) that
previously produced zero labels — language and base codec are
spec-mandated in MPLS STN tables on every Blu-ray ever made.

**bdmt.rs** (~350 LOC, 10 tests): reads `/BDMV/META/DL/bdmt_<lang>.xml`
files into a new `DiscMetadata` struct (localized title names per
ISO 639-2 code, descriptions, optional box-set position). Runs
independently of the parser registry — disc-level metadata, not
per-stream, so the registry's confidence selection doesn't apply.
Surfaced on a new `LabelAnalysis::disc_metadata` field.

**png_filenames.rs** (noop stub): pattern documentation + dead-code
detect/parse for future reactivation. Deferred because MPLS already
delivers per-stream lang/codec/type on every disc; PNG filename
language tokens only add studio variant disambiguation (FRC vs FRP,
LAS vs CSP) — niche enough to not justify the implementation cost
right now.

Wiring changes in `mod.rs`:
- New `Confidence::Low` variant (PartialOrd places it below Medium/High)
- New `ParseResult::low()` constructor
- `mpls_universal` appended last to `PARSERS` registry
- `LabelAnalysis::disc_metadata: Option<DiscMetadata>` field
- `analyze()` runs `bdmt::parse` independently and surfaces result
- `pub use bdmt::DiscMetadata` re-export so the labels-analyze tool
  in freemkv-tools can construct the JSON payload

Total: 151 of 151 labels tests passing (was 132 — added 19 new).
2026-05-10 20:53:56 -07:00
MattJackson 7fb2e07aed v0.18.13: bump version 2026-05-10 19:57:44 -07:00
MattJackson 376aadb335 disc/patch: adaptive batching — 32 sectors, drop to 1 on failure
Pass N now reads at 32 sectors per attempt and drops to 1 only on
batch-read failure to probe each sector individually. After 16
consecutive clean single-sector reads it climbs back to 32. Net
effect: NonTrimmed regions walk ~32x faster in clean stretches
without sacrificing per-sector recovery quality — the drop-to-1
retry from the same cursor position guarantees every sector in a
failed batch is individually attempted.

Design contract:
- A batch-read failure (count > 1) is NOT a recorded failure: no
  NonTrimmed mark, no consecutive_failures bump, no damage_window
  push, cursor stays put. We just drop current_batch to 1 and the
  loop re-attempts the same position at single-sector granularity.
- A single-sector failure (count == 1) follows the existing path:
  NonTrimmed mark, consecutive_failures++, damage_window.push(false),
  post-failure pause, wedge probes.
- Backtrack always at count=1: this path fills a gap that the main
  loop's damage-window skip jumped over. Using batched reads there
  would lump good sectors into NonTrimmed marks when the gap
  contains even one bad sector.

State machine adds:
- `initial_batch` (from opts.block_sectors, default 32 in patch_internal)
- `current_batch` (mutable, starts at initial_batch, drops to 1 on
  batch failure)
- `consecutive_singles_ok` (counter, resets on upscale + failure)
- `ADAPTIVE_UPSCALE_THRESHOLD = 16` (matches sweep's pattern for
  "16 consecutive good = back to fast mode")

Tests:
- pass_n_size_aware_skip.rs PatternedSectorReader now fills each
  sector with its OWN LBA byte (not the starting LBA's byte). This
  matches real drive behavior — the pre-0.18.13 fixture's
  "fill whole batch with one byte" was a shortcut that only worked
  when patch read 1 sector at a time. Existing recovery-quality
  assertions all still pass under adaptive batching.

User spec: "try 32, pass, great, fail -> do 1 sector"
2026-05-10 19:57:27 -07:00
MattJackson 586495b2ee v0.18.12: bump version 2026-05-10 19:44:57 -07:00
MattJackson 72f2224efe disc/patch: leave failed reads NonTrimmed mid-multipass, not Unreadable
User design call after watching Pass 2 mark ~20 KB as "Cosmetic"
(permanently Unreadable) after just 10 retries within a single pass:
"i think it's good or maybe until all passes are done. then it's
gone."

That contradicts what the multi-pass design promises a user. The
project goal in project docs is "recover 100% of readable data from any
optical disc, automatically." Marking sectors Unreadable after a
SINGLE pass's per-range retry budget gives up on sectors that
subsequent passes might recover — drive reads are stochastic, the
sector that fails 10 times in Pass 2 may succeed on attempt 1 in
Pass 3 after temperature / bus state / prior-read patterns shift.
The patch.rs doc comment already noted ~36% of patch-marked
Unreadable sectors turned out to be readable in re-rip experiments.

Three sites in `Disc::patch` were emitting `PatchItem::Unreadable`
mid-pass:
  - backtrack hit damage (line ~2659)
  - all-retries-exhausted on a single LBA (line ~2846)
  - redundant second mark after the wedge-suspicion log (line ~2970)

All three now emit `PatchItem::NonTrimmed` instead. Failed bytes
stay "maybe" (NonTrimmed) so the next pass gets another shot. The
per-range skip-limit (10) and per-pass wedge-threshold (50) still
bound time-per-pass; they just no longer turn the bytes terminal.

The `PatchItem::Unreadable` variant stays in the enum (with
#[allow(dead_code)]) because the orchestrator-side end-of-recovery
promotion will use it: autorip, after the final retry pass
completes, scans the mapfile and promotes still-NonTrimmed →
Unreadable. That promotion lands in a follow-up commit on the
autorip side — separable from this libfreemkv change.

Loss accounting unchanged: `bytes_pending + bytes_unreadable` is
the "lost or pending" total that `abort_on_lost_secs` consults
(disc/mod.rs:1327). Moving bytes from one bucket to the other
mid-pass doesn't affect whether the rip would abort; it only
affects display (UI shows "Maybe" vs "Cosmetic") and whether
subsequent passes retry the bytes (the actual fix).

Test update: `test_pass_progress_separates_unreadable_from_pending`
was renamed to `test_pass2_leaves_failed_reads_as_pending_not_unreadable`
and rewritten to assert the new invariant — Pass 2 leaves all
failed bytes as bytes_pending (no mid-pass Unreadable promotion).
Original assertions were checking the pre-design-call behavior.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 18:47:05 -07:00
MattJackson 0bca7a11bd disc/read_error: unify Pass 1 and Pass N error handling
User's design call after watching the avoidance work prevent a wedge
on the live rip (no wedge events across 6 read errors): "Pass N
and 1 should both be very very similar in recovery. almost identical
just smaller sectors imo in pass n. pause times the same imo as a
failed read is a failed read."

The error-handling code path was already centralized (one
handle_read_error fn, called by both Disc::sweep and Disc::patch).
The TUNING was split — Pass 1 used 5 s inter-error pauses + a
wedge-skip-and-continue policy; Pass N used 1 s pauses + immediate
AbortPass on HARDWARE_ERROR / ILLEGAL_REQUEST. That asymmetry made
Pass N vulnerable to the same wedge that Pass 1's avoidance fixed.

Changes:

1. FAIL_PAUSE_SECS = 5 — single constant, applied uniformly to both
   passes. Dropped PASS_1_FAIL_PAUSE_SECS and POST_FAILURE_PAUSE_SECS
   in favor of one value. CONSECUTIVE_FAIL_LONG_PAUSE_SECS kept as a
   distinct (but currently equal) value for future tuning escalation.

2. HARDWARE_ERROR / ILLEGAL_REQUEST path is now symmetric:
   - Pass 1: JumpAhead WEDGE_JUMP_SECTORS (1 GB) + WEDGE_PAUSE_SECS
     cooldown, mark skipped region NonTrimmed.
   - Pass N: JumpAhead WEDGE_PASS_N_SKIP_SECTORS (64 sectors / 128 KB)
     + WEDGE_PAUSE_SECS cooldown. Pass N's batch=1 means a 1 GB skip
     would abandon the entire current NonTrimmed range; small skip
     moves past the bricked LBA + buffer, outer patch loop picks up
     the next sector.
   - Both share WEDGE_ABORT_THRESHOLD — same 16-skip budget before
     real AbortPass on a permanently stuck drive.

3. wedge_skip / wedge_abort tracing logs now include `pass=1|N`
   so post-mortems can see which pass hit the wedge condition.

Cost analysis:

Pre-reframe worry was "5 s × 5500 NonTrimmed sectors per Pass N
pass × 7 passes = 53 hours." Reality: most NonTrimmed sectors
recover on first or second retry, so most reads are successful and
pay 0 pause. The few that DON'T recover hit the 10-skip budget and
get marked Unreadable — bounded at 10 × 5 s = 50 s per truly-bad
sector. Worst-case Pass N pause overhead on a typical damaged disc
is single-digit minutes, not hours. And it's strictly cheaper than
the alternative (wedge kills the entire multi-pass recovery).

Tests:

- `both_passes_pause_on_failed_read_for_wedge_avoidance` — locks the
  unified pause-tuning policy (was pass_1_pauses_briefly).
- `pass_n_hardware_error_also_skips_not_aborts` — was
  `pass_n_hardware_error_still_aborts`. New behavior verified:
  JumpAhead with WEDGE_PASS_N_SKIP_SECTORS + WEDGE_PAUSE_SECS.
- `pass_n_hardware_error_aborts_after_threshold` — new. Confirms
  Pass N respects the same WEDGE_ABORT_THRESHOLD as Pass 1.
- pass_1_does_not_pause_on_skip is gone (it was the old "Pass 1
  pause=0" assertion, irrelevant after the avoidance work).

Empirical validation: avoidance was already proven on a live rip
tonight — 6 read errors on a damaged disc, sense_family=Medium
throughout, wedge_count=0, Pass 1 continued cleanly past 40%
where it previously died at 48%. This commit extends the same
discipline to Pass N's recovery loop.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 18:04:01 -07:00
MattJackson 72ab714c1e labels/deluxe: Phase D rewrite against ground-truth binding pattern
Replaces the speculative arg-position heuristic with type-presence
detection driven by real disc bytecode. Ground truth captured in
(internal)/research/deluxe-poc/data/ via POC v0.3 binding-
bytecode dumps against disc-01 (Disney) and disc-09 (Warner).

What changed:

1. StackVal::CodingType(String) — new variant. getstatic against
   org/bluray/ti/CodingType (the BD-J spec codec enum) now pushes
   this, carrying the field name (e.g. DOLBY_LOSSLESS_AUDIO). The
   pre-fix code was treating codecs as a Deluxe-internal enum
   subclass walk (Phase B), which is the wrong model — codecs are
   standard BD-J API references.

2. coding_type_to_codec_hint(field) — new function. Maps
   org.bluray.ti.CodingType field names to human-readable codec
   strings (DOLBY_LOSSLESS_AUDIO -> "Dolby TrueHD", DOLBY_AC3_AUDIO
   -> "Dolby Digital", etc.). Unknown field names pass through
   verbatim so future codec values still surface something.

3. find_binding_classes — multi-class variant. Some Deluxe discs
   split per-stream tables across two binding classes (audio +
   subtitle). Returns top-K candidates by getstatic count, filtered
   to >=40% of the top count and capped at 4. Replaces the old
   single-class find_binding_class (which was unused after this
   change).

4. interpret_streams — rewritten. Args identified by TYPE not
   position:
   - First EnumRef{kind:"Language"} -> language
   - First EnumRef{kind:"Purpose"}  -> purpose
   - First CodingType(name)         -> codec_hint
   - First Int(n)                   -> stream index hint (traced
     only; per-type sequential stream_number still wins because BD
     spec stream-numbering is anchored on MPLS)
   - Construction has CodingType -> Audio stream; otherwise Subtitle
   - No Language -> skip (not a stream construction)

   This handles BOTH the Disney 5-arg pattern (I, Lbe, Llp, I,
   LCodingType) and the Warner 4-arg pattern (I, Law, Lgp,
   LCodingType) automatically — same code path because args are
   identified by type rather than constructor-signature shape.

5. parse() now walks all binding-class candidates and unions
   their constructions before calling interpret_streams. Logs each
   candidate at INFO with getstatic_count for diagnosis.

Tests:
- 2 new tests verify the CodingType -> codec_hint mapping for
  known + unknown field names.
- Existing interpret_streams tests updated to use the new
  signature (dropped CodecTable arg).
- Audio-emission test rewritten to use CodingType arg instead of
  the old binding_type substring-match approach.

Confidence is still Medium for now (single-corpus verification);
ready to promote to High once tested against a third Deluxe disc.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 17:32:29 -07:00
MattJackson d7fb1b35ed disc: emit Pass 1 summary INFO log at sweep exit
Wires the existing PassSummary infrastructure (in read_error.rs as
of a832bad) into the sweep loop's exit path. One INFO log line per
Pass 1 completion gives operators an at-a-glance damage profile
without grepping per-error WARN lines:

  INFO pass1_summary  total_reads_ok=384521 total_errors=5
                      zones_entered=1 jumps_taken=2
                      bytes_good=38_725_644_288 bytes_pending=46_GB
                      copy_elapsed_ms=1751650

Particularly useful for post-mortem analysis when combined with
the per-error structured WARN logs (ms_since_last_error /
ms_since_last_success / sense_family / wedge_transition) shipped
in 0.18.10. Single line tells you the pass shape; preceding WARN
lines tell you the per-error detail.

Pass N (Disc::patch) intentionally NOT covered in this commit —
Pass N has its own retry-budget summary semantics that warrant a
separate design pass. Pass 1 sweep is where wedge incidents
originate, so it gets the diagnostic surface first.

Staged for 0.18.11. 0.18.10 already shipped the per-error WARN
layer; this is the finishing companion log.
2026-05-10 17:21:37 -07:00
MattJackson 06be4defd2 v0.18.10: bump version 2026-05-10 17:14:52 -07:00
MattJackson a832bad697 disc: structured timing + transition diagnostics for read errors
Adds the observability we need to debug wedge incidents from logs
alone — without needing to enable verbose TRACE-level SCSI tracing.
Goal stated by user: "when error occurs we can debug and code
correctly."

Pre-fix the WARN log on each read error showed only sense codes
and consecutive_failures. Missing: timing context (was the failed
read fast or slow?), gap to previous events (cumulative vs.
immediate failure?), and family transitions (did the drive just
flip into wedge mode, or has it been there?).

New fields on ReadCtx (no caller signature change):

  last_success_at: Option<Instant>
  last_error_at: Option<Instant>
  last_error_family: Option<SenseFamily>
  total_errors: u64
  total_reads_ok: u64
  zones_entered: u64
  jumps_taken: u64
  in_damage_zone: bool

New SenseFamily enum (NotReady / Medium / Hardware / IllegalRequest
/ Other) with is_wedge_family predicate.

handle_read_error WARN log now carries:
  consecutive_failures
  consecutive_outer_failures
  ms_since_last_error    NEW gap between this and previous error
  ms_since_last_success  NEW gap to last good read
  total_errors           NEW aggregate this pass
  total_reads_ok         NEW
  wedge_count
  sense_family           NEW typed category, easier to filter
  sense_key / asc / ascq (existing)

NEW WARN log "wedge_transition" fires once when the sense family
changes from non-wedge to wedge (Medium to Hardware/IllegalRequest).
That's the moment the drive's firmware flipped into fast-fail
mode. Single timestamped event in the log so post-mortems can
pinpoint the transition without scanning thousands of TRACE lines.

Worked example: if the next wedge incident shows

  read_error  ms_since_last_success=18234  ms_since_last_error=null
  read_error  ms_since_last_success=28000  ms_since_last_error=10000
  read_error  ms_since_last_success=43000  ms_since_last_error=68
                                          (drive returned <100ms = wedge symptom)
  wedge_transition  errors_in_zone=5  ms_since_last_success=43000

we can immediately tell cumulative damage, 5 errors over 43 s,
drive went into fast-fail mode at the 5th. If instead we see

  read_error  ms_since_last_success=200  ms_since_last_error=null  sense_family=Hardware
  wedge_transition  errors_in_zone=1

the wedge was triggered by ONE read at a physically-bricked LBA
(immediate fast-fail, no warm-up).

These two patterns demand different tuning responses (longer
pause vs. larger initial jump), and now we can distinguish them
from a single WARN log line each instead of needing TRACE
verbose for the whole rip.

Plus jumps_taken / zones_entered counters that feed an end-of-pass
INFO summary (PassSummary). Caller invokes pass_summary at sweep
end and logs structured stats: "Pass 1 saw N errors / M ok reads
/ K zones / J jumps". Single-line post-mortem for any rip.

No caller signature change (timing is internal to the handler;
end-of-pass summary is a new method callers opt into). Precommit
green; 433+ tests pass. Staged for the 0.18.10 release once we
have user-validation data on 0.18.9's avoidance tuning.
2026-05-10 17:10:52 -07:00
MattJackson 26e2d847d5 v0.18.9: bump version 2026-05-10 17:02:24 -07:00
MattJackson 4442fa2df6 disc: wedge AVOIDANCE on Pass 1 — inter-error pause + larger jumps
Complements the wedge-skip backstop (fbdb50c) with proactive
avoidance so we don't HIT the wedge in the first place. User's
take after seeing the Dune Pt 2 rip wedge at 48%: 'we shouldn't be
wedging.'

Empirical observations from the 23:09:12-23:09:55 wedge timeline:

  5 read errors over 43 s, ~8 s apart (drive's own ECC recovery
  takes 5-10 s per failure). Not 'hammering' in any usual sense,
  but cumulative firmware-state buildup over 5 in-cluster errors
  was enough to tip the BU40N into wedge mode at the 5th error.

  Damage cluster spanned ~140 MB (LBAs 19.898M-19.965M). Current
  damage-jump base of 256 sectors × batch=32 = 16 MB first jump,
  doubling to 32 MB, 64 MB... Each jump landed BACK INSIDE the
  140 MB cluster, exposing the drive to MORE in-cluster errors.

Two avoidance levers:

1. Inter-error pause on Pass 1 (PASS_1_FAIL_PAUSE_SECS = 5 s):
   pre-fix Pass 1 ran pause_secs=0 on all errors to 'zoom past'
   damage zones. Successful reads still zoom at zero pause — the
   pause applies only to FAILED reads, giving the drive's firmware
   cool-down between cluster exposures. Cost: ~5 s per scattered
   failure (~30-60 s total on a damage cluster); trivial vs.
   crashing the rip at 48%.

2. Larger damage-jump base (JUMP_BASE_SECTORS = 1024, up from
   256): first jump at batch=32 now covers 64 MB instead of 16 MB,
   second jump 128 MB instead of 32 MB. Two jumps clear 192 MB —
   well past most single-cluster damage patterns. Smaller jumps
   were landing inside the cluster and adding to the wedge counter.

Plus a halt-aware sleep helper (sleep_secs_or_halt) so the new
inter-error pause doesn't degrade halt response time. Halt poll
granularity 100 ms — halt fires within ~100 ms regardless of
remaining pause time. Updated three sleep call sites in disc/mod.rs
(SkipBlock pause, JumpAhead post-pause, Retry pause).

The wedge-SKIP backstop (fbdb50c) stays — combined with this
avoidance work, the flow becomes:
  damage cluster encountered →
    pause 5 s, mark NonTrimmed →
    second failure →
    pause 5 s, mark NonTrimmed →
    ...
    threshold hit →
    damage-jump 64 MB (clears 95% of clusters) →
    if jump lands in another cluster: 128 MB next jump →
    only if drive STILL wedges after all this:
      wedge-skip kicks in (1 GB jump + 30 s cooldown × 16 budget)

Tests:
  pass_1_pauses_briefly_on_skip_for_wedge_avoidance — locks the
    new 5 s pause behavior in place (replaces the old pause=0 test).
  integration test threshold bumped from 5 s to 60 s with comment
    explaining the new bound is 'not infinite' rather than
    'milliseconds-fast'.
  All 433+ tests green on cargo +1.86 fmt + clippy + test.

Precommit green.
2026-05-10 16:55:02 -07:00
MattJackson 23dc55661c labels: apply_labels integration tests + class_reader robustness fuzz tests
Closes the final two audit items from this session.

labels::apply_labels: factored out of apply() so the matching logic
is unit-testable without needing a SectorReader / UdfFs. 11 new
tests in apply_tests cover:
  - codec_hint + variant flow through to AudioStream.label
  - purpose set on audio with no label English text
  - name fallback only when purpose=Normal (CLI owns purpose i18n)
  - subtitle SDH qualifier set; forced flag flipped on Forced
  - per-type 1-based indexing (audio #2 maps to 2nd audio stream,
    not 2nd stream overall)
  - labels for nonexistent streams are no-ops
  - empty labels list leaves streams untouched
  - fill_defaults generates audio + video labels; preserves existing

class_reader: robustness smoke tests. ClassFile::parse must NEVER
panic on adversarial input — only return Err. 9 new tests:
  - empty input
  - short magic (0..4 bytes)
  - wrong magic
  - truncated after magic
  - bad CP tag
  - truncated UTF-8 in CP
  - 200 random byte buffers (deterministic xorshift)
  - 100 magic + random tail (most adversarial — magic check passes,
    everything else garbage)
  - instructions iter on random code (200 buffers)
  - instruction_size on every opcode 0..255 with varied tail buffers
  - modified_utf8 on random byte buffers (500)

The xorshift PRNG keeps the tests deterministic (no rand dep) and
reproducible — failures will be the same buffer every time. This is
the lightweight alternative to a cargo-fuzz setup; if/when we adopt
cargo-fuzz, these tests stay as regression cases.

All 451 tests passing on cargo +1.86 fmt + clippy + test.
2026-05-10 16:38:19 -07:00
MattJackson fbdb50c79f disc: Pass 1 wedge-skip instead of abort-on-first-wedge
Pre-fix: when the drive returned HARDWARE_ERROR or ILLEGAL_REQUEST
during Pass 1 sweep, libfreemkv immediately returned ReadAction::
AbortPass. Autorip surfaced this as a fatal error and stopped the
rip at whatever progress percentage Pass 1 had reached — typically
40-50%. On a disc with one physical-damage cluster, the user would
see Pass 1 die at ~48% with the cryptic message 'E6000: <lba>
0x02/0x04/0x3e' and have no rip output to work with.

Root cause analysis: BU40N firmware transitions into a fast-fail
state when it hits cumulative read failures in a small LBA range —
returns HARDWARE_ERROR for every subsequent read near that LBA, even
sectors that aren't physically damaged. Per project docs 'Bad-sector
handling' rule #2, 'Recovery requires eject+reload OR significant
cool-down.' Aborting on first wedge throws away the rest of the
disc; the right response is to SKIP the wedged region (mark as
NonTrimmed for Pass N), pause for drive cooldown, and continue.

Fix: in handle_read_error, the HARDWARE_ERROR / ILLEGAL_REQUEST arm
now branches on bisect_on_marginal:

  Pass 1 (bisect_on_marginal=false): JumpAhead with WEDGE_JUMP_SECTORS
    (1 GB at 2048 bytes/sector) and WEDGE_PAUSE_SECS (30 s cooldown).
    Tracks wedge_count in ReadCtx; resets on any successful read.
    Truly aborts only after WEDGE_ABORT_THRESHOLD (16) consecutive
    wedges with no good read in between — generous enough to clear
    most physical-damage clusters, bounded enough to not loop forever
    on a permanently bricked drive.

  Pass N (bisect_on_marginal=true): unchanged AbortPass. Pass N's
    job is single-sector recovery; if the drive won't talk near a
    specific LBA, skipping doesn't help. Pass N exits and lets the
    outer layer decide retry/eject/surface.

5 unit tests cover the new policy:
  pass_1_hardware_error_jumps_ahead_not_aborts — JumpAhead emitted
    with correct sectors+pause, wedge_count incremented.
  pass_1_hardware_error_aborts_after_threshold — AbortPass kicks in
    on the WEDGE_ABORT_THRESHOLD-th consecutive wedge.
  pass_1_good_read_resets_wedge_count — on_success clears
    wedge_count; subsequent wedge gets fresh skip budget.
  pass_n_hardware_error_still_aborts — Pass N's AbortPass behavior
    intact.
  pass_1_illegal_request_also_routes_to_wedge_skip — both wedge
    sense families get the skip treatment.

Impact: on the Dune Pt 2 disc that consistently wedged at 48%
(physical damage at LBA ~19.9M), Pass 1 will now jump ahead 1 GB
on the wedge, give the drive 30 s cooldown, and continue scanning
the rest of the disc. The damaged region becomes Pass N's job to
revisit. Worst case if the drive stays wedged: 16 GB of NonTrimmed
disc area before honest AbortPass.

Precommit (cargo +1.86 fmt + clippy + test) green; 430 passing.
2026-05-10 16:37:59 -07:00
MattJackson 7cc74f0087 labels/xml: shared tolerant XML helper, paramount + criterion onto it
Replaces two near-duplicate hand-rolled XML scrapers in paramount.rs
and criterion.rs with a single labels::xml module that's robust to:

- Case-insensitive tag / attribute names ('<Playlist>' matches the
  same as '<playlist>'; 'Name=...' matches 'name=...').
- XML namespace prefixes (matches '<ns:tag>' for tag='tag').
- Arbitrary whitespace inside open tags and around '=' separators
  ('<tag  name = "X">' works).
- Both quote styles for attribute values (" and ').
- Self-closing tag forms ('<tag/>' and '<tag />').
- '>' chars inside quoted attribute values (no premature end-of-tag).

Three functions:
  xml::attr(element, name) -> Option<String>
      Extract attribute value from an open-tag fragment.
  xml::text(xml, tag) -> Option<String>
      Trimmed text content of first <tag>...</tag>.
  xml::find_element(xml, tag, from) -> Option<(start, end)>
      Locate next <tag>...</tag> for iteration; handles self-closing.

22 unit tests cover the robustness properties: case-insensitivity,
namespace stripping, whitespace tolerance, quote styles, self-close
forms, no-substring-false-positive (looking for 'lang' must NOT
match 'lang_id' or 'language'), '>' inside quoted attrs, iteration
across repeated elements.

paramount.rs: drops local extract_attr; find_feature_playlist now
walks xml::find_element('playlist', ...) so it works regardless of
case and self-closing style. Pre-refactor: required exactly
'<playlist ' (single space, exact case) and '/>' for self-close.

criterion.rs: drops local extract_tag; parse_stream_infos and
parse_playback_config iterate via xml::find_element. Same case-
sensitivity + namespace gains. The 'COMMENTARY' / 'SDH' / 'DS'
content-value match is now case-insensitive too (previously a disc
authored with 'commentary' would have been miscategorized as Normal).

Pre-refactor known failure modes (none observed yet, but trivial
to trip on a future disc): vendor switches whitespace around '=',
uses single quotes, capitalizes a tag, prefixes a namespace. All
now handled.

Out of scope by design: XML entity decoding (&amp;, &lt;), CDATA
sections, comments, processing instructions. None observed in BD-J
authored label data. If a future disc trips them, the entity
decoder is a localized addition.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:21:51 -07:00
MattJackson 3aa1e528c5 labels/deluxe: full Phase B/C/D buildout — codec walk, binding decode
Completes the Deluxe parser pipeline. Phase A (master enums) was
already shipping; this commit lands Phases B/C/D so the parser now
emits per-stream StreamLabel records on Deluxe-authored discs.

Phase B (decode_codec_enum): walks the codec enum's subclass
references (one .class per codec ordinal) and extracts the codec
name string from each subclass's constant pool. Heuristic: pick the
first Utf8 entry that's uppercase + underscored + >=4 chars, or one
of the known codec roots (ATMOS/DOLBY/DTS/TRUEHD/MLP/AC3/EAC3/PCM)
when no underscored candidate is found. CodecTable maps ordinal ->
codec string; empty string for ordinals where extraction failed
(logged via tracing, not fatal).

Phase C (find_binding_class): identifies the class that builds the
per-stream label table by counting getstatic operations targeting
any of the master enum classes from Phase A. Class with the highest
count >= 4 wins. Threshold is empirical (real binding classes have
50+ matches; floor of 4 admits small discs while rejecting incidental
single-reference classes).

Phase D (decode_binding + BindingDecoder): symbolic stack machine
that walks the binding class's <clinit> bytecode. Handles:
  - constant pushes: iconst_<n>/bipush/sipush/ldc(Integer)
  - new <X>: pushes uninit-object marker
  - dup: stack copy
  - getstatic <Y.Z>: pushes EnumRef when Y is in MasterEnumTable,
    else Unknown
  - invokespecial X.<init>(...)V: pops args per descriptor; when the
    receiver is NewObj(X), emits a Construction { binding_type: X,
    args: [...] }
  - invokevirtual/invokestatic/invokeinterface: pop args per
    descriptor, push return placeholder unless void
  - pop/pop2/aastore/putstatic/putfield: standard stack effects
  - branches/returns: clear stack (conservative resync — binding
    <clinit> is straight-line in practice)
  parse_method_arg_count: JVMS field-descriptor parser, handles
  primitives, references (L...;), arrays ([...).

interpret_streams: converts Constructions to StreamLabels using
the master enum table + CodecTable. Each construction with a
Language ref becomes a stream. Audio when codec_hint resolves via
binding_type substring match against CodecTable; subtitle otherwise.
Purpose ordinal -> LabelPurpose via the verified Deluxe Purpose enum
order (Normal/Commentary/PiP/Trivia/Descriptive/Score/NoForced/
NoForcedDescriptive). Stream index = sequential per type. Language
goes through vocab::lang for ISO code + variant.

deluxe::parse now returns Some(ParseResult::medium(labels)) when
all four phases produce labels. Medium confidence — the bytecode
mechanism is rigorously tested but the signal-to-StreamLabel
mapping (which arg is which, audio vs subtitle classification) is
heuristic until corpus binding-class bytecode confirms the exact
pattern.

Test coverage: 13 new unit tests in deluxe.rs
  parse_method_arg_count: 3 tests (basic types, references, malformed)
  BindingDecoder: 4 tests (simple construction, with int pushes,
    skips unmatched invokespecial, resolves master-enum ordinal)
  interpret_streams: 4 tests (subtitle on no codec, audio on codec
    match, purpose routing, skips no-language)
  MasterEnumTable: 3 tests (resolve, value, class_name_set)
  extract_codec_name: 1 test (uppercase+underscore matching)

class_reader.rs gained a #[cfg(test)] ConstantPool::from_entries
test-only constructor so Phase D tests can build synthetic CP
fixtures without writing raw .class bytes.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:15:39 -07:00
MattJackson 7c6b0f82ab labels: per-parser confidence + highest-confidence-wins registry
Replaces 'first-match-wins by array order' with 'highest-confidence-
wins, array order tiebreaker'. Removes the arbitrariness when more
than one parser can claim a disc (e.g. one with both
bluray_project.bin and playlists.xml).

New types in labels::mod:
  pub enum Confidence { Medium, High }
  pub struct ParseResult { labels: Vec<StreamLabel>, confidence }
  ParseResult::high(labels) / ::medium(labels) constructors

Parser signature change: every parse() now returns
Option<ParseResult> instead of Option<Vec<StreamLabel>>. Updated all
six parsers in lockstep:
  paramount: High (fully structured XML)
  criterion: High (fully structured XML)
  pixelogic: High by default, Medium when an unknown token component
             is encountered (the skip-unknown path now propagates the
             coverage gap to the caller instead of silently degrading)
  ctrm:      High (structured key-value)
  dbp:       High (anchor scan with vocab routing)
  deluxe:    still returns None pending Phase D — signature aligned

Registry behavior:
  extract() iterates all detect-positive parsers, picks highest
  Confidence with non-empty labels. Equal confidence falls to array
  order (deterministic). Same selection logic in analyze().

LabelAnalysis grew a confidence: Option<Confidence> field so the
diagnostic surface (freemkv-tools labels-analyze) exposes which
confidence tier the selected parser claimed. labels-analyze JSON
and labels-corpus-check structural diff both gained the field.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:01:20 -07:00
MattJackson 4226a53e73 labels: fresh-eyes audit — capture variant, dedupe detect, lock registry
Three targeted fixes from a second-pass audit of the labels module.

1. vocab::lang now returns Option<LangInfo> with both code AND a
   human-readable variant string. Pre-fix: 'Brazilian Portuguese 5.1'
   became language=por, variant='', dropping the dialect info the
   disc had explicitly authored. Post-fix: language=por,
   variant='Brazilian' — matches the convention pixelogic / ctrm /
   criterion already use for their region variants. dbp now
   populates StreamLabel::variant from this. Compound table grew a
   3-tuple (needle, code, variant); bare matches still return
   variant=''.

2. dbp and deluxe had duplicated detect() boilerplate (any top-level
   .jar in /BDMV/JAR/). Both now call jar::has_any_top_level_jar.
   The trait-level detect contract — see super::PARSERS — can't peek
   inside a jar without a SectorReader, so loose-detect-plus-real-
   check-in-parse is the unavoidable pattern for jar-content parsers.
   Consolidating in jar.rs at least makes the duplication visible.

3. mod.rs comment about parser ordering said 'dbp last'; deluxe is
   actually now last. Updated to explain the dbp-before-deluxe order
   is by cost (cp-iteration cheaper than bytecode walking when Phase
   D lands).

Plus a registry-level lock test in mod.rs::registry_tests — asserts
the PARSERS array order is exactly [paramount, criterion, pixelogic,
ctrm, dbp, deluxe]. This was previously implicit; if someone reorders
the array (which changes which parser wins on overlapping signals),
unit tests would have stayed green. Now they fail with an explanatory
message about why the order matters.

Audit findings deferred to follow-ups (each its own commit + design
discussion):
- Stronger detect contract — current loose-detect-real-check pattern
  is forced by SectorReader-not-in-detect-signature; could be fixed
  by changing the trait to take an Option<&mut dyn SectorReader> or
  similar.
- Per-parser confidence scoring — registry currently first-match-wins.
  A high-confidence parser ought to beat a low-confidence one
  regardless of array order.
- class_reader fuzzing — handles malformed input via Result but no
  adversarial corpus yet.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 15:50:32 -07:00
MattJackson 92f34a289e labels: refactor pixelogic + ctrm onto shared platform + hardening
Closes the platform unification: every label parser now routes
purpose/qualifier/codec classification through one source of truth
(vocab.rs) instead of N hand-rolls, and every binary-blob byte
scanner goes through one helper (text::extract_ascii_strings).

pixelogic.rs:
- Drop local extract_strings (~20 lines) — use text::extract_ascii_strings.
- HARDENING: replace  with skip-unknown-component +
  trace log. Pre-refactor behavior: any single uncatalogued token part
  (e.g. a future codec ID, new framework variant) silently dropped the
  entire stream record. New behavior: skip just the unknown part,
  surface what we know about the stream.
- 8 new unit tests cover basic audio/subtitle paths, commentary,
  descriptive, region variant, the new skip-unknown-component
  regression, and the non-audio/non-subtitle early-out.

ctrm.rs:
- Replace  with
  vocab::purpose(&name). Now word-boundary matched — 'Commenter Pro
  Track' no longer false-matches Commentary.
- Replace  with vocab::qualifier(&name).
  Same word-boundary tightening, plus picks up Forced and
  DescriptiveService for free.
- Preserved structural commentary signal via
  as a fallback when name is silent (e.g. 'audio_commentary_1.name=Track 2').
- 6 new unit tests including the 'Commenter' false-positive regression
  and the SDH-only-on-subtitles boundary.

text.rs:
- Drop module-level dead_code allow now that pixelogic uses
  extract_ascii_strings.

Net: all 5 framework parsers now on the unified platform. Future work
(deluxe Phase D, paramount/criterion XML hardening) builds on the
same scaffolding.

Precommit green.
2026-05-10 15:33:22 -07:00
MattJackson fdbe469d50 labels: add deluxe parser (Phase A — master enum identification)
Closes the 'detected but no parser' gap on Deluxe-authored BD-J discs
(com/bydeluxe/ package signature; ~20% of UHD discs in our corpus per
the 2026-05-10 11-disc capture session).

What ships in this commit:

- detect() — registers the parser in the chain (loose pre-check at the
  /BDMV/JAR/ level; real signal in parse via has_path_prefix on
  'com/bydeluxe/').
- Phase A master enum identification — walks every .class's <clinit>
  ldc sequence and matches against framework-stable fingerprints for:
    Language    (70 ldcs starting English/French/Spanish/Dutch)
    Purpose     (8  ldcs starting Normal/Commentary/PiP/Trivia)
    VideoFormat (7  ldcs starting HD/HDR10 Plus/HD Dolby)
    Region      (22 ldcs starting USA_D1/LIC1/LIC2/LIC3 — Disney-only)
    Studio      (6  ldcs starting Disney/Marvel/Pixar — Disney-only)
  All identifications verified out-of-tree on corpus disc-01 (Disney,
  The Amateur) and disc-09 (Warner, Dune Part 1) via the standalone
  POC.
- Phase B structural skeleton (find_codec_enum) — identifies the
  codec enum class by structural signature (>=20 'new' ops, 0 ldcs),
  returns the ordered subclass list. Codec string extraction from
  subclasses is dead-coded pending the follow-up commit.

What does NOT ship yet:

- Phase D (per-stream binding-class decoder). parse() returns None
  intentionally — the master enums alone don't yield StreamLabels
  without the streamTable.put(...) bytecode walker. analyze() will
  show 'deluxe' in parsers_detected with the enum identification in
  tracing logs, so the analyzer reports honestly: 'detected, can't
  emit labels yet' rather than silent failure.

Why ship A without D: A is proven on real corpus discs; D's design
needs ground-truth binding bytecode from at least 2 corpus discs
side-by-side to verify the stack-machine pattern. The Phase A
infrastructure (master enum identification + ordinal->name table)
is what D will consume — landing it now unblocks D's design without
holding back the parser registration.

5 unit tests cover the fingerprint matcher logic + a roster lock that
forces explicit consideration when adding/removing fingerprints.

Precommit green.
2026-05-10 15:22:11 -07:00
MattJackson 4ec75a03f2 labels: shared platform (vocab/text/jar) + dbp refactor
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.

New modules:

  vocab.rs       expanded from 27 -> ~370 lines
                 + lang(text) -> Option<&'static str>      (English/multi-word
                                                            -> ISO 639-2; ~45
                                                            languages, compound
                                                            phrases like
                                                            'Brazilian Portuguese'
                                                            and 'Castilian Spanish')
                 + purpose(text) -> LabelPurpose            (Commentary,
                                                            Descriptive, Score,
                                                            Ime; word-boundary
                                                            matched)
                 + qualifier(text) -> LabelQualifier        (SDH, Forced,
                                                            DescriptiveService)
                 + has_word internal primitive — enforces word-boundary
                   matching so 'Commenter' no longer matches 'commentary' and
                   'engineering' no longer matches 'english'. Existing parsers
                   used .contains() and got lucky on the corpus; vocab now
                   guarantees the boundary in one place. 20+ unit tests.

  text.rs        NEW (~85 lines)
                 + extract_ascii_strings(data, min_len) — promoted from two
                   near-duplicate copies (pixelogic min=4, dbp min=5);
                   threshold passed in. 7 unit tests including
                   trailing-without-terminator + high-bit-byte handling.

  jar.rs         NEW (~120 lines)
                 + for_each_jar(reader, udf, fn)  — walk every top-level
                                                    .jar under /BDMV/JAR/,
                                                    yield to callback.
                 + has_path_prefix(archive, prefix) — cheap 'is this MY
                                                      framework's jar?' check
                                                      via central-dir filenames.
                 + for_each_class(archive, fn)    — parse every .class entry
                                                    through class_reader,
                                                    yield (name, &ClassFile).
                 + try_each_class(archive, fn)    — same with early-return on
                                                    first Some(R) match.

Refactored:

  dbp.rs         v2 on the new platform:
                 - dropped extract_printable raw byte scan
                 - dropped its own English -> ISO 639-2 map
                 - dropped its own parse_attributes hand-roll
                 + iterates CpInfo::Utf8 via class_reader (structurally clean,
                   no false-positive risk from method bytecode bytes)
                 + routes language/purpose/qualifier through vocab
                 All 7 prior dbp tests still pass; +2 new ones cover
                 vocab routing.

dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.

Precommit green (cargo +1.86 fmt + clippy + test).
2026-05-10 15:16:25 -07:00
MattJackson 99236ffd55 labels: add class_reader, hand-rolled JVM .class file parser
Foundation for label parsers that need structured access to .class
files inside /BDMV/JAR/<x>.jar. Replaces noak (~3KLOC dep) with a
~1000-line std-only reader.

Public API:
- ClassFile::parse(&[u8]) -> Result<ClassFile>
- ConstantPool::{get, utf8, class_name, string, integer, member_ref, iter}
- Member::code(&pool) -> Option<CodeAttribute>
- CodeAttribute::instructions() -> Instructions iterator
- Instruction::{name, operand_u8, operand_u16, cp_index}
- Opcode constants (LDC, AASTORE, NEW, GETSTATIC, INVOKESPECIAL, ...)

Spec coverage:
- Constant pool: all 17 tag types incl. Long/Double 2-slot quirk
- Modified UTF-8 incl. 0xC0 0x80 -> U+0000 special case
- Bytecode iteration with full opcode size table
- Variable-length tableswitch / lookupswitch / wide

12 unit tests cover the opcode table edge cases (padded switch tables,
wide-iinc 6-byte form), modified-UTF-8 decoder, and iterator
stop-on-truncated behaviour.

Module is currently #![allow(dead_code)] — the public API is staged
for labels::deluxe (Phases A-E bytecode walker) and a labels::dbp
refactor onto the constant-pool iterator. Tests exercise the API
in isolation. The allow comes off as those callers land.

Also fixes two pre-existing clippy lints that 1.86's stricter checks
flagged after I touched the labels module:
- src/mux/disc.rs: while-let-loop in test fixture
- tests/pass_n_size_aware_skip.rs: type_complexity in helper signature

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 15:06:34 -07:00
MattJackson 9e3327a569 v0.18.8: bump version (Pass-1 fast-skip never reached autorip 0.18.7 — Cargo.lock mis-resolved) 2026-05-10 14:01:23 -07:00
MattJackson 0081955686 labels: add dbp parser (Magnolia Pictures BD-J framework)
5th BD-J authoring framework recognized. Discriminator: any top-
level .jar in /BDMV/JAR/ that contains com/dbp/ package paths.
Identified during the 2026-05-10 corpus session via string-mining
disc-07's BD-J jar — perm files reference bd-live.magpictures.com
(Magnolia / Magnet Releasing).

Stream labels live as plain ASCII strings inside compiled .class
files in the form

    LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,...
    HTextField,Subtitle1,English SDH,Fontstrip_Composite,...
    ATextField,Subtitle0,None,Fontstrip_Composite,...

— a quirk of the menu-rendering layer encoding TextField positions
and content as constant strings the Java compiler retained in the
class string pool. The leading single-letter prefix is string-pool
ordering noise; parser anchors on `TextField,`. Subtitle0 is the
disable-subtitles button and is skipped.

The parser:
  - reads each top-level .jar via udf.read_file
  - opens it with the existing zip dependency
  - confirms com/dbp/ presence in the central directory
  - walks .class entries, extracts printable strings, matches the
    `TextField,(Audio|Subtitle)<N>,<label>,...` pattern
  - maps human-readable language names ("English", "Castilian
    Spanish", "Brazilian Portuguese", "Canadian French", ...) to
    ISO 639-2 codes via a parser-local table (per the rules-of-
    engagement memo, each parser knows its own format)
  - preserves the full disc-authored label string in `name` so
    consumers display it raw without the lib guessing further
    structure
  - detects SDH / Forced qualifiers and Commentary / Descriptive
    purposes from substring matches; everything else falls through
    to fill_defaults using BD-spec MPLS data

Verified live on the corpus: disc-07 (Civil War UHD) now matches
parser=dbp with 3 audio + 2 subtitle labels, exactly the count
visible in the disc's authored TextField definitions and what BD
spec MPLS reports.

Limitation: dbp's detect() returns true for ANY top-level .jar in
/BDMV/JAR/ (every BD-J disc has one), since the discriminator
trait function takes only `&UdfFs` and can't read jar contents.
parse() does the real com/dbp/ check — a non-dbp disc gets
parsed-as-dbp, archive_has_dbp returns false, parse() returns
None, and we fall through. Diagnostic noise: parsers_detected
includes "dbp" on non-dbp BD-J discs. Real fix is refactoring the
DetectFn signature to take a SectorReader; deferred.

7 unit tests cover the TextField extraction, language detection
(simple + compound: "Brazilian Portuguese", "Castilian Spanish",
"Canadian French", "Latin American Spanish", "Australian English"
plus the disc-corpus typo "Austrailian English"), SDH/Forced/RNIB
qualifier detection, and Commentary/Descriptive purpose detection.
Don't-guess discipline preserved: unknown languages return ""
(consumer falls back to MPLS spec data via fill_defaults).
2026-05-10 13:15:07 -07:00
MattJackson 3244fdd683 v0.18.7: Pass 1 fast-skip, defer recovery to Pass N
Pass 1 sweep was grinding through damage zones because the marginal-
media handler returned `Bisect` for every failed 32-sector batch —
forcing 32 single-sector reads per bad block at ~5s each on a real
BU40N-vs-Dune-Pt-2 trace. AND the JumpAhead trigger required a 16-
block damage window to fill before firing, so entry into a
contiguous damage zone took ~40 minutes of grinding before the
first jump fired. Architecturally wrong: Pass 1's job is "fast and
accurate, get the most data in the shortest time." Bisection +
recovery is Pass N's purpose-built role.

ReadCtx now carries two new fields:
  - `consecutive_outer_failures: u64` — outer-batch failures since
    last outer success. Bisect inner failures don't count.
  - `bisect_on_marginal: bool` — whether to return Bisect on a
    marginal-media batch failure.
  - `fast_jump_threshold: u64` — outer-failures count that triggers
    JumpAhead before the damage window has filled.

`for_sweep` (Pass 1) sets `bisect_on_marginal=false`,
`fast_jump_threshold=4`, and zeroes the post-failure pause. Failed
batches become SkipBlock → whole block NonTrimmed → advance, no
sleep. After 4 consecutive outer failures: JumpAhead with the
existing escalating multiplier.

`for_patch` (Pass N) sets `bisect_on_marginal=true`,
`fast_jump_threshold=u64::MAX`, keeps the original cooldown pauses.
Pass N's whole reason to exist is to grind on bad ranges with
proper recovery semantics — single-sector reads, 60s recovery
timeout, retry budget, escalating skip — and that's unchanged.

`on_success` resets `consecutive_outer_failures` only when not
bisecting, so a good single-sector read inside Pass N's bisect
doesn't pretend we've escaped the damaged batch.

Tests:
  - `pass_n_marginal_with_batch_gt_1_bisects` — Pass N still bisects.
  - `pass_1_marginal_skips_instead_of_bisecting` — Pass 1 doesn't.
  - `pass_1_jumps_after_4_consecutive_outer_failures` — fast-entry.
  - `pass_n_does_not_fast_jump` — fast-entry is Pass-1-only.
  - `outer_success_resets_consecutive_outer_failures` — counter reset.
  - `bisect_inner_success_does_not_reset_outer_counter` — semantics.
  - `pass_1_does_not_pause_on_skip` — explicit zero-pause contract.
  - `long_failure_streak_extends_pause_on_pass_n` — Pass N still
    extends pauses on long failure streaks (renamed from the old
    sweep-based test).

Integration test `test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed`
updated: it used to assert Pass 1 recovers all sectors via bisect
(bytes_good=total). New contract: Pass 1 marks NonTrimmed; Pass N
recovers. Test now asserts Pass-1-only outcome (bytes_pending=total,
complete=false) consistent with the redesign.

Real-world impact on the user's BU40N + Dune Pt 2 trace from this
session: a damage zone that was on track to take ~40 minutes of
Pass-1 grinding will now jump in ~20 seconds. Pass N still has the
full 7-pass recovery budget to revisit those NonTrimmed ranges.
2026-05-10 12:50:12 -07:00
MattJackson ef5487e5a5 labels: distinguish "parser detected" from "parser succeeded"
Adds `parsers_detected: Vec<&'static str>` to `LabelAnalysis`. Records
every parser whose discriminator matched, regardless of whether its
parse() then returned Some/None.

Why: when `parser=None` we currently can't tell apart:
  (a) no parser recognized this disc — missing parser, candidate for a
      new module
  (b) a parser recognized it but parse() returned None / empty —
      capture truncated, or genuine empty authoring data, or a parser
      bug

Surfaced concretely on the 11-disc capture session 2026-05-10:
disc-04 had `bluray_project.bin` in jar_inventory (pixelogic detect()
returned true) but parse() returned None because file content was
past the 1 GB capture window. Old API: parser=None — looked like
"missing parser." New API: detected=[pixelogic], parser=None — clearly
"capture problem, not a parser gap."

The tracing log line on the no-parser-emitted-labels path now
distinguishes the two cases too.

No behavior change to production rip path. extract() is unchanged;
only the diagnostic analyze() returns the richer result.
2026-05-10 12:36:05 -07:00
MattJackson e7dffa63d2 v0.18.6: bump version (unified release with bdemu/freemkv/autorip) 2026-05-10 10:03:03 -07:00
MattJackson e5989c8270 labels: expose analyze() for corpus regression tooling
Promotes `mod labels` to `pub mod labels` and adds `analyze()` plus
`LabelAnalysis` (both `#[doc(hidden)]`) so an out-of-tree diagnostic
binary (freemkv-tools labels-analyze) can introspect which BD-J parser
matched a given disc, what JAR files the discriminators saw, and what
labels came out — without going through the production `apply()` path
that mutates DiscTitles.

Also adds `tracing::info!(parser = name, "label parser matched")` /
"no label parser matched" inside `extract()`. Dev-only signal: users
get the same seamless behavior; developers can finally tell whether a
disc hit a real parser or fell through to the codec-name fallback in
fill_defaults().

The new `jar_inventory()` helper deduplicates and sorts filenames
under any `/BDMV/JAR/<x>/` subdirectory — same plumbing the existing
`jar_file_exists()` discriminators use, just enumerated rather than
predicate-tested. Used by `analyze()` to surface unrecognized
parser-source files when no parser matches, which is the input to
"do we need a new parser?" triage.

No behavior change to the production label path. `apply()` and
`extract()` remain functionally identical; the new public surface
exists alongside.
2026-05-10 07:42:45 -07:00
MattJackson e1c938b82a ci: drop --locked from libfreemkv workflows
libfreemkv is a library — Cargo.lock is gitignored (standard for
libs). --locked refuses to create a lockfile on a fresh runner,
so it always fails CI. --locked stays in the binary crates
(freemkv, autorip, bdemu) which DO track Cargo.lock and benefit
from the dependency-race hard-fail behaviour.
2026-05-09 20:40:31 -07:00
MattJackson ed801a708b fmt: rustfmt-mandated reflow of canonical_title_order tests
The 0.18.4 commit landed with rustfmt diffs in the new
canonical_order tests because my local validation script piped
'cargo fmt --check' to 'tail -1', masking the diff output and
reporting green when fmt was actually unhappy. CI's lint job
caught it immediately. No code change — pure formatting.
2026-05-09 20:34:40 -07:00
MattJackson d94451954c v0.18.4: cargo --locked everywhere — hard-fail dependency races 2026-05-09 20:30:34 -07:00
MattJackson 385c9f094c v0.18.3: canonical_title_order — main feature first on branching UHDs 2026-05-09 20:08:56 -07:00
MattJackson 9f2a13739d Disc title order: main feature first on branching UHDs
Disc::titles previously sorted purely by duration_secs descending,
which puts a play-all virtual playlist at index 0 on UHDs that ship
one. Such playlists reference the same source clips multiple times
for seamless alternate-angle / alternate-ending playback and report
inflated duration AND inflated size_bytes that exceeds the disc's
physical capacity.

Concrete observed case (The Amateur 2025 4K UHD, 58.5 GB BD-100):
  Title 1 — 00020.mpls — 4h13m — 92.4 GB — 253 clips  ← impossible
  Title 2 — 00800.mpls — 2h02m — 57.2 GB — 1 clip      ← the movie

92.4 GB > 58.5 GB capacity is proof of clip double-counting. With
the duration-only sort, freemkv -t 1 / disc.titles.first() / autorip's
main-feature picker all selected the 4-hour composite instead of the
2-hour movie.

New canonical_title_order:
  1. Real titles (size_bytes <= capacity_bytes) before virtual
     composites — capacity gate is hard physical truth.
  2. Among real titles, fewer clips first (1-clip wins as the
     canonical main feature; multi-clip is either chapter-stitched
     or composite).
  3. Tiebreak on longer duration first.

Behaviour:
- Non-branching discs: unchanged. The longest 1-clip title is
  already the movie.
- Branching UHDs: virtual composite drops to the back, the real
  movie surfaces at index 0.

Comparator exposed as Disc::canonical_title_order for downstream
consumers that need the same logic on custom title sets.

Three regression tests (disc::tests::canonical_order_*):
- pushes_oversize_play_all_behind_real_main (The Amateur)
- preserves_natural_ranking_on_normal_disc
- fewer_clips_wins_tiebreak
2026-05-09 19:57:07 -07:00
MattJackson d9ce69bc9d v0.18.2: fix AACS nav-file scramble + sweep progress non-regression
decrypt::decrypt_sectors now restores chunks when decrypt_unit_full's
TS-sync verification fails, preventing 0.18.1's silent corruption of
MPLS/CLPI navigation files when DecryptingSectorSource decorates the
sweep reader. Fixes E6009 NoStreams on info iso:// for AACS-encrypted
UHDs ripped without --raw.

Disc::sweep progress takes max(snapshot.bytes_good, bytes_done) so
the user-visible counter never regresses below what the producer has
already sent.
2026-05-09 17:19:47 -07:00
MattJackson 7cd2c937ed 0.18.1 docs: refresh README, CHANGELOG, and docs/ for the trait split
The library's public-facing docs were sitting on the 0.17 trait
surface — Disc::copy, pes::Stream, SectorReader, etc. — even though
all in-tree callers migrated in 0.18 rounds 1-3. With 0.18.1 about
to ship, a user copy-pasting the README sample from crates.io would
have hit a compile error.

This commit is purely doc-side:

- README.md: Quick Start rewritten onto Disc::sweep + Disc::patch
  with caller-orchestrated multipass; Streams table footnote and
  Architecture row reference FrameSource / FrameSink.
- CHANGELOG.md: 0.18.1 entry describing the redesign — primitives,
  trait splits, deprecations (kept alive through 0.18.x, deletion
  target 0.18.2), throughput numbers.
- docs/{rip-recovery,api-design,architecture,disc-to-rip,
  drive-access,udf}.md: every Disc::copy / pes::Stream /
  SectorReader reference updated to the 0.18 trait surface.
- FEATURES.md: deleted (8+ versions stale; capabilities live in
  README.md and CHANGELOG.md now, matching the workspace-top
  FEATURES.md removal in 84acd65).
- examples/iso_dump.rs: verified compiles against 0.18.1.

No code changes.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 12:13:51 -07:00
MattJackson 107524da5e 0.18.1: bump version 2026-05-09 11:38:24 -07:00
MattJackson 055a3c5276 0.18 round 3: make Disc::sweep + Disc::patch pub (was pub(crate))
Round 3 step 1: lift the visibility on the two flat rip-phase verbs
so consumers (autorip + freemkv CLI) can call them directly instead
of going through Disc::copy's multipass dispatcher. Also lift their
option/outcome types and re-export at crate root.

- fn sweep -> pub fn sweep (with rustdoc explaining its role)
- fn patch -> pub fn patch (ditto)
- pub(crate) struct SweepOptions -> pub struct SweepOptions
- pub(crate) struct PatchOpts -> pub struct PatchOptions (renamed
  for consistency — both are 'Options')
- pub(crate) struct PatchOutcome -> pub struct PatchOutcome
- libfreemkv::{SweepOptions, PatchOptions, PatchOutcome} re-exports
  at crate root.

Disc::copy still exists and still calls Disc::sweep / Disc::patch
through the now-private sweep_internal / patch_internal wrappers.
Migration of the two autorip callers + the freemkv CLI's
disc_to_iso to direct sweep/patch is a follow-up; once those land
Disc::copy + CopyOptions + CopyResult delete in the same commit.

See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.

Single contributor: MattJackson.
2026-05-09 11:19:46 -07:00
MattJackson 50b04790ce 0.18 round 2: thread Halt through DiscStream construction
# Conflicts:
#	src/mux/disc.rs
2026-05-09 11:04:07 -07:00
MattJackson a00b2c884b 0.18 round 2: adopt DecryptingSectorSource decorator at sweep + patch + DiscStream 2026-05-09 11:03:43 -07:00
MattJackson 44e2c64d7e 0.18 round 2 (Halt threading): DiscStream accepts Halt at construction
Adds a Halt field to DiscStream, populated via the new
`with_halt(halt)` builder. The internal recovery / fill_extents
loops check `halt.is_cancelled()` directly. The existing
`set_halt(Arc<AtomicBool>)` method stays through the deprecation
window for callers (autorip mux) that haven't migrated; marked
#[deprecated] with a pointer to the constructor-time path.

Both signals are unified inside DiscStream: either Halt or the
legacy Arc<AtomicBool> triggers cancellation, so callers can mix
during the deprecation window without breaking stop behaviour.

See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.

Single contributor: MattJackson.
2026-05-09 10:58:54 -07:00
MattJackson d6535b8f57 0.18 round 2 (decrypt dedup): adopt DecryptingSectorSource at the two
existing call sites — sweep producer and DiscStream demux

Round 1 shipped the DecryptingSectorSource decorator
(libfreemkv/src/sector/decrypting.rs) but the existing decrypt
sites kept calling crate::decrypt::decrypt_sectors inline. This
commit migrates both:

- Disc::sweep (disc/mod.rs): producer wraps the input reader
  in DecryptingSectorSource::new(reader, keys) before the read loop.
  The inline decrypt_sectors call goes away — read_sectors yields
  plaintext directly.

- DiscStream (mux/disc.rs): constructor wraps the underlying
  Box<dyn SectorReader> in DecryptingSectorSource so the internal
  fill_extents / read path sees plaintext bytes. The DecryptKeys
  field stays on DiscStream for metadata-side use; it just no
  longer drives decryption.

Disc::patch carried the same inline decrypt step at three call
sites (main read, backtrack read, non-NOT_READY retry read). All
three migrated onto the same wrapping for a single audit surface.

Two small support changes carry the migration without touching
the round-1 decorator shape:
- sector/mod.rs gains specific SectorSource impls for
  &mut dyn SectorReader and Box<dyn SectorReader>, mirroring
  std's Read forwarding pattern. Generic blankets would conflict
  with the existing SectorReader → SectorSource blanket under the
  orphan rule (downstream could impl SectorReader for &mut U), so
  the impls are scoped to the dyn-trait shape we actually consume.
- sector/decrypting.rs gains DecryptingSectorSource::set_keys so
  DiscStream::set_raw() can flip the wrapped reader to a
  DecryptKeys::None pass-through without rebuilding the decorator
  (which would require moving the inner Box out from behind &mut self).

After this commit, grep `decrypt_sectors` in src/ shows the
function definition, its single use inside DecryptingSectorSource,
plus comments only. One audit surface for AACS / CSS / passthrough
correctness.

Behaviour-preserving: same plaintext bytes flow through; the only
difference is which type owns the decrypt step.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:49:53 -07:00
MattJackson 90ab00ed45 0.18 round 2: cargo fmt after sweep+patch+DiscStream merge
Auto-fmt nit on the multi-line map.flush() expression that landed
when the sweep + patch + DiscStream-FrameSource branches were merged
together. No semantic change.
2026-05-09 10:36:40 -07:00
MattJackson 6bfd7dfd13 0.18 round 2: confirm DiscStream as FrameSource via blanket impl 2026-05-09 10:32:56 -07:00
MattJackson b9a7f601d4 0.18 round 2: refactor Disc::patch onto Pipeline + PatchSink
# Conflicts:
#	src/io/mod.rs
#	src/io/pipeline.rs
2026-05-09 10:32:51 -07:00
MattJackson cf0a61f8b4 0.18 round 2: refactor Disc::sweep onto Pipeline + SweepSink (delete sweep_pipeline.rs) 2026-05-09 10:32:05 -07:00
MattJackson f98f07b2d3 0.18 round 2: refactor Disc::sweep onto Pipeline + SweepSink
Sweep was the original producer/consumer split that motivated the
generic Pipeline primitive (round 1, commit 198268b). Now that
Pipeline + Sink exist, sweep stops shipping its own bespoke
threading.

- New SweepSink: Sink<WorkItem> impl in src/disc/sweep.rs. Owns
  WritebackFile + Mapfile + ProgressSnapshot back-channel. apply()
  carries the file-write + mapfile.record per WorkItem; close()
  drains writeback, fsyncs, flushes mapfile.
- Disc::sweep: constructs SweepSink, calls Pipeline::spawn_named
  (so the consumer thread keeps showing up as
  freemkv-sweep-consumer), sends WorkItems, calls pipe.finish().
  The producer-side ReadCtx state machine, decrypt, set_speed,
  halt — all unchanged.
- Pipeline gains spawn_named(name, depth, sink) so callers can
  preserve identifiable thread names without the primitive baking
  one in. Also adds Pipeline::try_send for the throttled
  StatsRequest path that must not block the producer.
- Deleted src/disc/sweep_pipeline.rs entirely. WorkItem,
  ProgressSnapshot, ConsumerSummary moved into disc/sweep.rs as
  module-private types. WorkItem::Finish dropped — dropping the
  channel is the end-of-stream signal Pipeline already uses.

Behaviour-preserving: the sweep algorithm, mapfile invariants,
back-pressure via channel depth (DEFAULT_PIPELINE_DEPTH = 4) all
match the 0.17.13 implementation. New synthetic regression test
(sweep_pipeline_full_good_100_batches) exercises ~100 batches of
clean reads end-to-end through the new Pipeline path and verifies
bytes_good and ISO file size.

See (internal)/memory/0_18_redesign.md.
2026-05-09 10:31:05 -07:00
MattJackson b53454fa09 0.18 round 2: refactor Disc::patch onto Pipeline + PatchSink
Patch was strictly serial (per-sector recovery: read → seek+write
→ mapfile.record → next). Lifting the write+record onto a consumer
thread lets the drive issue the next per-sector retry while the
previous block's recovered bytes are being committed — small but
real win on damaged discs with many bad sectors, and uniform with
sweep's threading model.

- New PatchSink: Sink<PatchItem> impl in src/disc/patch.rs. Owns
  WritebackFile + Mapfile. apply() seeks+writes recovered bytes
  and records mapfile state per item; close() runs sync_all and
  mapfile.flush.
- Channel depth: WRITE_THROUGH_DEPTH (1). Patch wants minimum
  buffering — back-pressure should kick in immediately so the
  drive's per-sector retry budget isn't ahead of the writer.
- Disc::patch: keeps every existing recovery decision on the
  producer (reverse walk, damage-window skip, NOT_READY pauses,
  bridge-degradation handling, wedge exit, range watchdog).
  WritebackFile ownership moves to the sink.

Behaviour-preserving: per-sector single-shot read budget unchanged
(BU40N+Initio bridge wedge concern still respected); recovery
algorithm bit-identical.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:28:14 -07:00
MattJackson f28b6ee6d9 0.18 round 2: re-export Pipeline + Sink + Flow at the crate root
Round 2 #1 landed `Pipeline` / `Sink` / `Flow` / `DEFAULT_PIPELINE_DEPTH`
in `crate::io::pipeline` but only re-exported them through
`crate::io` (which is `pub(crate)`), so no out-of-tree consumer could
reach them. autorip's round 2 #2 (lifting the mux loop onto Pipeline +
MuxSink) is the first such consumer; surface the primitives at the
crate root for ergonomic access.

No behaviour change — the items themselves are unchanged from round
2 #1; this is just `pub use` plumbing.

Single contributor: MattJackson.
2026-05-09 10:22:44 -07:00
MattJackson d9d778fa64 0.18 round 2: confirm DiscStream as FrameSource
Mirror of the FrameSink concrete migrations slice (f52e4c5) on the
read side. DiscStream is the only meaningful source impl in tree;
all the mux/* impls are sinks.

The round-1 blanket impl<T: Stream + Send> FrameSource for T
already covers DiscStream if it's Send. This slice:

- Audits DiscStream's interior types for Send (its Box<dyn
  SectorReader> already requires Send via the trait's super-bound;
  verify nothing else interior breaks Send).
- Adds a synthetic-input test that constructs Box<dyn FrameSource>
  over a DiscStream, reads frames through the trait object, and
  exercises info() / headers_ready() / codec_private().
- (Conditional) Adds a direct FrameSource impl on DiscStream only
  if call-site ergonomics demand it; otherwise relies on the
  blanket.

No caller migrated. mux::resolve::input still returns
Box<dyn Stream>; autorip / CLI consumers still call Stream::read.
Per-caller migration is a later slice.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:13:14 -07:00
MattJackson bbfb887a35 0.18 round 1+2 integration fixes
Two clippy issues surfaced when round 1 polish + round 2 FrameSink
migrations both landed on libfreemkv main:

- src/halt.rs: clippy::new_without_default fires when a public new()
  exists without Default. The polish pass dropped the derive thinking
  it was redundant — clippy disagrees, so add a manual impl that
  forwards to new(). Doc-comment notes why both exist.

- src/disc/read_error.rs:372: pre-existing
  assert_eq!(.., true) trips clippy::bool_assert_comparison. Pre-0.18
  precommits passed because that lint sat outside the gate; the
  round-2 commits brought enough new clippy surface that it now
  shows up. Trivial cleanup: assert!(...) instead of assert_eq!.

Single contributor: MattJackson.
2026-05-09 10:00:06 -07:00
MattJackson 9884346c24 0.18 round 2: FrameSink concrete migrations for mux/* sinks 2026-05-09 09:53:12 -07:00
MattJackson 766b7c6636 0.18 round 1 polish: address libfreemkv code-review findings 2026-05-09 09:53:06 -07:00
MattJackson 925c30686b 0.18 round 1 polish: address libfreemkv code-review findings
Applies must-fix + in-scope should-fix items from the round-1 code
review:

- M1: FileSectorSource::open takes &Path (was &str — non-UTF-8 panic)
- M2: drop FileSectorSource's BufReader (defeated by absolute seeks)
- M3: WritebackFile Drop impl finalises the writeback pipeline
- M4: Pipeline::finish preserves panic payload in error message
- M5: pes::Stream is left without a : Send supertrait — concrete
  in-tree impls (MkvStream, M2tsStream) hold Box<dyn Read> /
  Box<dyn Write> trait objects that aren't Send, so the simple
  trait tightening would cascade into a wider Send audit. Per the
  review's escape clause the FrameSource blanket impl keeps its
  T: Send bound and the constraint is documented loudly there.
- S6: document Pipeline::send post-Flow::Stop semantics
- S9: truncate stale Stream docs (E9001/E9000 was runtime-only)
- S10: document WritebackPipeline.fd lifetime invariant
- S11: pub use pes::Stream as PesStream to disambiguate from
  disc::Stream codec enum at crate root
- S12: rename DEFAULT_DEPTH → DEFAULT_PIPELINE_DEPTH; add
  WRITE_THROUGH_DEPTH constant
- N14: drop Halt's Default derive (redundant with Halt::new)
- N17: Pipeline::spawn propagates thread-spawn error instead of expect
- N19: deprecation since = "0.18.0" (was "0.18.0-dev", non-conventional)
- N21: rename Apply enum to Flow

Deferred to follow-up commits: SectorReader/SectorSource competition
(migration commit), WritebackFile::create/open orphans (migration
commit), AACS round-trip test (design doc defers), various nits.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:52:25 -07:00
MattJackson f52e4c5d22 0.18 round 2: add FrameSink impls to concrete mux sinks
Per-impl migration of MkvStream / M2tsStream / NetworkStream /
NullStream / StdioStream from the deprecated pes::Stream trait
to the typed pes::FrameSink trait. Both impls coexist during
the 0.18 deprecation window — the existing Stream impls are
unchanged.

The FrameSink::finish signature differs (Box<Self> vs &mut self),
which is why this couldn't be a blanket impl. Each migration
re-borrows the box and delegates to the underlying Stream::finish
body.

FrameSink: Send forced two struct fields (M2tsStream's boxed
Write/Read, MkvStream's boxed WriteSeek/Read) to gain `+ Send`
bounds — minimum surface needed to make the Send-bounded trait
impl-able. mux::resolve::output's local Box<dyn WriteSeek>
construction picks up the same `+ Send`. tests/streams.rs's
shared `stream.write/.finish/.info/.read` calls were
disambiguated to `PesStream::*` to resolve the now-multiple
candidates from coexisting trait impls.

Caller migration (mux::resolve::output return type, autorip,
CLI) is a later slice. This commit only adds new impls; nothing
removed.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:49:18 -07:00
MattJackson 8d67790d2e 0.18: thread WritebackFile rename through FileSectorSink
The SectorSource/Sink agent and the WritebackFile-rename agent both
branched from main concurrently; the sector branch wrote against the
0.17 Writer name and only the rename branch knew about WritebackFile.
This integration commit reconciles the two: FileSectorSink::create /
::open / the inner-field type all use WritebackFile directly, and the
module-level + struct-level docs are corrected.
2026-05-09 09:17:39 -07:00
MattJackson 246a439990 0.18: FrameSource/FrameSink trait split (deprecate Stream) 2026-05-09 09:13:39 -07:00
MattJackson bd19041648 0.18: SectorSource/SectorSink trait split + DecryptingSectorSource decorator 2026-05-09 09:13:34 -07:00
MattJackson 198268b725 0.18: add crate::io::Pipeline + Sink trait
# Conflicts:
#	src/io/mod.rs
2026-05-09 09:13:30 -07:00
MattJackson 31424c8203 0.18: add crate::halt::Halt cancellation token 2026-05-09 09:13:07 -07:00
MattJackson c747ecc589 0.18: rename crate::io::Writer → WritebackFile 2026-05-09 09:12:59 -07:00
MattJackson 283a561c12 0.18 primitive: SectorSource/SectorSink trait split + DecryptingSectorSource
Splits the unidirectional read trait from a (planned) write trait at
the sector level, eliminating runtime "wrong direction" potential.
Keeps SectorReader alive as a pre-deprecation alias via blanket impl
so existing callers compile unchanged through the migration window.

Adds DecryptingSectorSource decorator: wrap any SectorSource in this
to get plaintext sectors out. Replaces the duplicate decrypt code
paths in sweep_pipeline and DiscStream (those migrations are
follow-up commits).

The formal #[deprecated] attribute on SectorReader is held back to a
follow-up commit because internal call sites in disc/, udf/, mux/,
and verify/ still go through the legacy trait, and the CI gauntlet
treats deprecation lints as errors. Behavioural intent — "this trait
is going away" — is documented on the trait itself.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:04:16 -07:00
MattJackson 2667d68675 0.18 primitive: FrameSource/FrameSink trait split (deprecate Stream)
Splits the bidirectional pes::Stream into one-direction traits so
calling read() on a write-only sink is a compile error instead of
runtime E9001. Keeps Stream alive as deprecated through 0.18 with a
blanket FrameSource impl so existing concrete types compile unchanged.

FrameSink can't be blanket-impl'd from Stream (different finish
signature), so concrete impls migrate per-type in a follow-up.

Concrete `impl pes::Stream for X` blocks in mux/* and the existing
tests gain a one-line `#[allow(deprecated)]` to keep `-D warnings`
clean during the deprecation window — no behavior changes.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 08:57:48 -07:00
MattJackson 57f2c22e30 0.18 primitive: crate::io::Pipeline + Sink trait
Generic bounded producer/consumer pipeline. The same shape applies to
sweep, patch, and mux today via three near-duplicate implementations
(or, in patch's and mux's case, no implementation at all). 0.18
collapses them onto one primitive. See
(internal)/memory/0_18_redesign.md for full context.

Single contributor: MattJackson.
2026-05-09 08:56:31 -07:00
MattJackson 5f3545d244 0.18 primitive: rename crate::io::Writer → WritebackFile
The type's job is the bounded-cache writeback pipeline (sync_file_range
+ posix_fadvise(DONTNEED)) — not generic writing. The 0.17 name was
ambiguous; reading `Writer::new(file)` gave no hint about what was
special. New name makes the role obvious at every call site.

Adds `WritebackFile::create(path)` and `WritebackFile::open(path)`
constructors so callers don't have to assemble a `File` first.

No alias kept; this is a clean 0.18 rename. See
(internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 08:53:17 -07:00
MattJackson 6e17ef0859 0.18 primitive: crate::halt::Halt cancellation token
One-bit cooperative cancellation flag. Replaces ad-hoc Arc<AtomicBool>
patterns scattered across libfreemkv (DiscStream::set_halt) and the
HALT_FLAGS global registry in autorip. See
(internal)/memory/0_18_redesign.md.

Single contributor: MattJackson (no attribution trailers).
2026-05-09 08:50:13 -07:00
MattJackson 6ec97af104 v0.17.13: thread Writer through patch + mux for big-write consistency
The bounded-cache writeback wrapper (crate::io::Writer) was added in
0.17.10 and wired into Disc::sweep in 0.17.11, but the other two
paths in the crate that write large amounts of data sequentially —
Disc::patch and the MKV/M2TS mux — were still operating on raw
std::fs::File. That meant the dirty-page burst pathology the wrapper
exists to prevent could still bite on slow / network-attached staging
during recovery and mux phases.

This release plugs those gaps:

- Disc::patch (disc/mod.rs:1981) now wraps the reopened ISO in
  Writer before any seek / write. sync_all on Writer cleanly drains
  the in-flight chunk before the existing fsync.
- mux/resolve.rs MKV and M2TS branches wrap the output File in
  Writer underneath BufWriter. UHD MKV mux routinely produces 70+ GB
  of sequential output; the page cache no longer absorbs that as a
  single hot blast on slow targets.

Mapfile, log, settings, history, and stream-pipeline byte buffers
remain unchanged: those are either small one-shot writes (where
the wrapper has zero benefit and adds a stream_position syscall) or
already use bounded persistence (mapfile time-batched in 0.17.12).
The principle: any path that writes substantial sequential data to
a single file uses Writer; trivial writes don't.
2026-05-09 06:32:08 -07:00
MattJackson 3a6c1aa5a3 v0.17.12: mapfile time-batched persistence — unblock NFS staging
Pre-0.17.12 every Mapfile::record() persisted the full mapfile via
tempfile-create + write + atomic-rename. On local LVM that's
microseconds; on NFS each rename is multiple RPCs through the
unraid user-share fuse layer, dragging a Black Mass UHD rip from
~11 MB/s on local to ~1.5 MB/s on NFS — the mapfile path alone burned
multiple seconds of wall time per real-world second of work.

Mapfile now batches the rename to once per second:

- record() always updates in-memory state and stats; only fires
  write_to_disk when last_flushed.elapsed() >= FLUSH_INTERVAL (1 s).
- New flush() API forces a persist; called by sweep_pipeline's
  consumer at end-of-sweep and by Disc::patch at end-of-patch,
  after the file's sync_all.
- Drop impl best-effort flushes so an early-return / unwind doesn't
  silently lose pending state.

Crash-safety changes from "lose at most one block" to "lose at most
1 s of recorded progress" — the ISO file's payload bytes are unaffected;
only the mapfile's authority over which sectors are already-good is at
risk, and a resume re-reads anything Pass 1 had already covered.

Measured on the BU40N test bed against Black Mass UHD inner zone:
- NFS staging: 1.5 MB/s → 16.48 MB/s (10.9× recovery)
- Local LVM staging: 11.09 MB/s → 11.83 MB/s (+6.7 % bonus)

Internal round_trip_load test now flushes before reading back from
disk. External patch / copy tests are unaffected: patch and
sweep_pipeline flush at completion before returning.
2026-05-08 23:00:52 -07:00
MattJackson 1ba3264747 v0.17.11: sweep producer/consumer split — overlap drive read with file write
Pre-0.17.11 sweep ran strictly serialised: SCSI read → decrypt → seek
+ write → mapfile.record → next read. Drive idled for the post-read
work; throughput capped at the sum of both costs. On a healthy disc
that's ~7-12 ms read + ~5-15 ms write/record per 64 KB batch, limiting
sustained throughput to ~10-12 MB/s on the test bed (BU40N + UHD inner
zone), well below the ~14-16 MB/s drive ceiling.

Decouples them: producer thread (caller's) owns SectorReader +
read_error state + decrypt + set_speed + halt; consumer thread (one
spawn) owns Writer + Mapfile, receives WorkItem messages, applies
file write + mapfile record. Bounded mpsc::sync_channel(4) gives
natural back-pressure. While the consumer writes batch N, the
producer is already reading batch N+1 — steady-state throughput is
now bound by the slower of the two pipelines (drive on healthy
discs), not their sum.

Side effects:
- Bisect path now decrypts. Pre-0.17.11 the bisect inner loop wrote
  raw cyphertext for single-sector recoveries on encrypted discs —
  quiet correctness bug exercised only by batch-fail-then-
  bisect-succeed on encrypted media. New producer-side decrypt
  covers main + bisect success paths uniformly.
- All read_ctx state stays single-threaded on producer (damage
  window, jump multiplier, etc.). No locking added.
- Mapfile remains single-writer on consumer. No locking.
- Halt latency: producer breaks loop, sends Finish, consumer drains
  ≤4 in-flight items + sync_all. ~1 batch (~12 ms) typical.
- BU40N + Initio bridge wedge concern unchanged: still single SCSI
  command in flight, error-path timing identical, no new retries.

New module: src/disc/sweep_pipeline.rs (WorkItem, ProgressSnapshot,
ConsumerInputs, spawn_consumer, consumer_loop, helpers). Public API
unchanged — Disc::copy / CopyOptions / CopyResult identical.

Patch (Pass N) is NOT changed; it's bound by drive recovery time, not
the read/write serialisation.
2026-05-08 21:32:11 -07:00
MattJackson ae2909fe8d v0.17.10: bounded-cache writeback pipeline for big sequential writes
Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
settings. Empirical cause: the kernel's vm.dirty_ratio (~20% of RAM)
lets hundreds of MB of dirty pages accumulate, then bursts a flush at
99% disk utilisation that blocks app writes for ~1 s. Confirmed on
the BU40N test bed — dirty pages grew 112 → 563 MB between bursts;
lowering vm.dirty_bytes to 64 MB at the host sysctl level eliminated
the dips. Shipping the equivalent inside libfreemkv so users do not
need to tune the host kernel.

- New crate::io::Writer: drop-in File wrapper (impl Write + Seek).
  Wraps a per-platform WritebackPipeline that on Linux schedules
  sync_file_range(WRITE) + lagging sync_file_range(WAIT_AFTER) +
  posix_fadvise(DONTNEED) in 32 MB chunks, bounding dirty cache at
  ~64 MB. macOS and Windows ship a no-op stub.
- Disc::sweep wraps its output File in Writer. Loop body unchanged.
- Module is purpose-built so any large sequential output (patch,
  mux) can adopt the same wrapper as a one-line change later.
2026-05-08 19:54:25 -07:00
MattJackson 6a3d19a453 v0.17.7: sync release — no functional changes
Version bump to keep the four freemkv crates at unified versioning
after autorip's v0.17.6 + v0.17.7 work today. No libfreemkv code
changes; republished to crates.io so downstream consumers stay
aligned on the latest patch version.
2026-05-08 16:20:28 -07:00
MattJackson a35596d2d1 v0.17.5: Pass N kernel block-device fallback + per-range fixes
Direct-SATA BU40N + Dune Part Two UHD live testing exposed that the
v0.17.3 single-shot SCSI READ path matched 0/22 of the small bad-
sector LBAs that dd if=/dev/sr0 recovers on the same drive. This
release closes that gap and fixes adjacent bugs silently capping
recovery.

- /dev/sr0 pread fallback in Drive::read (Linux only): on SCSI READ
  Err, fall back to posix_fadvise(DONTNEED) + pread() against the
  corresponding block device. Kernel sr_mod runs ~5 internal retries
  with no per-attempt mid-layer escalation overhead — the mechanism
  behind dd's recovery advantage. End-to-end byte verification
  confirms the fallback path returns real disc data.

- Disc::patch per-range watchdog fix: MAX_RANGE_SECS was breaking
  'outer (one slow range killed the entire patch). Now skips to the
  next range. Pre-fix patch died after 4 sectors of range 1 of 47.

- Per-sector range budget: range_budget = sectors × 25 s, capped at
  1800 s. Replaces the flat 180 s/range that was unfair to medium
  ranges and pointlessly generous to single-sector ones.

- consecutive_failures resets per range. The wedge-exit detector is
  for stuck-on-one-range, not many-small-ranges-with-one-fail-each.

- Reverted inline 5× retry experiment (was hurting: each retry paid
  kernel SCSI escalation overhead). Restored READ_RECOVERY_TIMEOUT_MS
  to 60 s. The kernel-auto-retry pattern is now provided by sr0
  fallback.

Empirical: pass 1 recovered 94.6 MB / 11 s of main title (33 sr0
saves). Pass 2 added 0.6 MB. Remaining ~233 MB on the test disc
appears physically unrecoverable on this hardware.
2026-05-08 12:58:48 -07:00
MattJackson 318f654fed v0.17.3: fix scsi_recovery.rs Linux compile (missing Duration import)
tests/scsi_recovery.rs:
- Add `use std::time::Duration` inside both `#[cfg(target_os = "linux")]`
  blocks. Locally on macOS the linux blocks are cfg-out so the missing
  import was invisible to precommit on macOS.
- Bug pre-dated this branch but only surfaced when v0.17.2 release CI
  ran the test compile on Linux.

Cargo.toml: 0.17.2 -> 0.17.3.
2026-05-07 19:16:21 -07:00
MattJackson 5e82ef65de v0.17.2: bump version (skip 0.17.1 — stray tag exists)
Cargo.toml: 0.17.1 -> 0.17.2. Functionally identical to the prior
commit; 0.17.1 was never published to crates.io but a tag exists on
the remote pointing at an unrelated commit. Bumping past it.
2026-05-07 19:10:39 -07:00
MattJackson 8534607329 v0.17.1: cache priming, NonTrimmed marking, decrypt regression test
src/disc/mod.rs:
- Cache priming (3-sector lookback) before patch's single-sector reads.
  Drive read-ahead pulls in adjacent pages so the target may already be
  cached when we ask for it. Throwaway reads — failures here don't
  update mapfile state.
- When patch hits skip-limit on a range, leave remaining sectors
  NonTrimmed instead of marking Unreadable. We never tried to read those
  sectors, so don't give them terminal status — drive state evolves
  between passes (cache, mechanical settle), and a later pass may
  succeed.

tests/pass_n_patch_fix.rs:
- New regression test for the decrypt key inversion bug at
  src/disc/mod.rs:1938-1942. Asserts decrypt_sectors is invoked with
  the correct key when opts.decrypt=true.

tests/pass_n_size_aware_skip.rs:
- rustfmt-only changes.

Cargo.toml: 0.17.0 -> 0.17.1.
2026-05-07 19:09:17 -07:00
MattJackson 97e1a4cad3 unified read-error handler + pass N size-aware skip
New disc/read_error.rs as the single entry point all read failures flow
through. Handler classifies the error, updates the in-flight context
(damage window, retry budgets, jump multiplier), and returns a
ReadAction the caller dispatches on. Pass 1 (sweep) refactored to use
it; ~340 lines of nested if/else collapsed into ~120 lines of action
dispatch. Adding a new error class = one match arm. Logging is in one
place. Bisect inner failures don't poison the damage window. Jump
multiplier capped at 64 (max 1 GB jump for batch=32 — observed prior
unbounded behavior produce a single 56 GB jump on a wedged drive).

Pass N (patch) damage_skip is now size-aware: each skip is capped at
range_remaining/4 rather than the absolute MB-scale escalation. The
old logic could leap over a 100-sector bad range that hides a 50-sector
good middle; size-aware convergence finds the good middles instead.

Tests in tests/pass_n_size_aware_skip.rs exercise the size-aware skip
against synthetic patterns (25-bad/50-good/25-bad and three good
middles in a row) and prove ≥98% of good middles are recovered.
Existing test test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed
updated to reflect that MEDIUM_ERROR now triggers single-sector
bisect (which the BlockSizeFailingReader succeeds at).
2026-05-07 09:08:43 -07:00
MattJackson 3f6bf33de5 Format code 2026-05-04 20:06:10 -07:00
MattJackson 359301e8fc Pass 1 transport failure recovery without user intervention 2026-05-04 20:02:01 -07:00
MattJackson bfa527162a 0.17.0: unwrap safety fix, patch pass algorithm, clippy compliance
- Fix unwrap in disc/mod.rs sweep() hot path using pattern matching
- Patch pass excludes Unreadable sectors from work list
- Expose bytes_bad_in_title for accurate UI reporting
- All 256 tests pass, cargo clippy clean with -D warnings
2026-05-04 09:42:07 -07:00
MattJackson 45defd47f3 fix patch pass: exclude Unreadable from work list; expose bytes_bad_in_title; clippy 1.86 fixes
- patch(): only process NonTrimmed + NonScraped ranges (Unreadable=terminal, NonTried=not-yet-swept)
- bytes_bad_in_title: pub fn for autorip main-movie lost_ms computation
- Clippy 1.86: saturating_sub, unused assignments, unused variable
- fmt: rustfmt formatting
2026-05-03 16:35:06 -07:00
72 changed files with 14604 additions and 1365 deletions
+4
View File
@@ -14,6 +14,10 @@ jobs:
with:
components: clippy, rustfmt
- run: cargo fmt --check
# libfreemkv is a library — Cargo.lock is gitignored. --locked
# would always fail on a fresh runner because there's no committed
# lockfile to lock against. The binary crates (freemkv, autorip,
# bdemu) track Cargo.lock and DO use --locked.
- run: cargo clippy -- -D warnings
test:
+2
View File
@@ -28,6 +28,8 @@ jobs:
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
# libfreemkv is a library — Cargo.lock isn't tracked, so --locked
# would always fail (no lockfile to lock against on a fresh runner).
- run: cargo test
publish:
+404
View File
@@ -1,5 +1,409 @@
# Changelog
## 0.18.4 (2026-05-09)
### Build / CI hardening — no library code changes
- All `cargo` invocations in `.github/workflows/*.yml` now use
`--locked`. Previously `cargo build` / `cargo test` could silently
re-resolve `Cargo.lock` if a lock-pinned version wasn't yet on
crates.io — masking dependency races between same-tag releases of
sibling crates. The 0.18.3 release hit this: autorip's docker image
built `libfreemkv v0.18.2` because libfreemkv 0.18.3 hadn't
published yet at the time autorip's CI ran, and cargo silently fell
back to the previous version. With `--locked`, that race now hard-
fails the build with a clear "package X is not in registry" error
and we retrigger after the upstream lands.
- `cargo publish --locked` in the release workflow guards against
publishing a lockfile-mismatched release.
## 0.18.3 (2026-05-09)
### Behaviour change
- **`Disc::titles[0]` is now the canonical main feature on branching
discs, not the longest playlist.** Previously titles were sorted
purely by `duration_secs` descending, which puts a "play-all"
virtual playlist (alternate angles / seamless branching) at index 0
on UHDs that ship one — its inflated duration overshoots the real
movie. Concrete example: *The Amateur (2025)* 4K UHD on a 58.5 GB
disc has Title 1 = `00020.mpls` 4h13m / 92.4 GB / 253 clips
(impossible — the size exceeds the disc capacity, proving it's a
virtual composite) and Title 2 = `00800.mpls` 2h02m / 57.2 GB /
1 clip (the actual film, matching TMDB).
New sort priority:
1. Real titles (`size_bytes <= capacity_bytes`) before virtual
composites.
2. Among real titles, fewer clips first (1-clip wins).
3. Tiebreak on longer duration first.
**Migration:** consumers calling `disc.titles.first()` /
`disc.titles[0]` automatically get the corrected title — no code
changes needed. CLI users invoking `freemkv -t 1 disc:// …` now hit
the actual main feature on branching discs (this was the user-
visible bug). On non-branching discs the order is unchanged.
The comparator is exposed as
[`Disc::canonical_title_order`](#) for callers that need to
re-sort a custom title set with the same logic.
Regression tests: `disc::tests::canonical_order_*` (three cases:
branching-UHD, normal disc, clip-count tiebreak).
## 0.18.2 (2026-05-09)
### Bug fixes
- **Nav-file scramble during AACS rip** (`decrypt::decrypt_sectors`). 0.18's
`DecryptingSectorSource` decorator broadened the call surface of
`decrypt_sectors` to every sector flowing through sweep, including UDF
navigation files (MPLS playlists, CLPI clip-info). The byte-0 heuristic
in `aacs::is_unit_encrypted` correctly fires on m2ts source-packet copy
markers but false-positives on any binary file whose first byte happens
to have the top 2 bits set — most notably MPLS files (start with 'M' =
0x4D) and CLPI files (start with 'H' = 0x48). `decrypt_unit_full`
already self-checks the result via TS-sync verification and returns
`false` on a misfire, but the chunk had been mutated by then;
`decrypt_sectors` discarded the return value, leaving scrambled bytes
in the ISO. Fix: snapshot the chunk before decryption and restore on
verification failure (same pattern `decrypt_unit_try_keys` already used
for multi-key discs). Symptom: `freemkv info iso://UHD.iso` and
`iso:// → mkv://` returned E6009 NoStreams on freshly-ripped UHD ISOs;
affected only the iso-source path, not the disc-source path or the
m2ts video payload itself. Regression test:
`decrypt::tests::nav_file_unit_survives_decrypt_attempt`.
- **Sweep progress display can regress to zero** (`Disc::sweep`). The
consumer-side `bytes_good` snapshot lags producer-side `bytes_done`
whenever the consumer is behind on draining the work channel. Until
the first snapshot arrived, the placeholder branch reported
`bytes_done` correctly; once a stale snapshot landed, the report
switched to `snap.bytes_good` and could regress below `bytes_done`.
Fix: `bytes_good_total = max(snap.bytes_good, bytes_done)`, so the
user-visible counter never moves backward.
## 0.18.1 (2026-05-09)
### I/O stack redesign — primitives over orchestration
0.18 reshapes the read/write surface from "library does the multipass dance"
to "library hands the caller flat verbs and a few composable primitives." All
in-tree consumers (autorip, the `freemkv` CLI) drove their own multipass
loops in 0.18 rounds 1-3; 0.18.1 lands the cleanup with the deprecated names
still alive for one minor-version window.
The 0.18 design notes are in
`(internal)/memory/0_18_redesign.md` (private) — this entry sticks to
what changed at the public surface.
#### Flat verbs
`Disc::sweep` is the forward Pass 1. `Disc::patch` is one retry pass over the
mapfile. Neither knows about pass index, retry budget, or accept-loss policy
— the caller invokes them in whatever sequence its use case dictates. The
old multipass-aware `Disc::copy` dispatcher is deprecated and slated for
deletion in 0.18.2; no in-tree caller still uses it.
#### Trait splits — direction-typed at compile time
- `pes::Stream` (combined read+write) is split into `FrameSource` and
`FrameSink`. Calling `read()` on a write-only sink is now a compile error,
not the runtime `E9001` (`StreamWriteOnly`) it used to be.
- `SectorReader` is split into `SectorSource` (read) and `SectorSink`
(write). `Drive` impls `SectorSource` only; `FileSectorSource` /
`FileSectorSink` replace `FileSectorReader` for ISO-backed I/O.
- A blanket impl bridges legacy `SectorReader` callers onto the new
`SectorSource` so existing code keeps compiling through the deprecation
window.
#### New primitives
- `Halt` — one cancellation token (cloneable, `Arc<AtomicBool>` under the
hood) replaces the three near-duplicate halt flags scattered through the
workspace. Threaded through every long-running loop.
- `Pipeline<I, R>` + `Sink<I>` — generic producer/consumer primitive in
`crate::io`. Replaces the bespoke `disc/sweep_pipeline.rs` and now also
drives `Disc::patch` and the autorip mux loop. `DEFAULT_PIPELINE_DEPTH`
for streaming reads (`4`); `WRITE_THROUGH_DEPTH` for write-through patch
semantics (`1`).
- `WritebackFile` — was `crate::io::Writer`. The renamed type makes its job
explicit: a `File` wrapper that runs continuous `sync_file_range` +
`posix_fadvise(DONTNEED)` to keep the kernel dirty-page cache bounded on
long sequential writes.
- `DecryptingSectorSource<S>` — a single decorator wrapping any
`SectorSource` to yield plaintext sectors. One audit surface for AACS /
CSS / passthrough; the previous two-site decrypt (sweep producer +
`DiscStream` demux) is gone.
#### Throughput
The round-2 producer/consumer split is now applied uniformly to sweep,
patch, and mux. Mux on NFS-staged UHD measured ~16 MB/s sustained on the
test bed (was ~12 MB/s pre-round-2).
#### Module reorg
- `sector/` and `io/` are now module directories.
- `disc/sweep.rs`, `disc/patch.rs`, and `disc/mapfile.rs` split out of the
monolithic `disc/mod.rs`.
#### Renames (no behavior change)
- `crate::io::Writer` → `crate::io::WritebackFile`.
- `Apply` → `Flow` (`Sink::apply` return value).
- `DEFAULT_DEPTH` → `DEFAULT_PIPELINE_DEPTH`.
- `PatchOpts` → `PatchOptions`.
#### Deprecated (alive in 0.18.1, deletion target 0.18.2)
`Disc::copy`, `pes::Stream`, `SectorReader`, `FileSectorReader`,
`CopyOptions`, `CopyResult`, `DiscStream::set_halt`. Each compiles with a
deprecation warning; all in-tree call sites have migrated.
## 0.17.13 (2026-05-09)
### Use `crate::io::Writer` uniformly for all binary file output
0.17.10 introduced the bounded-cache writeback wrapper (`sync_file_range`
+ `posix_fadvise(DONTNEED)` per chunk) and 0.17.11 wired it into
`Disc::sweep`. The other two big-write paths in the crate were still
opening raw `std::fs::File` and would have hit the same dirty-page
burst pathology against slow / network-attached staging.
This release threads `crate::io::Writer` through the remaining sites:
- **`Disc::patch`** (`disc/mod.rs:1981`): the ISO file reopened for
Pass-N recovery now wraps in `Writer` before any seek / write.
Recovery writes are sparse, but the wrapper costs nothing when
there's no chunk crossing — and on heavily-damaged discs it
matters as patch accumulates GB of recovered data.
- **MKV mux output** (`mux/resolve.rs:243`): `BufWriter::with_capacity`
now wraps `Writer::new(file)` instead of a raw `File`. UHD MKVs
routinely exceed 70 GB of sequential writes; pre-0.17.13 those
bursts went straight to the kernel writeback queue.
- **M2TS mux output** (`mux/resolve.rs:251`): same change for
parity with the MKV path — anyone using `m2ts://` URLs gets it
too.
No new public surface. `Writer::sync_all()` is called from `Disc::patch`
and `sweep_pipeline`'s consumer at end of pass; mux exits via Drop on
the wrapping `BufWriter`, which propagates flush down to `Writer`'s
final `note_progress` (kernel finishes the in-flight chunk on file
close — there's no explicit `sync_all` for mux today, same behaviour
as before).
## 0.17.12 (2026-05-09)
### Mapfile time-batched persistence — unblock NFS staging
Pre-0.17.12 every `Mapfile::record()` wrote the entire mapfile to disk
via tempfile-create + buffered-write + atomic-rename. On local LVM
that's effectively free (page cache + microsecond-scale renames). On
NFS (autorip's intended staging path for centralised media) each
record() became three RPCs through the unraid user share's shfs-fuse
layer — measured end-to-end at multiple ms each. With ~170 record()
calls per second of sustained sweep, the mapfile path alone burned
multiple seconds of wall time per real-world second of work, dragging
the rip from ~11 MB/s on local to ~1.5 MB/s on NFS.
Fix: time-batch the persistence inside `Mapfile`.
- `record()` always updates in-memory state and stats (so callers'
`stats()` reads stay coherent in the same process).
- The `write_to_disk()` rename only fires when ≥ `FLUSH_INTERVAL`
(1 s) has elapsed since the last persist.
- New `flush()` method forces an out-of-band persist; called by
`sweep_pipeline`'s consumer at end-of-sweep and by `Disc::patch` at
end-of-patch, after the file's `sync_all()`.
- `Drop` impl best-effort flushes so an early-return / unwind doesn't
silently lose pending state.
Crash-safety changes from "lose at most one block" to "lose at most
1 s of recorded progress" — the ISO file's payload bytes are
unaffected; only the mapfile's authority over which sectors are
already-good is at risk, and a resume re-reads anything Pass 1 had
already covered. Acceptable for a 7× throughput recovery on the
target deployment.
The internal `round_trip_load` test now calls `flush()` before
`Mapfile::load` to read back what the in-memory state asserts.
External patch / copy tests are unaffected: `patch` and
`sweep_pipeline` flush at completion before returning, so any
`Mapfile::load` at the call-site sees fully persisted state.
## 0.17.11 (2026-05-09)
### Sweep producer/consumer split — overlap drive read with file write
Pre-0.17.11 the sweep loop ran strictly serialised: SCSI read → decrypt
→ seek + write → mapfile.record → next read. The drive idled for the
post-read work; throughput capped at the **sum** of both costs. On a
healthy disc that's ~7-12 ms read + ~5-15 ms write/record per 64 KB
batch — limiting sustained throughput to ~10-12 MB/s on the test bed
(BU40N + UHD inner zone), well below the drive's ceiling of ~14-16 MB/s.
This release decouples them with a producer / consumer split:
- **Producer** (caller's thread): owns the `SectorReader`, the entire
`read_error` state machine (Retry, Bisect, SkipBlock, JumpAhead,
AbortPass), `set_speed` damage-zone transitions, halt check, and
decrypt. Hands plaintext bytes to the consumer.
- **Consumer** (one spawned thread): owns the `crate::io::Writer` and
the `Mapfile`. Receives `WorkItem` messages and applies the file
write + mapfile record per item.
- **Channel**: bounded `mpsc::sync_channel(4)` — natural back-pressure
via blocking `send` when consumer falls behind.
While the consumer is writing batch N to disk and updating the mapfile,
the producer is already reading batch N+1 from the drive. Steady-state
throughput is now bound by the slower of the two pipelines (the drive,
on a healthy disc) instead of their sum.
Side effects of the refactor:
- **Bisect path now decrypts.** Pre-0.17.11 the bisect inner loop wrote
raw cyphertext when `decrypt=true` and a single sector was recovered
via single-sector retry — a quiet correctness bug exercised only by
the (rare) batch-fail-then-bisect-succeed path on encrypted discs.
The new producer-side decrypt covers both the main success path and
the bisect inner success path.
- All `read_error::ReadCtx` state stays single-threaded on the producer
(damage window, jump multiplier, consecutive-good count, etc.). No
locking added.
- Mapfile remains single-writer (consumer-only). No locking.
- Halt-flag responsiveness unchanged: producer breaks the loop on
signal, sends `Finish`, consumer drains its ≤4 in-flight items and
exits within ~1 batch (~12 ms typical).
- BU40N + Initio bridge wedge concern unchanged: still one SCSI command
in flight, error-path timing identical, no new retry logic.
New module: `src/disc/sweep_pipeline.rs` (`WorkItem`, `ProgressSnapshot`,
`ConsumerInputs`, `spawn_consumer`, `consumer_loop`, send/recv helpers).
Public API surface unchanged — `Disc::copy` / `CopyOptions` /
`CopyResult` look identical to callers.
Patch (Pass N) is **not** affected by this release. Patch is bound by
drive recovery time (60 s timeouts on bad sectors), not the read↔write
serialisation; a similar split there would yield negligible benefit.
## 0.17.10 (2026-05-09)
### Bounded-cache writeback for big sequential writes
Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
settings. Cause: the kernel's `vm.dirty_ratio` (~20 % of RAM) lets
hundreds of MB of dirty pages accumulate, then bursts a flush at
99 % disk utilisation that blocks app writes for ~1 s. Confirmed
empirically on the BU40N test bed: dirty pages grew 112 → 563 MB
between bursts; lowering `vm.dirty_bytes` to 64 MB at the host
sysctl level eliminated the dips.
This release ships the equivalent inside libfreemkv so users don't
need to tune the host kernel:
- New `crate::io::Writer` — drop-in `File` wrapper implementing
`Write` + `Seek`. Wraps a per-platform `WritebackPipeline` that on
Linux schedules `sync_file_range(WRITE)` + lagging
`sync_file_range(WAIT_AFTER)` + `posix_fadvise(DONTNEED)` calls in
32 MB chunks, keeping dirty cache bounded at ~64 MB. macOS and
Windows ship a no-op stub — their default cache policies don't
exhibit the same pathology for our access pattern.
- Disc::sweep wraps its output `File` in `Writer`. No changes to the
loop body — `Writer` forwards `seek`/`write_all` to `File` and
drives the pipeline transparently.
- Module is purpose-built for any large sequential output (sweep,
patch, future mux) — they can adopt `crate::io::Writer` with a
one-line wrapper and inherit the same behaviour.
## 0.17.7 (2026-05-08)
### Sync release — no functional libfreemkv changes
Version bump to keep the four freemkv crates at unified versioning.
autorip shipped UI smoothness + audit fixes today (v0.17.6, v0.17.7);
libfreemkv carries no code changes, but is bumped + republished to
crates.io so downstream `libfreemkv = "0.17"` consumers stay aligned
on the latest patch version. See autorip CHANGELOG for the
user-visible work in this release window.
## 0.17.5 (2026-05-08)
### Pass N recovery — kernel block-device fallback + per-range fixes
Live testing on direct-SATA BU40N + AACS-encrypted UHD disc revealed that
the v0.17.3 single-shot SCSI READ path matched 0/22 of the small bad-sector
LBAs that `dd if=/dev/sr0` recovers on the same drive. This release closes
that gap and fixes several adjacent bugs that were silently capping
recovery.
- **`/dev/sr0` pread fallback in `Drive::read` (Linux)**: when a SCSI READ
via `/dev/sg*` returns Err, fall back to `posix_fadvise(POSIX_FADV_DONTNEED)
+ pread()` against the corresponding block device. The kernel `sr_mod`
driver runs ~5 internal retries per command without the per-attempt
error-escalation overhead that userspace SG_IO retries pay, which is the
source of dd's recovery advantage. End-to-end byte-verification confirms
the fallback path returns real disc data (md5-equivalent to fresh dd from
the same drive session). The block fd is opened in `Drive::open` by
resolving `/sys/class/scsi_generic/sgN/device/block`; best-effort, with no
fallback if open fails.
- **`Disc::patch` per-range watchdog fix**: when a range hit `MAX_RANGE_SECS`,
the old code did `wedged_exit = true; break 'outer;` — a single slow range
killed the entire patch. Now `break;` (skip this range, advance the outer
for loop). Pre-fix, patch was dying after 4 sectors of the first slow
range and never reaching the other 46.
- **Per-sector range budget**: replaced flat `MAX_RANGE_SECS = 180` with
`range_budget_secs = (range_sectors × SECONDS_PER_SECTOR).min(RANGE_BUDGET_CAP_SECS)`.
Tiny ranges exit fast (1-sector range = 25 s); medium ranges get
proportional time (51-sector range ≈ 1275 s); large ranges still bounded
by the 1800 s cap so they cannot monopolise pass 1.
- **`consecutive_failures` resets per range**: the wedge-exit detector
(`>= 50 consecutive failures`) is for "stuck on the same range"; pre-fix
the counter persisted across ranges, so 50 small post-bisection ranges
with one failure each falsely tripped the wedge mid-pass. Reset at every
range boundary.
- **Reverted inline 5× retry in patch**: a brief experiment that was
measurably harmful — each "2 s" SCSI timeout paid ~1.5 s kernel SCSI
mid-layer error-escalation overhead, so 5× retry took ~17 s per LBA and
triggered the per-range watchdog in 4 sectors. Restored
`READ_RECOVERY_TIMEOUT_MS = 60_000` (the v0.17.3 baseline; the kernel-
auto-retry pattern is now provided by the `/dev/sr0` fallback above).
### Empirical results (Dune Part Two UHD, BU40N direct SATA)
- Pass 1: **94.6 MB recovered** (28% of formerly-bad data, 33 sr0 fallback
saves), **11 s of main-title content** restored. Patch completed all 47
retryable ranges naturally (was wedging at range 1 of 47 in v0.17.3).
- Pass 2 cumulative: 95.2 MB (+0.6 MB; diminishing returns curve).
- Remaining ~233 MB on the test disc appears physically unrecoverable on
this hardware (kernel auto-retry can't decode it either).
### Behavioural notes
- The `/dev/sr0` fallback is Linux only; macOS and Windows fall back to
the existing single-shot SCSI behaviour. The fallback is gated to
`recovery=true` reads (only fires from the patch path, not the sweep
path) to avoid page-cache pressure during multi-GB sequential ripping.
## 0.17.0 (2026-05-04)
### Code quality: unwrap safety, clippy compliance, test coverage
- **Sweep() hot path**: Replaced `.err().unwrap()` in `disc/mod.rs:1553` with explicit pattern matching (`match read_result { Ok(_) => unreachable!(), Err(e) => e }`). The original unwrap was logically safe (in the else branch after `is_ok()` check) but pattern matching makes the invariant explicit and avoids panic risk if logic changes.
- **cargo clippy --lib**: Clean build with `-D warnings` across all targets. No lint failures.
- **cargo test --lib**: All 256 tests pass (0 failed, 1 ignored). Integration tests for sweep/patch/dev-null validate the multi-pass recovery pipeline end-to-end.
### Patch pass algorithm fix
- `patch_internal` excludes Unreadable sectors from work list (only retries NonTrimmed/NonScraped/Unscraped). Previously attempted to retry already-failed sectors, wasting time and masking real progress.
- Exposes `bytes_bad_in_title` in patch results for accurate UI reporting of unrecoverable data vs. recovered data.
## 0.16.1 (2026-04-30)
### Unified progress display for sweep and patch
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "libfreemkv"
version = "0.16.3"
version = "0.20.2"
edition = "2024"
rust-version = "1.86"
license = "AGPL-3.0-only"
-34
View File
@@ -1,34 +0,0 @@
# libfreemkv — Feature List
## v0.10.10 (current)
### Done
- [x] Drive access: open, identify, unlock, firmware upload, speed calibration, eject
- [x] 206 bundled drive profiles (MediaTek MT1959 A + B variants)
- [x] SCSI transport: Linux SG_IO, macOS IOKit, Windows SPTI
- [x] UDF 2.50 filesystem parser (metadata partitions, Blu-ray profile)
- [x] MPLS playlist parser (play items, STN table, secondary streams)
- [x] CLPI clip info parser (EP map, sector extents)
- [x] AACS 1.0 decryption (4 VUK paths: KEYDB, media key, processing key, device key)
- [x] AACS 2.0 SCSI handshake (P-256/SHA-256 ECDH, bus decryption, read data key)
- [x] DVD IFO parser (VMG, VTS, PGC chains, cell addresses)
- [x] DVD CSS decryption (bus auth, disc key via player keys, title key, sector descramble)
- [x] CSS Stevenson plaintext attack for ISO key recovery
- [x] MPEG-2 PS demuxer (DVD Program Stream with PES extraction)
- [x] MPEG-2 video codec parser (sequence header, quantizer matrices, keyframe detection)
- [x] KEYDB.cfg download, verify, save (raw TCP, zero HTTP deps)
- [x] Content reading with adaptive batch sizing, error recovery, 12+ MB/s
- [x] Stream labels: 5 BD-J format parsers (Paramount, Criterion, Pixelogic, CTRM, Deluxe)
- [x] MKV muxer: 14 codec parsers (H.264, HEVC, MPEG-2, AC-3, DTS, TrueHD, PGS, DVD Sub, VC-1, LPCM, +4 more)
- [x] SectorReader trait: decouples disc scanning from SCSI
- [x] 7 stream types: Disc, ISO, MKV, M2TS, Network, Stdio, Null
- [x] PES pipeline with unified Stream trait (any source → any dest)
- [x] FMKV metadata header for M2TS and network streams
- [x] Numeric error codes only (no English text in library)
- [x] Event system for progress callbacks
### Planned
- [ ] Windows testing on real hardware
- [ ] Pioneer Renesas platform support (48 drives, need GET_CONFIG 010C)
- [ ] TranscodeStream (ffmpeg integration)
- [ ] ISO write with BD-compliant UDF structure
+41 -19
View File
@@ -18,7 +18,7 @@ Part of the [freemkv](https://github.com/freemkv) project.
```toml
[dependencies]
libfreemkv = "0.13"
libfreemkv = "0.18"
```
## Quick Start
@@ -53,27 +53,48 @@ output.finish()?;
### Multi-pass recovery rip
For damaged discs, the library offers a two-stage rip model: fast sweep with zero-fill and a ddrescue-format mapfile, then targeted retry of bad ranges. See [`docs/rip-recovery.md`](docs/rip-recovery.md) for the full architecture.
For damaged discs the library exposes two flat verbs — `Disc::sweep` for the
forward Pass 1 and `Disc::patch` for retrying bad ranges. The library never
loops; the multipass policy is the caller's job. See
[`docs/rip-recovery.md`](docs/rip-recovery.md) and the design notes in
`(internal)/memory/0_18_redesign.md`.
```rust
use libfreemkv::disc::{CopyOptions, PatchOptions};
use libfreemkv::{SweepOptions, PatchOptions};
use libfreemkv::disc::{mapfile, mapfile_path_for};
use std::path::Path;
// Pass 1: disc → ISO. Fast 64 KB reads, skip-forward on failure,
// zero-fill bad blocks, write a sidecar .mapfile.
let mut result = disc.copy(
&mut drive,
Path::new("disc.iso"),
&CopyOptions { skip_on_error: true, skip_forward: true, ..Default::default() },
)?;
let iso = Path::new("disc.iso");
// Pass 2..N: retry bad ranges with full drive recovery.
// Idempotent — call as many times as you want.
while result.bytes_unreadable + result.bytes_pending > 0 {
let pr = disc.patch(&mut drive, Path::new("disc.iso"), &PatchOptions::default())?;
if pr.bytes_recovered_this_pass == 0 { break; }
// Pass 1: disc → ISO. Skip-on-error, zero-fill, write the sidecar mapfile.
disc.sweep(&mut drive, iso, &SweepOptions {
decrypt: true,
resume: false,
batch_sectors: None,
skip_on_error: true,
progress: None,
halt: None,
})?;
// Pass 2..N: retry every non-finished range. Idempotent.
loop {
let map = mapfile::Mapfile::load(&mapfile_path_for(iso))?;
let stats = map.stats();
if stats.bytes_pending + stats.bytes_unreadable == 0 { break; }
let outcome = disc.patch(&mut drive, iso, &PatchOptions {
decrypt: true,
block_sectors: None,
full_recovery: true,
reverse: true,
wedged_threshold: 50,
progress: None,
halt: None,
})?;
if outcome.bytes_recovered_this_pass == 0 { break; }
}
// Then mux from the ISO via the normal stream pipeline (no drive involvement).
// Mux from the ISO via the normal stream pipeline (no drive involvement).
```
## What It Does
@@ -92,14 +113,14 @@ while result.bytes_unreadable + result.bytes_pending > 0 {
| Stream | Input | Output | Transport |
|--------|-------|--------|-----------|
| DiscStream | Yes | -- | Optical drive via SCSI |
| IsoStream | Yes | -- | Blu-ray ISO image file (read via stream pipeline; written via `Disc::copy()`) |
| IsoStream | Yes | -- | Blu-ray ISO image file (read via stream pipeline; written via `Disc::sweep()`) |
| MkvStream | Yes | Yes | Matroska container |
| M2tsStream | Yes | Yes | BD transport stream with FMKV metadata header |
| NetworkStream | Yes (listen) | Yes (connect) | TCP with FMKV metadata header |
| StdioStream | Yes (stdin) | Yes (stdout) | Raw byte pipe |
| NullStream | -- | Yes | Discard sink (byte counter for benchmarks) |
Streams implement `pes::Stream` (frame-level). `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
Streams implement `FrameSource` (read) and/or `FrameSink` (write); direction is type-checked. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
AACS decryption requires a KEYDB.cfg file. If available at `~/.config/aacs/KEYDB.cfg` or passed via `ScanOptions`, the library handles everything — handshake, key derivation, and per-sector decryption — without the application needing to know anything about encryption.
@@ -122,7 +143,8 @@ Disc — scan titles, streams, AACS/CSS state
└── KEYDB — download + verify + save
Streams — unified PES pipeline
├── pes::Stream — read()/write() PES frames
├── FrameSource — read() PES frames (direction-typed)
├── FrameSink — write() PES frames (direction-typed)
├── DiscStream — sectors → decrypt → TS demux → PES
├── IsoStream — ISO file → decrypt → TS demux → PES
├── MkvStream — MKV mux/demux
+39 -37
View File
@@ -44,35 +44,30 @@ while let Ok(Some(frame)) = input.read() {
output.finish()?;
```
The `pes::Stream` trait:
The `FrameSource` and `FrameSink` traits — direction is type-checked, so
calling `read()` on a write-only sink (or `write()` on a read-only source)
is a compile error rather than a runtime fault:
```rust
pub trait Stream {
fn read(&mut self) -> io::Result<Option<PesFrame>>;
fn write(&mut self, frame: &PesFrame) -> io::Result<()>;
fn finish(&mut self) -> io::Result<()>;
pub trait FrameSource: Send {
fn read(&mut self) -> Result<Option<PesFrame>, Error>;
fn info(&self) -> &DiscTitle;
fn codec_private(&self, track: usize) -> Option<Vec<u8>>;
fn headers_ready(&self) -> bool;
fn codec_private(&self, track: usize) -> Option<Vec<u8>> { None }
fn headers_ready(&self) -> bool { true }
}
```
## IOStream (byte-level API)
For raw byte copies (disc→ISO, resume, benchmarks). Lower level than PES.
```rust
let opts = InputOptions::default();
let mut input = open_input("iso://Disc.iso", &opts)?;
let mut output = open_output("mkv://Movie.mkv", input.info())?;
io::copy(&mut *input, &mut *output)?;
output.finish()?;
pub trait FrameSink: Send {
fn write(&mut self, frame: &PesFrame) -> Result<(), Error>;
fn finish(self: Box<Self>) -> Result<(), Error>;
fn info(&self) -> &DiscTitle;
}
```
## Streams
All streams implement `IOStream` (byte-level) and/or `pes::Stream` (frame-level).
URL-based resolvers open any stream by string.
All streams implement `FrameSource` (read) and/or `FrameSink` (write); the
directional split prevents runtime "wrong-direction" errors. URL-based
resolvers open any stream by string.
| Stream | Input | Output | URL | Transport |
|--------|-------|--------|-----|-----------|
@@ -87,21 +82,14 @@ URL-based resolvers open any stream by string.
All URLs require a `scheme://path` format. Bare paths are rejected.
```rust
// PES pipeline (frame-level)
// PES pipeline (frame-level) — input() returns Box<dyn FrameSource>,
// output() returns Box<dyn FrameSink>.
let input = libfreemkv::input("disc:///dev/sg4", &opts)?; // DiscStream
let input = libfreemkv::input("iso://Dune.iso", &opts)?; // IsoStream
let output = libfreemkv::output("mkv://Dune.mkv", &title)?; // MkvOutputStream
let output = libfreemkv::output("m2ts://Dune.m2ts", &title)?; // M2tsOutputStream
let output = libfreemkv::output("network://10.0.0.1:9000", &title)?; // NetworkOutputStream
let output = libfreemkv::output("null://", &title)?; // NullOutputStream
// IOStream (byte-level)
let input = open_input("disc://", &opts)?; // DiscStream
let input = open_input("iso://Dune.iso", &opts)?; // IsoStream
let output = open_output("iso://Copy.iso", &meta)?; // IsoStream (write)
let output = open_output("mkv://Dune.mkv", &meta)?; // MkvStream
let output = open_output("m2ts://Dune.m2ts", &meta)?; // M2tsStream
let output = open_output("null://", &meta)?; // NullStream
```
### FMKV Metadata Header
@@ -175,20 +163,34 @@ libfreemkv/src/
├── lib.rs Public exports
├── error.rs Error codes (no English)
├── event.rs Event types for callbacks
├── halt.rs Halt cancellation token (Arc<AtomicBool> wrapper)
├── io/ Pipeline + WritebackFile primitives
│ ├── mod.rs Re-exports WritebackFile, Pipeline, Sink, Flow
│ ├── pipeline.rs Generic Pipeline<I, R> + Sink trait
│ ├── writeback_file.rs WritebackFile (was crate::io::Writer)
│ └── writeback.rs sync_file_range pipeline
├── drive/ Drive (open, init, single-shot read)
│ ├── mod.rs Drive struct, init, read (single-shot), reset, eject
│ ├── capture.rs Drive profile capture for contribution
│ ├── linux.rs Linux drive discovery
│ ├── macos.rs macOS drive discovery
│ └── windows.rs Windows drive discovery
├── disc/ Disc (scan, titles, AACS setup)
├── disc/ Disc (scan, titles, AACS setup, sweep, patch)
│ ├── mod.rs Disc struct, scan, titles, formats
│ ├── sweep.rs Disc::sweep (Pass 1 forward sweep)
│ ├── patch.rs Disc::patch (Pass N retry over mapfile)
│ ├── mapfile.rs ddrescue-format mapfile
│ └── read_error.rs ReadCtx / ReadAction state machine
├── scsi/ SCSI transport (Linux SG_IO, macOS IOKit, Windows SPTI)
├── platform/ Drive unlock (MT1959 A/B)
├── aacs/ AACS decryption (handshake, keys, keydb, decrypt)
├── css/ DVD CSS cipher
├── decrypt.rs Unified decrypt dispatcher (AACS/CSS/None)
├── pes.rs PES frame types, Stream trait
├── sector.rs SectorReader trait
├── pes.rs PES frame types, FrameSource / FrameSink traits
├── sector/ Sector I/O (was sector.rs in 0.17)
│ ├── mod.rs SectorSource, SectorSink traits
│ ├── file.rs FileSectorSource, FileSectorSink (ISO-backed)
│ └── decrypting.rs DecryptingSectorSource decorator
├── udf.rs UDF 2.50 filesystem parser
├── mpls.rs MPLS playlist parser
├── clpi.rs CLPI clip info parser
@@ -199,15 +201,15 @@ libfreemkv/src/
├── profile.rs Bundled drive profiles
├── speed.rs DriveSpeed enum
├── mux/
│ ├── mod.rs IOStream trait, public exports
│ ├── resolve.rs URL parser + open_input/open_output + input/output
│ ├── mod.rs Public mux exports
│ ├── resolve.rs URL parser + input/output (Box<dyn FrameSource/Sink>)
│ ├── meta.rs FMKV header format
│ ├── disc.rs DiscStream (optical drive → PES)
│ ├── iso.rs IsoStream (ISO image read/write)
│ ├── iso.rs IsoStream (ISO image read)
│ ├── isowriter.rs ISO image writer (UDF, AVDP, multi-extent)
│ ├── mkvstream.rs MkvStream (bidirectional Matroska, IOStream)
│ ├── mkvstream.rs MkvStream (bidirectional Matroska)
│ ├── mkvout.rs MkvOutputStream (PES → MKV)
│ ├── m2ts.rs M2tsStream (BD-TS, IOStream)
│ ├── m2ts.rs M2tsStream (BD-TS)
│ ├── pesout.rs PES output streams (M2ts, Network, Stdio, Null)
│ ├── network.rs NetworkStream (TCP + FMKV header)
│ ├── stdio.rs StdioStream (stdin/stdout pipe)
+6 -2
View File
@@ -65,8 +65,12 @@ libfreemkv (lib.rs)
│
├── Streaming
│ ├── mux/ Stream implementations (Disc, ISO, MKV, M2TS, Network, Stdio, Null)
│ ├── pes PES frame types, Stream trait (read/write frames)
│ └── sector SectorReader trait — abstracts disc vs ISO vs file
│ ├── pes PES frame types; FrameSource / FrameSink direction-typed traits
│ └── sector/ SectorSource / SectorSink traits, FileSector{Source,Sink}, DecryptingSectorSource
│
├── I/O Primitives
│ ├── halt Halt cancellation token (one Arc<AtomicBool>, cloneable)
│ └── io/ Pipeline<I, R> + Sink trait + WritebackFile (bounded-cache writer)
│
├── Support
│ ├── keydb KEYDB.cfg download, parse, verify, save
+8 -4
View File
@@ -97,7 +97,9 @@ drive.probe_disc()?;
// Scan disc (UDF + playlists + AACS — all automatic)
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
// Stream pipeline — PES frames from any source to any output
// Stream pipeline — PES frames from any source to any output.
// 0.18: input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
// direction is type-checked, so calling .write() on an input is a compile error.
let opts = InputOptions::default();
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
let title = input.info().clone();
@@ -121,11 +123,13 @@ output.finish()?;
| aacs/ | [aacs.md](aacs.md) | Key resolution + content decrypt + bus handshake |
| css/ | -- | DVD CSS cipher |
| decrypt.rs | -- | Unified decrypt dispatcher (AACS/CSS/None) |
| disc/ | -- | High-level scan + read API |
| disc/ | [rip-recovery.md](rip-recovery.md) | Disc::scan + Disc::sweep + Disc::patch + mapfile |
| labels/ | -- | BD-J stream labels (5 format parsers) |
| mux/ | -- | Stream implementations (7 stream types) |
| pes.rs | -- | PES frame types + Stream trait |
| sector.rs | -- | SectorReader trait |
| pes.rs | -- | PES frame types + FrameSource / FrameSink traits |
| sector/ | -- | SectorSource / SectorSink + DecryptingSectorSource decorator |
| io/ | -- | Pipeline<I, R> + Sink trait + WritebackFile |
| halt.rs | -- | Halt cancellation token |
| keydb.rs | -- | KEYDB download, parse, save |
| error.rs | -- | Error codes (E1xxx-E8xxx) |
| event.rs | -- | Drive event system |
+2 -2
View File
@@ -52,8 +52,8 @@ selects the per-CDB timeout:
| `recovery` | Timeout | Used by |
|------------|----------|------------------------------------------|
| `false` | 1.5 s | `Disc::copy` fast skip-forward sweep, `DiscStream::fill_extents` |
| `true` | 30 s | `Disc::patch` multi-pass over the mapfile |
| `false` | 1.5 s | `Disc::sweep` fast skip-forward pass, `DiscStream::fill_extents` |
| `true` | 30 s | `Disc::patch` retry pass over the mapfile |
On any SCSI failure or timeout, `read` returns `Err(DiscRead)` immediately.
There are no inline retries, no SCSI reset, no Phase 1/2/3 escalation.
+43 -22
View File
@@ -12,21 +12,24 @@ reach into the others.
| Layer | Where it lives | What it does |
|-------|---------------|--------------|
| 1 — Bad-range retry | `Disc::patch` (multi-pass over the mapfile) | Re-reads non-`+` ranges with the long timeout. Idempotent; call N times. |
| 1 — Bad-range retry | `Disc::patch` (one pass over the mapfile per call) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
| 2 — Single-shot primitive | `Drive::read` in `src/drive/mod.rs` | One CDB, one timeout, one result. No inline retries, no SCSI reset. |
| 3 — In-loop request adaptation | `DiscStream::fill_extents` adaptive batch sizer | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
The caller orchestrates layer 1. Autorip's `rip_disc` loops `copy` then
N × `patch` per the `MAX_RETRIES` config, then hands the ISO off to the
existing mux pipeline. Layer 3 runs inside any consumer of `DiscStream`
(direct PES pipeline, ISO playback, etc.) without caller involvement.
The library exposes flat verbs; the caller drives the multipass loop. Autorip
runs `Disc::sweep` once, then loops `Disc::patch` until either the mapfile is
clean or the configured retry budget is exhausted, then hands the ISO off to
the mux pipeline. The `freemkv` CLI does the same shape with a
terminal-output progress sink. Layer 3 runs inside any consumer of
`DiscStream` (direct PES pipeline, ISO playback, etc.) without caller
involvement.
Three primitives compose the disc-side flow:
| Primitive | What it does |
|---------------------------|-----------------------------------------------------------------------|
| `Disc::copy` | disc → ISO. Writes a sidecar `.mapfile`. Opt-in skip-forward on failure. |
| `Disc::patch` | Re-reads bad ranges from the drive. Idempotent; call N times. |
| `Disc::sweep` | disc → ISO, one forward pass. Writes a sidecar `.mapfile`. Opt-in skip-on-error. |
| `Disc::patch` | Re-reads bad ranges from the drive. One pass per call; caller invokes N times. |
| `DiscStream` (ISO source) | Reads sectors from the ISO, feeds decrypt → demux → codec → mux. |
## Data model
@@ -60,28 +63,46 @@ Status characters match ddrescue:
Position and size are hex byte offsets into the ISO.
### `CopyOptions` and `PatchOptions`
### `SweepOptions` and `PatchOptions`
`Disc::copy()` auto-detects the pass from mapfile state:
The library no longer dispatches between sweep and patch internally — the
caller picks the verb explicitly per pass. The two option structs are flat
and have no overlap:
```rust
CopyOptions {
decrypt: true, // decrypt AACS/CSS sectors
multipass: true, // enable mapfile + skip-on-error + damage-jump
progress: Some(&reporter), // progress callback
halt: Some(flag), // halt flag for graceful stop
SweepOptions {
decrypt: true,
resume: false,
batch_sectors: None,
skip_on_error: true, // damage-jump + zero-fill on read failure
progress: Some(&reporter),
halt: Some(flag),
}
PatchOptions {
decrypt: true,
block_sectors: None,
full_recovery: true,
reverse: true, // walk bad ranges high → low LBA
wedged_threshold: 50,
progress: Some(&reporter),
halt: Some(flag),
}
```
Dispatch logic:
- No mapfile → sweep (fresh Pass 1)
- Mapfile with NonTried (`?`) → sweep with resume
- Mapfile covering full disc, only NonTrimmed/NonScraped/Unreadable → patch
- Mapfile clean → no-op
Caller-orchestrated dispatch (the policy `Disc::copy` used to embed):
- No mapfile → `sweep` (fresh Pass 1).
- Mapfile with `?` ranges → `sweep` with `resume: true`.
- Mapfile covers full disc, only `*` / `/` / `-` ranges → `patch`.
- Mapfile clean → done; no further pass needed.
Each consumer (autorip, `freemkv` CLI) implements the loop in roughly five
lines of `Mapfile::stats()` checks.
## Algorithm
### Pass 1 — fast sweep (`Disc::copy` → `sweep_internal`)
### Pass 1 — fast sweep (`Disc::sweep`)
1. Read one ECC block (32 sectors for UHD, 16 for BD/DVD) at the current LBA.
2. On success: write data to ISO, mark `+`, advance.
@@ -95,7 +116,7 @@ Dispatch logic:
Pass 1 completes when every byte has been visited (either `+` or `*`).
### Pass 2+ — patch (`Disc::copy` → `patch_internal`)
### Pass 2+ — patch (`Disc::patch`)
`Disc::patch` reads the mapfile and iterates every non-`+` range. Default: **reverse** mode
(walks ranges from highest LBA to lowest, within each range from end to start).
@@ -177,4 +198,4 @@ scrape vs. retry with direction reversal) if there's measured benefit.
- [ddrescue manual, Algorithm chapter](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)
- [ddrescue optical media notes](https://www.electric-spoon.com/doc/gddrescue/html/Optical-media.html)
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/mod.rs`](../src/disc/mod.rs) (`Disc::copy`, `Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/sweep.rs`](../src/disc/sweep.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
+1 -1
View File
@@ -126,7 +126,7 @@ USB optical drives have ~500ms round-trip latency per SCSI command. Since `read_
`Disc::scan()` wraps the drive in a `BufferedSectorReader` before reading. On a single-sector read, the buffer prefetches a batch of sectors (sized from the kernel's `max_hw_sectors_kb` for the device) and caches them. Subsequent reads to nearby LBAs return from cache with zero SCSI overhead. After parsing the UDF directory structure, the entire metadata partition is pre-read into the cache, so all ICB lookups during title scanning and encryption resolution are instant.
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorReader` implementation.
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorSource` implementation.
### UDF Filename Encoding
+57
View File
@@ -247,6 +247,7 @@ fn validate_processing_key(
/// Find Verify Media Key Record (type 0x10) in MKB.
fn mkb_find_mk_dv(mkb: &[u8]) -> Option<[u8; 16]> {
let mut pos = 0;
let mut type10_seen: Vec<(usize, usize)> = Vec::new();
while pos + 4 <= mkb.len() {
let rec_type = mkb[pos];
let rec_len = u32::from_be_bytes([0, mkb[pos + 1], mkb[pos + 2], mkb[pos + 3]]) as usize;
@@ -254,14 +255,32 @@ fn mkb_find_mk_dv(mkb: &[u8]) -> Option<[u8; 16]> {
break;
}
if rec_type == 0x10 {
type10_seen.push((pos, rec_len));
}
if rec_type == 0x10 && rec_len >= 20 {
// mk_dv is at offset 4 (after record header)
let mut dv = [0u8; 16];
dv.copy_from_slice(&mkb[pos + 4..pos + 20]);
tracing::warn!(
target: "freemkv::disc",
phase = "mkb_mk_dv_found",
pos,
rec_len,
"mk_dv extracted from MKB"
);
return Some(dv);
}
pos += rec_len;
}
tracing::warn!(
target: "freemkv::disc",
phase = "mkb_mk_dv_not_found",
type10_seen = ?type10_seen,
scanned_bytes = pos,
"no 0x10 record with rec_len>=20 found"
);
None
}
@@ -628,32 +647,70 @@ pub fn resolve_keys(
}
};
tracing::warn!(
target: "freemkv::disc",
phase = "resolve_keys_start",
aacs2,
bus_encryption,
disc_hash = %hash_hex,
mkb_present = mkb_data.is_some(),
"resolve_keys: starting"
);
// Path 1: Look up VUK by disc hash in KEYDB
if let Some(entry) = keydb.find_disc(&hash_hex) {
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path1_hit_entry", "disc hash found in keydb");
if let Some(vuk) = entry.vuk {
return Some(build(vuk, 1));
}
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path1_no_vuk", "disc hash entry has no VUK");
} else {
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path1_miss", "disc hash NOT in keydb");
}
// Path 2: Find entry with matching VID → derive VUK from MK + VID
let mut path2_mk_did_count = 0usize;
for entry in keydb.disc_entries.values() {
if let (Some(mk), Some(did)) = (entry.media_key, entry.disc_id) {
path2_mk_did_count += 1;
if did == *volume_id {
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path2_hit", "MK+VID entry matched volume_id");
return Some(build(derive_vuk(&mk, volume_id), 2));
}
}
}
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path2_miss", mk_did_entries = path2_mk_did_count, "no MK+VID entry matched volume_id");
// Path 3: MKB + processing keys → media key → VUK
if let Some(mkb) = mkb_data {
let mk_dv = mkb_find_mk_dv(mkb);
let subdiff = mkb_find_subdiff_records(mkb);
let cvalues = mkb_find_cvalues(mkb);
tracing::warn!(
target: "freemkv::disc",
phase = "resolve_keys_mkb_records",
mk_dv_found = mk_dv.is_some(),
subdiff_found = subdiff.is_some(),
subdiff_len = subdiff.as_ref().map(|s| s.len()).unwrap_or(0),
cvalues_found = cvalues.is_some(),
cvalues_len = cvalues.as_ref().map(|c| c.len()).unwrap_or(0),
"MKB record scan results"
);
if let Some(mk) = derive_media_key_from_pk(mkb, &keydb.processing_keys) {
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path3_hit", "media key derived from processing key");
return Some(build(derive_vuk(&mk, volume_id), 3));
}
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path3_miss", pk_count = keydb.processing_keys.len(), "PK derivation failed");
// Path 4: MKB + device keys → processing key → media key → VUK
if let Some(mk) = derive_media_key_from_dk(mkb, &keydb.device_keys) {
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path4_hit", "media key derived from device key");
return Some(build(derive_vuk(&mk, volume_id), 4));
}
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_path4_miss", dk_count = keydb.device_keys.len(), "DK derivation failed");
} else {
tracing::warn!(target: "freemkv::disc", phase = "resolve_keys_no_mkb", "no MKB data available; paths 3/4 skipped");
}
None
+158 -1
View File
@@ -20,6 +20,35 @@ pub struct ClipInfo {
pub ep_coarse: Vec<EpCoarse>,
/// Fine EP entries for the primary video stream
pub ep_fine: Vec<EpFine>,
/// Per-stream metadata from the ProgramInfo section (BD spec).
/// Cross-validates the MPLS STN view — see `labels/clpi.rs`.
/// Empty when program_info is missing or malformed.
pub streams: Vec<ClpiStream>,
}
/// One stream descriptor from the CLPI ProgramInfo / stream_coding_info
/// table. Mirrors the same fields the MPLS STN table carries — see
/// `mpls::StreamEntry` for the playlist-side equivalent.
#[derive(Debug, Clone)]
#[allow(dead_code)]
pub struct ClpiStream {
/// PID of the stream in the MPEG-TS (matches MPLS).
pub pid: u16,
/// SCSI/BD coding type byte (0x80 LPCM, 0x83 TrueHD, 0x86 DTS-HD MA,
/// 0x90 PG, etc.). See `labels::mpls_universal::coding_type_to_codec_hint`.
pub coding_type: u8,
/// ISO 639-2 3-char language code. Empty for video streams.
pub language: String,
/// Audio format byte (1=mono, 3=stereo, 6=5.1, 12=7.1).
/// Zero for non-audio streams.
pub audio_format: u8,
/// Audio sample rate (1=48kHz, 4=96kHz, 5=192kHz). Zero for non-audio.
pub audio_rate: u8,
/// Video format byte (1=480i, 4=1080i, 5=720p, 6=1080p, 8=2160p).
/// Zero for non-video.
pub video_format: u8,
/// Video rate (1=23.976, 2=24, 3=25, 4=29.97, 6=50, 7=59.94).
pub video_rate: u8,
}
#[derive(Debug, Clone)]
@@ -128,7 +157,7 @@ pub fn parse(data: &[u8]) -> Result<ClipInfo> {
// Header offsets
let _seq_info_start = u32::from_be_bytes([data[8], data[9], data[10], data[11]]) as usize;
let _prog_info_start = u32::from_be_bytes([data[12], data[13], data[14], data[15]]) as usize;
let prog_info_start = u32::from_be_bytes([data[12], data[13], data[14], data[15]]) as usize;
let cpi_start = u32::from_be_bytes([data[16], data[17], data[18], data[19]]) as usize;
// ClipInfo section at offset 40
@@ -139,6 +168,16 @@ pub fn parse(data: &[u8]) -> Result<ClipInfo> {
0
};
// Parse ProgramInfo (per-stream language + codec). Best-effort:
// malformed program_info doesn't fail the parse, just gives an
// empty streams list. EP map is unaffected — sector-range lookups
// continue to work.
let streams = if prog_info_start > 0 && prog_info_start + 6 < data.len() {
parse_program_info(&data[prog_info_start..])
} else {
Vec::new()
};
// Parse CPI / EP Map
let (ep_coarse, ep_fine) = if cpi_start > 0 && cpi_start + 8 < data.len() {
parse_cpi(&data[cpi_start..])?
@@ -151,9 +190,127 @@ pub fn parse(data: &[u8]) -> Result<ClipInfo> {
source_packet_count,
ep_coarse,
ep_fine,
streams,
})
}
/// Parse the ProgramInfo section: per-stream (pid, coding_type,
/// language, codec sub-fields). Layout per BD spec / libbluray
/// clpi_parse.c:
///
/// ```text
/// ProgramInfo:
/// length: 4 bytes
/// reserved: 1 byte
/// num_programs: 1 byte
/// for each program:
/// spn_program_sequence_start: 4 bytes
/// program_map_pid: 2 bytes
/// num_streams: 1 byte
/// num_groups: 1 byte
/// for each stream:
/// pid: 2 bytes
/// stream_coding_info_length: 1 byte
/// stream_coding_info: (varies by coding_type)
/// coding_type: 1 byte
/// per-type bytes (see match arms below)
/// ```
///
/// Returns `Vec::new()` on any structural mismatch — we don't propagate
/// errors because the EP map is the primary CLPI output, and a corrupt
/// program_info shouldn't break sector-range lookups.
fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
let mut out = Vec::new();
if data.len() < 6 {
return out;
}
// length: 4 bytes (skipped — we trust the section bounds in the
// caller's slice and read the bytes that follow). Reserved 1 byte
// at offset 4. num_programs at offset 5.
let num_programs = data[5] as usize;
let mut pos = 6usize;
for _ in 0..num_programs {
// Program header: 4 (spn) + 2 (pmt_pid) + 1 (num_streams) + 1 (num_groups) = 8 bytes
if pos + 8 > data.len() {
return out;
}
let num_streams = data[pos + 6] as usize;
pos += 8;
for _ in 0..num_streams {
// Stream header: 2 (pid) + 1 (sci_length) + sci bytes
if pos + 3 > data.len() {
return out;
}
let pid = u16::from_be_bytes([data[pos], data[pos + 1]]);
let sci_len = data[pos + 2] as usize;
let sci_end = pos + 3 + sci_len;
if sci_end > data.len() || sci_len < 1 {
return out;
}
let sci = &data[pos + 3..sci_end];
let coding_type = sci[0];
let mut audio_format = 0u8;
let mut audio_rate = 0u8;
let mut video_format = 0u8;
let mut video_rate = 0u8;
let mut language = String::new();
match coding_type {
// Video — MPEG-2 (0x02), H.264 (0x1B), HEVC (0x24)
0x02 | 0x1B | 0x24 => {
if sci.len() >= 2 {
video_format = (sci[1] >> 4) & 0x0F;
video_rate = sci[1] & 0x0F;
}
}
// Primary audio — LPCM(0x80), AC-3(0x81), DTS(0x82),
// TrueHD(0x83), AC-3+(0x84), DTS-HD(0x85), DTS-HD MA(0x86)
0x80..=0x86 => {
if sci.len() >= 2 {
audio_format = (sci[1] >> 4) & 0x0F;
audio_rate = sci[1] & 0x0F;
}
if sci.len() >= 5 {
language = String::from_utf8_lossy(&sci[2..5]).to_string();
}
}
// Secondary audio (0xA1 AC-3+, 0xA2 DTS-HD)
0xA1 | 0xA2 => {
if sci.len() >= 2 {
audio_format = (sci[1] >> 4) & 0x0F;
audio_rate = sci[1] & 0x0F;
}
if sci.len() >= 5 {
language = String::from_utf8_lossy(&sci[2..5]).to_string();
}
}
// PG (0x90), IG (0x91): coding_type + 3-byte language [+ char_code for PG]
0x90 | 0x91 => {
if sci.len() >= 4 {
language = String::from_utf8_lossy(&sci[1..4]).to_string();
}
}
_ => {}
}
out.push(ClpiStream {
pid,
coding_type,
language,
audio_format,
audio_rate,
video_format,
video_rate,
});
pos = sci_end;
}
}
out
}
/// Parse the CPI section containing the EP map.
fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
if data.len() < 8 {
+2 -2
View File
@@ -16,7 +16,7 @@ pub mod lfsr;
pub(crate) mod tables;
use crate::disc::Extent;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
/// CSS decryption state for a DVD title.
#[derive(Debug, Clone)]
@@ -34,7 +34,7 @@ pub struct CssState {
/// 0x80 (start of the encrypted region). This only happens when a new PES
/// packet begins at exactly sector offset 128. We scan up to 50000
/// scrambled sectors sequentially across all extents.
pub fn crack_key(reader: &mut dyn SectorReader, extents: &[Extent]) -> Option<CssState> {
pub fn crack_key(reader: &mut dyn SectorSource, extents: &[Extent]) -> Option<CssState> {
let mut tried = 0u32;
let max_tries = 50_000;
+50 -1
View File
@@ -62,7 +62,23 @@ pub fn decrypt_sectors(
for chunk in buf.chunks_mut(unit_len) {
if chunk.len() == unit_len && aacs::is_unit_encrypted(chunk) {
aacs::decrypt_unit_full(chunk, &uk, rdk);
// `is_unit_encrypted` is a byte-0 heuristic: it fires on any
// unit whose first byte has the top 2 bits set, which is
// correct for m2ts source packets (where those bits are the
// copy-control marker) but false-positives on any other binary
// data with similarly-shaped first bytes — notably MPLS/CLPI
// navigation files that begin with ASCII magic ('M', 'H'…)
// and survive sweep mixed in with encrypted m2ts payloads.
// `decrypt_unit_full` self-checks via TS-sync verification and
// returns false on a misfire, but it has already mutated the
// chunk by then. Snapshot and restore on verification failure
// — same pattern `decrypt_unit_try_keys` uses for multi-key
// discs. Real m2ts units verify and stay decrypted; nav-file
// sectors get scrambled briefly and then put back as-was.
let original: Vec<u8> = chunk.to_vec();
if !aacs::decrypt_unit_full(chunk, &uk, rdk) {
chunk.copy_from_slice(&original);
}
}
}
}
@@ -74,3 +90,36 @@ pub fn decrypt_sectors(
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// Regression for the 0.18.1 nav-file scramble bug. A non-m2ts unit whose
/// first byte has the top 2 bits set (here: the ASCII letter 'M' that
/// MPLS files start with, 0x4D = 0b01001101) trips `is_unit_encrypted`,
/// gets AES-decrypted with the unit key, fails the TS-sync verification,
/// and must be restored to its original bytes — not left scrambled.
#[test]
fn nav_file_unit_survives_decrypt_attempt() {
let mut unit = vec![0u8; aacs::ALIGNED_UNIT_LEN];
unit[0] = b'M';
unit[1] = b'P';
unit[2] = b'L';
unit[3] = b'S';
for (i, b) in unit.iter_mut().enumerate().skip(4) {
*b = (i as u8).wrapping_mul(31);
}
let snapshot = unit.clone();
let keys = DecryptKeys::Aacs {
unit_keys: vec![(0, [0xAB; 16])],
read_data_key: None,
};
decrypt_sectors(&mut unit, &keys, 0).unwrap();
assert_eq!(
unit, snapshot,
"non-m2ts unit must be restored after failed decrypt"
);
}
}
+4 -4
View File
@@ -3,13 +3,13 @@
use super::*;
use crate::clpi;
use crate::mpls;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf;
impl Disc {
/// Scan Blu-ray titles from MPLS playlists.
pub(super) fn scan_bluray_titles(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
) -> Vec<DiscTitle> {
let mut titles = Vec::new();
@@ -31,7 +31,7 @@ impl Disc {
}
pub(super) fn parse_playlist(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
filename: &str,
data: &[u8],
@@ -199,7 +199,7 @@ impl Disc {
/// Prefers English, falls back to first available language.
/// Returns None if META directory is empty or XML has no usable title.
pub(super) fn read_meta_title(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
) -> Option<String> {
let meta_dir = udf_fs.find_dir("/BDMV/META")?;
+2 -2
View File
@@ -2,13 +2,13 @@
use super::*;
use crate::ifo;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf;
impl Disc {
/// Scan DVD titles from IFO files (VIDEO_TS.IFO + VTS_XX_0.IFO).
pub(super) fn scan_dvd_titles(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
) -> Vec<DiscTitle> {
let dvd_info = match ifo::parse_vmg(reader, udf_fs) {
+104 -8
View File
@@ -2,7 +2,7 @@
use super::*;
use crate::error::{Error, Result};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf;
/// Result of SCSI AACS handshake (ECDH authentication).
@@ -22,42 +22,103 @@ impl Disc {
) -> Option<HandshakeResult> {
use crate::aacs::{self, KeyDb};
let keydb_path = opts.resolve_keydb()?;
let keydb = KeyDb::load(&keydb_path).ok()?;
tracing::warn!(
target: "freemkv::disc",
phase = "handshake_entry",
"do_handshake entered"
);
let keydb_path = match opts.resolve_keydb() {
Some(p) => p,
None => {
tracing::warn!(
target: "freemkv::disc",
phase = "handshake_no_keydb",
"no KEYDB found in search paths; handshake skipped"
);
return None;
}
};
let keydb = match KeyDb::load(&keydb_path) {
Ok(db) => db,
Err(e) => {
tracing::warn!(
target: "freemkv::disc",
phase = "handshake_keydb_load_failed",
io_error_kind = ?e.kind(),
keydb = %keydb_path.display(),
"KEYDB load failed; handshake skipped"
);
return None;
}
};
let host_cert_count = keydb.host_certs.len();
tracing::warn!(
target: "freemkv::disc",
phase = "handshake_start",
host_cert_count,
keydb = %keydb_path.display(),
"handshake starting"
);
const MAX_CERT_ATTEMPTS: usize = 16;
for hc in keydb.host_certs.iter().take(MAX_CERT_ATTEMPTS) {
let mut last_err_code: Option<u16> = None;
for (idx, hc) in keydb.host_certs.iter().take(MAX_CERT_ATTEMPTS).enumerate() {
match aacs::handshake::aacs_authenticate(session, &hc.private_key, &hc.certificate) {
Ok(mut auth) => {
let volume_id = match aacs::handshake::read_volume_id(session, &mut auth) {
Ok(vid) => vid,
Err(_) => return None, // handshake succeeded but can't read VID
Err(e) => {
tracing::warn!(
target: "freemkv::disc",
phase = "handshake_vid_read_failed",
cert_index = idx,
error_code = e.code(),
"auth ok but volume ID read failed"
);
return None;
}
};
let read_data_key = aacs::handshake::read_data_keys(session, &mut auth)
.ok()
.map(|(rdk, _)| rdk);
tracing::debug!(
target: "freemkv::disc",
phase = "handshake_ok",
cert_index = idx,
has_read_data_key = read_data_key.is_some(),
);
return Some(HandshakeResult {
volume_id,
read_data_key,
});
}
Err(_) => {
Err(e) => {
last_err_code = Some(e.code());
continue;
}
}
}
tracing::warn!(
target: "freemkv::disc",
phase = "handshake_all_certs_failed",
host_cert_count,
tried = host_cert_count.min(MAX_CERT_ATTEMPTS),
last_error_code = last_err_code,
"all host certs in KEYDB rejected by drive"
);
// All host certs failed — return None, not a fake success
None
}
/// Resolve disc encryption — AACS 1.0, AACS 2.0, CSS, or none.
///
/// Reads AACS files from UDF (via SectorReader), resolves keys through
/// Reads AACS files from UDF (via SectorSource), resolves keys through
/// whatever path works: KEYDB VUK lookup, media key derivation, processing
/// keys, device keys. Uses handshake result (volume ID, bus key) if available.
pub(super) fn resolve_encryption(
udf_fs: &udf::UdfFs,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
keydb_path: &std::path::Path,
handshake: Option<&HandshakeResult>,
) -> Result<AacsState> {
@@ -73,6 +134,19 @@ impl Disc {
.or_else(|_| udf_fs.read_file(reader, "/AACS/DUPLICATE/Unit_Key_RO.inf"))
.map_err(|_| Error::AacsNoKeys)?;
// Log the disc hash so we can confirm whether it's present in KEYDB
// when key resolution fails. The disc hash is SHA-1 of the full
// Unit_Key_RO.inf file bytes — same value KEYDB.cfg keys VUK entries by.
let dh = crate::aacs::disc_hash(&uk_ro_data);
let dh_hex = crate::aacs::disc_hash_hex(&dh);
tracing::warn!(
target: "freemkv::disc",
phase = "scan_aacs_disc_hash",
disc_hash = %dh_hex,
uk_ro_len = uk_ro_data.len(),
"disc hash computed (compare with keydb.cfg entries)"
);
let cc_data = udf_fs
.read_file(reader, "/AACS/Content000.cer")
.or_else(|_| udf_fs.read_file(reader, "/AACS/Content001.cer"))
@@ -84,6 +158,28 @@ impl Disc {
.ok();
let mkb_ver = mkb_data.as_deref().and_then(aacs::mkb_version);
let mkb_first_64_hex = mkb_data
.as_deref()
.map(|m| {
m.iter()
.take(64)
.map(|b| format!("{b:02x}"))
.collect::<String>()
})
.unwrap_or_default();
tracing::warn!(
target: "freemkv::disc",
phase = "scan_aacs_mkb_info",
mkb_present = mkb_data.is_some(),
mkb_len = mkb_data.as_deref().map(|m| m.len()).unwrap_or(0),
mkb_version = ?mkb_ver,
mkb_first_64 = %mkb_first_64_hex,
keydb_disc_count = keydb.disc_entries.len(),
keydb_dk_count = keydb.device_keys.len(),
keydb_pk_count = keydb.processing_keys.len(),
"AACS resolution inputs"
);
// Use handshake volume ID if available, otherwise zeros
// (KEYDB VUK lookup by disc hash works without volume ID)
let volume_id = handshake.map(|h| h.volume_id).unwrap_or([0u8; 16]);
+148 -8
View File
@@ -17,11 +17,22 @@
//!
//! Status chars: `?` non-tried · `*` non-trimmed · `/` non-scraped · `-` unreadable · `+` finished.
//!
//! The mapfile is flushed to disk on every `record()` call so a crashed
//! rip loses at most one block of recorded state.
//! The mapfile is flushed to disk at most once per `FLUSH_INTERVAL`
//! during `record()` calls, plus on explicit `flush()` and on `Drop`.
//! This bounds atomic-rename RPC rate on networked staging (e.g. NFS)
//! where per-record persists otherwise serialize the rip pipeline.
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
/// Minimum interval between mapfile persists. `record()` updates in-memory
/// state every call but only writes to disk when this interval has elapsed
/// since the last persist (or when `flush()` is called explicitly, or on
/// `Drop`). Bounds RPC rate on NFS staging where atomic-rename per record
/// otherwise dominates throughput. On crash the worst-case progress loss
/// is one interval's worth of records.
const FLUSH_INTERVAL: Duration = Duration::from_millis(1000);
/// Status of a byte range in the mapfile. ddrescue-compatible.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -91,15 +102,34 @@ pub struct MapStats {
/// retry" UI bucket; `bytes_pending` over-counts because it folds
/// in `bytes_nontried`.
pub bytes_retryable: u64,
/// Number of unreadable ranges (for UI display). Computed from
/// `ranges_with(&[Unreadable])`.
pub num_bad_ranges: u32,
/// Largest gap among unreadable ranges in milliseconds. Computed as
/// largest range size / bytes_per_sec * 1000. Set by caller (autorip)
/// since bytes_per_sec is application-specific.
pub main_lost_ms: f64,
}
/// Write-through mapfile. Every `record()` persists to disk immediately
/// so a crash during rip loses at most one block.
/// Time-batched mapfile. `record()` keeps in-memory state up-to-date on
/// every call; persists to disk at most once per `FLUSH_INTERVAL`.
/// Explicit `flush()` and `Drop` guarantee state is on disk after a sweep
/// or patch finishes. On hard crash the worst-case loss is one flush
/// interval of records — the file's payload bytes are unaffected.
pub struct Mapfile {
path: PathBuf,
entries: Vec<MapEntry>,
total_size: u64,
version: String,
/// Incrementally maintained stats — updated on every `record()` call
/// so `stats()` is O(1) instead of O(n).
stats: MapStats,
/// True when in-memory state has changed but `write_to_disk` has not
/// yet captured it.
dirty: bool,
/// Wall-clock timestamp of the last successful `write_to_disk` (or
/// the moment the mapfile was constructed, whichever is later).
last_flushed: Instant,
}
impl Mapfile {
@@ -107,7 +137,7 @@ impl Mapfile {
/// Writes to disk immediately so a resume can pick up even if the caller
/// never records anything.
pub fn create(path: &Path, total_size: u64, version: &str) -> io::Result<Self> {
let mf = Self {
let mut mf = Self {
path: path.to_path_buf(),
entries: vec![MapEntry {
pos: 0,
@@ -116,8 +146,19 @@ impl Mapfile {
}],
total_size,
version: version.to_string(),
stats: MapStats {
bytes_total: total_size,
bytes_pending: total_size,
bytes_nontried: total_size,
..Default::default()
},
dirty: false,
last_flushed: Instant::now(),
};
// Eager initial persist so a resume can pick this up even if
// `record()` is never called.
mf.write_to_disk()?;
mf.last_flushed = Instant::now();
Ok(mf)
}
@@ -177,11 +218,15 @@ impl Mapfile {
}
entries.sort_by_key(|e| e.pos);
let total_size = entries.last().map(|e| e.pos + e.size).unwrap_or(0);
let stats = Self::compute_stats(&entries, total_size);
Ok(Self {
path: path.to_path_buf(),
entries,
total_size,
version,
stats,
dirty: false,
last_flushed: Instant::now(),
})
}
@@ -244,8 +289,30 @@ impl Mapfile {
merged.push(e);
}
// Recompute stats from merged entries. record() is already O(n) due to
// drain-and-rebuild, so this is a constant-factor overhead. The critical
// win is that stats() is now O(1) — called millions of times in the hot
// path during sweep/patch, it just returns the cached value.
self.stats = Self::compute_stats(&merged, self.total_size);
self.entries = merged;
self.write_to_disk()?;
self.dirty = true;
if self.last_flushed.elapsed() >= FLUSH_INTERVAL {
self.write_to_disk()?;
self.dirty = false;
self.last_flushed = Instant::now();
}
Ok(())
}
/// Persist any pending in-memory changes to disk. No-op if clean.
/// Callers (sweep/patch finalisation) invoke this after their last
/// `record()` to guarantee state is durable before returning.
pub fn flush(&mut self) -> io::Result<()> {
if self.dirty {
self.write_to_disk()?;
self.dirty = false;
self.last_flushed = Instant::now();
}
Ok(())
}
@@ -283,11 +350,15 @@ impl Mapfile {
}
pub fn stats(&self) -> MapStats {
self.stats
}
fn compute_stats(entries: &[MapEntry], total_size: u64) -> MapStats {
let mut s = MapStats {
bytes_total: self.total_size,
bytes_total: total_size,
..Default::default()
};
for e in &self.entries {
for e in entries {
match e.status {
SectorStatus::Finished => s.bytes_good += e.size,
SectorStatus::Unreadable => s.bytes_unreadable += e.size,
@@ -336,6 +407,16 @@ impl Mapfile {
}
}
impl Drop for Mapfile {
/// Best-effort flush on drop so a sweep / patch that returns early
/// (or unwinds) doesn't lose its in-memory state. Errors here are
/// swallowed because Drop has no way to surface them; explicit
/// `flush()` on the success path gives callers proper error handling.
fn drop(&mut self) {
let _ = self.flush();
}
}
fn parse_hex(s: &str) -> io::Result<u64> {
let s = s.strip_prefix("0x").unwrap_or(s);
u64::from_str_radix(s, 16).map_err(|_| {
@@ -436,6 +517,8 @@ mod tests {
let mut mf = Mapfile::create(&p, 1000, "test").unwrap();
mf.record(100, 200, SectorStatus::Finished).unwrap();
mf.record(500, 100, SectorStatus::Unreadable).unwrap();
// record() batches; explicit flush before reading back from disk.
mf.flush().unwrap();
let loaded = Mapfile::load(&p).unwrap();
assert_eq!(loaded.entries(), mf.entries());
let _ = std::fs::remove_file(&p);
@@ -467,4 +550,61 @@ mod tests {
assert_eq!(bad, vec![(100, 50), (300, 50)]);
let _ = std::fs::remove_file(&p);
}
#[test]
fn stats_consistent_after_overlapping_records() {
let p = tmpfile("stats_consistent_after_overlapping");
let _ = std::fs::remove_file(&p);
let mut mf = Mapfile::create(&p, 1000, "test").unwrap();
// Record some finished, some unreadable, some nontrimmed
mf.record(0, 300, SectorStatus::Finished).unwrap();
mf.record(300, 200, SectorStatus::NonTrimmed).unwrap();
mf.record(500, 100, SectorStatus::Unreadable).unwrap();
mf.record(600, 400, SectorStatus::Finished).unwrap();
// Final entries: [0..300 Finished, 300..500 NonTrimmed, 500..600 Unreadable, 600..1000 Finished]
let s = mf.stats();
assert_eq!(s.bytes_good, 700); // 300 + 400
assert_eq!(s.bytes_unreadable, 100); // 100
assert_eq!(s.bytes_pending, 200); // NonTrimmed only (NonTried=0)
assert_eq!(s.bytes_nontried, 0);
assert_eq!(s.bytes_retryable, 200); // NonTrimmed
assert_eq!(s.bytes_total, 1000);
// Overwrite a NonTrimmed range with Finished
mf.record(300, 100, SectorStatus::Finished).unwrap();
// Entries: [0..400 Finished, 400..500 NonTrimmed, 500..600 Unreadable, 600..1000 Finished]
let s2 = mf.stats();
assert_eq!(s2.bytes_good, 800); // 400 + 400
assert_eq!(s2.bytes_unreadable, 100);
assert_eq!(s2.bytes_pending, 100); // NonTrimmed only
assert_eq!(s2.bytes_retryable, 100);
let _ = std::fs::remove_file(&p);
}
#[test]
fn stats_consistent_after_split_record() {
let p = tmpfile("stats_consistent_after_split");
let _ = std::fs::remove_file(&p);
let mut mf = Mapfile::create(&p, 1000, "test").unwrap();
// Mark middle as NonTrimmed
mf.record(200, 400, SectorStatus::NonTrimmed).unwrap();
// Entries: [0..200 NonTried, 200..600 NonTrimmed, 600..1000 NonTried]
let s = mf.stats();
assert_eq!(s.bytes_pending, 1000); // NonTried(600) + NonTrimmed(400)
assert_eq!(s.bytes_retryable, 400); // NonTrimmed only
assert_eq!(s.bytes_nontried, 600); // 200 + 400
// Overwrite the NonTrimmed with Finished (splitting the remaining NonTried)
mf.record(200, 400, SectorStatus::Finished).unwrap();
// Entries: [0..200 NonTried, 200..600 Finished, 600..1000 NonTried]
let s2 = mf.stats();
assert_eq!(s2.bytes_good, 400);
assert_eq!(s2.bytes_pending, 600); // NonTried(200 + 400)
assert_eq!(s2.bytes_nontried, 600);
assert_eq!(s2.bytes_retryable, 0);
let _ = std::fs::remove_file(&p);
}
}
+759 -737
View File
File diff suppressed because it is too large Load Diff
+1460
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+239
View File
@@ -0,0 +1,239 @@
//! `Disc::sweep`'s consumer-side `Sink<WorkItem>`.
//!
//! Background: the original sweep loop runs strictly serialised —
//! SCSI read → decrypt → seek + write → mapfile.record → next iter.
//! On a healthy disc the SCSI read costs ~5-12 ms per 64 KB batch and
//! the post-read work (decrypt 1-3 ms + file write + mapfile fsync
//! 5-15 ms) adds another batch's worth of latency. The drive idles
//! during the post-read work; throughput tops out at the *sum* of
//! both costs.
//!
//! 0.17.11 introduced a bespoke producer/consumer split (the now-
//! removed `disc/sweep_pipeline.rs`) to overlap the two stages. 0.18
//! collapses that split — together with the analogous splits patch
//! and mux need — onto the generic [`crate::io::Pipeline`] +
//! [`crate::io::Sink`] primitive. This module is the sweep-specific
//! `Sink` impl; the producer-side state machine (read_error context,
//! decrypt, set_speed, halt) stays in `Disc::sweep` in `disc/mod.rs`.
//!
//! Correctness invariants preserved (same as 0.17.11):
//! - Mapfile is single-writer (consumer-only). No locking.
//! - All `read_error::ReadCtx` state stays on the producer thread.
//! - `set_speed` calls happen on the producer thread (same thread that
//! owns the `SectorSource`). No new SCSI concurrency.
//! - Per-iteration ordering of file-write → mapfile-record is kept
//! intact in the consumer (write before record), so the on-disk
//! invariant "mapfile only marks Finished what the file has
//! received" survives a crash mid-pass.
//! - The BU40N+Initio bridge wedge concern is unchanged: only one
//! SCSI command in flight at a time, error-path timing identical,
//! no new retry logic.
use std::io::{Seek, SeekFrom, Write};
use std::sync::mpsc::{Receiver, SyncSender, sync_channel};
use crate::error::Error;
use crate::io::{Flow, Sink};
use super::mapfile::{MapStats, Mapfile, SectorStatus};
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. 64 KB
/// matches the existing zero_gap chunk size used by the pre-split
/// sweep loop.
const ZERO_CHUNK: usize = 65 * 1024;
/// Producer → Consumer messages. The consumer applies these in FIFO
/// order; ordering of file writes and mapfile records across items is
/// preserved.
pub(super) enum WorkItem {
/// Successful batch read. Producer has already decrypted `buf` if
/// `opts.decrypt` was set. Consumer writes `buf` at `pos` and
/// records the range as `Finished`.
Good { pos: u64, buf: Vec<u8> },
/// Bisect inner-loop good single sector (already decrypted by the
/// producer). 2048 bytes.
BisectGood { pos: u64, buf: Box<[u8; 2048]> },
/// Bisect inner-loop bad single sector. Consumer writes 2048
/// zeros at `pos` and records the sector as `NonTrimmed`.
BisectBad { pos: u64 },
/// Whole-batch zero-fill (failed batch on `SkipBlock`, or the
/// failed batch portion of `JumpAhead`). Consumer streams zeros
/// across `[pos, pos+len)` and records the range as `NonTrimmed`.
SkipFill { pos: u64, len: u64 },
/// Gap fill following a `JumpAhead`. Same effect as `SkipFill`;
/// distinguished only so future logging / instrumentation can
/// tell them apart without parsing a flag.
GapFill { pos: u64, len: u64 },
/// Producer wants the latest mapfile stats for the progress
/// callback. Consumer responds on `prog_tx` with a fresh
/// [`ProgressSnapshot`]. Best-effort: if the producer hasn't
/// drained the previous snapshot, the new one is silently
/// dropped — the producer's local cache stays current enough.
StatsRequest,
}
/// Snapshot the consumer sends back to the producer for the progress
/// callback.
pub(super) struct ProgressSnapshot {
pub stats: MapStats,
pub bad_ranges: Vec<(u64, u64)>,
}
/// Final summary returned by the consumer thread on shutdown — what
/// `SweepSink::close` produces, surfaced to the producer via
/// `Pipeline::finish`.
pub(super) struct ConsumerSummary {
pub stats: MapStats,
}
/// Drain any pending progress snapshots from the consumer. Returns
/// the most recent one, if any. The producer caches it and uses it
/// for subsequent progress callbacks until a fresh one arrives.
pub(super) fn try_recv_progress(rx: &Receiver<ProgressSnapshot>) -> Option<ProgressSnapshot> {
let mut latest = None;
while let Ok(snap) = rx.try_recv() {
latest = Some(snap);
}
latest
}
/// `Sink<WorkItem>` for sweep. Owns the writeback file + mapfile +
/// progress back-channel. `apply` carries the file-write +
/// mapfile.record per item; `close` drains the writeback pipeline,
/// fsyncs the ISO, and flushes the mapfile.
pub(super) struct SweepSink {
file: crate::io::WritebackFile,
map: Mapfile,
/// `sync_all`-on-failure-is-an-error iff the output is a regular
/// file. `/dev/null` and pipes always fail `sync_all`; that's not
/// a real error.
is_regular: bool,
/// Back-channel for `StatsRequest` responses. The producer caches
/// the latest snapshot and uses it for the progress callback;
/// dropped sends on a full channel are by design.
prog_tx: SyncSender<ProgressSnapshot>,
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. Held
/// in the sink so each apply call doesn't reallocate.
zero: Box<[u8; ZERO_CHUNK]>,
}
impl SweepSink {
/// Construct a new `SweepSink` plus the matching progress
/// receiver. Channel depth on the back-channel is `1` — the
/// producer's cache is the source of truth between snapshots.
pub(super) fn new(
file: crate::io::WritebackFile,
map: Mapfile,
is_regular: bool,
) -> (Self, Receiver<ProgressSnapshot>) {
let (prog_tx, prog_rx) = sync_channel::<ProgressSnapshot>(1);
let sink = SweepSink {
file,
map,
is_regular,
prog_tx,
zero: Box::new([0u8; ZERO_CHUNK]),
};
(sink, prog_rx)
}
}
impl Sink<WorkItem> for SweepSink {
type Output = ConsumerSummary;
fn apply(&mut self, item: WorkItem) -> Result<Flow, Error> {
match item {
WorkItem::Good { pos, buf } => {
// Decrypt is on the producer; consumer assumes plaintext.
let len = buf.len() as u64;
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf)
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, len, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::BisectGood { pos, buf } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf[..])
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, 2048, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::BisectBad { pos } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&self.zero[..2048])
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, 2048, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::SkipFill { pos, len } | WorkItem::GapFill { pos, len } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
// Subsequent writes are sequential; `WritebackFile`'s
// seek-elision keeps them on the writeback pipeline path.
let mut filled = 0u64;
while filled < len {
let chunk = (len - filled).min(self.zero.len() as u64) as usize;
self.file
.write_all(&self.zero[..chunk])
.map_err(|e| Error::IoError { source: e })?;
filled += chunk as u64;
}
self.map
.record(pos, len, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::StatsRequest => {
let stats = self.map.stats();
let bad_ranges = self.map.ranges_with(&[
SectorStatus::NonTrimmed,
SectorStatus::Unreadable,
SectorStatus::NonScraped,
SectorStatus::NonTried,
]);
// Best-effort: drop on backpressure; producer's cache
// stays current enough.
let _ = self
.prog_tx
.try_send(ProgressSnapshot { stats, bad_ranges });
}
}
Ok(Flow::Continue)
}
fn close(mut self) -> Result<Self::Output, Error> {
// Drain the writeback pipeline + fsync the ISO, then persist
// any pending mapfile state. Same finalisation order as the
// pre-Pipeline consumer loop.
if let Err(e) = self.file.sync_all() {
if self.is_regular {
return Err(Error::IoError { source: e });
}
// Non-regular outputs (/dev/null, pipes) always fail
// sync_all; that's not a real error.
}
self.map.flush().map_err(|e| Error::IoError { source: e })?;
Ok(ConsumerSummary {
stats: self.map.stats(),
})
}
}
+120 -2
View File
@@ -29,7 +29,7 @@ use crate::platform::PlatformDriver;
use crate::platform::mt1959::Mt1959;
use crate::profile::{self, DriveProfile};
use crate::scsi::ScsiTransport;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use std::path::Path;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
@@ -69,6 +69,16 @@ pub struct Drive {
halt: Arc<AtomicBool>,
/// Event handler — fires for read errors and library-level state changes.
event_fn: Option<Box<dyn Fn(Event) + Send>>,
/// Linux only: raw fd for the corresponding block device (`/dev/sr*`)
/// used as a recovery fallback when SCSI READ via `/dev/sg*` returns
/// an error. The kernel `sr_mod` driver auto-retries failed reads
/// (~5× per command) — historically the reason `dd if=/dev/sr0`
/// recovers ~50% of bad sectors that single-shot `SG_IO` READ
/// misses on the same drive. `None` when the block device couldn't
/// be resolved or opened (no fallback in that case; SCSI read
/// errors propagate as before).
#[cfg(target_os = "linux")]
block_dev_fd: Option<std::os::unix::io::RawFd>,
}
impl Drive {
@@ -87,6 +97,9 @@ impl Drive {
None => (None, None, None),
};
#[cfg(target_os = "linux")]
let block_dev_fd = open_block_device_for_sg(device);
Ok(Drive {
scsi: transport,
driver,
@@ -96,6 +109,8 @@ impl Drive {
device_path: device.to_string_lossy().to_string(),
halt: Arc::new(AtomicBool::new(false)),
event_fn: None,
#[cfg(target_os = "linux")]
block_dev_fd,
})
}
@@ -483,6 +498,54 @@ impl Drive {
scsi_status = status,
"Drive::read checked_exec failed"
);
// /dev/sr0 pread fallback (Linux only). The kernel
// sr_mod driver auto-retries failed reads (~5× per
// command). Empirically (BU40N + Dune Part 2 UHD,
// 2026-05-08) dd via /dev/sr0 recovers ~50% of bad
// sectors that a single-shot SG_IO READ misses.
#[cfg(target_os = "linux")]
if recovery {
if let Some(fd) = self.block_dev_fd {
let len = count as usize * 2048;
if buf.len() >= len {
let offset = lba as i64 * 2048;
// Drop kernel cache for this region so we get
// a fresh device read, not stale page-cache
// data from a prior successful neighbour read.
let _ = unsafe {
libc::posix_fadvise(
fd,
offset,
len as i64,
libc::POSIX_FADV_DONTNEED,
)
};
let n = unsafe {
libc::pread(fd, buf.as_mut_ptr() as *mut libc::c_void, len, offset)
};
if n == len as isize {
tracing::info!(
target: "freemkv::drive",
lba,
count,
bytes = len,
"Drive::read recovered via /dev/sr0 pread fallback"
);
return Ok(len);
}
tracing::debug!(
target: "freemkv::drive",
lba,
count,
pread_ret = n as i64,
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
"/dev/sr0 pread fallback also failed"
);
}
}
}
Err(Error::DiscRead {
sector: lba as u64,
status: Some(status),
@@ -586,10 +649,65 @@ impl Drop for Drive {
fn drop(&mut self) {
self.cleanup();
// SgIoTransport::drop() runs next, calling libc::close(fd)
#[cfg(target_os = "linux")]
if let Some(fd) = self.block_dev_fd.take() {
unsafe { libc::close(fd) };
}
}
}
impl SectorReader for Drive {
/// Resolve a `/dev/sg*` path to the corresponding `/dev/sr*` block
/// device by walking sysfs, then open it for read (no `O_DIRECT` —
/// `posix_fadvise(POSIX_FADV_DONTNEED)` flushes the cache before each
/// pread, which avoids buffer-alignment requirements while still
/// forcing fresh device reads).
///
/// Returns `None` on any error (sysfs not present, no matching block
/// device, open failed). Callers treat that as "no fallback available"
/// and propagate the original SCSI READ error.
#[cfg(target_os = "linux")]
fn open_block_device_for_sg(sg_path: &Path) -> Option<std::os::unix::io::RawFd> {
let basename = sg_path.file_name()?.to_str()?;
if !basename.starts_with("sg") {
return None;
}
let sysfs_dir = format!("/sys/class/scsi_generic/{}/device/block", basename);
let entries = std::fs::read_dir(&sysfs_dir).ok()?;
let block_name = entries
.flatten()
.find_map(|e| e.file_name().into_string().ok())?;
let block_path = format!("/dev/{}", block_name);
let mut bytes = block_path.as_bytes().to_vec();
bytes.push(0);
let fd = unsafe {
libc::open(
bytes.as_ptr() as *const libc::c_char,
libc::O_RDONLY | libc::O_CLOEXEC,
)
};
if fd < 0 {
tracing::debug!(
target: "freemkv::drive",
sg = basename,
block_path,
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
"Failed to open block device for fallback; sr0 fallback disabled"
);
None
} else {
tracing::info!(
target: "freemkv::drive",
sg = basename,
block_path,
fd,
"Opened /dev/sr* as recovery fallback for failed SCSI reads"
);
Some(fd)
}
}
impl SectorSource for Drive {
fn read_sectors(
&mut self,
lba: u32,
+166
View File
@@ -0,0 +1,166 @@
//! One-bit cooperative cancellation flag.
//!
//! `Halt` is a clonable token wrapping `Arc<AtomicBool>`. Pass clones into
//! every long-running loop; the loop polls `is_cancelled()` and bails out
//! cleanly. Calling `cancel()` from any clone flips the shared flag, and
//! every other clone observes it on its next poll.
//!
//! Why: `Ordering::Relaxed` is sufficient on both load and store because
//! this flag is purely advisory — no other memory operations piggyback on
//! it for happens-before ordering. Callers that need to publish data
//! across threads do so via channels or other synchronization, not via
//! this bit.
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
/// Clonable, infallible cooperative-cancellation token.
///
/// Clones share the same underlying flag. `cancel()` is one-way; there is
/// no `reset()` by design — construct a fresh `Halt` for a fresh
/// operation.
///
/// Construct with [`Halt::new`] (or [`Halt::default`]). The `Default`
/// impl forwards to `new()` — both produce a fresh, uncancelled token.
/// The pair exists because clippy's `new_without_default` lint requires
/// `Default` whenever a public `new()` is present, even when the two
/// would do exactly the same thing.
#[derive(Clone, Debug)]
pub struct Halt(Arc<AtomicBool>);
impl Halt {
/// Construct a fresh, uncancelled token.
pub fn new() -> Self {
Self(Arc::new(AtomicBool::new(false)))
}
/// Wrap an existing `Arc<AtomicBool>` as a `Halt`. Useful as a
/// bridge during the 0.18 deprecation window: callers that already
/// hold an `Arc<AtomicBool>` (e.g. `Drive::halt_flag()`, the
/// deprecated `DiscStream::set_halt`) can adopt the new token API
/// without changing the underlying flag.
///
/// Cancelling either side flips the same bit — the wrapping `Halt`
/// and the original `Arc` are two views over one shared flag.
pub fn from_arc(flag: Arc<AtomicBool>) -> Self {
Self(flag)
}
/// Borrow the underlying `Arc<AtomicBool>`. Used at boundaries with
/// pre-`Halt` APIs that still take an `Arc<AtomicBool>` directly
/// (`CopyOptions::halt`, the deprecated `DiscStream::set_halt`).
/// Round 3 deletes those boundaries and this accessor with them.
pub fn as_arc(&self) -> &Arc<AtomicBool> {
&self.0
}
/// Flip the shared flag to cancelled. Idempotent.
pub fn cancel(&self) {
self.0.store(true, Ordering::Relaxed);
}
/// Read the shared flag.
pub fn is_cancelled(&self) -> bool {
self.0.load(Ordering::Relaxed)
}
}
impl Default for Halt {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn fresh_is_not_cancelled() {
let h = Halt::new();
assert!(!h.is_cancelled());
}
#[test]
fn cancel_flips_state() {
let h = Halt::new();
assert!(!h.is_cancelled());
h.cancel();
assert!(h.is_cancelled());
}
#[test]
fn cancel_is_idempotent() {
let h = Halt::new();
h.cancel();
h.cancel();
assert!(h.is_cancelled());
}
#[test]
fn clone_shares_state() {
let original = Halt::new();
let cloned = original.clone();
assert!(!original.is_cancelled());
assert!(!cloned.is_cancelled());
// Cancel via the clone; the original observes it.
cloned.cancel();
assert!(original.is_cancelled());
assert!(cloned.is_cancelled());
}
#[test]
fn clone_shares_state_reverse_direction() {
let original = Halt::new();
let cloned = original.clone();
// Cancel via the original; the clone observes it.
original.cancel();
assert!(cloned.is_cancelled());
}
#[test]
fn clone_shares_state_across_threads() {
let h = Halt::new();
let h2 = h.clone();
let handle = std::thread::spawn(move || {
h2.cancel();
});
handle.join().unwrap();
assert!(h.is_cancelled());
}
#[test]
fn from_arc_shares_state() {
// The 0.18 deprecation-window bridge: a Halt built from an
// existing Arc<AtomicBool> must be a *view* over the same bit,
// not a fresh copy. Cancelling either side flips both.
let arc = Arc::new(AtomicBool::new(false));
let halt = Halt::from_arc(arc.clone());
assert!(!halt.is_cancelled());
assert!(!arc.load(Ordering::Relaxed));
// Cancel via the wrapping Halt; the original Arc observes it.
halt.cancel();
assert!(arc.load(Ordering::Relaxed));
// Conversely: flip the Arc directly; the Halt view observes it.
let arc2 = Arc::new(AtomicBool::new(false));
let halt2 = Halt::from_arc(arc2.clone());
arc2.store(true, Ordering::Relaxed);
assert!(halt2.is_cancelled());
}
#[test]
fn as_arc_returns_backing_flag() {
// `as_arc()` must hand back the *same* Arc, not a clone of a
// different bit. Verified by writing through the borrowed Arc
// and observing through the Halt.
let halt = Halt::new();
let arc = halt.as_arc().clone();
assert!(!halt.is_cancelled());
arc.store(true, Ordering::Relaxed);
assert!(halt.is_cancelled());
}
}
+3 -3
View File
@@ -9,7 +9,7 @@
use crate::disc::{Codec, Resolution};
use crate::error::{Error, Result};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
// ── Public types ────────────────────────────────────────────────────────────
@@ -184,7 +184,7 @@ fn bcd_byte(b: u8) -> u32 {
///
/// Reads the VMG (Video Manager) to discover title sets, then reads each
/// VTS IFO to extract PGC chains, cell addresses, and stream attributes.
pub fn parse_vmg(reader: &mut dyn SectorReader, udf: &UdfFs) -> Result<DvdInfo> {
pub fn parse_vmg(reader: &mut dyn SectorSource, udf: &UdfFs) -> Result<DvdInfo> {
let vmg_data = udf.read_file(reader, "/VIDEO_TS/VIDEO_TS.IFO")?;
// Validate VMG magic
@@ -263,7 +263,7 @@ pub fn parse_vmg(reader: &mut dyn SectorReader, udf: &UdfFs) -> Result<DvdInfo>
///
/// `titles_info` is a list of (chapter_count, vts_title_number) from TT_SRPT.
fn parse_vts(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf: &UdfFs,
vts_number: u8,
titles_info: &[(u16, u8)],
+33
View File
@@ -0,0 +1,33 @@
//! File I/O helpers that bound kernel cache pressure on big writes.
//!
//! `WritebackFile` is a drop-in wrapper around `std::fs::File` for any
//! call site that performs large sequential writes (sweep, patch, mux,
//! etc.). It implements `Write` and `Seek` so existing code paths can
//! swap `File` for `WritebackFile` with no body changes. Internally it
//! drives a `WritebackPipeline` that, on Linux, drains dirty pages
//! continuously at 32 MB granularity to avoid the kernel's
//! accumulate-then-burst flush behaviour. macOS and Windows use a
//! no-op pipeline — their default cache policies have not been shown
//! to exhibit the same pathology for this access pattern.
//!
//! `Pipeline` + `Sink` (0.18) is the generic producer/consumer primitive
//! used by sweep, patch, and mux to overlap reads with writes via a
//! bounded channel + dedicated consumer thread.
mod writeback;
mod writeback_file;
pub mod pipeline;
pub(crate) use writeback_file::WritebackFile;
// Re-exports for the 0.18 redesign. Sweep + patch are both wired up
// (disc/sweep.rs, disc/patch.rs); mux migrates separately in autorip.
// `WRITE_THROUGH_DEPTH` is patch-specific and has no other in-tree
// caller — the targeted `#[allow]` keeps the re-export visible without
// dragging the rest of the module under `dead_code`.
#[allow(unused_imports)]
pub use pipeline::{
DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, READ_PIPELINE_DEPTH, Sink, WRITE_PIPELINE_DEPTH,
WRITE_THROUGH_DEPTH,
};
+556
View File
@@ -0,0 +1,556 @@
//! Generic bounded producer/consumer pipeline.
//!
//! `Pipeline<I, R>` spawns a single consumer thread, hands it items
//! through a bounded `mpsc::sync_channel`, and joins it on `finish()`.
//! The consumer's behaviour is supplied by a [`Sink`] implementation:
//! `apply` is called once per item, `close` is called once at the end.
//!
//! Three call sites in libfreemkv want a producer/consumer split —
//! sweep (migrated to `disc/sweep.rs::SweepSink`), patch, and mux.
//! 0.18 collapses all three onto this primitive; sweep is in,
//! patch and mux migrate in later 0.18 slices.
//! See `(internal)/memory/0_18_redesign.md` for the full picture.
//!
//! ## Cancellation and error semantics
//!
//! - Producer dropping the channel (via `Pipeline::finish` dropping
//! `tx`) signals end-of-stream; consumer flushes via `close()` and
//! returns its `Output`.
//! - Consumer returning [`Flow::Stop`] also calls `close()` and
//! returns its `Output`. `send()` from the producer will then either
//! succeed (if the item already fit in the channel buffer) or fail
//! with `Err(item)` once the consumer has dropped its receiver.
//! - Consumer returning `Err` from `apply` skips `close()` entirely;
//! the consumer keeps draining the channel so the producer never
//! blocks on a dead receiver, and the first error is propagated as
//! the `JoinHandle` result.
//! - Consumer panic is converted into
//! `Error::IoError { source: io::Error::other(...) }`.
//!
//! ## Debug logging
//!
//! Set `FREEMKV_DEBUG=1` environment variable to enable verbose debug
//! logging throughout the pipeline (channel sends/receives, backpressure,
//! consumer lag detection). This is critical for diagnosing stalls.
use std::io;
use std::sync::mpsc::{SyncSender, sync_channel};
use std::thread::{self, JoinHandle};
use crate::error::Error;
/// Check if verbose debug logging is enabled via FREEMKV_DEBUG env var.
pub fn debug_enabled() -> bool {
std::env::var("FREEMKV_DEBUG")
.ok()
.map(|v| v == "1" || v == "true" || v == "yes")
.unwrap_or(false)
}
/// Default channel depth for callers without a specific reason to
/// pick another value. Kept conservative (4) — most callers should
/// use READ_PIPELINE_DEPTH or WRITE_PIPELINE_DEPTH instead.
pub const DEFAULT_PIPELINE_DEPTH: usize = 4;
/// Read pipeline depth. Larger buffer compensates for drive variability
/// and NFS sync_file_range stalls; keeps ISO reader thread fed even when
/// consumer blocks on write.
pub const READ_PIPELINE_DEPTH: usize = 32;
/// Write pipeline depth. Smaller buffer reduces backpressure risk when
/// sync_file_range blocks; prevents producer from accumulating too much
/// work while consumer waits for NFS to drain.
pub const WRITE_PIPELINE_DEPTH: usize = 16;
/// Channel depth for write-through pipelines. Each `send` fully
/// drains before the next can enqueue. Use this when the producer
/// must observe consumer side-effects (e.g. mapfile state) before
/// emitting the next item.
#[allow(dead_code)]
pub const WRITE_THROUGH_DEPTH: usize = 1;
/// Outcome of [`Sink::apply`]: either keep feeding items
/// ([`Flow::Continue`]), or stop the pipeline early and run `close()`
/// ([`Flow::Stop`]).
///
/// `Stop` has no in-tree caller in this slice — sweep never returns
/// it (it always processes the producer's full work-list before the
/// channel is dropped). Patch and mux are the intended consumers and
/// migrate in later 0.18 slices. The variant ships now so the contract
/// is fixed; the targeted `#[allow]` is removed when patch lands.
pub enum Flow {
Continue,
#[allow(dead_code)]
Stop,
}
/// Consumer-side of a [`Pipeline`]. The pipeline owns one of these on
/// its consumer thread and calls `apply` once per received item, then
/// `close` once at end-of-stream.
pub trait Sink<I>: Send + 'static {
/// Type returned from `close()` and surfaced via
/// [`Pipeline::finish`].
type Output: Send + 'static;
/// Apply one item. Returning [`Flow::Continue`] keeps the
/// pipeline running; [`Flow::Stop`] ends it cleanly (still calls
/// `close()`). An error short-circuits: `close()` is *not* called
/// and the error is what `finish()` will return, but the consumer
/// keeps draining the channel so the producer never blocks on a
/// dead receiver.
fn apply(&mut self, item: I) -> Result<Flow, Error>;
/// Called once at end-of-stream — either because the producer
/// dropped `tx` or because `apply` returned [`Flow::Stop`]. Use
/// this to flush, fsync, finalise. Skipped if any prior `apply`
/// returned `Err`.
fn close(self) -> Result<Self::Output, Error>;
}
/// Bounded producer/consumer pipeline. Holds the producer-side
/// channel and the consumer thread's join handle.
pub struct Pipeline<I: Send + 'static, R: Send + 'static> {
tx: SyncSender<I>,
handle: JoinHandle<Result<R, Error>>,
}
impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
/// Spawn the consumer thread with the given channel depth and
/// [`Sink`].
///
/// The thread is named `freemkv-pipeline-consumer` so it shows up
/// distinctly in stack traces and `top -H`. Callers that want a
/// more specific name (e.g. `freemkv-sweep-consumer`) should use
/// [`Pipeline::spawn_named`] instead. Returns an `Error::IoError`
/// if the OS refuses the thread spawn (resource exhaustion);
/// callers already operate in fallible context, so this is
/// propagated rather than panicked.
///
/// Sweep uses [`Pipeline::spawn_named`] directly so the consumer
/// thread shows up as `freemkv-sweep-consumer`; this function has
/// no in-tree caller yet. Patch and mux migrate in later 0.18
/// slices. The targeted `#[allow]` is removed when one of them
/// lands on the default name.
#[allow(dead_code)]
pub fn spawn<S: Sink<I, Output = R>>(depth: usize, sink: S) -> Result<Self, Error> {
Self::spawn_named("freemkv-pipeline-consumer", depth, sink)
}
/// Like [`Pipeline::spawn`] but lets the caller supply the
/// consumer thread's name. Useful when several pipelines run in
/// the same process and stack traces / `top -H` need to tell them
/// apart (e.g. `freemkv-sweep-consumer`, `freemkv-mux-consumer`).
pub fn spawn_named<S: Sink<I, Output = R>>(
name: &str,
depth: usize,
sink: S,
) -> Result<Self, Error> {
let (tx, rx) = sync_channel::<I>(depth);
let handle = thread::Builder::new()
.name(name.into())
.spawn(move || -> Result<R, Error> {
let mut sink = sink;
let mut first_err: Option<Error> = None;
let mut stopped = false;
while let Ok(item) = rx.recv() {
if debug_enabled() {
tracing::debug!("Pipeline receive: item={}", std::any::type_name::<I>());
}
let apply_start = std::time::Instant::now();
if first_err.is_some() || stopped {
// Drain remaining items so the producer never
// blocks on a dead receiver. `apply` is not
// called once we've decided to stop.
continue;
}
match sink.apply(item) {
Ok(Flow::Continue) => {}
Ok(Flow::Stop) => {
stopped = true;
if debug_enabled() {
tracing::debug!("Pipeline: consumer returned Flow::Stop");
}
}
Err(e) => {
if debug_enabled() {
tracing::debug!("Pipeline: apply error, stopping, err={:?}", e);
}
first_err = Some(e);
}
}
let apply_elapsed = apply_start.elapsed();
if debug_enabled() && apply_elapsed > std::time::Duration::from_millis(100) {
tracing::debug!(
"Pipeline apply: took {:.2}s, item={}",
apply_elapsed.as_secs_f64(),
std::any::type_name::<I>()
);
} else if debug_enabled() {
tracing::debug!(
"Pipeline apply: OK in {:.3}ms, item={}",
apply_elapsed.as_micros(),
std::any::type_name::<I>()
);
}
}
match first_err {
Some(e) => Err(e),
None => sink.close(),
}
})
.map_err(|e| Error::IoError { source: e })?;
Ok(Pipeline { tx, handle })
}
/// Push one item. Blocks if the channel is full — that's the
/// back-pressure the whole primitive exists to provide. Returns
/// the item back if the consumer thread is gone (panicked or
/// already returned).
///
/// After the consumer returns [`Flow::Stop`], `send` will silently
/// buffer items into the channel until the channel fills, then
/// return `Err(item)` once the consumer has dropped its receiver.
/// Producers that need to stop pushing on `Stop` should track an
/// independent signal (e.g. `Halt`) — `send` alone is not the
/// notification edge.
pub fn send(&self, item: I) -> Result<(), I> {
let start = std::time::Instant::now();
match self.tx.send(item) {
Ok(()) => {
let elapsed = start.elapsed();
if debug_enabled() && elapsed > std::time::Duration::from_millis(10) {
tracing::debug!(
"Pipeline send: blocked {:.2}s, item={}",
elapsed.as_secs_f64(),
std::any::type_name::<I>()
);
} else if debug_enabled() {
tracing::debug!("Pipeline send: OK in {:.3}ms", elapsed.as_micros());
}
Ok(())
}
Err(e) => {
let elapsed = start.elapsed();
if debug_enabled() && elapsed > std::time::Duration::from_millis(10) {
tracing::debug!(
"Pipeline send: blocked {:.2}s before channel closed, item={}",
elapsed.as_secs_f64(),
std::any::type_name::<I>()
);
} else if debug_enabled() {
tracing::debug!("Pipeline send: failed after {:.3}ms", elapsed.as_micros());
}
Err(e.0)
}
}
}
/// Non-blocking variant of [`Pipeline::send`]. If the channel is
/// full or the consumer has hung up, the item is returned in
/// `Err`. Useful for best-effort signalling (e.g. sweep's
/// throttled `StatsRequest`) where dropping the message is
/// preferable to blocking the producer.
pub fn try_send(&self, item: I) -> Result<(), std::sync::mpsc::TrySendError<I>> {
self.tx.try_send(item)
}
/// Drop the producer-side channel and wait for the consumer
/// thread to finish. Returns whatever the consumer's `close()`
/// produced, or the first `apply` error, or — on consumer panic —
/// an `Error::IoError` whose source is `io::Error::other(...)`
/// with a "pipeline consumer panicked: <payload>" message
/// (callers can match on the constant prefix).
pub fn finish(self) -> Result<R, Error> {
let Pipeline { tx, handle } = self;
// Explicit drop, although the destructure already drops `tx`
// at end-of-scope. Being explicit keeps the intent obvious.
drop(tx);
match handle.join() {
Ok(result) => result,
Err(payload) => {
// Preserve the original panic message when the
// consumer's panic payload was a `&str` or `String`
// (the two stdlib formats that `panic!` produces).
// Anything else falls back to "(no message)".
let msg = payload
.downcast_ref::<&'static str>()
.copied()
.or_else(|| payload.downcast_ref::<String>().map(|s| s.as_str()))
.unwrap_or("(no message)");
Err(Error::IoError {
source: io::Error::other(format!("pipeline consumer panicked: {msg}")),
})
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
/// Sums u64s; returns the total from `close`.
struct SumSink {
total: u64,
}
impl Sink<u64> for SumSink {
type Output = u64;
fn apply(&mut self, item: u64) -> Result<Flow, Error> {
self.total += item;
Ok(Flow::Continue)
}
fn close(self) -> Result<u64, Error> {
Ok(self.total)
}
}
#[test]
fn happy_path_sums_items() {
let pipe = Pipeline::spawn(DEFAULT_PIPELINE_DEPTH, SumSink { total: 0 })
.expect("spawn should succeed");
let mut expected = 0u64;
for i in 0..100u64 {
expected += i;
pipe.send(i).expect("send should succeed");
}
let total = pipe.finish().expect("finish should succeed");
assert_eq!(total, expected);
assert_eq!(total, (0..100u64).sum::<u64>());
}
/// Sleeps `delay` per apply; counts how many it received.
struct SlowSink {
delay: Duration,
count: Arc<AtomicUsize>,
}
impl Sink<()> for SlowSink {
type Output = usize;
fn apply(&mut self, _item: ()) -> Result<Flow, Error> {
std::thread::sleep(self.delay);
self.count.fetch_add(1, Ordering::SeqCst);
Ok(Flow::Continue)
}
fn close(self) -> Result<usize, Error> {
Ok(self.count.load(Ordering::SeqCst))
}
}
#[test]
fn back_pressure_blocks_sender() {
// depth=2 + 5 sends + 50ms/apply: with the consumer pinned at
// 50 ms per item, the producer can buffer 2 (channel cap) +
// 1 (consumer in flight) = 3 items before sends 4 and 5 must
// block on consumer progress. Wall-clock floor across all 5
// sends is therefore ~2 * 50ms = 100ms (sends 4 and 5 each
// wait roughly one apply-cycle). Use 80 ms as the assertion
// floor to stay above the 50ms-per-item progress floor while
// tolerating CI jitter — it still proves blocking is real.
let count = Arc::new(AtomicUsize::new(0));
let sink = SlowSink {
delay: Duration::from_millis(50),
count: count.clone(),
};
let pipe = Pipeline::spawn(2, sink).expect("spawn should succeed");
let start = Instant::now();
for _ in 0..5 {
pipe.send(()).expect("send should succeed");
}
let elapsed_send = start.elapsed();
let total = pipe.finish().expect("finish should succeed");
assert_eq!(total, 5);
assert!(
elapsed_send >= Duration::from_millis(80),
"back-pressure not observed: 5 sends with depth=2 and 50ms/apply \
took {elapsed_send:?}, expected ≥ ~100ms (one or more sends \
should have blocked behind the consumer)"
);
}
/// Returns `Err` on the Nth apply (1-indexed). Tracks all calls.
struct FailOnNthSink {
n: usize,
seen: Arc<AtomicUsize>,
close_called: Arc<AtomicUsize>,
}
impl Sink<u64> for FailOnNthSink {
type Output = ();
fn apply(&mut self, _item: u64) -> Result<Flow, Error> {
let i = self.seen.fetch_add(1, Ordering::SeqCst) + 1;
if i == self.n {
Err(Error::DecryptFailed)
} else {
Ok(Flow::Continue)
}
}
fn close(self) -> Result<(), Error> {
self.close_called.fetch_add(1, Ordering::SeqCst);
Ok(())
}
}
#[test]
fn apply_error_drains_then_propagates() {
let seen = Arc::new(AtomicUsize::new(0));
let close_called = Arc::new(AtomicUsize::new(0));
let pipe = Pipeline::spawn(
DEFAULT_PIPELINE_DEPTH,
FailOnNthSink {
n: 3,
seen: seen.clone(),
close_called: close_called.clone(),
},
)
.expect("spawn should succeed");
// Send 10 items. Subsequent sends after the 3rd error must
// still succeed (the consumer is draining).
for i in 0..10u64 {
pipe.send(i).expect("send should succeed even after error");
}
let res = pipe.finish();
assert!(matches!(res, Err(Error::DecryptFailed)));
assert_eq!(
close_called.load(Ordering::SeqCst),
0,
"close() must not be called when apply returned Err"
);
// The consumer kept calling `recv` to drain after the error;
// it just stopped invoking `apply`. So `seen` is exactly 3
// (apply was called for items 1, 2, 3).
assert_eq!(seen.load(Ordering::SeqCst), 3);
}
/// Returns `Flow::Stop` on the Nth apply.
struct StopOnNthSink {
n: usize,
seen: Arc<AtomicUsize>,
close_called: Arc<AtomicUsize>,
}
impl Sink<u64> for StopOnNthSink {
type Output = usize;
fn apply(&mut self, _item: u64) -> Result<Flow, Error> {
let i = self.seen.fetch_add(1, Ordering::SeqCst) + 1;
if i >= self.n {
Ok(Flow::Stop)
} else {
Ok(Flow::Continue)
}
}
fn close(self) -> Result<usize, Error> {
self.close_called.fetch_add(1, Ordering::SeqCst);
Ok(self.seen.load(Ordering::SeqCst))
}
}
#[test]
fn apply_stop_calls_close_and_returns_output() {
let seen = Arc::new(AtomicUsize::new(0));
let close_called = Arc::new(AtomicUsize::new(0));
let pipe = Pipeline::spawn(
DEFAULT_PIPELINE_DEPTH,
StopOnNthSink {
n: 3,
seen: seen.clone(),
close_called: close_called.clone(),
},
)
.expect("spawn should succeed");
// Send 10 items. After Stop, subsequent sends may either
// succeed (already buffered) or fail with Err(I) (channel
// closed). Both are valid — we don't assert on the send
// results.
for i in 0..10u64 {
let _ = pipe.send(i);
}
let out = pipe.finish().expect("finish should succeed after Stop");
assert_eq!(close_called.load(Ordering::SeqCst), 1);
// At least 3 items processed (the one that returned Stop).
assert!(
out >= 3,
"expected ≥ 3 applies before Stop took effect, got {out}"
);
}
/// Panics on the first apply call.
struct PanickingSink;
impl Sink<u64> for PanickingSink {
type Output = ();
fn apply(&mut self, _item: u64) -> Result<Flow, Error> {
panic!("synthetic test panic");
}
fn close(self) -> Result<(), Error> {
Ok(())
}
}
#[test]
fn consumer_panic_becomes_io_error() {
// Silence the panic message that would otherwise pollute the
// test output — we expect this panic.
let prev = std::panic::take_hook();
std::panic::set_hook(Box::new(|_| {}));
let pipe =
Pipeline::spawn(DEFAULT_PIPELINE_DEPTH, PanickingSink).expect("spawn should succeed");
// First send may succeed (item buffered before panic) or fail
// (channel closed after panic) — either is fine.
let _ = pipe.send(1);
// Drain a few more sends; once the channel is closed they'll
// return Err(I), which we just discard.
for i in 0..5u64 {
let _ = pipe.send(i);
}
let res = pipe.finish();
std::panic::set_hook(prev);
match res {
Err(Error::IoError { source }) => {
let msg = source.to_string();
// Constant prefix lets callers match without parsing
// the variable payload tail.
assert!(
msg.contains("pipeline consumer panicked"),
"expected constant panic prefix, got: {msg}"
);
// The original `panic!` payload (a `&'static str`) must
// be preserved — without the downcast the message
// would just be the prefix.
assert!(
msg.contains("synthetic test panic"),
"expected original panic payload, got: {msg}"
);
}
other => panic!("expected Err(IoError), got {other:?}"),
}
}
}
+17
View File
@@ -0,0 +1,17 @@
//! Per-platform writeback pipeline. On Linux, drains dirty pages
//! continuously at chunk granularity to keep the kernel's writeback
//! queue bounded. On macOS and Windows, a no-op stub.
//!
//! The platform decision lives entirely in this file (the cfg-gated
//! `pub use` below). Callers — and `DiskWriter` itself — are
//! platform-independent.
#[cfg(target_os = "linux")]
mod linux;
#[cfg(not(target_os = "linux"))]
mod noop;
#[cfg(target_os = "linux")]
pub(super) use linux::WritebackPipeline;
#[cfg(not(target_os = "linux"))]
pub(super) use noop::WritebackPipeline;
+104
View File
@@ -0,0 +1,104 @@
//! Linux writeback pipeline using `sync_file_range` + `posix_fadvise`.
//!
//! Pathology this fixes: the kernel's default `vm.dirty_ratio` (~20 %
//! of RAM) lets dirty pages accumulate to hundreds of MB during a
//! big sequential write, then bursts a flush at 99 % disk utilisation.
//! While the burst runs, app writes block on the writeback queue —
//! observed empirically as instantaneous speed dropping from ~15 MB/s
//! to ~1 MB/s every ~30 s during a Pass 1 sweep.
//!
//! Strategy: every `chunk_bytes` of new sequential output, kick async
//! writeback (`SYNC_FILE_RANGE_WRITE`) on the just-completed chunk and
//! finalise the *previous* chunk via `WAIT_AFTER` + `posix_fadvise
//! (DONTNEED)`. By the time we finalise, that previous chunk has had
//! a full chunk's worth of work to flush — the wait is near-instant.
//! Dirty cache stays bounded at ~2 × `chunk_bytes` and writes drain
//! continuously instead of in bursts.
use std::fs::File;
use std::os::unix::io::{AsRawFd, RawFd};
pub(crate) struct WritebackPipeline {
/// Aliases the wrapping `WritebackFile::file`. Only valid for the
/// lifetime of that struct — moving the `File` independently
/// would silently UAF this fd. The pipeline is a private field of
/// `WritebackFile` and never exposed outside that wrapper, which
/// is what keeps the alias sound.
fd: RawFd,
chunk_bytes: u64,
last_flush_pos: u64,
pending: Option<(u64, u64)>,
}
impl WritebackPipeline {
/// Construct a pipeline aliasing `file`'s file descriptor. The
/// returned `WritebackPipeline` MUST be dropped before `file`
/// itself, or kept inside the same struct that owns `file` — the
/// alias is unchecked.
pub(crate) fn new(file: &File, start_pos: u64, chunk_bytes: u64) -> Self {
Self {
fd: file.as_raw_fd(),
chunk_bytes,
last_flush_pos: start_pos,
pending: None,
}
}
/// Caller advanced the file position to `pos`. If a chunk boundary
/// was crossed, kick async writeback for the just-completed chunk
/// and finalise the previous one.
pub(crate) fn note_progress(&mut self, pos: u64) {
if pos < self.last_flush_pos.saturating_add(self.chunk_bytes) {
return;
}
unsafe {
let chunk_off = self.last_flush_pos as i64;
let chunk_len = (pos - self.last_flush_pos) as i64;
libc::sync_file_range(self.fd, chunk_off, chunk_len, libc::SYNC_FILE_RANGE_WRITE);
if let Some((prev_off, prev_len)) = self.pending.take() {
libc::sync_file_range(
self.fd,
prev_off as i64,
prev_len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
self.pending = Some((chunk_off as u64, chunk_len as u64));
}
self.last_flush_pos = pos;
}
/// Caller is about to seek away from the current write region.
/// Drain any in-flight chunk and reset tracking.
pub(crate) fn handle_seek(&mut self, new_pos: u64) {
self.finalize();
self.last_flush_pos = new_pos;
}
/// Drain any in-flight chunk. Idempotent. Call before `sync_all()`
/// or when discarding the pipeline.
pub(crate) fn finalize(&mut self) {
if let Some((prev_off, prev_len)) = self.pending.take() {
unsafe {
libc::sync_file_range(
self.fd,
prev_off as i64,
prev_len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
}
}
}
+18
View File
@@ -0,0 +1,18 @@
//! No-op writeback pipeline for non-Linux targets. macOS and Windows
//! page cache policies have not been shown to exhibit the Linux
//! accumulate-then-burst flush pathology for our access pattern.
//! If that changes, replace this stub with a real implementation
//! (e.g. `F_NOCACHE` on macOS, `FILE_FLAG_WRITE_THROUGH` on Windows).
use std::fs::File;
pub(crate) struct WritebackPipeline;
impl WritebackPipeline {
pub(crate) fn new(_file: &File, _start_pos: u64, _chunk_bytes: u64) -> Self {
Self
}
pub(crate) fn note_progress(&mut self, _pos: u64) {}
pub(crate) fn handle_seek(&mut self, _new_pos: u64) {}
pub(crate) fn finalize(&mut self) {}
}
+129
View File
@@ -0,0 +1,129 @@
//! `WritebackFile` — a `File` wrapper whose reason for existing is the
//! bounded-cache writeback pipeline.
//!
//! Why: large sequential writes (sweep, patch, mux on UHD-scale output)
//! left to the kernel's default writeback policy accumulate hundreds of
//! megabytes of dirty pages and then burst-flush, stalling subsequent
//! writes for seconds at a time. `WritebackFile` drives a continuous
//! [`super::writeback::WritebackPipeline`] that on Linux issues
//! incremental `sync_file_range` + `posix_fadvise(DONTNEED)` calls at
//! 32 MB granularity so dirty pages drain at the same rate they're
//! produced. macOS and Windows fall through to a no-op pipeline — their
//! default cache policies have not been shown to exhibit the same
//! pathology for this access pattern.
//!
//! It implements `Write` and `Seek` so any call site that wrote to a
//! plain `File` through those traits (sweep, patch, mux) can swap in
//! `WritebackFile` without touching the body of the loop. The wrapper
//! also tracks the current file position to feed the pipeline with
//! progress + seek boundaries.
//!
//! See `super::writeback::linux` for the underlying pathology and the
//! strategy.
use std::fs::{File, OpenOptions};
use std::io::{self, Seek, SeekFrom, Write};
use std::path::Path;
use super::writeback::WritebackPipeline;
const WRITEBACK_CHUNK_BYTES: u64 = 32 * 1024 * 1024;
pub(crate) struct WritebackFile {
file: File,
pipeline: WritebackPipeline,
pos: u64,
}
impl WritebackFile {
/// Wrap an open `File`. The current OS file position is queried
/// once so the pipeline starts tracking from wherever the file
/// already is (typically 0 for fresh files; non-zero for resumed
/// or appended files).
pub(crate) fn new(mut file: File) -> io::Result<Self> {
let pos = file.stream_position()?;
let pipeline = WritebackPipeline::new(&file, pos, WRITEBACK_CHUNK_BYTES);
Ok(Self {
file,
pipeline,
pos,
})
}
/// Create a new file at `path` (truncating any existing contents)
/// and wrap it. Convenience for the common
/// `File::create(path)` + `WritebackFile::new(file)` pair so callers
/// don't have to assemble a `File` first.
pub(crate) fn create(path: &Path) -> io::Result<Self> {
let file = File::create(path)?;
Self::new(file)
}
/// Open an existing file at `path` for writing (no truncation) and
/// wrap it. Mirrors `File::open` semantics for the writable case
/// — used by patch / resume paths that mutate an existing ISO in
/// place.
pub(crate) fn open(path: &Path) -> io::Result<Self> {
let file = OpenOptions::new().write(true).open(path)?;
Self::new(file)
}
/// Drain in-flight writeback then issue a full fsync. Use this in
/// place of `File::sync_all`.
pub(crate) fn sync_all(&mut self) -> io::Result<()> {
self.pipeline.finalize();
self.file.sync_all()
}
}
impl Write for WritebackFile {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let n = self.file.write(buf)?;
self.pos += n as u64;
self.pipeline.note_progress(self.pos);
Ok(n)
}
fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {
self.file.write_all(buf)?;
self.pos += buf.len() as u64;
self.pipeline.note_progress(self.pos);
Ok(())
}
fn flush(&mut self) -> io::Result<()> {
self.file.flush()
}
}
impl Seek for WritebackFile {
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
let p = self.file.seek(from)?;
// Only treat seeks that actually move the position as
// boundaries — sweep does a redundant `seek(Current(pos))`
// before every write, and we don't want that to drain the
// pipeline on every iteration.
if p != self.pos {
self.pipeline.handle_seek(p);
self.pos = p;
}
Ok(p)
}
}
impl Drop for WritebackFile {
fn drop(&mut self) {
// Run the pipeline's tail finalize so the last in-flight chunk
// gets its `WAIT_AFTER` + `posix_fadvise(DONTNEED)`. Without
// this, callers that drop a `WritebackFile` without calling
// `sync_all` (panic, early-return, idiomatic `let _ = w;`)
// leave the trailing chunk in cache; the kernel still flushes
// on close, but the bounded-cache invariant fails at the tail.
// We deliberately do *not* call `self.file.sync_all()` here —
// close already triggers a flush, and an `fsync` from `Drop`
// would silently swallow its `io::Error` anyway. `finalize` is
// idempotent so an explicit `sync_all` followed by drop is
// still safe.
self.pipeline.finalize();
}
}
+402
View File
@@ -0,0 +1,402 @@
//! BDMV disc-library metadata (`/BDMV/META/DL/bdmt_<lang>.xml`).
//!
//! Every commercial Blu-ray carries a disc-library metadata directory
//! with one XML file per shipped language. The schema is the Blu-ray
//! "disc library metadata" namespace (`urn:BDA:bdmv;disclibmeta`),
//! conventionally prefixed `di:`. Fields commonly present:
//!
//! - `<di:title>` or `<di:name>` — the title string. Vendor practice
//! varies (Paramount discs tend to use `<di:name>`).
//! - `<di:description>` — optional synopsis (often absent on retail
//! discs; common on box sets and special editions).
//! - `<di:discNumber>` / `<di:numSets>` (or `<di:numberOfSets>`) —
//! set position for multi-disc releases.
//!
//! This module is intentionally separate from the BD-J `StreamLabel`
//! parsers under `labels/*.rs`. The XML here is disc-level (title,
//! description, set position), not per-stream — wiring into the main
//! parser registry happens elsewhere.
//!
//! Real-world XML is irregular: missing description elements, multiple
//! title elements (first one wins), and occasional malformed content.
//! Extraction is best-effort — a malformed file is treated as "no
//! metadata" (returns `None` from the helper), and the caller can
//! still get metadata from sibling-language XML files.
// The module wiring (registry hook + public re-export) is added
// separately. Until then the parse/detect entry points have no
use super::xml;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::BTreeMap;
/// Disc-level metadata extracted from `/BDMV/META/DL/bdmt_*.xml`.
///
/// All maps are keyed by 3-char ISO 639-2 language code (e.g.
/// `"eng"`, `"fra"`, `"jpn"`) — the same key segment used in the
/// `bdmt_<lang>.xml` filename.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Default)]
pub struct DiscMetadata {
/// Localized titles, keyed by 3-char ISO 639-2 lang code
/// (e.g. "eng" → "Dune Part Two")
pub titles: BTreeMap<String, String>,
/// First-line / short description, per lang
pub descriptions: BTreeMap<String, String>,
/// Disc N of M for box sets (None if not a box set)
pub disc_number: Option<(u32, u32)>,
}
/// True if `/BDMV/META/DL/` exists and contains at least one
/// `bdmt_*.xml` file.
pub fn detect(udf: &UdfFs) -> bool {
let Some(dir) = udf.find_dir("/BDMV/META/DL") else {
return false;
};
dir.entries
.iter()
.any(|e| !e.is_dir && is_bdmt_filename(&e.name))
}
/// Read every `bdmt_<lang>.xml` under `/BDMV/META/DL/` and return the
/// aggregated [`DiscMetadata`]. Returns `None` if no titles could be
/// extracted from any file.
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<DiscMetadata> {
let dir = udf.find_dir("/BDMV/META/DL")?;
let mut out = DiscMetadata::default();
for entry in &dir.entries {
if entry.is_dir {
continue;
}
let Some(lang) = lang_code_from_filename(&entry.name) else {
continue;
};
let path = format!("/BDMV/META/DL/{}", entry.name);
let Ok(bytes) = udf.read_file(reader, &path) else {
continue;
};
let Ok(text) = std::str::from_utf8(&bytes) else {
continue;
};
let Some((title, description, disc_set)) = parse_bdmt_xml(&lang, text) else {
continue;
};
out.titles.insert(lang.clone(), title);
if let Some(desc) = description {
out.descriptions.insert(lang.clone(), desc);
}
// Disc-set position is disc-global; first one we successfully
// read wins. (All bdmt_*.xml on a given disc carry the same
// value in practice.)
if out.disc_number.is_none() {
if let Some(ds) = disc_set {
out.disc_number = Some(ds);
}
}
}
if out.titles.is_empty() {
None
} else {
Some(out)
}
}
/// True if `name` matches the `bdmt_<lang>.xml` convention with a
/// 3-character ISO 639-2 lang code segment. Case-insensitive.
fn is_bdmt_filename(name: &str) -> bool {
lang_code_from_filename(name).is_some()
}
/// Extract the 3-char language code from a `bdmt_<lang>.xml` filename.
/// Returns `None` if the filename doesn't match. Lang code is
/// lowercased so callers always see e.g. `"eng"` not `"ENG"`.
fn lang_code_from_filename(name: &str) -> Option<String> {
let lower = name.to_ascii_lowercase();
let stem = lower.strip_suffix(".xml")?;
let lang = stem.strip_prefix("bdmt_")?;
// ISO 639-2 codes are exactly 3 ASCII letters. Be strict — keeps
// us from picking up unrelated `bdmt_foo.xml` siblings.
if lang.len() != 3 || !lang.chars().all(|c| c.is_ascii_alphabetic()) {
return None;
}
Some(lang.to_string())
}
/// Tuple returned by [`parse_bdmt_xml`]: `(title, description?, disc_set?)`.
/// Aliased so the function signature isn't a clippy::type-complexity offender.
pub(crate) type BdmtFields = (String, Option<String>, Option<(u32, u32)>);
/// Parse one `bdmt_<lang>.xml` document and return
/// `(title, description?, disc_set?)`. Returns `None` if no title
/// could be located — the caller treats this as "skip this file".
///
/// Title-element preference: `<di:name>` → `<di:title>` →
/// `<di:tableOfContents>/<di:titleName>` (first match wins, per the
/// authoring-tool conventions documented at the module level).
pub(crate) fn parse_bdmt_xml(_lang_code: &str, xml_text: &str) -> Option<BdmtFields> {
let title = extract_title(xml_text)?;
let description = xml::text(xml_text, "description")
.filter(|s| !s.is_empty())
.filter(|s| !looks_like_xml(s))
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty());
let disc_set = extract_disc_set(xml_text);
Some((title, description, disc_set))
}
/// Reject candidate description strings that are themselves XML
/// fragments — observed on disc-04 (Top Gun: Maverick), where
/// `<di:description>` contained `<di:thumbnail href="…"/>` child
/// elements and no actual prose. Surfacing that raw to the JSON
/// output is worse than dropping the field entirely.
fn looks_like_xml(s: &str) -> bool {
let t = s.trim_start();
t.starts_with('<')
}
/// Try title-bearing element variants in priority order. The `xml`
/// helpers are case- and namespace-insensitive, so callers pass the
/// bare local name (no `di:` prefix).
fn extract_title(xml_text: &str) -> Option<String> {
// Order matches the module-level convention: <di:name> first
// (Paramount-style), then <di:title>, then the nested
// tableOfContents/titleName form.
for tag in ["name", "title"] {
if let Some(s) = xml::text(xml_text, tag) {
let trimmed = s.trim();
if !trimmed.is_empty() {
return Some(trimmed.to_string());
}
}
}
// tableOfContents/titleName: search inside the toc block so we
// don't accidentally pick a stray <titleName> from elsewhere.
if let Some((s, e)) = xml::find_element(xml_text, "tableOfContents", 0) {
let block = &xml_text[s..e];
if let Some(t) = xml::text(block, "titleName") {
let trimmed = t.trim();
if !trimmed.is_empty() {
return Some(trimmed.to_string());
}
}
}
None
}
/// Extract `(discNumber, numSets)` if both are present and parse as
/// `u32`. Accepts either `<di:numSets>` or `<di:numberOfSets>` for
/// the denominator (both forms appear in the wild).
fn extract_disc_set(xml_text: &str) -> Option<(u32, u32)> {
let n = xml::text(xml_text, "discNumber")?
.trim()
.parse::<u32>()
.ok()?;
let total = xml::text(xml_text, "numSets")
.or_else(|| xml::text(xml_text, "numberOfSets"))?
.trim()
.parse::<u32>()
.ok()?;
Some((n, total))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extract_simple_title() {
// Minimal Paramount-style document: <di:name> as the title
// carrier inside a <discInfo> root.
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<discInfo xmlns:di="urn:BDA:bdmv;disclibmeta">
<di:name>Dune Part Two</di:name>
</discInfo>"#;
let (title, desc, set) = parse_bdmt_xml("eng", xml).expect("title should parse");
assert_eq!(title, "Dune Part Two");
assert_eq!(desc, None);
assert_eq!(set, None);
}
#[test]
fn extract_title_element_variant() {
// <di:title> is the alternate carrier; should be picked up
// when <di:name> is absent.
let xml = r#"<discInfo xmlns:di="urn:BDA:bdmv;disclibmeta">
<di:title>The Matrix</di:title>
<di:description>A film about computers.</di:description>
</discInfo>"#;
let (title, desc, _) = parse_bdmt_xml("eng", xml).unwrap();
assert_eq!(title, "The Matrix");
assert_eq!(desc.as_deref(), Some("A film about computers."));
}
#[test]
fn extract_title_from_table_of_contents_fallback() {
// Some authoring tools nest the title under tableOfContents.
// No <di:name> or <di:title> at top level → fall back to
// titleName inside tableOfContents.
let xml = r#"<discInfo xmlns:di="urn:BDA:bdmv;disclibmeta">
<di:tableOfContents>
<di:titleName>Inside Out 2</di:titleName>
</di:tableOfContents>
</discInfo>"#;
let (title, _, _) = parse_bdmt_xml("eng", xml).unwrap();
assert_eq!(title, "Inside Out 2");
}
#[test]
fn extract_box_set_position() {
let xml = r#"<discInfo xmlns:di="urn:BDA:bdmv;disclibmeta">
<di:name>LOTR Disc 2</di:name>
<di:discNumber>2</di:discNumber>
<di:numSets>5</di:numSets>
</discInfo>"#;
let (_, _, set) = parse_bdmt_xml("eng", xml).unwrap();
assert_eq!(set, Some((2, 5)));
}
#[test]
fn extract_box_set_position_alternate_total_tag() {
// <di:numberOfSets> is an alternate spelling we've seen.
let xml = r#"<discInfo xmlns:di="urn:BDA:bdmv;disclibmeta">
<di:name>X</di:name>
<di:discNumber>3</di:discNumber>
<di:numberOfSets>6</di:numberOfSets>
</discInfo>"#;
let (_, _, set) = parse_bdmt_xml("eng", xml).unwrap();
assert_eq!(set, Some((3, 6)));
}
#[test]
fn extract_box_set_requires_both_fields() {
// discNumber alone (no total) yields None — we don't fabricate
// a denominator.
let xml = r#"<discInfo xmlns:di="urn:BDA:bdmv;disclibmeta">
<di:name>X</di:name>
<di:discNumber>1</di:discNumber>
</discInfo>"#;
let (_, _, set) = parse_bdmt_xml("eng", xml).unwrap();
assert_eq!(set, None);
}
#[test]
fn multiple_languages_keyed_correctly() {
// Simulate driving parse_bdmt_xml from two synthetic XML
// blobs and aggregating into DiscMetadata the same way parse()
// would. This exercises the BTreeMap key handling without
// needing a UdfFs.
let eng_xml = r#"<discInfo xmlns:di="urn:BDA:bdmv;disclibmeta">
<di:name>Dune Part Two</di:name>
</discInfo>"#;
let fra_xml = r#"<discInfo xmlns:di="urn:BDA:bdmv;disclibmeta">
<di:name>Dune Deuxième Partie</di:name>
<di:description>Suite du film de 2021.</di:description>
</discInfo>"#;
let mut meta = DiscMetadata::default();
for (lang, blob) in [("eng", eng_xml), ("fra", fra_xml)] {
let (title, desc, ds) = parse_bdmt_xml(lang, blob).unwrap();
meta.titles.insert(lang.to_string(), title);
if let Some(d) = desc {
meta.descriptions.insert(lang.to_string(), d);
}
if meta.disc_number.is_none() {
if let Some(d) = ds {
meta.disc_number = Some(d);
}
}
}
assert_eq!(
meta.titles.get("eng").map(String::as_str),
Some("Dune Part Two")
);
assert_eq!(
meta.titles.get("fra").map(String::as_str),
Some("Dune Deuxième Partie")
);
assert!(meta.descriptions.get("eng").is_none());
assert_eq!(
meta.descriptions.get("fra").map(String::as_str),
Some("Suite du film de 2021.")
);
assert_eq!(meta.disc_number, None);
}
#[test]
fn malformed_xml_returns_none() {
// Random gibberish has no recognizable title element. We
// document the contract: parse_bdmt_xml returns None, and
// parse() (the caller) skips the file. Aggregating across
// zero files leaves DiscMetadata::default() — which parse()
// surfaces as None to its caller. Either is documented as
// acceptable per the module spec.
let bad = "this is not xml &&& <<< nope";
assert!(parse_bdmt_xml("eng", bad).is_none());
// Half-open tag, no body, no close: also yields no title.
let truncated = "<discInfo><di:name>";
assert!(parse_bdmt_xml("eng", truncated).is_none());
}
#[test]
fn description_with_only_child_xml_is_dropped() {
// Real-world bug from disc-04 (Top Gun: Maverick, 2026-05-11
// capture): <di:description> contained only <di:thumbnail/>
// child elements with no actual prose. The previous parser
// surfaced the raw XML fragment as the description string.
// Now we reject candidates that begin with `<`.
let xml = r#"<discInfo>
<di:name>Top Gun: Maverick</di:name>
<di:description>
<di:thumbnail href="tgm_meta_sm.jpg" />
<di:thumbnail href="tgm_meta_lg.jpg" />
</di:description>
</discInfo>"#;
let (title, description, _) =
parse_bdmt_xml("eng", xml).expect("title is present so parse must succeed");
assert_eq!(title, "Top Gun: Maverick");
assert!(
description.is_none(),
"description containing only XML children must be dropped, got {description:?}"
);
}
#[test]
fn description_with_plain_text_passes_through() {
// The legitimate case still works: a description with actual
// prose survives the looks_like_xml filter.
let xml = r#"<discInfo>
<di:name>Some Movie</di:name>
<di:description>An epic tale of one man's quest for tea.</di:description>
</discInfo>"#;
let (_, description, _) = parse_bdmt_xml("eng", xml).expect("must parse");
assert_eq!(
description.as_deref(),
Some("An epic tale of one man's quest for tea.")
);
}
#[test]
fn whitespace_in_title_is_trimmed() {
let xml = r#"<discInfo><di:name>
Dune Part Two
</di:name></discInfo>"#;
let (title, _, _) = parse_bdmt_xml("eng", xml).unwrap();
assert_eq!(title, "Dune Part Two");
}
#[test]
fn lang_code_extraction() {
assert_eq!(lang_code_from_filename("bdmt_eng.xml"), Some("eng".into()));
assert_eq!(lang_code_from_filename("BDMT_FRA.XML"), Some("fra".into()));
assert_eq!(lang_code_from_filename("bdmt_jpn.xml"), Some("jpn".into()));
// Non-matching cases:
assert_eq!(lang_code_from_filename("bdmt_.xml"), None);
assert_eq!(lang_code_from_filename("bdmt_engl.xml"), None);
assert_eq!(lang_code_from_filename("bdmt_e1g.xml"), None);
assert_eq!(lang_code_from_filename("bdmt_eng.txt"), None);
assert_eq!(lang_code_from_filename("foo.xml"), None);
}
}
File diff suppressed because it is too large Load Diff
+291
View File
@@ -0,0 +1,291 @@
//! CLPI vs MPLS cross-validation diagnostic.
//!
//! Empirical question (raised 2026-05-11): is CLPI's per-stream
//! language and codec data truly redundant with MPLS's STN-table data
//! on real-world Blu-rays?
//!
//! Build a quick audit that walks both sources, normalizes their stream
//! lists by (PID, language, coding_type), and flags any disagreement.
//!
//! Three classes of mismatch we want to detect:
//!
//! 1. **CLPI has streams MPLS doesn't reference.** Orphan streams in
//! the .m2ts that no playlist's STN table includes. Means the user
//! can't reach them through the menu but they're physically on the
//! disc.
//! 2. **MPLS has streams CLPI doesn't list.** Should never happen if
//! both parsers are correct — playlists reference clips which
//! reference streams. If it happens, one of our parsers has a bug.
//! 3. **Same PID, different language / coding_type.** The playlist re-
//! tagged a stream's metadata. Rare but spec-permitted. Means CLPI
//! and MPLS disagree about the same physical stream's properties.
//!
//! If audits across the corpus show zero mismatches of any class, CLPI
//! program_info extraction is **empirically redundant** for labels and
//! we can leave it out of the registry. If even one mismatch surfaces,
//! we add a CLPI label parser to the registry as belt-and-suspenders.
//!
//! This module exposes `audit(reader, udf)` returning a structured
//! report. Surfaced via the labels-analyze tool — not part of the
//! `analyze()` pipeline (no impact on the label output).
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::BTreeMap;
/// One row in the audit: a stream PID that's known to one source or
/// both, with the fields each source reported.
#[derive(Debug, Clone)]
pub struct ClpiVsMplsRow {
pub pid: u16,
pub clpi_coding_type: Option<u8>,
pub clpi_language: Option<String>,
pub mpls_coding_type: Option<u8>,
pub mpls_language: Option<String>,
}
impl ClpiVsMplsRow {
/// Three rules for classification:
/// - both sources missing (impossible — caller wouldn't insert)
/// - one source missing → class A or B (orphan-on-disc / playlist-only)
/// - both present but fields differ → class C (metadata divergence)
/// - both present and identical → no mismatch
pub fn class(&self) -> ClpiVsMplsClass {
match (
self.clpi_coding_type.is_some(),
self.mpls_coding_type.is_some(),
) {
(true, false) => ClpiVsMplsClass::ClpiOnly,
(false, true) => ClpiVsMplsClass::MplsOnly,
(true, true) => {
let coding_match = self.clpi_coding_type == self.mpls_coding_type;
let lang_match = self.clpi_language == self.mpls_language;
if coding_match && lang_match {
ClpiVsMplsClass::Match
} else {
ClpiVsMplsClass::Divergent
}
}
(false, false) => ClpiVsMplsClass::Match,
}
}
}
/// Classification of one (PID) row.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ClpiVsMplsClass {
/// PID seen in CLPI ProgramInfo but no MPLS STN table references
/// it. Orphan on disc.
ClpiOnly,
/// PID seen in MPLS STN table but no CLPI ProgramInfo includes it.
/// One of our parsers probably has a bug.
MplsOnly,
/// Both sources see this PID with the same coding_type + language.
Match,
/// Both sources see this PID but disagree on coding_type or language.
/// MPLS wins for label rendering (playlist-authoritative view); CLPI
/// is the per-clip ground truth.
Divergent,
}
/// Full audit report.
#[derive(Debug, Clone, Default)]
pub struct ClpiVsMplsAudit {
pub rows: Vec<ClpiVsMplsRow>,
}
impl ClpiVsMplsAudit {
pub fn class_counts(&self) -> (usize, usize, usize, usize) {
let mut clpi_only = 0;
let mut mpls_only = 0;
let mut matches = 0;
let mut divergent = 0;
for r in &self.rows {
match r.class() {
ClpiVsMplsClass::ClpiOnly => clpi_only += 1,
ClpiVsMplsClass::MplsOnly => mpls_only += 1,
ClpiVsMplsClass::Match => matches += 1,
ClpiVsMplsClass::Divergent => divergent += 1,
}
}
(clpi_only, mpls_only, matches, divergent)
}
}
/// Walk `/BDMV/CLIPINF/*.clpi` and `/BDMV/PLAYLIST/*.mpls`, build a
/// dedup-by-PID table of (CLPI fields, MPLS fields), return the
/// merged view. Missing files (read errors, parse failures) are
/// silently skipped — this is diagnostic, not correctness-critical.
pub fn audit(reader: &mut dyn SectorSource, udf: &UdfFs) -> ClpiVsMplsAudit {
// Aggregate by PID across all CLPI files. If a PID appears in
// multiple clips (typical — main movie clip + trailers reference
// the same audio stream PIDs), first encountered wins (they should
// all agree per BD spec).
let mut clpi_by_pid: BTreeMap<u16, (u8, String)> = BTreeMap::new();
if let Some(dir) = udf.find_dir("/BDMV/CLIPINF") {
let names: Vec<String> = dir
.entries
.iter()
.filter(|e| !e.is_dir && e.name.to_ascii_lowercase().ends_with(".clpi"))
.map(|e| e.name.clone())
.collect();
for name in names {
let path = format!("/BDMV/CLIPINF/{}", name);
let Ok(data) = udf.read_file(reader, &path) else {
continue;
};
let Ok(clip) = crate::clpi::parse(&data) else {
continue;
};
for s in clip.streams {
clpi_by_pid
.entry(s.pid)
.or_insert((s.coding_type, s.language));
}
}
}
// Same for MPLS streams.
let mut mpls_by_pid: BTreeMap<u16, (u8, String)> = BTreeMap::new();
if let Some(dir) = udf.find_dir("/BDMV/PLAYLIST") {
let names: Vec<String> = dir
.entries
.iter()
.filter(|e| !e.is_dir && e.name.to_ascii_lowercase().ends_with(".mpls"))
.map(|e| e.name.clone())
.collect();
for name in names {
let path = format!("/BDMV/PLAYLIST/{}", name);
let Ok(data) = udf.read_file(reader, &path) else {
continue;
};
let Ok(pl) = crate::mpls::parse(&data) else {
continue;
};
for s in pl.streams {
if s.pid == 0 {
// PID 0 means "no PID in stream entry" — skip rather
// than collide with other entries.
continue;
}
mpls_by_pid
.entry(s.pid)
.or_insert((s.coding_type, s.language));
}
}
}
// Merge views: every PID seen anywhere gets a row.
let mut all_pids: std::collections::BTreeSet<u16> = std::collections::BTreeSet::new();
all_pids.extend(clpi_by_pid.keys().copied());
all_pids.extend(mpls_by_pid.keys().copied());
let mut rows = Vec::with_capacity(all_pids.len());
for pid in all_pids {
let clpi = clpi_by_pid.get(&pid);
let mpls = mpls_by_pid.get(&pid);
rows.push(ClpiVsMplsRow {
pid,
clpi_coding_type: clpi.map(|(c, _)| *c),
clpi_language: clpi.map(|(_, l)| l.clone()),
mpls_coding_type: mpls.map(|(c, _)| *c),
mpls_language: mpls.map(|(_, l)| l.clone()),
});
}
ClpiVsMplsAudit { rows }
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn class_match_when_identical() {
let r = ClpiVsMplsRow {
pid: 0x1100,
clpi_coding_type: Some(0x83),
clpi_language: Some("eng".into()),
mpls_coding_type: Some(0x83),
mpls_language: Some("eng".into()),
};
assert_eq!(r.class(), ClpiVsMplsClass::Match);
}
#[test]
fn class_clpi_only_when_mpls_missing() {
let r = ClpiVsMplsRow {
pid: 0x1100,
clpi_coding_type: Some(0x83),
clpi_language: Some("eng".into()),
mpls_coding_type: None,
mpls_language: None,
};
assert_eq!(r.class(), ClpiVsMplsClass::ClpiOnly);
}
#[test]
fn class_mpls_only_when_clpi_missing() {
let r = ClpiVsMplsRow {
pid: 0x1100,
clpi_coding_type: None,
clpi_language: None,
mpls_coding_type: Some(0x90),
mpls_language: Some("fra".into()),
};
assert_eq!(r.class(), ClpiVsMplsClass::MplsOnly);
}
#[test]
fn class_divergent_on_lang_disagreement() {
let r = ClpiVsMplsRow {
pid: 0x1100,
clpi_coding_type: Some(0x83),
clpi_language: Some("eng".into()),
mpls_coding_type: Some(0x83),
mpls_language: Some("und".into()),
};
assert_eq!(r.class(), ClpiVsMplsClass::Divergent);
}
#[test]
fn class_counts_sum_rows() {
let audit = ClpiVsMplsAudit {
rows: vec![
ClpiVsMplsRow {
pid: 0x1100,
clpi_coding_type: Some(0x83),
clpi_language: Some("eng".into()),
mpls_coding_type: Some(0x83),
mpls_language: Some("eng".into()),
}, // Match
ClpiVsMplsRow {
pid: 0x1101,
clpi_coding_type: Some(0x83),
clpi_language: Some("fra".into()),
mpls_coding_type: None,
mpls_language: None,
}, // ClpiOnly
ClpiVsMplsRow {
pid: 0x1102,
clpi_coding_type: None,
clpi_language: None,
mpls_coding_type: Some(0x90),
mpls_language: Some("eng".into()),
}, // MplsOnly
ClpiVsMplsRow {
pid: 0x1103,
clpi_coding_type: Some(0x86),
clpi_language: Some("spa".into()),
mpls_coding_type: Some(0x86),
mpls_language: Some("ita".into()),
}, // Divergent
],
};
let (co, mo, m, d) = audit.class_counts();
assert_eq!(co, 1);
assert_eq!(mo, 1);
assert_eq!(m, 1);
assert_eq!(d, 1);
}
}
+56 -92
View File
@@ -3,8 +3,8 @@
//! Clean structured XML with Content/Qualifier per stream and
//! stream number mapping via playbackconfig.
use super::{LabelPurpose, LabelQualifier, StreamLabel, StreamLabelType};
use crate::sector::SectorReader;
use super::{LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, xml};
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::HashMap;
@@ -12,7 +12,7 @@ pub fn detect(udf: &UdfFs) -> bool {
super::jar_file_exists(udf, "streamproperties.xml")
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
let sp_data = super::read_jar_file(reader, udf, "streamproperties.xml")?;
let sp_text = std::str::from_utf8(&sp_data).ok()?;
@@ -66,7 +66,8 @@ pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLab
if labels.is_empty() {
return None;
}
Some(labels)
// High confidence: streamproperties.xml is fully structured.
Some(ParseResult::high(labels))
}
struct StreamInfo {
@@ -78,105 +79,68 @@ struct StreamInfo {
qualifier: LabelQualifier,
}
fn parse_stream_infos(xml: &str) -> Vec<StreamInfo> {
fn parse_stream_infos(text: &str) -> Vec<StreamInfo> {
let mut infos = Vec::new();
let mut pos = 0;
while pos < xml.len() {
let (tag, stream_type) = if let Some(p) = xml[pos..].find("<AudioStreamInfos>") {
(p + pos, StreamLabelType::Audio)
} else if let Some(p) = xml[pos..].find("<SubtitleStreamInfos>") {
(p + pos, StreamLabelType::Subtitle)
} else {
break;
};
for (tag_name, stream_type) in [
("AudioStreamInfos", StreamLabelType::Audio),
("SubtitleStreamInfos", StreamLabelType::Subtitle),
] {
let mut from = 0;
while let Some((start, end)) = xml::find_element(text, tag_name, from) {
let block = &text[start..end];
let id = xml::text(block, "ID").unwrap_or_default();
let lang_id = xml::text(block, "LangInfoID").unwrap_or_default();
let content = xml::text(block, "Content").unwrap_or_default();
let qualifier_str = xml::text(block, "Qualifier").unwrap_or_default();
let end_tag = match stream_type {
StreamLabelType::Audio => "</AudioStreamInfos>",
StreamLabelType::Subtitle => "</SubtitleStreamInfos>",
};
let (language, variant) = if lang_id.contains('_') {
let parts: Vec<&str> = lang_id.splitn(2, '_').collect();
(parts[0].to_lowercase(), parts[1].to_string())
} else {
(lang_id.to_lowercase(), String::new())
};
let block_end = match xml[tag..].find(end_tag) {
Some(p) => tag + p + end_tag.len(),
None => break,
};
let purpose = if content.eq_ignore_ascii_case("COMMENTARY") {
LabelPurpose::Commentary
} else {
LabelPurpose::Normal
};
let block = &xml[tag..block_end];
let id = extract_tag(block, "ID").unwrap_or_default();
let lang_id = extract_tag(block, "LangInfoID").unwrap_or_default();
let content = extract_tag(block, "Content").unwrap_or_default();
let qualifier_str = extract_tag(block, "Qualifier").unwrap_or_default();
let qualifier = match qualifier_str.to_ascii_uppercase().as_str() {
"SDH" => LabelQualifier::Sdh,
"DS" => LabelQualifier::DescriptiveService,
_ => LabelQualifier::None,
};
let (language, variant) = if lang_id.contains('_') {
let parts: Vec<&str> = lang_id.splitn(2, '_').collect();
(parts[0].to_lowercase(), parts[1].to_string())
} else {
(lang_id.to_lowercase(), String::new())
};
let purpose = match content.as_str() {
"COMMENTARY" => LabelPurpose::Commentary,
_ => LabelPurpose::Normal,
};
let qualifier = match qualifier_str.as_str() {
"SDH" => LabelQualifier::Sdh,
"DS" => LabelQualifier::DescriptiveService,
_ => LabelQualifier::None,
};
infos.push(StreamInfo {
id,
stream_type,
language,
variant,
purpose,
qualifier,
});
pos = block_end;
infos.push(StreamInfo {
id,
stream_type,
language,
variant,
purpose,
qualifier,
});
from = end;
}
}
infos
}
fn parse_playback_config(xml: &str, map: &mut HashMap<String, u16>) {
let mut pos = 0;
while pos < xml.len() {
let tag_start = if let Some(p) = xml[pos..].find("<AudioStreams>") {
Some(p + pos)
} else {
xml[pos..].find("<SubtitlesStreams>").map(|p| p + pos)
};
let tag_start = match tag_start {
Some(p) => p,
None => break,
};
let block_end = xml[tag_start..]
.find("</AudioStreams>")
.or_else(|| xml[tag_start..].find("</SubtitlesStreams>"))
.map(|p| tag_start + p + 20)
.unwrap_or(xml.len());
let block = &xml[tag_start..block_end];
if let (Some(stream_id_str), Some(info_id)) = (
extract_tag(block, "StreamID"),
extract_tag(block, "StreamInfo_ID"),
) {
if let Ok(stream_num) = stream_id_str.parse::<u16>() {
map.insert(info_id, stream_num);
fn parse_playback_config(text: &str, map: &mut HashMap<String, u16>) {
for tag_name in ["AudioStreams", "SubtitlesStreams"] {
let mut from = 0;
while let Some((start, end)) = xml::find_element(text, tag_name, from) {
let block = &text[start..end];
if let (Some(stream_id_str), Some(info_id)) = (
xml::text(block, "StreamID"),
xml::text(block, "StreamInfo_ID"),
) {
if let Ok(stream_num) = stream_id_str.parse::<u16>() {
map.insert(info_id, stream_num);
}
}
from = end;
}
pos = block_end;
}
}
fn extract_tag(xml: &str, tag: &str) -> Option<String> {
let open = format!("<{tag}>");
let close = format!("</{tag}>");
let start = xml.find(&open)? + open.len();
let end = xml[start..].find(&close)? + start;
Some(xml[start..end].trim().to_string())
}
+185 -17
View File
@@ -4,8 +4,8 @@
//! When both exist, language_streams.txt provides structured types while
//! menu_base.prop provides stream number → button name mapping.
use super::{LabelPurpose, LabelQualifier, StreamLabel, StreamLabelType, vocab};
use crate::sector::SectorReader;
use super::{LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, vocab};
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::HashMap;
@@ -14,7 +14,7 @@ pub fn detect(udf: &UdfFs) -> bool {
|| super::jar_file_exists(udf, "language_streams.txt")
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
// Try language_streams.txt first (richer structured data)
let ls_labels = parse_language_streams(reader, udf);
@@ -22,12 +22,18 @@ pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLab
let mb_labels = parse_menu_base(reader, udf);
// If we have both, merge: language_streams for structure, menu_base for names
match (ls_labels, mb_labels) {
(Some(ls), Some(mb)) => Some(merge(ls, mb)),
(Some(ls), None) => Some(ls),
(None, Some(mb)) => Some(mb),
(None, None) => None,
let labels = match (ls_labels, mb_labels) {
(Some(ls), Some(mb)) => merge(ls, mb),
(Some(ls), None) => ls,
(None, Some(mb)) => mb,
(None, None) => return None,
};
if labels.is_empty() {
return None;
}
// High confidence: both language_streams.txt and menu_base.prop
// are structured key-value formats with documented types.
Some(ParseResult::high(labels))
}
fn merge(ls: Vec<StreamLabel>, mb: Vec<StreamLabel>) -> Vec<StreamLabel> {
@@ -49,7 +55,7 @@ fn merge(ls: Vec<StreamLabel>, mb: Vec<StreamLabel>) -> Vec<StreamLabel> {
// ── language_streams.txt parser ────────────────────────────────────────────
fn parse_language_streams(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
fn parse_language_streams(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
let data = super::read_jar_file(reader, udf, "language_streams.txt")?;
let text = std::str::from_utf8(&data).ok()?;
@@ -172,9 +178,167 @@ fn parse_language_streams(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<
Some(labels)
}
#[cfg(test)]
mod tests {
use super::*;
/// Build a minimal menu_base.prop text and run `parse_menu_base`'s
/// inner logic via a temporary closure. This isolates the prop
/// parsing without needing a SectorSource.
fn parse_props(text: &str) -> Vec<StreamLabel> {
// Mirror the inner loop of parse_menu_base exactly. Kept
// separate so the test doesn't need disc fixtures.
use std::collections::HashMap;
let mut entries: HashMap<String, HashMap<String, String>> = HashMap::new();
for line in text.lines() {
let line = line.trim();
if line.is_empty() || line.starts_with('#') {
continue;
}
let Some(eq_pos) = line.find('=') else {
continue;
};
let full_key = &line[..eq_pos];
let value = &line[eq_pos + 1..];
if let Some(dot_pos) = full_key.rfind('.') {
entries
.entry(full_key[..dot_pos].to_string())
.or_default()
.insert(full_key[dot_pos + 1..].to_string(), value.to_string());
}
}
let mut labels = Vec::new();
for (prefix, props) in &entries {
let is_audio = props
.get("class")
.is_some_and(|c| c.contains("AudioButton"))
|| prefix.starts_with("audio_");
let is_subtitle = props
.get("class")
.is_some_and(|c| c.contains("SubtitleButton"))
|| prefix.starts_with("subtitle_");
let stream_num_str = props
.get("streamNumber")
.or_else(|| props.get("audioStream"))
.or_else(|| props.get("subtitleStream"));
let stream_num: u16 = match stream_num_str.and_then(|s| s.parse().ok()) {
Some(n) if n > 0 => n,
_ => continue,
};
if !is_audio && !is_subtitle {
continue;
}
let name = props.get("name").cloned().unwrap_or_default();
let purpose = match vocab::purpose(&name) {
LabelPurpose::Normal if prefix.contains("comm") => LabelPurpose::Commentary,
p => p,
};
let qualifier = if is_subtitle {
vocab::qualifier(&name)
} else {
LabelQualifier::None
};
let stream_type = if is_audio {
StreamLabelType::Audio
} else {
StreamLabelType::Subtitle
};
let language = props
.get("audioLanguage")
.or_else(|| props.get("subtitleLanguage"))
.cloned()
.unwrap_or_default();
labels.push(StreamLabel {
stream_number: stream_num,
stream_type,
language,
name,
purpose,
qualifier,
codec_hint: String::new(),
variant: String::new(),
});
}
labels.sort_by_key(|l| (l.stream_type as u8, l.stream_number));
labels
}
#[test]
fn commentary_via_name() {
let labels = parse_props(
"audio_1.class=AudioButton\n\
audio_1.streamNumber=2\n\
audio_1.name=Director's Commentary\n\
audio_1.audioLanguage=eng\n",
);
assert_eq!(labels.len(), 1);
assert_eq!(labels[0].purpose, LabelPurpose::Commentary);
assert_eq!(labels[0].language, "eng");
}
#[test]
fn commentary_via_prefix_when_name_silent() {
let labels = parse_props(
"audio_commentary_1.class=AudioButton\n\
audio_commentary_1.streamNumber=2\n\
audio_commentary_1.name=Track 2\n\
audio_commentary_1.audioLanguage=eng\n",
);
assert_eq!(labels[0].purpose, LabelPurpose::Commentary);
}
#[test]
fn commenter_does_not_false_match_commentary() {
// Regression for the pre-refactor `name.contains("comment")`
// bug: this would wrongly classify a "Commenter Pro" track as
// Commentary. vocab::purpose enforces a word boundary.
let labels = parse_props(
"audio_1.class=AudioButton\n\
audio_1.streamNumber=2\n\
audio_1.name=Commenter Pro Track\n\
audio_1.audioLanguage=eng\n",
);
assert_eq!(labels[0].purpose, LabelPurpose::Normal);
}
#[test]
fn descriptive_via_name() {
let labels = parse_props(
"audio_1.class=AudioButton\n\
audio_1.streamNumber=3\n\
audio_1.name=English Descriptive Audio\n",
);
assert_eq!(labels[0].purpose, LabelPurpose::Descriptive);
}
#[test]
fn sdh_only_on_subtitles() {
// SDH applied to a subtitle stream.
let labels = parse_props(
"subtitle_1.class=SubtitleButton\n\
subtitle_1.streamNumber=4\n\
subtitle_1.name=English SDH\n",
);
assert_eq!(labels[0].qualifier, LabelQualifier::Sdh);
}
#[test]
fn sdh_not_applied_to_audio_stream_even_if_name_contains_sdh() {
// Audio streams should not pick up SDH (it's a subtitle
// concept). Edge case: badly-authored name happens to include
// "SDH" — we don't propagate it to audio metadata.
let labels = parse_props(
"audio_1.class=AudioButton\n\
audio_1.streamNumber=5\n\
audio_1.name=English SDH (track?)\n",
);
assert_eq!(labels[0].qualifier, LabelQualifier::None);
}
}
// ── menu_base.prop parser ──────────────────────────────────────────────────
fn parse_menu_base(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
fn parse_menu_base(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
let data = super::read_jar_file(reader, udf, "menu_base.prop")?;
let text = std::str::from_utf8(&data).ok()?;
@@ -231,16 +395,20 @@ fn parse_menu_base(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<Str
}
let name = props.get("name").cloned().unwrap_or_default();
let name_lower = name.to_lowercase();
let purpose = if name_lower.contains("comment") || prefix.contains("comm") {
LabelPurpose::Commentary
} else {
LabelPurpose::Normal
// Purpose: ask vocab first (word-boundary matched — avoids the
// "Commenter" false positive the prior `name.contains("comment")`
// had). Then fall back to the structural prefix check
// (`audio_commentary.foo`-style keys group commentary streams
// regardless of display name).
let purpose = match vocab::purpose(&name) {
LabelPurpose::Normal if prefix.contains("comm") => LabelPurpose::Commentary,
p => p,
};
let qualifier = if is_subtitle && name_lower.contains("sdh") {
LabelQualifier::Sdh
// Qualifier: only apply to subtitles (SDH is a subtitle concept).
let qualifier = if is_subtitle {
vocab::qualifier(&name)
} else {
LabelQualifier::None
};
+261
View File
@@ -0,0 +1,261 @@
//! "dbp" framework — Magnolia Pictures BD-J authoring shop (per
//! `bd-live.magpictures.com` referenced in the disc's
//! `com/dbp/bluray.MenuXlet.perm`). Detected on UHD discs whose
//! `/BDMV/JAR/<x>.jar` (top-level, not in a subdir) contains
//! `com/dbp/` package paths.
//!
//! Stream labels live as plain ASCII strings inside compiled `.class`
//! files in the jar — a quirk of the menu-rendering layer encoding
//! its TextField positions and content as constant strings the
//! Java compiler retained in the class string pool. Format observed
//! in the corpus (Civil War UHD, 2024):
//!
//! ```text
//! LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,...
//! RTextField,Audio2,English Descriptive Audio,Fontstrip_Composite,...
//! HTextField,Subtitle1,English SDH,Fontstrip_Composite,...
//! ATextField,Subtitle0,None,Fontstrip_Composite,...
//! ```
//!
//! The single uppercase letter before `TextField` is string-pool
//! prefix noise — the parser anchors on `TextField,` regardless of
//! what precedes it. `Subtitle0` is the disable-subtitles menu
//! button and is skipped (not a real subtitle stream).
//!
//! ## Implementation
//!
//! v2 (2026-05-10): rewritten on top of [`super::class_reader`] —
//! iterates `CpInfo::Utf8` constant-pool entries instead of raw byte
//! scanning each class file. Equivalent label coverage (the literal
//! `TextField,...` strings live in the CP as Utf8 entries), but
//! structurally cleaner: no false-positive risk from method bytecode
//! or attribute names happening to contain `TextField,`. Language /
//! purpose / qualifier classification moved to [`super::vocab`] so all
//! Java-parser families share one source of truth.
use super::class_reader::CpInfo;
use super::{ParseResult, StreamLabel, StreamLabelType, jar, vocab};
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::BTreeMap;
/// dbp detect can't peek inside a jar without a SectorSource (the
/// trait function only takes `&UdfFs`), so we trigger on the cheap
/// signal "any top-level .jar in /BDMV/JAR/." That fires on every
/// BD-J disc, but parse() does the real `com/dbp/` check and
/// returns None on a mismatch — so this parser only ever consumes
/// time on discs that fell through every earlier parser.
pub fn detect(udf: &UdfFs) -> bool {
jar::has_any_top_level_jar(udf)
}
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
jar::for_each_jar(reader, udf, |_entry_name, archive| {
if !jar::has_path_prefix(archive, "com/dbp/") {
return None;
}
let labels = scan_jar(archive);
if labels.is_empty() {
None
} else {
// High confidence: TextField,Audio1,... is a stable anchor
// pattern + vocab routes language/purpose/qualifier.
Some(ParseResult::high(labels))
}
})
}
fn scan_jar(archive: &mut jar::Jar) -> Vec<StreamLabel> {
// BTreeMap so we keep the highest-numbered (last-written) label
// for each stream slot deterministic across runs. The same
// TextField,Audio1,... string can appear in multiple classes
// (button-state variants, localization fallbacks). Last write
// wins — they should all agree, but the structure is defensive.
let mut audios: BTreeMap<u16, String> = BTreeMap::new();
let mut subs: BTreeMap<u16, String> = BTreeMap::new();
jar::for_each_class(archive, |_class_name, class| {
for (_idx, cp) in class.constant_pool.iter() {
if let CpInfo::Utf8(s) = cp {
collect_textfield(s, &mut audios, &mut subs);
}
}
});
let mut out = Vec::new();
for (num, label) in audios {
out.push(make_label(num, label, StreamLabelType::Audio));
}
for (num, label) in subs {
out.push(make_label(num, label, StreamLabelType::Subtitle));
}
out
}
fn collect_textfield(
s: &str,
audios: &mut BTreeMap<u16, String>,
subs: &mut BTreeMap<u16, String>,
) {
// Anchor on "TextField," — the prefix character before it varies
// (string-pool ordering inside compiled Java) and is irrelevant.
let Some(idx) = s.find("TextField,") else {
return;
};
let after = &s[idx + "TextField,".len()..];
let mut parts = after.splitn(3, ',');
let kind_n = parts.next().unwrap_or("").trim();
let label = parts.next().unwrap_or("").trim();
if label.is_empty() {
return;
}
if let Some(rest) = kind_n.strip_prefix("Audio") {
if let Ok(n) = rest.parse::<u16>() {
audios.insert(n, label.to_string());
}
} else if let Some(rest) = kind_n.strip_prefix("Subtitle") {
if let Ok(n) = rest.parse::<u16>() {
// Subtitle0 is conventionally the "None / Off" disable
// button, not an actual subtitle stream.
if n > 0 {
subs.insert(n, label.to_string());
}
}
}
}
fn make_label(num: u16, label: String, stream_type: StreamLabelType) -> StreamLabel {
let lang_info = vocab::lang(&label);
let language = lang_info.map(|l| l.code).unwrap_or("").to_string();
let variant = lang_info.map(|l| l.variant).unwrap_or("").to_string();
let qualifier = vocab::qualifier(&label);
let purpose = vocab::purpose(&label);
StreamLabel {
stream_number: num,
stream_type,
language,
name: label,
purpose,
qualifier,
codec_hint: String::new(),
variant,
}
}
#[cfg(test)]
mod tests {
use super::super::{LabelPurpose, LabelQualifier};
use super::*;
#[test]
fn collect_extracts_audio_and_subtitle_indices() {
let mut audios = BTreeMap::new();
let mut subs = BTreeMap::new();
let lines = [
"LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,296,763,275,25,left",
"RTextField,Audio2,English Descriptive Audio,Fontstrip_Composite,296,803,275,25,left",
"RTextField,Audio3,Spanish 5.1 Dolby Digital,Fontstrip_Composite,296,843,275,25,left",
"ATextField,Subtitle0,None,Fontstrip_Composite,1312,843,275,25,left",
"HTextField,Subtitle1,English SDH,Fontstrip_Composite,1312,763,275,25,left",
"DTextField,Subtitle2,Spanish,Fontstrip_Composite,1312,803,275,25,left",
];
for s in &lines {
collect_textfield(s, &mut audios, &mut subs);
}
assert_eq!(audios.len(), 3);
assert_eq!(audios[&1], "English Dolby Atmos");
assert_eq!(audios[&2], "English Descriptive Audio");
assert_eq!(audios[&3], "Spanish 5.1 Dolby Digital");
// Subtitle0 ("None") is skipped — disable button, not a stream.
assert_eq!(subs.len(), 2);
assert_eq!(subs[&1], "English SDH");
assert_eq!(subs[&2], "Spanish");
}
#[test]
fn collect_ignores_non_textfield_strings() {
let mut audios = BTreeMap::new();
let mut subs = BTreeMap::new();
for s in [
"GraphicButton,SU_Audio",
"AudioMenu",
"CommentaryMenuAlternateScenes",
"PrimaryAudioControl",
] {
collect_textfield(s, &mut audios, &mut subs);
}
assert!(audios.is_empty());
assert!(subs.is_empty());
}
#[test]
fn make_label_routes_via_vocab() {
let l = make_label(1, "English SDH".to_string(), StreamLabelType::Subtitle);
assert_eq!(l.language, "eng");
assert_eq!(l.qualifier, LabelQualifier::Sdh);
assert_eq!(l.purpose, LabelPurpose::Normal);
}
#[test]
fn make_label_descriptive_audio() {
let l = make_label(
2,
"English Descriptive Audio".to_string(),
StreamLabelType::Audio,
);
assert_eq!(l.language, "eng");
assert_eq!(l.purpose, LabelPurpose::Descriptive);
}
#[test]
fn make_label_commentary() {
let l = make_label(
3,
"English Director's Commentary".to_string(),
StreamLabelType::Audio,
);
assert_eq!(l.language, "eng");
assert_eq!(l.purpose, LabelPurpose::Commentary);
}
#[test]
fn make_label_compound_languages_populate_variant() {
let brazilian = make_label(1, "Brazilian Portuguese 5.1".into(), StreamLabelType::Audio);
assert_eq!(brazilian.language, "por");
assert_eq!(brazilian.variant, "Brazilian");
let castilian = make_label(1, "Castilian Spanish".into(), StreamLabelType::Audio);
assert_eq!(castilian.language, "spa");
assert_eq!(castilian.variant, "Castilian");
let canadian = make_label(
1,
"Canadian French Dolby Digital".into(),
StreamLabelType::Audio,
);
assert_eq!(canadian.language, "fra");
assert_eq!(canadian.variant, "Canadian");
}
#[test]
fn make_label_bare_language_has_empty_variant() {
let l = make_label(1, "English Dolby Atmos".into(), StreamLabelType::Audio);
assert_eq!(l.language, "eng");
assert_eq!(l.variant, "");
}
#[test]
fn make_label_unknown_language_is_empty() {
// vocab::lang returns None — make_label converts both fields to "".
let l = make_label(1, "Klingon Dolby Atmos".into(), StreamLabelType::Audio);
assert_eq!(l.language, "");
assert_eq!(l.variant, "");
}
#[test]
fn make_label_rnib_descriptive_service() {
let l = make_label(1, "English RNIB".into(), StreamLabelType::Subtitle);
assert_eq!(l.language, "eng");
assert_eq!(l.qualifier, LabelQualifier::DescriptiveService);
}
}
+1644
View File
File diff suppressed because it is too large Load Diff
+151
View File
@@ -0,0 +1,151 @@
//! BD-J jar utilities — common scaffolding for parsers that read
//! `/BDMV/JAR/*.jar`.
//!
//! Composes with [`class_reader`](super::class_reader) for structured
//! `.class` access. Used by `dbp` (string-pool scan via constant pool)
//! and `deluxe` (bytecode pattern matching) — those parsers express
//! "open every top-level jar, look at every .class inside" without
//! repeating the zip-archive boilerplate.
// `try_each_class` is staged for `labels::deluxe`, which needs the
// early-return form to short-circuit class iteration on a match.
// dead-code allow comes off when deluxe lands.
#![allow(dead_code)]
use super::class_reader::ClassFile;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::io::Cursor;
use zip::ZipArchive;
/// In-memory zip archive: backed by a `Vec<u8>` read from UDF. Owns
/// the buffer; callers pass it to [`has_path_prefix`], [`for_each_class`],
/// etc.
pub type Jar = ZipArchive<Cursor<Vec<u8>>>;
/// True if `/BDMV/JAR/` contains at least one top-level `.jar` file
/// (not under a subdir). Used by `detect()` in parsers whose real
/// signal lives inside a jar — they can't open the jar without a
/// `SectorSource`, so they use this cheap pre-check and do the real
/// `com/<vendor>/` discriminator in `parse()`.
pub fn has_any_top_level_jar(udf: &UdfFs) -> bool {
let Some(jar_dir) = udf.find_dir("/BDMV/JAR") else {
return false;
};
jar_dir
.entries
.iter()
.any(|e| !e.is_dir && e.name.to_lowercase().ends_with(".jar"))
}
/// Open every top-level `*.jar` entry in `/BDMV/JAR/` and yield each
/// `(entry_name, Jar)` to `f`. Returns the first `Some(R)` the callback
/// produces, or `None` if every jar was visited without a hit.
///
/// "Top-level" means entries directly under `/BDMV/JAR/`, not nested
/// under a subdir. (Pixelogic, Criterion, Paramount, etc. put their
/// data files inside `/BDMV/JAR/<x>/`; dbp and Deluxe put their jar
/// directly at `/BDMV/JAR/<name>.jar`.)
///
/// Entries that fail to read from UDF or that aren't valid zips are
/// silently skipped — same defensive shape as the existing dbp parser.
pub fn for_each_jar<R, F>(reader: &mut dyn SectorSource, udf: &UdfFs, mut f: F) -> Option<R>
where
F: FnMut(&str, &mut Jar) -> Option<R>,
{
let jar_dir = udf.find_dir("/BDMV/JAR")?;
for entry in &jar_dir.entries {
if entry.is_dir {
continue;
}
if !entry.name.to_lowercase().ends_with(".jar") {
continue;
}
let path = format!("/BDMV/JAR/{}", entry.name);
let Ok(bytes) = udf.read_file(reader, &path) else {
continue;
};
let Ok(mut archive) = ZipArchive::new(Cursor::new(bytes)) else {
continue;
};
if let Some(r) = f(&entry.name, &mut archive) {
return Some(r);
}
}
None
}
/// True if any entry in this jar's central directory starts with
/// `prefix`. Fast — only reads filenames, never extracts bytes.
///
/// Used by parsers as a cheap "is this MY framework's jar?" check
/// (e.g. `has_path_prefix(archive, "com/dbp/")` for dbp,
/// `has_path_prefix(archive, "com/bydeluxe/")` for Deluxe).
pub fn has_path_prefix(archive: &mut Jar, prefix: &str) -> bool {
for i in 0..archive.len() {
if let Ok(f) = archive.by_index(i) {
if f.name().starts_with(prefix) {
return true;
}
}
}
false
}
/// Iterate every `.class` entry in the jar, parse it with
/// [`class_reader`], and call `f` with `(entry_name, &ClassFile)`.
///
/// Entries that fail to read or parse are silently skipped — this is
/// label-extraction code, robustness matters more than completeness.
/// Callers that need to know which classes failed should use the
/// lower-level [`class_reader`] API directly.
pub fn for_each_class<F>(archive: &mut Jar, mut f: F)
where
F: FnMut(&str, &ClassFile),
{
for i in 0..archive.len() {
let Ok(mut entry) = archive.by_index(i) else {
continue;
};
if !entry.name().ends_with(".class") {
continue;
}
let name = entry.name().to_string();
let mut bytes = Vec::with_capacity(entry.size() as usize);
if std::io::Read::read_to_end(&mut entry, &mut bytes).is_err() {
continue;
}
let Ok(class) = ClassFile::parse(&bytes) else {
continue;
};
f(&name, &class);
}
}
/// Like [`for_each_class`] but allows the callback to short-circuit
/// iteration. Returns the first `Some(R)` the callback produces.
pub fn try_each_class<R, F>(archive: &mut Jar, mut f: F) -> Option<R>
where
F: FnMut(&str, &ClassFile) -> Option<R>,
{
for i in 0..archive.len() {
let Ok(mut entry) = archive.by_index(i) else {
continue;
};
if !entry.name().ends_with(".class") {
continue;
}
let name = entry.name().to_string();
let mut bytes = Vec::with_capacity(entry.size() as usize);
if std::io::Read::read_to_end(&mut entry, &mut bytes).is_err() {
continue;
}
let Ok(class) = ClassFile::parse(&bytes) else {
continue;
};
if let Some(r) = f(&name, &class) {
return Some(r);
}
}
None
}
+1130 -15
View File
File diff suppressed because it is too large Load Diff
+597
View File
@@ -0,0 +1,597 @@
//! Universal MPLS-based stream labels.
//!
//! Unlike the framework-specific parsers in this directory (dbp,
//! pixelogic, ctrm, criterion, ...), this module is the *floor*:
//! every Blu-ray ships with MPLS playlists under `/BDMV/PLAYLIST/`,
//! and every MPLS file has an STN table with per-stream ISO 639-2
//! language codes plus coding-type / channel-layout / sample-rate
//! bytes from the BD spec.
//!
//! The framework parsers extract richer editorial labels ("English
//! Dolby Atmos", "Director's Commentary") when the disc was authored
//! with a recognized tool. When none of them match (e.g. a "no BD-J"
//! disc, or an authoring framework we haven't catalogued), MPLS still
//! gives us language + codec on every stream — enough to render
//! something more useful than the bare PID.
//!
//! Output confidence is Low: MPLS carries language + codec but
//! never purpose/qualifier info (no way to tell "Commentary" from
//! "Normal" from the STN table alone). Higher-confidence framework
//! parsers, when present, always win on the registry's max-by-confidence
//! tiebreaker — MPLS is only chosen when nothing else matched.
use super::{
LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType,
vocab::{self, LangInfo},
};
use crate::sector::SectorSource;
use crate::udf::UdfFs;
/// True iff `/BDMV/PLAYLIST/` exists and contains at least one
/// `.mpls` file. Cheap directory walk only — no sector reads.
pub fn detect(udf: &UdfFs) -> bool {
let Some(dir) = udf.find_dir("/BDMV/PLAYLIST") else {
return false;
};
dir.entries
.iter()
.any(|e| !e.is_dir && has_mpls_extension(&e.name))
}
/// Walk every `*.mpls` in `/BDMV/PLAYLIST/`, parse it, and convert
/// each StreamEntry to a [`StreamLabel`]. Streams shared across
/// playlists (same PID) are deduped.
///
/// Returns `None` if no labels could be produced (e.g. no .mpls files
/// parsed successfully, or every parsed stream was a type we skip
/// like IG / DV EL).
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
let playlist_dir = udf.find_dir("/BDMV/PLAYLIST")?;
// Collect mpls filenames first so we don't hold a borrow on udf
// while we call udf.read_file (which takes &self).
let mpls_names: Vec<String> = playlist_dir
.entries
.iter()
.filter(|e| !e.is_dir && has_mpls_extension(&e.name))
.map(|e| e.name.clone())
.collect();
if mpls_names.is_empty() {
return None;
}
let mut labels: Vec<StreamLabel> = Vec::new();
// (stream_type_tag, language, codec_hint, pid) — PID is the
// canonical "same physical stream" key; type+lang+codec round
// out the rare case where two distinct logical streams happen
// to share a PID across playlists with different metadata.
let mut seen: Vec<(u8, String, String, u16)> = Vec::new();
// Global 1-based counters keyed by StreamLabelType. Incremented
// only when an entry survives dedup, so stream_numbers are dense
// (1, 2, 3, ...) per type across the whole disc — not reset per
// playlist. A disc with 2 MPLS files that each list the same
// 8 audio streams ends up with audio_1..audio_8, not audio_1..
// audio_16 or audio_1..audio_8 with audio_1 duplicated.
let mut audio_idx: u16 = 0;
let mut sub_idx: u16 = 0;
for name in &mpls_names {
let path = format!("/BDMV/PLAYLIST/{}", name);
let Ok(data) = udf.read_file(reader, &path) else {
continue;
};
let Ok(playlist) = crate::mpls::parse(&data) else {
continue;
};
for entry in &playlist.streams {
let label_type = match entry.stream_type {
2 | 5 => StreamLabelType::Audio, // primary + secondary audio
3 => StreamLabelType::Subtitle, // PG subtitle
// 1 = primary video, 6 = secondary video, 7 = DV EL
// → no StreamLabelType variant for video, skip.
// 4 = IG (interactive graphics) — not a user-facing
// stream, skip.
_ => continue,
};
let language = normalize_language(&entry.language);
let name = language_display_name(&language);
let codec_hint = build_codec_hint(label_type, entry);
let type_tag = type_tag(label_type);
let key = (type_tag, language.clone(), codec_hint.clone(), entry.pid);
if seen.contains(&key) {
continue;
}
seen.push(key);
let stream_number = match label_type {
StreamLabelType::Audio => {
audio_idx += 1;
audio_idx
}
StreamLabelType::Subtitle => {
sub_idx += 1;
sub_idx
}
};
labels.push(StreamLabel {
stream_number,
stream_type: label_type,
language,
name,
purpose: LabelPurpose::Normal,
qualifier: LabelQualifier::None,
codec_hint,
variant: String::new(),
});
}
}
if labels.is_empty() {
return None;
}
// MPLS gives language + codec but never editorial info (no
// commentary/SDH/director's cut). Low confidence means framework
// parsers (paramount, criterion, pixelogic, ctrm, dbp, deluxe) always
// win when they match. MPLS only gets chosen as the parser when
// nothing else fired — exactly the universal-fallback role we want.
Some(ParseResult::low(labels))
}
fn has_mpls_extension(name: &str) -> bool {
// Case-insensitive ".mpls" suffix. Some discs use uppercase,
// some lowercase; UDF filenames preserve case but we don't.
let n = name.len();
if n < 5 {
return false;
}
name[n - 5..].eq_ignore_ascii_case(".mpls")
}
/// Lowercase + trim the raw 3-char ISO 639-2 code. If the lowered
/// string maps via [`vocab::lang`] (it won't for plain "eng" — that
/// matcher is for English-name fragments, not codes) use its
/// canonical code; otherwise return the trimmed lowercase string.
fn normalize_language(raw: &str) -> String {
let trimmed = raw.trim().to_ascii_lowercase();
if trimmed.is_empty() {
return String::new();
}
// vocab::lang() matches free-form English names, not ISO 639-2
// codes — so for the typical MPLS payload ("eng", "fra", ...)
// it returns None and we keep the trimmed code.
if let Some(LangInfo { code, .. }) = vocab::lang(&trimmed) {
return code.to_string();
}
trimmed
}
/// Human-readable English name for an ISO 639-2 code, or empty if
/// the code is unknown. Kept inline rather than in vocab because
/// vocab is the *reverse* mapping (name → code).
pub(crate) fn language_display_name(iso: &str) -> String {
match iso {
"eng" => "English",
"fra" | "fre" => "French",
"spa" => "Spanish",
"deu" | "ger" => "German",
"ita" => "Italian",
"jpn" => "Japanese",
"zho" | "chi" => "Chinese",
"kor" => "Korean",
"por" => "Portuguese",
"pol" => "Polish",
"ces" | "cze" => "Czech",
"hun" => "Hungarian",
"nld" | "dut" => "Dutch",
"ara" => "Arabic",
"hin" => "Hindi",
"tur" => "Turkish",
"tha" => "Thai",
"swe" => "Swedish",
"nor" => "Norwegian",
"dan" => "Danish",
"fin" => "Finnish",
"heb" => "Hebrew",
"rus" => "Russian",
"ell" | "gre" => "Greek",
"vie" => "Vietnamese",
"ind" => "Indonesian",
"msa" | "may" => "Malay",
"ukr" => "Ukrainian",
"ron" | "rum" => "Romanian",
"bul" => "Bulgarian",
"hrv" => "Croatian",
"srp" => "Serbian",
"slk" | "slo" => "Slovak",
"slv" => "Slovenian",
"est" => "Estonian",
"lav" => "Latvian",
"lit" => "Lithuanian",
"isl" | "ice" => "Icelandic",
"eus" | "baq" => "Basque",
"cat" => "Catalan",
"glg" => "Galician",
_ => "",
}
.to_string()
}
/// Map BD coding_type byte → codec name. Returns empty for unknown
/// bytes (the table covers everything the spec defines, but unknown
/// values are still possible on malformed discs).
pub(crate) fn codec_name(coding_type: u8) -> &'static str {
match coding_type {
0x02 => "MPEG-2",
0x1B => "H.264",
0x24 => "HEVC",
0x80 => "LPCM",
0x81 => "AC-3",
0x82 => "DTS",
0x83 => "TrueHD",
0x84 => "AC-3+",
0x85 => "DTS-HD",
0x86 => "DTS-HD MA",
0x90 => "PG",
0x91 => "IG",
0xA1 => "AC-3+ Secondary",
0xA2 => "DTS-HD Secondary",
_ => "",
}
}
/// Build the final `codec_hint`. For audio streams, optionally
/// append " <channels>" and/or " <rate>" suffixes. Sample rate is
/// only spelled out for non-48k (the universal default).
fn build_codec_hint(label_type: StreamLabelType, entry: &crate::mpls::StreamEntry) -> String {
let base = codec_name(entry.coding_type);
if base.is_empty() {
return String::new();
}
if label_type != StreamLabelType::Audio {
return base.to_string();
}
let mut out = base.to_string();
let channels = match entry.audio_format {
1 => Some("mono"),
3 => Some("2.0"),
6 => Some("5.1"),
12 => Some("7.1"),
_ => None,
};
if let Some(ch) = channels {
out.push(' ');
out.push_str(ch);
}
// 1 = 48 kHz (universal default, omit). Only call out higher rates.
let rate = match entry.audio_rate {
4 => Some("96kHz"),
5 => Some("192kHz"),
_ => None,
};
if let Some(r) = rate {
out.push(' ');
out.push_str(r);
}
out
}
/// Dedup-tag for the label type. `u8` instead of `StreamLabelType`
/// itself because the enum does not derive `Hash` / `Eq`-by-discriminant
/// in a way that we want to couple to (and `==` works fine for the
/// linear `Vec::contains` lookup we do).
fn type_tag(t: StreamLabelType) -> u8 {
match t {
StreamLabelType::Audio => 1,
StreamLabelType::Subtitle => 2,
}
}
// ── Tests ────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use crate::mpls::{Playlist, StreamEntry};
fn audio_entry(pid: u16, coding: u8, fmt: u8, rate: u8, lang: &str) -> StreamEntry {
StreamEntry {
stream_type: 2,
pid,
coding_type: coding,
video_format: 0,
video_rate: 0,
audio_format: fmt,
audio_rate: rate,
language: lang.to_string(),
dynamic_range: 0,
color_space: 0,
secondary: false,
}
}
fn pg_entry(pid: u16, lang: &str) -> StreamEntry {
StreamEntry {
stream_type: 3,
pid,
coding_type: 0x90,
video_format: 0,
video_rate: 0,
audio_format: 0,
audio_rate: 0,
language: lang.to_string(),
dynamic_range: 0,
color_space: 0,
secondary: false,
}
}
fn playlist_with(streams: Vec<StreamEntry>) -> Playlist {
Playlist {
version: "0200".to_string(),
play_items: Vec::new(),
streams,
marks: Vec::new(),
}
}
/// Drive the same conversion logic that `parse()` runs on real
/// disc data, but starting from already-parsed Playlists so we
/// don't have to synthesize valid MPLS bytes.
fn labels_from_playlists(playlists: &[Playlist]) -> Vec<StreamLabel> {
let mut labels: Vec<StreamLabel> = Vec::new();
let mut seen: Vec<(u8, String, String, u16)> = Vec::new();
for playlist in playlists {
let mut audio_idx: u16 = 0;
let mut sub_idx: u16 = 0;
for entry in &playlist.streams {
let label_type = match entry.stream_type {
2 | 5 => StreamLabelType::Audio,
3 => StreamLabelType::Subtitle,
_ => continue,
};
let stream_number = match label_type {
StreamLabelType::Audio => {
audio_idx += 1;
audio_idx
}
StreamLabelType::Subtitle => {
sub_idx += 1;
sub_idx
}
};
let language = normalize_language(&entry.language);
let name = language_display_name(&language);
let codec_hint = build_codec_hint(label_type, entry);
let key = (
type_tag(label_type),
language.clone(),
codec_hint.clone(),
entry.pid,
);
if seen.contains(&key) {
continue;
}
seen.push(key);
labels.push(StreamLabel {
stream_number,
stream_type: label_type,
language,
name,
purpose: LabelPurpose::Normal,
qualifier: LabelQualifier::None,
codec_hint,
variant: String::new(),
});
}
}
labels
}
#[test]
fn mpls_audio_streams_become_labels() {
// Two audio streams: English TrueHD 7.1 48k, French AC-3 5.1 48k.
let pl = playlist_with(vec![
audio_entry(0x1100, 0x83, 12, 1, "eng"),
audio_entry(0x1101, 0x81, 6, 1, "fra"),
]);
let labels = labels_from_playlists(&[pl]);
assert_eq!(labels.len(), 2);
// English TrueHD 7.1
let a = &labels[0];
assert_eq!(a.stream_type, StreamLabelType::Audio);
assert_eq!(a.stream_number, 1);
assert_eq!(a.language, "eng");
assert_eq!(a.name, "English");
assert_eq!(a.codec_hint, "TrueHD 7.1");
assert_eq!(a.purpose, LabelPurpose::Normal);
assert_eq!(a.qualifier, LabelQualifier::None);
assert_eq!(a.variant, "");
// French AC-3 5.1
let b = &labels[1];
assert_eq!(b.stream_type, StreamLabelType::Audio);
assert_eq!(b.stream_number, 2);
assert_eq!(b.language, "fra");
assert_eq!(b.name, "French");
assert_eq!(b.codec_hint, "AC-3 5.1");
}
#[test]
fn mpls_pg_streams_become_subtitle_labels() {
let pl = playlist_with(vec![
pg_entry(0x1200, "eng"),
pg_entry(0x1201, "spa"),
pg_entry(0x1202, "fra"),
]);
let labels = labels_from_playlists(&[pl]);
assert_eq!(labels.len(), 3);
for label in &labels {
assert_eq!(label.stream_type, StreamLabelType::Subtitle);
assert_eq!(label.codec_hint, "PG");
}
assert_eq!(labels[0].stream_number, 1);
assert_eq!(labels[0].language, "eng");
assert_eq!(labels[0].name, "English");
assert_eq!(labels[1].stream_number, 2);
assert_eq!(labels[1].language, "spa");
assert_eq!(labels[1].name, "Spanish");
assert_eq!(labels[2].stream_number, 3);
assert_eq!(labels[2].language, "fra");
}
#[test]
fn dedup_streams_across_playlists() {
// Two playlists, same English TrueHD 7.1 PID 0x1100 in both.
// Expect one Audio label, not two.
let pl1 = playlist_with(vec![
audio_entry(0x1100, 0x83, 12, 1, "eng"),
audio_entry(0x1101, 0x81, 6, 1, "fra"),
]);
let pl2 = playlist_with(vec![
audio_entry(0x1100, 0x83, 12, 1, "eng"), // duplicate
audio_entry(0x1102, 0x82, 6, 1, "deu"), // new
]);
let labels = labels_from_playlists(&[pl1, pl2]);
// Expected: eng@0x1100, fra@0x1101, deu@0x1102 — three uniques.
assert_eq!(labels.len(), 3);
// PID isn't stored on StreamLabel, so assert on the surviving
// language set instead.
let mut langs: Vec<String> = labels.iter().map(|l| l.language.clone()).collect();
langs.sort();
assert_eq!(langs, vec!["deu", "eng", "fra"]);
}
#[test]
fn coding_type_to_codec_hint_table() {
// Spot-check every entry in the spec table. Audio entries
// come back bare (no channels/rate set) so codec_hint is the
// codec name alone.
let cases: &[(u8, &str)] = &[
(0x02, "MPEG-2"),
(0x1B, "H.264"),
(0x24, "HEVC"),
(0x80, "LPCM"),
(0x81, "AC-3"),
(0x82, "DTS"),
(0x83, "TrueHD"),
(0x84, "AC-3+"),
(0x85, "DTS-HD"),
(0x86, "DTS-HD MA"),
(0x90, "PG"),
(0x91, "IG"),
(0xA1, "AC-3+ Secondary"),
(0xA2, "DTS-HD Secondary"),
];
for (ct, expected) in cases {
assert_eq!(
codec_name(*ct),
*expected,
"coding_type 0x{:02X} should map to {}",
ct,
expected
);
}
// Unknown bytes return empty.
assert_eq!(codec_name(0x00), "");
assert_eq!(codec_name(0xFF), "");
}
#[test]
fn audio_format_appends_channel_layout() {
let mono = audio_entry(1, 0x83, 1, 1, "eng");
let stereo = audio_entry(2, 0x83, 3, 1, "eng");
let surround_51 = audio_entry(3, 0x83, 6, 1, "eng");
let surround_71 = audio_entry(4, 0x83, 12, 1, "eng");
let unknown = audio_entry(5, 0x83, 0, 1, "eng");
assert_eq!(
build_codec_hint(StreamLabelType::Audio, &mono),
"TrueHD mono"
);
assert_eq!(
build_codec_hint(StreamLabelType::Audio, &stereo),
"TrueHD 2.0"
);
assert_eq!(
build_codec_hint(StreamLabelType::Audio, &surround_51),
"TrueHD 5.1"
);
assert_eq!(
build_codec_hint(StreamLabelType::Audio, &surround_71),
"TrueHD 7.1"
);
assert_eq!(build_codec_hint(StreamLabelType::Audio, &unknown), "TrueHD");
}
#[test]
fn audio_rate_only_shows_above_48k() {
// 48 kHz (rate=1) is the universal default → not surfaced.
let r48 = audio_entry(1, 0x83, 6, 1, "eng");
// 96 kHz (rate=4) → surfaced.
let r96 = audio_entry(2, 0x83, 6, 4, "eng");
// 192 kHz (rate=5) → surfaced.
let r192 = audio_entry(3, 0x83, 6, 5, "eng");
assert_eq!(build_codec_hint(StreamLabelType::Audio, &r48), "TrueHD 5.1");
assert_eq!(
build_codec_hint(StreamLabelType::Audio, &r96),
"TrueHD 5.1 96kHz"
);
assert_eq!(
build_codec_hint(StreamLabelType::Audio, &r192),
"TrueHD 5.1 192kHz"
);
}
#[test]
fn unknown_iso_code_passes_through_without_display_name() {
// Made-up code: keep the raw lowercase code as `language`,
// but `name` is empty because we don't know it.
let pl = playlist_with(vec![audio_entry(0x1100, 0x83, 6, 1, "xyz")]);
let labels = labels_from_playlists(&[pl]);
assert_eq!(labels.len(), 1);
assert_eq!(labels[0].language, "xyz");
assert_eq!(labels[0].name, "");
}
#[test]
fn ig_and_dv_streams_are_skipped() {
// stream_type 4 = IG, 7 = DV EL — both must not surface.
let mut ig = pg_entry(0x1400, "eng");
ig.stream_type = 4;
let mut dv = audio_entry(0x1011, 0x24, 0, 0, "");
dv.stream_type = 7;
let pl = playlist_with(vec![ig, dv]);
let labels = labels_from_playlists(&[pl]);
assert!(labels.is_empty());
}
#[test]
fn secondary_audio_becomes_audio_label() {
// stream_type 5 = secondary audio. The conversion should
// still produce an Audio label (the registry's apply path
// can ignore secondary if it wants — this module just
// surfaces what's there).
let mut sec = audio_entry(0x1A00, 0x83, 3, 1, "eng");
sec.stream_type = 5;
sec.secondary = true;
let pl = playlist_with(vec![sec]);
let labels = labels_from_playlists(&[pl]);
assert_eq!(labels.len(), 1);
assert_eq!(labels[0].stream_type, StreamLabelType::Audio);
assert_eq!(labels[0].codec_hint, "TrueHD 2.0");
}
}
+20 -31
View File
@@ -12,15 +12,15 @@
//! sub_com1_idx="23,24,25" />
//! ```
use super::{LabelPurpose, LabelQualifier, StreamLabel, StreamLabelType};
use crate::sector::SectorReader;
use super::{LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, xml};
use crate::sector::SectorSource;
use crate::udf::UdfFs;
pub fn detect(udf: &UdfFs) -> bool {
super::jar_file_exists(udf, "playlists.xml")
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
let data = super::read_jar_file(reader, udf, "playlists.xml")?;
let text = std::str::from_utf8(&data).ok()?;
@@ -30,8 +30,8 @@ pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLab
let mut labels = Vec::new();
// Parse audio streams
if let Some(aud) = extract_attr(&feature, "aud") {
let com_idx = extract_attr(&feature, "aud_com1_idx").and_then(|s| s.parse::<usize>().ok());
if let Some(aud) = xml::attr(&feature, "aud") {
let com_idx = xml::attr(&feature, "aud_com1_idx").and_then(|s| s.parse::<usize>().ok());
for (i, lang) in aud.split(',').enumerate() {
let lang = lang.trim();
@@ -57,12 +57,12 @@ pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLab
}
// Parse subtitle streams
if let Some(sub) = extract_attr(&feature, "sub") {
let forced: Vec<bool> = extract_attr(&feature, "forced_sub")
if let Some(sub) = xml::attr(&feature, "sub") {
let forced: Vec<bool> = xml::attr(&feature, "forced_sub")
.map(|s| s.split(',').map(|f| f.trim() == "1").collect())
.unwrap_or_default();
let com_indices: Vec<usize> = extract_attr(&feature, "sub_com1_idx")
let com_indices: Vec<usize> = xml::attr(&feature, "sub_com1_idx")
.map(|s| s.split(',').filter_map(|i| i.trim().parse().ok()).collect())
.unwrap_or_default();
@@ -100,32 +100,29 @@ pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLab
if labels.is_empty() {
return None;
}
Some(labels)
// High confidence: paramount's playlists.xml is fully structured
// and we extract every documented field.
Some(ParseResult::high(labels))
}
/// Find the feature playlist element (the one with the most audio tracks).
fn find_feature_playlist(xml: &str) -> Option<String> {
fn find_feature_playlist(text: &str) -> Option<String> {
let mut best: Option<String> = None;
let mut best_aud_count = 0;
let mut from = 0;
let mut pos = 0;
while let Some(start) = xml[pos..].find("<playlist ") {
let abs_start = pos + start;
let end = match xml[abs_start..].find("/>") {
Some(p) => abs_start + p + 2,
None => break,
};
let element = &xml[abs_start..end];
while let Some((start, end)) = xml::find_element(text, "playlist", from) {
let element = &text[start..end];
// Prefer name="Feature" explicitly
if let Some(name) = extract_attr(element, "name") {
// Prefer name="Feature" explicitly.
if let Some(name) = xml::attr(element, "name") {
if name.eq_ignore_ascii_case("Feature") {
return Some(element.to_string());
}
}
// Otherwise pick the one with the most audio streams
if let Some(aud) = extract_attr(element, "aud") {
// Otherwise pick the one with the most audio streams.
if let Some(aud) = xml::attr(element, "aud") {
let count = aud.split(',').count();
if count > best_aud_count {
best_aud_count = count;
@@ -133,15 +130,7 @@ fn find_feature_playlist(xml: &str) -> Option<String> {
}
}
pos = end;
from = end;
}
best
}
/// Extract an XML attribute value from an element string.
fn extract_attr(element: &str, name: &str) -> Option<String> {
let needle = format!("{name}=\"");
let start = element.find(&needle)? + needle.len();
let end = element[start..].find('"')? + start;
Some(element[start..end].to_string())
}
+102 -23
View File
@@ -5,9 +5,13 @@
//!
//! Token format: `{lang}_{codec?}_{purpose?}_{region?}_`
use super::{LabelPurpose, LabelQualifier, StreamLabel, StreamLabelType, vocab};
use crate::sector::SectorReader;
use super::{
Confidence, LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, text,
vocab,
};
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::sync::atomic::{AtomicBool, Ordering};
/// Known audio codec tokens
const AUDIO_CODECS: &[&str] = &["MLP", "AC3", "DTS", "DDL", "WAV", "AC"];
@@ -20,9 +24,18 @@ pub fn detect(udf: &UdfFs) -> bool {
super::jar_file_exists(udf, "bluray_project.bin")
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
let data = super::read_jar_file(reader, udf, "bluray_project.bin")?;
let strings = extract_strings(&data);
// min_len=4 matches the prior local extract_strings impl. The token
// grammar is `{lang3}_{codec?}_{purpose?}_{region?}_` so the
// shortest meaningful run is 4 chars (lang + underscore).
let strings = text::extract_ascii_strings(&data, 4);
// Tracked across all parse_token calls in this run: did any stream
// hit an unrecognized token component (skip-unknown path)? If yes
// we downgrade confidence to Medium — the labels are still valid
// but the corpus surfaced something we don't catalogue.
let saw_unknown = AtomicBool::new(false);
let mut labels = Vec::new();
let mut in_feature = false;
@@ -50,7 +63,7 @@ pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLab
continue;
}
if let Some(label) = parse_token(s) {
if let Some(label) = parse_token_inner(s, Some(&saw_unknown)) {
match label.stream_type {
StreamLabelType::Audio => {
audio_num += 1;
@@ -73,10 +86,15 @@ pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLab
if labels.is_empty() {
return None;
}
Some(labels)
let confidence = if saw_unknown.load(Ordering::Relaxed) {
Confidence::Medium
} else {
Confidence::High
};
Some(ParseResult { labels, confidence })
}
fn parse_token(s: &str) -> Option<StreamLabel> {
fn parse_token_inner(s: &str, saw_unknown: Option<&AtomicBool>) -> Option<StreamLabel> {
let clean = s.trim().trim_start_matches('\t').trim_end_matches('_');
let parts: Vec<&str> = clean.split('_').collect();
if parts.len() < 2 {
@@ -127,7 +145,18 @@ fn parse_token(s: &str) -> Option<StreamLabel> {
} else if part.starts_with("PGStream") {
is_subtitle = true;
} else {
return None;
// Unknown token component — skip this single part rather
// than discarding the entire stream record. Pre-refactor
// behavior was `return None` here, which silently dropped
// any stream containing a single uncatalogued token (e.g.
// a new codec ID or framework variant). Better to surface
// what we know than discard a whole stream over one part,
// but flag the parse as Medium-confidence so callers know
// some data was elided.
tracing::debug!(part = %part, "pixelogic: unrecognized token component, skipping");
if let Some(flag) = saw_unknown {
flag.store(true, Ordering::Relaxed);
}
}
}
@@ -153,22 +182,72 @@ fn parse_token(s: &str) -> Option<StreamLabel> {
})
}
fn extract_strings(data: &[u8]) -> Vec<String> {
let mut strings = Vec::new();
let mut current = String::new();
// extract_strings removed — replaced by super::text::extract_ascii_strings(data, 4).
for &b in data {
if (0x20..0x7f).contains(&b) {
current.push(b as char);
} else {
if current.len() > 3 {
strings.push(current.clone());
}
current.clear();
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_token_basic_audio() {
let l = parse_token_inner("eng_MLP_", None).unwrap();
assert_eq!(l.stream_type, StreamLabelType::Audio);
assert_eq!(l.language, "eng");
assert_eq!(l.codec_hint, "TrueHD");
assert_eq!(l.purpose, LabelPurpose::Normal);
}
if current.len() > 3 {
strings.push(current);
#[test]
fn parse_token_basic_subtitle_sdh() {
let l = parse_token_inner("eng_SDH_", None).unwrap();
assert_eq!(l.stream_type, StreamLabelType::Subtitle);
assert_eq!(l.language, "eng");
assert_eq!(l.qualifier, LabelQualifier::Sdh);
}
#[test]
fn parse_token_commentary() {
let l = parse_token_inner("eng_MLP_ACOM_", None).unwrap();
assert_eq!(l.stream_type, StreamLabelType::Audio);
assert_eq!(l.purpose, LabelPurpose::Commentary);
}
#[test]
fn parse_token_descriptive() {
let l = parse_token_inner("eng_AC3_ADES_", None).unwrap();
assert_eq!(l.purpose, LabelPurpose::Descriptive);
}
#[test]
fn parse_token_with_region() {
let l = parse_token_inner("eng_MLP_US_", None).unwrap();
assert_eq!(l.language, "eng");
assert_eq!(l.variant, "US");
}
#[test]
fn parse_token_unknown_component_does_not_kill_stream() {
// Regression: pre-refactor, an unrecognized token part returned
// None for the whole stream, silently dropping it. New
// behavior: skip the unknown part, surface what we know.
let l = parse_token_inner("eng_MLP_FUTUREFLAG_FOR_", None).unwrap();
assert_eq!(l.stream_type, StreamLabelType::Audio);
assert_eq!(l.language, "eng");
assert_eq!(l.codec_hint, "TrueHD");
assert_eq!(l.qualifier, LabelQualifier::Forced);
}
#[test]
fn parse_token_no_audio_or_subtitle_signal_returns_none() {
// A token that has only a language and an unknown part with
// no audio/subtitle classifier should still return None —
// there's no way to file it as a stream.
assert!(parse_token_inner("eng_UNKNOWN_", None).is_none());
}
#[test]
fn parse_token_rejects_non_lang_prefix() {
assert!(parse_token_inner("XX_MLP_", None).is_none());
assert!(parse_token_inner("ENG_MLP_", None).is_none()); // uppercase not accepted as ISO 639-2
}
strings
}
+89
View File
@@ -0,0 +1,89 @@
//! Text-extraction helpers used by parsers that scan binary blobs for
//! embedded label strings.
//!
//! Promoted from two near-duplicate implementations:
//! - `pixelogic::extract_strings` (`bluray_project.bin`, min_len=4)
//! - `dbp::extract_printable` (`.class` files in jars, min_len=5)
//!
//! Single implementation, threshold passed in. Callers that have a
//! more structured parse path (e.g. `class_reader` for .class) should
//! prefer that — this helper is for genuinely unstructured input.
/// Walk `data`, emit every maximal run of printable-ASCII bytes
/// (`0x20..=0x7E`) whose length is at least `min_len`.
///
/// Non-printable bytes (including `\t`, `\n`, NUL) terminate the
/// current run. Output strings are guaranteed valid UTF-8 (they're
/// pure 7-bit ASCII). Strings shorter than `min_len` are dropped.
pub fn extract_ascii_strings(data: &[u8], min_len: usize) -> Vec<String> {
let mut out = Vec::new();
let mut current = String::new();
for &b in data {
if (0x20..=0x7E).contains(&b) {
current.push(b as char);
} else if current.len() >= min_len {
out.push(std::mem::take(&mut current));
} else {
current.clear();
}
}
if current.len() >= min_len {
out.push(current);
}
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extracts_simple_runs() {
let got = extract_ascii_strings(b"hello\0world\0", 3);
assert_eq!(got, vec!["hello", "world"]);
}
#[test]
fn applies_minimum_length() {
let got = extract_ascii_strings(b"hi\0ok\0longer\0", 5);
assert_eq!(got, vec!["longer"]);
}
#[test]
fn treats_tab_and_newline_as_separators() {
// \t (0x09) and \n (0x0A) are below 0x20, so they break runs.
let got = extract_ascii_strings(b"alpha\tbeta\ngamma", 3);
assert_eq!(got, vec!["alpha", "beta", "gamma"]);
}
#[test]
fn handles_trailing_run_without_terminator() {
// Run at the very end of the buffer should still be emitted.
let got = extract_ascii_strings(b"prefix\0tail", 3);
assert_eq!(got, vec!["prefix", "tail"]);
}
#[test]
fn rejects_high_bit_bytes() {
// 0x80+ is non-printable per this helper's definition.
let mut buf = b"good".to_vec();
buf.push(0xC3);
buf.push(0xA9);
buf.extend_from_slice(b"more");
let got = extract_ascii_strings(&buf, 3);
assert_eq!(got, vec!["good", "more"]);
}
#[test]
fn empty_input_returns_empty() {
let got = extract_ascii_strings(&[], 1);
assert!(got.is_empty());
}
#[test]
fn min_len_zero_emits_singletons() {
// Pathological but well-defined.
let got = extract_ascii_strings(b"a\0b", 0);
assert_eq!(got, vec!["a", "b"]);
}
}
+432 -9
View File
@@ -1,19 +1,40 @@
//! Shared label vocabulary — values we are 100% confident about.
//! Shared label vocabulary — canonical mappings used by ≥1 label parser.
//!
//! Labels come from BD-J authoring tool files (bluray_project.bin,
//! playlists.xml, menu_base.prop, etc.) — NOT from BD spec fields.
//! This is NOT for MPLS/CLPI/STN data. Those follow the BD spec directly.
//! playlists.xml, menu_base.prop, .class string pools, etc.) — NOT
//! from BD spec fields. This module is the central, regression-tested
//! source of truth for:
//!
//! Rules:
//! - Only map values we are 100% certain about (published codec names).
//! - Unknown codes (csp, eda, cf, etc.) pass through raw from disc.
//! - The app/CLI handles display text, not the lib.
//! - Codec brand name aliases (`MLP` → `TrueHD`).
//! - English / multi-word language name → ISO 639-2 code.
//! - English text → [`LabelPurpose`] (Commentary / Descriptive / etc.).
//! - English text → [`LabelQualifier`] (SDH / Forced / Descriptive Service).
//!
//! Rules of engagement (carried over from
//! `(internal)/memory/feedback_label_data_rules.md`):
//!
//! 1. Only map values we are 100% certain about — published codec
//! names, well-known ISO 639-2 mappings, vendor-documented purpose
//! keywords.
//! 2. Unknown codes / unrecognized phrases pass through raw or return
//! `None`. We never guess.
//! 3. Matching is case-insensitive and word-boundary-aware where
//! relevant (so "Commenter" doesn't match "commentary"). Anchoring
//! on whole tokens is the responsibility of this module — callers
//! pass raw text, we handle it.
//!
//! This module is NOT for BD spec STN codec IDs; those decode in
//! `mpls.rs` separately.
use super::{LabelPurpose, LabelQualifier};
// ── Codec aliases ────────────────────────────────────────────────────────────
/// Map a codec identifier found in label data to its display name.
///
/// These are well-known codec identifiers used across multiple BD-J
/// authoring tools. NOT BD spec STN codec IDs — those are decoded
/// separately in mpls.rs.
/// authoring tools. Unknown codes pass through unchanged so callers
/// can still surface vendor-specific tokens we haven't catalogued.
pub fn codec(code: &str) -> &str {
match code {
"MLP" => "TrueHD",
@@ -25,3 +46,405 @@ pub fn codec(code: &str) -> &str {
_ => code,
}
}
// ── Language: English / multi-word names → ISO 639-2 ─────────────────────────
/// Result of [`lang`] — ISO code + human-readable regional variant.
///
/// `code` is ISO 639-2 (always 3 lowercase letters).
/// `variant` is the regional dialect as a human-readable English word
/// (`"Brazilian"`, `"Castilian"`, `"Canadian"`, `"Simplified"`, ...)
/// or `""` when the input names just a bare language without
/// dialect ("Spanish" → variant=""). The variant matches the
/// convention pixelogic / ctrm / criterion already use for their
/// `StreamLabel::variant` field: a short display token the UI can
/// surface verbatim.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct LangInfo {
pub code: &'static str,
pub variant: &'static str,
}
/// Map a free-form language label fragment to an ISO 639-2 code AND
/// (where applicable) its regional variant.
///
/// Handles both bare English names ("English", "Spanish") and the
/// multi-word vendor variants we've seen in the corpus ("Brazilian
/// Portuguese", "Castilian Spanish", "Canadian French"). Match is
/// case-insensitive; longer compound phrases win over their bare
/// counterparts (so "Brazilian Portuguese" returns
/// `LangInfo { code: "por", variant: "Brazilian" }`, not consumed by
/// the bare "Portuguese" entry).
///
/// Bare-name matches return `variant: ""`.
///
/// Returns `None` for unrecognized input — callers decide whether to
/// fall back to MPLS spec codes, pass through raw, or drop the stream.
/// Never guesses.
///
/// Why the variant: the prior `lang() -> Option<&str>` shape silently
/// dropped regional dialect info. "Brazilian Portuguese 5.1" became
/// `language="por", variant=""` — UI displayed plain "Portuguese"
/// even though the disc had explicitly labeled this stream Brazilian.
/// Capturing the variant here parallels how pixelogic and ctrm
/// populate `StreamLabel::variant` from their own region tables.
pub fn lang(text: &str) -> Option<LangInfo> {
let lower = text.to_lowercase();
// Multi-word compounds first — longest-match wins.
for (needle, code, variant) in COMPOUND_LANGS {
if lower.contains(needle) {
return Some(LangInfo { code, variant });
}
}
// Bare names: word-boundary match (avoid "english" inside "englishman"
// or any other accidental substring).
for (needle, code) in BARE_LANGS {
if has_word(&lower, needle) {
return Some(LangInfo { code, variant: "" });
}
}
None
}
const COMPOUND_LANGS: &[(&str, &str, &str)] = &[
("brazilian portuguese", "por", "Brazilian"),
("euro portuguese", "por", "European"),
("european portuguese", "por", "European"),
("castilian spanish", "spa", "Castilian"),
("latin american spanish", "spa", "Latin American"),
("latin spanish", "spa", "Latin American"),
("canadian french", "fra", "Canadian"),
("parisian french", "fra", "Parisian"),
("australian english", "eng", "Australian"),
("austrailian english", "eng", "Australian"), // disc-corpus typo, keep matching
("british english", "eng", "British"),
("simplified chinese", "zho", "Simplified"),
("traditional chinese", "zho", "Traditional"),
("mandarin chinese", "zho", "Mandarin"),
("cantonese chinese", "zho", "Cantonese"),
];
const BARE_LANGS: &[(&str, &str)] = &[
("english", "eng"),
("spanish", "spa"),
("french", "fra"),
("german", "deu"),
("italian", "ita"),
("japanese", "jpn"),
("chinese", "zho"),
("mandarin", "zho"),
("cantonese", "zho"),
("portuguese", "por"),
("polish", "pol"),
("czech", "ces"),
("hungarian", "hun"),
("dutch", "nld"),
("korean", "kor"),
("arabic", "ara"),
("hindi", "hin"),
("turkish", "tur"),
("thai", "tha"),
("swedish", "swe"),
("norwegian", "nor"),
("danish", "dan"),
("finnish", "fin"),
("hebrew", "heb"),
("russian", "rus"),
("greek", "ell"),
("vietnamese", "vie"),
("indonesian", "ind"),
("malay", "msa"),
("ukrainian", "ukr"),
("romanian", "ron"),
("bulgarian", "bul"),
("croatian", "hrv"),
("serbian", "srp"),
("slovak", "slk"),
("slovenian", "slv"),
("estonian", "est"),
("latvian", "lav"),
("lithuanian", "lit"),
("icelandic", "isl"),
("basque", "eus"),
("catalan", "cat"),
("galician", "glg"),
];
// ── Purpose ──────────────────────────────────────────────────────────────────
/// Classify a free-form English label string into a [`LabelPurpose`].
///
/// Recognized keywords (case-insensitive, word-boundary matched):
/// - "commentary", "director's commentary" → `Commentary`
/// - "descriptive", "description", "audio description", "described" → `Descriptive`
/// - "score", "music only" → `Score`
/// - "ime" (alternate music for closing themes etc.) → `Ime`
/// - anything else → `Normal`
///
/// Word-boundary matching means "Commentary track" matches but
/// "Commenter Pro audio" does not.
pub fn purpose(text: &str) -> LabelPurpose {
let lower = text.to_lowercase();
// Multi-word compounds first — they're more specific.
if lower.contains("audio description") || lower.contains("descriptive service") {
return LabelPurpose::Descriptive;
}
if lower.contains("music only") {
return LabelPurpose::Score;
}
if has_word(&lower, "commentary") {
return LabelPurpose::Commentary;
}
if has_word(&lower, "descriptive")
|| has_word(&lower, "description")
|| has_word(&lower, "described")
{
return LabelPurpose::Descriptive;
}
if has_word(&lower, "score") {
return LabelPurpose::Score;
}
if has_word(&lower, "ime") {
return LabelPurpose::Ime;
}
LabelPurpose::Normal
}
// ── Qualifier ────────────────────────────────────────────────────────────────
/// Classify a free-form English label string into a [`LabelQualifier`].
///
/// Recognized keywords (case-insensitive, word-boundary matched):
/// - "sdh", "captions" → `Sdh`
/// - "forced", "forced narrative" → `Forced`
/// - "rnib", "descriptive service" → `DescriptiveService`
/// - anything else → `None`
///
/// SDH (Subtitles for the Deaf and Hard of hearing) wins over Forced
/// when both keywords are present, because an SDH track is its own
/// stream regardless of whether the player flags it as "forced".
pub fn qualifier(text: &str) -> LabelQualifier {
let lower = text.to_lowercase();
if has_word(&lower, "sdh") || has_word(&lower, "captions") {
return LabelQualifier::Sdh;
}
if lower.contains("descriptive service") || has_word(&lower, "rnib") {
return LabelQualifier::DescriptiveService;
}
if has_word(&lower, "forced") {
return LabelQualifier::Forced;
}
LabelQualifier::None
}
// ── Internal: word-boundary matching ────────────────────────────────────────
/// True if `needle` appears in `haystack` surrounded by non-alphanumeric
/// boundaries (or string ends). `haystack` is assumed lowercase already.
///
/// This is the load-bearing primitive for `lang` / `purpose` /
/// `qualifier`: bare-token matchers MUST use it, otherwise we match
/// "english" inside "englishman" and "sdh" inside "lambdash". The
/// existing parsers used `.contains()` and got lucky on the corpus;
/// vocab guarantees the boundary.
fn has_word(haystack: &str, needle: &str) -> bool {
if needle.is_empty() {
return false;
}
let bytes = haystack.as_bytes();
let nb = needle.as_bytes();
let mut i = 0;
while i + nb.len() <= bytes.len() {
if &bytes[i..i + nb.len()] == nb {
let before = if i == 0 { None } else { Some(bytes[i - 1]) };
let after = bytes.get(i + nb.len()).copied();
let bound = |c: Option<u8>| match c {
None => true,
Some(b) => !b.is_ascii_alphanumeric(),
};
if bound(before) && bound(after) {
return true;
}
}
i += 1;
}
false
}
// ── Tests ────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn codec_known_aliases() {
assert_eq!(codec("MLP"), "TrueHD");
assert_eq!(codec("AC3"), "Dolby Digital");
assert_eq!(codec("AC"), "Dolby Digital");
assert_eq!(codec("DDL"), "Dolby Digital Plus");
assert_eq!(codec("atmos"), "Dolby Atmos");
}
#[test]
fn codec_unknown_passes_through() {
assert_eq!(codec("FX9"), "FX9");
assert_eq!(codec(""), "");
}
fn li(code: &'static str, variant: &'static str) -> LangInfo {
LangInfo { code, variant }
}
#[test]
fn lang_bare_names_have_empty_variant() {
assert_eq!(lang("English"), Some(li("eng", "")));
assert_eq!(lang("english"), Some(li("eng", "")));
assert_eq!(lang("Spanish 5.1 Dolby Digital"), Some(li("spa", "")));
assert_eq!(lang("japanese"), Some(li("jpn", "")));
assert_eq!(lang("Italian"), Some(li("ita", "")));
}
#[test]
fn lang_compounds_carry_variant() {
assert_eq!(
lang("Brazilian Portuguese 5.1"),
Some(li("por", "Brazilian"))
);
assert_eq!(lang("Castilian Spanish"), Some(li("spa", "Castilian")));
assert_eq!(
lang("Canadian French Dolby Digital"),
Some(li("fra", "Canadian"))
);
assert_eq!(
lang("Latin American Spanish"),
Some(li("spa", "Latin American"))
);
assert_eq!(lang("Simplified Chinese"), Some(li("zho", "Simplified")));
assert_eq!(lang("British English"), Some(li("eng", "British")));
}
#[test]
fn lang_compounds_win_over_bare() {
// Brazilian Portuguese must map to (por, Brazilian) via the
// compound rule, not be intercepted by bare "portuguese"
// (which would yield (por, "") and lose the variant).
assert_eq!(lang("Brazilian Portuguese").unwrap().variant, "Brazilian");
assert_eq!(lang("Canadian French").unwrap().variant, "Canadian");
}
#[test]
fn lang_unknown_returns_none() {
assert_eq!(lang("Klingon Dolby Atmos"), None);
assert_eq!(lang(""), None);
assert_eq!(lang("eng"), None); // 3-letter codes are not English names
}
#[test]
fn lang_word_boundary_avoids_substring_false_positive() {
// No false positive — "engineering" must NOT match "english".
assert_eq!(lang("Audio Engineering Demo"), None);
}
#[test]
fn purpose_recognizes_commentary() {
assert_eq!(purpose("English Commentary"), LabelPurpose::Commentary);
assert_eq!(purpose("Director's Commentary"), LabelPurpose::Commentary);
}
#[test]
fn purpose_recognizes_descriptive() {
assert_eq!(
purpose("English Descriptive Audio"),
LabelPurpose::Descriptive
);
assert_eq!(purpose("Audio Description"), LabelPurpose::Descriptive);
assert_eq!(purpose("Described Video"), LabelPurpose::Descriptive);
}
#[test]
fn purpose_descriptive_service_routes_to_descriptive() {
// "Descriptive Service" is qualifier territory but the purpose
// implication is Descriptive — vocab::purpose treats it as such.
assert_eq!(
purpose("English Descriptive Service"),
LabelPurpose::Descriptive
);
}
#[test]
fn purpose_word_boundary_avoids_commenter_false_positive() {
// "Commenter Pro audio" does NOT match "commentary" — the
// existing dbp/ctrm hand-rolls would have. Vocab is stricter.
assert_eq!(purpose("Commenter Pro audio track"), LabelPurpose::Normal);
}
#[test]
fn purpose_recognizes_score() {
assert_eq!(purpose("Music Only"), LabelPurpose::Score);
assert_eq!(purpose("Isolated Score"), LabelPurpose::Score);
}
#[test]
fn purpose_unknown_is_normal() {
assert_eq!(purpose("English Dolby Atmos"), LabelPurpose::Normal);
assert_eq!(purpose(""), LabelPurpose::Normal);
}
#[test]
fn qualifier_recognizes_sdh() {
assert_eq!(qualifier("English SDH"), LabelQualifier::Sdh);
assert_eq!(qualifier("English Captions"), LabelQualifier::Sdh);
}
#[test]
fn qualifier_recognizes_forced() {
assert_eq!(qualifier("English Forced"), LabelQualifier::Forced);
assert_eq!(qualifier("Forced Narrative"), LabelQualifier::Forced);
}
#[test]
fn qualifier_recognizes_descriptive_service() {
assert_eq!(
qualifier("English RNIB"),
LabelQualifier::DescriptiveService
);
assert_eq!(
qualifier("English Descriptive Service"),
LabelQualifier::DescriptiveService
);
}
#[test]
fn qualifier_sdh_wins_over_forced_when_both_present() {
// SDH track is its own stream regardless of forced flag.
assert_eq!(qualifier("English Forced SDH"), LabelQualifier::Sdh);
}
#[test]
fn qualifier_unknown_is_none() {
assert_eq!(qualifier("English"), LabelQualifier::None);
assert_eq!(qualifier(""), LabelQualifier::None);
}
#[test]
fn has_word_basic() {
assert!(has_word("english forced", "english"));
assert!(has_word("english forced", "forced"));
assert!(has_word("english", "english"));
}
#[test]
fn has_word_rejects_substring() {
assert!(!has_word("engineering", "english"));
assert!(!has_word("englishman", "english"));
assert!(!has_word("aenglish", "english"));
}
#[test]
fn has_word_punctuation_boundary() {
// "(SDH)" is a valid boundary — parentheses count as non-alphanum.
assert!(has_word("english (sdh)", "sdh"));
assert!(has_word("commentary,extra,info", "commentary"));
}
}
+452
View File
@@ -0,0 +1,452 @@
//! Tolerant XML scraping helpers — promoted from two near-duplicate
//! hand-rolls in `paramount.rs` (attribute extraction) and `criterion.rs`
//! (tag-text extraction).
//!
//! These are NOT a full XML parser. They handle the subset of XML the
//! BD-J authoring tools we've seen actually emit: ASCII tag/attr
//! names, no entity references inside label strings, optional XML
//! namespaces. Hardening goals over the prior `find("<tag>")` /
//! `find(r#"name=""#)` matchers:
//!
//! 1. **Case-insensitive** tag and attribute names — vendors casing
//! is inconsistent across authoring-tool revisions.
//! 2. **Namespace-aware** — strip an optional `ns:` prefix so
//! `<ns:playlist>` and `<playlist>` both match.
//! 3. **Whitespace-tolerant** — multiple/tab/newline characters
//! around `=` between attribute name and value; whitespace inside
//! the opening tag.
//! 4. **Quote-style tolerant** — both `"value"` and `'value'`.
//! 5. **Self-closing tag handling** — `<tag />` and `<tag/>` both
//! work; [`text`] returns `Some("")` for empty content.
//!
//! Out of scope (intentionally simple): XML entity decoding
//! (`&amp;`, `&lt;`, etc.), CDATA sections, comments, processing
//! instructions, DTD declarations. None of the BD-J authored disc
//! data we've observed exercises any of those — labels are plain
//! ASCII/Latin-1 in attribute values.
/// Extract the value of attribute `name` from one XML element
/// fragment (e.g. `<playlist name="Feature" id="00222" />`).
///
/// Returns the raw attribute text (no entity decoding) or `None` if
/// the attribute isn't present. Empty string for `name=""` is
/// represented as `Some("")`.
pub fn attr(element: &str, name: &str) -> Option<String> {
let bytes = element.as_bytes();
let name_lower = name.to_ascii_lowercase();
let name_bytes = name_lower.as_bytes();
let mut i = 0;
while i + name_bytes.len() < bytes.len() {
// Find the next position where `name=` could start. We need
// a word boundary before the name (whitespace or `<` or `:`).
if i > 0 && is_name_char(bytes[i - 1]) {
i += 1;
continue;
}
if !slice_eq_ignore_case(&bytes[i..i + name_bytes.len()], name_bytes) {
i += 1;
continue;
}
let after_name = i + name_bytes.len();
// The character immediately after the name must not be a
// name-continuation (otherwise we matched a prefix like
// `lang_id` when looking for `lang`).
if after_name < bytes.len() && is_name_char(bytes[after_name]) {
i = after_name;
continue;
}
// Walk past whitespace, then `=`, then more whitespace, then
// the opening quote.
let mut j = after_name;
while j < bytes.len() && is_ws(bytes[j]) {
j += 1;
}
if j >= bytes.len() || bytes[j] != b'=' {
i = j.max(i + 1);
continue;
}
j += 1; // past '='
while j < bytes.len() && is_ws(bytes[j]) {
j += 1;
}
if j >= bytes.len() {
return None;
}
let quote = bytes[j];
if quote != b'"' && quote != b'\'' {
// Unquoted attribute values aren't part of well-formed
// XML (HTML5 allows them, XML doesn't). Skip.
i = j;
continue;
}
let value_start = j + 1;
let close = bytes[value_start..].iter().position(|&b| b == quote)?;
let value = &element[value_start..value_start + close];
return Some(value.to_string());
}
None
}
/// Extract the trimmed text content of the first occurrence of
/// `<tag>...</tag>` in `xml`. Returns `None` if the tag isn't found
/// or its opening tag is malformed.
///
/// Whitespace around the inner text is stripped. Self-closing
/// `<tag />` yields `Some("")`. Nested same-name tags are NOT
/// handled — the first close encountered wins (this matches the
/// prior behavior in criterion.rs).
pub fn text(xml: &str, tag: &str) -> Option<String> {
let (open_end, body_start) = find_open_tag(xml, tag, 0)?;
// Self-closing — already consumed in find_open_tag if `/>`.
if open_end == body_start {
// Means find_open_tag returned the same offset twice for
// self-closing form. (Not currently the case in our impl,
// but defensive.)
return Some(String::new());
}
// For self-closing tags, body_start is past `/>` and we have no
// content. Detect by checking the char at body_start - 1 was `/`.
if body_start >= 2 && &xml[body_start - 2..body_start] == "/>" {
return Some(String::new());
}
// Find the matching close tag. Case-insensitive + namespace-aware.
let close_start = find_close_tag(xml, tag, body_start)?;
Some(xml[body_start..close_start].trim().to_string())
}
/// Locate the next `<tag>` opening AND its closing `</tag>` in
/// `xml`, starting at byte offset `from`. Returns `(element_start,
/// element_end)` — `element_start` is the `<` of the opening tag,
/// `element_end` is one past the `>` of the closing tag. Useful for
/// iterating over repeated elements like `<playlist>` blocks in
/// `paramount`.
///
/// Self-closing elements return the same offset for body_end as the
/// element_end (i.e. `element_end - element_start` includes only the
/// `<tag .../>` text).
pub fn find_element(xml: &str, tag: &str, from: usize) -> Option<(usize, usize)> {
let bytes = xml.as_bytes();
let mut i = from;
while i < bytes.len() {
if bytes[i] != b'<' {
i += 1;
continue;
}
// Try matching tag name at i+1 (after `<`).
let after_lt = i + 1;
if !matches_tag_name_at(bytes, after_lt, tag) {
i += 1;
continue;
}
// Found an open tag at offset i. Walk to find the closing `>`
// of the open tag itself.
let mut j = after_lt;
// Skip past the tag name (and optional namespace prefix).
while j < bytes.len() && (is_name_char(bytes[j]) || bytes[j] == b':') {
j += 1;
}
// Walk attributes — track quoting state.
let mut self_closing = false;
while j < bytes.len() {
match bytes[j] {
b'>' => {
j += 1;
break;
}
b'/' if j + 1 < bytes.len() && bytes[j + 1] == b'>' => {
self_closing = true;
j += 2;
break;
}
b'"' | b'\'' => {
let q = bytes[j];
j += 1;
while j < bytes.len() && bytes[j] != q {
j += 1;
}
if j < bytes.len() {
j += 1;
}
}
_ => j += 1,
}
}
if self_closing {
return Some((i, j));
}
// Find matching close. Doesn't handle nested same-name; OK
// for our authoring-tool subset.
let close_start = find_close_tag(xml, tag, j)?;
let close_end = find_byte(bytes, b'>', close_start)? + 1;
return Some((i, close_end));
}
None
}
// ── Internal helpers ───────────────────────────────────────────────────────
/// True if `bytes[start..]` opens a tag named `tag`, allowing an
/// optional `ns:` namespace prefix. Comparison is case-insensitive.
/// The character after the tag name must not be a name-continuation
/// (so `<player>` doesn't match `<play>`).
fn matches_tag_name_at(bytes: &[u8], start: usize, tag: &str) -> bool {
let tag_lower = tag.to_ascii_lowercase();
let tag_bytes = tag_lower.as_bytes();
// Skip optional `prefix:` (one or more name chars + `:`).
let mut name_start = start;
let mut scan = start;
while scan < bytes.len() && is_name_char(bytes[scan]) {
scan += 1;
}
if scan < bytes.len() && bytes[scan] == b':' {
name_start = scan + 1;
}
if name_start + tag_bytes.len() > bytes.len() {
return false;
}
if !slice_eq_ignore_case(&bytes[name_start..name_start + tag_bytes.len()], tag_bytes) {
return false;
}
// Boundary: char after the tag name must be `>`, `/`, whitespace.
let after = name_start + tag_bytes.len();
if after >= bytes.len() {
return false;
}
matches!(bytes[after], b'>' | b'/' | b' ' | b'\t' | b'\n' | b'\r')
}
/// Find the offset of the next `</tag>` (or `</ns:tag>`) in `xml`
/// starting at `from`. Case-insensitive; returns the offset of the
/// `<`. None if not found.
fn find_close_tag(xml: &str, tag: &str, from: usize) -> Option<usize> {
let bytes = xml.as_bytes();
let mut i = from;
while i + 2 < bytes.len() {
if bytes[i] == b'<' && bytes[i + 1] == b'/' {
// Check tag name (with optional namespace).
if matches_tag_name_at(bytes, i + 2, tag) {
return Some(i);
}
}
i += 1;
}
None
}
/// Find the open tag of `<tag>` in `xml` starting at `from`. Returns
/// `(after_open_lt, after_open_gt)` — the offsets are: just past
/// the `<` of the open tag, and just past the `>` of the open tag.
fn find_open_tag(xml: &str, tag: &str, from: usize) -> Option<(usize, usize)> {
let bytes = xml.as_bytes();
let (elem_start, _) = find_element(xml, tag, from)?;
let after_lt = elem_start + 1;
// Find the `>` that ends the open tag (handling quoted attrs).
let mut j = after_lt;
while j < bytes.len() {
match bytes[j] {
b'>' => return Some((after_lt, j + 1)),
b'/' if j + 1 < bytes.len() && bytes[j + 1] == b'>' => {
return Some((after_lt, j + 2));
}
b'"' | b'\'' => {
let q = bytes[j];
j += 1;
while j < bytes.len() && bytes[j] != q {
j += 1;
}
if j < bytes.len() {
j += 1;
}
}
_ => j += 1,
}
}
None
}
fn find_byte(bytes: &[u8], target: u8, from: usize) -> Option<usize> {
bytes[from..]
.iter()
.position(|&b| b == target)
.map(|p| p + from)
}
/// True if `c` can be part of an XML name token (rough). We accept
/// alphanumerics, `_`, `-`, `.`.
fn is_name_char(c: u8) -> bool {
c.is_ascii_alphanumeric() || c == b'_' || c == b'-' || c == b'.'
}
fn is_ws(c: u8) -> bool {
matches!(c, b' ' | b'\t' | b'\n' | b'\r')
}
fn slice_eq_ignore_case(a: &[u8], b_lower: &[u8]) -> bool {
if a.len() != b_lower.len() {
return false;
}
a.iter()
.zip(b_lower.iter())
.all(|(&x, &y)| x.to_ascii_lowercase() == y)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn attr_basic() {
assert_eq!(
attr(r#"<playlist name="Feature" id="00222" />"#, "name"),
Some("Feature".into())
);
assert_eq!(
attr(r#"<playlist name="Feature" id="00222" />"#, "id"),
Some("00222".into())
);
}
#[test]
fn attr_case_insensitive_name() {
assert_eq!(
attr(r#"<playlist Name="Feature" />"#, "name"),
Some("Feature".into())
);
assert_eq!(
attr(r#"<playlist NAME="Feature" />"#, "Name"),
Some("Feature".into())
);
}
#[test]
fn attr_accepts_single_quotes() {
assert_eq!(
attr(r#"<playlist name='Feature' />"#, "name"),
Some("Feature".into())
);
}
#[test]
fn attr_whitespace_around_equals() {
assert_eq!(
attr(r#"<playlist name = "Feature" />"#, "name"),
Some("Feature".into())
);
assert_eq!(
attr("<playlist name\n=\n\"Feature\" />", "name"),
Some("Feature".into())
);
}
#[test]
fn attr_missing_returns_none() {
assert_eq!(attr(r#"<playlist name="X" />"#, "id"), None);
assert_eq!(attr("", "name"), None);
}
#[test]
fn attr_no_substring_false_positive() {
// Looking for "lang" should NOT match "lang_id" or
// "language" because of the name-char boundary check.
assert_eq!(attr(r#"<x lang_id="fra" language="eng" />"#, "lang"), None);
}
#[test]
fn attr_empty_value() {
assert_eq!(attr(r#"<x name="" id="1" />"#, "name"), Some("".into()));
}
#[test]
fn text_basic() {
assert_eq!(text("<x>hello</x>", "x"), Some("hello".into()));
assert_eq!(
text("<x> hello world </x>", "x"),
Some("hello world".into())
);
}
#[test]
fn text_case_insensitive_tag() {
assert_eq!(text("<X>foo</X>", "x"), Some("foo".into()));
assert_eq!(text("<Foo>bar</foo>", "foo"), Some("bar".into()));
}
#[test]
fn text_namespace_prefix() {
assert_eq!(text("<ns:tag>value</ns:tag>", "tag"), Some("value".into()));
assert_eq!(text("<foo:Bar>v</foo:Bar>", "bar"), Some("v".into()));
}
#[test]
fn text_self_closing() {
assert_eq!(text("<x/>", "x"), Some("".into()));
assert_eq!(text("<x />", "x"), Some("".into()));
assert_eq!(text("<x attr=\"y\" />", "x"), Some("".into()));
}
#[test]
fn text_with_attrs() {
assert_eq!(
text(r#"<x id="1" name="y">hello</x>"#, "x"),
Some("hello".into())
);
}
#[test]
fn text_missing_close_returns_none() {
assert_eq!(text("<x>hello", "x"), None);
}
#[test]
fn text_skips_inner_tags_naively() {
// Limitation noted: nested same-name tags aren't handled.
// Different-name nesting works (we just return everything
// between the open and close).
assert_eq!(
text("<x><y>nested</y></x>", "x"),
Some("<y>nested</y>".into())
);
}
#[test]
fn find_element_basic() {
let xml = r#"<x /> <y attr="1">body</y>"#;
let (s, e) = find_element(xml, "y", 0).unwrap();
assert_eq!(&xml[s..e], r#"<y attr="1">body</y>"#);
}
#[test]
fn find_element_self_closing() {
let xml = r#"<x />"#;
let (s, e) = find_element(xml, "x", 0).unwrap();
assert_eq!(&xml[s..e], "<x />");
}
#[test]
fn find_element_handles_quoted_gt_in_attr() {
// A `>` inside a quoted attribute value should not terminate
// the open tag prematurely.
let xml = r#"<x attr="foo>bar">body</x>"#;
let (s, e) = find_element(xml, "x", 0).unwrap();
assert_eq!(&xml[s..e], r#"<x attr="foo>bar">body</x>"#);
}
#[test]
fn find_element_iteration() {
let xml = "<p>a</p><p>b</p><p>c</p>";
let mut positions = Vec::new();
let mut from = 0;
while let Some((s, e)) = find_element(xml, "p", from) {
positions.push(&xml[s..e]);
from = e;
}
assert_eq!(positions, vec!["<p>a</p>", "<p>b</p>", "<p>c</p>"]);
}
#[test]
fn find_element_with_namespace() {
let xml = r#"<root><ns:item id="1" /></root>"#;
let (s, e) = find_element(xml, "item", 0).unwrap();
assert_eq!(&xml[s..e], r#"<ns:item id="1" />"#);
}
}
+52 -8
View File
@@ -79,10 +79,12 @@ pub mod disc;
pub mod drive;
pub mod error;
pub mod event;
pub mod halt;
pub(crate) mod identity;
pub(crate) mod ifo;
pub mod io;
pub mod keydb;
pub(crate) mod labels;
pub mod labels;
pub(crate) mod mpls;
pub mod mux;
pub mod pes;
@@ -95,6 +97,9 @@ pub(crate) mod speed;
pub(crate) mod udf;
pub mod verify;
// Re-export verify types at the crate root for ergonomic imports.
pub use verify::{SectorRange, SectorStatus, VerifyResult, verify_title};
// ─── Drive lifecycle ────────────────────────────────────────────────────────
//
// `Drive::open(path)` → `wait_ready()` → `init()` → `Disc::scan()`. `Drive`
@@ -112,6 +117,31 @@ pub use drive::{Drive, DriveStatus, find_drive};
// codes to localized messages. See `error.rs` for the full taxonomy.
pub use error::{Error, Result};
// ─── 0.18 primitives ────────────────────────────────────────────────────────
//
// One-bit cooperative cancellation token, shared by every long-running loop
// in libfreemkv (sweep, patch, mux). Replaces the ad-hoc `Arc<AtomicBool>`
// flags scattered across 0.17 (`DiscStream::set_halt`, autorip's
// `HALT_FLAGS` registry). Clone it cheaply; pass it by value into each
// component; poll `is_cancelled()` inside the loop body.
pub use halt::Halt;
// Generic bounded producer/consumer primitive used by sweep, patch, and
// mux to overlap reads with writes via a dedicated consumer thread.
// `Pipeline::spawn(name, depth, sink)` spawns a named consumer; `pipe.send(item)`
// pushes one item with back-pressure; `pipe.finish()` joins the
// consumer and surfaces its `close()` output. Callers implement `Sink`
// to define per-item behaviour and end-of-stream finalisation.
//
// `DEFAULT_PIPELINE_DEPTH` (=4) is for callers without specific needs;
// most should use READ_PIPELINE_DEPTH or WRITE_PIPELINE_DEPTH instead.
// Patch uses `WRITE_THROUGH_DEPTH` (=1). Returning `Flow::Stop` from
// `apply` ends the consumer cleanly (still calls `close()`).
pub use io::pipeline::{
DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, READ_PIPELINE_DEPTH, Sink, WRITE_PIPELINE_DEPTH,
WRITE_THROUGH_DEPTH,
};
// ─── Drive events (low-level callbacks) ─────────────────────────────────────
pub use event::{Event, EventKind};
pub use identity::DriveId;
@@ -133,11 +163,15 @@ pub use decrypt::{DecryptKeys, decrypt_sectors};
// for displaying disc name + format quickly while a full scan runs in the
// background. The codec / channel / resolution enums are the canonical
// structured representation; never compare against display strings.
// Note: `disc::Stream` here is the codec enum (audio / video / sub kind)
// — not the `pes::Stream` trait re-exported below as `PesStream`. Two
// different concepts, the same short name; the trait gets the `Pes`
// prefix at the crate root to keep both addressable.
pub use disc::{
AacsState, AudioChannels, AudioStream, Clip, Codec, ColorSpace, ContentFormat, DamageSeverity,
Disc, DiscFormat, DiscId, DiscTitle, Extent, FrameRate, HdrFormat, KeySource, LabelPurpose,
LabelQualifier, Resolution, SampleRate, ScanOptions, Stream, SubtitleStream, VideoStream,
classify_damage,
LabelQualifier, PatchOptions, PatchOutcome, Resolution, SampleRate, ScanOptions, Stream,
SubtitleStream, SweepOptions, VideoStream, classify_damage,
};
// ─── Streams ────────────────────────────────────────────────────────────────
@@ -145,7 +179,7 @@ pub use disc::{
// All stream types implement `pes::Stream` — read PES frames from a source,
// write PES frames to a sink. Pick the right type at construction:
//
// - `DiscStream` — physical drive or ISO (any `SectorReader`). Always read.
// - `DiscStream` — physical drive or ISO (any `SectorSource`). Read-only.
// - `MkvStream` — Matroska container. Read on `open()`, write on `create()`.
// - `M2tsStream` — Blu-ray Transport Stream. Read on `open()`, write on `create()`.
// - `NetworkStream` — TCP. Read on `listen()`, write on `connect()`.
@@ -155,6 +189,12 @@ pub use disc::{
// Most consumers use the URL resolvers (`input()` / `output()`) which pick
// the right type from a scheme:// URL. Direct construction is for callers
// that need to wire custom readers (e.g. autorip's drive-session reuse).
// The trait is re-exported as `PesStream` here to disambiguate from
// `disc::Stream` (the codec-kind enum re-exported above), which would
// otherwise collide at the crate root.
pub use pes::PesFrame;
pub use pes::Stream as PesStream;
pub use mux::DiscStream;
pub use mux::M2tsStream;
pub use mux::MkvStream;
@@ -166,10 +206,14 @@ pub use mux::{InputOptions, StreamUrl, input, output, parse_url};
// ─── Lower-level surfaces ───────────────────────────────────────────────────
//
// `ScsiTransport` is the platform-abstraction trait Drive uses; expose for
// out-of-tree platform backends. `SectorReader` lets callers feed any byte
// source (test harness, network image, SMB share) into the disc scan
// pipeline; `FileSectorReader` is the standard ISO-on-disk implementation.
// out-of-tree platform backends. `SectorSource` / `SectorSink` are the
// direction-typed read/write traits; `FileSectorSource` and `FileSectorSink`
// are the ISO-on-disk implementations. [`DecryptingSectorSource`] is the
// single decrypt-on-read decorator (AACS / CSS / none) — wrap any
// `SectorSource` to get plaintext sectors out.
pub use scsi::{DriveInfo, ScsiSense, ScsiTransport, drive_has_disc, list_drives};
pub use sector::{FileSectorReader, SectorReader};
pub use sector::{
DecryptingSectorSource, FileSectorSink, FileSectorSource, SectorSink, SectorSource,
};
pub use speed::DriveSpeed;
pub use udf::{UdfFs, read_filesystem};
+218 -26
View File
@@ -1,6 +1,6 @@
//! DiscStream — read any disc (physical drive or ISO file) → PES frames.
//!
//! One stream type for all disc sources. The source is a SectorReader —
//! One stream type for all disc sources. The source is a SectorSource —
//! Drive (hardware) or IsoSectorReader (file). DiscStream doesn't care.
//!
//! Read-only. For disc→ISO (raw sector copy), use `Disc::copy()`.
@@ -8,10 +8,11 @@
use crate::disc::{Disc, DiscTitle, Extent};
use crate::drive::extract_scsi_context;
use crate::event::{BatchSizeReason, Event, EventKind};
use crate::sector::SectorReader;
use crate::halt::Halt;
use crate::sector::{DecryptingSectorSource, SectorSource};
use std::io;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::AtomicBool;
/// Ramp back up to the preferred batch size after this many sectors
/// of clean reading at the current (reduced) size. 100 MiB = 51,200 sectors.
@@ -96,12 +97,22 @@ impl AdaptiveBatch {
/// Disc stream. Reads sectors from any source → PES frames.
///
/// Sources: physical drive, ISO file, or any SectorReader.
/// Sources: physical drive, ISO file, or any SectorSource.
/// Decrypt, demux, and codec parsing happen internally.
pub struct DiscStream {
reader: Box<dyn SectorReader>,
/// Underlying sector source wrapped in the 0.18
/// [`DecryptingSectorSource`] decorator. Every `read_sectors`
/// call yields plaintext, so `fill_extents` no longer needs an
/// inline `decrypt::decrypt_sectors` step. `DecryptKeys::None`
/// (raw / unencrypted disc) makes the decorator a pass-through.
reader: DecryptingSectorSource<Box<dyn SectorSource>>,
title: DiscTitle,
disc: Option<Disc>,
/// Mirror of the keys handed in at construction. The decorator
/// owns the cryptographic state; this field is kept for
/// metadata-side callers (`info()` and friends) that want to
/// know whether the disc was encrypted, without reaching through
/// the wrapper.
decrypt_keys: crate::decrypt::DecryptKeys,
// Extents to read
@@ -120,11 +131,13 @@ pub struct DiscStream {
adaptive: AdaptiveBatch,
pub errors: u64,
pub skip_errors: bool,
/// When set and the flag is raised, fill_extents returns Err(Halted) at the
/// next retry boundary. Unlike skip_errors, this propagates the error up so
/// the rip terminates cleanly. Share the Arc with Drive::halt_flag() to get
/// unified Stop behavior across drive reads and sector processing.
halt: Option<Arc<AtomicBool>>,
/// When set and the token is cancelled, fill_extents returns Err(Halted)
/// at the next retry boundary. Unlike skip_errors, this propagates the
/// error up so the rip terminates cleanly. Construct with
/// [`DiscStream::with_halt`] (preferred) or set post-hoc via the
/// deprecated [`DiscStream::set_halt`] bridge — both populate this same
/// field and either entry point yields one source of truth.
halt: Option<Halt>,
event_fn: Option<Box<dyn Fn(Event) + Send>>,
eof: bool,
@@ -147,11 +160,11 @@ pub struct DiscStream {
impl DiscStream {
/// Create a disc stream from any sector reader.
///
/// Works with physical drives and ISO files — both implement SectorReader.
/// Works with physical drives and ISO files — both implement SectorSource.
/// The caller opens the source, scans for titles/keys, and passes them in.
/// The stream handles demuxing, decryption, and codec parsing internally.
pub fn new(
reader: Box<dyn SectorReader>,
reader: Box<dyn SectorSource>,
title: DiscTitle,
decrypt_keys: crate::decrypt::DecryptKeys,
batch_sectors: u16,
@@ -160,6 +173,13 @@ impl DiscStream {
let extents = title.extents.clone();
let bytes_total_extents: u64 = extents.iter().map(|e| e.sector_count as u64 * 2048).sum();
// Debug log reader type at construction — critical for diagnosing mux reading from drive instead of ISO
tracing::debug!(
target: "mux",
"DiscStream constructed with reader type: {}",
std::any::type_name::<dyn SectorSource>()
);
let mut pids = Vec::new();
let mut parsers = Vec::new();
let mut pid_to_track = Vec::new();
@@ -189,7 +209,11 @@ impl DiscStream {
}
Self {
reader,
// Wrap the input reader in DecryptingSectorSource so the
// internal fill_extents path sees plaintext bytes. For
// DecryptKeys::None (unencrypted / raw / test fixtures)
// the decorator is a pass-through.
reader: DecryptingSectorSource::new(reader, decrypt_keys.clone()),
title,
disc: None,
decrypt_keys,
@@ -219,19 +243,39 @@ impl DiscStream {
self.event_fn = Some(Box::new(f));
}
/// Share a halt flag — typically from `Drive::halt_flag()`. When raised,
/// the next read-retry boundary inside fill_extents returns Err(Halted)
/// instead of continuing. Required for Stop to work during dense bad-sector
/// regions (where read() loops internally waiting for enough clean data to
/// emit a frame and would otherwise never check an external stop signal).
/// Constructor-time builder: attach a [`Halt`] token so that when
/// any clone is cancelled, the next read-retry boundary inside
/// `fill_extents` returns `Err(Halted)`. Required for Stop to work
/// during dense bad-sector regions (where the outer PES read() loop
/// can spend minutes inside fill_extents before emitting a frame).
///
/// Preferred over the post-hoc [`DiscStream::set_halt`] bridge —
/// pass the same `Halt` clone you hand to sweep / patch / mux so
/// every phase observes a single Stop signal.
pub fn with_halt(mut self, halt: Halt) -> Self {
self.halt = Some(halt);
self
}
/// Bridge for callers that haven't migrated to the
/// [`DiscStream::with_halt`] constructor-time path yet. Wraps the
/// supplied `Arc<AtomicBool>` as a [`Halt`] (`Halt::from_arc`) and
/// stores it in the same internal slot, so a halt installed via
/// either entry point goes through one halt-check inside
/// `fill_extents`. Calling `set_halt` after `with_halt` (or vice
/// versa) replaces the previous token with the new one.
#[deprecated(
since = "0.18.0",
note = "use `DiscStream::with_halt(Halt)` at construction instead"
)]
pub fn set_halt(&mut self, flag: Arc<AtomicBool>) {
self.halt = Some(flag);
self.halt = Some(Halt::from_arc(flag));
}
fn is_halted(&self) -> bool {
self.halt
.as_ref()
.map(|h| h.load(Ordering::Relaxed))
.map(|h| h.is_cancelled())
.unwrap_or(false)
}
@@ -241,9 +285,12 @@ impl DiscStream {
}
}
/// Skip decryption — return raw encrypted bytes.
/// Skip decryption — return raw encrypted bytes. Updates both
/// the metadata-side key field and the wrapped reader's keys so
/// subsequent `read_sectors` calls become a pass-through.
pub fn set_raw(&mut self) {
self.decrypt_keys = crate::decrypt::DecryptKeys::None;
self.reader.set_keys(crate::decrypt::DecryptKeys::None);
}
/// Get the scanned Disc (for listing all titles).
@@ -275,10 +322,19 @@ impl DiscStream {
// this loop can spend minutes shrinking and skipping sectors; without
// the check, Stop wouldn't take effect until the outer PES read() loop
// finally emits a frame, which may never happen.
let start_time = std::time::Instant::now();
loop {
if self.is_halted() {
return Err(crate::error::Error::Halted.into());
}
// Debug: log slow reads during mux — helps diagnose stalls
if cfg!(debug_assertions) && start_time.elapsed().as_secs() > 5 {
tracing::debug!(target: "mux", "fill_extents waiting at LBA {} ({}s elapsed, sectors={})", lba, start_time.elapsed().as_secs(), remaining);
}
let mut sectors = remaining.min(self.adaptive.current() as u32) as u16;
// Align to 3-sector AACS units when possible. Partial units at
// extent boundaries are safely handled by decrypt_sectors().
@@ -419,11 +475,10 @@ impl crate::pes::Stream for DiscStream {
}
let bytes = self.buf_valid;
if let Err(e) =
crate::decrypt::decrypt_sectors(&mut self.read_buf[..bytes], &self.decrypt_keys, 0)
{
return Err(e.into());
}
// Plaintext: the wrapped reader (DecryptingSectorSource)
// applied AACS / CSS in-place during fill_extents'
// read_sectors call. The pre-0.18 inline decrypt step
// lived here.
if let Some(ref mut demuxer) = self.ts_demuxer {
let packets = demuxer.feed(&self.read_buf[..bytes]);
@@ -525,3 +580,140 @@ impl crate::pes::Stream for DiscStream {
true
}
}
#[cfg(test)]
mod tests {
//! `DiscStream` is the only read-only `Stream` impl in tree (every
//! other concrete impl in `mux/*` is bidirectional or write-only).
//! These tests lock down a static `Send` assertion plus a
//! `Box<dyn Stream>` round trip exercising every method through the
//! trait object, so future Send-breaking edits to `DiscStream`'s
//! interior types fail at compile time.
use super::*;
use crate::disc::{ContentFormat, DiscTitle};
use crate::pes::Stream;
/// Static-assert `DiscStream: Send`. The `Stream` trait has `Send` as a
/// supertrait — if a future field on `DiscStream` is non-`Send` (e.g.
/// a `Box<dyn Read>` instead of `Box<dyn SectorSource>`), this fails
/// at compile time, before the runtime trait-object test below.
fn _assert_disc_stream_is_send() {
fn requires_send<T: Send>() {}
requires_send::<DiscStream>();
}
/// Trivial `SectorSource` that yields zeroed sectors. Empty title means
/// the demuxer produces no PES frames, so `read()` walks the extents to
/// EOF and returns `Ok(None)`. That's enough to exercise the trait-object
/// dispatch — the goal here is the bridge, not the demuxer.
struct ZeroReader {
capacity: u32,
}
impl crate::sector::SectorSource for ZeroReader {
fn read_sectors(
&mut self,
_lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> crate::error::Result<usize> {
let bytes = count as usize * 2048;
buf[..bytes].fill(0);
Ok(bytes)
}
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
fn synthetic_title(sector_count: u32) -> DiscTitle {
DiscTitle {
extents: vec![crate::disc::Extent {
start_lba: 0,
sector_count,
}],
..DiscTitle::empty()
}
}
/// Smallest credible witness that `DiscStream` flows through `dyn Stream`:
/// build a `Box<dyn Stream>`, drive `read()` to EOF, exercise `info()` /
/// `headers_ready()` / `codec_private()` through the trait object.
#[test]
fn stream_via_dyn_object() {
let reader = ZeroReader { capacity: 8 };
let title = synthetic_title(8);
let stream = DiscStream::new(
Box::new(reader),
title,
crate::decrypt::DecryptKeys::None,
8,
ContentFormat::BdTs,
);
let mut src: Box<dyn Stream> = Box::new(stream);
// Empty-title fixture has no streams configured, so headers are
// trivially ready and codec_private() yields nothing on track 0.
assert!(src.headers_ready());
assert!(src.codec_private(0).is_none());
let _ = src.info();
// Drive read() to EOF through the trait object — empty-title fixture
// produces no frames, but the call still routes through the blanket
// dispatch into Stream::read.
let mut frames = 0usize;
while src.read().expect("read").is_some() {
frames += 1;
if frames > 1024 {
panic!("unexpected unbounded frame stream from empty title");
}
}
assert_eq!(frames, 0);
}
/// `is_halted()` must observe a cancellation signal regardless of
/// which entry point installed the token. The deprecated
/// `set_halt(Arc<AtomicBool>)` and the new `with_halt(Halt)` are
/// two views over one slot — flipping either bit must cause the
/// next `fill_extents` retry boundary to bail.
#[test]
fn halt_via_with_halt_observed_by_is_halted() {
let halt = Halt::new();
let stream = DiscStream::new(
Box::new(ZeroReader { capacity: 8 }),
synthetic_title(8),
crate::decrypt::DecryptKeys::None,
8,
crate::disc::ContentFormat::BdTs,
)
.with_halt(halt.clone());
assert!(!stream.is_halted());
halt.cancel();
assert!(
stream.is_halted(),
"with_halt token cancellation must be observed by is_halted()"
);
}
#[test]
fn halt_via_set_halt_bridge_observed_by_is_halted() {
let arc = Arc::new(AtomicBool::new(false));
let mut stream = DiscStream::new(
Box::new(ZeroReader { capacity: 8 }),
synthetic_title(8),
crate::decrypt::DecryptKeys::None,
8,
crate::disc::ContentFormat::BdTs,
);
stream.set_halt(arc.clone());
assert!(!stream.is_halted());
arc.store(true, std::sync::atomic::Ordering::Relaxed);
assert!(
stream.is_halted(),
"set_halt(Arc<AtomicBool>) bridge must observe Arc-side flips"
);
}
}
+6 -6
View File
@@ -1,10 +1,10 @@
//! ISO sector reader — file-backed SectorReader for Blu-ray ISO images.
//! ISO sector reader — file-backed SectorSource for Blu-ray ISO images.
//!
//! An ISO is a flat image of 2048-byte sectors. Sector N starts at byte offset N * 2048.
//! Used by DiscStream::open_iso() and Disc::scan_image().
use crate::error::{Error, Result};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use std::fs::File;
use std::io::{Read, Seek, SeekFrom};
use std::path::Path;
@@ -32,12 +32,12 @@ impl IsoSectorReader {
Ok(Self { file, capacity })
}
pub fn capacity(&self) -> u32 {
pub fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
impl SectorReader for IsoSectorReader {
impl SectorSource for IsoSectorReader {
fn read_sectors(
&mut self,
lba: u32,
@@ -73,7 +73,7 @@ mod tests {
std::fs::write(&dir, &data).unwrap();
let mut reader = IsoSectorReader::open(dir.to_str().unwrap()).unwrap();
assert_eq!(reader.capacity(), 4);
assert_eq!(reader.capacity_sectors(), 4);
let mut buf = [0u8; 2048];
reader.read_sectors(0, 1, &mut buf, true).unwrap();
@@ -94,7 +94,7 @@ mod tests {
std::fs::write(&dir, &data).unwrap();
let reader = IsoSectorReader::open(dir.to_str().unwrap()).unwrap();
assert_eq!(reader.capacity(), 10);
assert_eq!(reader.capacity_sectors(), 10);
std::fs::remove_file(&dir).ok();
}
+7 -7
View File
@@ -18,10 +18,10 @@ const SCAN_SIZE: usize = 1024 * 1024;
enum Mode {
Write {
muxer: super::tsmux::TsMuxer<Box<dyn Write>>,
muxer: super::tsmux::TsMuxer<Box<dyn Write + Send>>,
},
Read {
reader: Box<dyn Read>,
reader: Box<dyn Read + Send>,
},
}
@@ -57,7 +57,7 @@ pub struct M2tsStream {
impl M2tsStream {
/// Create for writing PES frames → BD-TS output.
/// Writes FMKV metadata header, then muxes PES frames into BD transport stream.
pub fn create(mut writer: impl Write + 'static, title: &DiscTitle) -> io::Result<Self> {
pub fn create(mut writer: impl Write + Send + 'static, title: &DiscTitle) -> io::Result<Self> {
// Write FMKV metadata header
if !title.streams.is_empty() {
let m = meta::M2tsMeta::from_title(title);
@@ -72,7 +72,7 @@ impl M2tsStream {
DiscStream::Subtitle(s) => s.pid,
})
.collect();
let boxed: Box<dyn Write> = Box::new(writer);
let boxed: Box<dyn Write + Send> = Box::new(writer);
let mut muxer = super::tsmux::TsMuxer::new(boxed, &pids);
for (i, cp) in title.codec_privates.iter().enumerate() {
if let Some(data) = cp {
@@ -111,7 +111,7 @@ impl M2tsStream {
/// Open an M2TS stream for reading. Takes any Read source — file, pipe, socket.
///
/// Tries FMKV metadata header first. Falls back to PMT scan of first 1 MB.
pub fn open(mut reader: impl Read + 'static) -> io::Result<Self> {
pub fn open(mut reader: impl Read + Send + 'static) -> io::Result<Self> {
// Read first chunk — enough for FMKV header or PMT scan
let mut head = vec![0u8; SCAN_SIZE];
let head_len = read_fill(&mut reader, &mut head)?;
@@ -125,7 +125,7 @@ impl M2tsStream {
let (pids, parsers, pid_to_track) = Self::setup_pes(&title.streams);
// Chain: remaining head bytes + rest of reader
let remaining_head = &head[header_end..];
let chain: Box<dyn Read> =
let chain: Box<dyn Read + Send> =
Box::new(io::Cursor::new(remaining_head.to_vec()).chain(reader));
return Ok(Self {
disc_title: title.clone(),
@@ -150,7 +150,7 @@ impl M2tsStream {
let (pids, parsers, pid_to_track) = Self::setup_pes(&streams);
// Chain: full head (it's all TS data) + rest of reader
let chain: Box<dyn Read> = Box::new(io::Cursor::new(head).chain(reader));
let chain: Box<dyn Read + Send> = Box::new(io::Cursor::new(head).chain(reader));
Ok(Self {
disc_title: DiscTitle {
+4 -4
View File
@@ -18,7 +18,7 @@ use crate::disc::*;
use std::io::{self, Read};
struct ReadState {
reader: Box<dyn Read>,
reader: Box<dyn Read + Send>,
cluster_ts_ms: i64,
/// Codec private data per track (track_number, hvcC/avcC bytes).
codec_privates: Vec<(u16, Vec<u8>)>,
@@ -26,7 +26,7 @@ struct ReadState {
enum Mode {
Write {
muxer: Option<MkvMuxer<Box<dyn WriteSeek>>>,
muxer: Option<MkvMuxer<Box<dyn WriteSeek + Send>>>,
},
Read(ReadState),
}
@@ -40,7 +40,7 @@ pub struct MkvStream {
impl MkvStream {
/// Create for writing PES frames → MKV container.
/// Codec privates come from title.codec_privates (populated by input stream).
pub fn create(writer: Box<dyn WriteSeek>, title: &DiscTitle) -> io::Result<Self> {
pub fn create(writer: Box<dyn WriteSeek + Send>, title: &DiscTitle) -> io::Result<Self> {
let mut tracks = Vec::new();
let mut has_default_video = false;
let mut has_default_audio = false;
@@ -79,7 +79,7 @@ impl MkvStream {
}
/// Open an MKV file for reading → PES frames.
pub fn open(mut reader: impl Read + 'static) -> io::Result<Self> {
pub fn open(mut reader: impl Read + Send + 'static) -> io::Result<Self> {
let (disc_title, codec_privates) = parse_mkv_header(&mut reader)?;
Ok(Self {
disc_title,
+23
View File
@@ -30,3 +30,26 @@ impl crate::pes::Stream for NullStream {
&self.disc_title
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::pes::Stream;
/// Verify NullStream routes through the `Stream` trait object cleanly.
#[test]
fn stream_via_dyn_object_writes_and_finishes() {
let title = DiscTitle::empty();
let mut sink: Box<dyn Stream> = Box::new(NullStream::new(&title));
let frame = crate::pes::PesFrame {
track: 0,
pts: 0,
keyframe: true,
data: vec![0x01, 0x02, 0x03],
};
sink.write(&frame).unwrap();
let _ = sink.info();
sink.finish().unwrap();
}
}
+20 -7
View File
@@ -186,7 +186,7 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
None => crate::disc::ScanOptions::default(),
};
let mut reader = super::iso::IsoSectorReader::open(&path.to_string_lossy())?;
let capacity = reader.capacity();
let capacity = reader.capacity_sectors();
let disc = crate::disc::Disc::scan_image(&mut reader, capacity, &scan_opts)
.map_err(|e| -> io::Error { e.into() })?;
if disc.titles.is_empty() {
@@ -203,7 +203,11 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
let title = disc.titles[idx].clone();
let keys = disc.decrypt_keys();
let format = disc.content_format;
let mut stream = DiscStream::new(Box::new(reader), title, keys, 64, format);
// ISO file: use large batch size (16 MB) — sequential read from fast storage, no bad sectors.
// Physical drives need small batches for adaptive error handling and retry logic.
const ISO_MUX_BATCH_SECTORS: u16 = 8192;
let mut stream =
DiscStream::new(Box::new(reader), title, keys, ISO_MUX_BATCH_SECTORS, format);
if opts.raw {
stream.set_raw();
}
@@ -242,15 +246,24 @@ pub fn output(
match parsed {
StreamUrl::Mkv { ref path } => {
validate_file_path(path, "mkv")?;
let file = std::fs::File::create(path)?;
let writer: Box<dyn super::WriteSeek> =
Box::new(std::io::BufWriter::with_capacity(IO_BUF_SIZE, file));
// Wrap the output in `crate::io::WritebackFile` (bounded-cache
// writeback) so a UHD-scale MKV mux to slow / network-attached
// staging doesn't hit the dirty-page burst pathology that
// sweep already side-steps. BufWriter sits on top to coalesce
// mux's many small EBML element writes.
let writer: Box<dyn super::WriteSeek + Send> =
Box::new(std::io::BufWriter::with_capacity(
IO_BUF_SIZE,
crate::io::WritebackFile::create(path)?,
));
Ok(Box::new(MkvStream::create(writer, title)?))
}
StreamUrl::M2ts { ref path } => {
validate_file_path(path, "m2ts")?;
let file = std::fs::File::create(path)?;
let writer = std::io::BufWriter::with_capacity(IO_BUF_SIZE, file);
let writer = std::io::BufWriter::with_capacity(
IO_BUF_SIZE,
crate::io::WritebackFile::create(path)?,
);
Ok(Box::new(M2tsStream::create(writer, title)?))
}
StreamUrl::Network { ref addr } => {
+104 -34
View File
@@ -78,43 +78,20 @@ impl PesFrame {
}
}
/// A PES frame source or sink. Each implementor is **either** read-only or
/// write-only — never both.
/// A PES frame stream. One trait per format — same type opens for read
/// (`open()` / `listen()` / `input()`) or write (`create()` / `connect()` /
/// `output()`). Calling the wrong-direction method returns a typed
/// `StreamReadOnly` / `StreamWriteOnly` error.
///
/// Implementors fall into two camps:
///
/// - **Read sources**: `DiscStream` (drive or ISO), `M2tsStream` (when
/// constructed from an existing file), `MkvStream` (demux), `NetworkStream`
/// (TCP listener), `StdioStream::input()`. These return frames from
/// `read()` and surface `StreamWriteOnly` (E9001) from `write()`.
/// - **Write sinks**: `MkvStream::create`, `M2tsStream::create`,
/// `NetworkStream::connect`, `StdioStream::output()`, `NullStream`.
/// These accept frames in `write()` and surface `StreamReadOnly` (E9000)
/// from `read()`. Always call `finish()` when done — that's where MKV
/// writes its `Cues` index and `M2tsStream` flushes the TS muxer.
///
/// Direction is established at construction; mixing produces an error code,
/// not a panic. Most consumers don't construct streams directly — call
/// `mux::input(url, opts)` / `mux::output(url, title)` and let URL parsing
/// pick the right type.
///
/// `info()` returns the stream's `DiscTitle` metadata (track list, codec
/// info, duration). For sources it's parsed from the input; for sinks it's
/// the metadata supplied at creation. Stable across all reads.
///
/// `codec_private(track)` exposes per-track initialization data
/// (H.264 SPS/PPS, HEVC VPS/SPS/PPS, AC-3 fscod, etc.) that some output
/// formats need before any frame can be written. `headers_ready()` returns
/// false until enough input frames have been seen to populate every video
/// track's codec-private blob — callers buffer frames they read until
/// `headers_ready()` returns true.
pub trait Stream {
/// `Send` is required so streams can move across the producer/consumer
/// threads in autorip's mux pipeline.
pub trait Stream: Send {
/// Read the next frame, or `Ok(None)` at end of stream. Returns
/// `StreamWriteOnly` (E9001) on a write-only sink.
/// `StreamWriteOnly` on a stream opened for writing.
fn read(&mut self) -> std::io::Result<Option<PesFrame>>;
/// Write a frame to the sink. Returns `StreamReadOnly` (E9000) on a
/// read-only source.
/// Write a frame to the sink. Returns `StreamReadOnly` on a stream
/// opened for reading.
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()>;
/// Finalize the stream: flush buffered frames, write any container
@@ -143,7 +120,7 @@ pub trait Stream {
/// Wraps any output stream and counts bytes written.
///
/// Progress tracking is a CLI concern — streams don't know their size.
/// Wrap the output with CountingStream, then query bytes_written().
/// Wrap the output with `CountingStream`, then query `bytes_written()`.
///
/// ```text
/// let mut output = CountingStream::new(libfreemkv::output(dest, &title)?);
@@ -194,3 +171,96 @@ impl Stream for CountingStream {
self.inner.headers_ready()
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::disc::DiscTitle;
fn make_frame(track: usize, pts: i64) -> PesFrame {
PesFrame {
track,
pts,
keyframe: track == 0 && pts == 0,
data: vec![track as u8, (pts & 0xff) as u8, 0xAA],
}
}
/// Minimal in-memory `Stream` for trait-shape tests. `read` replays
/// pre-seeded frames; `write` collects them.
struct MockStream {
read_queue: std::vec::IntoIter<PesFrame>,
written: Vec<PesFrame>,
title: DiscTitle,
}
impl MockStream {
fn new(read_frames: Vec<PesFrame>) -> Self {
Self {
read_queue: read_frames.into_iter(),
written: Vec::new(),
title: DiscTitle::empty(),
}
}
}
impl Stream for MockStream {
fn read(&mut self) -> std::io::Result<Option<PesFrame>> {
Ok(self.read_queue.next())
}
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()> {
self.written.push(frame.clone());
Ok(())
}
fn finish(&mut self) -> std::io::Result<()> {
Ok(())
}
fn info(&self) -> &DiscTitle {
&self.title
}
}
#[test]
fn stream_read_yields_frames_then_eof() {
let frames = vec![make_frame(0, 0), make_frame(1, 1_000), make_frame(0, 2_000)];
let mut s = MockStream::new(frames.clone());
let f0 = s.read().unwrap().expect("first frame");
assert_eq!(f0.track, frames[0].track);
assert_eq!(f0.pts, frames[0].pts);
assert!(f0.keyframe);
let f1 = s.read().unwrap().expect("second frame");
assert_eq!(f1.pts, frames[1].pts);
let f2 = s.read().unwrap().expect("third frame");
assert_eq!(f2.pts, frames[2].pts);
assert!(s.read().unwrap().is_none());
assert!(s.read().unwrap().is_none()); // idempotent at EOF
}
#[test]
fn stream_write_collects_then_finishes() {
let mut s = MockStream::new(Vec::new());
let frames = [make_frame(0, 0), make_frame(1, 100), make_frame(2, 200)];
for f in &frames {
s.write(f).unwrap();
}
assert_eq!(s.written.len(), 3);
s.finish().unwrap();
}
#[test]
fn stream_via_dyn_object() {
let mut s: Box<dyn Stream> = Box::new(MockStream::new(vec![make_frame(0, 0)]));
let frame = s.read().unwrap().expect("first frame");
s.write(&frame).unwrap();
let _ = s.info();
s.finish().unwrap();
}
}
+54 -5
View File
@@ -1,7 +1,7 @@
//! Pipeline-progress reporting for the rip pipeline.
//!
//! v0.13.16 architecture rule: ONE progress signal type. Every long-running
//! pipeline operation (`Disc::copy`, `Disc::patch`, mux) emits the same
//! pipeline operation (`Disc::copy`, `Disc::patch`, `verify_title`) emits the same
//! `PassProgress` shape via the `Progress` trait. Consumers (autorip) compute
//! a single `PipelineStats` derived view and never reach into per-pass
//! internals.
@@ -28,6 +28,8 @@ pub enum PassKind {
/// Demux ISO → output (MKV / M2TS / network). Single phase that runs
/// after all rip passes complete.
Mux,
/// Sector verification — reads every sector and classifies health.
Verify,
}
/// One progress sample from a pipeline phase.
@@ -36,6 +38,13 @@ pub enum PassKind {
/// regardless of which kind of pass is running. `bytes_good_total` is the
/// cumulative count of confirmed-clean bytes across the whole rip; useful
/// for the "data recovered" stat the user sees.
///
/// For `PassKind::Verify`, the fields map as follows:
/// - `work_done` = sectors read so far
/// - `work_total` = total sectors in title
/// - `bytes_good_total` = good + slow + recovered sectors × 2048
/// - `bytes_unreadable_total` = bad sectors × 2048
/// - `bytes_pending_total` = 0 (verify processes sequentially, nothing pending)
#[derive(Debug, Clone, Copy)]
pub struct PassProgress {
pub kind: PassKind,
@@ -57,20 +66,60 @@ pub struct PassProgress {
pub main_title_size_bytes: Option<u64>,
}
impl PassProgress {
/// Percentage of work completed for this pass (0..=100).
///
/// Returns `100.0` if `work_total` is zero to avoid division by zero.
pub fn work_pct(&self) -> f64 {
if self.work_total == 0 {
return 100.0;
}
self.work_done as f64 / self.work_total as f64 * 100.0
}
/// Percentage of the disc that is confirmed clean (0..=100).
///
/// Computed from `bytes_good_total / bytes_total_disc`.
pub fn good_pct(&self) -> f64 {
if self.bytes_total_disc == 0 {
return 100.0;
}
self.bytes_good_total as f64 / self.bytes_total_disc as f64 * 100.0
}
/// Percentage of the disc that is unreadable (0..=100).
pub fn bad_pct(&self) -> f64 {
if self.bytes_total_disc == 0 {
return 0.0;
}
self.bytes_unreadable_total as f64 / self.bytes_total_disc as f64 * 100.0
}
/// Percentage of the disc that is still pending (not yet attempted or needs retry).
pub fn pending_pct(&self) -> f64 {
if self.bytes_total_disc == 0 {
return 0.0;
}
self.bytes_pending_total as f64 / self.bytes_total_disc as f64 * 100.0
}
}
/// A consumer of pipeline progress events. Library code calls
/// `Progress::report` once per inner-loop iteration (throttling is the
/// consumer's job — `report` is cheap; the library doesn't gate it).
///
/// Returns `true` to continue, `false` to request early stop.
///
/// No `Send`/`Sync` bound — `report` is always called from the same thread
/// running the rip pipeline, so closures with non-`Sync` captures (e.g.
/// running the pipeline, so closures with non-`Sync` captures (e.g.
/// `RefCell<PassProgressState>`) work directly. Blanket impl below lets
/// callers pass closures without explicit struct types.
pub trait Progress {
fn report(&self, p: &PassProgress);
fn report(&self, p: &PassProgress) -> bool;
}
impl<F: Fn(&PassProgress)> Progress for F {
fn report(&self, p: &PassProgress) {
impl<F: Fn(&PassProgress) -> bool> Progress for F {
fn report(&self, p: &PassProgress) -> bool {
(self)(p)
}
}
+15 -4
View File
@@ -53,7 +53,7 @@ static io_registry_entry_t find_iomedia_child(io_registry_entry_t parent) {
char cls[128];
kr = IOObjectGetClass(child, cls);
if (kr == KERN_SUCCESS) {
if (strcmp(cls, "IOMedia") == 0) {
if (strcmp(cls, "IOMedia") == 0 || strcmp(cls, "IOBDMedia") == 0) {
IOObjectRelease(iter);
return child;
}
@@ -214,8 +214,15 @@ int shim_open_exclusive(const char *bsd_name) {
return 0;
}
char cmd[128];
snprintf(cmd, sizeof(cmd), "diskutil unmountDisk force %s 2>/dev/null", bsd_name);
// Use a shell wrapper so the device path is not subject to buffer limits.
// snprintf into 128 bytes could truncate long BSD names (e.g. disk12s3s1),
// producing a broken command. sh -c with $1 passes the arg via argv.
const char *shell_fmt = "sh -c 'diskutil unmountDisk force \"$1\" >/dev/null 2>&1' _ %s";
char cmd[512];
int written = snprintf(cmd, sizeof(cmd), shell_fmt, bsd_name);
if (written < 0 || (size_t)written >= sizeof(cmd)) {
return -1;
}
system(cmd);
usleep(500000);
@@ -256,7 +263,11 @@ int shim_open_exclusive(const char *bsd_name) {
return -4;
}
kr = (*g_handle.scsi)->ObtainExclusiveAccess(g_handle.scsi);
for (int retry = 0; retry < 10; retry++) {
kr = (*g_handle.scsi)->ObtainExclusiveAccess(g_handle.scsi);
if (kr == kIOReturnSuccess) break;
usleep(500000);
}
if (kr != kIOReturnSuccess) {
(*g_handle.scsi)->Release(g_handle.scsi);
(*g_handle.mmc)->Release(g_handle.mmc);
+19 -4
View File
@@ -68,10 +68,25 @@ pub(crate) const TUR_TIMEOUT_MS: u32 = 5_000;
pub(crate) const READ_TIMEOUT_MS: u32 = 10_000;
/// Timeout for content READ commands on the recovery path —
/// [`disc::Disc::patch`]'s targeted retries on bad ranges. Doubles
/// the fast-path budget so a sector that fails at 30 s gets one more
/// honest attempt. Matches sg_dd's default per-command timeout
/// (`DEF_TIMEOUT = 60000`).
/// [`disc::Disc::patch`]'s targeted retries on bad ranges. Matches
/// `sg_dd`'s 60 s ceiling: long enough that any sector the drive can
/// recover at all gets the time to do so, short enough that an
/// unresponsive bus is detected before the per-range watchdog fires.
///
/// In practice failed reads return in 1–4 s (the drive itself gives up
/// on uncorrectable ECC before the timeout); the 60 s value is a
/// safety ceiling, not a steady-state cost.
///
/// Historical note (2026-05-08): briefly lowered to 2 s with a 5×
/// inline retry loop in `Disc::patch` to mimic the kernel `sr_mod`
/// driver's auto-retry pattern. The synthetic logic worked but on the
/// live drive each "2 s" read paid ~1.5 s of kernel SCSI mid-layer
/// error escalation on top, so 5× retries took ~17 s per LBA and
/// triggered MAX_RANGE_SECS after 4 sectors — pushing recovery to
/// 0/22 ranges (worse than the 0/22 baseline of v0.17.3 single-shot
/// at 60 s, since that at least visited every range). Reverted; the
/// kernel-auto-retry approach is being pursued via a `/dev/sr0` pread
/// fallback instead.
pub(crate) const READ_RECOVERY_TIMEOUT_MS: u32 = 60_000;
// ── SCSI status bytes (SPC-4 §4.5.5) ────────────────────────────────────────
-82
View File
@@ -1,82 +0,0 @@
//! SectorReader — trait for reading 2048-byte disc sectors.
//!
//! Implemented by Drive (SCSI) and IsoFile (file-backed).
//! Used by UDF parser, disc scanner, label parsers — anything that
//! reads sectors doesn't need to know where they come from.
use crate::error::Result;
/// Read 2048-byte sectors from a disc or disc image.
pub trait SectorReader: Send {
/// Read `count` sectors starting at `lba` into `buf`.
/// `buf` must be at least `count * 2048` bytes.
/// `recovery`: true = full retry/reset loop (ripping), false = single attempt (verify).
/// File-backed readers ignore the flag.
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize>;
/// Total capacity in sectors, if known.
fn capacity(&self) -> u32 {
0
}
fn set_speed(&mut self, _kbs: u16) {}
}
/// SectorReader backed by a file (ISO image).
/// Seeks to lba * 2048, reads count * 2048 bytes.
pub struct FileSectorReader {
file: std::io::BufReader<std::fs::File>,
capacity: u32,
}
impl FileSectorReader {
pub fn open(path: &str) -> std::io::Result<Self> {
let file = std::fs::File::open(path)?;
let len = file.metadata()?.len();
let sectors = len / 2048;
if sectors > u32::MAX as u64 {
// ~8 TB hard cap (u32::MAX × 2048 bytes). Path lives in the
// typed Error variant — no English in the message.
return Err(crate::error::Error::IsoTooLarge {
path: path.to_string(),
}
.into());
}
let capacity = sectors as u32;
Ok(Self {
file: std::io::BufReader::with_capacity(4 * 1024 * 1024, file),
capacity,
})
}
}
impl SectorReader for FileSectorReader {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
use std::io::{Read, Seek, SeekFrom};
let offset = lba as u64 * 2048;
let bytes = count as usize * 2048;
self.file
.seek(SeekFrom::Start(offset))
.map_err(|e| crate::error::Error::IoError { source: e })?;
self.file
.read_exact(&mut buf[..bytes])
.map_err(|e| crate::error::Error::IoError { source: e })?;
Ok(bytes)
}
fn capacity(&self) -> u32 {
self.capacity
}
}
+200
View File
@@ -0,0 +1,200 @@
//! `DecryptingSectorSource` — wrap any [`SectorSource`] to apply
//! AACS / CSS in-place decryption on every read.
//!
//! This is the 0.18 single-source-of-truth for decrypt-on-read. The
//! actual cipher code lives in [`crate::aacs`] and [`crate::css`];
//! we just call the existing [`crate::decrypt::decrypt_sectors`]
//! helper that already drives both of them. In follow-up commits
//! `sweep_pipeline` and `DiscStream` migrate onto this decorator
//! and delete their duplicate decrypt call sites.
//!
//! Composition: `Drive` → `DecryptingSectorSource` → caller sees
//! plaintext. For `DecryptKeys::None` discs the decorator is a
//! pass-through, so callers can wire it unconditionally and keep
//! their pipeline shape uniform regardless of encryption state.
use crate::decrypt::{DecryptKeys, decrypt_sectors};
use crate::error::Result;
use super::SectorSource;
/// Decorator: read from `inner`, then run the configured
/// AACS / CSS decrypt over the bytes that landed in `buf`.
///
/// `unit_key_idx` selects the AACS unit key for the disc (0 for
/// the vast majority of titles; the rare multi-CPS-unit discs pick
/// the index that covers the title being read). For
/// [`DecryptKeys::None`] and [`DecryptKeys::Css`] the index is
/// ignored.
pub struct DecryptingSectorSource<S: SectorSource> {
inner: S,
keys: DecryptKeys,
unit_key_idx: usize,
}
impl<S: SectorSource> DecryptingSectorSource<S> {
/// Wrap `inner` with the given keys. The default unit-key
/// index is 0; use [`with_unit_key_idx`] for the multi-CPS-unit
/// case.
///
/// [`with_unit_key_idx`]: Self::with_unit_key_idx
pub fn new(inner: S, keys: DecryptKeys) -> Self {
Self {
inner,
keys,
unit_key_idx: 0,
}
}
/// Override the AACS unit-key index. Only meaningful for
/// [`DecryptKeys::Aacs`]; other variants ignore it.
pub fn with_unit_key_idx(mut self, idx: usize) -> Self {
self.unit_key_idx = idx;
self
}
/// Replace the configured keys without unwrapping the decorator.
/// Used by `DiscStream::set_raw()` to flip from encrypted-disc
/// decryption to a pass-through after the inner reader is already
/// owned by the wrapper. For new construction prefer [`new`].
///
/// [`new`]: Self::new
pub fn set_keys(&mut self, keys: DecryptKeys) {
self.keys = keys;
}
/// Borrow the inner source. Useful for tests and for adapters
/// that want to introspect the underlying drive / file without
/// unwrapping the decorator.
pub fn inner(&self) -> &S {
&self.inner
}
/// Mutable borrow of the inner source.
pub fn inner_mut(&mut self) -> &mut S {
&mut self.inner
}
/// Consume the decorator and return the underlying source.
pub fn into_inner(self) -> S {
self.inner
}
}
impl<S: SectorSource> SectorSource for DecryptingSectorSource<S> {
fn capacity_sectors(&self) -> u32 {
self.inner.capacity_sectors()
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
let n = self.inner.read_sectors(lba, count, buf, recovery)?;
// Reuse the existing crate-wide decrypt entry point — same
// path the 0.17 sweep_pipeline and DiscStream call, so we
// inherit their AACS / CSS / None semantics verbatim. The
// helper is a no-op for DecryptKeys::None.
decrypt_sectors(&mut buf[..n], &self.keys, self.unit_key_idx)?;
Ok(n)
}
fn set_speed(&mut self, kbs: u16) {
self.inner.set_speed(kbs)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::error::Result;
/// Synthetic SectorSource that yields a deterministic byte
/// pattern keyed by LBA. Used to verify the decorator's
/// pass-through behaviour for `DecryptKeys::None`.
struct PatternedSource {
capacity: u32,
}
impl PatternedSource {
fn fill(lba: u32, count: u16, buf: &mut [u8]) {
let bytes = count as usize * 2048;
for (i, slot) in buf[..bytes].iter_mut().enumerate() {
let abs = lba as u64 * 2048 + i as u64;
*slot = ((abs.wrapping_mul(2654435761) >> 16) & 0xff) as u8;
}
}
}
impl SectorSource for PatternedSource {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
Self::fill(lba, count, buf);
Ok(count as usize * 2048)
}
}
#[test]
fn passthrough_with_no_keys() {
let src = PatternedSource { capacity: 16 };
let mut wrapped = DecryptingSectorSource::new(src, DecryptKeys::None);
// capacity_sectors delegates.
assert_eq!(wrapped.capacity_sectors(), 16);
let mut got = vec![0u8; 4 * 2048];
let n = wrapped.read_sectors(3, 4, &mut got, false).unwrap();
assert_eq!(n, 4 * 2048);
let mut expected = vec![0u8; 4 * 2048];
PatternedSource::fill(3, 4, &mut expected);
assert_eq!(got, expected);
}
#[test]
fn passthrough_set_speed_delegates() {
struct SpeedRecorder {
last: Option<u16>,
}
impl SectorSource for SpeedRecorder {
fn capacity_sectors(&self) -> u32 {
0
}
fn read_sectors(
&mut self,
_lba: u32,
_count: u16,
_buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
Ok(0)
}
fn set_speed(&mut self, kbs: u16) {
self.last = Some(kbs);
}
}
let mut wrapped =
DecryptingSectorSource::new(SpeedRecorder { last: None }, DecryptKeys::None);
wrapped.set_speed(7200);
assert_eq!(wrapped.inner().last, Some(7200));
}
// TODO: AACS round-trip test — needs a fixture-encrypted unit
// (6144-byte aligned) plus the matching unit key. The cipher
// path itself is exercised by `crate::aacs` unit tests; here
// we only assert the decorator wires the existing helper, not
// that AES-128 is correct.
}
+243
View File
@@ -0,0 +1,243 @@
//! File-backed sector I/O — read and write 2048-byte sectors against
//! an ISO image on disk.
//!
//! [`FileSectorSource`] is the read side (open-only). [`FileSectorSink`]
//! is the write side (create or open-rw); writes go through
//! [`crate::io::WritebackFile`] so big sequential ISO writes share
//! the same bounded-cache writeback pipeline used by sweep / patch /
//! mux.
use std::fs::{File, OpenOptions};
use std::io::{Read, Seek, SeekFrom, Write};
use std::path::Path;
use crate::error::{Error, Result};
use super::{SectorSink, SectorSource};
/// SectorSource backed by a file (ISO image).
///
/// Seeks to `lba * 2048`, reads `count * 2048` bytes per call. The
/// file is held directly: every `read_sectors` call performs an
/// absolute seek, so a wrapping `BufReader` would have its buffer
/// invalidated on every call (its internal cursor moves with the
/// `Seek` impl) — pure overhead. Callers that benefit from buffered
/// reads should compose their own `BufReader` at the `read_sectors`
/// granularity they care about.
pub struct FileSectorSource {
file: File,
capacity: u32,
}
impl FileSectorSource {
/// Open an existing ISO file for reading. Capacity is derived
/// from `metadata().len() / 2048`. Returns
/// [`Error::IsoTooLarge`] if the file would exceed the 32-bit
/// LBA address space (~8 TB).
pub fn open(path: &Path) -> std::io::Result<Self> {
let file = File::open(path)?;
let len = file.metadata()?.len();
let sectors = len / 2048;
if sectors > u32::MAX as u64 {
return Err(Error::IsoTooLarge {
path: path.to_string_lossy().into_owned(),
}
.into());
}
let capacity = sectors as u32;
Ok(Self { file, capacity })
}
}
// Implement the legacy `SectorSource` trait. The blanket impl in
// `super` produces the `SectorSource` impl automatically — no need
// to write both, and writing both would conflict. This keeps the
// 0.17 method-resolution path intact (callers with `SectorSource`
// in scope can still write `fsr.read_sectors(..)` against a
// `FileSectorSource`).
impl SectorSource for FileSectorSource {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
let offset = lba as u64 * 2048;
let bytes = count as usize * 2048;
self.file
.seek(SeekFrom::Start(offset))
.map_err(|e| Error::IoError { source: e })?;
self.file
.read_exact(&mut buf[..bytes])
.map_err(|e| Error::IoError { source: e })?;
Ok(bytes)
}
}
/// SectorSink backed by a file (ISO image).
///
/// Writes go through [`crate::io::WritebackFile`], which on Linux drives
/// continuous `sync_file_range` + `posix_fadvise(DONTNEED)` to keep
/// the kernel dirty page cache bounded during multi-GB sequential
/// writes. macOS / Windows fall through to a no-op pipeline.
///
/// `finish` runs `sync_all` before dropping the underlying file.
pub struct FileSectorSink {
inner: crate::io::WritebackFile,
}
impl FileSectorSink {
/// Create a new ISO file at `path`, truncating any existing
/// file. The file is opened read-write so the same handle can
/// later be reused for verification reads if needed (sweep
/// doesn't, but it costs nothing here).
pub fn create(path: &Path) -> std::io::Result<Self> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(path)?;
Ok(Self {
inner: crate::io::WritebackFile::new(file)?,
})
}
/// Open an existing ISO file for in-place updates (e.g. patch
/// pass writing recovered sectors over zero-filled holes).
/// Does not truncate.
pub fn open(path: &Path) -> std::io::Result<Self> {
let file = OpenOptions::new().read(true).write(true).open(path)?;
Ok(Self {
inner: crate::io::WritebackFile::new(file)?,
})
}
}
impl SectorSink for FileSectorSink {
fn write_sectors(&mut self, lba: u32, buf: &[u8]) -> Result<()> {
debug_assert!(
buf.len() % 2048 == 0,
"FileSectorSink::write_sectors: buf len {} not a multiple of 2048",
buf.len()
);
let offset = lba as u64 * 2048;
self.inner
.seek(SeekFrom::Start(offset))
.map_err(|e| Error::IoError { source: e })?;
self.inner
.write_all(buf)
.map_err(|e| Error::IoError { source: e })?;
Ok(())
}
fn finish(mut self: Box<Self>) -> Result<()> {
self.inner
.sync_all()
.map_err(|e| Error::IoError { source: e })?;
Ok(())
}
}
#[cfg(test)]
mod tests {
// Bring the 0.18 trait into scope (not super::*: the super
// module also re-exports the legacy `SectorSource`, and
// having both `SectorSource::read_sectors` and
// `SectorSource::read_sectors` visible would force every
// call site to disambiguate). External consumers see the
// same surface this test exercises.
use super::{FileSectorSink, FileSectorSource};
use crate::sector::{SectorSink, SectorSource};
use tempfile::tempdir;
#[test]
fn round_trip_single_sector() {
let dir = tempdir().unwrap();
let path = dir.path().join("rt.iso");
let mut sink = FileSectorSink::create(&path).unwrap();
// Pre-extend the file to 4 sectors of zeros so we can write
// sector 2 in place. Easiest way: write zeros first.
let zeros = [0u8; 4 * 2048];
sink.write_sectors(0, &zeros).unwrap();
let mut payload = [0u8; 2048];
for (i, b) in payload.iter_mut().enumerate() {
*b = (i as u8).wrapping_mul(17);
}
sink.write_sectors(2, &payload).unwrap();
Box::new(sink).finish().unwrap();
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 4);
let mut got = [0u8; 2048];
let n = src.read_sectors(2, 1, &mut got, false).unwrap();
assert_eq!(n, 2048);
assert_eq!(got, payload);
// Sectors 0,1,3 still zero.
let mut z = [0xffu8; 2048];
src.read_sectors(0, 1, &mut z, false).unwrap();
assert!(z.iter().all(|b| *b == 0));
}
#[test]
fn round_trip_multi_sector() {
let dir = tempdir().unwrap();
let path = dir.path().join("multi.iso");
let mut sink = FileSectorSink::create(&path).unwrap();
let mut payload = vec![0u8; 8 * 2048];
for (i, b) in payload.iter_mut().enumerate() {
*b = ((i * 31) ^ (i >> 7)) as u8;
}
sink.write_sectors(0, &payload).unwrap();
Box::new(sink).finish().unwrap();
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 8);
let mut got = vec![0u8; 8 * 2048];
let n = src.read_sectors(0, 8, &mut got, false).unwrap();
assert_eq!(n, 8 * 2048);
assert_eq!(got, payload);
}
#[test]
fn open_existing_does_not_truncate() {
let dir = tempdir().unwrap();
let path = dir.path().join("open.iso");
// Create with 4 sectors of pattern A.
let mut sink = FileSectorSink::create(&path).unwrap();
let pat_a = [0xaau8; 4 * 2048];
sink.write_sectors(0, &pat_a).unwrap();
Box::new(sink).finish().unwrap();
// Reopen and overwrite sector 1 only.
let mut sink = FileSectorSink::open(&path).unwrap();
let pat_b = [0xbbu8; 2048];
sink.write_sectors(1, &pat_b).unwrap();
Box::new(sink).finish().unwrap();
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 4);
let mut got = [0u8; 2048];
src.read_sectors(0, 1, &mut got, false).unwrap();
assert_eq!(got, [0xaau8; 2048]);
src.read_sectors(1, 1, &mut got, false).unwrap();
assert_eq!(got, [0xbbu8; 2048]);
src.read_sectors(2, 1, &mut got, false).unwrap();
assert_eq!(got, [0xaau8; 2048]);
}
}
+113
View File
@@ -0,0 +1,113 @@
//! Sector-level I/O traits.
//!
//! The sector layer is direction-typed: [`SectorSource`] reads
//! 2048-byte sectors, [`SectorSink`] writes them. Concrete impls
//! never do both — physical drives are read-only, file-backed
//! ISO images are opened for read OR write at construction time.
//!
//! - [`SectorSource`] is implemented by `Drive` (hardware) and
//! [`FileSectorSource`] / `IsoSectorReader` (file-backed).
//! - [`SectorSink`] is implemented by [`FileSectorSink`]
//! (ISO-backed) and sweep/patch consumer adapters.
//! - [`DecryptingSectorSource`] is a decorator that wraps any
//! `SectorSource` and applies AACS / CSS in-place decrypt to
//! yield plaintext sectors.
pub mod decrypting;
pub mod file;
use crate::error::Result;
/// Read 2048-byte sectors from a disc, image, or composed source.
///
/// Wrap the inner source in [`DecryptingSectorSource`] to get
/// plaintext sectors out of an encrypted disc.
pub trait SectorSource: Send {
/// Total capacity in sectors, if known. Default `0` = unknown
/// (e.g. live drives that haven't completed `READ CAPACITY` yet).
fn capacity_sectors(&self) -> u32 {
0
}
/// Read `count` sectors starting at `lba` into `buf`.
/// `buf` must be at least `count * 2048` bytes.
/// `recovery`: true = full retry/reset loop (ripping),
/// false = single attempt (verify). File-backed sources ignore
/// the flag.
///
/// Returns the number of bytes written into `buf` on success.
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize>;
/// Optional speed control for sources that map to a physical
/// drive. No-op for everything else.
fn set_speed(&mut self, _kbs: u16) {}
}
// Forwarding impls so `Box<dyn SectorSource>` and `&mut dyn SectorSource`
// satisfy the `SectorSource` trait bound when wrapped by generic
// decorators like `DecryptingSectorSource<S: SectorSource>`.
impl SectorSource for Box<dyn SectorSource> {
fn capacity_sectors(&self) -> u32 {
(**self).capacity_sectors()
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
(**self).read_sectors(lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
(**self).set_speed(kbs)
}
}
impl SectorSource for &mut (dyn SectorSource + '_) {
fn capacity_sectors(&self) -> u32 {
(**self).capacity_sectors()
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
(**self).read_sectors(lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
(**self).set_speed(kbs)
}
}
/// Write 2048-byte sectors to a disc image or composed sink.
///
/// The terminal [`finish`] takes `Box<Self>` so it can run on `dyn
/// SectorSink` and consume the sink (`fsync` + close).
///
/// [`finish`]: SectorSink::finish
pub trait SectorSink: Send {
/// Write the sectors in `buf` starting at `lba`. `buf.len()`
/// must be a multiple of 2048; the implementation seeks to
/// `lba * 2048` before writing.
fn write_sectors(&mut self, lba: u32, buf: &[u8]) -> Result<()>;
/// Flush, fsync, and close. Consumes the sink. Always called
/// last; subsequent operations are not defined.
fn finish(self: Box<Self>) -> Result<()>;
}
pub use decrypting::DecryptingSectorSource;
pub use file::{FileSectorSink, FileSectorSource};
+17 -17
View File
@@ -19,7 +19,7 @@
//! BD-ROM Part 3 — Blu-ray filesystem profile
use crate::error::{Error, Result};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
/// A UDF filesystem parsed from disc.
#[derive(Debug)]
@@ -84,7 +84,7 @@ impl UdfFs {
/// Reads sector by sector from disc — no buffering.
/// Get the absolute starting LBA of a file on disc.
/// Used by the rip pipeline to locate m2ts content sectors.
pub fn file_start_lba(&self, reader: &mut dyn SectorReader, path: &str) -> Result<u32> {
pub fn file_start_lba(&self, reader: &mut dyn SectorSource, path: &str) -> Result<u32> {
let parts: Vec<&str> = path.trim_matches('/').split('/').collect();
let mut current = &self.root;
for part in &parts[..parts.len() - 1] {
@@ -115,7 +115,7 @@ impl UdfFs {
Ok(self.partition_start + data_lba)
}
pub fn read_file(&self, reader: &mut dyn SectorReader, path: &str) -> Result<Vec<u8>> {
pub fn read_file(&self, reader: &mut dyn SectorSource, path: &str) -> Result<Vec<u8>> {
let parts: Vec<&str> = path.trim_matches('/').split('/').collect();
let mut current = &self.root;
@@ -175,7 +175,7 @@ impl UdfFs {
///
/// Skips: STREAM/ (video), BACKUP/, DUPLICATE/,
/// MKB_RO.inf, ContentHash*, ContentRevocation*
pub fn metadata_sector_ranges(&self, reader: &mut dyn SectorReader) -> Result<Vec<(u32, u32)>> {
pub fn metadata_sector_ranges(&self, reader: &mut dyn SectorSource) -> Result<Vec<(u32, u32)>> {
let mut ranges = Vec::new();
// UDF structure: sector 0 through end of metadata partition
@@ -194,7 +194,7 @@ impl UdfFs {
/// All sector ranges that contain data (metadata + all files including STREAM).
/// For full disc-to-ISO dumps — reads only allocated sectors, skips gaps.
pub fn all_sector_ranges(&self, reader: &mut dyn SectorReader) -> Result<Vec<(u32, u32)>> {
pub fn all_sector_ranges(&self, reader: &mut dyn SectorSource) -> Result<Vec<(u32, u32)>> {
let mut ranges = Vec::new();
// UDF structure sectors
@@ -212,7 +212,7 @@ impl UdfFs {
fn collect_all_file_ranges(
&self,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
entry: &DirEntry,
ranges: &mut Vec<(u32, u32)>,
) -> Result<()> {
@@ -238,7 +238,7 @@ impl UdfFs {
fn collect_file_ranges(
&self,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
entry: &DirEntry,
ranges: &mut Vec<(u32, u32)>,
) -> Result<()> {
@@ -276,7 +276,7 @@ impl UdfFs {
/// Read an Extended File Entry (tag 266) or File Entry (tag 261)
/// and return its first allocation extent: (data_lba, data_length).
/// The data_lba is partition-relative.
fn read_icb_extent(&self, reader: &mut dyn SectorReader, meta_lba: u32) -> Result<(u32, u32)> {
fn read_icb_extent(&self, reader: &mut dyn SectorSource, meta_lba: u32) -> Result<(u32, u32)> {
let extents = self.read_icb_extents(reader, meta_lba)?;
extents.first().copied().ok_or(Error::DiscRead {
sector: 0,
@@ -290,7 +290,7 @@ impl UdfFs {
/// Handles files with many extents (e.g. 88 GB m2ts files have ~90 extents).
fn read_icb_extents(
&self,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
meta_lba: u32,
) -> Result<Vec<(u32, u32)>> {
let mut icb = [0u8; 2048];
@@ -367,7 +367,7 @@ impl UdfFs {
/// Returns Vec of (absolute_lba, sector_count) covering the entire file.
pub fn file_extents(
&self,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
path: &str,
) -> Result<Vec<(u32, u32)>> {
let parts: Vec<&str> = path.trim_matches('/').split('/').collect();
@@ -417,7 +417,7 @@ impl UdfFs {
/// 3. Metadata partition file → metadata content location
/// 4. FSD → root directory ICB
/// 5. Root directory → file tree
pub fn read_filesystem(reader: &mut dyn SectorReader) -> Result<UdfFs> {
pub fn read_filesystem(reader: &mut dyn SectorSource) -> Result<UdfFs> {
// Step 1: Anchor Volume Descriptor Pointer at sector 256
// ECMA-167 §10.2 — always at sector 256
let mut avdp = [0u8; 2048];
@@ -593,7 +593,7 @@ pub fn read_filesystem(reader: &mut dyn SectorReader) -> Result<UdfFs> {
/// and points to its ICB.
#[allow(clippy::only_used_in_recursion)]
fn read_directory(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
part_start: u32,
meta_start: u32,
meta_lba: u32,
@@ -758,7 +758,7 @@ fn read_directory(
}
/// Read file size (info_length) from an Extended File Entry ICB.
fn read_file_size(reader: &mut dyn SectorReader, meta_start: u32, meta_lba: u32) -> Result<u64> {
fn read_file_size(reader: &mut dyn SectorSource, meta_start: u32, meta_lba: u32) -> Result<u64> {
let mut icb = [0u8; 2048];
read_sector(reader, meta_start + meta_lba, &mut icb)?;
@@ -874,7 +874,7 @@ fn parse_dstring(data: &[u8]) -> String {
/// Each SCSI command has ~500ms overhead on USB drives, so reading 32 sectors
/// at once (one command) is 32x faster than 32 individual reads.
pub(crate) struct BufferedSectorReader<'a> {
inner: &'a mut dyn SectorReader,
inner: &'a mut dyn SectorSource,
cache_start: u32,
cache: Vec<u8>,
cache_sectors: u32,
@@ -884,7 +884,7 @@ pub(crate) struct BufferedSectorReader<'a> {
}
impl<'a> BufferedSectorReader<'a> {
pub(crate) fn new(inner: &'a mut dyn SectorReader, batch: u16) -> Self {
pub(crate) fn new(inner: &'a mut dyn SectorSource, batch: u16) -> Self {
Self {
inner,
cache_start: u32::MAX,
@@ -952,7 +952,7 @@ impl BufferedSectorReader<'_> {
}
}
impl SectorReader for BufferedSectorReader<'_> {
impl SectorSource for BufferedSectorReader<'_> {
fn read_sectors(
&mut self,
lba: u32,
@@ -996,7 +996,7 @@ impl SectorReader for BufferedSectorReader<'_> {
}
}
fn read_sector(reader: &mut dyn SectorReader, lba: u32, buf: &mut [u8]) -> Result<()> {
fn read_sector(reader: &mut dyn SectorSource, lba: u32, buf: &mut [u8]) -> Result<()> {
reader.read_sectors(lba, 1, buf, true)?;
Ok(())
}
+34 -14
View File
@@ -1,7 +1,8 @@
//! Disc sector verification — read every sector and classify health.
use crate::disc::{Chapter, DiscTitle};
use crate::sector::SectorReader;
use crate::progress::Progress;
use crate::sector::SectorSource;
use std::time::Instant;
/// Health status of a single sector read.
@@ -77,18 +78,14 @@ impl VerifyResult {
}
}
/// Progress callback: (sectors_done, total_sectors, current_status)
/// Return false to stop verification early.
pub type ProgressFn = Box<dyn FnMut(u64, u64, SectorStatus) -> bool>;
/// Verify all sectors in a title's extents.
/// Reads in batches for speed, falls back to single-sector on failure.
/// The progress callback returns false to request early stop.
pub fn verify_title(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
title: &DiscTitle,
batch_sectors: u16,
mut on_progress: Option<ProgressFn>,
on_progress: Option<&dyn Progress>,
) -> VerifyResult {
let start = Instant::now();
let mut good: u64 = 0;
@@ -97,7 +94,6 @@ pub fn verify_title(
let mut bad: u64 = 0;
let mut ranges: Vec<SectorRange> = Vec::new();
let mut sectors_done: u64 = 0;
let mut _stopped = false;
let mut byte_offset: u64 = 0;
let total_sectors: u64 = title.extents.iter().map(|e| e.sector_count as u64).sum();
@@ -140,9 +136,21 @@ pub fn verify_title(
}
sectors_done += count as u64;
if let Some(ref mut cb) = on_progress {
if !cb(sectors_done, total_sectors, status) {
_stopped = true;
if let Some(cb) = on_progress {
let pp = crate::progress::PassProgress {
kind: crate::progress::PassKind::Verify,
work_done: sectors_done,
work_total: total_sectors,
bytes_good_total: (good + slow + recovered) * 2048,
bytes_unreadable_total: bad * 2048,
bytes_pending_total: 0,
bytes_total_disc: total_sectors * 2048,
disc_duration_secs: Some(title.duration_secs),
bytes_bad_in_main_title: 0,
main_title_duration_secs: Some(title.duration_secs),
main_title_size_bytes: Some(total_sectors * 2048),
};
if !cb.report(&pp) {
break 'outer;
}
}
@@ -214,9 +222,21 @@ pub fn verify_title(
}
sectors_done += 1;
if let Some(ref mut cb) = on_progress {
if !cb(sectors_done, total_sectors, status) {
_stopped = true;
if let Some(cb) = on_progress {
let pp = crate::progress::PassProgress {
kind: crate::progress::PassKind::Verify,
work_done: sectors_done,
work_total: total_sectors,
bytes_good_total: (good + slow + recovered) * 2048,
bytes_unreadable_total: bad * 2048,
bytes_pending_total: 0,
bytes_total_disc: total_sectors * 2048,
disc_duration_secs: Some(title.duration_secs),
bytes_bad_in_main_title: 0,
main_title_duration_secs: Some(title.duration_secs),
main_title_size_bytes: Some(total_sectors * 2048),
};
if !cb.report(&pp) {
break 'outer;
}
}
+2 -2
View File
@@ -1,6 +1,6 @@
//! Disc scanning pipeline tests.
use libfreemkv::SectorReader;
use libfreemkv::SectorSource;
use libfreemkv::error::Result;
use libfreemkv::{Disc, DiscTitle, ScanOptions};
use std::collections::HashMap;
@@ -20,7 +20,7 @@ impl MockSectorReader {
}
}
impl SectorReader for MockSectorReader {
impl SectorSource for MockSectorReader {
fn read_sectors(
&mut self,
lba: u32,
+73 -44
View File
@@ -5,8 +5,8 @@ use libfreemkv::disc::{CopyOptions, DiscRegion};
use libfreemkv::error::Result;
use libfreemkv::pes::Stream as PesStream;
use libfreemkv::{
ContentFormat, Disc, DiscFormat, DiscStream, DiscTitle, EventKind, Extent, FileSectorReader,
SectorReader,
ContentFormat, Disc, DiscFormat, DiscStream, DiscTitle, EventKind, Extent, FileSectorSource,
SectorSource,
};
use std::io::Write;
use std::sync::Arc;
@@ -32,7 +32,7 @@ impl ZeroSectorReader {
}
}
impl SectorReader for ZeroSectorReader {
impl SectorSource for ZeroSectorReader {
fn read_sectors(
&mut self,
_lba: u32,
@@ -46,7 +46,7 @@ impl SectorReader for ZeroSectorReader {
Ok(bytes)
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
@@ -67,7 +67,7 @@ impl SlowZeroSectorReader {
}
}
impl SectorReader for SlowZeroSectorReader {
impl SectorSource for SlowZeroSectorReader {
fn read_sectors(
&mut self,
_lba: u32,
@@ -81,7 +81,7 @@ impl SectorReader for SlowZeroSectorReader {
Ok(bytes)
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
@@ -101,6 +101,7 @@ fn synthetic_disc(capacity_sectors: u32) -> Disc {
aacs: None,
css: None,
encrypted: false,
aacs_error: None,
content_format: ContentFormat::BdTs,
}
}
@@ -181,9 +182,10 @@ fn test_disc_copy_progress_callback_fires() {
last_bytes: Arc<AtomicU64>,
}
impl libfreemkv::progress::Progress for CountingReporter {
fn report(&self, p: &libfreemkv::progress::PassProgress) {
fn report(&self, p: &libfreemkv::progress::PassProgress) -> bool {
self.calls.fetch_add(1, Ordering::Relaxed);
self.last_bytes.store(p.bytes_good_total, Ordering::Relaxed);
true
}
}
let reporter = CountingReporter {
@@ -302,7 +304,7 @@ fn test_drop_impls_do_not_panic_or_block() {
panic!("DiscStream drop did not complete within 100ms");
}
// ── 5. FileSectorReader round trip ────────────────────────────────────────
// ── 5. FileSectorSource round trip ────────────────────────────────────────
#[test]
fn test_file_sector_reader_round_trip() {
@@ -318,10 +320,14 @@ fn test_file_sector_reader_round_trip() {
tmp.write_all(&data).expect("write data");
tmp.flush().expect("flush");
let path = tmp.path().to_str().expect("path utf-8").to_string();
let mut fsr = FileSectorReader::open(&path).expect("open FileSectorReader");
let path = tmp.path().to_path_buf();
let mut fsr = FileSectorSource::open(&path).expect("open FileSectorSource");
assert_eq!(fsr.capacity(), N_SECTORS as u32, "capacity mismatch");
assert_eq!(
fsr.capacity_sectors(),
N_SECTORS as u32,
"capacity mismatch"
);
// Read each sector individually and compare.
let mut buf = vec![0u8; SECTOR_SIZE];
@@ -389,7 +395,7 @@ impl FailingSectorReader {
}
}
impl SectorReader for FailingSectorReader {
impl SectorSource for FailingSectorReader {
fn read_sectors(
&mut self,
_lba: u32,
@@ -416,7 +422,7 @@ impl SectorReader for FailingSectorReader {
})
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
@@ -453,11 +459,17 @@ fn test_disc_copy_completes_full_disc_with_failing_reader() {
let _ = std::fs::remove_file(&iso_path);
let _ = std::fs::remove_file(libfreemkv::disc::mapfile_path_for(&iso_path));
// Hard bound — even at 0 ms per read, 1024 sectors with skip-forward
// should complete in well under a second on any host.
// Hard bound — Pass 1 must NOT infinite-loop on a fully-failing
// reader. The threshold accommodates the 2026-05-10 wedge-
// avoidance pause (PASS_1_FAIL_PAUSE_SECS = 5 s on each failed
// batch). With batch=32 and 1024 sectors that's up to ~5 batch
// failures + a few damage-jump pauses before fast-trigger jumps
// us past end-of-disc — well-bounded total, ~20-30 s typical.
// The point of this test is "finishes cleanly, not infinitely",
// not "completes in milliseconds."
assert!(
elapsed < Duration::from_secs(5),
"Pass 1 took {elapsed:?} on a 2 MB synthetic disc — expected < 5 s"
elapsed < Duration::from_secs(60),
"Pass 1 took {elapsed:?} on a 2 MB synthetic disc — expected < 60 s (not infinite)"
);
// Per RIP_DESIGN.md §2.1: Pass 1 must reach end of disc regardless of
@@ -559,7 +571,7 @@ struct BlockSizeFailingReader {
capacity: u32,
}
impl SectorReader for BlockSizeFailingReader {
impl SectorSource for BlockSizeFailingReader {
fn read_sectors(
&mut self,
lba: u32,
@@ -585,7 +597,7 @@ impl SectorReader for BlockSizeFailingReader {
}
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
@@ -593,7 +605,7 @@ impl SectorReader for BlockSizeFailingReader {
#[test]
fn test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed() {
let capacity_sectors: u32 = 256;
let _total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
let mut reader = BlockSizeFailingReader {
capacity: capacity_sectors,
@@ -618,31 +630,41 @@ fn test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed() {
let _ = std::fs::remove_file(&iso_path);
let _ = std::fs::remove_file(libfreemkv::disc::mapfile_path_for(&iso_path));
// Pass 1 reads every batch at bpt=32 (no batch reduction, no skip-ahead).
// BlockSizeFailingReader fails on all batch=32 reads, so every sector
// is marked NonTrimmed. bytes_good=0 is correct — Pass 2 recovers them.
// Pass 1's job is "fast and accurate, get the most data in the
// shortest time." It no longer bisects on marginal media — that's
// Pass N's purpose-built role. So a BlockSizeFailingReader that
// fails on multi-sector reads and succeeds on single-sector
// results in: every batch fails → SkipBlock → whole 32-sector
// ECC block marked NonTrimmed → Pass N (Disc::patch) revisits and
// recovers via single-sector reads with proper recovery semantics.
//
// Pass 1 alone:
assert_eq!(
result.bytes_good, 0,
"Pass 1 at bpt=32 should not recover any sectors when all batches fail"
"Pass 1 doesn't bisect on marginal media — failed batches become NonTrimmed for Pass N to revisit"
);
assert!(
result.bytes_pending > 0,
"all sectors should be NonTrimmed pending Pass 2"
assert_eq!(
result.bytes_pending, total_bytes,
"every sector is NonTrimmed (pending) after Pass 1, awaiting Pass N"
);
assert!(
!result.complete,
"complete=false when sectors remain NonTrimmed"
"complete=false because NonTrimmed regions remain (Pass N's work)"
);
}
// ── 9. PassProgress carries separate unreadable vs pending byte counts ─────
//
// The video-damage-time display needs bytes_unreadable_total (confirmed dead)
// separate from bytes_pending_total (might still recover). This test verifies
// that a Pass 2 with some confirmed failures produces correct field values.
// 2026-05-11 design call: Pass N never marks bytes as `Unreadable` mid-multipass —
// failed reads stay `NonTrimmed` so the next pass can retry them. The orchestrator
// (autorip) promotes still-NonTrimmed bytes to Unreadable after the FINAL retry
// pass completes. This test was rewritten from its pre-design-call shape (which
// asserted Pass 2 produced bytes_unreadable > 0) to verify the new invariant:
// pass-level retries keep failed bytes in `bytes_pending` so subsequent passes
// get more shots at them.
#[test]
fn test_pass_progress_separates_unreadable_from_pending() {
fn test_pass2_leaves_failed_reads_as_pending_not_unreadable() {
let capacity_sectors: u32 = 128;
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
@@ -680,7 +702,7 @@ fn test_pass_progress_separates_unreadable_from_pending() {
dur: Arc<AtomicU64>,
}
impl libfreemkv::progress::Progress for SnapshotReporter {
fn report(&self, p: &libfreemkv::progress::PassProgress) {
fn report(&self, p: &libfreemkv::progress::PassProgress) -> bool {
self.unreadable
.store(p.bytes_unreadable_total, Ordering::Relaxed);
self.pending.store(p.bytes_pending_total, Ordering::Relaxed);
@@ -688,6 +710,7 @@ fn test_pass_progress_separates_unreadable_from_pending() {
if let Some(d) = p.disc_duration_secs {
self.dur.store((d * 1000.0) as u64, Ordering::Relaxed);
}
true
}
}
let reporter = SnapshotReporter {
@@ -715,24 +738,30 @@ fn test_pass_progress_separates_unreadable_from_pending() {
pass2.bytes_good, 0,
"pass2: still no good sectors (reader always fails)"
);
assert!(
pass2.bytes_unreadable > 0,
"pass2: some sectors confirmed unreadable"
// 2026-05-11 design: pass-level retries do NOT promote failed bytes
// to Unreadable. Failed bytes stay NonTrimmed (pending) so a later
// pass can retry. End-of-recovery promotion is an orchestrator
// concern (autorip), not the patch loop's.
assert_eq!(
pass2.bytes_unreadable, 0,
"pass2: Disc::patch never marks Unreadable mid-multipass — orchestrator promotes after final pass"
);
assert!(
pass2.bytes_pending < pass1.bytes_pending,
"pass2: fewer pending sectors than pass1"
// bytes_pending stays at total_bytes because everything still
// failed and nothing got recovered or promoted out of pending.
assert_eq!(
pass2.bytes_pending, total_bytes,
"pass2: failed bytes remain NonTrimmed for the next pass to retry"
);
let observed_unreadable = last_unreadable.load(Ordering::Relaxed);
let observed_pending = last_pending.load(Ordering::Relaxed);
assert!(
observed_unreadable > 0,
"progress should report confirmed unreadable bytes"
assert_eq!(
observed_unreadable, 0,
"progress should report zero confirmed-unreadable mid-pass under the new design"
);
assert!(
observed_pending == 0 || observed_pending < total_bytes,
"pending should shrink as sectors are confirmed unreadable"
observed_pending > 0,
"progress should report pending bytes as the reader keeps failing"
);
// Video damage time: unreadable / total * duration
+119
View File
@@ -0,0 +1,119 @@
//! Regression tests for Pass N (patch) fix — decrypt key inversion bug.
//!
//! Tests that decrypt_sectors is invoked correctly when opts.decrypt=true.
//! The 2026-05-03 bug at `libfreemkv/src/disc/mod.rs:1938-1942` inverted
//! the decrypt key arms, causing patch to pass DecryptKeys::None on encrypted discs.
use libfreemkv::{aacs, decrypt::DecryptKeys};
/// Test: decrypt_sectors with AACS keys actually decrypts units.
#[test]
fn decrypt_sectors_with_aacs_keys_works() {
// Build an encrypted aligned unit
let mut unit = vec![0xFFu8; aacs::ALIGNED_UNIT_LEN];
// Set encryption flag (bits 6-7 of byte 0)
unit[0] |= 0xC0;
// Fill with recognizable pattern
for (i, byte) in unit
.iter_mut()
.enumerate()
.take(aacs::ALIGNED_UNIT_LEN)
.skip(1)
{
*byte = ((i * 3 + 7) & 0xFF) as u8;
}
let unit_key: [u8; 16] = [0xAAu8; 16];
// Encrypt the unit using AACS algorithm
aacs::decrypt_unit(&mut unit, &unit_key); // decrypt_unit is idempotent on already-encrypted data
// Now we have encrypted data - create DecryptKeys with actual keys
let keys = DecryptKeys::Aacs {
unit_keys: vec![(0u32, unit_key)],
read_data_key: None,
};
// decrypt_sectors should handle this without error
let result = libfreemkv::decrypt::decrypt_sectors(&mut unit, &keys, 0);
assert!(
result.is_ok(),
"decrypt_sectors with AACS keys should not error"
);
}
/// Test: decrypt_sectors with DecryptKeys::None is a no-op.
#[test]
fn decrypt_sectors_with_none_keys_is_noop() {
let mut sector = vec![0x42u8; 2048];
let keys = DecryptKeys::None;
let result = libfreemkv::decrypt::decrypt_sectors(&mut sector, &keys, 0);
assert!(result.is_ok());
assert_eq!(
&sector[..],
&[0x42u8; 2048][..],
"DecryptKeys::None should not modify buffer"
);
}
/// Test: decrypt_sectors with CSS keys descrambles sectors.
#[test]
fn decrypt_sectors_with_css_keys_works() {
let mut sector = vec![0xFFu8; 2048];
// Set CSS scramble flag (bits 4-5 of byte 0x14)
sector[0x14] |= 0x30;
let title_key: [u8; 5] = [0x42, 0x13, 0x37, 0xBE, 0xEF]; // Not used - defined later
let keys = DecryptKeys::Css { title_key };
// Descramble (CSS uses same operation for encrypt/decrypt)
libfreemkv::decrypt::decrypt_sectors(&mut sector, &keys, 0).unwrap();
// Flag should be cleared
assert_eq!(sector[0x14] & 0x30, 0x00, "CSS flag should be cleared");
}
/// Test: AACS unit encryption detection works.
#[test]
fn aacs_encryption_flag_detection() {
let mut unit = vec![0u8; aacs::ALIGNED_UNIT_LEN];
// No encryption flag
assert!(!aacs::is_unit_encrypted(&unit));
// Set bit 6
unit[0] |= 0x40;
assert!(aacs::is_unit_encrypted(&unit));
// Set bit 7
unit[0] = 0x80;
assert!(aacs::is_unit_encrypted(&unit));
// Both bits set
unit[0] = 0xC0;
assert!(aacs::is_unit_encrypted(&unit));
}
/// Test: DecryptKeys::is_encrypted() correctly identifies encrypted state.
#[test]
fn decrypt_keys_is_encrypted_variants() {
let none = DecryptKeys::None;
assert!(!none.is_encrypted());
let aacs = DecryptKeys::Aacs {
unit_keys: vec![],
read_data_key: None,
};
assert!(aacs.is_encrypted());
let css = DecryptKeys::Css {
title_key: [0u8; 5],
};
assert!(css.is_encrypted());
}
+408
View File
@@ -0,0 +1,408 @@
//! Pass N (Disc::patch) size-aware-skip targeted tests.
//!
//! The user's failure mode (2026-05-07): "what if we have a 100 sector zone
//! and its really 2 25 sector zones and we keep jumping over the good in
//! the middle." Today's pre-fix patch escalates skip-distance based on
//! `consecutive_skips_without_recovery` with hardcoded 32 → 4096 sector
//! caps. A 100-sector bad range whose actual layout is 25 bad + 50 good +
//! 25 bad would have the patch skip 32-4096 sectors after a couple of
//! failures, leaping over the entire range AND the good middle.
//!
//! The fix: cap each skip at `range_remaining/4`. These tests exercise
//! that boundary.
use libfreemkv::disc::CopyOptions;
use libfreemkv::disc::DiscRegion;
use libfreemkv::disc::mapfile::{Mapfile, SectorStatus};
use libfreemkv::error::Result;
use libfreemkv::{ContentFormat, Disc, DiscFormat, SectorSource};
use std::collections::HashSet;
use std::sync::{Arc, Mutex};
const SECTOR_SIZE: usize = 2048;
/// Reader where you specify exactly which LBAs return Err. Everything else
/// returns Ok with the LBA encoded in each byte for verification.
struct PatternedSectorReader {
capacity: u32,
bad_lbas: HashSet<u32>,
/// Trace every read so tests can assert what was actually attempted.
trace: Arc<Mutex<Vec<(u32, u16)>>>,
}
type ReadTrace = Arc<Mutex<Vec<(u32, u16)>>>;
impl PatternedSectorReader {
fn new(capacity: u32, bad_lbas: HashSet<u32>) -> (Self, ReadTrace) {
let trace = Arc::new(Mutex::new(Vec::new()));
(
Self {
capacity,
bad_lbas,
trace: trace.clone(),
},
trace,
)
}
}
impl SectorSource for PatternedSectorReader {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
self.trace.lock().unwrap().push((lba, count));
// Whole-batch fails if ANY sector in the batch is bad. (Models a
// real drive: a multi-sector READ aborts on the first ECC failure.)
for offset in 0..count as u32 {
if self.bad_lbas.contains(&(lba + offset)) {
return Err(libfreemkv::error::Error::ScsiError {
opcode: libfreemkv::scsi::SCSI_READ_10,
status: libfreemkv::scsi::SCSI_STATUS_CHECK_CONDITION,
sense: Some(libfreemkv::ScsiSense {
sense_key: libfreemkv::scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
}),
});
}
}
// Fill each sector with ITS OWN LBA byte, not the starting LBA's
// byte. This matches real drive behavior: a multi-sector READ
// returns per-sector-correct data. Pre-0.18.13 only single-sector
// reads were exercised by patch tests, so the cheaper "fill the
// whole batch with one byte" worked; adaptive batching needs the
// per-sector pattern to verify correct positioning.
for (i, chunk) in buf.chunks_mut(SECTOR_SIZE).enumerate() {
chunk.fill(((lba + i as u32) & 0xff) as u8);
}
Ok(buf.len())
}
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
fn synthetic_disc(capacity_sectors: u32) -> Disc {
Disc {
volume_id: String::new(),
meta_title: None,
format: DiscFormat::BluRay,
capacity_sectors,
capacity_bytes: capacity_sectors as u64 * SECTOR_SIZE as u64,
layers: 1,
titles: Vec::new(),
region: DiscRegion::Free,
aacs: None,
css: None,
encrypted: false,
aacs_error: None,
content_format: ContentFormat::BdTs,
}
}
/// Pre-populate a mapfile with one large NonTrimmed range so patch's work-
/// list has something to do. Caller pre-allocates the ISO at `total_bytes`
/// so seeks don't fail.
fn prep_iso_and_mapfile(
iso_path: &std::path::Path,
total_bytes: u64,
finished_ranges: &[(u64, u64)],
nontrimmed_ranges: &[(u64, u64)],
) {
use std::fs::OpenOptions;
use std::io::{Seek, SeekFrom, Write};
let mut f = OpenOptions::new()
.create(true)
.write(true)
.truncate(true)
.open(iso_path)
.unwrap();
f.set_len(total_bytes).unwrap();
f.seek(SeekFrom::Start(0)).unwrap();
f.write_all(&[]).unwrap();
let map_path = libfreemkv::disc::mapfile_path_for(iso_path);
let mut mf = Mapfile::create(&map_path, total_bytes, "test").unwrap();
for &(pos, size) in finished_ranges {
mf.record(pos, size, SectorStatus::Finished).unwrap();
}
for &(pos, size) in nontrimmed_ranges {
mf.record(pos, size, SectorStatus::NonTrimmed).unwrap();
}
}
/// THE critical test. A 100-sector "bad" range hides 50 good sectors in
/// the middle (LBAs 125-174). Pre-fix patch would skip-escalate at 32+
/// sectors and leap over the whole range. Post-fix: skip is capped at
/// range_remaining/4 (=25 sectors initially), which forces convergence.
#[test]
fn patch_recovers_good_middle_of_a_bad_range() {
let capacity_sectors: u32 = 1024;
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
// Bad range layout: LBAs 100-124 bad, 125-174 GOOD, 175-199 bad.
let mut bad_lbas = HashSet::new();
for lba in 100..125 {
bad_lbas.insert(lba);
}
for lba in 175..200 {
bad_lbas.insert(lba);
}
let (mut reader, _trace) = PatternedSectorReader::new(capacity_sectors, bad_lbas);
let disc = synthetic_disc(capacity_sectors);
let tmp = tempfile::NamedTempFile::new().unwrap();
let iso_path = tmp.path().to_path_buf();
drop(tmp);
// Pre-populate: 0..100 already Finished from an imagined Pass 1,
// 100..200 NonTrimmed (the range we want patch to retry),
// 200..1024 already Finished.
let finished = [
(0, 100 * 2048),
(200 * 2048, (capacity_sectors as u64 - 200) * 2048),
];
let nontrimmed = [(100 * 2048, 100 * 2048)];
prep_iso_and_mapfile(&iso_path, total_bytes, &finished, &nontrimmed);
// Run patch.
// disc.copy() with multipass=true auto-dispatches to patch when the
// mapfile already covers the disc and has retryable ranges.
let opts = CopyOptions {
decrypt: false,
multipass: true,
..Default::default()
};
let pr = disc
.copy(&mut reader, &iso_path, &opts)
.expect("copy returns Ok");
// Re-load mapfile and inspect.
let map_path = libfreemkv::disc::mapfile_path_for(&iso_path);
let map = Mapfile::load(&map_path).unwrap();
// The good middle (125..175) MUST end up Finished. If size-aware skip
// is not enabled, patch would skip 32+ sectors after a few failures
// and leap clean over LBA 125 → middle stays NonTrimmed/Unreadable.
let finished_ranges = map.ranges_with(&[SectorStatus::Finished]);
let total_finished_in_middle: u64 = finished_ranges
.iter()
.map(|&(pos, sz)| {
let start = pos.max(125 * 2048);
let end = (pos + sz).min(175 * 2048);
end.saturating_sub(start)
})
.sum();
// Allow 2 sectors (4 KB) of boundary slop — patch's bisection may
// not converge exactly on the good/bad boundary in a single pass,
// and that's acceptable. The pre-fix behaviour would have left the
// entire good middle as NonTrimmed (~0 bytes recovered).
let good_middle_bytes: u64 = 50 * 2048;
let min_acceptable: u64 = good_middle_bytes - 2 * 2048;
// Cleanup before assertions
let _ = std::fs::remove_file(&iso_path);
let _ = std::fs::remove_file(&map_path);
assert!(
total_finished_in_middle >= min_acceptable,
"size-aware skip should have discovered most of the 50 good sectors in the middle. \
Recovered {} of {} good middle bytes (min acceptable {}). bytes_good={} bytes_total={}",
total_finished_in_middle,
good_middle_bytes,
min_acceptable,
pr.bytes_good,
pr.bytes_total,
);
}
/// A second test: a bad range that's actually 4 small bad sub-zones
/// separated by good sectors. Demonstrates the bisection behaviour
/// converges when zones are non-uniform.
#[test]
fn patch_recovers_multiple_good_middles() {
let capacity_sectors: u32 = 2048;
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
// Bad pattern: 1000-1024 bad, 1025-1099 good, 1100-1124 bad,
// 1125-1199 good, 1200-1224 bad, 1225-1299 good.
let mut bad_lbas = HashSet::new();
for lba in 1000..1025 {
bad_lbas.insert(lba);
}
for lba in 1100..1125 {
bad_lbas.insert(lba);
}
for lba in 1200..1225 {
bad_lbas.insert(lba);
}
let (mut reader, _trace) = PatternedSectorReader::new(capacity_sectors, bad_lbas);
let disc = synthetic_disc(capacity_sectors);
let tmp = tempfile::NamedTempFile::new().unwrap();
let iso_path = tmp.path().to_path_buf();
drop(tmp);
let finished = [
(0, 1000 * 2048),
(1300 * 2048, (capacity_sectors as u64 - 1300) * 2048),
];
let nontrimmed = [(1000 * 2048, 300 * 2048)];
prep_iso_and_mapfile(&iso_path, total_bytes, &finished, &nontrimmed);
let opts = CopyOptions {
decrypt: false,
multipass: true,
..Default::default()
};
let pr = disc
.copy(&mut reader, &iso_path, &opts)
.expect("copy returns Ok");
let map_path = libfreemkv::disc::mapfile_path_for(&iso_path);
let map = Mapfile::load(&map_path).unwrap();
let finished_ranges = map.ranges_with(&[SectorStatus::Finished]);
let recovered: u64 = finished_ranges
.iter()
.map(|&(pos, sz)| {
let start = pos.max(1000 * 2048);
let end = (pos + sz).min(1300 * 2048);
end.saturating_sub(start)
})
.sum();
let _ = std::fs::remove_file(&iso_path);
let _ = std::fs::remove_file(&map_path);
// Three good middles of 75 sectors each = 225 good sectors in the
// bad range. Total bad = 75. So we want at least most of 225 sectors
// (= 460800 bytes) to be Finished after patch.
let target = 200 * 2048; // be generous — anything over 200 sectors is convincing
assert!(
recovered >= target,
"size-aware skip should find most of the 3 good middles. \
Recovered {} bytes; expected ≥ {}. bytes_good={} bytes_total={}",
recovered,
target,
pr.bytes_good,
pr.bytes_total,
);
}
/// 0.18 Pass N pipeline split: exercises the new producer/consumer
/// path end-to-end on a synthetic patterned reader. Bad range layout
/// is small (5 bad LBAs surrounded by good middle) so the producer
/// emits a mix of `Recovered` and `Unreadable` items and the consumer
/// thread must apply both kinds. Verifies:
///
/// - `bytes_good` advances (good sectors flow producer→consumer→file
/// →mapfile with the data preserved).
/// - The recovered LBAs end up Finished; the bad LBAs end up Unreadable.
/// - Bytes written at the recovered offsets match what the producer
/// read from the patterned source (proves the channel hand-off
/// didn't drop or reorder buffers, and the consumer's seek+write
/// landed at the right offsets).
#[test]
fn patch_pipeline_split_recovers_and_records_correctly() {
let capacity_sectors: u32 = 512;
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
// Layout: LBAs 200-204 inclusive are bad (5 sectors), 205-249 good.
// The pre-existing range is LBAs 200-249 NonTrimmed (100 KB).
let mut bad_lbas = HashSet::new();
for lba in 200..205 {
bad_lbas.insert(lba);
}
let (mut reader, _trace) = PatternedSectorReader::new(capacity_sectors, bad_lbas.clone());
let disc = synthetic_disc(capacity_sectors);
let tmp = tempfile::NamedTempFile::new().unwrap();
let iso_path = tmp.path().to_path_buf();
drop(tmp);
let finished = [
(0, 200 * 2048),
(250 * 2048, (capacity_sectors as u64 - 250) * 2048),
];
let nontrimmed = [(200 * 2048, 50 * 2048)];
prep_iso_and_mapfile(&iso_path, total_bytes, &finished, &nontrimmed);
let opts = CopyOptions {
decrypt: false,
multipass: true,
..Default::default()
};
let pr = disc
.copy(&mut reader, &iso_path, &opts)
.expect("copy returns Ok");
// Bytes_good_total should advance — the good LBAs in the bad range
// (205-249, 45 sectors) are all reachable via per-sector retry.
// Initial bytes_good = 200 * 2048 + (512-250) * 2048 = 462 sectors.
// After patch, bytes_good should be ≥ 462 + 45 = 507 sectors worth.
let initial_good_sectors: u64 = 200 + (capacity_sectors as u64 - 250);
let min_expected_good_bytes = (initial_good_sectors + 30) * 2048;
assert!(
pr.bytes_good >= min_expected_good_bytes,
"patch should have recovered most good LBAs in the bad range via the pipeline. \
bytes_good={} (expected ≥ {}); bytes_total={}",
pr.bytes_good,
min_expected_good_bytes,
pr.bytes_total,
);
// Verify the mapfile records: every good LBA is Finished, every
// bad LBA is either Unreadable or NonTrimmed (not Finished).
let map_path = libfreemkv::disc::mapfile_path_for(&iso_path);
let map = Mapfile::load(&map_path).unwrap();
let finished_ranges = map.ranges_with(&[SectorStatus::Finished]);
let in_finished = |lba: u32| -> bool {
let pos = lba as u64 * 2048;
finished_ranges
.iter()
.any(|&(p, sz)| pos >= p && pos < p + sz)
};
for lba in 205..250 {
assert!(
in_finished(lba),
"good LBA {lba} should be Finished after pipeline patch run"
);
}
for lba in 200..205 {
assert!(
!in_finished(lba),
"bad LBA {lba} should NOT be Finished after pipeline patch run"
);
}
// Verify the consumer wrote the producer's bytes at the right
// offsets. PatternedSectorReader fills each sector with `(lba & 0xff)
// as u8` — picking LBA 220 (well inside the recovered region) gives
// a clean signature byte to check.
use std::io::{Read, Seek, SeekFrom};
let mut iso = std::fs::File::open(&iso_path).unwrap();
iso.seek(SeekFrom::Start(220 * 2048)).unwrap();
let mut sector = [0u8; 2048];
iso.read_exact(&mut sector).unwrap();
let expected_byte = (220u32 & 0xff) as u8;
let _ = std::fs::remove_file(&iso_path);
let _ = std::fs::remove_file(&map_path);
assert!(
sector.iter().all(|&b| b == expected_byte),
"consumer should have written PatternedSectorReader's pattern \
(byte {expected_byte:#x} for LBA 220) to the recovered offset; \
got first 8 bytes = {:?}",
&sector[..8]
);
}
+3 -1
View File
@@ -9,7 +9,6 @@
//! These tests require a real /dev/sg* device and are therefore #[ignore].
use std::path::Path;
use std::time::Duration;
#[test]
#[ignore]
@@ -25,6 +24,7 @@ fn test_sgio_transport_timeout_does_not_kill_transport() {
use libfreemkv::scsi::{DataDirection, SCSI_STATUS_TRANSPORT_FAILURE};
use std::sync::Arc;
use std::sync::atomic::{AtomicI32, Ordering};
use std::time::Duration;
let mut transport = SgIoTransport::open(_path).expect("open device");
let fd_before = transport.fd;
@@ -94,6 +94,8 @@ fn test_drive_read_per_cdb_timeout_bounds_call() {
#[cfg(target_os = "linux")]
{
use std::time::Duration;
let mut drive = libfreemkv::Drive::open(_path).expect("open drive");
let timeout_ms: u32 = 5_000;
+20 -19
View File
@@ -302,12 +302,12 @@ fn m2ts_stream_write_read() {
keyframe: i == 0,
data: vec![i; 100],
};
stream.write(&frame).unwrap();
PesStream::write(&mut stream, &frame).unwrap();
}
stream.finish().unwrap();
PesStream::finish(&mut stream).unwrap();
// Verify the info is correct
let info = stream.info();
let info = PesStream::info(&stream);
assert_eq!(info.streams.len(), 4);
assert_eq!(info.duration_secs, 7200.0);
}
@@ -356,7 +356,7 @@ fn m2ts_read_returns_error_on_write_stream() {
let dt = sample_disc_title();
let output = Cursor::new(Vec::new());
let mut stream = M2tsStream::create(output, &dt).unwrap();
assert!(stream.read().is_err());
assert!(PesStream::read(&mut stream).is_err());
}
// ── DiscTitle::empty ──────────────────────────────────────────
@@ -578,7 +578,7 @@ fn meta_all_stream_types() {
#[test]
fn mkvstream_write_finish() {
let dt = sample_disc_title();
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(Cursor::new(Vec::new()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> = Box::new(Cursor::new(Vec::new()));
let mut stream = MkvStream::create(writer, &dt).unwrap();
// Write some fake PES frames (they won't produce valid MKV content
@@ -590,20 +590,20 @@ fn mkvstream_write_finish() {
keyframe: i == 0,
data: vec![i; 100],
};
stream.write(&frame).unwrap();
PesStream::write(&mut stream, &frame).unwrap();
}
// finish should not panic even without valid codec data
stream.finish().unwrap();
PesStream::finish(&mut stream).unwrap();
}
#[test]
fn mkvstream_meta_sets_title() {
let dt = sample_disc_title();
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(Cursor::new(Vec::new()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> = Box::new(Cursor::new(Vec::new()));
let stream = MkvStream::create(writer, &dt).unwrap();
let info = stream.info();
let info = PesStream::info(&stream);
assert_eq!(info.playlist, "Test Movie");
assert_eq!(info.duration_secs, 7200.0);
assert_eq!(info.streams.len(), 4);
@@ -640,7 +640,7 @@ fn mkvstream_roundtrip_bdts() {
codec_privates: Vec::new(),
};
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(Cursor::new(Vec::new()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> = Box::new(Cursor::new(Vec::new()));
let mut stream = MkvStream::create(writer, &dt).unwrap();
// Write PES frames targeting the audio track
@@ -651,13 +651,13 @@ fn mkvstream_roundtrip_bdts() {
keyframe: true,
data: vec![i; 100],
};
stream.write(&frame).unwrap();
PesStream::write(&mut stream, &frame).unwrap();
}
stream.finish().unwrap();
PesStream::finish(&mut stream).unwrap();
// Verify the info is correct
let info = stream.info();
let info = PesStream::info(&stream);
assert_eq!(info.streams.len(), 1);
assert_eq!(info.playlist, "Audio Only");
}
@@ -724,10 +724,10 @@ fn mkvstream_meta_preserves_all_streams() {
codec_privates: Vec::new(),
};
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(Cursor::new(Vec::new()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> = Box::new(Cursor::new(Vec::new()));
let stream = MkvStream::create(writer, &dt).unwrap();
let info = stream.info();
let info = PesStream::info(&stream);
assert_eq!(info.streams.len(), 5, "all 5 streams should be preserved");
assert_eq!(info.playlist, "Stream Test");
assert_eq!(info.duration_secs, 3600.0);
@@ -832,7 +832,8 @@ fn mkvstream_e2e_h264_produces_valid_mkv() {
}
}
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(SharedWriter(output2.clone()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> =
Box::new(SharedWriter(output2.clone()));
let mut stream2 = MkvStream::create(writer, &dt).unwrap();
// Write the ES data (SPS+PPS+IDR) as a keyframe PES frame.
@@ -842,7 +843,7 @@ fn mkvstream_e2e_h264_produces_valid_mkv() {
keyframe: true,
data: es_data,
};
stream2.write(&frame1).unwrap();
PesStream::write(&mut stream2, &frame1).unwrap();
// Write a second non-IDR frame
let frame2 = libfreemkv::pes::PesFrame {
@@ -851,8 +852,8 @@ fn mkvstream_e2e_h264_produces_valid_mkv() {
keyframe: false,
data: es_data2,
};
stream2.write(&frame2).unwrap();
stream2.finish().unwrap();
PesStream::write(&mut stream2, &frame2).unwrap();
PesStream::finish(&mut stream2).unwrap();
let data = output2.lock().unwrap().clone().into_inner();
+4 -4
View File
@@ -1,7 +1,7 @@
//! UDF parser tests using a MockSectorReader.
use libfreemkv::error::Result;
use libfreemkv::{SectorReader, read_filesystem};
use libfreemkv::{SectorSource, read_filesystem};
use std::collections::HashMap;
const SECTOR_SIZE: usize = 2048;
@@ -38,7 +38,7 @@ impl MockSectorReader {
}
}
impl SectorReader for MockSectorReader {
impl SectorSource for MockSectorReader {
fn read_sectors(
&mut self,
lba: u32,
@@ -463,11 +463,11 @@ fn find_dir_case_insensitive() {
#[test]
fn sector_reader_is_object_safe() {
// Verify SectorReader can be used as a trait object
// Verify SectorSource can be used as a trait object
let mut reader = MockSectorReader::new();
reader.set_sector(0, vec![42u8; SECTOR_SIZE]);
let dyn_reader: &mut dyn SectorReader = &mut reader;
let dyn_reader: &mut dyn SectorSource = &mut reader;
let mut buf = vec![0u8; SECTOR_SIZE];
let n = dyn_reader.read_sectors(0, 1, &mut buf, true).unwrap();
assert_eq!(n, SECTOR_SIZE);