Compare commits

Author SHA1 Message Date
matthew 125a8e5bf0 aacs: full-ladder resolve_candidate + correct mk_from_dk
leak-guard / leak-guard (push) Successful in 24s
resolve_candidate(candidate, mkb, unit_key_ro, vid) -> ResolvedChain: one
composed, PURE-DERIVATION boil-down for a candidate key at any ladder rung
(KeyCandidate::{Uk,Vuk,Mk,Pk,Dk}, each carrying the module's newtype). Walks
DK/PK/MK/VUK -> terminal unit keys, parsing Unit_Key_RO.inf at the version
the disc's MKB declares, and returns every CPS unit key as (cps_unit, key)
(matching ResolvedKeys) plus the intermediate chain. No sampling/validation
(that's the caller's unit_key_validates) and no position recovery (Dk is a
positioned DeviceKey; recover_dk_position first). Adds ProcessingKey newtype.
Consumers stop re-composing the ladder; every client hardens one impl.

Correct mk_from_dk: real Subset-Difference walk (derive_media_key_and_pk_from_dk)
instead of the Media-Key-Variant path, which needed integrator KCD absent
in-tree and Err'd for every real disc -- dead for both consumers. Drops the
now-unused vid arg.
2026-07-01 22:34:52 -07:00
Matthew Jackson 741c1ea11d restore freemkv-unlock path dep for local dev (post-v1.2.1) 2026-07-01 19:08:56 -07:00
Matthew Jackson e713b26b87 v1.2.1: bump version (freemkv-unlock git-pinned for the tag) 2026-07-01 19:08:53 -07:00
Matthew Jackson fcbd667add changelog: describe the shipped re-base DTS fix + faithful-passthrough note 2026-07-01 19:02:04 -07:00
Matthew Jackson 88c03152e2 mux/dts: re-base PTS per PES instead of a running clock (fix long-title drift)
The first cut used a global running clock (max(next, own-PES PTS) + advance),
which fixed the same-PES collision but DRIFTED: once accumulated frame
durations exceeded the PES-timestamp spacing, it never re-based, so a
feature-long DVD DTS track ran minutes past its real length (2h44 for a 2h03
film) while AC-3 from the same source stayed exact.

Match the AC-3 path: re-base to each PES's own container timestamp, and advance
by one frame duration ONLY within a run of AUs sharing one PES. Fixes the DVD
multi-frame-per-PES collision without drift; the UHD DTS-HD MA per-PES
attribution (da85f56) is preserved (each AU still takes its own core PES's PTS).
Adds new_pes_rebases_to_its_own_pts_no_drift; full mux suite green (905).
2026-07-01 18:21:25 -07:00
Matthew Jackson c8e7ad5e56 v1.2.1: bump version + changelog (DVD DTS timestamp fix) 2026-07-01 17:30:25 -07:00
Matthew Jackson f122f08628 mux/dts: monotonic per-frame PTS + real frame duration (fixes DVD DTS)
DVD packs several DTS core frames into one PES; the parser stamped every
access unit with that single PES PTS and duration_ns=None, so consecutive
frames collided on one timestamp — ffmpeg rejected the output as 'non
monotonically increasing dts to muxer: X >= X' (deep-decode = corrupt,
e.g. The Punisher). The UHD DTS-HD MA path (one AU per PES, distinct PTS)
was unaffected, which is why this only surfaced on DVD.

Parse the DTS core header for samples ((NBLKS+1)*32) and sample rate
(SFREQ, 48kHz fallback) to derive each AU's duration, and stamp a running
monotonic PTS: max(next_clock, own-core-PES PTS), then advance by the
frame duration. A later PES whose PTS is ahead of the clock still wins
(preserves the UHD per-PES attribution from da85f56/c49a180); frames
sharing one PES advance frame-by-frame instead of colliding.

Tests: the 3 that encoded 'same PES -> same PTS' now assert monotonic
advance; new dvd_many_cores_one_pes_are_strictly_monotonic reproduces the
Punisher bug; duration/SFREQ-fallback unit tests added.
2026-07-01 17:17:35 -07:00
Matthew Jackson b2c3540989 restore freemkv-unlock path dep for local dev (post-v1.2.0) 2026-07-01 16:53:06 -07:00
Matthew Jackson 3a4307def6 v1.2.0: bump version (freemkv-unlock git-pinned for the tag) 2026-07-01 16:53:03 -07:00
Matthew Jackson 917026d566 changelog: DVD DTS/LPCM silent-audio fix + Pass-N recovery specialists + flat scheduler 2026-07-01 16:50:08 -07:00
Matthew Jackson 6b0bcbb43f recovery: flat-pool bandit scheduler (FREEMKV_PATCH_FLAT, opt-in)
The tier ladder (bucket->mug->thimble) forces tier 0 to attempt ALL
ranges before tier 1, and tier 1 all before tier 2 — so on a hardened
residual (late resume, flood already gone) the marginal specialists are
unreachable for hours: you bail near-empty buckets over hundreds of dead
ranges before a thimble ever touches water.

FREEMKV_PATCH_FLAT collapses the 3 tiers into ONE flat pool of all 16
handler configs. run_handlers already sorts best-first by the live
decayed-yield scorecard, so this is a data-driven bandit: the first
ranges try them all (explore), the ranking floats whatever is actually
landing sectors to the front (exploit), re-measured per range; a handler
that doesn't fit stays last but is never dropped (floor -> can revive).
The new handlers self-limit (yield after 4 dead reads), so the flat
scheduler does NOT reintroduce the old depth-first per-range starvation.

Opt-in; unset keeps the proven tier ladder as the default for 1.2.0.
2026-07-01 16:44:29 -07:00
Matthew Jackson 516ff4e581 test: widen profile_03 efficiency guard for tier-2 roster
The tier-2 marginal specialists now additionally probe permanently-bad
residual sectors before leaving them NonTrimmed, so the alternating-bad
profile's finite read count rose past the old tier-0/1 bound. Guard
still catches runaway; behavior asserts unchanged.
2026-07-01 13:55:20 -07:00
Matthew Jackson c27009d443 recovery: wire tier 2 marginal specialists into the Pass-N chain
Add a third breadth-first tier (PATCH_TIERS 2->3) that runs the marginal
specialists on the hardened residual tiers 0-1 leave: SlowSpin (Linear
fwd+rev @ min), FuaRetry (Linear fwd+rev+Bisect @ FUA), SlowFua (Linear @
min+FUA), CachePrime, Oscillate (@ max and @ min), SpeedSweep. Every read is
a wedge-safe read_span, so they inherit wedge-abort / unproductive-yield /
deadline for free. All are new configs, so the EWMA scorecard calibrates each
once then ranks by decayed rate — a specialist that doesn't fit self-
deprioritises. Tiers 0-1 (fast scouts, slow-deep) are unchanged; this is
purely additive. Also switch the scorecard log sort to sort_by_key.

cargo test -p libfreemkv green (2200 passed).
2026-07-01 13:50:35 -07:00
Matthew Jackson 789d988314 recovery: CachePrime + Oscillate handlers + fixtures
CachePrime reads the good run immediately preceding a residual island to lock
the servo/PLL, then reads the island warm (boundary sectors the drive can't
cold-seek). Oscillate reads each residual sector by alternating approach —
forward-into then reverse-into — for direction-dependent tracking. Both go
through the wedge-safe read_span (primes included). FakeDisc models a
direction-dependent sector and a servo-primed boundary sector; fixtures prove
a forward/cold Linear misses each while Oscillate/CachePrime recover them.
2026-07-01 13:50:35 -07:00
Matthew Jackson 1e0dc4c514 recovery: FuaRetry + SlowFua fixtures (FUA / min+FUA levers)
FuaRetry (Linear fwd+rev+Bisect at {max,fua,deep}) and SlowFua (Linear at
{min,fua,deep}) are Linear/Bisect at FUA params, no new struct. FakeDisc now
models a stochastic sector that lands only on the Nth physical (FUA) read (a
cached re-read keeps missing) and a hardest sector needing BOTH min speed and
FUA. Fixtures prove cached reads keep missing while the FUA group lands the
stochastic sector, and that neither lever alone recovers the hardest sector —
only the min+FUA combination does.
2026-07-01 13:50:35 -07:00
Matthew Jackson 629ed32e9e recovery: SpeedSweep handler + slow-spindle fixtures (SlowSpin/SpeedSweep)
SlowSpin is Linear pinned to min speed (no new struct). SpeedSweep is a new
per-sector handler that tries Max->Min until one reads (speed resonance).
FakeDisc now models a sector that reads ONLY at min speed; fixtures prove a
max-speed Linear misses it while SlowSpin and SpeedSweep recover it (and that
SpeedSweep tries fast-then-slow).
2026-07-01 13:50:35 -07:00
Matthew Jackson ac3b3fcfa4 recovery: decayed (EWMA) handler scorecard so the ranking flips
Grade handlers by an exponentially-decayed recovery rate (bytes/second,
alpha=0.5) instead of a cumulative rate. The residual hardens mid-pass, so
the best technique changes: cumulative froze the early winner in the lead
forever; the EWMA re-prices continuously — a handler that stops earning
decays down, a late-starting specialist climbs. Keeps rank()'s untried->top
one-shot calibration and attempted-but-zero-time->bottom.

New flip fixture proves a handler that recovers a lot early then nothing
loses its lead to one that starts recovering later.
2026-07-01 13:50:35 -07:00
Matthew Jackson 0c8153304e recovery: parameterize the read primitive with ReadParams (speed/FUA/timeout)
Add ReadParams { speed: SpeedPref, fua: bool, timeout: TimeoutPref } and
thread it through read_span so every wedge-safe handler read can request a
spindle speed (SET CD SPEED issued only on change, restored to max when the
handler exits), set the READ(10) FUA bit, and pick the 10s vs 60s timeout.

- SectorSource gains read_sectors_fua (default ignores fua); Drive sets the
  CDB bit, DecryptingSectorSource threads fua to its inner read.
- recovery_read gains a fua param.
- Linear becomes { direction, params }; Bisect/Jump take params. Existing
  tier-0/1 instances keep identical behavior (max speed, no FUA, fast/deep).
- Scoreboard keys on the full-config String name (linear:fwd:max:fast, ...).
- FakeDisc observes speed + FUA + approach so specialist techniques are
  provably exercised in later commits.

cargo test -p libfreemkv green (2193 passed).
2026-07-01 13:50:35 -07:00
Matthew Jackson e94319c099 dvd: route private_stream_1 audio sub-ids by positional stream number
The private_stream_1 sub-stream id low nibble is the DVD audio-stream
NUMBER (0-7), shared across codecs, not a per-codec ordinal. A DTS (or
LPCM) track that isn't the disc's first audio stream got a sub-id one
too low, so the demux routing key (0xBD00 | sub_id) never matched and
every packet was dropped -> present-but-silent track. AC-3 at position
0 coincidentally worked (ordinal==position). Route by positional index.

Fixes silent DTS 5.0 track on The Punisher (AC-3 5.1 @ pos0 plays,
DTS 5.0 @ pos1 was 0xBD88, real wire id 0xBD89).
2026-07-01 13:49:55 -07:00
Matthew Jackson a93da78621 lib: remove standalone sector-health verify feature
Delete src/verify.rs (standalone 'read every sector, classify health'
module) + its lib.rs pub mod/pub use. It was old, unused code. The
decrypt-verify gate (disc::verify / UnitVerifier) is a DIFFERENT module and
stays. Relocated the pure chapter_at_offset helper (used by locate_ranges +
autorip's done-card) into disc/mod.rs as a pub fn.
2026-07-01 13:09:21 -07:00
Matthew Jackson cf13838f12 recovery: audit fixes — live at-risk excludes NonTried, latency-gated wedge, drop dead block counters
- Live located drilldown (sweep + patch progress snapshots) now excludes
  NonTried, matching the one-shot path. Including the unread remainder made
  main_at_risk_ms show ~full-movie at sweep start and melt to 0 as it
  progressed — unread is unknown, not damage.
- Wedge abort now requires the failure to be FAST (< WEDGE_FASTFAIL_MS,
  500ms) as well as wedge-family sense: a real uncorrectable sector on
  Hardware-error media spends ECC-recovery time before failing, so it no
  longer false-trips the fast-fail wedge abort. New regression test.
- Removed the always-zero blocks_attempted/read_ok/read_failed/
  unreadable_count from PatchOutcome + PatchLoopState (dead residue from
  the old grind loop; the HandlerScoreboard supersedes them) so the
  patch.done log no longer emits misleading zeros.
2026-07-01 12:07:08 -07:00
Matthew Jackson 9519628954 docs: 1.2.0 changelog — Pass-N handler chain, DVD drive-prep unlock fix, unlocker matrix 2026-07-01 11:05:31 -07:00
Matthew Jackson 21f8cec419 disc: unlocker_matrix reports 'did work this rip', not 'matched'
yes now means the unlocker actually ran and did its job: LibreDrive from
the runtime firmware-unlock success, AACS host-cert only when LibreDrive
didn't do the bus (stock-drive fallback), CSS from the crack succeeding.
On a LibreDrive UHD that correctly reads LibreDrive: yes, AACS: no (LD
removed the bus, the cert route never ran) instead of the misleading
AACS: yes. Names stay registry-driven; runtime logic lives here.
2026-07-01 10:44:39 -07:00
Matthew Jackson d5d72c4e22 drive: disable FUA on READ(10) — the ~10x sequential-read regression
Forcing FUA (Force Unit Access) on EVERY READ(10) bypassed the drive's
readahead/streaming cache on the bulk Pass-1 sweep, collapsing sequential
throughput ~10x (UHD 15-25 → ~2 MB/s, DVD → ~0.5 MB/s), disc-type-
agnostic — the cache IS the streaming throughput. This was the real speed
regression (not unlock/riplock, which the earlier chase suspected). Clear
byte-1 bit 0x08. FUA will return as a dedicated Pass-N recovery handler
that sets/clears it per marginal-sector re-read, where cache-masking of a
stochastic sector actually matters — never blanket on the bulk path (#55).
2026-07-01 10:26:56 -07:00
Matthew Jackson c781fb7193 disc: Disc::unlocker_matrix() — registry-driven unlocker report
Returns each registered unlocker's name + whether it applies to this
drive+disc (via the unlock bridge over all_unlockers()). Disc crypto kind
derived in the library so the CLI and autorip render an identical, always-
current report with no hardcoded names. Also adds a Drive::unlocker_name()
runtime getter.
2026-07-01 10:00:59 -07:00
Matthew Jackson 541ca2139f drive: run drive-prep unlock for EVERY disc, incl. DVD (fix riplock regression)
The v1.0.0-rc.1 unlocker refactor added an `if disc_is_dvd() { return }`
early-return in Drive::init() that skipped the whole drive-prep unlock for
DVDs. That firmware unlock is what lifts riplock and readies max read
speed — a DRIVE-level, disc-independent feature — so skipping it left
every DVD stuck at stock/riplock speed (~0.4x, 3h ETA). UHD was unaffected
because it flows through the unlock.

Remove the skip: init() now runs the identity-keyed drive unlocker for all
discs (disc kind is Unknown at init, so only the drive unlocker matches;
the AACS host-cert handshake and CSS bus-auth still run later, gated on the
real disc kind, on top of the unlocked drive). Speed stays where it
belongs — SET CD SPEED(0xFFFF) at pass-1 start (disc/mod.rs) — not in the
unlocker. Also drop the matching probe_disc DVD skip.

Reverts the SET STREAMING stopgap (separate freemkv-unlock revert): that
treated the symptom; the real bug was the skipped unlock. Drive features
come from the firmware unlock, not a stock speed CDB.
2026-07-01 09:38:51 -07:00
Matthew Jackson 2dd98c3e32 recovery: fresh-eyes audit fixes (handlers + Pass-N engine + sweep)
Handlers (section_recover.rs):
- Bisect expand loops now honor ctx.halted() (were deadline-only, so a
  Stop could hang up to 60s vacuuming a readable island).
- read_span: explicit Transport arm so a bus-abort read isn't counted as
  unproductive grinding; debug_assert the sector-aligned span invariant.
- Scoreboard rank: an attempted-but-zero-time handler (e.g. returned Halted
  on its first check) now ranks BOTTOM, not top — it no longer crowds out
  proven performers.
- Document the wedge tier-size coupling + new regression test that a
  2-handler (tier-1) chain still catches a wedge via cross-section streak.

Pass-N engine (patch.rs):
- Rebuild PatchOutcome stats AFTER the post-read re-verify downgrade flush
  (was snapshotting before it, over-reporting bytes_good / recovered and
  risking a 'perfect rip' verdict on an imperfect one).
- Progress 'recovered' composes the still-bad set to MATCH work_total
  (subtract NonTried, add Unreadable) so the bar can't pin at 0 on a
  partially-swept disc or run backward on the Unreadable→NonTrimmed relabel.
- Remove dead work_done field; rewrite the stale 'adaptive batching' comment
  to describe the handler chain and mark block_sectors/full_recovery as
  informational-only.

Sweep (disc/mod.rs):
- Saturating arithmetic at the damage-jump position math (honor the
  read_error side's documented defence-in-depth guarantee).

Deferred (noted, need focused passes): fast_capture re-introduction,
Pass-1 halt-misclassified-as-jump, bytes_good display inflation, the
always-zero blocks_* telemetry, Pass-1 jump-on-first-error policy.
2026-07-01 09:14:32 -07:00
Matthew Jackson ceaa1da369 drive: apply drive-features for every disc, incl. DVD (fix riplock)
DVD ran the drive at ~0.4x stock speed: init() early-returned for DVD
before any speed was set, and its only speed command was the sweep's lone
legacy SET CD SPEED, which this BU40N ignores for DVD. BD/UHD got up to
speed only via the firmware unlocker's calibration — a path a DVD can't
take (it breaks stock CSS).

Call the new unlock-crate drive-features capability at init() for ALL disc
kinds, before the DVD stock-mode early-return, via a bridge shim. Stock
MMC only (SET STREAMING + SET CD SPEED), no bus unlock, so CSS is
undisturbed.
2026-07-01 08:56:19 -07:00
Matthew Jackson fe14a2d5e5 Pass-N: detect drive fast-fail wedge and abort instead of grinding
The 2026-07-01 overnight rip ground a wedged BU40N for 28 minutes at
0 B/s. Root cause: the handler chain's read_span split errors only into
Transport (bus gone) vs Bad (everything else), so the drive's fast-fail
wedge sense (ILLEGAL REQUEST 0x05/0x24 — it rejects every CDB in <100ms
without attempting recovery) was treated as an ordinary bad sector. The
chain hopped to the next section forever, and the existing wedge detector
in read_error.rs never saw it because the chain reads through
recovery_read directly.

Add a pass-level wedge streak: read_span counts consecutive wedge-family
(Hardware/IllegalRequest) senses; at WEDGE_ABORT_STREAK (16) it escalates
the read to Transport, which every handler already propagates as
TransportFault — aborting the whole pass and setting wedged_exit so the
caller spin-cycles. The streak is carried across sections via PatchCtx
(seeded into and read back from each per-section HandlerCtx) so a wedge is
caught even when every bad sub-range is smaller than the streak. Any Good
or non-wedge (medium-error) read resets it, so scattered bad sectors on
real media never trip it. New fixture test: a wholly-wedged 1000-sector
section aborts in <100 reads, not 1000.
2026-07-01 08:02:52 -07:00
Matthew Jackson 8f6a92ccd4 Bisect: halve at dead boundary + exempt boundary probes from stall
The expand loops broke out of the batch sweep on the first failing read,
overshooting the good/dead boundary by up to a batch and leaving the
readable sectors adjacent to a dead edge for the re-bisect to re-pin. On
top of that, early-yield counted those boundary-probe failures as a
stall, so the forward expand quit early AND the backward expand inherited
a poisoned unproductive counter and yielded instantly.

Now on a failed expand batch we halve the step (down to a single sector)
to recover right up to the dead edge in ~log2(batch) reads, drive the
expand loops off the deadline only (they self-terminate via halving), and
clear the unproductive streak once an island is located. patch_recovers_
good_middle_of_a_bad_range now recovers 50/50 good-middle sectors.
2026-06-30 22:06:04 -07:00
Matthew Jackson 29f76aad68 section_recover: per-rip handler scorecard + Bisect-leads-scouts + proportional Jump
Scorecard: grade each handler by recovery rate (MB/s) per rip, order
best-first on later sections, log the ranking at pass end. Untried
handlers rank top so each is calibrated once before ranking narrows to
the winners. Ephemeral (reset per pass), no persistence.

Tier 0 scouts are now [Bisect, Jump, Linear-fast x2], scorecard-ordered.
Bisect leads: probing the MIDDLE of a range lands on a readable island in
one read where a linear scan grinds the dead front to reach it. Jump now
jumps to the middle of the REMAINING span (proportional) instead of a
fixed 8 MiB that leapt clean over small ranges and missed their readable
middles. Tier 1 is slow deep-recovery on the residue.
2026-06-30 21:26:35 -07:00
Matthew Jackson 840ba8390c section_recover: per-rip handler scorecard + Bisect leads the scouts
Scorecard (ephemeral, reset each pass, no persistence): grades every
handler by recovery rate (bytes/sec). run_handlers orders handlers
best-first by that rate; an untried handler ranks top so each is
calibrated once, then the ranking narrows to the winners. Logged at pass
end (phase=scorecard) so the operator sees which handler is pulling the
weight on this drive/disc and which is a dud.

Tier 0 scouts are now [Bisect, Jump, Linear-fast x2], scorecard-ordered.
Bisect leads because probing a range's MIDDLE lands on a readable island
in one read, where Jump (linear from the front, big skip) can grind the
dead front or overshoot a small range entirely. The scorecard confirms
or overturns that order with real per-disc data.
2026-06-30 21:25:39 -07:00
Matthew Jackson e803905265 section_recover: Jump starts at 8 MiB base to clear big dead spots faster
Each dead probe read costs the drive's full ~10s timeout, so a large dead
region took a dozen escalating probes. Starting the jump at 8 MiB (vs
1 MiB) clears it in a handful; a skipped span stays bad for Bisect to
reclaim readable islands, so an over-jump loses nothing.
2026-06-30 21:12:12 -07:00
Matthew Jackson 11d4c33477 patch: single Jump-scout tier 0, expand-Bisect, live in-handler progress
Tier 0 is now a single fast Jump scout: it streams the big readable
ranges back and skips dead runs in seconds, so the pass reaches every
section fast and converges to the small genuine-dead residue instead of
grinding three handlers x 60s on each dead fragment. Tier 1 (fast
mop-up + slow deep reads + Bisect) works only that residue.

Bisect now expands: on a good probe it reads outward forward and
backward in full batches until a read fails, recovering the whole
readable island in large reads; the two failing ends become smaller bad
sub-ranges it bisects again. One huge bad range becomes many precisely
located small dead clusters.

Progress heartbeat: HandlerCtx gains a throttled tick (250ms) called
from every read, pushing a fresh snapshot to the reporter DURING a
handler. The bar and speed now move continuously as recovery happens
instead of jumping once per section (the reason speed read 0 B/s and the
% looked frozen between range boundaries).
2026-06-30 20:57:36 -07:00
Matthew Jackson 8d775cd341 patch: add Jump handler (lead fast tier) + recovery-based progress %
Jump: on sustained batch failures skip ahead an escalating distance
(1 MiB doubling to 256 MiB) to find where readable data resumes, leaving
the skipped span for Bisect to pin — mirrors the Pass-1 damage-jump. It
leads the fast tier so a large dead run is skipped in seconds instead of
the linear sweeps grinding every dead batch (10 s each) first; on a
readable range it just streams it back. Recovers readable data buried
behind a big dead front (the 192 MB Dune range).

Progress %: report bytes RECOVERED (initial-bad minus still-pending)
instead of a per-range counter that only advanced on the final tier — so
the bar reflects the readable bulk recovered during tier 0 the instant it
lands, matching the 'MB remaining' number.
2026-06-30 20:40:22 -07:00
Matthew Jackson bc07011bcb patch: breadth-first two-tier recovery, largest ranges first
The per-range walk was depth-first: each bad range ran the full handler
chain (fast + slow deep-recovery + bisect) before the next range was
touched. So a handful of tiny dead fragments at one end of the disc
burned the whole pass and the big NonTrimmed ranges elsewhere — usually
sweep-jump over-marks that read straight back — were never attempted.

Now recovery runs in two breadth-first tiers over ALL sections:
- Tier 0 gives every section one fast full-batch attempt (fast reads
  only), largest ranges first, so the recoverable bulk of the disc comes
  back in the first minutes.
- Tier 1 deep-recovers only the residue tier 0 could not pull.
Per-section still-bad sets persist across tiers. Largest-first ordering
means a big readable region is reached before time is spent on tiny dead
fragments.

Linear no longer collapses a failed batch to count=1 single-sector reads
(live probing: a marginal sector recovers in a large read, not a lone
one) — a failed 32-batch stays 32 and is re-attempted at full size by the
next handler/pass; Bisect salvages readable islands.

Adds a handler-start trace line so the debug log shows which handler is
running and the hand-off to the next.
2026-06-30 20:25:48 -07:00
Matthew Jackson d65b776a8e patch: replace grind-until-wedge loop with bounded handler chain
Pass-N recovery is now a chain of time-bounded recovery handlers instead
of one monolithic per-range loop that could grind the front of a bad
range for up to 30 min, wedge the drive, and abort the whole pass.

A bad range is a SubRanges set; recovery is an ordered list of
SectionHandlers (Linear{reverse,fast} covering back/forward x fast/slow,
and Bisect). A coordinator runs each handler with a hard per-handler
deadline: a handler recovers what it can (removing it from the still-bad
set) and hands the rest to the next handler; whatever is still bad after
the chain becomes NonTrimmed residue and we move on to the next range.

Guarantees, now structural rather than bolted-on:
- never hangs: every handler is deadline-bounded; the loop always drains
  to recovered-or-residue.
- always moves on: a range that cannot be finished leaves residue and
  advances; only a genuine transport fault or user halt ends the pass.
- extensible: a new recovery idea is one SectionHandler impl added to the
  chain; a proven-ineffective one is removed. The engine never changes.

Removes ~1.9k lines of the old inner loop (watchdogs, skip escalation,
NOT_READY grind, wedge counters) and their tests. fast_capture is now
inert (the chain supersedes it); breadth-first ordering becomes a future
scheduler concern. New module: disc/section_recover.rs (8 fixture tests,
injectable clock — bounded/never-hang proven without touching a drive).
Two A/B tests updated to the chain's strictly-better recovery counts.
2026-06-30 19:55:37 -07:00
Matthew Jackson ff7f3028a5 drive: always set FUA on READ(10) to bypass the drive cache
Every Drive::read now forces the read from physical media instead of
letting the drive satisfy it from cache. A recovery tool must not trust
the cache: on a re-read of a marginal sector the BU40N can return a
cached miss (~4 ms) instead of giving the surface a fresh physical
attempt (~40-250 ms), masking a sector that would otherwise read on a
real media hit. FUA (READ(10) byte 1 bit 3) is validated on the
BU40N/Initio bridge.
2026-06-30 18:49:14 -07:00
Matthew Jackson 25a7b131b4 drive: Drive::spin_cycle() — soft un-wedge via START STOP UNIT (no eject)
Spin the disc down (START=0, LOEJ=0) then back up (START=1) to clear the
BU40N/Initio fast-fail wedge state a run of HARDWARE_ERROR reads leaves the drive
in — the non-eject power-cycle our notes say the wedge needs. The disc stays
loaded; we never eject (slot-loading — a human eject is a product failure for an
unattended service). Validated live 2026-07-01: took the drive from
failing-every-read back to reading at MB/s.
2026-06-30 17:54:51 -07:00
Matthew Jackson 0cab32a08a disc/patch: fast-capture reads fail-fast (no deep recovery)
A fast_capture pass defers every failed block to a granular pass anyway, so
spending the drive's 60s deep-recovery timeout on it here only freezes the
breadth-first sweep on a pure-bad cluster (~25s per 32-block, incl. NOT_READY
grind). Read with recovery=false in fast_capture so a bad block fails on the
short timeout (~10s) and the sweep steps on; the granular passes
(fast_capture=false) do the deep recovery on what's left. No recovery lost —
the block stays NonTrimmed for the granular retry.
2026-06-30 17:08:26 -07:00
Matthew Jackson 22f0f5eb6f docs: 1.2.0 changelog — fast_capture breadth-first patch recovery 2026-06-30 16:38:56 -07:00
Matthew Jackson 2013ef8c44 disc/patch: fast_capture mode — breadth-first recovery (#50)
A PatchOptions.fast_capture pass reads each bad range ONCE at the full batch and
leaves every FAILED block NonTrimmed for a later pass — no bisect, no per-sector
grind, no retry. This lets a first retry pass grab the readable blocks (the
sweep's good skip-ahead overshoot) of EVERY section quickly, before any single
section's slow per-sector recovery — instead of grinding section 1 to exhaustion
before even touching section 2. A later pass (fast_capture = false) does the
granular bisect/retry on what's left.

Load-bearing invariant (fixture test): NO data is dropped. A failed block becomes
NonTrimmed (pending, retried by a granular pass), NEVER Unreadable. The test pins
that the readable half of a range recovers, the bad half stays NonTrimmed (not
Unreadable), and the bad block is marked in ONE batch read with zero bisection.
Disc::copy's internal patch keeps fast_capture=false (single-call full recovery).
2026-06-30 16:37:59 -07:00
Matthew Jackson acaae3d0a4 docs: 1.2.0 changelog — PassProgress mapfile-free contract + located drilldown 2026-06-30 16:13:05 -07:00
Matthew Jackson 65c6835363 disc: add progress_snapshot_from_mapfile one-shot + locate_ranges test
The library now offers a one-shot PassProgress snapshot built from a mapfile on
disk + title, so a client gets the fully-rendered drilldown without parsing the
mapfile itself — used for autorip's pass-boundary and mux-entry paints (no live
callback yet at those points). Adds a locate_ranges test covering in-feature vs
out-of-feature at-risk time (ported from autorip's removed from_map tests).
2026-06-30 16:09:28 -07:00
Matthew Jackson 0f0c496a4d progress: PassProgress carries the located drilldown (emit side)
Add LocatedRange + LocatedProgress to the progress contract and a 'located'
field on PassProgress, populated by the sweep + patch emitters from the
in-memory bad-range set + title. Move the range->chapter/time annotation
(locate_ranges, range_chapter, byte_offset_in_title) into the library so a
client renders the disc map + at-risk movie time straight from PassProgress
and never reads the mapfile itself — if the mapfile becomes a mapdb, this type
and its producer change, clients don't.

PassProgress is no longer Copy (located carries a Vec); it's built once per
throttled emission and passed by reference. Non-locating phases (verify,
extract) emit LocatedProgress::default(). Adds consts::MILLIS_PER_SEC.
Consumer-side wiring (autorip drops Mapfile::load) follows.
2026-06-30 15:26:49 -07:00
Matthew Jackson 923b9edbf4 disc/patch: remove scatter-recovery (ruled out by live probing); add SubRanges
Live drive probing (cold-single vs in-stream batch vs centered window,
one clean uncached read per sector) showed recovery RATE is identical
across approach: 3/5 every mode. The drive's per-sector ECC is
media-bound and partly stochastic, NOT approach-bound. So scatter
(seek-away recalibration + far-anchor re-read) does not improve recovery
and is dead weight — removed (function, read_good_sectors, SCATTER_*
consts, and its tests).

Add SubRanges: the still-bad (pos,len) sub-ranges of a section with
remove()/split, the work-list foundation for the upcoming per-section
recovery decomposition (recover_section -> stream/bisect/retry phase
helpers). Unit-tested; unused until the decomposition lands.

25 patch + 8 AB profile tests green; precommit clean on Rust 1.86.
2026-06-30 13:56:49 -07:00
Matthew Jackson caf1b03fd4 disc/patch: keep MAX speed on recovery reads + geometric batch re-grow
Live probing on the BU40N/UHD testbed (rprobe/dprobe) settled the speed
question empirically: a marginal sector reads ~12x FASTER at MAX speed
than at slow, and slow NEVER recovered one MAX didn't. Cleanly-tested
(one uncached read per sector) the recovery RATE is identical across
speed/direction/window/batch — the drive's per-sector ECC is media-bound,
not approach-bound. The only host lever that matters is read SPEED.

So the old 'drop to 0x0000 on first failure and hold it for the whole
range' only slowed the GOOD sectors of a bad range — measured ~3x slower
overall. Now the range stays at MAX; the first failure just re-attempts
once (stochastic media) and falls through. Live A/B: 11 KB/s -> 33 KB/s,
bytes_lost unchanged.

Also: batch adaptation is now halve-on-failure / double-on-success
(geometric), so a bisected batch climbs back through clean runs from any
size (the old count==1-only upscale left it stuck at mid sizes). Removed
the dead consecutive_singles_ok counter and the inert batch-retry probe.

25 patch tests green; precommit clean on Rust 1.86.
2026-06-30 13:25:07 -07:00
Matthew Jackson 9cc422bd00 disc/patch: drop cache-priming, bisect batch failures instead of count=1
Two patch-recovery speed fixes, grounded in live rip1 telemetry on the
Dune UHD bad regions.

1. Pull prime_cache. In the reverse walk it issued throwaway reads of
   lba-3..lba-1 — i.e. straight into the not-yet-visited dead L-EC core —
   grinding each ~7-9s at the fast timeout, then discarding the result.
   The live trace showed a good target read (read_duration_ms=1) taking
   9s wall-clock purely from the prime grind ahead of it. The dead-zone
   skip (Tier 1) + scatter's recalibration read (read_good_sectors)
   cover the cache-warm intent without grinding the dead zone, so prime
   is redundant harm — removed (function + CACHE_PRIME_SECTORS + call).

2. Bisect on batch failure. handle_read_failure used to collapse a
   failed count>1 batch straight to count=1 and single-walk the entire
   batch. Now it halves (count/2) and retries the same start: a good
   half recovers in BULK, the bad sector is isolated in O(log n) reads
   instead of O(n). Invariant preserved — only a count==1 failure marks
   NonTrimmed, so no good sector is lumped into a bad mark.

These are Tier 2 (bisect) of the progressive recovery model
(fast-mass-skip -> bisect -> scatter); composable so order can be tuned.
25 lib + 8 AB fixture tests green; precommit clean on Rust 1.86.
2026-06-30 11:20:44 -07:00
Matthew Jackson 5b39a0af2b disc/patch: add scatter-recovery for stuck bad sectors
The 'reset, read good data, come back for one sector' technique. A
genuinely-damaged sector makes the drive grind its full recovery timeout
and still fail; re-reading consecutive bad LBAs at identical conditions
re-fails AND is the rapid-failure cadence that drops the BU40N into a
firmware fast-fail wedge (CLAUDE.md hard-rule #2).

scatter_recover: on a stuck single sector, make up to SCATTER_MAX_ATTEMPTS
fresh tries, each preceded by read_good_sectors() — a seek to a far
known-good anchor + a sustained read that re-seats the head/servo and
breaks the failure cadence. The fresh re-read uses the FAST timeout
(recovery=false) so a recalibrated marginal sector reads quick and a
truly-dead one fails fast instead of burning 60s per attempt. The seek +
good read IS the settle (matches ddrescue/MakeMKV) — no idle sleep.

Gated to genuine single-sector MEDIUM_ERROR (0x03): transport faults
still abort the pass, NOT_READY keeps its own retry path, wedge-family
senses are left for cooldown/eject. A scatter recovery is recorded
exactly like a normal read success (write-then-mark-Finished); nothing
marks a range good without a successful read + write.

Logs anchor_ms/reread_ms per attempt for live tuning. 5 fixture tests
(recover-marginal, give-up-dead, skip-non-medium, skip-batch, AACS
widen); 2 AB read-count guards raised for scatter's bounded extra reads.
Precommit green on Rust 1.86.
2026-06-30 10:29:55 -07:00
Matthew Jackson bd0a21bdb4 disc/patch: extract PatchCtx/patch_region orchestration + unify log taxonomy
Pure structural refactor of the Pass-N recovery path, no behavior change.

- PatchCtx<R> holds the one-pass context (reader, pipe, shared, opts,
  reused read buffer, cross-region state) and owns orchestration.
- PatchCtx::run() walks the ordered bad ranges, applies the inter-range
  cooldown (armed on a grind, consumed at the next range's entry), and
  stops the pass the moment a range reports halt/wedge/transport-fault.
- PatchCtx::patch_region() is the focused single-range loop: read ->
  success/failure -> damage-skip -> per-range watchdog. Every former
  break / break 'outer is now a typed RegionOutcome (Completed,
  SkipLimit, BudgetExceeded, Wedged, Halted, TransportFault) that run()
  maps back to next-range vs stop-pass exactly as before.
- recovery_read() extracts the AACS mid-unit-window widening + plain
  read into one reusable helper (the building block the upcoming
  scatter re-read will share).
- Logging unified onto a single dotted patch.* taxonomy (patch.region.*,
  patch.read.*, patch.probe.*, patch.wedge.*, patch.batch.*,
  patch.mapfile.*, patch.skip.*, patch.stall, patch.speed); the
  freemkv::scan pass-boundary markers are left intact.

20/20 disc::patch tests green; precommit (fmt + clippy -D warnings +
tests) clean on Rust 1.86.
2026-06-30 10:04:51 -07:00
Matthew Jackson c8822ddea3 error: treat IoError/DeviceNotFound as SCSI transport failures (dead bus)
is_scsi_transport_failure recognized only the 0xFF SCSI sentinel, so the two
NON-SCSI dead-bus faults the Linux SG_IO transport returns — Error::IoError
(ioctl(SG_IO) == -1: ENODEV/EIO on an unplugged bridge) and Error::DeviceNotFound
(fd gone) — were classified as recoverable bad sectors. Every caller
(sweep read_error, patch, single-pass fill_extents) would then zero-fill /
skip-ahead across a wedged bus instead of aborting the pass and re-enumerating
the bridge — potentially zero-filling a huge span. Add both to the classifier so
a dead bus aborts the pass everywhere. (Companion to the unlock_bridge fix, which
maps the same two faults to 0xFF across the freemkv-unlock seam.)
2026-06-30 07:19:03 -07:00
Matthew Jackson 7a7ab2c9d8 unlock_bridge: map non-SCSI transport faults (IoError/DeviceNotFound) to 0xFF
Round-2 audit gap in the prior bridge fix: extract_scsi_context only recognizes
Error::ScsiError / DiscRead and collapses every other variant to (0, None). But
the Linux SG_IO transport returns genuine dead-bus faults as non-SCSI variants —
Error::IoError (ioctl(SG_IO) == -1: ENODEV/EIO on an unplugged bridge) and
Error::DeviceNotFound (fd gone) — which were crossing the seam as status 0x00, so
the unlock crate classified neither as a transport fault (needs 0xFF) nor as a
sense-bearing rejection, and kept hammering a dead bus.

Match the error variant in the adapter: a SCSI status (CHECK CONDITION or a
drive-tagged 0xFF) keeps its real status + sense; any other variant is a
non-SCSI transport/IO fault → SCSI_STATUS_TRANSPORT_FAILURE / None. Keying off
`sense.is_none()` would be wrong (a senseless CHECK CONDITION is a rejection).
Add bridge tests covering CHECK-CONDITION sense preservation, 0xFF passthrough,
and IoError/DeviceNotFound → 0xFF.
2026-06-29 22:57:09 -07:00
Matthew Jackson 7a0ef5412c mux: add the B1 drop-to-keyframe resync gate to DiscStream
DiscStream (the live-drive single-pass path) enables AACS decrypt-loss
concealment (NULL-TS fill on an undecryptable unit) but, unlike the file-backed
PipelinedPesStream, had no B1 resync gate — so after a concealed gap it forwarded
inter-coded video frames referencing the now-missing data, producing
decode-broken MKV output (dangling-reference frames).

Mirror PipelinedPesStream: add per-stream ResyncGate + is_video, and route every
codec-frame emit on the TS path (in-stream parse, demuxer EOF flush, parser EOF
flush) through gate.admit(is_video, frame.discontinuity, frame.keyframe) so a
video track drops forward to the next keyframe after a concealment event. Warn
once if a gate is still armed at EOF (a concealed gap in the final GOP with no
trailing keyframe). Audio/subtitle always admit.
2026-06-29 22:27:52 -07:00
Matthew Jackson 8c38cb0918 unlock/patch: preserve SCSI sense across the bridge; init aborts on dead bus; slow-retry first patch failure
Audit fixes (v1.1.0..HEAD regressions in the unlock migration + adaptive patch
speed):

- unlock_bridge ScsiAdapter: libfreemkv's transport returns Err on ANY non-zero
  SCSI status (a normal CHECK CONDITION), not only transport faults. The adapter
  was collapsing every such Err to { status: 0xFF, sense: None }, which discarded
  the parsed sense and defeated the AACS handshake's ILLEGAL_REQUEST wedge guard
  (so it kept hammering the drive — hard-rule #2) and inverted its
  transport-vs-rejection diagnosis. Now reconstruct status + the 32-byte sense
  buffer (sense_key@2, asc@12, ascq@13) and only emit 0xFF/None for a genuine
  transport fault.

- Drive::init: a genuine transport fault during the drive-prep unlock means the
  bus is dead — propagate it (the v1.1.0 invariant) instead of silently
  swallowing it via `if let Ok`. Other errors (no matching unlocker) still fall
  through to stock mode. SET CD SPEED max now runs only when the bus is alive.

- disc::patch: on the first read failure in a range, drop to slow recovery speed
  and RE-ATTEMPT the same position at slow speed before marking it. A
  single-sector range's first failing sector was being marked from a MAX-speed
  read it never got to recover.

- docs: lib.rs architecture diagram (handshake → host_certs) and README (stale
  pluggable-unlock-seam / register-unlocker / crates.io / docs.rs references).
2026-06-29 22:16:28 -07:00
Matthew Jackson b36896564f disc: log adaptive patch speed transitions; issue SET CD SPEED max unconditionally
Two recovery-path fixes:
- patch: log the per-range speed transitions (INFO, phase="patch_speed") — each
  range enters at 0xFFFF (max) and drops to 0x0000 (slow recovery) on its first
  read failure. Previously the adaptive-speed behavior was invisible in the logs.
- drive init: issue the generic SET CD SPEED max UNCONDITIONALLY at drive-open,
  not only when a firmware unlocker matched. A stock-mode BD/UHD drive (no
  firmware unlock) was left riplocked because the call sat inside the
  unlocker-matched branch.
2026-06-29 21:47:50 -07:00
Matthew Jackson e648bde94b README: drop crates.io/docs.rs badges + stale plugin-model line (off crates.io now) 2026-06-29 21:17:06 -07:00
Matthew Jackson c86fa9bfc6 Cargo: publish = false — off crates.io (git-deps the firmware crate)
libfreemkv now git-deps freemkv-unlock (which carries drive firmware and can
never be on crates.io), so libfreemkv itself can only be consumed by git tag.
The freemkv-unlock dep stays an interim path dep for local cross-repo dev; the
release script swaps it to a git tag in the tagged commit.
2026-06-29 21:04:05 -07:00
Matthew Jackson 2ba6274eae unlock: dispatch via freemkv-unlock; delete in-tree handshake/css-auth/registry
Rewire the three unlock dispatch points through the freemkv-unlock crate via a
private `unlock_bridge`: drive-prep (kind=Unknown) at `Drive::init`, AACS cert
(kind=Aacs) at `do_handshake_cert`, CSS bus-auth (kind=Css) at scan. The bridge
news up `all_unlockers()` and runs the first matching one, mapping its
`Unlocked` result to the bus-key gate. After a successful drive unlock,
libfreemkv issues a generic SET CD SPEED (max) itself — the old per-unlocker
trait method is gone.

Delete the in-tree unlock code now owned by freemkv-unlock: the AACS cert
handshake (`aacs/handshake.rs`), the CSS bus-auth (`css/auth.rs`), and the
unlock registry (`unlock.rs`). Host-cert collection (a keysource concern) stays
in a small `aacs/host_certs.rs`. No public unlock surface remains — clients
touch libfreemkv only, oblivious to unlockers (as they are to SCSI). 2277 tests
pass.
2026-06-29 20:45:00 -07:00
Matthew Jackson 3bdb6f8b1a stage 4a: bridge libfreemkv to the freemkv-unlock crate
Add the freemkv-unlock dependency (path dep for local dev; re-pinned to a git
tag at release) and the consumption bridge: one generic ScsiAdapter mapping
libfreemkv's transport to the unlock contract, host-cert + DriveId mapping, and
run_unlockers() which news up all_unlockers() and runs the first matching one.
Additive — validates the dep resolves and the adapter compiles; the in-tree
registry/handshake/css path is still active and gets replaced in 4b.
2026-06-29 19:48:50 -07:00
Matthew Jackson 80e8523db2 aacs: extract run_cert_handshake + collect_host_certs as shared primitives
Pull the wedge-guarded cert loop and host-cert collection out of the in-tree
AacsCertUnlocker into public aacs::handshake primitives (run_cert_handshake +
CertHandshake, collect_host_certs). The in-tree path now delegates to them, so
the external freemkv-unlock-aacs plugin runs the IDENTICAL cert handshake — one
implementation, two callers. Pure refactor of the live AACS path; the existing
handshake + collect_host_certs unit tests validate it unchanged.
2026-06-29 17:33:26 -07:00
Matthew Jackson ab8f09645f unlock: route_unlock returns a structured UnlockRoute; ctx carries opts
Replace route_unlock's Option<(name, Vid)> with a structured UnlockRoute
{ Unlocked(name, Unlocked) | Failed(UnlockError) | NoMatch } so a single
dispatch serves every caller: drive-prep wants "did anything unlock", and the
AACS cert route (next) needs the FAILURE REASON to render "missing keys" vs
"host cert rejected" instead of collapsing it to a bare None. Only a genuine
SCSI transport fault still returns Err (abort). UnlockCtx gains an optional
ScanOptions (the cert route's host-cert source), and read_mkb_from_drive now
takes &mut dyn ScsiTransport — both prerequisites for the cert handshake to
become an external freemkv-unlock-aacs unlocker. Drive-prep + CSS callers fold
the new outcome; no behavior change.
2026-06-29 17:18:11 -07:00
Matthew Jackson 326d17c2f4 unlock: Unlocked carries the bus-key gate nuance; route_unlock returns it
Extend Unlocked with drive_unlocked + read_data_key_err so the registry result
carries everything the downstream bus-key gate needs — a firmware unlocker sets
drive_unlocked=true (bus encryption removed at the drive, no bus key), the cert
route will set read_data_key (+ read_data_key_err on a failed bus-key read).
route_unlock now returns the full Unlocked rather than just a Vid, so a single
dispatch can fold the firmware, cert, and CSS results uniformly. Prep for the
AACS cert handshake becoming a registry unlocker.
2026-06-29 16:59:04 -07:00
Matthew Jackson bac105a022 css: CSS bus-auth becomes a uniform registry Unlocker
Convert the CSS read-unlock into a first-class registry Unlocker (CssUnlocker)
dispatched through route_unlock like every other barrier removal, instead of
a direct call in scan. libfreemkv appends the built-in CSS unlocker (and, next,
the AACS cert handshake) exactly once via ensure_builtins(), AFTER any
client-registered firmware unlocker — so the registry order is firmware → cert
→ css, owned by the lib, not the client.

Defense in depth: the unlocker does NOT trust the caller-declared DiscKind.
matches() filters on the declared kind (Css), but unlock() self-verifies
against the drive's GET CONFIGURATION profile and refuses (UnlockError::
NotApplicable, a new shared "this unlocker doesn't apply" variant) WITHOUT
issuing a single CSS CDB if the drive reports a non-DVD profile — so a
mis-routed Blu-ray is never sent CSS bus-auth. Guard the firmware unlocker the
same structural way (it matches only the drive-prep phase, kind == Unknown).

Tests: CssUnlocker matches only DiscKind::Css; a BD-profile drive yields
NotApplicable with zero CSS CDBs issued.
2026-06-29 16:49:52 -07:00
Matthew Jackson cfcc524367 unlock: introduce UnlockCtx + DiscKind; trait keys off context
Reshape the Unlocker seam so every unlocker is dispatched at ONE place from
ONE ordered registry — the firmware, cert, and CSS routes are all "remove the
bus-encryption barrier", differing only in what they key off. matches() and
unlock() now take an UnlockCtx { drive_id, kind: DiscKind } instead of a bare
DriveId: a firmware unlocker keys off drive_id (kind irrelevant), the cert
unlocker will match DiscKind::Aacs, the CSS unlocker DiscKind::Css. UnlockCtx
is #[non_exhaustive] so a host-cert source can be added without breaking
external unlockers. Drive-prep dispatch passes DiscKind::Unknown (no disc
probed yet); the cert/CSS registry impls + the single post-probe dispatch
point follow in subsequent commits.
2026-06-29 16:27:30 -07:00
Matthew Jackson 03820c68f8 css: auth primitives operate on &mut dyn ScsiTransport; clean up bus_auth
Thread &mut dyn ScsiTransport through unlock_css_reads and its bus-auth /
disc-key primitives instead of &mut Drive — they only ever issued SCSI via
drive.scsi_mut(), so this is mechanical and lossless, and it makes the CSS
unlock speak the same transport interface as the registry Unlocker trait
(prerequisite for CSS becoming a uniform unlocker).

Also clean up the badly-named, half-dead bus_auth:
- rename bus_auth -> establish_authenticated_session: it is run for its
  side effect (sets the drive's ASF=1, unlocking scrambled-sector reads),
  which the name now states.
- drop the derived CSS bus key from the return + computation: it had no
  consumer (descrambling is keyless via the Stevenson attack), so it was
  dead crypto computed on every DVD unlock. Return just the negotiated AGID,
  which the caller genuinely needs for the best-effort disc-key REPORT KEY.
2026-06-29 16:01:07 -07:00
Matthew Jackson f682405973 aacs: handshake primitives operate on &mut dyn ScsiTransport
The AACS cert-auth primitives (aacs_authenticate, the AACS 2.0 P-256
variants, read_volume_id, read_data_keys) and their scsi_read/scsi_write
helpers touched the drive ONLY through Drive::scsi_execute — a pure
pass-through to the transport. Thread &mut dyn ScsiTransport instead of
&mut Drive so these primitives are transport-level, matching the firmware
Unlocker seam (which hands out &mut dyn ScsiTransport for testability).
Pure mechanical signature change, no logic change; the cert orchestrator
(do_handshake_cert) keeps &mut Drive for the OEM-VID shortcut and passes
session.scsi_mut() into the primitives. Step toward making the cert
handshake a uniform registry unlocker.
2026-06-29 15:47:30 -07:00
Matthew Jackson 93fbfac6f0 unlock: Unlocker::unlock returns Unlocked { vid, read_data_key }
Generalize the unlock seam toward uniform unlockers (firmware / cert / CSS).
`unlock` now reports what it LEARNED — a Volume ID and/or an AACS bus key —
instead of just a Vid; libfreemkv files those onto the disc/drive in one
place, so a plugin never touches Disc. The hardware side-effect (extended
mode) still happens inside unlock. route_unlock walks the registry in
registration order (the single ordered place that decides run order); no
priority field. Firmware route reports {vid: Some, read_data_key: None}.
2026-06-29 15:09:17 -07:00
Matthew Jackson a7c8ee09b0 disc: credit firmware unlock in the bus-encryption gate
The bus-key gate only credited the cert handshake's read_data_key as proof
bus encryption was removed. A firmware unlocker removes it AT THE DRIVE
(serves clear content) and yields no read_data_key — so a SUCCESSFUL
firmware unlock (VID present, read_data_key None) tripped the gate and
blocked ALL key resolution, including the online source. That was the
root cause of live UHD discs reporting "missing keys" after an unlock.

Now a single predicate answers "is bus encryption gone?": never-had-it ||
file/ISO || firmware-unlocked || cert-bus-key. The gate is just
`if !bus_encryption_removed { error }` — no enumerated cases. HandshakeResult
gains `drive_unlocked`, and the read_data_key failure reason is captured so
the warn says WHY the bus key is missing.

Also: reword the first hardware-sense escalation as "fast-fail escalation"
(it is often transient — the drive recovers), reserving "wedge" for a
persistent run; and scrub the product name from core comments (it belongs
only in the unlocker crate).
2026-06-29 15:09:16 -07:00
Matthew Jackson 263950622f disc: make bus-key unavailability self-explaining in the log
When a bus-encrypted disc's handshake yields no read_data_key, the gate
logged a bare "bus_key_unavailable" with no indication of WHY — turning
every occurrence into archaeology (is the bus key not-attempted, or did
the read fail?).

Now:
- read_data_keys failure is captured (error code) instead of swallowed by
  `.ok()`, and logged at the handshake with its consequence.
- HandshakeResult carries `read_data_key_err: Option<u16>` so the gate
  distinguishes "never attempted" (None — VID-only/OEM path) from "read
  FAILED" (a code), and the bus_key_unavailable warn now reports the code
  plus whether a Volume ID was present.

No behavior change — purely diagnostic. The handshake_ok debug also now
reports has_volume_id.
2026-06-29 13:33:09 -07:00
Matthew Jackson 9e6af4a729 mux: harden audio discontinuity handling (audit follow-up)
Two defensive hardenings from the post-fix audit (vs FFmpeg/GStreamer):

1. Move the `pes.discontinuity` partial-drop ABOVE the empty-data guard in
   all three audio parsers (ac3/dts/truehd), so a discontinuity signal can
   never be stranded by an empty post-gap PES. The demuxer only emits
   non-empty PES today; this is defense-in-depth for any future caller.

2. A PES with no PTS must not reset the timeline to 0. ac3 now carries
   `flush_pts_ns`, dts continues from the most recent known base; truehd
   already kept its running cadence on a None PTS. Matches OSS behavior
   (PTS rebases off the next PES that actually carries a PTS).

Adds an ac3 regression test (empty-payload discontinuity PES still drops
the stranded partial). Loss accounting was reviewed: TS-demux CC-gaps are
NOT counted toward lost_video_secs / abort (that is sector-based via
DiscStream::errors / mapfile bytes_unreadable), so a source splice never
inflates loss — no gating needed there.
2026-06-29 12:50:52 -07:00
Matthew Jackson be08e3938b mux: drop truncated partial audio frame on concealed gap
The AC-3, DTS and TrueHD parsers buffer access units across PES
boundaries. At a concealed-loss gap the buffered unit is truncated:
splicing post-gap bytes onto it manufactures a corrupt frame on top of
the real loss (FFmpeg "Failed to decode block code(s)" / "Invalid data
found" at the gap) and, for TrueHD, strands the PTS cadence into the
non-monotonic audio-DTS band seen on multi-clip titles.

The video parsers already handle this via the ResyncGate, but the
discontinuity signal was only wired into video — audio parsers ignored
pes.discontinuity and spliced across the gap.

Now, when pes.discontinuity is set, each audio parser drops the partial
(clears buf, and for DTS its PTS marks / pending base) so the post-gap
PES re-bases a fresh unit. A lost gap degrades to a clean single-frame
drop instead of a corrupt spliced frame. No effect on perfect rips: the
branch only runs when concealment inserted a discontinuity marker.

Adds a per-parser test feeding a partial frame then a discontinuity PES,
asserting the truncated partial is dropped (not spliced) and the post-gap
PTS is adopted.
2026-06-29 11:23:08 -07:00
Matthew Jackson cb7d78ac6a disc: adaptive patch speed — enter each range fast, slow on first failure
Pass N (patch) pinned the slow recovery speed (0x0000) for the whole pass.
But Pass 1's damage-jump overshoots, so most of a jumped range is clean data
the reverse-walk reads first — grinding it at slow speed wastes minutes per
gap. Now each range mirrors Pass 1's model:

- Enter at max speed (0xFFFF) + reset to the initial batch (current_batch
  carries across ranges, so a prior single-sector grind would otherwise start
  the next range slow). Reads the clean overshoot fast.
- First read failure in the range → drop to 0x0000 (once; idempotent SET CD
  SPEED) for the rest of that range — grind only the genuine damage.
- 10 s halt-responsive cooldown between ranges, gated on "this range actually
  grinded" so a many-small-range pass (100+ ranges) doesn't stall ~20 min on
  unconditional pauses.

No change to good/bad classification, the mapfile, or the abort math — purely
read speed + an inter-range cooldown. Synthetic speed-transition test to follow.
2026-06-29 10:34:18 -07:00
Matthew Jackson 039a8f9f19 docs: 1.2.0 changelog date → 2026-06-29 (release prep); fix neither/nor grammar in 1.1.0 entry 2026-06-29 09:57:29 -07:00
Matthew Jackson c5515d310f mux/decrypt: audit-nit cleanup + defensive conceal fallback
Non-blocking follow-ups from the 1.2.0 audit:
- conceal loop: if decrypt reported loss but the padding-aware predicate
  matched nothing to conceal (a ~256^-31 contradiction), fall back to the
  strict predicate and conceal whatever it flags, loudly — belt-and-
  suspenders so ciphertext can never reach the mux.
- decrypt_dropped doc: reflect 1.2.0 (mux-path loss is concealed + tallied,
  not silently dropped).
- direct unit test for aacs_unit_still_ciphertext (the padding-aware
  conceal predicate): clear/all-zero/full-decrypted/full-ciphertext/
  decrypted-short-tail.
- fix three stale "v1.1.1" comment refs (the fragment-tail fix ships in
  1.2.0; there is no v1.1.1 release).
2026-06-29 09:51:28 -07:00
Matthew Jackson 789b699f95 mux: make B1 concealment decode-clean on every gap shape
Closes the three residual holes where a concealed/lost gap could still let
a dangling-reference frame reach the muxer (degraded/undecryptable-disc
path only; clean rips are byte-identical and untouched). Root cause: the
discontinuity signal was reconstructed from the 4-bit continuity counter
and applied per-PES, both of which are lossy.

Three coordinated changes:

1. CC-INDEPENDENT marker. fill_null_ts_unit now tags its NULL packets with
   an adaptation-field discontinuity_indicator; the demuxer recognises a
   0x1FFF packet carrying it as a concealed gap and forces a discontinuity
   on every tracked PID (the lost unit's PID is unknowable). This survives
   a loss that is an exact multiple of 16 packets (CC aliases to in-sequence
   — hole 3) and a loss at a PID's very start (no prior CC — hole 4); it
   also drops any open, potentially-truncated partial PES.

2. PUSI ATTRIBUTION. A gap landing on a PES boundary now flags the PES
   STARTING after it, not the one flushed at the boundary (hole 1) —
   stamping the pre-gap frame could arm-then-disarm the gate on a keyframe
   and admit the real post-gap inter frame.

3. PER-FRAME signal. codec::Frame gains `discontinuity`; each parser
   propagates it onto the first post-gap frame. MPEG-2 buffers whole GOPs
   asynchronously, so it associates the gap by ES OFFSET (like PTS/source),
   landing it on the exact post-gap picture mid-GOP (hole 2) — a per-PES
   flag stamped the previous picture. consume_ts (and the EOF flush drain)
   gate on frame.discontinuity.

Tests: CC-independent marker with in-sequence CC + leading-loss; PUSI
attribution flags the post-gap PES; MPEG-2 offset-mark stamps the post-gap
picture through GOP reorder, not the previous one. Existing B1 gate + EOF
tests still green (2270 lib tests).
2026-06-29 09:39:03 -07:00
Matthew Jackson 71b4b09c93 docs: 1.2.0 CHANGELOG — document the breaking public-API changes
Per the release decision: ship the AACS-version-stride API changes under
the 1.2.0 minor bump, with an explicit Breaking section (DiscInputs new
field, DiscInputsCtx::new arity, read_aacs_inputs* 3-tuple return).
2026-06-29 08:41:26 -07:00
Matthew Jackson 067fd207d5 mux: conceal only genuinely-undecryptable units (padding-aware)
The P3 concealment loop in DecryptingSectorSource::read_sectors keyed on
aacs_unit_needs_decrypt, whose sync check is the majority-vote
ts_sync_destroyed (<=16 of 32 syncs). A successfully padding-aware-
decrypted content-fragment TAIL unit (e.g. 11 content packets + 21 zero
padding) has only 11 syncs, so the majority vote called it "still
encrypted" — and when such a good unit shared a read buffer with a
genuinely-undecryptable one (dropped>0), the loop overwrote the GOOD
decrypted tail with NULL-TS, silently discarding correct video and
over-counting concealed units vs the tallied dropped bytes.

Add aacs_unit_still_ciphertext (padding-aware): encrypted AND at least
one non-zero (non-padding) 192-byte packet missing its 0x47 sync — the
same discriminator decrypt_unit uses to accept a fragment tail. The
conceal loop now uses it, so only genuinely-unrestored ciphertext is
concealed. Full and fully-ciphertext units are unchanged. Regression
test: a decrypted short-padding-tail co-resident with a failed unit is
left byte-for-byte intact while the failed unit is concealed.
2026-06-29 00:29:20 -07:00
Matthew Jackson e6180a429b v1.2.0: bump version + reconcile CHANGELOG to [1.2.0] 2026-06-28 23:34:19 -07:00
Matthew Jackson d715a0943a mux: B1 drop-to-keyframe resync after a concealed gap
Pairs with A2 (read-path NULL-TS concealment). When the demux assembler
sees a TS continuity gap it now stamps `discontinuity` on the next
completed PES; the codec-parse stage carries that onto a per-track
ResyncGate. After a gap on an inter-coded video track the gate drops
forward to the next IRAP/IDR keyframe so no frame with a dangling
reference reaches the muxer (an ffmpeg deep scan would otherwise report
a missing-reference / non-existing-PPS error). Audio and subtitle tracks
have no cross-frame references, so the gate is a no-op there.

- ts.rs: PesPacket gains `discontinuity`; PesAssembler tracks a sticky
  pending_discontinuity flag set on CC gap / discontinuity_indicator and
  carried to the next completed/flushed PES.
- resync.rs (new): ResyncGate — per-track arm-on-gap, drop non-keyframes
  until the next keyframe disarms and resumes. Logs the resync + drop
  count once at the keyframe.
- pipelined_stream.rs: precompute per-track is_video, apply the gate in
  consume_ts. Out-of-range track index emits as-is (defensive).

Tests: ResyncGate unit tests; ts.rs gap-stamps-discontinuity; end-to-end
B1 video-drops-to-keyframe and audio-never-drops through PipelinedPesStream.
2026-06-28 23:01:47 -07:00
Matthew Jackson 9a7be7a1a5 1.2.0: mux loss-concealment read path (P3/Edit-2, A2 NULL-TS fill)
Decrypt-verify is a RIP gate, not a MUX gate. On the mux read path an
undecryptable content unit must never abort the mux:

- DecryptingSectorSource gains tolerate_decrypt_loss(): when set, an
  undecryptable in-content unit is tallied, overwritten with valid NULL
  TS packets (PID 0x1FFF) via aacs::fill_null_ts_unit, logged loud with
  its LBA, and the read returns Ok — the stream keeps flowing. The rip
  paths keep the fail-loud DECRYPT_VERIFY_READ decorator (re-read off the
  disc); only the mux opts in.
- Wire it into both mux read paths: the file-backed highway
  (build_iso_pipeline) and the inline DiscStream.
- NULL-TS fill keeps the demuxer byte-synced on the 192-byte stride; the
  lost video/audio PID packets surface as a CC gap the TS assembler
  already drops a partial PES on (the B1 foundation). Ciphertext is never
  passed downstream either way.
- Fix stale resolve_vid_only no-cert test: default is UHD (audit #4).

Tests: conceal-as-NULL-TS, fill well-formedness, fail-loud still holds.
2026-06-28 22:44:19 -07:00
Matthew Jackson a731e7b26b 1.2.0: single MKB framing walker + AACS resolve hardening
- mkb_records() as the one record-framing iterator; rebuild walk_mkb,
  find_record_body, mkb_find_subdiff_records, mkb_content_len, mkb_version,
  mkb_type_raw, mkb_find_mk_dv on it (D4).
- resolve_vid_only / read_aacs_version default to UHD (+warn) on a missing
  content cert instead of BD; route MKB through bounded read_mkb_content.
- AacsVersion major()/from_major() + AACS_MAJOR_BD/UHD as the stride source;
  table + stride-discriminating regression tests.
- read_encrypted_units probes 8 evenly-spaced points per extent (off-midpoint
  scrambled content now sampled); decrypt source-zero mask uses PKT.
2026-06-28 22:28:09 -07:00
Matthew Jackson 4d6f5c0a98 1.2.0: unify hex parsing (one parser) + robust encrypted-unit sampling
- One workspace hex parser (libfreemkv::hex): the keydb / online / mapfile
  parsers had divergent prefix handling (0x vs 0X vs none) — a key written with
  a prefix one parser didn't expect was silently dropped. All three now call it.
- read_encrypted_units probes several points spread across each extent instead
  of only the midpoint-forward window, so a late-starting / sparse encrypted
  body still yields samples (empty samples make decrypt_with skip wrong-key
  validation). A read error at one probe no longer abandons the extent.
2026-06-28 22:12:06 -07:00
Matthew Jackson cad5929afe 1.1.1: unify AACS key-input path on Disc::inputs() + named constants
- read_aacs_inputs* now returns the AACS major version; DiscInputs carries it,
  and DiscInputsCtx parses Unit_Key_RO.inf at the disc's own stride (fixes the
  hardcoded-V20 read-time fetch for V10 discs). One source of truth, no version
  argument to drift.
- Disc::inputs() is the single complete AACS-input source (inf/MKB/VID/hash/
  version); the out-of-band duplicate readers go away.
- Named constants for AACS file paths (aacs::PATH_*) and the AACS majors
  (aacs::AACS_MAJOR_*, AacsVersion::major/from_major) replace magic strings/ints.
- push_ranges saturating (corrupt-disc panic guard).
2026-06-28 21:45:58 -07:00
Matthew Jackson 59681dfd4b 1.1.1: AACS decrypt + key-resolution hardening
- decrypt_unit: padding-aware acceptance — recover real video at content-
  fragment tails (the phantom mux-loss class) without weakening wrong-key
  rejection (a full content unit still needs all 32 TS syncs).
- scan: read the MKB via the bounded read_mkb_content so Disc::inputs()
  carries it. Online key resolution was shipping mkb=0 (a full read of the
  ~128 MiB MKB_RO allocation fails) → the decode service 404'd.
- resolve_vid_only: surface an MKB read error instead of silently emptying.
- fetch: a per-sample dry-set replaces the global fetch_spent latch, so a
  second CPS unit's key can still be fetched after the first came back empty.
- verify::push_ranges: saturating arithmetic (corrupt-disc panic guard).
- Tests for all of the above.
2026-06-28 21:12:57 -07:00
Matthew Jackson eba34f4c20 v1.1.0: bump version 2026-06-28 18:14:18 -07:00
Matthew Jackson 49e5627a69 changelog: 1.1.0 (drop -beta.1 pre-release tag) 2026-06-28 17:58:46 -07:00
Matthew Jackson fb8405385e aacs: rustfmt verify-gate files + correct cert/unit-key test fixtures to libaacs-strict layout; 1.1.0-beta.1 changelog 2026-06-28 16:49:16 -07:00
Matthew Jackson f23338b5dd verify: container-kind seam (HD-DVD-ready) + skip units with unread sectors
ContainerKind {Ts,Ps} + ClipLayout.container thread the post-decrypt structural check per clip; decryptability() dispatches it (TS: unit_is_clean_ts, PS: unit_is_clean_ps). New decrypt_unit_checked(unit,key,accept) decouples the container-agnostic AACS crypto from the format-specific acceptance (decrypt_unit delegates with the TS check). unit_is_clean_ps is the MPEG-2 PS pack-start check, documented UNVALIDATED for HD-DVD (.evo unit/seed/pack alignment must be confirmed on real media). clip_layouts assigns Ts today; .evo->Ps is the one-line HD-DVD hook.

reverify_iso now takes an is_finished predicate and SKIPS any unit with a non-Finished backing sector: we can't verify what wasn't read (a non-Finished sector is zero-filled because the drive read failed there), and must never waste a key lookup on a block the read already knows is bad. observe() (sweep) was already safe (only fed Good bytes).
2026-06-28 16:05:25 -07:00
Matthew Jackson a7bd574c34 verify: post-read decrypt-verify gate + libaacs-strict verify + audit fixes
Post-read verify gate (new src/disc/verify.rs): UnitVerifier buffers/aligns the disc-absolute read stream into clip-file 6144-byte units, then makes one decryptability() decision per unit (CPI gate -> held keys -> key_fetch -> strict TS). POST_READ_VERIFY const kill-switch; fail-safe contract (only ever downgrades units it is confident are undecryptable; every doubt skips). Hooked into Disc::sweep (producer observes ciphertext -> WorkItem::MarkBad after the Good, FIFO-ordered) and Disc::patch (post-loop reverify_iso reads recovered units whole from the patched ISO). extract::clip_layouts enumerates AACS clips for the gate.

Standards-correct AACS verify: aacs::unit_is_clean_ts is a strict port of libaacs _verify_ts (all 32 TS syncs, not a majority vote); decrypt_unit accepts a key only on it; the majority verify_ts is removed. Deleted the Disc::verify_clips post-pass bolt-on (its primitive is absorbed by the read-path gate).

libaacs/DVD audit fixes: content-cert bus_encryption flag now read from bit 7 (was bit 0 - defeated the bus-key fail-loud gate); cc_id read from offset 14; title_cps_unit range-validated + 1->0 index-converted per libaacs. Corrected attack_crib ("functionally-equivalent" not "exact" port) and read_disc_key (READ DVD STRUCTURE 0xAD, not REPORT KEY) doc comments.

Also includes accumulated uncommitted work: key-fetch seam and TrueHD/DTS audio fix.
2026-06-28 15:03:52 -07:00
Matthew Jackson f49ef023cf changelog: enrich 1.1.0-beta.1 and rc.5.2 entries with concrete mechanisms (version label SoT, per-extent AACS anchor, E7017/E7022 no-key split, 3-sector unit grid)
leak-guard / leak-guard (push) Successful in 28s
2026-06-26 22:26:15 -07:00
Matthew Jackson ba5e4fdafa changelog: add 1.1.0-beta.1 version-stamp consistency, DTS-HD MA core-sync, and TrueHD DTS-monotonicity fixes 2026-06-26 22:16:45 -07:00
Matthew Jackson ced89133cc docs(fvi): standardize table headers and multi-line dense cells
Use 'Semantics / reference' as the last-column header across all spec
tables, and break enum/option/note cells onto separate lines with <br>
for readability. Formatting only; no semantic changes.
2026-06-26 21:42:31 -07:00
Matthew Jackson aefd6b6342 mux: single source of truth for the version label (VERSION_LABEL/MUX_APP)
The CLI --version, the MKV muxing/writing-application field, and the FVI generator
all derive from one libfreemkv const, so a binary reports the exact same label it
stamps into the files it produces — no split-brain where an MKV claims one version
and the binary another.
2026-06-26 21:33:19 -07:00
Matthew Jackson 78f78d285e docs(fvi): pretty-print JSON blocks in FVI format spec
Reformat all three JSON code blocks (section 8 ext example, Appendix A
Header schema, Appendix A Record schema) with consistent 2-space
indentation and valid syntax. No field or semantic changes.
2026-06-26 21:21:26 -07:00
Matthew Jackson 80ecb671fd mux: allow build-time version label override (FREEMKV_BUILD_LABEL)
build.rs emits FREEMKV_VERSION = FREEMKV_BUILD_LABEL when that env is set
(non-empty), else the Cargo package version. The muxing/writing-application
field and the FVI generator tag use it, so a pre-release/test build can be
stamped without bumping Cargo.toml and disturbing the tag-pinned [patch]
version matching. The git short hash is still appended either way.
2026-06-26 19:38:13 -07:00
Matthew Jackson c49a180ce7 mux: fix non-monotonic audio DTS (TrueHD + DTS-HD MA) and stamp builds with git hash
TrueHD: when the PES PTS lags the access-unit cadence, resync to the PTS
but never snap the running timestamp backward, so the emitted DTS stays
monotonic across the resync (next_pts_ns = max(next_pts_ns, pts)).

DTS-HD MA: size each EXSS extension substream exactly from its header
(exss_frame_size) and skip it as a unit, so a false 0x7FFE8001 core sync
inside the lossless extension payload can no longer split the access unit
and truncate the extension. Falls back to a bounded scan when the header
is unparseable.

Provenance: build.rs bakes the git short hash into GIT_SUFFIX; the muxing/
writing-application field and the FVI generator tag now record the exact
build (e.g. "freemkv 1.1.0-beta.1 (g835cc99)"), so any output file is
traceable to the revision that produced it.
2026-06-26 19:26:24 -07:00
Matthew Jackson afa218fc8f Remove keydb download/save from the library
keydb I/O moves out of libfreemkv into freemkv-keysources
(KeydbSource::save / ::update). Delete src/keydb.rs entirely (save,
http_get, default_path, write_atomic, UpdateResult) and drop `pub mod
keydb;` — http_get had no real callers. The shared Keydb* Error variants
stay in error.rs (keysources raises them; the every-error-has-a-code
contract depends on them). flate2 is no longer used here, so drop it
from Cargo.toml (zip stays for labels/jar.rs).

CHANGELOG: note the keydb-I/O move; reword the rc.5.2
DefaultDecodedFieldDuration entry to state only the action taken (the
revert) rather than an unverified Windows-fps outcome.
2026-06-26 17:33:57 -07:00
Matthew Jackson 835cc990ad DVD vob_start absolute rebase + rc.5.3 audit fixes
- ifo.rs: rebase VTS title VOBS to absolute disc LBA (file_start_lba +
  vtstt_vobs); fixes DVD rips opening on the menu region instead of the
  movie (e.g. SOTL). Adds absolute-placement regression test.
- aacs/boil.rs: add mk_from_pk primitive (PK -> MK via MKB walk).
- dvdnav/: nav-VM command decoder + start-cell resolver seam, parked
  behind USE_NAV_RESOLVER (kept compiled, never executed).
- mux: FVI src.byte within-sector per spec; Unknown colour -> CICP
  unspecified (2,2,2,1); demux clear PCS -> NORMAL; ts.rs feed() base
  reset + boundary provenance fix.
- Assorted audit fixes (doc/comment/test accuracy) across the crate.
2026-06-26 17:03:58 -07:00
Matthew Jackson d8c323bf9f Magic-number/taxonomy pass: central wire-format + sector + unit consts
- libfreemkv::consts: coding_type::* (ES coding-type bytes), pes_stream_id::*
  + PAYLOAD_RANGE, SECTOR_BYTES (usize) + SECTOR_BYTES_U64 (offset math)
- replace bare wire-code/sector literals across disc, mpls, clpi, labels,
  m2ts_mux, ps, tsmux, file_sector_source, extract
- remove two unreachable secondary-stream match arms in mpls parse_stream_entry
2026-06-26 13:20:21 -07:00
Matthew Jackson decb87a250 AACS pipeline reshape + TrueHD metadata + central consts + clippy/fmt clean
- AACS: delete in-lib keydb parser (Step 3); boil-down primitives
  (mk_from_dk/vuk_from_mk/uk_from_vuk) + newtypes; KeySource->get_uk(ctx)+
  ResolveCtx; Unlocker->unlock()->Result<Vid,UnlockError> + AacsCertUnlocker;
  OEM bus-key gate (AacsBusKeyUnavailable); structured ResolutionTrace (Step 4).
- TrueHD: sample-rate from major-sync, Atmos label, 44.1k AU duration.
- consts: central media/format constants module; 17 duplicate const-defs
  centralized (sector/TS-packet/source-packet); mpls stream-entry + category
  codes named.
- clippy --all-targets -D warnings clean (1.86); fmt clean; 2199 lib tests.
2026-06-26 12:19:24 -07:00
Matthew Jackson 05729f5dfe fix(libfreemkv): rc6 hardening pass — mux timeline/colour/PCR, demux panic sentinel, parser robustness + doc accuracy
Surgical fixes (each with a regression test that fails without the change):

mux/mkv.rs, mux/demux_sink.rs: drive the clip-boundary timeline epoch
off the resolved PRIMARY VIDEO track, not the literal stream index 0.
An M2TS/PMT title can list an audio ES before video, so streams[0] may
be audio; a non-video epoch driver ratchets the frontier and inflates
the timeline. mkv cluster-opening falls back to track 0 for audio-only
titles so they still open clusters.

mux/codec/ac3.rs: correct ACMOD_CHANNELS — acmod=5 (3/1) is 4 channels,
not 3 (was undercounting a 3/1 stream); fix the A/52 Table 5.8 doc.

disc/mod.rs: HDMV coding_type 0x91 (Interactive Graphics / menus) no
longer maps to PGS subtitle — it falls through to Unknown so the PMT/STN
walker drops it instead of surfacing a bogus subtitle track.

mux/videomap.rs + mux/mkv.rs: FVI colour now mirrors the MKV muxer's CICP
precedence (measured CICP authoritative; HDR-driven PQ/HLG transfer
override) via a shared cicp_for_video helper, so the two sinks can't
disagree (HDR10 BT.2020 no longer emits SDR transfer 14).

mux/mkvstream.rs: saturating_add on cluster_ts + rel_ts so an adversarial
CLUSTER_TIMESTAMP near i64::MAX can't overflow/panic before the existing
saturating_mul.

mux/timeline.rs: tighten the tail-straggler clamp so a normal new-epoch
non-video frame leading the sparse video frontier by >3s is not demoted
into the previous clip's epoch.

mux/m2ts_mux/mod.rs: re-stamp PCR per video TS packet (mid-PES), not only
at PES boundaries, so a large UHD I-frame can't open a multi-second PCR
gap; modular 33-bit PTS rebasing so a real 90 kHz clock wrap is not
collapsed to PTS 0 (pre-base frames still floor to 0).

io/byte_prefetcher.rs, sector/prefetched.rs: wrap the producer feed loop
in catch_unwind and emit a typed error sentinel on panic, so a mid-stream
producer panic is not read as a clean EOF at the demux boundary (which
would silently truncate the mux).

mux/codec/h264.rs: extend HIGH_PROFILES to the full ISO/IEC 14496-15 set
that mandates the avcC chroma/bit-depth extension (adds 244 et al.).

Doc/comment accuracy: css/mod.rs (50000 sectors, not scrambled-sectors),
aacs/decrypt.rs (decrypt_unit already-clear path), ifo.rs (TT_SRPT at
0xC4), css/lfsr.rs (LFSR0 24-bit; TAB1-then-XOR cipher; real scramble-flag
predicate), disc/read_error.rs (for_sweep does bounded transient retries).

Skipped: keydb.rs SSRF guard (low/latent, no live caller) — a hard
loopback block breaks an existing behavioral test that exercises the
header-EOF path over a loopback server; a clean fix needs a resolver test
seam beyond this surgical pass. The sibling keydb_fetch.rs comment fix is
out of scope (freemkv crate).
2026-06-25 23:39:03 -07:00
Matthew Jackson dc1d05985b feat(mux): emit HDR10 static metadata from HEVC SEI
Parse the two HDR10 HEVC SEI messages and emit the corresponding
Matroska Colour metadata, only when actually present in the bitstream
(SDR / no-SEI tracks omit it; nothing is fabricated).

Parse (Rec. ITU-T H.265 Annex D):
- Mastering Display Colour Volume SEI, payloadType 137 (D.2.28):
  display_primaries_x/y[3] (SEI order G,B,R), white_point_x/y
  (0.00002 units), max/min_display_mastering_luminance (0.0001 cd/m²).
- Content Light Level Info SEI, payloadType 144 (D.2.35):
  MaxCLL / MaxFALL (cd/m² integers).
HevcParser::scan_sei walks the sei_rbsp ff-extension payloadType/
payloadSize coding and de-emulates (00 00 03) before reading, reusing
the existing strip_emulation_prevention helper. Both SEI are required
before any metadata is surfaced; SEI NALs still pass through unchanged.

Carry: the measured Hdr10Metadata rides PictureInfo (the same per-coded-
picture seam FieldOrder uses), flowing through from_codec_frame onto
PesFrame.coding to the deferred-muxer activate path, where
apply_coding_to_track stamps it on the video track before the header is
written. Set only when both SEI were seen.

Emit (RFC 9559 / Matroska): new Colour children in ebml.rs
(MasteringMetadata 0x55D0, Primary R/G/B + WhitePoint chromaticity
0x55D1..0x55D8, Luminance max/min 0x55D9/0x55DA, MaxCLL 0x55BC,
MaxFALL 0x55BD). write_hdr10 converts chromaticity SEI int × 0.00002 →
Matroska float, luminance SEI int × 0.0001 → cd/m² float; MaxCLL/MaxFALL
are uints verbatim. SEI primary index 0/1/2 (G/B/R) mapped to the
Matroska R/G/B element layout. Emitted only when hdr10 is present.

Tests: SEI parse with exact raw values, requires-both-SEI, SDR omission,
and emulation-prevention stripping (hevc.rs); muxer emit with exact unit
scaling + SDR omission of MasteringMetadata/MaxCLL/MaxFALL (mkv.rs);
apply_coding_to_track HDR10 plumbing (mkvstream.rs).
2026-06-25 21:59:43 -07:00
Matthew Jackson 539b170f7e test(disc): pin the -t 1 = main-feature contract (DVD); fix 2 test clippy nits
Owner-flagged invariant: freemkv -t 1 ALWAYS selects the main feature because the
CLI's title 1 maps to titles[0] and the list is ordered by canonical_title_order
(main feature first). Adds a DVD-shaped contract pin asserting titles[0] is the
movie after sorting (a regression there is a title-ordering bug, not a remux issue),
complementing the existing branching-UHD / normal-disc order tests.

Also clears two pre-existing test-only clippy nits surfaced under --all-targets:
unused `lba` in ClearStubReader::read_sectors, and an unneeded `mut` on the
h264 population test's closure.
2026-06-25 21:42:52 -07:00
Matthew Jackson 998e21c544 test: real-executing coverage for mux codecPrivate/DefaultDuration, sweep damage-jump, patch watchdog clock seam, and AACS CBC KAT
mux/mkv: assert emitted CODEC_PRIVATE bytes verbatim for H.264/HEVC/VC-1/
MPEG-2 (direct TrackEntry child, not nested in Video) and DefaultDuration ns
for all eight frame rates, read back out of a real MkvMuxer.

disc/sweep: end-to-end Disc::sweep against a synthetic MockReader with an
injected bad region, asserting the resulting mapfile marks the clean lead
Finished and the failed batch + zero-filled skip-ahead gap NonTrimmed,
proving the Pass-1 damage-jump engaged.

disc/patch: introduce a minimal clock seam (fn() -> Instant on the internal
PatchLoopState, defaulting to Instant::now) so the per-range and whole-pass
watchdogs are deterministically testable; public API and callers unchanged,
production behavior identical. Add tests that advance a fake clock to trip the
range budget and whole-pass stall predicate.

aacs: add an AES-128-CBC known-answer test for aes_cbc_decrypt using the
published NIST SP 800-38A F.2.2 vector (blocks 1..3 exact; block 0 via the
documented fixed-AACS-IV substitution).
2026-06-25 21:37:20 -07:00
Matthew Jackson 7f55271adb mux: reconcile fvi:// video-index sink onto rc6
Port + adapt the freemkv native per-picture video index (FVI) from the
old feat/fvi-sink branch onto rc6's codec-agnostic PictureInfo model.
This is a surgical adaptation, not a merge.

Adaptations (fvi_sink.rs, videomap.rs, tests/fvi_pipeline.rs):
- Retarget from the removed crate::mux::codec::mpeg2::PictureInfo (raw
  public fields) to rc6's authoritative crate::mux::codec::PictureInfo
  in codec/coding.rs, via its accessors.
- type from coding_type() -> CodingType{I,P,B}; emitted for ANY frame
  that carries coding (every video codec now fills it), with the
  keyframe-flag I/P fallback only when coding is absent.
- Replace the mpeg2-only tff/rff/progressive members with codec-agnostic
  members derived through the accessors: field_order (tff/bff/progressive)
  and progressive, emitted ONLY when the codec measured the signal
  (Option::Some) and omitted otherwise; plus nb_fields.
- Test fixtures rebuilt via PictureInfo::mpeg2(CodingType, Mpeg2Coding{..})
  / coding_type_only(..); added measured_cicp: None to VideoStream
  literals for rc6's struct.

Honesty decision (key / random-access):
- The codec-agnostic PictureInfo carries NO GOP-closure (no closed_gop /
  gop_start), so key is set from the frame's intra / decode-restart flag
  (frame.keyframe == coding.keyframe() for video), NOT a fabricated
  clean-RAP claim. The old gop member is honestly omitted. FVI_FORMAT.md
  is updated to document this as a limitation: key is an intra picture /
  parser-flagged decode-restart point; MPEG-2 open-GOP clean-RAP precision
  (closed_gop) is not currently distinguished. §7.1 rewritten for the
  new field_order/progressive/nb_fields members.

Wiring:
- mux/mod.rs: pub(crate) mod fvi_sink; pub(crate) mod videomap
  (#[allow(dead_code)] on videomap — the VideoMap accumulator is staged
  for side-channel reuse, sink builds records directly); pub use
  fvi_sink::FviSink.
- mux/resolve.rs: add the fvi:// output scheme to StreamUrl, parse_url,
  scheme(), path_str(), input() (write-only reject) and output()
  (constructs FviSink), mirroring the mkv:///demux:// patterns.

Provenance fix surfaced by the end-to-end test:
- pipelined_stream::consume_ps was dropping the PS demuxer's byte-exact
  source stamp (source: None) when rebuilding PesPacket, so PS/DVD-path
  frames reached the mux/index with no provenance (FVI src null). Carry
  ps.source through, matching the TS path; the real-pipeline fvi test now
  sees the stamped src sectors.

Gate: cargo +1.86 fmt + clippy --lib -D warnings clean; cargo +1.86 test
--lib (2182 passed) and --test fvi_pipeline (2 passed); precommit.sh
libfreemkv green.
2026-06-25 21:18:15 -07:00
Matthew Jackson e064bc7055 mux/vc1: measure coding type from progressive PTYPE; carry source
Honest PictureInfo population for VC-1 — completes the set (MPEG-2, H.264, HEVC,
VC-1 all populate now).

- Read the advanced-profile sequence header INTERLACE flag (SMPTE 421M §6.1.1,
  bit 41), de-escaping emulation-prevention bytes as parse_vc1_resolution does.
- For a PROGRESSIVE sequence, decode the picture-layer PTYPE VLC (0=P, 10=B,
  110=I, 1110=BI→I, 1111=Skipped→P) from the first bits after the frame start
  code. For INTERLACED (FCM/FPTYPE precede PTYPE) or simple/main/unknown, decline
  → coding stays None: honestly absent, never read at a wrong bit offset.
- Set coding = coding_type_only(...) and source = pes.source; field order is not
  decoded, so field_order() is honestly None.
- Reuses the shared startcode::BitReader (no new primitive).
- Tests: progressive I/P/B from real PTYPE VLCs + source carry + field-order
  absence; interlaced honest-decline.
2026-06-25 20:50:36 -07:00
Matthew Jackson 6acc26a802 mux/hevc: measure coding type from slice_type; reuse shared BitReader
Honest PictureInfo population for HEVC, on the same principle as H.264.

- Consolidate the bit reader: hevc.rs had its own BitReader (used by the SPS
  parser); h264 just gained one in startcode. Promote startcode's to the single
  shared pub(crate) reader (adds read_bits) and delete hevc's copy — one proven
  primitive, reused (SPS parse + both slice-type decoders).
- hevc: decode slice_type from the first coded slice's slice_segment_header
  (H.265 §7.3.6.1) → I/P/B (§7.4.7.1). The offset to slice_type depends on
  num_extra_slice_header_bits, which lives in the PPS — so we parse it from the
  ACTIVE PPS (§7.3.2.3) and only measure slice_type when that PPS is known.
  With no active PPS we decline rather than guess: coding stays None, honestly
  absent. Set coding = coding_type_only(...) and source = pes.source; field
  order (pic_struct SEI) is not decoded, so field_order() is honestly None.
- Tests: I/P/B from real slice headers, source carry, field-order absence, and
  the no-PPS honest-omission case.
2026-06-25 20:44:15 -07:00
Matthew Jackson 534eca502c mux/h264: measure coding type from slice_type; carry source provenance
Honest PictureInfo population for H.264 — the foundation that lets the video
index claim a frame's type and be believed (a faked "P" on a B-frame would make
the index confidently wrong).

- startcode: shared minimal MSB-first BitReader with Exp-Golomb ue(v), reused by
  H.264 (and next HEVC). Documents the emulation-prevention caveat: only the
  leading slice-header fields are read, where 00 00 03 cannot intervene.
- h264: decode first_mb_in_slice + slice_type (H.264 §7.3.3) from the first
  coded slice and map to I/P/B (§7.4.3 Table 7-6; SP→P, SI→I). Set
  coding = PictureInfo::coding_type_only(...) and source = pes.source. Field
  order is NOT decoded here, so field_order() stays None — honestly absent,
  never guessed.
- Tests: I/P/B from real ue-encoded slice headers, source carry, field-order
  absence; BitReader Exp-Golomb table + truncation.
2026-06-25 20:38:09 -07:00
Matthew Jackson e3dbafcebd mux: codec-agnostic PictureInfo + provenance; measure field order, never guess
Carry per-picture truth and byte-exact source provenance THROUGH the stream so
the muxer (and the upcoming video index) read MEASURED facts instead of
assuming them. Honest data in, honest data out.

- codec/coding.rs: codec-agnostic PictureInfo (CodingType / FieldOrder + the
  accessors field_order/coding_type/nb_fields/progressive/keyframe). Each codec
  folds its raw signals in; consumers use only accessors, never branch on codec.
- mpeg2: builds PictureInfo from the picture coding extension and carries it +
  SourcePos (source_marks, parallel to pts_marks) on every emitted frame.
- pes / codec::Frame: additive `coding` + `source`, forwarded through the
  highway; None for audio/subtitle and the network/stdio deserialize hop.
- mkvstream: DEFER muxer construction until the first coded picture, set the
  video track's FieldOrder from the MEASURED value, THEN write the header —
  right the first time, no guess, no seek-back. An interlaced track that arrives
  with no measured order is LOGGED loudly and left UNDETERMINED, never faked.
- mkv: MkvTrack::video no longer guesses TFF (a bitstream property the scan
  cannot know is UNDETERMINED at build). Removed VideoStream::top_field_first
  (the dead scan-time guess) crate-wide.
- Tests: parser population (every PictureInfo facet + per-PES source carry) and
  mux-stream consumption (measured -> correct; missing -> UNDETERMINED, not
  faked). Two obsolete tests updated only after confirming (their own comments)
  they existed to enforce the deleted hardcoded-TFF.
2026-06-25 20:13:55 -07:00
Matthew Jackson 43fb97f71f mux/mkv: spec-conformance fixes (field order, CICP, VobSub idx, SeekHead, AC-3)
- FieldOrder now derives from the bitstream's measured top_field_first
  (Some(true)→TFF, Some(false)→BFF) instead of hardcoding TFF for all
  interlaced content; falls back to TFF when unmeasured. Adds
  VideoStream::top_field_first; DVD/BD scan sets None with a precise
  TODO(spec) for parser→title plumbing.
- CICP (matrix/transfer/primaries/range) now prefers measured CICP from
  the bitstream (VideoStream::measured_cicp) over the coarse ColorSpace
  enum, so the container stops assuming a colour space the stream may
  contradict. Enum remains the fallback.
- VobSub S_VOBSUB CodecPrivate now emits a `size: WxH` line ahead of the
  palette per the .idx format so players place/scale subs correctly.
- SeekHead: when zero cues are written, the CUES Seek entry is Voided
  instead of leaving a dangling pointer to the Cues offset (now Tags/EOF).
- AC-3 Channels back-patch offset is captured from the writer instead of
  the hardcoded chan_elem_pos+2 (decoupled from the VINT width choice).
- Hoisted inline CICP codes and the dvcC fourcc to named constants citing
  ITU-T H.273 / RFC 9559; fixed the stale FieldOrder comment.
- DefaultDuration vs pulldown: precise TODO(spec) left (needs the same
  parser→title channel as top_field_first).

Tests: BFF-from-measured-flag, measured-CICP-overrides-enum, VobSub size:
line present/omitted, zero-cue SeekHead Void. precommit (1.86) green.
2026-06-25 18:40:30 -07:00
Matthew Jackson 8e0797eab0 wip: rc6 VFR/DVD/CSS base (held for bulletproofing + split) 2026-06-25 18:17:08 -07:00
Matthew Jackson 9b6a48e9d9 demux: solidify sink — reuse canonical primitives, fix 3 bugs
Delete re-implementations in the demux:// sink and wire to proven helpers;
keep only genuinely-new functionality.

- AnnexB reframing: delete the sink's local length_prefixed_to_annexb (it
  break'd on a zero-length NAL, dropping the rest of the access unit) and
  call the canonical append_length_prefixed_as_annex_b in mux::hevc, which
  skips just the empty NAL.
- HEVC param sets: delete hvcc_param_sets; reuse hvcc_to_annex_b.
- avcC param sets: hoist as the new canonical avcc_to_annex_b in mux::hevc,
  next to hvcc_to_annex_b (the symmetry point); the sink calls it.
- PGS .sup: emit a synthetic clear display set (empty PCS + END) at
  pts + duration_ns so subtitles time out instead of lingering to EOF.
- TimelineContinuity: move verbatim into the shared mux::timeline module
  (with the prev_offset straggler-remap intact) and use it from both the
  MKV muxer and the demux sink; delete the sink's drifted TimelineRebase
  copy (which lacked the straggler branch).
- VobSub .idx: emit the conventional 'id: <lang2>, index: 0' line mkvmerge
  reads to assign the subtitle language; palette reuse unchanged.
- output(): seed DemuxOptions.base from title.playlist when non-empty.

New constants for the PGS clear-segment framing and avcC header cite the
public HDMV PGS (BD-ROM Part 3) and ISO/IEC 14496-15 specs.

Tests: a zero-length NAL mid-frame no longer truncates the AU; a frame with
duration_ns produces a .sup clear segment; existing demux tests stay green.
2026-06-25 17:58:49 -07:00
Matthew Jackson 8e2e22af5c mux: highway-level regression — DVD video keyframe + duration survive the pipelined path
The DVD seek-index concern is that the mux highway
(PsDemuxer -> PipelinedPesStream codec parse -> frame out -> muxer)
might drop the keyframe flag or per-frame duration that Mpeg2Parser sets
on each Frame, which would stop the muxer's cluster/cue open
(keyframe && track 0) from ever firing and leave a DVD MKV with many
clusters and zero cues.

Add dvd_highway_preserves_video_keyframe_and_duration: drives a real
Mpeg2Parser through PipelinedPesStream via DemuxBatch::Ps batches (one
PTS-stamped MPEG-2 PS video PES per GOP, decode-order I + P/B), reads the
frames back through the highway's read(), and asserts the GOP-opening
I-frames arrive as keyframes, every frame keeps its duration (BlockGroup
path), and video routes to track 0. Guards from_codec_frame's
keyframe/duration propagation across the demux-thread + recycled-buffer
highway, which the prior cue tests (codec parser straight into the muxer)
did not exercise.

Verified against the real current-VFR-build output
(Greenland-feature.mkv, today): 3365 clusters / 3365 CuePoints, all
BlockGroups, cues resolve to clusters and track the video — the highway
preserves the flags and the seek index is complete.
2026-06-25 16:58:04 -07:00
Matthew Jackson b2e1698b9a mkv: regression test — DVD VFR/BlockGroup output gets per-cluster cues
The existing cue tests (cue_count_equals_cluster_count,
cue_positions_resolve_to_clusters, cue_times_match_cluster_timestamps)
all feed frames with duration_ns=None, so they exercise only the
SimpleBlock write path (UHD/HEVC). DVD MPEG-2 video is now VFR: every
coded picture carries a per-frame duration_ns=Some(..), so it is written
as a BlockGroup, not a SimpleBlock. That cue path was untested.

Add cue_count_equals_cluster_count_blockgroup_vfr, which drives the real
Mpeg2Parser end-to-end (decode-order frames, non-monotonic B-frame
display PTS, telecine field durations) into the muxer and asserts the
output is wholly BlockGroup (no SimpleBlock) and that the Cues index has
exactly one cue per cluster, with every cue resolving to a real cluster.
Guards the DVD seek index (scrub/fast-forward) against regressing to the
chapter-seek-only, zero-cue state.
2026-06-25 16:32:32 -07:00
Matthew Jackson 730af6b1d9 demux: add demux:// per-track elementary-stream sink
New write-only pes::Stream sink that taps the per-track PesFrame stream
(the seam right before MKV muxing) and writes each track to its own
elementary-stream file, plus chapters and per-audio-track delay metadata.
Purely additive — the MKV mux path is untouched.

- mux/demux_sink.rs: DemuxSink + EsWriter dispatch. Pass-through for
  codecs whose Frame.data is already standalone ES (MPEG-2, VC-1, AC3/
  E-AC3, DTS/DTS-HD, TrueHD, LPCM). Non-trivial writers:
  - AnnexBWriter: reframes hvcC/avcC 4-byte-length-prefixed NALs to
    Annex-B and prepends VPS/SPS/PPS parsed out of the codec_private
    configuration record (HEVC .hevc / H.264 .h264).
  - PgsSupWriter: rebuilds the HDMV 'PG' segment framing the parser
    strips, with 90kHz PTS/DTS (.sup).
  - VobSubWriter: writes raw SPUs to .sub and synthesizes the .idx
    sidecar (palette + per-SPU timestamp/filepos).
  - Delay-in-filename (mkvmerge-readable 'DELAY <n>ms') + chapter XML/OGM
    export. TimelineRebase ports the MKV muxer's seamless-branch epoch
    logic so per-track ES timestamps stay continuous across clip joins.
- mux/resolve.rs: StreamUrl::Demux variant + scheme/path_str/parse_url/
  input(write-only)/output arms.
- mux/mod.rs: module + public type re-exports.

16 unit tests: Annex-B reframing, hvcC/avcC param extraction, delay
sign/rounding + mkvmerge-regex match, PGS .sup framing, VobSub .idx
synthesis, chapter XML/OGM, timeline rebase, and end-to-end file-keying
by track + track selection.
2026-06-25 16:23:37 -07:00
Matthew Jackson 52d2e85e3c css: skip clear/uncrackable extra titles instead of failing the whole mux
A genuinely-clear or uncrackable extra title (a tiny menu/nav stub) no
longer poisons a multi-title rip with a false CssKeyMissing (E7023).

- decrypt_keys_for_title_checked: re-crack a non-overlapping VTS via
  crack_key_outcome and report title_is_clear when the title's own
  extents show no scrambling. A genuinely-clear stub on an otherwise-CSS
  disc needs no key.
- ensure_title_decryptable: pass a clear stub without a key; a scrambled-
  but-uncrackable title still hard-fails with CssKeyMissing.
- is_scrambled_pack: hardened scramble-evidence gate for the crack scan —
  requires the MPEG-PS pack-start signature before trusting the 0x14
  scramble bits, so a clear stub with stray 0x14 bits can't flip
  saw_scrambled. The descramble loop keeps the looser is_scrambled.
- mux/resolve: ISO per-title gate routes through the clear-aware check.
2026-06-25 13:43:04 -07:00
Matthew Jackson 6b3014f3e8 progress: expose bytes_retryable so 'lost' counts only failed reads
Add PassProgress::bytes_retryable_total (NonTrimmed/NonScraped — failed
and awaiting retry), distinct from bytes_pending_total which also folds
in not-yet-attempted (NonTried) bytes. Set it at every construction site
(Sweep from the snapshot, Patch from stats, 0 for sequential/placeholder
paths). The disc-level 'lost' display in the CLI can now use
unreadable+retryable instead of unreadable+pending, so a healthy
in-progress rip no longer reports its unread remainder as lost.
2026-06-25 13:04:37 -07:00
Matthew Jackson 8f5968a18f Resume plain disc->iso copy after interrupt
Disc::copy gated its entire mapfile-resume dispatch behind
`if opts.multipass`, so a plain (non-multipass) `disc:// iso://`
copy always called sweep_internal(resume=false) — wiping the
mapfile + ISO and re-sweeping from sector 0. The CLI help and
rip_iso examples promise "auto-resumes if interrupted", and the
sweep already flushes a crash-safe mapfile per block, so the only
missing piece was consulting it on re-run.

Lift the mapfile-resume dispatch out of the multipass guard:
clean->no-op, size-mismatch->fresh sweep, NonTried tail->resume
sweep now apply to both modes. The patch (Pass N) dispatch on
retryable bytes stays multipass-only; a plain copy with a
fully-attempted mapfile returns a terminal result instead of
restarting. autorip (always multipass=true) is unchanged.

Add a regression test that an interrupted plain copy resumes the
NonTried tail and does NOT re-read the Finished prefix.
2026-06-25 08:24:00 -07:00
Matthew Jackson 62d2dfe96a libfreemkv: align E7021→E7017 in comments/test-names (keys-but-no-VID is AacsVidUnavailable) 2026-06-25 07:59:54 -07:00
Matthew Jackson 3c42950ecd libfreemkv: changelog — rc.6 entry (+ backfill rc.5.3, mark rc.5.2 released) 2026-06-24 23:56:41 -07:00
Matthew Jackson 0127c274d2 test: strengthen AACS resolve-reason + unlocker seam coverage (rc.6 WS3)
Targeted tests for the rc.6 surfaces, strengthening (not duplicating) the
regression tests the rc.6 commits already shipped. No production code changes.

aacs/keys.rs — resolve_keys_with_reason / classify_resolve_failure:
The E7021/E7022 split is already proven end-to-end through the
ensure_decryptable gate (disc/mod.rs). These pin the classifier directly at
the keys.rs seam for the branches the gate test does not reach:
  - processing-keys-only + zero VID -> VidUnavailable (the gate test only
    exercises the device-keys arm of has_derivation_material).
  - VID PRESENT + material -> NoMaterial: a non-zero VID must never be
    reported as VidUnavailable however much material is on hand (the has_vid
    short-circuit; the gate test only uses the zero-VID sentinel).
  - VID present + no material -> NoMaterial.
  - version dispatch: version 1 routes the V10 resolver (stamps V10), any
    other value routes the V20->V21 chain; a resolved disc returns Ok, never
    Err(ResolveFailure).

unlock.rs — Unlocker seam introspection + ordering:
  - matching_name reports the first matching unlocker without running it, and
    is None for an unsupported drive; registered_count grows after a
    registration (monotonic check — the registry is process-wide and shared
    across the unlock tests, so no exact-delta assertion).
  - route_unlock first-registered-match-wins: two unlockers matching the same
    identity, the earlier-registered one runs and the later is never consulted.
2026-06-24 23:40:36 -07:00
Matthew Jackson f407c4c693 fix(extract): anchor AACS unit base per-extent + crash-safety hardening
The dir:// extractor set the AACS unit-alignment base ONCE to the first
extent's start, then read every extent against that single base. For a
multi-extent (fragmented / Long-AD / continuation-ICB) file the second
and later extents start at arbitrary LBAs whose offset from the first
extent is generally not a multiple of 3 sectors, so the first read of
each later extent failed the decrypt-on-read gate
(is_unit_aligned(lba, unit_base)), returned DecryptFailed, and recorded
the whole extent as a zero-filled hole even though the data was readable.
Re-anchor the unit base PER extent (matching mux/disc.rs and
sector/prefetched.rs), so each clip's encrypted region gates on its own
unit grid. Same bug class as the rc.5.2 clip-anchor fix.

Also harden the extract + keydb write paths:
- finalize_file: fsync the .partial after set_len (the truncation runs
  on a second handle the content fsync never touched) and fsync the
  parent dir after rename so the new dirent is crash-durable.
- keydb write_atomic: fsync the parent dir after rename (POSIX dirent
  durability), matching the finalize_file pattern.
- AACS tail batch: document that decrypt_sectors' trailing-partial
  contract already handles the short final unit; no math change.
- decrypt-loss delta loads use Acquire (defensive happens-before if
  file extraction is ever parallelised).
- is_windows_reserved: add CONIN$/CONOUT$/CLOCK$; reserved names are
  now substituted (prefix _) instead of aborting the whole tree walk, so
  a legal Linux-authored NUL.cfg extracts.
- http_get header cap: >= MAX_HEADER_BYTES (was > , one byte over).

Regression tests: multi-extent AACS file (Δ4-sector extents) extracts
both extents with zero loss; focused per-extent alignment-arithmetic
test; reserved-name substitution assertions.
2026-06-24 23:20:15 -07:00
Matthew Jackson d4a0f5b786 aacs: split no-key resolution reason into E7021 vs E7022
When key resolution had derivation material (device or processing keys)
but no Volume ID was available to derive the unit key, surface
Error::AacsVidUnavailable instead of the generic NoDiscKey. When there
was no usable key material at all, keep NoDiscKey.

resolve_keys_classical / resolve_keys_v21 still return a bare
Option<ResolvedKeys> (all existing callers unchanged); a new
resolve_keys_with_reason wrapper threads the typed ResolveFailure
(VidUnavailable | NoMaterial) out. decrypt_with uses it; the
ensure_decryptable_keys gate maps a captured AacsVidUnavailable reason to
E7021, otherwise E7022. No decryption math, key derivation, or descramble
logic changed -- only the reason reported on a resolution failure.

Adds ensure_decryptable_aacs_vid_unavailable_vs_no_key proving both
branches (device-keys + zero VID -> E7021; no keys -> E7022).
2026-06-24 22:38:54 -07:00
Matthew Jackson 1854869ab3 v1.0.0-rc.5.3: bump version 2026-06-24 21:50:00 -07:00
Matthew Jackson 275d9eebe0 libfreemkv: plain-English, source-agnostic AACS scan logs
Rewrite the AACS scan/VID trace lines so a reader understands them without
opening the source: name the real thing (AACS host certificate, Volume ID,
decryption key), say "key source" not "keydb", and describe what happened.
The VID flow is unchanged (unlocker OEM VID → cert handshake → continue); a
missing VID is logged, never fatal. All strings are in tracing macros (the
sanctioned debug-log channel) — no English added to any Error.
2026-06-24 21:47:37 -07:00
Matthew Jackson 05b9befc64 libfreemkv: keydb default location is local to the executable
default_path() now resolves <dir of current exe>/keydb.cfg with no OS-config
fallback — freemkv is a portable standalone binary, so the keydb lives next to
it. --keydb still overrides.
2026-06-24 20:46:31 -07:00
Matthew Jackson 1767cf67b6 libfreemkv: dir:// — decrypted file-tree extraction (Disc::extract_tree)
Sibling of Disc::copy specialized to write per-file instead of a whole ISO
image, decrypting on the way out: walk the UDF tree, read each file's extents
through the shared DecryptingSectorSource (AACS unit-aligned, CSS per-VTS),
strip AACS/, sanitize host paths per component, .partial+rename, 1-shot with
per-file loss accounting (no mapfile; recovery stays the iso:// multipass
path). Reuses UdfFs + the decrypt seam; only the per-file orchestration is new.
2026-06-24 20:46:31 -07:00
Matthew Jackson 9a3f6b7313 libfreemkv: hard-error when decryption is needed but no key is available
Adds Disc::ensure_decryptable / ensure_decryptable_keys, the single decrypt
gate consulted before any copy or mux. When the source is encrypted and no
key resolved (and not --raw), abort with a typed error and write nothing,
instead of silently emitting ciphertext at exit 0. Unifies the prior ad-hoc
CSS/AACS checks.
2026-06-24 19:33:09 -07:00
Matthew Jackson 1f91eebb9a v1.0.0-rc.5.2: bump version 2026-06-24 17:14:10 -07:00
Matthew Jackson f4a475c7b9 libfreemkv: rc.5.2 changelog — audio routing, AACS clip-anchor, recovery, TopGun, keysources move 2026-06-24 17:12:17 -07:00
Matthew Jackson 5b0976859f libfreemkv: rc.5.2 SOTL video — full Windows fps, opening-GOP proof, self-sufficient log-level 3
Three Silence-of-the-Lambs (R2 PAL SD-DVD) follow-ups for rc.5.2.

SUB-TASK 1 — Windows Explorer showed 12.5 fps (half) for the 576i25 track.
Root cause: the DefaultDecodedFieldDuration (20 ms field) element rc.5.1
added to "fix" Windows fps did the opposite. With FlagInterlaced=1 +
DefaultDuration=40 ms + DefaultDecodedFieldDuration=20 ms, Explorer halved
to 12.5 fps and MediaInfo flipped to VFR. MakeMKV's correct rip omits the
field-duration element, keeps FlagInterlaced=1 + FieldOrder=TFF +
full-frame DefaultDuration (40 ms), and Explorer shows 25 fps / MediaInfo
CFR. Fix: MkvTrack::video now passes field_duration_ns == 0 so the element
is no longer written; the 1/DefaultDuration = 25 fps signal (the only one
tools trust) is the full-frame value. Interlace signalling (FlagInterlaced,
FieldOrder=TFF) is retained — MediaInfo reads scan type from the MPEG-2 ES
picture coding extension, so it still reports Interlaced / Top Field First.
Tests pin the new TrackEntry elements (element present/absent + values).

SUB-TASK 2 — opening "menu"/still-frame video. Traced the MPEG-2
opening-GOP path; the wrong/last seq header and PTS-floor-to-0 hypotheses
are RULED OUT with file:line evidence: codecPrivate is the FIRST sequence
header (read once at headers-ready, mkvstream.rs:115 + pipelined_stream.rs:289),
DVD VOBU structure guarantees each title opens on seq header + I-frame (no
mid-GOP open), the parser back-anchors leading still-frames to the disc's
real timeline (mpeg2.rs:296-303), and the muxer anchors base on the opening
keyframe's real PTS so the t=0 floor (mkv.rs:963) never corrupts it.
Regression tests pin all three (parser + muxer level).

SUB-TASK 3 — make --log-level 3 self-sufficient (diag.rs + minimal hooks).
(a) dump the ACTUAL MKV TrackEntry elements written per track
(tag=mkv.track: FlagInterlaced, FieldOrder, DefaultDuration, field duration,
Display dims, codecPrivate hex) so Windows-fps-class metadata is verifiable
from a log alone. (b) capture the first ~100 coded frames per track (raw)
to <output>.opening.bin with a per-frame summary line (tag=mkv.opening.frame:
track, key/delta, size, PTS) so opening-GOP/menu issues are diagnosable from
a future log without the disc. Both gated to log-level 3; normal runs open
no side file and record nothing.

CI gate (Rust 1.86): fmt --check, clippy -D warnings, and test --tests all
green.
2026-06-24 17:04:56 -07:00
Matthew Jackson f72a956b5b fix(dvd-audio-probe): read each sub-stream's true max channel count
The DVD AC-3 sub-stream probe recorded the FIRST decodable frame of each
physical 0x8x sub-stream as its channel count. A DVD feature opens with
logos/warnings whose audio is often a thin 2.0 bed on 0x80 before the
real 5.1 main mix begins a fraction of a second later. The probe locked
onto that opening 2.0 frame and reported 0x80=2, missing the 5.1
entirely (confirmed on Greenland: 0x80's head frames are acmod=2, then
acmod=7+lfe). With no 6-channel sub-stream found, channel-match routing
fell back to the ordinal map — harmless on Greenland, but on a disc where
the 5.1 lives on a non-ordinal sub-stream the wrong-substream bug stays
unfixed.

Fix: scan EVERY 0x0B77 frame of each sub-stream in the probe window and
keep the MAXIMUM channel count (the sub-stream's real main-mix
capability), advancing frame-by-frame via ac3_frame_size so a frame body
can't be mistaken for a new sync. Also bump PROBE_SECTORS 512->1024: the
1 MiB head window saw ONLY 0x80; 2 MiB reliably contains a frame of every
physical sub-stream.

Greenland tag=dvd.substream: before 0x80=2 (only); after 0x80=6, 0x81=2,
0x82=2 — matching the IFO and the decoded output.

Adds probe_reads_max_channels_no_cross_contamination regression test.
2026-06-24 16:55:34 -07:00
Matthew Jackson 9cd36427be libfreemkv: fix rc.5.2 audit code findings
1. HEVC CRA->BLA false-trigger on 33-bit PTS wraparound
   (src/mux/codec/hevc.rs): the clip-boundary auto-detect compared the
   RAW 33-bit PES PTS against the high-water mark, so a single-clip title
   crossing 2^33->0 (~26.5h) false-armed pending_clip_boundary and rewrote
   a legitimate in-clip CRA(21)->BLA_W_LP(16), dropping valid RASL pictures
   (visible corruption) and breaking the single-clip byte-identical
   guarantee. Now unwrap the PTS onto a monotonic 64-bit timeline first
   (a near-full-period backstep is a wrap: add 2^33, update the watermark,
   do not arm). Regression test cra_after_33bit_pts_wrap_not_rewritten;
   the genuine-clip-join test still passes.

2. Single-pass recovery read bypassed the transport-failure abort
   (src/mux/disc.rs): the line-442 short-circuit only inspected the 10s
   read res. A transport failure (status 0xFF, wedged USB bridge) on the
   60s recovery read fell into the skip_errors branch and zero-filled/
   advanced, marching the disc at one bridge-recovery per probe
   (run-forever, hard rule #2). Re-check the recovery error for
   is_scsi_transport_failure() before the skip block and abort with
   Error::DiscRead. Test transport_failure_on_recovery_read_aborts_even_with_skip_errors.

3. Recovery-read SUCCESS branch had no coverage (src/mux/disc.rs tests):
   added RecoverableReader (errors when recovery=false, succeeds when
   recovery=true) and test recovery_read_success_muxes_recovered_data_no_skip
   driving fill_extents to the size-1 bottom-out and asserting the recovered
   data is muxed (counters advance, no skip).

4. TrueHD channel-correction probe omitted set_unit_base
   (src/disc/mod.rs correct_truehd_channels): the probe read via a
   DecryptingSectorSource without anchoring the AACS unit-alignment gate,
   so it degraded to absolute start_lba % 3 and returned DecryptFailed on a
   non-3-aligned extent, silently understating Atmos/7.1 as 5.1. Now call
   set_unit_base(ext.start_lba) before the probe read (no-op for CSS/None).

5. is_unit_aligned lba<unit_base latent trap (src/aacs/decrypt.rs):
   wrapping_sub mis-gated when lba < unit_base (2^32 == 1 mod 3). Switched
   to saturating_sub (clamps offset to 0, a unit boundary) and pinned the
   contract with is_unit_aligned_lba_below_base_is_well_defined plus
   is_unit_aligned_relative_to_base.

cargo +1.86 fmt --check / clippy -D warnings / test --tests all green.
2026-06-24 16:31:28 -07:00
Matthew Jackson 674a7dd867 dvd: route AC-3 audio to the physically-correct sub-stream by probed channel count
Fixes the "Silence of the Lambs" R2 PAL wrong-substream rip: the feature's
IFO declares one 5.1 AC-3 stream, but the scan assigned it the on-wire
sub-stream id 0x80 purely by per-codec ordinal (ifo::assign_audio_sub_stream_ids).
On this disc the physical 0x80 carries the 2.0 down-mix and the 5.1 main mix
lives at a different 0x8x sub-stream, so the rip muxed 2.0 while labelling it
"Dolby Digital 5.1" (the acmod fixup in mkv.rs then corrected only the Channels
element, surfacing the mismatch as the "IFO claimed 6 but acmod says 2" warning
— too late to re-route).

New src/disc/dvd_audio_probe.rs probes each physical AC-3 sub-stream's real
channel count from the head of the feature (the acmod/lfeon of its first frame
after the 0x0B77 sync) and re-routes each IFO-declared AC-3 stream onto the
physical sub-stream whose actual channel count matches the declared count,
instead of trusting the ordinal. Wired into both mux demux paths
(DiscStream::new and resolve::build_iso_pipeline) over the decrypting reader,
so it works on CSS discs and the autorip ISO-remux path alike. Bounded
512-sector best-effort read; an empty/unreadable probe degrades to the original
ordinal mapping (no regression on normal discs).

The cell selection is left unchanged: the feature's cell 0 (cat=0x02, 302.4s)
is chapter 1 of the movie (matches MakeMKV's chapter map and 1h53 duration
exactly), so it must NOT be dropped — the perceived "wrong video at the start"
was the wrong 2.0 audio over the opening, the same root cause.

Diagnostics (--log-level 3): new tag=dvd.substream rows dump the ACTUAL acmod
channel count of each physical 0x8x sub-stream read from the VOB, and the
per-cell tag=dvd.cell verdict now spells out the keep/skip reason. With the
existing tag=dvd.aattr (IFO declared sub_id + channels) a bug log alone now
shows whether the ordinal 0x80 really carries the declared layout — no disc
needed to diagnose this class.

expose ac3::find_ac3_sync as pub(crate) for the probe.
2026-06-24 16:28:21 -07:00
Matthew Jackson 1cec2aaaf3 mux/hevc: auto-detect non-seamless clip boundary, rewrite splice CRA→BLA
The "TopGun bug" (Top Gun 1986 UHD, DV Profile 7 dual-layer): the auditor
flags the rip `corrupt` with a flood of HEVC "Could not find ref with POC N"
decode errors (POC 114/210/228/234/240/246/252/318/336/354/372/399 — one
cluster per clip join).

Root cause: the title is multiple .m2ts clips joined at non-seamless
boundaries (mpls connection_condition 0x05/0x06), read as one concatenated
stream. Each next clip opens with a CRA whose RASL leading pictures reference
pre-join frames gone after concatenation. hevc.rs already has the spec remedy
(rewrite splice CRA_NUT 21 → BLA_W_LP 16 so a decoder sets NoRaslOutput and
drops the dangling RASL), behind mark_clip_boundary() — but nothing ever
called it: connection_condition is not plumbed through the threaded mux
pipeline, so the whole CRA→BLA mechanism was dead code.

Fix: detect the boundary inside the parser from the bitstream. Each clip
carries its own PES PTS base, so a non-seamless join is a large backward
PTS reset. HevcParser::parse now tracks a PTS high-water mark and, on a
backward step beyond 3 s (270000 ticks @ 90 kHz — mirroring the mux-side
DISCONTINUITY_BACKSTEP_NS), arms the existing CRA→BLA rewrite for the new
clip's first IRAP. Self-contained: works for every mux path, no cross-thread
plumbing. The 3 s threshold sits above any HEVC B-frame reorder dip and far
below any clip duration, so it never false-triggers in-clip; single-clip /
seamless titles never arm it and stay byte-identical.

Adds regression test cra_at_auto_detected_pts_backstep_rewritten_to_bla
(in-clip dip must not trigger; splice CRA after a backward reset must become
BLA; one-shot). hevc suite 55/55, mux suite 780/780.
2026-06-24 15:41:26 -07:00
Matthew Jackson 794d88f6e7 libfreemkv: rc.5.2 DVD test coverage — depth-aware mux, colour codes, CSS scan
Implements the rc.5.2 quick-units list from the DVD coverage audit and
corrects the "passes-but-encodes-the-bug" tests that could not
distinguish correct from wrong behaviour.

New tests (each with the bug it guards):

mux/mkv.rs
- field_duration_is_direct_trackentry_child_not_in_video: depth-aware
  check that DefaultDecodedFieldDuration (and DefaultDuration) are direct
  TrackEntry children, NOT nested in the Video master. Replaces the flat
  find_id byte-scan that passed either way. Adds master_children /
  first_track_entry depth-walking helpers.
- pal_576i_emits_bt470bg_colour_codes / ntsc_480i_emits_smpte170m_colour_codes:
  assert the actual CICP tuples written into the MKV Colour master —
  PAL (5,5,5,1) vs NTSC (6,6,6,1) — not just stream-layer ColorSpace.
- ntsc_480i_field_order_is_tff_and_encoded: pins NTSC 480i hardcoded TFF
  and its ~33.37ms/16.68ms frame/field durations, asserting the encoded
  FlagInterlaced/FieldOrder bytes (480i was never exercised before).

ifo.rs
- video_attr_absolute_bytes_pin_real_layout: drives parse_video_attr with
  HARDCODED real DVD-Video bytes (PAL/NTSC x 4:3/16:9, plus mpeg_version
  in bits 7-6) instead of v_atr_byte, so a co-edit of the shift constants
  can't re-seed the PAL-as-NTSC bug. Anchors that permitted_df bits (1-0)
  are not read as the TV system.

disc/dvd.rs
- scan_dvd_titles_lpcm_routes_to_a0_pid_range: LPCM (coding 4) → sub-id
  0xA0 → PID 0xBDA0, disjoint from the AC-3 0xBD8x space, channels kept.
- scan_dvd_titles_multiple_vobsub_tracks_distinct_pids: three VobSub
  tracks → distinct 0x20+ordinal PIDs, per-language, shared palette.

css/mod.rs
- crack_outcome_reaches_cracked_with_span: drives the full crack scan to
  CrackOutcome::Cracked via a Stevenson-crackable synthetic sector and
  asserts crack_span recording (the Cracked branch was never exercised).
- recrack_succeeds_on_other_vts_extents: per-VTS re-crack SUCCESS path.
- all_locked_synthetic_iso_yields_css_key_missing_signal: all-locked
  multi-extent ISO → ScrambledUncracked, the signal the scan converts to
  css_error = Some(CssKeyMissing).

Corrected fixtures (passes-but-encodes-the-bug):
- scan_dvd_titles_mixed_audio_codecs_distinct_pids: real channel nibbles
  (AC-3 5.1 = 6ch, DTS 2.0 = 2ch) replacing the 1ch placeholders; asserts
  channel counts and exact canonical PIDs (0xBD80 / 0xBD88).
- ebml.rs FieldOrder comment: drop the stale "PAL DVD (576i) is
  bottom-field-first" line that contradicted the TFF-for-all code.
2026-06-24 15:41:26 -07:00
Matthew Jackson 63ed05bd63 libfreemkv: clip-anchored AACS unit gate + consolidate key mechanism
The AACS unit-alignment gate measured `lba % 3` against absolute disc LBA 0,
but aligned units are anchored at each clip's encrypted-region start. A clip
whose start_lba is not 3-aligned had its readable units wrongly rejected with
"Decryption failed" (the big-title-only failure on some Blu-rays). One
canonical clip-anchored helper (`aacs::is_unit_aligned`) is now the single
source of truth for the decrypt-on-read gate; both mux read paths set the
per-extent `unit_base = start_lba` via a new `SectorSource::set_unit_base`.

Also moves key *mechanism* into the library: the encrypted sample reader
(`read_encrypted_units`) and the candidate-key resolution loop
(`resolve_and_apply`) now live here, so a key source is purely a lookup.
Regression test covers a clip based at a non-3-aligned LBA.
2026-06-24 15:40:50 -07:00
Matthew Jackson 987e26e44d libfreemkv: single-pass disc->MKV recovers marginal sectors before giving up
Single-pass disc->MKV has no Pass N, so its read bottom-out now issues one
bounded recovery read (recovery=true, ~60s ECC) before skipping or aborting,
matching the multipass patch. Fixes a transient/marginal sector surfaced as a
read failure direct-to-MKV while multipass recovered it. One read, not a loop
(hard rule #2); recovered data is used so no bogus-status hole reopens.
2026-06-24 14:58:32 -07:00
Matthew Jackson b76e9d38c5 v1.0.0-rc.5.1: bump version 2026-06-24 14:36:29 -07:00
Matthew Jackson 6592f2a590 libfreemkv: rc.5.1 DVD correctness fixes
- CSS: unlock scrambled-sector reads on enforcing drives via bus-auth
  only; classify sense 6F/03 as CSS-locked; early-bail on a fully locked
  scan; gate the AACS handshake off DVD discs.
- DVD first-play menu no longer prepended to the feature: read the title
  VOBS base from vtstt_vobs (0xC4), not the menu VOBS vtsm_vobs (0xC0).
- Interlaced field-duration (DefaultDecodedFieldDuration) written as a
  direct TrackEntry child rather than inside Video, so Windows reports
  the correct frame rate.
- Audio channel count read from the AC-3 bitstream; FieldOrder set to
  TFF; per-track BPS tags.
- Structured disc diagnostics at --log-level 3; reduced per-operation
  log spam.
2026-06-24 14:34:55 -07:00
Matthew Jackson 315276dd13 Add Contributor Covenant v2.1 Code of Conduct 2026-06-24 10:44:58 -07:00
Matthew Jackson 9d40da3982 v1.0.0-rc.5: bump version 2026-06-24 10:27:30 -07:00
Matthew Jackson e0ce035765 Preserve interlaced scan type in label and MKV output
Interlacing is detected upstream (PAL DVD -> R576i) but was dropped in
two places: the video label hardcoded a 'p' suffix, and the muxer never
wrote any scan-type flag, so MediaInfo inferred progressive and reported
576p for a 576i source.

- Add Resolution::is_interlaced() for the R*i variants.
- generate_video_label now branches i/p for the heights that can be
  interlaced (1080, 576, 480) instead of always emitting 'p'.
- MkvTrack carries interlaced + field_order; the video serializer emits
  FlagInterlaced (0x9A; 1=interlaced, 2=progressive) and, for interlaced
  content, FieldOrder (0x9D) - bottom-field-first for PAL 576i,
  top-field-first otherwise. Adds the EBML constants.
2026-06-24 09:47:16 -07:00
Matthew Jackson 2613a81f09 Fix AC-3 channel count read from wrong nibble in VTS audio attr
The VTS_AST_ATR byte 1 stores (channels - 1) in the low 3 bits, but we
were reading the high nibble. A normal 2.0 AC-3 track has 0x1 in the low
nibble and 0x0 in the high nibble, so we computed 0+1=1 and mislabeled
every stereo track as mono in the muxed Channels element.

Read the low 3 bits instead, and correct the two audio_attr tests that
had encoded (channels - 1) in the high nibble to match the buggy reader.
2026-06-24 09:39:02 -07:00
Matthew Jackson 45a7b74ab9 fix: correct stale FAIL_PAUSE_SECS doc claiming uniform sweep+patch application 2026-06-24 05:57:39 -07:00
Matthew Jackson 8b8ada7802 fix: drop BytePrefetcher channel endpoints before join to prevent deadlock
Drop rx and recycle_tx (now Option fields) before joining the producer
thread in BytePrefetcher::Drop. Without this, a non-EOF source fills the
depth-2 forward channel then spins in send_timeout(POLL_INTERVAL) forever
because rx is never drained; join() deadlocks. Adds a regression test
(drop_endless_prefetcher_joins_cleanly) that directly exercises the path.
2026-06-24 05:50:58 -07:00
Matthew Jackson cdd11ccb25 fix: correct stale doc on PatchItem::Unreadable — promotion shipped via Mapfile::record, not PatchSink 2026-06-24 05:02:24 -07:00
Matthew Jackson c706a94312 fix: correct stale css::crack comment reference to css::crack_key 2026-06-24 04:57:47 -07:00
Matthew Jackson 06ee748689 fix: remove stale sweep_pipeline.rs cross-references in patch.rs comments 2026-06-24 04:52:04 -07:00
Matthew Jackson 95b9762c51 Revert "fix: discard duplicate TS packets in non-PUSI continuation path"
This reverts commit f3c3614a17.
2026-06-24 01:59:49 -07:00
Matthew Jackson fb13f975df fix: enforce WEDGE_FAMILY_COOLDOWN_SECS == ZONE_ENTRY_COOLDOWN_SECS at compile time 2026-06-24 01:52:43 -07:00
Matthew Jackson dbc2225315 fix: correct Profile-07 comment — prime_cache(lba) reads lba-3..lba-1, not lba itself 2026-06-24 01:51:21 -07:00
Matthew Jackson ee0c7cebe3 fix(vc1): assemble keyframe prefix in fixed seq-then-entry order
When a keyframe AU carried an unchanged seq_header (stripped) but a
redefined entry_point (appended), the old append-then-reassert path
produced [entry_point, seq_header] — entry_point before seq_header,
violating SMPTE 421M which requires seq+entry before every RAP.

Replace the single shared prefix Vec + reassert_active() with per-type
temporaries (redefined_seq / redefined_ep) collected during the scan,
then assembled in canonical seq-then-entry order at keyframe time.
Non-keyframes still emit only genuine redefinitions, also seq-before-ep.
Removes the now-unused reassert_active() helper. Adds a regression test
covering the seq-unchanged / entry-redefined trigger case.
2026-06-24 01:47:51 -07:00
Matthew Jackson f3c3614a17 fix: discard duplicate TS packets in non-PUSI continuation path 2026-06-24 01:45:39 -07:00
Matthew Jackson c4f0566fb1 fix: correct comment — patch leaves middle NonTrimmed, not NonTrimmed/Unreadable 2026-06-24 01:11:39 -07:00
Matthew Jackson 77f67dd3ed fix: correct doc comment in patch pipeline test to say NonTrimmed not Unreadable 2026-06-24 01:10:51 -07:00
Matthew Jackson 7f195be894 fix: remove stale TDD-red comment from BytesRead emission test 2026-06-24 00:50:21 -07:00
Matthew Jackson 48e95a7b2c fix: correct hevc.rs doc comments — non-seamless BD join is connection_condition 0x05/0x06 not 0x01
Comments at lines ~63 and ~102 misidentified 0x01 (first-item/seamless) as the
non-seamless trigger and labelled 0x05/0x06 as seamless — inverted vs the BD-ROM
spec and mpls.rs (which documents 1=seamless, 5/6=non-seamless). Corrected all
affected doc blocks; no logic change.
2026-06-24 00:49:22 -07:00
Matthew Jackson 6268f6e5d9 fix: use trim_mkb in resolve_vid_only to avoid zeroing unrecognised MKB 2026-06-24 00:48:02 -07:00
Matthew Jackson 4c50ca2122 fix: correct stale comment in patch.rs work-list ranges_with call
The comment at line 404 claimed "every non-Finished range" but the
immediately-following ranges_with call lists only NonTrimmed,
NonScraped, and Unreadable — deliberately omitting NonTried.
Update the comment to accurately reflect the actual status list and
explain that NonTried is excluded because it is handled by a preceding
sweep pass, not by patch.
2026-06-24 00:09:13 -07:00
Matthew Jackson a324e5c62f fix: correct module doc — only Pass 1 routes through handle_read_error, not Pass N 2026-06-23 23:34:02 -07:00
Matthew Jackson dc7ab01907 fix: update stale doc comment in ReadCtx::for_patch — sync is automatic 2026-06-23 22:51:47 -07:00
Matthew Jackson 55849edd99 fix: correct skip_sectors_for_probe doc comment (8x per index, not 2x per 3) 2026-06-23 22:50:40 -07:00
Matthew Jackson af3666ff3b fix: correct module doc watchdog constant name in patch.rs 2026-06-23 22:49:42 -07:00
Matthew Jackson b82075b41a Fix rc5 audit findings: keydb doc, pipeline ordering, hot-loop Arc, tests
- keydb.rs: separate default_path()/no_home_dir() doc blocks; correct the
  false XDG lock-step claim (Linux write path uses $HOME, ignores
  XDG_CONFIG_HOME; read-side search also checks XDG_CONFIG_HOME).
- io/pipeline.rs: use Release/Acquire on the abandoned flag so a leaked
  consumer reliably skips close() on weak memory models (ARM64/POWER),
  not just x86 TSO.
- mux/disc.rs: cache the decrypt-loss Arc at construction; lost_bytes()
  no longer clones an Arc per frame on the mux hot path.
- disc/dvd.rs: assert display_aspect mapping for both 16:9 (PAL test) and
  4:3 (NTSC test).
- mux/resolve.rs: extract css_error_aborts() helper and unit-test the
  scrambled-but-uncracked CSS guard (Fix 6) incl. the --raw exemption.
- aacs/keys.rs: add unit tests for mkb_type_raw/mkb_type/mkb_is_uhd and
  MkbType (Category C 2.0 UHD, prerecorded 1.0, no-0x10-record None).
- release.yml: publish job needs [verify, test] so a failing test suite
  blocks crates.io publication.
2026-06-23 19:11:09 -07:00
Matthew Jackson 3c3e0b4341 Bump to 1.0.0-rc.4.3 2026-06-23 15:49:25 -07:00
Matthew Jackson e96528ad5b DVD: correct PAL/NTSC, anamorphic aspect, and SD colour
Fix three DVD video-attribute bugs surfaced by a PAL disc detected as
NTSC:

- PAL/NTSC: parse video_format from VTS_V_ATR bits 5-4, not bits 1-0
  (the old mask read permitted_df, so PAL 576i/25fps was mis-detected
  as NTSC 480i/29.97). Named consts replace the magic bit positions.
- Anamorphic aspect: write MKV DisplayWidth/Height from the disc's
  display_aspect (16:9 720x576 -> 1024x576) instead of square pixels,
  so 16:9 DVDs no longer render as 4:3.
- Colour: stamp SD colorimetry (PAL=BT.470BG, NTSC=SMPTE-170M) instead
  of BT.709 (HD).

Adds VideoStream.display_aspect (threaded through every muxer) plus
TvSystem/DvdAspect/ColorSpace plumbing, with regression tests. Removes
the deprecated Disc mux set_halt bridge (use with_halt).
2026-06-23 15:38:49 -07:00
Matthew Jackson 705857f117 docs: changelog entry for rc.4.2 Windows fixes 2026-06-23 12:49:08 -07:00
Matthew Jackson 845e20e508 v1.0.0-rc.4.2: bump version 2026-06-23 12:41:07 -07:00
Matthew Jackson d5afeb6088 io: add platform-aware fsync helpers (dir + durable file sync)
Add an io::fsync module with a per-OS split (posix/windows) mirroring the
writeback_file convention, replacing two duplicated dir-fsync copies:

- dir(): POSIX directory fsync; a no-op on Windows, where std cannot open
  a directory as a File and the failed open logged a spurious warning on
  every mapfile write.
- file_durable(): opens the target read+write before sync_all so the flush
  succeeds on Windows, where FlushFileBuffers rejects a read-only handle
  with ERROR_ACCESS_DENIED.

Point the mapfile writer at the shared dir() helper.
2026-06-23 12:38:02 -07:00
Matthew Jackson 008c1f143e v1.0.0-rc.4.1: bump version 2026-06-23 10:45:42 -07:00
Matthew Jackson e633a7d3af test(scsi/windows): cross-check all FFI structs + constants vs SDK headers
Audited every #[repr(C)] struct and IOCTL/flag constant in scsi/windows.rs
against the authoritative Windows SDK headers (ntddscsi.h, winioctl.h,
devioctl.h, winnt.h, fileapi.h). All correct except the already-reverted
ScsiPassThroughDirect packing. Add the missing regression guards:
- StoragePropertyQuery layout (STORAGE_PROPERTY_QUERY: 0/4/8, size 12).
- IOCTL/flag constants, with IOCTLs asserted against an independent CTL_CODE
  re-derivation (not a tautological literal) so a mistyped code is caught.
Validated compiling via cargo xwin check --target x86_64-pc-windows-msvc.
2026-06-23 10:44:21 -07:00
Matthew Jackson f177d61bbf fix(scsi/windows): revert wrong packed(4) on ScsiPassThroughDirect (rc.4 drive-detection regression)
rc.4 added #[repr(C, packed(4))] to ScsiPassThroughDirect on the false premise
that ntddscsi.h wraps SCSI_PASS_THROUGH_DIRECT in #pragma pack(push, 4). It does
NOT — verified against the Windows SDK ntddscsi.h: the struct has no pragma pack
and uses natural alignment. On 64-bit Windows (LLP64) that puts DataBuffer at
offset 24 and the struct at 56 bytes, which bare #[repr(C)] produces and which
DeviceIoControl expects.

packed(4) instead imposed offset 20 / 48 bytes — the layout of the SDK's
SEPARATE 32-bit thunk struct SCSI_PASS_THROUGH_DIRECT32 (VOID* POINTER_32). Using
that on a 64-bit host malformed every IOCTL_SCSI_PASS_THROUGH_DIRECT, so the
INQUIRY in drive enumeration failed and autorip/CLI reported zero drives
('RC4 no longer detects my drive'). rc.3.1 (bare repr(C)) worked for the same
users; this restores that layout.

Replace the tautological packed-layout test (which asserted the same wrong
offsets the struct produced) with one cross-checked against the SDK header:
DataBuffer@24, SenseInfoOffset@32, Cdb@36, size 56. Verified compiling via
cargo xwin check for x86_64-pc-windows-msvc.
2026-06-23 10:40:32 -07:00
Matthew Jackson 618524ecb8 ci: cache Rust builds with Swatinem/rust-cache to speed up release + CI 2026-06-23 10:01:37 -07:00
Matthew Jackson 4277ee32dd v1.0.0-rc.4: bump version 2026-06-23 09:31:56 -07:00
Matthew Jackson 6ace16293b keysource: add KeySource::label() for source identification 2026-06-23 09:09:06 -07:00
Matthew Jackson 8efe2fcbfd aacs: add MkbType (BD vs UHD generation) accessor API
Expose the MKB Type field (record 0x10) as a typed MkbType enum with mkb_type()
/ mkb_type_raw() / mkb_is_uhd() helpers, so callers can distinguish AACS 1.0
(Blu-ray) from AACS 2.0/2.1 (UHD) discs without poking raw bytes.
2026-06-23 08:19:40 -07:00
Matthew Jackson 31d07fde6e CHANGELOG: add 1.0.0-rc.4 — decrypt-loss accounting, durable writes, Windows SCSI, truthful error causes 2026-06-23 07:22:05 -07:00
Matthew Jackson ecee9f4ec0 Account for decrypt-time loss so partial AACS/CSS failures can't pass as a perfect rip
When a scrambled AACS unit fails to decrypt under every available key
(a missing/wrong CPS sub-key, or a marginal unit that fails the TS-sync
verify), decrypt_sectors restored the original encrypted bytes and
returned Ok with no signal. Those still-encrypted bytes flowed to the TS
assembler, which silently dropped the non-syncing packets with no loss
counter. The only loss accounting was DiscStream's read-error zero-fill
path, so mux reported lost_video_secs=0 for decrypt-dropped content and
the abort gate accepted the rip even under abort_on_lost_secs=0. A rip
missing real video/audio segments was published as a perfect success.

decrypt_sectors now returns the number of bytes in scrambled units that
no key could decrypt. DecryptingSectorSource accumulates that into a
shared counter exposed via decrypt_loss(); both mux pipelines fold it
into lost_bytes() — the inline DiscStream path directly, and the
file-backed highway via PipelinedPesStream sharing the producer's
counter. Restore-to-original is unchanged, so clear nav-files are never
corrupted; metadata-probe callers that don't read the counter are
unaffected. Adds regression tests at the decrypt and decorator layers.
2026-06-23 06:15:24 -07:00
Matthew Jackson 9220f03f3b keydb: correct read_capped_to_string doc for non-UTF-8 case
The doc claimed Error::KeydbInvalid for non-UTF-8 input, but the code
returns Error::KeydbParse (KeydbInvalid is reserved for the size-cap
violation). Correct the doc to match behavior and add a regression test
asserting non-UTF-8 yields KeydbParse.
2026-06-23 05:27:01 -07:00
Matthew Jackson e52689579b keydb: classify server-dropped connection as KeydbConnect, not KeydbParse
In http_get, when the server closes the TCP connection before the HTTP
header block completes (n == 0 on the byte-by-byte header read), or sends
a header block exceeding 64 KiB, the code returned KeydbParse (E8004).
Both are connection/protocol-level faults from the server, not parse
failures of keydb content — the keydb bytes were never received. Return
KeydbConnect (E8000) instead, which already covers TCP-level exchange
failures. A CLI user hitting a transient drop or a redirect target that
immediately closes now sees the correct 'server hung up' diagnostic
rather than 'the downloaded file was malformed'.

Add a regression test that stands up a loopback listener which accepts
then drops the connection before headers, asserting KeydbConnect.
2026-06-23 05:21:46 -07:00
Matthew Jackson 97ae47b3ea fix(scsi/windows): pack ScsiPassThroughDirect to match ntddscsi.h layout
ntddscsi.h wraps SCSI_PASS_THROUGH_DIRECT in #pragma pack(push, 4),
forcing the PVOID DataBuffer field to 4-byte alignment even on 64-bit
hosts. The Rust struct used bare #[repr(C)], so the compiler applied
natural 8-byte pointer alignment and inserted 4 padding bytes after
TimeOutValue. That shifted DataBuffer to offset 24 (SDK: 20),
SenseInfoOffset to 32 (28), and Cdb to 36 (32), and grew the struct to
56 bytes (48). DeviceIoControl reads at the SDK offsets, so every SPTI
ioctl on 64-bit Windows either got rejected or interpreted garbage as
the CDB and DataBuffer pointer.

Add #[repr(C, packed(4))] to ScsiPassThroughDirect and the companion
SptwbDirect (so offset_of!(SptwbDirect, sense) stays correct for
SenseInfoOffset), plus a layout regression test asserting DataBuffer at
offset 20 and a 48-byte struct size.
2026-06-23 05:16:22 -07:00
Matthew Jackson f596dcb40e prefetched: test that event_fn fires BytesRead per batch
The prefetch producer thread fires a BytesRead event after every
batch it reads, carrying a cumulative byte count. Nothing asserted
this callback actually fired, so a consumer that passed None for the
event_fn would silently get no progress events. Add a fixture-based
regression test that drives a finite extent through new_with_events,
captures the events, and asserts the cumulative count is
non-decreasing and reaches the full extent size at EOF.

This locks the contract autorip's mux progress bar and soft-stall
watchdog depend on.
2026-06-23 04:54:40 -07:00
Matthew Jackson 0e18bfa035 aacs: preserve transport-failure errors through the auth handshake
A SEND KEY / REPORT KEY step in the bus-auth handshake mapped every
SCSI error to a cert/key-specific code (AacsCertRejected, AacsCertRead,
AacsKeyRead, AacsKeyRejected, etc.) via map_err(|_| ...). That discarded
the underlying SCSI error, so a transport-layer wedge (bridge crash / USB
disconnect) mid-handshake was reported as 'drive rejected your host cert',
sending operators down a keydb/host-cert dead end for what is really a
replug/power-cycle situation.

Add a handshake_err() helper that keeps the original error when it is a
transport failure (is_scsi_transport_failure) and only substitutes the
handshake-specific code for genuine SCSI rejections. Apply it at every
SEND KEY / REPORT KEY / REPORT DISC STRUCTURE step in both the AACS 1.0
and AACS 2.0 paths. Add a regression test covering both branches.
2026-06-23 04:20:54 -07:00
Matthew Jackson 89fa0a791e disc: warn when READ CAPACITY fails instead of silently using 0 sectors
read_udf treated a READ CAPACITY SCSI failure as a 0-sector disc via
unwrap_or(0) with no diagnostic. capacity=0 then skews the layer
heuristic (always reports 1 layer, even for dual-layer discs) and the
canonical title-ordering sort, with nothing in /api/state or info to
indicate the command actually failed. Emit a tracing::warn carrying the
original error at the fallback site so a transient capacity failure is
visible. Recovery behavior is unchanged: 0 is still used as the
fallback.
2026-06-23 04:11:46 -07:00
Matthew Jackson f68a66c4be keydb: map missing home dir to NotFound, not keydb-parse error
default_path returned Error::KeydbParse (E8004, rendered as 'failed to
parse the keydb file') when HOME/USERPROFILE was unset. That misreports
an environment failure — a process with no home directory, typically a
stripped container or CI config — as a corrupt keydb file the code never
read. Map it to an IoError(NotFound) in the I/O category instead, so no
display path blames the keydb. Add a regression test.
2026-06-23 04:05:41 -07:00
Matthew Jackson 662594ff40 Preserve I/O error from read_aacs_inputs ISO open
Disc::read_aacs_inputs opened the ISO via FileSectorSource::open and
mapped any failure to Error::AacsNoKeys (E7000), discarding the real
Error::IoError (E5000) and its OS errno. A missing or unreadable ISO
(ENOENT/EPERM) is an I/O fault, not a key-resolution failure; callers
that dispatch on the error code would wrongly tell the user to check
their keys when the ISO simply does not exist.

Propagate the open error unchanged and add a regression test asserting
a nonexistent ISO yields E_IO_ERROR, not E_AACS_NO_KEYS.
2026-06-23 03:59:55 -07:00
Matthew Jackson c9bf92cd6f Fix oversized read batch on non-sysfs (Windows) optical drives
detect_max_batch_sectors() is a Linux-sysfs probe with no platform
gate. It derived the device name with rsplit('/'), which never splits a
Windows \.\CdRom0 / \.\D: path, so the whole path became the device
name, no /sys node matched, is_optical fell to false, and the function
returned the 8192-sector block default (16 MiB/request) instead of the
60-sector optical default. That value then took the Some(b) arm in
Disc::copy and bypassed the 510-sector optical clamp that lives only in
the sysfs branch, leaving every Windows rip/verify running ~16x over the
optical cap (coarser bad-sector recovery, 16 MiB UDF reads).

Gate the sysfs probe behind a new sysfs_batch_probe_supported() helper
(Linux-only, requires a '/'-delimited path) and return the optical
default for any path the probe can't handle. Add regression tests for
the \.\ device-path forms.
2026-06-23 03:51:51 -07:00
Matthew Jackson 4a76deadeb scsi(windows): fix StorageAdapterDescriptor.BusType width (u8 -> u32)
STORAGE_ADAPTER_DESCRIPTOR.BusType is STORAGE_BUS_TYPE, an int-sized
(4-byte) enum, but the repr(C) struct declared it as u8. The total
size stayed 32 bytes by coincidence of alignment padding, and the two
fields actually read today (MaximumTransferLength at offset 8 and
AlignmentMask at offset 16) sit ahead of BusType, so there was no
runtime impact. But BusMajorVersion and BusMinorVersion landed at
offsets 26 and 28 instead of the SDK's 28 and 30, so any future reader
of those fields would have gotten wrong values.

Widen BusType to u32 so BusMajorVersion/BusMinorVersion fall at the
correct offsets, and add a layout regression test asserting every
field offset and the 32-byte total against the winioctl.h layout.
2026-06-23 03:45:34 -07:00
Matthew Jackson 8e6d494e54 aacs: fix stale mkb_version offset doc comment
The doc comment claimed the version was a BE u32 at offset 8 of the
record body (offset 12 from pos), but the code correctly reads pos+8
(body offset 4): a 4-byte record header at pos, the Type field at body
offset 0, then the version at body offset 4. Rewrite the comment to
match the actual read so a maintainer does not 'correct' the offset and
break MKB version parsing. Clarify the matching test comment too.
2026-06-23 02:28:34 -07:00
Matthew Jackson 24ed1d1d19 disc: reject partial CPS-unit key coverage in AACS validation gate
aligned_unit_keys_validate accepted a unit-key set as soon as ONE
scrambled sample decrypted. On a multi-CPS-unit disc a set covering
CPS unit 0 but not CPS unit 1 therefore passed: decrypt_with committed
it, the sweep proceeded, and CPS-unit-1 sectors passed through as raw
encrypted bytes into the ISO/MKV with no error surfaced anywhere.

Require every scrambled sample to be descrambled by some unit key.
A sample no key covers now fails the gate, so an incomplete set is
rejected (AacsKeyRejected) and the caller falls through to the next
candidate, ultimately surfacing a key error instead of silently
writing ciphertext. Wholly-wrong-key rejection is unchanged.

Add a regression test for the partial-coverage case.
2026-06-23 01:43:36 -07:00
Matthew Jackson f2c2ff0eb3 disc: fix misleading bridge-degradation comment, add 04/3E regression test
The comment on the bridge-degradation branch in handle_read_error
claimed it matched the NOT_READY 04/3E sense signature, but
is_bridge_degradation() keys solely on a non-standard SCSI status byte
(anything that is not GOOD/CHECK CONDITION/TRANSPORT FAILURE) and
ignores sense_key/ASC/ASCQ. A real 04/3E bad-sector error arrives as
CHECK CONDITION (0x02), so it never took this branch — it falls through
to the generic NOT_READY retry. Rewrite the comment to describe the
status-byte condition the predicate actually detects, and drop the
parallel misleading note in the not_ready_err test helper.

Add a regression test asserting a NOT_READY 04/3E error is not
classified as bridge degradation and routes to the NOT_READY retry
(3 s pause) rather than the bridge cooldown (15 s pause).
2026-06-23 01:20:04 -07:00
Matthew Jackson a3987e67f2 scsi(windows): only sleep on successful device reset
SptiTransport::reset() unconditionally slept 2 seconds after sending
IOCTL_STORAGE_RESET_DEVICE, even when the IOCTL failed (e.g.
ERROR_INVALID_FUNCTION on a driver that does not support the reset).
On failure no reset occurred, so there is nothing to settle and the
2-second penalty was pure waste. Gate the settle sleep on the IOCTL
return so it only fires when the drive was actually reset.
2026-06-23 00:59:26 -07:00
Matthew Jackson 1b008008dd keydb: write keydb.cfg atomically (temp + fsync + rename)
keydb::save() overwrote the live keydb.cfg with a bare in-place
std::fs::write. keydb.cfg is the single source of AACS truth and this
path runs unattended (first-boot download + daily-refresh thread, with
a container restart on every release), so a SIGKILL, OOM-kill, power
loss, or ENOSPC mid-write could leave the file truncated with the prior
good copy already gone. A truncated keydb does not error at write time;
it surfaces later as failed key resolution on every AACS rip.

Factor the write into write_atomic(): create the parent dir, write a
unique sibling temp file, fsync, then rename (atomic within a
filesystem). On any write/fsync/rename failure the temp is removed and
the existing keydb is left untouched. Same pattern already used by the
settings and mover write paths. Add regression tests covering in-place
replacement (no stray temp) and prior-copy preservation on failure.
2026-06-23 00:24:58 -07:00
Matthew Jackson 980eeb3de9 mux: track skipped bytes for accurate loss estimation
DiscStream skips a whole AACS unit (3 sectors = 6144 bytes) per
read-error event, but only the skip-event count was exposed. Loss
estimates built from errors*2048 therefore undercounted AACS loss ~3x.

Add a lost_bytes field that accumulates the actual zero-filled byte
count at each skip, expose it via a new Stream::lost_bytes() accessor
(default 0; DiscStream and CountingStream override), so consumers can
scale lost-video time by real bytes lost rather than the event count.

Regression tests assert the AACS path records 6144 B/event (and
exceeds the errors*2048 undercount) while the align=1 path records
2048 B/event.
2026-06-23 00:12:56 -07:00
Matthew Jackson c3c5259f84 pipeline: leaked consumer must not finalise an abandoned output
When finish_with_halt's grace period expires it detaches from the
consumer thread and returns an error to the caller, but the leaked
consumer kept running to completion: once its wedged write syscall
returned it would fall through to sink.close(). For the mux writer
close() finalises the MKV (Cues block + segment-header patch), so a
leaked consumer could finalise — and keep writing to — an output file
the caller had already reported as failed, racing a fresh rip for the
same device over the same path.

Add a shared abandonment flag the consumer polls in its drain loop and
again before close(). finish_with_grace sets it before dropping the
JoinHandle, so the moment the wedged syscall returns the consumer skips
any further apply and skips close() entirely, then exits. This does not
interrupt the in-flight syscall (only its return or process exit can),
but it bounds the damage to the write already in flight instead of a
full finalise of an abandoned file.

Regression tests cover both sides: a consumer leaked past the grace
period skips close(), while one that finishes inside the grace window
still calls close() and finalises normally.
2026-06-22 23:22:57 -07:00
Matthew Jackson ae411df8f9 scsi/windows: surface IOCTL_STORAGE_RESET_DEVICE failures
SptiTransport::reset() discarded the DeviceIoControl return value, so a
wrong or unsupported reset IOCTL would fail with ERROR_INVALID_FUNCTION
and silently no-op while the unconditional 2s settle sleep made it look
like a reset happened. That is exactly the regression class the doc block
records for the two earlier (incorrect) code values.

Bind the result and warn (with GetLastError) when the reset fails, debug
on success. Lift IOCTL_STORAGE_RESET_DEVICE to module scope and add a
test recomputing it from the CTL_CODE formula so a wrong value can't slip
back in unnoticed.
2026-06-22 23:08:52 -07:00
Matthew Jackson 60daf63c09 mapfile: fsync parent directory after rename for durable resume checkpoint
Mapfile::flush() wrote the new state to a .tmp sibling, sync_all()'d the
temp file, then rename(2)'d it over the final mapfile path — but never
fsynced the parent directory. After the rename the new dirent lives only
in the directory's page cache, so a crash or power loss in the
rename-commit window (the wide window on NFS, the very case the temp
fsync guards) can lose it: resume then reads a stale or absent mapfile
even though the data bytes were durable, silently discarding multi-pass
recovery progress.

Add a best-effort fsync_dir() on the path's parent after the rename,
mirroring the established dirent-durability pattern in autorip's mover.
A directory that can't be opened or synced is logged and ignored rather
than failing the write, since the file bytes are already durable.

Adds a regression test exercising the parent-fsync branch against a real
subdirectory and asserting the helper is a no-op on a missing directory.
2026-06-22 23:00:04 -07:00
Matthew Jackson d3c58791ff release CI: parallelize test/publish/release off the critical path
Tests, crates.io publish, and the GitHub Release no longer serialize
behind each other: each job only needs the version-check (verify) gate.
Binary consumers git-tag-pin libfreemkv, so they start building the
instant the tag exists rather than waiting on the crates.io publish.
crates.io publish runs as an independent job for external consumers and
uses --no-verify (CI already compiled this commit in the test job).
2026-06-22 21:59:31 -07:00
Matthew Jackson b85744d120 Demote per-read Drive::read trace event to TRACE
Drive::read fires hundreds of thousands of times per rip. Logging its
entry at DEBUG floods a diagnostic log and buries the events that
actually matter. Move it to TRACE so a level-3 (debug) bug-report log
stays readable; level-4 (trace) still captures it for deep dives.
2026-06-22 21:39:27 -07:00
Matthew Jackson 633a22c6bd Merge branch 'feat/windows-hardening' 2026-06-22 18:08:41 -07:00
Matthew Jackson ab959dd770 v1.0.0-rc.3.1: silent-failure guards (mux empty/zero-frame, CSS crack-vs-unencrypted), Windows keydb path, AlignmentMask, English errors 2026-06-22 18:07:48 -07:00
Matthew Jackson 9f422e6ebb docs: describe the generic Unlocker seam, drop in-tree firmware specifics
The docs still documented the old in-tree firmware unlocker: the MediaTek
MT1959 variant table, the READ BUFFER unlock CDB bytes, the profiles.json
schema (unlock_mode/unlock_buf_id/unlock_cdb), the platform/mt1959 driver
listings, and the 'why unlock is needed' handshake mechanism. None of that
lives in libfreemkv anymore — the core is firmware-clean and ships only the
pluggable Unlocker trait + registry (src/unlock.rs).

Rewrite drive-access, architecture, api-design, disc-to-rip, and the README
to describe only the generic Unlocker seam: the trait, register_unlocker, the
registry routing, and the host-cert fallback when no unlocker matches. Point
readers to the freemkv-unlock repo for concrete unlockers. No source change.
2026-06-22 17:10:51 -07:00
Matthew Jackson 63ca840b7e v1.0.0-rc.3: bump version 2026-06-22 16:13:07 -07:00
Matthew Jackson e9108e8b6b scsi/windows: correct IOCTL_STORAGE_RESET_DEVICE to 0x002D5004
The dual-model Windows audit (Sonnet) caught that the prior 'fix' (0x002DD000)
was also wrong: that decodes to the OBSOLETE RESET_BUS code (function 0x400,
R|W access) which class drivers reject. Canonical ntddstor.h:
  IOCTL_STORAGE_RESET_DEVICE = CTL_CODE(0x2D, 0x0401, METHOD_BUFFERED, FILE_READ_ACCESS)
  = (0x2D<<16)|(1<<14)|(0x401<<2)|0 = 0x002D5004.

Only affects the best-effort drive-reset recovery path (return value is
discarded); the normal read/unlock/rip path never calls it, so this does not
change normal Windows operation. Verified correct-by-construction; the
AlignmentMask finding is deferred to rc4 (benign on USB optical bridges,
needs a real Windows SCSI-HBA rig to validate).
2026-06-22 15:53:45 -07:00
Matthew Jackson fa8913800c Fix 3 Opus-audit findings: patch transport-abort, patch AACS align, reset IOCTL
- patch (Pass N) now aborts immediately on transport failure (status=0xFF),
  symmetric with the sweep and single-pass mux. Previously a USB-bridge crash
  was treated as an ordinary bad sector and the pass hammered the crashed
  device sector-by-sector until the per-range watchdog expired. (medium)
- patch AACS recovery reads are now unit-aligned: a mid-unit single-sector
  read on an AACS disc was rejected by the decrypting reader (DecryptFailed)
  and the sector abandoned without asking the drive. The read is now widened
  to the enclosing whole 3-sector unit and the requested window copied out,
  leaving all recovery accounting (pos/block_bytes/cursor) untouched so it
  cannot desync. Only affected CLI decrypt-to-ISO --multipass re-runs. (low)
- IOCTL_STORAGE_RESET_DEVICE corrected 0x002D1004 -> 0x002DD000 (the old value
  decoded to function 0x401 with the access bits cleared, so DeviceIoControl
  would fail ERROR_INVALID_FUNCTION instead of resetting). Windows-only. (low)

Adds a transport-failure classification regression test.
2026-06-22 15:50:22 -07:00
Matthew Jackson f863e9a4be mux/disc: abort single-pass rip on transport failure (USB-bridge crash)
A direct disc://→mkv:// single-pass rip drives fill_extents in
skip_errors mode. On a read failure it shrank the batch, retried, and
once bottomed out zero-filled + skipped the unit and continued. A SCSI
transport failure (status=0xFF) is a USB-bridge crash, NOT a skippable
bad sector: the bridge is wedged and every subsequent read fails the
same way. So the loop marched the entire disc at one ~15s bridge-
recovery per probe, producing no MKV — the user-reported 'hundreds of
0x28/0xff warnings, runs forever, Movie.mkv never created'.

Fix: short-circuit to an error on transport failure before any
shrink/skip, even under skip_errors — mirroring the multipass sweep's
transport-failure rule in read_error::handle_read_error. The CLI
surfaces it so the user power-cycles the drive or switches to multipass
recovery. Regression test asserts exactly one read is issued and no skip
is counted (no infinite march).
2026-06-22 15:18:52 -07:00
Matthew Jackson 4d81affb45 Merge branch 'rc3-topgun' into rc3-integration 2026-06-22 15:08:06 -07:00
Matthew Jackson c73a3dbcb6 Fix Windows multi-drive selection, disk:// alias, and READ chunking
Three fixes for a Windows ASUS Blu-ray drive that failed/spammed errors:

- resolve.rs: accept disk:// as an alias for disc:// (identical behavior;
  empty = auto-detect, path = device). Windows users commonly type
  disk://i: after the drive-letter convention.

- drive::find_drive: prefer a drive that reports media present. Enumerate
  all optical drives, query Drive::drive_status() (GET EVENT STATUS, works
  regardless of firmware), and return the first reporting DiscPresent;
  fall back to the first enumerated drive when none report a disc so
  single-drive / quirky setups don't regress. Selection policy split into
  select_drive_with_media() for unit testing.

- READ chunking: add ScsiTransport::max_transfer_bytes() (default 1 MiB).
  Windows SPTI overrides it with the adapter MaximumTransferLength queried
  via IOCTL_STORAGE_QUERY_PROPERTY / StorageAdapterProperty, clamped to a
  64 KiB floor (fallback on query failure). Drive::read now caps each
  READ(10) to that limit: small reads take the unchanged single-CDB path,
  larger reads loop over read_one() chunks, reporting the failing chunk's
  LBA on error. This stops the 16 MiB single read that exceeded the
  adapter limit, made DeviceIoControl fail, and spammed transport-failure
  warnings with slow tiny-read fallbacks.

Tests added for the disk:// alias, media-preference selection, and READ
chunk decomposition / per-chunk error LBA.
2026-06-22 15:02:24 -07:00
Matthew Jackson 4f606ae9a3 aacs: source OEM host certs from keysource layer
Complete the OEM/AACS cert baseline so host certs are a KeySource output,
never compiled in. With an unlocker present the OEM route is unused
(unlocker_read_volume_id short-circuits); without one, the cert handshake
runs when a keysource supplies a host cert and fails gracefully when none
does.

- KeySource trait gains host_certs() (default empty), reusing the existing
  aacs::HostCert type. A source holds certs as its second kind of AACS
  material alongside decryption keys.
- ScanOptions gains key_sources so the handshake can collect certs across
  the app's keysource layer, unioned with DriveCredentials.
- do_handshake_cert collects certs via collect_host_certs (credentials +
  every key source). Zero certs from any source now returns the new
  graceful Error::AacsNoHostCert (code 7024, sentinel <no host cert>)
  instead of silently skipping; resolution still falls back to the
  path-1 disc-hash -> VUK lookup, which drops the error on a hit.
- error.rs: add E_AACS_NO_HOST_CERT / Error::AacsNoHostCert, wired into
  code(), Display, and the round-trip + sentinel tests.

HandshakeResult { volume_id, read_data_key } unchanged: the cert path
still yields both the VID and the bus key.
2026-06-22 11:23:38 -07:00
Matthew Jackson 25acd09504 unlock: finalize Unlocker 3-capability contract
Rename the trait to a generic, drive-neutral capability contract so future
unlockers don't conform to LibreDrive specifics:

  - unlock(...)   -> unlock_drive(...)        (the one required capability)
  - read_vid(...) -> read_volume_id(...)      (no-op default)
  - add set_max_read_speed(...)               (no-op default)

The trait doc now states the contract in one place: unlockers are optional
drive-capability providers; the AACS layer is the always-present baseline and
falls back to the full cert handshake when no unlocker matches. Implement only
the capabilities your drive supports.

Registry: route_unlock now calls unlock_drive; unlocker_read_vid renamed to
unlocker_read_volume_id; add unlocker_set_max_read_speed (mirrors route_unlock
resolution, first matching unlocker, no-op if none match). drive::init calls
it on a matched drive in the post-unlock path; a speed-set failure is logged
and does not fail the rip. encrypt.rs handshake updated to the new VID helper.

Tests updated for the renames; added a set_max_read_speed routing test
(match invokes, no-match is a safe no-op).
2026-06-22 11:05:21 -07:00
Matthew Jackson 159e967760 unlock: add OEM read_vid capability to Unlocker seam
An Unlocker unlocks drive functionality, not just the disc: unlock() is
one capability, OEM VID retrieval is another. Widen the Unlocker trait
with a default-no-op read_vid(), add an unlocker_read_vid registry helper
that mirrors route_unlock resolution, and consult it in do_handshake_cert
before the cert-based VID read. A matching unlocker that serves a VID via
its OEM path short-circuits the cert handshake — VID is obtained without
the host certificate + HRL (restoring the pre-refactor decoupled OEM VID
path, now living inside the unlocker). Non-matching drives, and unlockers
without an OEM VID path, fall through to cert auth unchanged.

is_unlocked() now reports the honest signal (a registered unlocker matched
this drive) instead of const false.
2026-06-22 10:50:47 -07:00
Matthew Jackson 6dc62bcd84 Extract drive unlock behind pluggable Unlocker seam
libfreemkv must stay firmware-clean for crates.io. Move ALL drive-unlock
knowledge — firmware blobs, WRITE_BUFFER/MODE SELECT upload, unlock CDBs,
the MT1959 variant-A/B handshake, the 800 KB profiles.json database, and
the DriveProfile parsing — out into the freemkv-unlock-ld crate.

libfreemkv now keeps only the seam:
  - Unlocker trait (name/matches/unlock) + a process-wide ordered registry
    (register_unlocker / route_unlock) in src/unlock.rs
  - Drive::init() walks the registry; the first unlocker whose matches(id)
    is true runs unlock(scsi, id); if none match the drive is left in
    stock mode and the host-cert AACS handshake (the OEM route) carries
    the disc.

The unlocker issues its own CDBs through the public ScsiTransport::execute,
so libfreemkv knows nothing about how unlocking happens.

Removed:
  - profiles.json
  - src/platform/mt1959/{mod,variant_a,variant_b}.rs
  - src/profile.rs (DriveProfile, ProfilesFile, find_by_drive_id, ...)
  - the PlatformDriver trait

Because the Unlocker seam reports only success/failure (no extended-access
marker), VID acquisition is now always via the cert-based handshake; the
per-drive OEM-VID-CDB shortcut and Drive::is_unlocked() (now const false)
are removed/neutralized. Disc-speed calibration moved into the unlocker's
unlock(); Drive::probe_disc() is a no-op.

git grep over src/ is firmware-blob/profiles/WRITE_BUFFER/mt1959-free.
All tests pass on Rust 1.86 (precommit green).
2026-06-22 10:31:51 -07:00
Matthew Jackson 9250f5bb30 wip: top gun EL/decrypt follow-up (in progress, rc3) 2026-06-22 10:02:53 -07:00
Matthew Jackson e960c2f1be mux/mkv: video-only timeline epochs + finer 0.1ms TimestampScale
Fixes corrupt MKV seek index on single-clip titles with many
interleaved tracks (Top Gun UHD: 2 video, 11 audio, 32 PGS).

TimelineContinuity previously shared one high_ns frontier + offset_ns
across ALL tracks. A sparse, lagging non-video frame (subtitle/audio)
ratcheted the frontier up; the next normal video frame then sat >3s
below it and was misread as a clip-boundary discontinuity, permanently
bumping offset_ns. On a one-clip title this fired thousands of times
and inflated Cue/cluster timestamps into the billions of ms, destroying
the seek index (ffmpeg then seeked to wrong positions and emitted
spurious 'Could not find ref with POC N' errors).

Now only the VIDEO track drives epoch decisions: video alone advances
the frontier and opens a new epoch on a real backward PTS jump.
Non-video tracks are remapped under the current offset and never touch
the frontier or offset. A lagging non-video tail straggler at a genuine
multi-clip boundary (old-epoch raw PTS under the new offset) is
recognised via the previous offset and remapped to the seam, so it
neither flies forward nor forces a back/forward-dated split cluster.
Genuine multi-clip seamless rebasing is preserved.

Also drop TimestampScale from 1ms to 0.1ms (100_000 ns/tick) so
23.976fps frames and 0.833ms TrueHD AUs stop colliding on a single
tick (the source of the non-monotonic-DTS warnings and the audio
cadence flattening). The finer scale shrinks the i16 block-relative
span to ~3.27s, so: cluster duration is set to 2s nominal (keeps
keyframe-driven clusters within the i16 range for typical GOPs), and
the i16-overflow cluster-split path now emits a Cue for the split
cluster so the seek index has no gaps.

Regression tests: single-clip late-subtitle must not inflate offset_ns;
non-video must not advance the frontier; non-video straggler remapped
to seam at a real boundary; every cluster (incl. i16-split) carries a
Cue. Existing tick/duration assertions updated for the new scale.
2026-06-22 09:45:45 -07:00
Matthew Jackson f74979bdb4 docs: scrub AACS internals + key-coverage claims from changelog; condense pre-1.0 history 2026-06-22 09:39:40 -07:00
Matthew Jackson dc2cac1b5f v1.0.0-rc.2: bump version 2026-06-22 09:06:54 -07:00
Matthew Jackson 337e77951c rc2: macOS cross-compile fix + security/recovery hardening
- build.rs: pass target -arch to cc so macos_shim cross-compiles (x86_64-apple-darwin)
- AACS/CSS: unit-aligned decrypting sweep; per-VTS CSS title keys (hard-fail on wrong VTS);
  reject truncated Unit_Key_RO; AACS 2.0 sig-verify skip; CSS bus-auth random nonce
- recovery: gap-filling mapfile load; sweep/copy resume reconciliation; stale-mapfile abort;
  patch wedge/damage-window range reset
- mux: TS continuity + PSI CC desync guards; HEVC numTemporalLayers clamp; MPEG-2 pending
  byte-cap; PS parse_pts marker-bit validation; HdrFormat strict parse; Unknown-variant metadata
- net/keydb: network:// SSRF parity (IPv4-mapped, CGNAT, 0.0.0.0/8, Class-E); bounded keydb
  header read + size cap + error context
- io: durable mapfile fsync; NFS writeback degrade; sync_file_range error capture;
  Windows SCSI u32 transfer guard
2026-06-22 08:58:10 -07:00
Matthew Jackson 5941c059c6 v1.0.0-rc.1
CSS keyless decrypt (Stevenson), AACS 1.0/2.0/2.1, MPEG-2 DVD, multi-OS SCSI, multipass recovery, mux highway, audit hardening
2026-06-21 21:06:07 -07:00
Matthew Jackson e8bb6225ac libfreemkv 0.31.10: parallelise recover_dk_position slot scan
The subset-difference slots are independent, so rayon find_map_any scans them
in parallel and cancels on first match. UHD MKB no-match scan ~26s -> ~4.6s on
8 cores. Bit-identical result; 58 aacs tests pass.
2026-06-17 19:56:07 -07:00
Matthew Jackson 9c80ef8245 libfreemkv 0.31.9: ~3x fewer AES ops in the subset-difference PK walk
calc_pk_from_dk derived all three children (left/pk/right) at every tree
level but used only the one it descended into; the Processing Key only
matters at the final node. Derive just the descended child per level + the
PK once at the end. Bit-for-bit identical; speeds every DK->MK derivation
(disc decryption + unpositioned-DK recovery). 60.8s -> 22.9s on a UHD
worst-case recovery scan.
2026-06-17 15:43:39 -07:00
Matthew Jackson dc87962e50 libfreemkv 0.31.8: recover unpositioned device key position from an MKB
Add recover_dk_position: boil a position-less device key down against a
disc MKB to its invariant subset-difference position (node/uv/u_mask_shift)
— zero-descent probe + ancestor walk-up, hoisted verify. Consolidate the
SD-walk surface (drop the research-only probe::walk_pk_against_tables;
make derive_media_key_from_pk_walked internal).
2026-06-17 15:29:51 -07:00
Matthew Jackson 4221cd6a86 libfreemkv 0.31.7: expose Processing Key from the DK walk 2026-06-17 13:18:34 -07:00
Matthew Jackson dda4e7482b aacs: unify the SD-walk; derive_media_key_from_dk now exposes the PK
Two byte-identical copies of the subset-difference walk lived in keys.rs
and variants.rs. Consolidate the pure helpers (aesg3, calc_v_mask,
calc_pk_from_dk) into keys.rs (pub(super)); variants.rs imports them.

Add derive_media_key_and_pk_from_dk(mkb, dks) -> Option<(mk, pk)>, which
returns the intermediate Processing Key the walk already computes;
derive_media_key_from_dk becomes a thin wrapper. This lets callers bank
the PK on a DK boil instead of re-deriving it via a second, divergent
walk (the classical-vs-variant cvalues order made that miss silently).

216 AACS tests pass (incl. a new (mk,pk) regression); precommit (1.86) green.
2026-06-17 13:11:03 -07:00
Matthew Jackson f80551f278 udf: read AACS MKB length-aware; honor inline allocation descriptors
read_file rejected the padded ~128 MiB MKB_RO.inf via the 0.31.0
MAX_FILE_BYTES cap, so read_aacs_inputs failed and the online
key-resolve path never contacted the keyserver. Read the MKB's real
record length from its header and read exactly that. Also honor
inline/embedded (AD type 3) files so small AACS .inf files read from
the ICB payload instead of being misparsed as allocation descriptors.

Release 0.31.6.
2026-06-08 19:20:41 -07:00
Matthew Jackson 41a6d89cd1 libfreemkv 0.31.5: do not force monotonic block timestamps on video
B-frame video PTS is legitimately non-monotonic in decode/storage order; the
audio-oriented monotonic nudge was clobbering it to prev+1ms, which decoders
flagged as non-monotonic DTS (thousands per title). Apply the nudge to
audio/subtitle only; video keeps its true PES PTS. + regression test.
2026-06-08 09:00:35 -07:00
Matthew Jackson f79c2a0aa9 libfreemkv 0.31.4: prune 144 vacuous tests (keep spec-grounded subset) 2026-06-08 07:28:55 -07:00
Matthew Jackson d181362460 libfreemkv 0.31.3: use RFC 5737 doc address in network URL test fixture 2026-06-07 22:42:09 -07:00
Matthew Jackson 8000bae177 libfreemkv 0.31.2: comprehensive spec-grounded test suite (~950 tests)
Test-hardening release, no runtime changes. Adds spec-grounded unit tests
across the silent-corruption surfaces — UDF/MPLS/CLPI/IFO parsing, BD/DVD
title + extent assembly, AACS/CSS key handling, TS/PS demux + codec parsers,
MKV/EBML container output, the mux pipeline, sector prefetch + decrypt
decorator, drive/SCSI sense decoding, label extraction, and core I/O. Each
test is grounded in the format spec or real on-disc behavior and verified to
fail under a targeted source mutation. No behavior changed.
2026-06-07 22:28:29 -07:00
Matthew Jackson 2a55bab3ed tests: read_aacs_inputs Long-AD coverage; scrub title names from comments
Add a precommit fixture proving Disc::read_aacs_inputs reads a Long-AD,
multi-extent /AACS/Unit_Key_RO.inf in full — the exact input the online
key-request path depends on (no disc/deploy needed). Make
read_aacs_inputs_from_reader pub(crate) for the test.

Reword internal comments/doc examples to generic descriptions instead of
specific media titles.
2026-06-07 21:25:29 -07:00
Matthew Jackson 222a596c55 libfreemkv 0.31.1: UDF Long-AD stride fix + MKB trim guard + regression tests 2026-06-07 20:49:55 -07:00
Matthew Jackson c1b4f3cbb3 tests: harden UDF allocation-descriptor + bad-sector recovery paths
Spec-grounded unit tests for the silent-corruption surfaces, each verified to
fail under a targeted source mutation (no vacuous tests).

udf (10): Extended-AD 20-byte stride + extent LBA at off+12, type-1 sparse
extents skipped not emitted, zero-length type-0 terminator, continuation-loop
bound (anti-hang), UTF-16BE and 8-bit name decoding, FID L_IU offset, parent
(..) FID skip, d-string length-byte cap. Locks the spec branches a future
allocation-descriptor refactor must not silently break.

recovery (9): Pass-N damage-skip range bounds (forward/reverse cursor stays in
range), one-quarter-of-remaining skip cap, below-threshold no-op, work-done
accounting, and bridge-degradation retry-to-budget fall-through.
2026-06-07 20:46:00 -07:00
Matthew Jackson 06c30aa466 udf: honor ICB allocation-descriptor type (Short/Long/Extended AD)
read_icb_extents hardcoded an 8-byte Short-AD stride for every file. Large
BD-ROM .m2ts streams use 16-byte Long ADs; striding them as Short ADs reads
descriptor #0 correctly (length+lba align) but lands #1 in the middle of the
first Long AD (its zero impl_use bytes). The AD-list terminator (data_len==0
=> break) then fired on that zero and stopped after the first extent, so every
multi-extent title truncated at ~1 GiB. The same reader backs read_file, so
disc AACS-input files (/AACS/*.inf) and the m2ts mux extents were both
affected.

Read the ICB Tag flags (AD type) and stride 8/16/20 bytes for Short/Long/
Extended ADs accordingly; Extended ADs carry the lba at off+12.

aacs: extract trim_mkb and restore its guard so an MKB whose content length
the parser cannot determine (mkb_content_len == 0) is returned intact instead
of truncated to empty.

Regression tests: Long-AD read_icb_extents returns all extents; Long-AD
read_file returns full content; trim_mkb never zeroes an unrecognised MKB.
2026-06-07 19:53:40 -07:00
Matthew Jackson 061f68594a 0.31.0: hardening and correctness pass across mux, codec, AACS/CSS, UDF/MPLS/CLPI, recovery, drive/SCSI, labels, and I/O
Library-wide review-and-fix pass: tightened AACS keydb/handshake/variant
handling and trailing-partial-unit policy, corrected MPLS mark offset and
added UDF allocation bounds, hardened the mux/codec framing and M2TS paths,
guarded SCSI READ CAPACITY short transfers and unified error mapping, added
overflow guards on untrusted disc input, and made prefetch shutdown
deterministic. Release profile now builds with thin LTO + single codegen unit.
2026-06-07 17:37:38 -07:00
MattJackson 5b6ea8f5c4 v0.30.7: mux correctness pass
- hevc: real chroma/bit-depth in hvcC (10-bit Main10), emit redefined
  param sets in-band; clamp oversized param sets
- h264: emit redefined SPS/PPS in-band, clamp oversized avcC
- ac3: flush final frame at EOS, per-frame PTS
- dts: attribute each access unit's PTS to its own core PES
- truehd: capture PTS only at access-unit boundary
- meta: preserve color_space round-trip
- dvd: fix subtitle/audio track PID collision
- pgs/vobsub: flush trailing subtitle at EOF; multi-PES SPU reassembly
- ts: unify codec table via from_coding_type (+LPCM), warn on unknown
2026-06-06 21:46:49 -07:00
MattJackson eeba94b21d Merge branch 'worktree-agent-a184f5a97a37e2c1a' 2026-06-06 21:40:42 -07:00
MattJackson cfc12774f2 mux: fix DVD subtitle/audio track collision, PGS/VobSub flush, unify TS codec table
Subtitle/DVD output-corruption + stream-mapping coverage fixes.

1. DVD subtitle/audio track-mapping collision (CRITICAL). The PS path
   routed 0xBD private-stream packets to a track via (sub_id & 0x1F)+1,
   so VobSub subtitle sub-id 0x20+j aliased audio track j+1: subtitle
   PES was fed to the AC-3 parser and the real subtitle track got
   nothing. Route by the canonical DVD PID instead via a new
   PsPacket::dvd_pid() that mirrors scan_dvd_titles' PID assignment
   (video 0xE0, audio 0xBD00+i, subtitle 0x20+j), then look up the
   track in pid_to_track. Fixed identically at all three sites
   (pipelined_stream consume_ps, disc.rs live feed, disc.rs EOF flush).
   Unmappable/unmapped packets now WARN instead of silently dropping.

2. PGS flush() missing. PgsParser inherited the no-op default flush, so
   the last subtitle of every PGS track (emitted only when a following
   PCS arrives) was dropped at EOF. Implemented flush() to drain the
   pending display set (duration_ns: None for the trailing block).

3. DVD VobSub multi-PES SPU not reassembled. A subpicture unit larger
   than one PES spans multiple PES (only the head carries a PTS).
   DvdSubParser is now stateful: it buffers per sub-stream until the
   leading 2-byte SPU_size is satisfied, inherits the head PTS, and
   emits one Frame. flush() drains a truncated trailing SPU at EOF.

4. One-table hygiene. scan_streams had a duplicate stream_type->Codec
   table that had drifted from Codec::from_coding_type (missing 0x80
   LPCM, 0x85 mapped to DTS-HD MA vs HR, etc.). scan_streams now uses
   from_coding_type plus a new Codec::kind()/CodecKind category split,
   so the two mappings can never diverge. Silent drops in
   scan_streams and bluray STN parsing now WARN with PID + type.

Tests: dvd_pid mapping + subtitle/audio collision regression, PGS
final-subtitle flush, VobSub multi-PES reassembly + EOF flush,
scan_streams 0x80 LPCM via from_coding_type.
2026-06-06 21:33:11 -07:00
MattJackson fd543c058b mux: rustfmt (Rust 1.86) on touched codec/meta files 2026-06-06 21:32:04 -07:00
MattJackson cd8ce708ac mux/h264: emit redefined SPS/PPS in-band + clamp oversized avcC
H.264 stored SPS/PPS in single Option slots and always stripped them
from frame data, so a mid-title parameter-set redefinition (same id,
different body) was lost — those frames decoded against the stale avcC
copy the player re-applies at each keyframe. Same defect class as the
HEVC PPS-redefinition bug. Mirror that fix: emit a changed SPS/PPS
in-band at every occurrence; strip only the first-seen / identical ones.

Also guard avcC's 16-bit NAL length fields: a param set > 65535 bytes
truncated the length while appending all bytes. Return None instead.
2026-06-06 21:24:16 -07:00
MattJackson 2eee777b8f mux/hevc: real chroma/bit-depth in hvcC + clamp oversized param sets
codec_private() hardcoded 8-bit 4:2:0 in the hvcC fixed header, wrong for
10-bit Main 10 UHD (essentially all UHD). Parse chroma_format_idc and
bit_depth_luma/chroma_minus8 from the SPS RBSP (with emulation-prevention
removal and sub-layer profile_tier_level handling) and emit the real
values; fall back to 8-bit 4:2:0 only if the SPS can't be parsed.

Also guard the 16-bit NAL length fields: a param set larger than 65535
bytes would truncate the length while appending all bytes, mis-framing
the record. Return None instead of emitting a corrupt hvcC.
2026-06-06 21:22:44 -07:00
MattJackson a5962be86c mux/truehd: capture PTS only at an access-unit boundary
A continuation PES (one that merely extends a TrueHD AU spanning PES
packets) carries its own later PTS, which the parser was adopting
mid-assembly, snapping the AU's timestamp and breaking the monotonic
per-AU cadence (A/V drift). Capture the PTS base only when the
reassembly buffer is empty, i.e. when a PES actually begins a new AU.
2026-06-06 21:19:53 -07:00
MattJackson 7306f661b9 mux/dts: attribute each access unit's PTS to its own core PES
When two access units flushed in one parse() call (a core arriving in an
earlier PES than the PES that closes the unit, the standard DTS-HD MA
core+extension-as-separate-PES layout), the second AU inherited the
latest PES's PTS instead of its own core's.

Track per-PES PTS markers keyed by buffer offset and stamp each emitted
AU with the PTS of the PES covering its first byte, rebasing markers on
every front drain. Each AU now keeps its own core's timestamp.

Extends two_cores_back_to_back to assert PTS; adds cross-PES coverage.
2026-06-06 21:19:07 -07:00
MattJackson 77f7aced83 mux/ac3: flush final frame at EOS and stamp per-frame PTS
Two defects:
- Ac3Parser inherited the no-op default flush(), so a complete final
  frame still buffered at end-of-stream was dropped (~32 ms of audio
  lost). Add a flush() that drains a complete buffered frame, mirroring
  dts.rs.
- Every frame in one parse() call was stamped with the single PES PTS,
  collapsing their timecodes and drifting A/V. Compute a base PTS once
  per call, then advance per frame by the frame's own duration (AC-3 =
  1536 samples; E-AC-3 from numblkscod), converting samples->ns at the
  stream sample rate (fscod). Each Frame now carries duration_ns.
2026-06-06 21:16:17 -07:00
MattJackson 97ae452e40 mux/meta: preserve color_space round-trip in m2ts metadata
MetaStream::Video dropped color_space on from_title/to_title, hardcoding
BT.709 on the way back. HDR titles (BT.2020) lost their color metadata.

Add a color_space field, populate it in from_title, and use it in
to_title. For pre-0.30.7 metadata that has no color_space, derive it from
the preserved hdr field (all HDR formats are BT.2020, SDR is BT.709).

Adds ColorSpace::id() + FromStr for serialization round-trip.
2026-06-06 21:13:29 -07:00
MattJackson eec0594a30 Merge branch 'main' into worktree-agent-a91cd2cf29779b84e 2026-06-06 21:11:05 -07:00
MattJackson dab6ea4359 mux/hevc: emit redefined param sets in-band at every keyframe
Fight Club redefines PPS id 0 mid-title; the parser froze the first PPS into
codecPrivate and stripped the rest, so the redefined segment decoded against
the wrong PPS (CABAC/cu_qp_delta desync, intact framing). Now any VPS/SPS/PPS
whose body differs from the codecPrivate copy is emitted in-band at every
occurrence, overriding the hvcC copy a player re-applies per keyframe. Proven:
Fight Club re-mux decode errors 320+ -> 0 across all corrupt regions.

Also adds aacs::unit_key_validates (1-block early-reject UK validation) and
ts_sync_count/ts_packet_total helpers.
2026-06-06 21:09:04 -07:00
MattJackson 9a5ed57044 mux: enforce strictly-monotonic per-track block timestamps
Fixes the non-monotonic DTS ffmpeg/players reject (observed on Fight Club audio
streams 16/18: "14061 >= 14060"). Some audio PES PTS truncate to the same
millisecond as the prior frame, or tick back 1ms from rounding. The MKV writer
now tracks the last block timecode per track and nudges a non-increasing one to
prev+1ms (sub-frame, inaudible, A/V sync unaffected at ms granularity), recorded
only for frames actually written. New helper monotonic_ts() + unit test.

Independent of the separate HEVC slice-payload corruption investigation.
2026-06-06 15:57:00 -07:00
MattJackson 7d58ba7b08 v0.30.0: DiscInputs carries the disc's volume label (title)
Add `DiscInputs.volume_label: Option<String>` — the disc's human title (UDF/ISO
volume identifier, else BDMV <di:name>), populated by Disc::inputs() from the
scan. Identity only, no secret, not used in any AACS derivation; lets a key
source forward the title so a key service can catalog disc_hash → title.
2026-06-06 09:05:15 -07:00
MattJackson 4f1dbfd042 v0.29.0: bump version 2026-06-05 20:36:19 -07:00
MattJackson 5b702a76a7 mux: write PTS (presentation), not DTS, as the MKV block timecode
The video codec parsers (HEVC, H.264, VC-1, MPEG-2) used
pes.dts.or(pes.pts) as each frame's timestamp. MKV block timecodes
are presentation timestamps; frames are stored in decode order and
the player reorders for display by timecode. Using DTS makes the
timecode monotonic in storage order, presenting B-frames in decode
order — visible motion judder / wrong frames on playback, and
PTS-based seeking lands on the wrong frame.

The compressed video was always byte-correct (verified by NAL-level
diff against a known-good demux); this was purely a timestamp defect
affecting every B-frame title. Fix: prefer PTS (pes.pts.or(pes.dts)).
Verified on a real UHD iso->mkv: emitted PTS now reorders for
B-frames identically to a reference muxer.

Update the two tests that asserted the old DTS-preferred behavior and
add an HEVC regression test pinning PTS as the block timecode.
2026-06-05 20:01:13 -07:00
MattJackson 6be5198886 libfreemkv: v1.0 hardening — codec/EBML/TS robustness + DTS parser fixes
Audit-driven fixes (rounds 1–3):
- hevc: correct hvcC profile/level SPS offsets (HEVC has a 2-byte NAL header)
- mkv: map all DTS variants to the registered A_DTS codec id; force a new
  cluster before the i16 cluster-relative timestamp can overflow
- ebml/mkvstream: bound untrusted EBML sizes (no multi-GB allocs); reject
  uint>8 (was an OOB panic) and non-{0,4,8} float widths (were a desync)
- ts: skip PES-header bytes that span a TS packet boundary; add the PMT
  section_len/prog_info_len bounds the PAT parser already had
- ac3: preserve a 0x0B77 syncword split across a PES boundary; cap buffer
- dts: validate each next-core boundary by decoded core size (a 0x7FFE8001
  pattern inside XLL payload no longer false-splits/drops the lossless
  extension); reject sub-minimum core frames; fix forced-emit PTS base
- lpcm: DVD program-stream PCM no longer double-strips the BD LPCM header
- vc1/mpeg2: do not emit a parameter-set-only PES as a standalone frame
- pgs/truehd: cap the pending reassembly buffer (parity with ac3/dts)
- aacs: ts_syncs_intact uses the exact packet count
- prefetched: capacity-guard the recycled-buffer set_len
- Cargo.toml: exclude project docs from the published crate

Convergence: a third independent audit pass found no remaining material
(CRITICAL/HIGH/MEDIUM) issues. Full precommit (fmt + clippy -D + tests,
Rust 1.86) green.
2026-06-05 16:23:39 -07:00
MattJackson e2aa9abd6d aacs: validate a resolved key against content before applying it
decrypt_with now takes the disc's encrypted content samples and, after
deriving the candidate unit keys, confirms at least one de-scrambles a real
aligned unit before committing them. A wrong key (a keydb VK that doesn't
match the disc, a stale UK) is rejected with AacsKeyRejected instead of
silently applying garbage unit keys. Conservative by design: with no samples
(resume / mapfile cache) it accepts as before, leaving those paths unchanged.

The KeySource trait becomes a stateful provider — next_key hands one candidate
at a time (the source owns the order) and reports exhaustion, replacing the
all-at-once resolve; errored() distinguishes a failed source from a clean
no-key.
2026-06-05 09:23:08 -07:00
MattJackson 3b7bee9ed4 Fix DTS-HD MA muxing as lossy core only (#10)
DTS-HD MA/HRA access units on Blu-ray are a DTS core frame (sync
0x7FFE8001) followed by one or more DTS extension substreams (sync
0x64582025) carrying the lossless audio. Ground-truthing the Dunkirk
ISO showed the m2ts demuxer hands these out as SEPARATE PES packets on
the same PID: one core PES (exactly core-sized, nothing trailing), then
the extension substreams in following PES packets with their own later
PTS.

The old DtsParser emitted one frame per PES the moment a core frame was
complete, and dropped any PES with no core sync. So every core became a
core-only (lossy) frame and the extension PES packets were discarded as
junk -- silently downgrading the track to lossy DTS core (1557 kb/s CBR,
16-bit) instead of DTS-HD MA (VBR, 24-bit lossless).

Rewrite the parser to assemble across PES boundaries: an access unit
runs from its core sync up to (but not including) the NEXT core sync, so
the core plus every following extension substream stays together. Add a
CodecParser::flush() (default empty) called at end-of-stream by both the
pipelined and inline DiscStream mux paths to drain the final buffered
unit. A 64 KiB cap guarantees forward progress and never stalls if a
boundary can't be found.

Validated on the rip1 testbed: Dunkirk eng+ger and Fight Club eng main
audio now ffprobe as profile=DTS-HD MA (Fight Club eng at 24-bit), with
VBR packet sizes (~2716-2788 B) well above the old fixed 2012 B lossy
core. Genuinely-lossy DTS dub tracks are left untouched.
2026-06-05 06:40:27 -07:00
MattJackson b7405e2d27 mux: fail iso:// with no usable AACS key instead of muxing garbage
When an AACS-encrypted ISO is muxed with decryption requested (not --raw)
but key resolution yielded no usable key, input() proceeded to mux the
still-encrypted stream — emitting ~100 MB of garbage (no TS syncs, demuxer
emits nothing) and sometimes spinning for tens of minutes.

Add a cheap result-check in resolve::input()'s Iso branch via the pure
predicate aacs_key_missing(raw, has_aacs, keys): when decryption is
requested AND the disc carries AACS state AND decrypt_keys() is None,
return new Error::NoDiscKey { disc_hash } (E7022) before muxing. The
40-hex disc hash is sourced from AacsState::disc_hash. --raw and
non-AACS (unencrypted / CSS) discs are unaffected. Unit-tested.
2026-06-05 05:06:15 -07:00
MattJackson e04d79c593 v0.28.1: bump version 2026-06-04 21:46:41 -07:00
MattJackson b34af1fa74 mux: wire TrueHD channel probe into iso:// mux setup
correct_truehd_channels() probes the first DECRYPTED access units of the
chosen title (TrueHD PIDs, bounded 8 MiB read of the first extent) and
sets AudioStream.channels from the MLP major sync — fixing the MPLS
audio_format understatement (5.1 declared on a 7.1/Atmos TrueHD track).
Regenerates the basic codec label for the corrected count; richer
editorial labels are left untouched. Wired in resolve.rs input() for
iso:// after decrypt_with (the m2ts is only decryptable post-key), using
a fresh reader so the mux reader is undisturbed. generate_audio_label
made pub(crate).
2026-06-04 20:28:37 -07:00
MattJackson a7317f8885 mux/truehd: decode true channel count from MLP major-sync format_info
Adds truehd_channels(format_info) + truehd_channels_from_stream(): the
real presentation channel count (8ch/7.1 when present, else 6ch/5.1)
from the TrueHD major sync, which the MPLS audio_format base field
understates (declares 5.1 even on a 7.1/Atmos track — the Dune case).
Per the MLP spec presentation channel-assignment masks. 3 unit tests.

This is the decoder; wiring it to correct AudioStream.channels requires
a mux-time probe of the DECRYPTED audio payload (the channel truth isn't
readable at scan — the m2ts is AACS-encrypted until the key is applied
at mux), to be added + validated on a real rip.
2026-06-04 19:51:46 -07:00
MattJackson 474273afc0 mux/mkv: signal Dolby Vision via dvcC BlockAdditionMapping
With the EL now carried (mpls PID fix), the MKV writer tags the Dolby
Vision layer so players/mediainfo recognise it: a BlockAdditionMapping
(0x41E4) with BlockAddIDType 'dvcC' and a DOVIDecoderConfigurationRecord
(profile 7, bl+el+rpu present) on the DV-layer video track. Disc
Profile 7 dual-layer is preserved losslessly as a second video track
(archival-correct, MakeMKV-equivalent); M2TS carries the EL as a native
second video PID with no extra signaling needed.

Adds dolby_vision_config() + a profile-7 dvcC unit test. End-to-end DV
recognition (mediainfo / playback) to be confirmed on a full rip.
2026-06-04 19:42:14 -07:00
MattJackson 0a2bab5789 mpls: resolve PID for sub-path / DV-EL stream entries (types 2/3/4)
parse_stream_entry only read the PID for stream-entry type 1 (a stream
in the PlayItem's own clip), returning PID 0 for every other type. The
Dolby Vision enhancement layer uses type 4 (verified on Wicked + Dune:
se_len=9 type=0x04 [00 10 15] -> PID 0x1015), so the EL fell through to
PID 0x0000 and was silently dropped by the demux/mux — Dolby Vision lost.

Now the PID offset is keyed off the entry type per the BD stream_entry()
layout: type 1 -> +2, type 2 -> +4, type 3/4 -> +3. The DV EL now
resolves to its real PID (0x1015) so it is demuxed and carried as a
stream through the PES layer to every writer (M2TS carries dual-PID DV
natively; the MKV writer's DV signaling is the format-specific piece).
2026-06-04 19:28:34 -07:00
MattJackson fa4d7ef871 mux/dts: keep DTS-HD extension substream across PES boundaries
The DTS parser emitted a core-only frame and discarded the trailing
DTS-HD extension substream whenever the extension straddled a PES
boundary (it advanced past only the core, then re-synced on the core
syncword, skipping the leftover extension bytes). That silently
downgrades DTS-HD MA / HRA to lossy DTS core.

Now: when an extension sync is visible after the core — full, or a
partial prefix at the buffer edge — wait for the full extension instead
of splitting it off; only 'nothing after the core' (final unit / EOF,
no parser flush) is taken as a genuine lossy core-only unit. Adds a
boundary-split regression test asserting core+extension is preserved.
2026-06-04 19:16:26 -07:00
MattJackson 36d1af1b7f labels+disc: codec from stream, audio-richness title tiebreak
Two validated audio-correctness fixes (proven on real discs Wicked/
Paddington/Dune/Fight Club via head-captures):

labels: apply_labels now derives the codec/channel descriptor from the
stream's OWN codec/channels unless the parser's codec_hint is BOTH
consistent with it AND richer (e.g. "Dolby Atmos" on a TrueHD stream).
A mis-bound hint ("AC-3 2.0" on a TrueHD track) is rejected and the
stream's own codec used — killing the cross-labeled shuffle (Wicked) and
the compat-core mislabel (Paddington), while keeping rich hints and
normalizing plain ones to uniform marketing names. (codec_hint_consistent
+ codec_hint_adds_detail, 5 tests.)

disc: canonical_title_order gains an audio-richness tiebreak
(lossless > channels > track-count) for titles that tie on
duration+clips — so a movie authored as a full-audio playlist plus a
stereo-only twin (Fight Club 00800 vs 00004) picks the full-audio one
instead of falling to array order.
2026-06-04 19:10:11 -07:00
MattJackson b96f6206fe 0.28.0: version bump (keydb decoupling) 2026-06-04 16:41:04 -07:00
MattJackson bb32fb993b 0.28.0 review fixes: propagate mux key-apply error; scrub key-shaped fixtures
- mux input(): propagate a failed decrypt_with rather than muxing an
  undecryptable stream silently (review M6).
- Scrub synthetic but key-SHAPED hex (HOST_PRIV_KEY/DEVICE_KEY/disc-entry) from
  the keydb parser tests to obvious zero/repeated-byte placeholders. No real or
  real-looking key material in code.
2026-06-04 16:38:58 -07:00
MattJackson c1eb74dfa5 AACS: cutover — lookup-free lib; DriveCredentials for the handshake
The library no longer loads a keydb anywhere. The scan path always captures the
disc's AACS inputs (MKB, VID, Unit_Key_RO.inf) and resolves NO key; a caller
resolves a Key from a key source and applies it via Disc::decrypt_with.

- ScanOptions loses keydb_path / unit_key / disable_keydb (and the path search);
  it now carries only optional DriveCredentials (host certs) for the live-drive
  AACS handshake. do_handshake_cert uses those instead of loading the keydb.
  An unlocked / LibreDrive drive takes the OEM Volume-ID path and needs none.
- The mux input() path takes caller-resolved unit_keys instead of a keydb_path,
  and applies them via decrypt_with.
- Deleted the now-dead inline resolve_encryption / resolve_encryption_static.

All 700+ lib tests pass.
2026-06-04 16:04:06 -07:00
MattJackson 46838c63ca AACS: trim the MKB captured at keyless scan to its record length
The VID-only scan path stashed MKB_RO/RW raw — those files are allocated to a
fixed ~128 MiB and zero-padded, so the MKB on AacsState (consumed by
Disc::inputs() and the device/processing-key decrypt_with derivation) was the
full pad, not the ~few-MB record stream. Trim to mkb_content_len, matching
read_aacs_inputs.
2026-06-04 15:32:12 -07:00
MattJackson 1e60220ff6 AACS: KeySource::needs_samples() gates the content-sample read
A source that validates server-side against ciphertext (an online key service)
needs encrypted content samples; one that keys on disc identity (keydb, mapfile)
does not. needs_samples() lets the caller skip the extra disc read unless a
configured source actually needs it. Defaults false.
2026-06-04 14:45:40 -07:00
MattJackson 080f03e8ed AACS: DiscInputs gains app-populated samples for ciphertext-validating sources
An online key service validates a candidate against real ciphertext, so it
needs a few encrypted content sample units. Add a samples field to DiscInputs;
Disc::inputs() leaves it empty (reading content needs the disc reader, which
scan does not retain) for the application to fill.
2026-06-04 14:42:11 -07:00
MattJackson 35de6101d4 AACS: add KeySource trait + DiscInputs; Media key carries a candidate pool
The library now describes the key-source abstraction it consumes, while the
concrete sources live in a companion crate. A KeySource looks a disc up from
its DiscInputs (disc hash, volume id, MKB, Unit_Key_RO.inf) and returns a Key;
the library derives down and decrypts. Sources do no derivation.

Disc::inputs() exposes those inputs from a scanned disc.

Key::Media now carries a Vec of candidate media keys rather than one: a media
key is MKB-scoped, so a source hands its whole pool and the library picks the
one that verifies against the disc's MKB. That mirrors Device/Processing and
keeps key selection (a derivation step) inside the library.
2026-06-04 14:29:32 -07:00
MattJackson 9012101573 AACS: decrypt_with accepts higher-level keys and derives down
Disc::decrypt_with now takes Device / Processing / Media / Volume keys in
addition to Unit. A caller hands in whatever level it resolved and the
library derives down the AACS chain to the per-CPS-unit keys, then
decrypts:

  Device   -> MKB walk         -> media key -> VUK -> per-CPS unit keys
  Processing -> MKB            -> media key -> VUK -> per-CPS unit keys
  Media      -> Volume ID      -> VUK              -> per-CPS unit keys
  Volume     -> Unit_Key_RO.inf, one unit key per CPS unit
  Unit       -> used directly (terminal)

Derivation stays centralized in the version-dispatched resolver
(1.0 / 2.0 / 2.1), fed by a single-key provider built from the supplied
key — no new crypto. Volume notably does NOT stop at the volume key: it
decrypts every CPS unit's key.

Scan stashes the AACS inputs (Unit_Key_RO.inf and MKB) on AacsState so an
out-of-band decrypt_with can derive without re-reading the disc.

Non-breaking: Key is #[non_exhaustive] and the existing Unit path is
unchanged. New tests cover the Volume -> per-CPS derive-down, the
missing-inputs error, and the no-units rejection.
2026-06-04 14:20:16 -07:00
MattJackson 8bc1de6c9b 0.27.5 (step 2, Phase 1): expose AACS inputs (uk_ro, mkb) on AacsState
scan now stashes the raw Unit_Key_RO.inf + MKB bytes on AacsState (via
resolve_vid_only, the disable_keydb path), so an external key-resolver can
derive unit keys from a resolved VUK without re-reading the disc — the
foundation for moving lookup/derivation out of libfreemkv. Additive: the keydb
path is untouched, all existing constructors default the new fields empty.
584 lib tests green. Builds on the KeyOrigin rename + the Key/decrypt_with API.
2026-06-04 13:20:27 -07:00
MattJackson d94a4d3444 0.27.5 (step 2, item 2): rename provenance enum KeySource -> KeyOrigin
Disambiguates the key vocabulary: Key (the input handed to decrypt_with),
key sources (the resolver's lookup list), and KeyOrigin (how a key was
resolved). Internal-only rename — no dependents import it.
2026-06-04 11:42:21 -07:00
MattJackson 1bd7e1de2a 0.27.4: lookup-free Key API (Disc::decrypt_with) + E8005 deferred-mux fix
Add libfreemkv::Key (non_exhaustive; Unit wired) + Disc::decrypt_with(Key) — the
caller resolves a key from any source and hands it in; the library decrypts with
no keydb/network of its own. inject_unit_keys now synthesizes a minimal
ExternalUk AACS state when a scan built none (no keydb at scan time), so a disc
keyed at sweep but scanned without a keydb still decrypts from its mapfile UK
instead of reporting E8005 and deferring the mux forever. Tests cover the
synthesize, replace, empty->None, and decrypt_with paths.
2026-06-04 11:08:33 -07:00
MattJackson da62ee7cf2 v0.27.3: bump version 2026-06-03 21:47:43 -07:00
MattJackson 575c76156f mapfile: persist resolved unit keys (keys XOR VID)
A keyed disc now writes its decrypted AACS unit keys to the mapfile header
(# freemkv-uk: <cps>:<hex>); an unresolved disc writes only the VID. The two are
mutually exclusive (set_unit_keys clears the VID) — unit keys are the final
answer, so deferred-mux / resume decrypts directly with no key lookup, while the
VID alone is the 'still unresolved, retry' marker. CopyOptions/SweepOptions carry
the keys (written when present, else the VID); Disc::inject_unit_keys applies
mapfile-recovered keys to a scanned disc. Round-trip test added.
2026-06-03 21:45:37 -07:00
MattJackson b518860d9c v0.27.0: detect AACS-scrambled units by raw TS sync, not flag bits
Rename is_unit_encrypted -> is_aacs_scrambled and decide encryption from the unit's MPEG-TS sync bytes (destroyed by the encrypted body) instead of the TP_extra copy-control (byte 0) or TS scrambling-control (byte 7) flags, which discs do not set reliably. One shared predicate now backs the decrypt gate and out-of-band key validation, so callers agree on what 'encrypted' means. Decryption restores the syncs, so a decrypted unit reads as clear and there is no flag to clear.
2026-06-03 07:35:50 -07:00
MattJackson c8eb42b490 v0.26.11: detect AACS unit encryption via TS scrambling-control bits
is_unit_encrypted read the TP_extra copy-control bits (byte 0), which are a
copy-permission flag, not an encryption flag. On discs whose sampled units are
clear navigation packets (PAT/PMT) those bits can be set while the unit is not
scrambled, so a correct Unit Key was used to 'decrypt' already-plaintext data,
produced garbage, and the key was wrongly treated as failing.

Read the actual flag instead: the TS transport_scrambling_control bits (top two
of TS-header byte 3 = byte 7 of the aligned unit, inside the clear seed). AACS
encrypts whole aligned units, so this one packet's TSC reflects the unit.
decrypt_unit now clears the TSC bits of every packet on the way out so the
result is valid unscrambled TS. Tests updated to the TSC flag.
2026-06-02 20:37:56 -07:00
MattJackson 48570ac065 Local resolve: MK-pool brute via km_verifies (path 2.5)
keydb stores Media Keys per-disc, but an MK is MKB-scoped (shared across a
pressing/MKB-family). A disc whose own hash/VID isn't keyed can still resolve
if any stored MK verifies against its MKB. New path 2.5 (between PK and the
VID lookup) collects the distinct MK pool from the providers, km_verifies each
against the disc MKB, and on a UNIQUE pass derives VUK (with the disc VID) then
the UK — matching the online resolver's behavior so local keydb mode resolves
the same discs (e.g. an MK present in keydb under a sibling pressing).
km_verifies is one AES-D + magic check per candidate (cheap). Adds
KeyProvider::media_keys() + a path-2.5 unit test.
2026-06-02 16:14:09 -07:00
MattJackson 08e46640fd v0.26.8: bump version 2026-06-02 15:41:38 -07:00
MattJackson a9195824ff Trim MKB to real record length; empty unit-keys means no keys
- mkb_content_len walks the MKB record stream and returns where it ends,
  so callers can drop the trailing zero padding. MKB files are allocated
  to a fixed size (~128 MiB) with records at the front; both MKB_RO and
  MKB_RW can be padded. read_aacs_inputs* now trim to the real records
  (~few MB) instead of shipping ~128 MiB of zeros.
- decrypt_keys() returns None when unit_keys is empty: an AACS state that
  carries only a VID (out-of-band resolution, pre-key) is 'encrypted, no
  keys', not a usable empty key set.
2026-06-02 15:41:27 -07:00
MattJackson 401fe23988 v0.26.7: bump version 2026-06-02 15:15:58 -07:00
MattJackson f89bce5851 Read MKB_RO not the padded MKB_RW; keep VID when keydb is disabled
- read_aacs_inputs / read_aacs_inputs_from_drive now read MKB_RO.inf first.
  MKB_RW.inf is a fixed ~128 MiB rewritable region that is mostly zero
  padding; reading it shipped 124 MiB of nothing. MKB_RO is the real,
  correctly-sized MKB (a few MB). Fall back to RW only if RO is absent.
- disable_keydb no longer drops the Volume ID. A caller resolving Unit Keys
  out-of-band needs the VID (on-disc content read during the handshake).
  New resolve_vid_only builds a keys-free AacsState carrying just the VID +
  version metadata, so the disc reports 'encrypted, no keys' (resolved
  out-of-band) instead of discarding the VID.
2026-06-02 15:15:45 -07:00
MattJackson dfccb85e15 v0.26.6: bump version 2026-06-02 14:11:13 -07:00
MattJackson ebedffb762 ScanOptions: add disable_keydb to skip all keydb lookup
A caller that resolves Unit Keys out-of-band can now set disable_keydb
so the scan consults no keydb at all — neither an explicit keydb_path
nor the standard search locations. Without it, a keydb that merely sits
in a default location ('~/.config/...') silently shadows the out-of-band
path. unit_key still takes precedence over everything.
2026-06-02 14:09:59 -07:00
MattJackson fdf63ccb7f v0.26.5: bump version 2026-06-02 13:37:32 -07:00
MattJackson c735d284da v0.26.4: bump version 2026-06-02 13:06:10 -07:00
MattJackson 94ab7bc73c scan: genericize the caller-supplied unit-key docs (no external-source naming)
The ScanOptions.unit_key path is a generic primitive — a caller-supplied Unit
Key that bypasses keydb lookup. Doc comments + a tracing log named a specific
external source; reworded to neutral 'out-of-band / external key service' so
the library makes no assumptions about where the key came from.
2026-06-02 13:00:11 -07:00
MattJackson 9f209fe066 v0.26.3: bump version 2026-06-02 10:56:56 -07:00
MattJackson 32a1a6e095 mapfile: persist AACS Volume ID as ddrescue-safe comment
Add an optional 16-byte AACS Volume ID to the Mapfile, persisted as a
'# freemkv-vid: <32-hex>' header comment so it survives to deferred-mux
and resume without touching the ISO payload. ddrescue treats the line
as a comment, so the pos/size/status data parser is unaffected.

- Mapfile gains set_vid/vid accessors; save() emits the comment in the
  header block, load() parses it back (malformed/absent -> None, never
  errors the load). Load->save round-trip preserves it (patch pass).
- Thread vid through CopyOptions and SweepOptions; sweep persists it
  when creating/opening the mapfile.
- Unit test: round-trips the VID, asserts the saved comment, and
  confirms data ranges parse identically with and without the comment.
2026-06-02 08:50:19 -07:00
MattJackson 1d3b8f5fb6 chore: gitignore scratch/ (prevent accidental commit of session artifacts) 2026-06-01 22:52:58 -07:00
MattJackson 97b0ae7be2 chore: scrub internal refs from changelog/makefile/cargo 2026-06-01 22:46:54 -07:00
MattJackson 7756f1feca aacs: restore read_aacs_inputs_from_drive (live-drive key-file extractor) 2026-06-01 21:52:27 -07:00
MattJackson 1565da610a chore: scrub non-shippable references from tests/comments 2026-06-01 21:36:57 -07:00
MattJackson e134616422 v0.26.2: bump version 2026-06-01 21:06:56 -07:00
MattJackson 8d54a3c64e aacs: KeyProvider abstraction + PK walker + external-UK key source
Two coherent additions to the AACS resolver:

KeyProvider abstraction (provider.rs) — key material comes from pluggable
backends; KeyDb implements it (device/processing keys, host certs,
disc-by-hash / disc-by-vid lookup) plus orphan-DK parsing. ResolveContext
takes a provider array. Adds the SD-tree PK walker
(derive_media_key_from_pk_walked) and a `probe` module (km_verifies MK
oracle, mkb_* record parsers) used for offline key verification. Cvalue
record selection prefers 0x05, falls back to 0x07.

External-UK key source — the second, mutually-exclusive key source for the
keyserver path. ScanOptions/InputOptions gain `unit_key`; when set,
resolve_encryption_static skips keydb entirely and uses the caller-supplied
Unit Key directly (KeySource::ExternalUk). Disc::read_aacs_inputs exposes a
disc's Unit_Key_RO.inf + MKB so a caller can fetch the UK out-of-band; the
library makes no network call itself.

CHANGELOG: redact test-disc title in historical notes.
2026-06-01 20:47:14 -07:00
MattJackson c0478e1273 labels(pixelogic): number audio from the FPL playlist, not menu segments
Pixelogic discs that authored a SEG_MainFeature menu/preview segment
containing a stray commentary token made the parser anchor on that
segment instead of the real FPL_MainFeature playlist. Combined with
'Audio Stream N' placeholder tokens not advancing the audio counter, a
lone eng_ACOM_ commentary at STN slot 4 collapsed onto slot 1 — tagging
the main feature track as Commentary (seen on Wicked: For Good).

Fix:
- Anchor the feature section on the FPL_ playlist when present; only
  fall back to SEG_MainFeature on discs with no FPL_ section.
- Count 'Audio Stream N' placeholders so editorial audio tokens get
  their true STN ordinal. Subtitle (PG Stream N) numbering is left
  unchanged — counting it regressed several corpus discs.

Extract the section/numbering loop into assign_labels with unit tests.
Verified: 11/11 labels-corpus discs still PASS; Wicked commentary now
lands on the AC-3 2.0 stereo track instead of the main TrueHD.
2026-05-31 21:16:11 -07:00
MattJackson 7d29168fec v0.26.1: AACS resolver path 5 — KEYDB unit-keys direct fallback
Adds a 5th key-resolution path that consumes pre-decrypted unit keys
directly from KEYDB when the entry has no VUK field. Covers ~4,572
entries in the public keydb (~2.5%), heavily skewed toward MKBv76+ UHD
discs where DVDFab/FindVUK can no longer extract a VUK but does extract
unit keys. Partial CPS-unit coverage is rejected so a disc is never
half-decrypted.

Resolver path order reordered root-to-leaf: DK (1) → PK (2) →
KEYDB-derived MK+VID (3) → KEYDB VUK (4) → KEYDB unit keys (5).
Previous order was leaf-first.

API:
- AacsState::vuk is now Option<[u8; 16]> (was [u8; 16])
- ResolvedKeys::vuk is now Option<[u8; 16]> (was [u8; 16])
- KeySource variants reordered + new KeyDbUnitKeys variant

3 new resolver tests (path 4 still works without VID; path 5 succeeds
with pre-decrypted unit keys; path 5 rejects partial CPS coverage).
2026-05-22 12:51:37 -07:00
MattJackson 8e14c9b850 v0.26.0: bump version 2026-05-21 15:18:39 -07:00
MattJackson e1c8343f77 aacs: OEM-driven VID retrieval — per-drive CDB from profile, cert fallback
When the drive is in extended-access state (unlocked), retrieve VID via
the per-drive `read_vid_cdb` from the bundled profile instead of the
cert-based AACS REPORT_KEY handshake. Cert handshake remains the
fallback for drives that don't enter extended-access state, or whose
profile lacks the required CDB.

Empirically verified on the BU40N (signature 999ec375) against
Barbie UHD: drive returns 36 bytes from buffer 0x44 at offset
0x10E291, VID at response[4..20]. The 16 bytes match Dune Part Two's
known VID in keydb.cfg byte-for-byte, cross-validating the path
against an independent oracle.

Architectural impact:

- Renames `Drive::is_libredrive_active()` → `Drive::is_unlocked()`.
  Internal `Mt1959::libredrive_active` becomes `Mt1959::unlocked`;
  the prior `unlocked` (init-success flag) becomes `init_complete`
  to avoid the name collision.
- `disc/encrypt.rs::Disc::read_vid` is the single entry point.
  When `is_unlocked()` is true, calls `read_vid_oem` (issues the
  per-drive CDB, validates the response signature high-3-bytes
  `00 22 00`, returns bytes [4..20]). Otherwise delegates to
  `read_vid_cert` (the existing AACS REPORT_KEY format 0x80 path).
- `DriveProfile` gains the per-drive CDB templates and identifier
  blocks extracted from each per-drive firmware payload — including
  `read_vid_cdb`, `read_disc_keys_cdb`, `drive_nominal_speed_cdb`,
  `set_speed_max_cdb`, two cache-prime canary CDBs, the buffer-0x45
  verify CDB, the firmware-upload CDB, and the unlock probe CDB.
  Variants A and B differ in which fields are populated. All optional;
  consumers fall back to the cert/handshake path when fields are
  absent.
- New error variants `Error::DriveProfileMissing` (E7020) and
  `Error::VidCdbUnavailable` (E7021). Both treated as
  "OEM unavailable → try cert path" by `read_vid`, not terminal.

Closes the v0.25.x gap where HRL-burned host certs (the public
libaacs leaked cert is on every recent drive's HRL) blocked all
post-handshake VID retrieval. With OEM-driven VID:

- AACS 1.0 BD on supported drives: rips end-to-end with our existing
  DKs walking the MKB.
- AACS 2.x UHD: fails honestly at the DK wall (E7018 "No usable DK"
  for v77+ MKBs) instead of the misleading E7017 "No Volume ID"
  the prior code surfaced. We have VID; we just don't have v77+ DK
  material — that gap is a key-acquisition problem, not a code
  problem.

Empirically verified on rip1 (BU40N + Barbie UHD, MKB v77,
2026-05-21): error code flipped from E7017 to E7018 as predicted.
The DK wall is now correctly the proximate failure for unrippable
modern UHD discs, instead of the indirect VID-retrieval wall the
v0.25.x cert-only path produced.

Renames and comment scrubs eliminate upstream-RE-vocabulary
references in the public crate per `feedback_no_breadcrumbs.md`.

674 tests pass (565 lib + 109 integration). No tradename leaks in
any modified file.
2026-05-21 15:18:21 -07:00
MattJackson a956c6ad94 v0.25.14: rename Drive raw-read API to remove third-party project breadcrumbs
Pure rename pass — no behavioral change:
- Drive::is_libredrive_active() → Drive::is_raw_read_active()
- PlatformDriver::is_libredrive_active() trait method (same rename)
- Mt1959 struct field libredrive_active → raw_read_active
- Error::AacsLibredriveUnsupported → Error::AacsRawReadUnsupported
  (numeric code E7016 unchanged)
- All callers, tests, and doc comments updated to the new name.

Old identifiers removed entirely; downstream consumers must update.
Mirrored in bdemu, freemkv, autorip, freemkv-tools.
2026-05-21 14:43:20 -07:00
MattJackson 1805d92ca4 v0.25.13: DrmScheme dispatcher + AACS 2.1 framework + libredrive cleanup
- Introduce DrmScheme enum (Css/Aacs10/Aacs20/Aacs21) + drm module with
  uniform detect/load dispatch across all four protection schemes.
- Land AACS 2.1 Media Key Variant framework in aacs::variants: chain
  derivation, MKB record types 0x82/0x83, bit-0x02 SoftKCD and bit-0x04
  online-challenge detection. Aacs21 dispatcher arm wired but commented
  out pending validation against a Variant-scheme disc.
- Replace aacs2: bool with AacsVersion enum across ContentCertificate,
  UnitKeyFile, ResolvedKeys. resolve_keys splits into _v1/_v2/_v21.
- Delete the libredrive raw-read VID shortcut from do_handshake; the
  drive enforces the AGID requirement regardless of firmware-upload
  state, so the shortcut spuriously dispatched E7017 instead of
  surfacing the real downstream walls.
2026-05-21 13:57:45 -07:00
MattJackson 823f0ad430 v0.25.12: bump version 2026-05-21 11:37:53 -07:00
MattJackson 477bdf1835 v0.25.11: bump version 2026-05-21 11:14:22 -07:00
MattJackson 4d83b69c20 aacs: libredrive raw-read VID path + revert v0.25.9 built-ins + walker fix follow-through
Three coherent threads landing for v0.25.11:

1. Libredrive raw-read VID path. When Mt1959::do_unlock sees both the
   MMkv active-mode marker at [12..16] and the LbDr mode-ID marker at
   [16..20], Drive::is_libredrive_active() returns true and
   do_handshake skips the AACS cert dance — VID is retrieved via
   READ_DISC_STRUCTURE format 0x80 with AGID=0 and bus encryption is
   already off. This unblocks UHD ripping on drives whose leaked host
   cert is on the AACS HRL.

   - platform/mt1959/mod.rs: detection + active flag + 4 unit tests.
   - platform/mod.rs: PlatformDriver::is_libredrive_active trait method.
   - drive/mod.rs: Drive::is_libredrive_active accessor.
   - disc/encrypt.rs: do_handshake branches on the flag; new
     read_volume_id_libredrive helper. Return type widened to
     (Option<HandshakeResult>, Option<Error>) so callers see which
     specific failure happened.
   - disc/mod.rs: scan_with plumbs the new tuple through and preserves
     handshake errors as disc.aacs_error.

2. Revert v0.25.9 built-in AACS keys + plugin slot. Single source of
   AACS truth: keydb.cfg. The compiled-in DKs/PKs were a slim
   convenience that didn't move the hard problem (no v77+ DKs) and
   added a maintenance surface. Plugin slot was overlapping
   functionality with the main keydb.

   - Deleted src/aacs/builtin_keys.rs (4 DKs + 3 PKs).
   - Removed KeyDb::with_builtins, load_or_builtins, merge_from,
     merge_local_plugin, local_plugin_path, internal dedup helpers.
     KeyDb::empty kept for unit-test use.
   - KeyDb::load reverts to pre-0.25.9 form: read file or return I/O
     error; no fallback.
   - disc::encrypt::resolve_encryption keydb_path back to required
     (&Path), not Option<&Path>.
   - disc::scan_with surfaces KeydbLoad { path: "<no keydb in search
     paths>" } sentinel when encrypted + no keydb — same sentinel
     autorip's message switch already handles.
   - CSS player keys in src/css/auth.rs stay compiled in; they're
     1999-era public inputs separate from AACS and pre-date the 0.25.9
     additions.

3. Walker fix follow-through (libaacs-parity validate_processing_key,
   cvalues 0x07-then-0x05 preference, path-2/3/4 short-circuit on
   zero VID) + NIST AES-CMAC KAT + VID MAC round-trip / mutation /
   zero-rejection tests.

5 new Error variants for finer-grained AACS failure reporting:
AacsHostCertRejected (E7015), AacsLibredriveUnsupported (E7016),
AacsVidUnavailable (E7017), AacsMkUnavailable (E7018),
AacsVukNotInKeydb (E7019). Lets CLIs/UIs render which piece of the
AACS chain failed instead of always saying "no keys."
2026-05-21 11:10:35 -07:00
MattJackson 7dbbfc6726 v0.25.10: bump version 2026-05-20 15:35:05 -07:00
MattJackson e635c9556f v0.25.9: built-in AACS keys + plugin slot + MKB record-type fix
Two changes that make AACS 1.0 / DVD self-sufficient:

1. MKB record-type identification bug fix. `mkb_find_mk_dv` was
   searching for type 0x10 (which is Type-and-Version, 12 bytes)
   when the Verify Media Key Record is actually type 0x81 for
   AACS 1.0 or type 0x86 for AACS 2.0/2.1. `mkb_version` had the
   inverse bug. PK and DK derivation paths therefore silently
   failed on every disc, masking how often the fallback paths
   could have worked. Fix searches the correct types; tests added
   covering both the 0x81 and 0x86 verify-record forms and the
   0x10 version record at offset 8 of the body.

2. Built-in AACS keys + operator plugin slot. Four device keys
   (covering MKB v01-v82+) and three processing keys (covering
   v63-v68) compiled directly into the library. Combined with the
   31 CSS player keys already in css/auth.rs, DVDs and Blu-rays
   (AACS 1.0) now decrypt with zero external files. New plugin
   path at ~/.config/freemkv/local_keys.cfg (same syntax as
   keydb.cfg) layered additively on top of built-ins and main
   keydb. `Disc::scan` no longer errors when keydb.cfg is absent;
   AACS 2.0 / UHD still surfaces a specific error when the disc
   needs keys none of the layers provide.

Public docstrings in project docs + README updated to describe the
three additive layers (built-ins → keydb.cfg → local_keys.cfg).
2026-05-20 09:00:32 -07:00
MattJackson d7b5c30f5d v0.25.8: bump version (unified release with autorip 0.25.8) 2026-05-19 21:45:45 -07:00
MattJackson 739a276a39 v0.25.7: BU40N firmware wedge fix in do_handshake
Pre-0.25.7 the AACS authenticate loop fired up to 16 host-cert
attempts back-to-back with no pause. Each attempt is 5-10 SCSI
REPORT_KEY/SEND_KEY exchanges, so on a disc whose host cert isn't
in our KEYDB (or one the drive rejects), the drive saw 80-160 SCSI
commands in a few hundred ms and entered a fast-fail firmware
wedge state where every subsequent CDB returns sense 05/24 until
power-cycled.

Three defences:
- MAX_CERT_ATTEMPTS capped at 3 (was 16)
- 1-second sleep between attempts
- Bail immediately on any sense_key == 0x05 (ILLEGAL_REQUEST) so
  the loop can't deepen the wedge if a regression undoes the
  attempt cap.
2026-05-19 21:18:45 -07:00
MattJackson 5f1028a62a v0.25.6: sync to autorip 0.25.6 (image diet) 2026-05-19 18:17:10 -07:00
MattJackson ea15d212de v0.25.5: sync to autorip 0.25.5 2026-05-19 18:01:14 -07:00
MattJackson 00673c8ec3 v0.25.4: sync to autorip 0.25.4 2026-05-19 17:44:23 -07:00
MattJackson bfefb4cb5b v0.25.3: sync to autorip 0.25.3 release (no behavioural changes) 2026-05-19 17:11:11 -07:00
MattJackson 1b95193517 v0.25.2: DTS-HD codec ID + PGS BlockDuration
- MkvTrack::audio emits A_DTS/MA, A_DTS/HR, A_DTS per the DTS family
  instead of mislabelling everything as A_DTS. Plex transcoder and
  strict hardware decoders reject DTS-HD MA payload under a plain
  A_DTS track.
- PgsParser is now stateful: pairs display PCS with the following
  empty PCS to compute a duration. Frame::duration_ns + PesFrame::duration_ns
  carry it through; MkvMuxer::write_frame gains a final Option<u64>
  parameter that emits BlockGroup + BlockDuration when set. Fixes
  subtitle bitmaps lingering past their intended end-time.
2026-05-19 16:11:54 -07:00
MattJackson 7dcac44136 v0.25.1: bump version 2026-05-19 14:20:44 -07:00
MattJackson 4da559e39f mux: PrefetchedSectorSource event_fn + delete DiscStream::new_pipeline
* `PrefetchedSectorSource::new_with_events` adds an optional
  `event_fn` callback that fires `BytesRead` after every successful
  batch from the producer thread. The original `new()` becomes a
  thin no-events wrapper. Lets autorip wire the highway and still
  get UI progress events without polling the consumer side.

* `build_iso_pipeline` grows an `event_fn` arg so the autorip
  multipass mux can pipe BytesRead straight through to its progress
  UI.

* Stream trait gains a default `errors() -> u64` method (= 0) so
  Box<dyn Stream> callers (autorip's mux loop) can read the
  skip-on-error counter without downcasting. `DiscStream` overrides
  to return its `errors` field.

* Delete `DiscStream::new_pipeline` and the pipeline-mode fields
  (`demux_thread`, `demux_rx`) plus the `read_pipeline` helper.
  All pipeline construction now goes through
  `PipelinedPesStream` via `build_iso_pipeline`; `DiscStream`
  becomes the single-thread-only inline path used by the drive
  single-pass read.

* `lib.rs` re-exports `build_iso_pipeline`.
2026-05-19 14:19:37 -07:00
MattJackson eeca250f69 v0.25.0: bump version 2026-05-19 13:37:24 -07:00
MattJackson c51b3181f2 mux: pipelined PES highway — read+decrypt → demux → parse on 3 threads
Introduces the freemkv mux throughput highway: a three-stage thread
pipeline that replaces the inline single-thread read path for any
file-backed source (ISO and m2ts file URLs both route through it).

  Thread A: read + decrypt  (PrefetchedSectorSource / BytePrefetcher)
  Thread B: M2TS demux      (DemuxThread)
  Thread C: codec parse     (PipelinedPesStream, on caller thread)

Each handoff uses a bounded crossbeam channel with a recycled buffer
pool — no allocations or memcpys in the steady-state hot loop.

Component map:

* io/byte_prefetcher.rs (new) — std::io::Read producer thread with
  recycled Vec<u8> pool. Pairs with PrefetchedSectorSource (sector
  side) so demux_thread::spawn_zero_copy can wire either upstream.
* sector/prefetched.rs — recycled buffer pool added; into_channels()
  peels off the rx/recycle_tx/shell triple for zero-copy demux.
* mux/demux_thread.rs (new) — owns the TsDemuxer/PsDemuxer, runs
  feed() on its thread, ships Vec<PesPacket> batches.
* mux/pipelined_stream.rs (new) — the read-side Stream impl. Pulls
  packets from the demux thread and runs codec parse on the caller.
* mux/resolve.rs — build_iso_pipeline (public) / build_m2ts_pipeline
  (private) assemble the three stages; iso:// and m2ts:// both
  return PipelinedPesStream.
* mux/m2ts.rs — collapsed to a write-only sink (Mode::Read deleted;
  the read direction lives on the highway now).
* mux/codec/h264.rs — find_start_code uses memchr SIMD memmem::find.
* mux/codec/hevc.rs — tightened frame_data initial capacity.
* mux/ts.rs — boundary-packet handling avoids the per-batch 16 MiB
  remainder copy; PesAssembler starts at 16 KiB to dodge the 64-page
  first-touch fault tax that the previous 256 KiB pre-alloc paid on
  every PES boundary.
* mux/disc.rs — gains DiscStream::new_pipeline + read_pipeline as
  the legacy autorip ingress (drive + multipass paths still need
  on_event / skip_errors before they migrate to the highway).
* io/file_sector_source/* — per-OS prefetch() syscall hook
  (Linux readahead, macOS F_RDADVISE, Windows/other no-op).
* decrypt.rs — FREEMKV_DECRYPT_THREADS renamed to FREEMKV_THREADS;
  pool sized to all cores by default.

Measured on rip1 testbed (Civil War UHD, 62 GiB ISO → null://):

  60 → 322 MB/s warm cache (old new_pipeline path)
  60 → 660 MB/s warm cache (highway path, this commit)
  60 → 126 MB/s sustained disk-bound

The IsoSectorReader baseline reader was deleted in favour of
FileSectorSource so the freemkv CLI and autorip exercise the same
read path.
2026-05-19 13:35:32 -07:00
MattJackson 2a31a47434 mux: keyframe-align MKV clusters + SeekHead; set TS RAI on keyframe PES
MKV: cluster boundaries now require a video keyframe in addition to the
5s minimum, so every cluster has a CuePoint at its start. Pre-first-
keyframe frames are dropped. Adds a SeekHead at Segment start with
fixed-width back-patched SeekPositions for Info/Tracks/Chapters/Cues.

Before this change a 2h26m UHD rip had 52 CuePoints across ~1750
clusters and a 16.5-minute gap between adjacent seek entries; players
seeking inside that gap had to scan from the prior cue. After, one
CuePoint per cluster.

TS (tsmux production path + m2ts_mux): PesFrame.keyframe is plumbed
end-to-end. Codec-private parameter sets are prepended on the first
keyframe (not the first frame); non-key video before any keyframe is
dropped. The first TS packet of a keyframe video PES carries an
adaptation field with random_access_indicator=1. m2ts_mux previously
hardcoded RAI=1 on every PCR packet; that is now gated on the current
PES being a keyframe video PES, combining correctly with PCR when both
land on the same packet.

Adds 17 tests covering keyframe alignment, cue count/position/timing,
SeekHead correctness, RAI set/clear, codec_private gating, non-key drop,
and PCR+RAI combination.
2026-05-17 15:31:16 -07:00
MattJackson b08662f95c Revert "iter14: re-engage FileSectorSource 4 MiB readahead buffer"
This reverts commit 4594196b6e.
2026-05-17 10:50:00 -07:00
MattJackson 4594196b6e iter14: re-engage FileSectorSource 4 MiB readahead buffer
iter13 strace finding: producer thread (mux ISO reader) spends 80%
of wall-clock in state D (NFS RPC wait), doing 207 preads/sec at
103 us each. Pipe is 78 MB/s isolated read; we use ~25 MB/s.

FileSectorSource was using direct per-sector pread (0.21.3 bypass).
That bypass was justified under Phase 2.5 + 0.21.7 producer-poll cap
("32 MiB refill bursts the TCP connection enough to starve the
writer thread"). Both of those conditions are gone now (iter8
baseline: no Phase 2.5, no producer poll cap).

The 4 MiB buffer infrastructure was preserved with #[allow(dead_code)]
in case re-engagement was ever wanted. iter14 just routes the hot
path through buffer_covers + refill + memcpy. Net diff: ~10 LOC of
business logic, 5 unit tests already cover the contract.

Expected: producer's effective read rate jumps from 25 → 60+ MB/s
(matches dd ceiling for 1+ MiB block reads). If consumer side keeps
up, mean mux climbs from 28.7 → 35-50 MB/s. If consumer is now the
cap, we see a clear ceiling around 30-35 and we know where to look
next.

Critically: this change ONLY affects mux-from-ISO. Disc→ISO sweep,
Pass N bad-sector retry, AACS, drive ops, mapfile, recovery — all
untouched (they use DriveSectorSource which is a separate impl).
2026-05-17 10:36:04 -07:00
MattJackson 9d6974df3e iter13 base: lock chunk back to 32 MiB after iter12 revert
iter12 revert restored chunk to 128 (iter11 value) by way of the
revert chain. Setting it back explicitly to 32 for iter13. We're
back to iter8 baseline config (no Phase 2.5, 32 MiB chunks,
DONTNEED, no app-level FileSectorSource buffer, channel depth 32).

iter13 is instrumentation: strace the mux thread during a known-dip
to see where time is actually going.
2026-05-17 09:49:03 -07:00
MattJackson 3394a5b3fe Revert "iter12: fallocate without KEEP_SIZE + truncate_at_sync; chunk back to 32 MiB"
This reverts commit e2c7e20329.
2026-05-17 09:48:35 -07:00
MattJackson e2c7e20329 iter12: fallocate without KEEP_SIZE + truncate_at_sync; chunk back to 32 MiB
iter11 (128 MiB) crashed to 16.6 mean — chunk-size sweet spot is
firmly 32 MiB. Locking in.

iter12 hypothesis: each NFS write past the file's EOF triggers a
server-side SETATTR (metadata commit) to update the file length.
With ~62 GiB of MKV output and writes at chunk boundaries, that's
thousands of SETATTRs per rip. By using plain fallocate (mode 0)
the file is pre-extended to size_bytes up front; subsequent writes
overwrite the pre-extended region with no metadata ops.

Adds high_water tracking + truncate_at_sync so we ftruncate down
to actual content size if the size hint was an overestimate.
2026-05-17 09:35:07 -07:00
MattJackson ff22385a61 iter11: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB
Prediction: ~26-27 mean (regression from iter8). Confirms chunk-size
sweet spot at 32 MiB. Then move off chunk-size as a lever entirely.
2026-05-17 09:21:00 -07:00
MattJackson d39d3933f5 Revert "Reapply "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point""
This reverts commit f7165a2020.
2026-05-17 09:19:44 -07:00
MattJackson 31dc3d35c3 iter10: chunk back to 32 MiB
iter9 (64 MiB on top of no-P2.5) regressed -1.2 from iter8. 32 MiB
remains best on the no-P2.5 path. Net iter10 = iter8 minus DONTNEED
(via the prior commit re-applying f4a881b).
2026-05-17 09:06:34 -07:00
MattJackson f7165a2020 Reapply "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point"
This reverts commit 8ec767339e.
2026-05-17 09:06:14 -07:00
MattJackson d836f8b1e7 iter9: 64 MiB chunks on top of iter8 (Phase 2.5 disabled)
iter8 (Phase 2.5 disabled, 32 MiB chunks) hit 28.7 MB/s mean
— 1.3 MB/s below R2 floor. Without Phase 2.5, WAIT_AFTER blocks the
mux thread directly once per chunk. Halving the WAIT_AFTER frequency
(32 → 64 MiB chunks) should raise the mean without changing the
underlying architecture.
2026-05-17 08:52:51 -07:00
MattJackson 8919de5b03 iter8: chunk back to 32 MiB on top of Phase-2.5 revert
iter7's 64 MiB chunks didn't help (23.1 vs iter4's 24.0). Reverting
to 32 MiB so iter8 differs from iter4 by exactly one variable:
Phase 2.5 disabled / direct passthrough mux thread.

Test question: is Phase 2.5 helping at all? If iter8 mean > iter4,
Phase 2.5 has been a net negative on this rig the whole time. If
iter8 mean < iter4, Phase 2.5 is doing what it claimed.
2026-05-17 08:39:27 -07:00
MattJackson 0dc1f12108 Revert "Reapply "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)""
This reverts commit 335e8b68cc.
2026-05-17 08:39:06 -07:00
MattJackson 6abc25a7e8 iter7: WRITEBACK_CHUNK_BYTES 32 -> 64 MiB
iter6 (8 MiB chunks) regressed mean -8.2 MB/s vs iter4 (32 MiB).
Trend: bigger = better in this workload, against the page-age
theory. Try 64 MiB to see if fewer/larger syncs raise mean further.
0.21.14 went to 128 MiB and was reverted; 64 is between.
2026-05-17 08:25:44 -07:00
MattJackson 4d22cf632f iter6 cleanup: stop tracking (agent dir)worktrees/ 2026-05-17 08:10:29 -07:00
MattJackson 611ad77586 iter6: revert depth bump + WRITEBACK_CHUNK_BYTES 32->8 MiB
iter5 (256-frame channel) regressed -1.8 MB/s vs iter4. Reverting to
32 frames.

iter4 sample pattern shows clear ~30 s oscillation (peak 45 MB/s →
dip 3 MB/s → recovery). Matches Linux vm.dirty_expire_centisecs
default (30 s). 32 MiB chunks at 25 MB/s issue WAIT_AFTER every
~1.3 s, which can't outrun the kernel's own page-age limit, so
pages buildup then flush in bursts. Smaller 8 MiB chunks
(WAIT_AFTER every ~0.33 s) should keep the dirty-page set young
and eliminate the periodic flush-burst dip.
2026-05-17 08:09:57 -07:00
MattJackson 60590193e3 iter5: READ_PIPELINE_DEPTH 32 -> 256 frames
iter4 measurement (Phase 2.5 + DONTNEED, no FileSectorSource buffer)
showed mean 24.0 MB/s but with dips to 2.8 MB/s. Channel at 32 frames
(~1.6 MiB at 50 KB/frame avg) drains in <100 ms on any producer pause,
starving the consumer.

256 frames = ~12 MiB ≈ 3-4 sec of consumer drain at the rolling-avg
floor we want to hit. Should coast through producer micro-stalls
without idle gaps.
2026-05-17 07:53:41 -07:00
MattJackson 5f24843d3c Revert "iter2: restore FileSectorSource readahead buffer (32 MiB)"
This reverts commit ecf9a4d01e.
2026-05-17 07:38:34 -07:00
MattJackson ecf9a4d01e iter2: restore FileSectorSource readahead buffer (32 MiB)
iter1 baseline (Phase 2.5 + DONTNEED restored) measured at 18.4 MB/s
mean on Civil War remux — well below the rig's 37 MB/s concurrent-r+w
ceiling. Per-sector pread is the producer-side bottleneck: ~50us per
pread on NFS = ~19k preads/sec = effective ceiling near what we see.

The 0.21.3 bypass commit cited an A/B test showing the buffer hurt
throughput on NFS bidirectional workloads. That test was taken under
the 0.21.7 producer polling cap; once the cap is gone, the cap was
the bottleneck, not the buffer. Same invalidation pattern as the
0.21.5 Phase 2.5 revert.

Restored the buffered path. DONTNEED page-cache eviction from 0.21.6
stays intact.
2026-05-16 22:46:22 -07:00
MattJackson 8ec767339e Revert "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point"
This reverts commit f4a881b25d.
2026-05-16 17:02:20 -07:00
MattJackson 335e8b68cc Reapply "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)"
This reverts commit 579c94bd23.
2026-05-16 17:02:15 -07:00
MattJackson 1137b4ea1f v0.23.2: bump version 2026-05-16 12:36:28 -07:00
MattJackson 7788c560d1 v0.23.1: bump version 2026-05-16 11:35:41 -07:00
MattJackson 84e0c2ca69 v0.23.0: bump version 2026-05-16 11:17:34 -07:00
MattJackson f4a881b25d io/writeback: medium-agnostic by construction, no DONTNEED, single detection point
Three coupled changes that together meet the "stable max throughput
on any medium" bar:

1. Remove `is_nfs` from `skip_wait`. WAIT_AFTER runs on every medium
   now. The `bounded_syscall` 30s safety net (already in place) covers
   the wedged-FS case generically — no need to predict NFS-hangs at
   compile time. Re-applies the 0.21.12 fix that landed the flat
   25 MB/s on NFS in the first place.

2. Drop the `posix_fadvise(DONTNEED)` calls after WAIT_AFTER. Bounded
   *dirty* pages (the actual invariant) does not require evicting
   *clean* pages. DONTNEED was forcing read-modify-write on any
   in-window seek-then-write — exactly the pattern matroska
   cluster-size backpatches produce. Empirical 2026-05-15: write side
   of mux fed NFS at 43 MB/s while the file grew at 25 MB/s, an 18
   MB/s overhead almost certainly composed of those RMW cycles. The
   kernel reclaims clean pages under LRU when memory is actually
   needed — we don't have to ask.

3. Replace `detect_nfs(fd) -> bool` with `detect_storage_class(fd)
   -> (StorageClass, chunk_bytes_seed)`. Medium detection happens
   *once* at construction and produces *one* output: the initial
   `chunk_bytes` seed for the autotuner. NFS gets 64 MiB (commit ack
   ~10-30 ms; needs bigger chunks to amortize); other media use the
   caller's hint. No hot-path branches on medium. The autotuner drives
   all subsequent decisions from measured WAIT_AFTER p95 latency,
   identically on every (OS, FS) combination.

Module-level doc rewritten to match: no more "NFS escape hatch", no
"is_nfs" framing. The whole writeback module is now medium-agnostic
except for one labelled detection point.
2026-05-16 11:17:30 -07:00
MattJackson 1edda7fedb v0.22.1: bump version 2026-05-16 10:12:34 -07:00
MattJackson 3dd71a0650 v0.22.0: bump version 2026-05-16 09:39:44 -07:00
MattJackson 579c94bd23 Revert "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)"
This reverts commit 2a33364253.
2026-05-16 09:30:44 -07:00
MattJackson 3b2a68fe7a Revert "io/pipeline: bump WRITE_PIPELINE_DEPTH 16 → 32"
This reverts commit d703ce439b.
2026-05-16 09:30:44 -07:00
MattJackson 5b56f112a0 Revert "io/writeback_file: coalesce consecutive Cmd::Writes in the writer thread"
This reverts commit fc13268dc6.
2026-05-16 09:30:44 -07:00
MattJackson f27e4c3088 Revert "io/writeback/linux: drop is_nfs skip — bounded cache works on every medium"
This reverts commit dffee56102.
2026-05-16 09:30:44 -07:00
MattJackson b3576f2ae6 Revert "io/writeback: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB"
This reverts commit 50867516b8.
2026-05-16 09:30:44 -07:00
MattJackson 39f2d0e992 v0.21.14: bump version 2026-05-15 12:02:43 -07:00
MattJackson 50867516b8 io/writeback: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB
The pipeline's adaptive autotuner grows chunk_bytes only when the p95
WAIT_AFTER latency exceeds 200 ms. On NFS, sync_file_range(WAIT_AFTER)
translates to an NFS COMMIT RPC whose ack lands within ~10 ms — so
the autotuner never triggered and the pipeline stayed at the original
32 MiB initial value forever.

That capped sustained mux throughput by paying NFS COMMIT-RPC overhead
roughly once per second of writes. Bidirectional mountstats on rip1
2026-05-15: write side 29 MB/s (RTT 30 ms but exec_time 257 ms — 220 ms
queue/serial waiting) while concurrent dd on the same disk shows
~91 MB/s write + ~65 MB/s read available.

128 MiB initial drops COMMIT cadence 4x while keeping the bounded-
cache invariant intact (worst-case dirty pages ~2 x chunk = 256 MiB,
well under vm.dirty_ratio = 6.6 GB on the 32 GB rig). The adaptive
autotuner can still grow further (up to 256 MiB) or shrink if
WAIT_AFTER ever measures sub-20 ms p95 on faster media.
2026-05-15 12:02:40 -07:00
MattJackson 193e7c1680 v0.21.13: bump version 2026-05-15 10:20:37 -07:00
MattJackson b6463729e8 v0.21.12: bump version 2026-05-15 09:55:41 -07:00
MattJackson dffee56102 io/writeback/linux: drop is_nfs skip — bounded cache works on every medium
The WritebackPipeline's WAIT_AFTER + posix_fadvise(DONTNEED) dance
keeps dirty pages bounded at ~2 × chunk_bytes by waiting for each
chunk's writeback to commit before issuing the DONTNEED hint to drop
it from cache. The original 0.18-era design unconditionally skipped
this on NFS on the premise that "NFS clients have their own buffering
and commit semantics that handle dirty-page bounds without us forcing
the issue."

Empirically wrong. On unraid-1 NFS the kernel client buffers dirty
pages up to vm.dirty_ratio (default 20% of RAM = ~6.6 GB on the rip1
host) before the kernel forces writeback and throttles app writes.
Result on 0.21.11 mux measured 2026-05-15: mux throughput cycled
between ~45 MB/s (cache absorbing) and ~7 MB/s (cache draining under
throttle) on a ~100 s period — exactly the burst-flush pathology this
pipeline was built to fix, but disabled on the medium it actually
runs on. /proc/meminfo Dirty: column climbed lockstep with mux
write rate during the slow half of every cycle, confirming the cause.

The original safety concern — `sync_file_range(WAIT_AFTER)` hanging
indefinitely on a wedged NFS server — is already handled by
`wait_after_with_timeout`'s `bounded_syscall` wrapper (30 s deadline).
If a real WAIT_AFTER call exceeds the deadline the pipeline flips to
the `degraded` state and skips WAIT_AFTER + DONTNEED for the rest of
its life — same effect as the old NFS branch, but only triggered when
something is genuinely broken rather than as a blanket exception.

This change is medium-agnostic: every medium goes through the same
path now, every medium gets the same safety net, and the
ADAPTIVE_WINDOW chunk-size autotuner (lines 226-255 — measures p95 of
WAIT_AFTER and resizes between 4 MiB and 256 MiB) finally activates
on NFS where previously it was dead code. Slow medium auto-grows
chunks to amortise per-chunk overhead; fast medium auto-shrinks to
keep cache pressure tight; nothing in the code special-cases the
filesystem type.

`is_nfs` is still detected (for logging + observability) but no
longer keys `skip_wait`. Module doc + startup log line updated to
match.
2026-05-15 09:55:38 -07:00
MattJackson d8f27cdee0 v0.21.11: bump version 2026-05-14 22:18:33 -07:00
MattJackson fc13268dc6 io/writeback_file: coalesce consecutive Cmd::Writes in the writer thread
Pre-coalescing the Phase-2.5 writer thread issued one `file.write_all`
syscall per `Cmd::Write` dequeued. The mux side calls
`WritebackFile::write_all(buf)` per PES frame, typically 30-200 KB.
On NFS that translates to one RPC per syscall, capping per-thread
throughput at `(wsize / rtt) × inflight` — well below what the same
disk delivers under a 1 MiB `dd oflag=direct` workload (empirical
2026-05-15: dd 71 MB/s vs mux ~25 MB/s sustained, with instantaneous
samples bursting 7→108 MB/s as the kernel page cache filled and
drained on its own cadence).

Coalesce instead: dequeue drains consecutive `Cmd::Write` items off
the ring up to a 1 MiB byte budget, returns them as
`DequeuedWork::Writes(Vec<Vec<u8>>)`, and the run loop concatenates
into one contiguous buffer and issues a single `file.write_all`.
Non-write commands (Seek, Flush, SyncAll, Finish) break the run and
are returned one at a time as `DequeuedWork::Other`, preserving their
ordering relative to the writes.

Single-buffer fast path avoids the concat allocation when only one
write is in the queue at dequeue time. A single oversize write (e.g.
the rare matroska cluster larger than 1 MiB) is admitted alone so it
still makes progress — the kernel splits internally.

This is generic across mediums: bigger app writes are at-least-as-
good on local SSD, HDD, or NFS. On fast storage the ring rarely fills
so coalescing is mostly a no-op; on slow storage with significant
per-RPC overhead it materially improves throughput.

Tests in `tests/` (write_then_drop_persists_bytes, sync_all_drains,
seek_then_patch_roundtrip, flush_is_observed_in_order) still pass —
ordering and durability semantics are unchanged.
2026-05-14 22:18:23 -07:00
MattJackson d9b55f02d0 v0.21.10: bump version 2026-05-14 20:43:21 -07:00
MattJackson 35d66d2059 v0.21.9: bump version 2026-05-14 20:21:36 -07:00
MattJackson d703ce439b io/pipeline: bump WRITE_PIPELINE_DEPTH 16 → 32
The depth was conservative because pre-0.21.8 a full sync_file_range
stall on NFS could traverse this channel and pin the producer. With
0.21.8's restored Phase 2.5 writer thread + 128 MiB byte-bounded ring
inside WritebackFile, every blocking syscall happens downstream of
this channel — never on it. The original "smaller buffer reduces
backpressure risk" rationale no longer applies.

Empirical (2026-05-15 Civil War UHD remux on 0.21.8): 30.9 MB/s
sustained but instantaneous samples spanning 9-53 MB/s, stdev 9.3.
Distribution clusters 52% of samples in 25-35 MB/s but has a long
9-15 MB/s tail. The tail corresponds to brief stalls in the matroska
builder when the sink momentarily lags — exactly the case a deeper
inter-thread channel covers. 32 frames at PES-frame sizes is still
well under a megabyte of additional memory, so the cost is zero.
2026-05-14 20:21:20 -07:00
MattJackson 6112a26d15 v0.21.8: bump version 2026-05-14 18:54:10 -07:00
MattJackson 2a33364253 io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)
Reverts 523af46. That revert was made on the premise that Phase 2.5
caused a ~60% mux throughput regression on NFS bidirectional workloads.
The premise was wrong: at the time of measurement the producer was
capped at ~8 MB/s by a 50 ms thread::sleep poll in
Pipeline::send_with_halt (fixed in v0.21.7's io/pipeline change), so
the comparison was measuring the polling cap on both sides.

With the polling cap removed, direct passthrough exposes the kernel's
default dirty-page writeback pathology on NFS: writes accumulate, the
kernel periodically bursts a flush, app writes block for the burst.
Observed empirically on Civil War UHD remux 2026-05-14: 5-45 MB/s
spiking around a ~21 MB/s sustained mean, dominated by burst-flush
back-pressure cycles.

Phase 2.5 decouples the mux thread from the file syscall:
  * mux writes complete instantly into a 128 MiB byte-bounded SPSC ring,
  * a dedicated writer thread executes the real File writes, seeks,
    and sync_file_range calls; can sit in a kernel burst without
    blocking the mux pipeline,
  * backpressure via Condvar notify/wait, no polling primitive,
  * the ActiveClusterBuffer fast-path preserves the original MKV
    cluster-backpatch optimisation so in-window seeks don't drain
    the current writeback chunk.

Halt-safety is preserved: every blocking writeback syscall on the
writer thread still routes through bounded_syscall with a 60 s
deadline. A wedged NFS server cannot trap the writer indefinitely;
the muxer keeps queueing into the ring; the kernel page cache and
the ring together absorb the stall.
2026-05-14 16:42:00 -07:00
MattJackson fa3872ddcc v0.21.7: bump version 2026-05-14 11:10:57 -07:00
MattJackson c427389f36 io/pipeline: replace polling-send with kernel-wakeup channel
The halt-aware send loop polled try_send on a 50 ms sleep slice when
the channel was full. That capped producer throughput at 1 / 50 ms =
20 frames/sec ≈ 1 MB/s at typical PES frame sizes — way below NFS,
let alone local SSD/NVMe. Multi-day diagnostic 2026-05-13/14 surfaced
it as the root cause of the 18 → 3 MB/s mux throughput regression on
0.21.x.

Replaced std::sync::mpsc::sync_channel with crossbeam_channel::bounded
and switched send_with_halt to send_timeout. Producer now BLOCKS on
consumer drain (kernel-wakeup) instead of polling — the timeout slice
(250 ms) only fires when the producer is waiting for stop signal
observation, never on the happy path.

Result: no throughput cap from this primitive at any medium speed.
Mux is bounded by actual storage / network bandwidth, not by our
channel implementation.

Same fix needs to land in autorip's mux.rs producer-side
sync_channel (separate sibling commit).

Documented in (internal)/memory/
feedback_send_with_halt_poll_throttle.md.
2026-05-14 11:10:39 -07:00
MattJackson 6980f562df v0.21.6: bump version 2026-05-14 09:16:34 -07:00
MattJackson e3b2c9d850 file_sector_source: restore read-side DONTNEED + SEQUENTIAL (the actual fix)
Empirical: isolated NFS read 70 MB/s + write 93 MB/s on the rip1 setup
right now, but mux throughput pinned at 2.7 MB/s on 0.21.5. NOT
environmental — code regression.

Root cause: Phase 1 silently dropped the read-side
posix_fadvise(POSIX_FADV_DONTNEED) eviction that the pre-Phase-1 (0.20.7)
hot path had. Without it, an 85 GB streaming ISO read pins the entire
file in the kernel page cache, starving concurrent MKV writeback. 0.21.2
then also dropped the POSIX_FADV_SEQUENTIAL hint on the same theory,
compounding the regression.

Restored both, per-OS split:
- linux: posix_fadvise(SEQUENTIAL) at open + posix_fadvise(DONTNEED)
  on consumed 32 MiB windows
- macos: F_RDADVISE hint at open (kept); drop_window no-op (macOS unified
  buffer cache less prone to the pin pathology)
- windows / other: both no-op stubs

Target mux speed restored to 20+ MB/s (per concurrent-NFS math:
70/2 read × 0.73 MKV/ISO ratio ≈ 25 MB/s achievable).
2026-05-14 09:16:23 -07:00
MattJackson a383200ef1 v0.21.5: bump version 2026-05-14 01:46:59 -07:00
MattJackson 51d15f551e file_sector_source: allow(dead_code) on residual buffer fields post-0.21.3 bypass 2026-05-14 01:46:51 -07:00
MattJackson 523af461a5 io/writeback_file: revert write path to direct passthrough (0.20.7 baseline)
The Phase 2.5 writer-thread + bounded ring + ActiveClusterBuffer
architecture introduced a ~60% mux throughput regression on NFS
bidirectional workloads (18 MB/s -> 7-8 MB/s in 0.21.x). Other
candidates (FileSectorSource readahead buffer, per-OS sink split)
ruled out empirically across 0.21.1-0.21.4.

Reverts WritebackFile's hot path to direct File passthrough matching
the 0.20.7 baseline:
  * Write/write_all/flush call straight through to self.file.
  * Seek calls through to self.file and notifies the pipeline.
  * sync_all runs pipeline.finalize() then per-OS durable_sync.

Kept (intentional): the per-OS file split established in earlier work
(writeback_file/{linux,macos,windows,other}.rs). create_with_size_hint
still dispatches to platform::preallocate and sync_all still routes
through platform::durable_sync (bounded_syscall 60s deadline on
Linux/macOS), so halt-safety on a wedged NFS fsync is preserved.

Removed:
  * Cmd / RingState / Shared / WriterState / writer_thread_main
  * ActiveClusterBuffer (active-cluster window for in-window seeks)
  * push_command / dequeue / publish_error / mark_writer_gone
  * RING_CAPACITY_BYTES, ACTIVE_CLUSTER_WINDOW_BYTES,
    MAX_WRITE_CHUNK_BYTES, WRITER_THREAD_NAME constants
  * writer JoinHandle field + Drop join logic
  * Tests that asserted internal writer-thread state
    (backpressure_blocks_when_ring_full, in/out_of_window_seek_*,
    active_cluster_buffer_*, writer_thread_panic_surfaces_on_drop)

Kept tests (correctness-of-output only):
  * write_then_drop_persists_bytes
  * sync_all_drains_and_flushes (renamed from sync_all_blocks_until_ring_drains)
  * seek_then_patch_roundtrip (replaces in_window_seek_then_patch_roundtrip)
  * flush_is_observed_in_order

mod.rs: 1026 -> 303 lines. linux/macos/windows/other.rs untouched.
2026-05-14 01:40:37 -07:00
MattJackson 695b65149e v0.21.4: bump version 2026-05-14 00:09:28 -07:00
MattJackson d3f9560689 test/file_sector_source: drop buf-state assertions after 0.21.3 bypass
The three tests (multi_sector_read_spanning_buffer_boundary,
backward_seek_rebuffers, partial_buffer_at_eof) were asserting
internal buf_start_lba / buf_len_sectors state. With 0.21.3's
read-path bypass, those fields are no longer mutated. The
byte-level contract assertions (read returns correct bytes for
every scenario the tests cover) remain intact.
2026-05-14 00:09:21 -07:00
MattJackson 1fa5a7d27f v0.21.3: bump version 2026-05-13 23:57:53 -07:00
MattJackson 55219070f1 io/file_sector_source: bypass app-level buffer — direct pread per call
The 32 MiB readahead window (0.21.0–0.21.1) regressed mux throughput
on NFS bidirectional workloads vs the pre-Phase-1 0.20.7 baseline
(18 -> 7-8 MB/s). The 0.21.2 4 MiB shrink made it worse (5-6 MB/s).
Both signs point at the application-level buffer itself, not the size.

This commit bypasses the buffer entirely on the read path — every
read_sectors call seeks and pread()s direct to the file. That matches
0.20.7's hot path. Kernel readahead handles the policy; on NFS that
interleaves naturally with concurrent writes on the same TCP
connection.

Buffer state fields and refill/buffer_covers are kept so the
structure is preserved for a future per-source-type policy (e.g. a
local-disk source where batched reads ARE beneficial), and so the
existing tests still exercise that machinery.
2026-05-13 23:57:46 -07:00
MattJackson a34c419521 v0.21.2: bump version 2026-05-13 22:28:09 -07:00
MattJackson b179846f5d io/file_sector_source: throttle readahead for NFS bidirectional workloads
Empirical regression observed during 0.21.1 mux test on rip1/unraid-1:
historical 0.20.7 baseline averaged ~18 MB/s mux throughput; 0.21.1
dropped to ~7-8 MB/s flat. Same NFS source + destination, same disc.

Suspect: 32 MiB FileSectorSource readahead + posix_fadvise(SEQUENTIAL)
together saturate the TCP connection on read bursts, starving the
writer thread's concurrent NFS writes (mux reads the source ISO and
writes the MKV over the same connection).

- READAHEAD_BUF_BYTES: 32 MiB -> 4 MiB. Matches NFS rsize=1 MiB * 4
  round-trips per refill, interleaves cleanly with writes.
- linux/hint_sequential: now no-op. Kernel's default ~128 KiB
  readahead is what we want on NFS-backed ISOs (the dominant case).
  Per-OS file stays so we can re-enable a hint cleanly later if a
  different path benefits.
2026-05-13 22:27:57 -07:00
MattJackson e110e80e6e lint: silence clippy::unnecessary_cast on glibc + fix doc list indent
CI's lint workflow runs clippy on linux target where:
- platform/fs_type/linux.rs and io/writeback/linux.rs: the i64 cast
  on buf.f_type / NFS_SUPER_MAGIC is unnecessary on glibc x86_64 (both
  already i64) but required on musl (c_ulong); silence the lint via
  inline allow with explanatory comment.
- mux/m2ts_mux/packet.rs: doc comment continuation across lines was
  parsed as an unindented list item. Reworded to a single flowing
  sentence.
2026-05-13 20:53:26 -07:00
MattJackson 597fa34099 v0.21.1: bump version 2026-05-13 20:30:09 -07:00
MattJackson 52eeb949e3 mux: Rust 1.86 compat + fmt
- m2ts_mux: replace u64::is_multiple_of (stable in 1.87+) with %.
  CI on the pinned 1.86 toolchain rejected the unstable feature use.
- mux/{fmp4,hevc,m2ts_mux}: rustfmt drift cleanup (test-code wrapping).
2026-05-13 20:30:01 -07:00
MattJackson 6ca62b8411 v0.21.0: bump version 2026-05-13 20:17:47 -07:00
MattJackson 04195c27d8 io+mux: phase 3 — streaming sinks + sequential container muxers
SocketSink + UdpSocketSink (`src/io/sink/socket.rs`) — sequential-only
TCP/UDP write destinations. SocketSink wraps BufWriter<TcpStream> with
1 MiB capacity, tunes SO_SNDBUF on construction, calls shutdown(Write)
on finish(). UdpSocketSink emits one datagram per write — caller
packetizes. Both impl Write+Send and thus satisfy SequentialSink via
the Phase 2 blanket; neither impls Seek, so RandomAccessSink is
correctly inaccessible (compile error to mux MKV onto a socket).

New sequential container muxers in src/mux/:

  - hevc/ — raw HEVC Annex B elementary stream. Length-prefixed NALU
    → 00 00 00 01 NALU. hvcC parsing emits VPS/SPS/PPS once at stream
    head. Fully ships.

  - m2ts_mux/ — standard MPEG-TS (188-byte packets). Single program,
    HEVC video on PID 0x100, optional AC3/TrueHD audio on PID 0x101.
    PAT+PMT re-emitted every 250 packets; PCR stamped on video every
    40 packets. Hand-rolled, no new deps. Distinct from the existing
    BD-TS (192-byte) `mux::m2ts::M2tsStream` — that path stays as-is.

  - fmp4/ — fragmented MP4. STUB: ftyp + minimal moov skeleton with
    one HEVC video trak + mvex/trex. Media fragments (moof+mdat) are
    TODO for v0.22.0 — write_video accumulates frames into a pending
    buffer that finish() clears. Init segment is well-formed enough
    that init_segment_starts_with_ftyp_then_moov asserts the box
    chain.

17 new unit tests added (socket round-trip, HEVC Annex B conversion,
M2TS packet alignment + PAT/PMT cadence + per-PID CC, fMP4 box chain).
All 514 lib tests + 17 new = pass on Rust 1.86 (fmt + clippy + test
via (internal)/scripts/precommit.sh libfreemkv).

No new dependencies. No version bump. Don't-touch list clean.
2026-05-13 20:17:33 -07:00
MattJackson 5a8f8e54e1 io: phase 2.5 — writer thread + bounded ring + active-cluster buffer
WritebackFile now offloads all File I/O to a dedicated writer thread.
Muxer's Write/Seek/sync_all push into a bounded byte ring; the writer
drains the ring and runs the syscalls.

- Commands: Write(Vec<u8>), Seek(SeekFrom), SyncAll(oneshot),
  Finish(oneshot). One ring carries all four; ordering preserved.
- ActiveClusterBuffer mirrors the last ACTIVE_CLUSTER_WINDOW_BYTES
  written. Seek-back within window → in-memory patch + re-emit
  (no forced drain). Seek-back outside window → real drain+seek.
  Wins for MKV: cluster size patches almost always land inside the
  active window; only the end-of-mux Cues + segment header backpatch
  fall outside.
- Ring capacity RING_CAPACITY_BYTES = 128 MiB. Backpressure on full
  blocks the muxer (correct semantics for archival workflows).
- All syscalls in writer thread wrapped in bounded_syscall(60s) so
  a wedged NFS doesn't pin the thread forever.
- Drop drains via Finish + joins the writer thread; sync_all blocks
  until ring drained AND underlying fsync completes.
2026-05-13 20:17:29 -07:00
MattJackson 1a2c830cf6 fmt: rustfmt reorder of cfg-gated use/mod decls in sink + fs_type 2026-05-13 20:10:57 -07:00
MattJackson 637c2bfe36 v0.20.10: bump version 2026-05-13 19:59:07 -07:00
MattJackson 0d28357b4d io+platform: phase 2 — sink trait split + fs_type detection
Introduces the SequentialSink / RandomAccessSink trait pair under
io::sink and an open_for_mkv dispatch helper that picks WritebackFile
on Linux+NFS and LocalFileSink everywhere else. LocalFileSink wraps
BufWriter<File> with a 4 MiB buffer and exposes a per-OS preallocate
path (fallocate on Linux, F_PREALLOCATE on macOS, no-op fallback).

Adds platform::fs_type::detect with a per-OS split (statfs on Linux /
macOS, UNC heuristic on Windows, Unknown elsewhere) so construction-
site dispatch has a single primitive to call.

Blanket impls cover the common shapes: any Write+Send is a
SequentialSink, and any SequentialSink+Seek is a RandomAccessSink.
WritebackFile satisfies the random-access trait via the blanket impl
without needing an explicit per-type impl. No callers wired yet — the
mux::resolve construction sites stay on WritebackFile pending Phase 3.

Tests: 5 new sink/preallocate tests + 3 fs_type tests (1 ignored,
needs a real NFS mount). cargo +1.86 fmt + clippy + tests all green.
2026-05-13 19:58:54 -07:00
MattJackson 5495332f07 v0.20.9: bump version 2026-05-13 19:52:48 -07:00
MattJackson e22fc6fd47 io: phase 1 buffering — read-side flatness
Three changes targeting 0.20.9's "muxer never read-stalls on NFS read
latency" invariant:

A. FileSectorSource gets a 32 MiB internal read-ahead buffer
   (READAHEAD_BUF_BYTES). Splits out from src/sector/file.rs into
   src/io/file_sector_source/ with per-OS open hints (Linux
   posix_fadvise(SEQUENTIAL), macOS fcntl(F_RDADVISE) with 64 MiB
   cap, Windows TODO stub, BSD/illumos no-op). Backward seeks
   rebuffer; partial reads at EOF return only the bytes that exist;
   oversize-request bypass for count > BUF_SECTORS.

B. WritebackFile inline #[cfg(target_os = "linux")] blocks split
   into per-OS files under src/io/writeback_file/. Linux unchanged
   (fallocate KEEP_SIZE, fsync via bounded_syscall). macOS gets a
   real F_PREALLOCATE + F_FULLFSYNC impl (was a "skipped (non-linux)"
   debug log before). Windows is a stub (FlushFileBuffers via
   std sync_all; TODO for SetFileValidData). BSDs/illumos fall back
   to std sync_all.

C. New byte_channel module — byte-bounded producer/consumer wrapping
   std sync_channel with Mutex/Condvar byte accounting. Sender blocks
   when used_bytes + item.byte_size() > capacity. HasByteSize impl
   for PesFrame. Default cap BYTE_CHANNEL_DEFAULT_CAPACITY = 64 MiB,
   sized to absorb worst-case NFS read p99 (~2 s × UHD peak compressed
   ~15 MB/s). The mux call site lives in autorip (out of scope here);
   this lands the primitive in libfreemkv for autorip to adopt.

Test counts: byte_channel +6, file_sector_source +5, sector::file
round-trip suite (3) preserved. passn_handler_ab.rs A/B fixture
(8 profiles) still green.

precommit.sh libfreemkv: fmt + clippy + test all green on Rust 1.86.

No version bump; no Cargo.lock changes; no forbidden-file edits
(disc/patch.rs, disc/read_error.rs, io/pipeline.rs,
tests/passn_handler_ab.rs).
2026-05-13 19:48:23 -07:00
MattJackson 5b98c13e47 v0.20.8: bump version 2026-05-13 19:18:51 -07:00
MattJackson 8f8f1a62a2 io+disc: bundle 0.20.8 dev work
- io/pipeline.rs: add send_with_halt + finish_with_halt for cooperative
  halt during blocking producer-consumer handoffs; 5 new tests
- disc/patch.rs: split Disc::patch body (1168 -> 316 LOC) into named
  helpers (compute_initial_state, prime_cache, check_range_watchdog,
  handle_skip_limit, compute_damage_skip, handle_read_success,
  handle_read_failure, report_patch_progress, build_outcome) with
  PatchLoopState / RangeFrame structs; references shared
  PATCH_DAMAGE_THRESHOLD_PCT constant
- disc/read_error.rs: add pub const PATCH_DAMAGE_THRESHOLD_PCT = 6;
  ReadCtx::for_patch() now references the shared constant (was a
  latent 12 / 6 inconsistency)
- tests/passn_handler_ab.rs: 8-profile A/B fixture locking current
  patch-side recovery behavior (clean / all-medium / alternating /
  edge-bad-good-middle / single-bad / deep-pit / medium-then-good /
  batch-fail). Goldens captured pre-unification; will catch any
  future refactor that breaks the size-aware skip cap.
2026-05-13 19:15:48 -07:00
MattJackson e3cfd27942 v0.20.7: version bump for unified release (no source changes)
The 0.20.7 work lives downstream in autorip — process-level safety net
(hard watchdog 5-min mux escalation + std::process::exit, restart-loop
counter + auto-quarantine after 3 attempts, partial-state preservation,
failure_reason surfaced in /api/state).

Bumping libfreemkv to 0.20.7 keeps the 4 crates on the unified release
train per project docs.
2026-05-13 14:29:40 -07:00
MattJackson aa12bdad62 v0.20.6: io::bounded — halt-safe wrapper for blocking syscalls
Generalizes 0.20.5's hand-written wait_after_with_timeout into a
reusable primitive. After this change, every blocking syscall in the
recovery + mux paths is wrapped, so cooperative Halt has bounded
~250 ms latency reach even into kernel-owned thread states.

New module src/io/bounded.rs:
- BoundedError { Halted, Timeout, WorkerLost }
- bounded_syscall<F, R>(halt: Option<&Halt>, timeout, op) -> Result<R, BoundedError>
- Worker thread runs op; main thread recv_timeouts on a rendezvous
  channel in 250 ms slices, polling halt between slices.
- Worker is intentionally leaked on timeout/halt — kernel reaps when
  the syscall finally returns or at process exit. Calling thread is
  NEVER trapped inside a kernel call.
- 6 unit tests cover the happy path + each error variant.

Refactored callsites:
- src/io/writeback/linux.rs::wait_after_with_timeout now delegates
  to bounded_syscall. ~30 LOC of duplicated channel/thread plumbing
  deleted. Same semantics, cleaner.
- src/io/writeback_file.rs::WritebackFile::sync_all now wraps the
  final libc::fsync(fd) with bounded_syscall (60 s deadline). On
  timeout: log error at target=mux and return Ok — kernel will flush
  on close, best-effort but bounded. Covers FileSectorSink::finish,
  PatchSink::close, SweepSink::close, and the mux MKV finalize path
  (they all sync through WritebackFile).

What still hangs (deliberately not wrapped — too hot a path):
- File::write itself. Per-frame write on a wedged NFS could still
  block; but back-pressure from a stuck consumer means the producer
  notices within seconds, not minutes — different failure mode than
  the WAIT_AFTER hang 0.20.5/0.20.6 fix.
2026-05-13 14:22:24 -07:00
MattJackson ef3895cdc5 v0.20.5: NFS-aware writeback + bounded sync_file_range timeout
Targets the recurring mux hang on NFS dest where the consumer thread
sits indefinitely inside libc::sync_file_range(SYNC_FILE_RANGE_WAIT_AFTER)
because the NFS server never returns a commit ack. The whole rip
wedges; halt is cooperative and can't reach inside a kernel syscall.

A. NFS detection at WritebackPipeline construction (fstatfs f_type ==
NFS_SUPER_MAGIC 0x6969). When NFS:
- Skip SYNC_FILE_RANGE_WAIT_AFTER entirely.
- Skip posix_fadvise(DONTNEED) — NFS client handles its own buffering.
- Still issue async SYNC_FILE_RANGE_WRITE (harmless hint).
Cannot hang on a syscall not made. fstatfs failure fails open (assume
local). Logged at info on construction so operators see which strategy
is active. The whole hang vector is removed for NFS deployments.

B. Hard timeout on WAIT_AFTER for non-NFS (defense in depth, since
even a degraded local disk could in principle hang the syscall).
Each WAIT_AFTER runs on a worker thread; main thread waits on a
sync_channel rendezvous with 30s deadline. On timeout: log error,
set per-pipeline 'degraded' Arc<AtomicBool>, downgrade to NFS-style
skip for the rest of the pipeline's life. Worker thread leaks
intentionally — it'll unwind when the syscall eventually returns or
the process exits. Converts indefinite freeze into 'log loud +
downgrade + keep ripping'.

C. Diagnostic logging for the 73%-of-this-movie reproduction:
- WritebackFile::seek logs every non-trivial seek (from, to, signed
  delta) at target=mux so we can see if MkvMuxer seeks back before
  a stall.
- WritebackPipeline::finalize logs the chunk being finalised before
  any WAIT_AFTER call, so a hung chunk is identifiable by offset.

No new dependencies. macOS / Windows noop stubs unchanged. Net
+198 LOC libfreemkv (mostly writeback/linux.rs).
2026-05-13 14:09:55 -07:00
MattJackson 2dcf969ac8 v0.20.4: mux performance + observability — universal across storage
Four targeted changes to maximize mux throughput regardless of storage
backend (local SSD, local HDD, NFS, network share) and surface enough
log data to diagnose 'mux slow' reports without a re-rip:

1. POSIX_FADV_SEQUENTIAL on FileSectorSource::open (Linux only).
   Widens the kernel readahead window for sequential ISO reads. One
   syscall at open, free on every storage type.

2. POSIX_FADV_DONTNEED on the ISO read side after every 32 MiB chunk.
   Mirrors the writeback DONTNEED that already runs on the write
   side. Keeps the read-side page cache bounded during multi-GB ISO
   reads — eliminates the OOM-pressure / eviction-storm risk on
   long mux runs. Linux only; per-drop trace at target="mux".

3. WritebackFile::create_with_size_hint(path, size_bytes) calls
   fallocate(FALLOC_FL_KEEP_SIZE) on Linux to pre-reserve extents
   for the output. Reported file size stays 0 (writes grow it
   naturally) but the on-disk extent allocation is contiguous —
   reduces extent fragmentation for big sequential muxes. Wired
   into mkv:// and m2ts:// output paths via DiscTitle::size_bytes.
   No-op on macOS/Windows; old create() kept with #[allow(dead_code)]
   for callers without a size hint.

4. Adaptive WRITEBACK_CHUNK_BYTES in the Linux writeback pipeline.
   Tracks sync_file_range(WAIT_AFTER) elapsed_ms in a rolling
   16-sample window. p95 > 200 ms → double chunk size (cap 256 MiB).
   p95 < 20 ms → halve (floor 4 MiB). One algorithm, both
   fast-storage (small chunks, responsive) and slow-storage (big
   chunks, fewer commit round-trips) optimized. Per-chunk trace +
   per-32-chunk debug snapshot + info-on-resize so an operator can
   see where the autoscaler settled.

All four are universal — no storage-type detection, no env vars to
flip, no per-deploy tuning required. Total +201/-6 across four files.
2026-05-13 13:50:44 -07:00
MattJackson 005f887bf9 v0.20.3: add halt check to Disc::patch backtrack inner loop
WO-5 (partial): the patch backtrack inner loop ('while bt_pos <
backtrack_end' in disc/patch.rs) issues per-sector reads to fill the
gap created by a damage-window skip. A long backtrack span can run
minutes; without an inline halt poll, the outer halt only takes
effect when control returns to the per-range loop. Adds a halt poll
at the top of each iteration so cancellation propagates inside the
backtrack span.

Per-sector read failures inside the backtrack already drop through
to the main fail path; this only changes the cancellation latency
between an /api/stop call and the producer actually unwinding. Drops
worst-case unwind from 'whole backtrack span × per-sector recovery
timeout' (minutes) to 'one in-flight SCSI command' (seconds).

The broader Arc<AtomicBool> → Halt migration on CopyOptions /
SweepOptions / PatchOptions / Drive::halt and Pipeline::send halt-
awareness is deferred — separate cycle, larger API impact.
2026-05-13 11:55:52 -07:00
MattJackson d7243a6044 v0.20.2: delete dead retry block + wedge-family cooldown in Disc::patch
WO-3c: Delete dead non-NOT_READY retry block (~100 LOC). The block
declared retry_count = 0 inside the per-iteration Err arm, so the
'MAX_NON_NOT_READY_RETRIES=3' budget actually fired exactly once
(1s pause + 1 retry) before falling through to NonTrimmed. The
'exponential backoff: 2s, 4s, 8s' comment was wrong by construction.
Cross-pass NonTrimmed retry (each pass gives the same sectors another
shot) already covers the recovery case, and gives the drive minutes
between attempts instead of 1-8 seconds — empirically what stochastic
recovery on the BU40N actually needs.

WO-4 (targeted slice): Add wedge-family cooldown on HARDWARE_ERROR /
ILLEGAL_REQUEST senses. These are what the BU40N's firmware fast-fail
state returns; every subsequent read in that state comes back in
<100ms. Pre-fix patch hammered the drive: mark NonTrimmed, sleep 1s,
advance, hit next wedge, mark, sleep 1s — exactly the rapid-retry
cadence the firmware is sensitive to. Now a wedge-family sense triggers
WEDGE_FAMILY_COOLDOWN_SECS=30 cooldown (matches read_error.rs's
ZONE_ENTRY_COOLDOWN_SECS), and WEDGE_ABORT_THRESHOLD=16 consecutive
wedges aborts the pass for autorip eject+reload. Any non-wedge read
clears the counter.

Also drops the duplicate NonTrimmed dispatch (Mapfile::record is
idempotent so it wasn't a correctness bug, but it doubled per-failure
consumer work).
2026-05-13 11:53:57 -07:00
MattJackson f1926c38dc v0.20.1: delete SectorReader, extract Disc::patch, doc/stub cleanup
WO-2 (delete SectorReader trait):
- The 0.18 trait split into SectorSource (read-only) and SectorSink
  (write-only) is final; the legacy SectorReader alias was a bridge.
- Renames every internal &mut dyn SectorReader (~25 sites) to
  &mut dyn SectorSource. The trait method capacity() becomes
  capacity_sectors() with a default of 0 (preserves SectorReader's
  default-0 behavior).
- Deletes the SectorReader trait, its blanket-to-Source bridge, and
  the FileSectorReader type alias. Adds explicit forwarding impls
  for Box<dyn SectorSource> and &mut dyn SectorSource so generic
  decorators like DecryptingSectorSource<S: SectorSource> compose.

WO-3a (extract Disc::patch):
- Moves Disc::patch (1230 lines) and bytes_bad_in_title from
  disc/mod.rs into disc/patch.rs as a split inherent impl. Zero
  behavior change — pure mechanical relocation. disc/mod.rs drops
  from 3,945 to 2,714 LOC.

WO-6 (partial):
- Deletes src/labels/png_filenames.rs — was a 72-LOC stub with
  detect() returning false, never wired into the PARSERS registry.

project docs doc drift fixes (audited 2026-05-13):
- JUMP_BASE_SECTORS: 256→1024 (64 MB base for UHD, not 8 MB)
- PASSN_DAMAGE_THRESHOLD_PCT: 12→6
- PASSN_SKIP_SECTORS_BASE: 64→32
- MAX_RANGE_SECS=180: replaced by proportional range_sectors × 25,
  capped at RANGE_BUDGET_CAP_SECS=1800.
2026-05-13 11:36:55 -07:00
MattJackson 4709a73c80 v0.20.0: delete FrameSource/FrameSink, keep single Stream trait
The 0.18 trait split into FrameSource (read-only) and FrameSink
(write-only) was an over-engineered API. Consumers don't think
"frame source backed by MKV" — they think "open MKV for reading".
The split paid a real API-complexity cost (two trait names, two
re-exports, dual impls per bidirectional type, deprecation bridge)
for one marginal property: compile-time direction-safety at the
trait-object boundary. The runtime error path on a wrong-direction
call (StreamReadOnly / StreamWriteOnly) is unambiguous and rare in
practice.

Deletions:
- pes::Stream is no longer #[deprecated]
- pes::FrameSource trait + its blanket-from-Stream bridge
- pes::FrameSink trait + the trampoline impls on every concrete type
- The compile-time-direction-safety test scaffolding
- Crate-root FrameSource / FrameSink re-exports

Additions:
- Stream is now Send-bounded (Stream: Send supertrait). Every
  concrete impl was already Send-compliant — Box<dyn Read + Send>
  and Box<dyn Write + Send> were already in place on the trait
  objects MkvStream / M2tsStream / etc hold internally. Promoting
  Send into the trait makes Box<dyn Stream> Send too, which lets
  autorip drop its SendStream unsafe newtype.

The public API is now: one Stream trait, one concrete type per
format, two constructors (open/create or input/output). Bidirectional
types route through internal Mode { Read | Write } discriminants.

Net: -347 lines libfreemkv, -38 lines autorip, -5 lines freemkv.
2026-05-13 08:42:14 -07:00
MattJackson 55bd1ee868 v0.19.1: repair Cargo.toml after botched 0.19.0 bump
v0.19.0 was tagged with a search-and-replace gone wrong:
rust-version, serde, and zip all had their version strings
replaced with "0.19.0". Edition 2024 rejected rust-version
0.19.0 (< 1.85), failing every CI build. No artifacts shipped
to crates.io.

Repair:
- rust-version: 0.19.0 → 1.86 (CI pin)
- serde: 0.19.0 → 1
- zip: 0.19.0 → 2

Also drops an unused start_lba binding in mux/disc.rs that
clippy 1.86 catches.
2026-05-12 21:29:07 -07:00
MattJackson 5285e8b6ec v0.19.0: bump version 2026-05-12 19:24:06 -07:00
MattJackson c5af61e46b v0.18.25: bump version 2026-05-12 16:27:28 -07:00
MattJackson ed73c1ab68 v0.18.25: add mux debug logging for reader type and stall detection 2026-05-12 16:27:16 -07:00
MattJackson 5ce2df0888 v0.18.24: bump version 2026-05-12 15:46:47 -07:00
MattJackson ea591a36a5 v0.18.23: bump version 2026-05-12 10:59:45 -07:00
MattJackson 368e10486e v0.18.22: bump version 2026-05-12 09:33:00 -07:00
MattJackson a99c4f8487 v0.18.21: format code 2026-05-12 09:18:52 -07:00
MattJackson 4df470e572 v0.18.21: fix borrow checker, make io public 2026-05-12 09:01:52 -07:00
MattJackson a0b13941fe v0.18.21: fix borrow checker error in debug logging 2026-05-12 08:49:33 -07:00
MattJackson 275c9da009 v0.18.21: verbose debug logging for pipeline stalls (fixed) 2026-05-12 08:37:42 -07:00
MattJackson da183722fa v0.18.20: bump version 2026-05-11 22:33:15 -07:00
MattJackson 4a5424eabe v0.18.20: bump version 2026-05-11 22:28:57 -07:00
MattJackson 293d89c586 v0.18.20: separate read/write pipeline depths 2026-05-11 22:18:56 -07:00
MattJackson 4840f5134d libfreemkv: increase DEFAULT_PIPELINE_DEPTH to 32 for smoother mux speed 2026-05-11 22:01:27 -07:00
MattJackson 11c8211605 v0.18.18: bump version 2026-05-11 20:16:30 -07:00
MattJackson 1038beaf06 v0.18.17: bump version 2026-05-11 19:58:36 -07:00
MattJackson f1df57196c libfreemkv: add Clone derive to MuxAtomics 2026-05-11 19:54:50 -07:00
MattJackson 8ba9dc1c0b v0.18.16: bump version 2026-05-11 15:57:03 -07:00
MattJackson 49b131f39a add debug logging for MKB processing 2026-05-11 15:53:11 -07:00
MattJackson 8238ec4ce7 optical vs block batch sizing 2026-05-11 15:35:45 -07:00
MattJackson 4fd5d27df8 v0.18.15: bump version 2026-05-11 11:30:19 -07:00
MattJackson 1fbe272832 disc/scan: surface AACS resolution error on Disc.aacs_error
scan_with() collapsed every failure path from resolve_encryption() into
None via .ok(), so callers couldn't tell the difference between "no
KEYDB found", "KEYDB failed to parse", "disc hash not in KEYDB and
fallback derivation failed", "AACS files unreadable on disc", and a
handshake that rejected every host cert. autorip's UI was stuck
printing "no decryption keys found (check KEYDB)" for all of them,
which is a particularly bad message when the user has actually loaded
a KEYDB and the real failure is something else.

Changes:
- New pub field Disc.aacs_error: Option<Error>. Populated by scan_with
  whenever encrypted && aacs.is_none(). Sentinel KeydbLoad path
  "<no keydb in search paths>" distinguishes the no-keydb case from
  a real load failure without adding a new Error variant (which would
  be a breaking change for downstream exhaustive matches).
- tracing::warn in scan_with at scan_aacs_resolve_failed and
  scan_aacs_no_keydb, with error_code and keydb path for grepping.
- tracing in do_handshake: keydb load failure, host-cert exhaustion
  (with cert count and last error code), VID read failure post-auth,
  and a debug-level success log. Lets us see whether handshake even
  got off the ground for a given disc.

Test fixtures updated to set aacs_error: None.
2026-05-11 11:29:48 -07:00
MattJackson 6bd635725e v0.18.14: bump version 2026-05-11 08:27:10 -07:00
MattJackson 09b77b4dea disc/patch: relabel "possible wedge" heuristic log to avoid confusion
The 'All probes failed — possible wedge condition' log fired during patch
probing whenever 10+ consecutive failures hit AND a probe sweep at the
local zone returned 0 successes. This was distinct from the read_error.rs
'wedge_transition' log that fires when the SCSI sense family ACTUALLY
flips into Hardware/IllegalRequest fast-fail mode.

Two logs both saying 'wedge' caused operator confusion during the
2026-05-11 Dune Pt 2 wedge investigation — was the drive wedged, or was
it just a zone of fully-bad sectors? They mean different things.

Relabel to 'patch_zone_fully_bad' with explicit pointer to read_error.rs
for the canonical wedge detection. Same triggering condition; just clearer
wording in the log stream.
2026-05-10 22:20:44 -07:00
MattJackson 688058b3e8 labels/bdmt: drop description fields that are just XML child elements
Disc-04 (Top Gun: Maverick) re-test 2026-05-11 surfaced a real-world
bdmt_eng.xml where <di:description> contained no prose, only nested
<di:thumbnail href="…"/> elements. The previous parser surfaced
the raw XML fragment as the description string ("<di:thumbnail
href=\"tgm_meta_sm.jpg\" />\\r\\n      <di:thumbnail
href=\"tgm_meta_lg.jpg\" />"). Worse than no description.

Fix: filter description candidates that begin with `<` after
trimming. Real prose never starts with an angle bracket; XML-only
content always does. Net: title extraction unaffected (it uses its
own element-priority path); description field drops when it would
otherwise carry XML noise.

Two new bdmt tests, 12 of 12 passing.
2026-05-10 22:12:59 -07:00
MattJackson c96bac7977 labels: append CLPI orphan streams after gap-fill
Three layered sources of stream labels now, in precedence order:
1. **Framework parser** (paramount/criterion/pixelogic/ctrm/dbp/deluxe)
   — editorial labels with purpose/qualifier ("English Atmos",
   "Director's Commentary", "English SDH"). High or Medium confidence.
2. **MPLS gap-fill** (`fill_gaps_from_mpls`) — every stream the
   playlist references gets at least a basic lang+codec label, even
   when the framework parser missed it.
3. **CLPI orphan append** (`append_clpi_orphans`) — streams in
   /BDMV/CLIPINF/*.clpi ProgramInfo that NO MPLS playlist references.
   Empirical (2026-05-11): ~5% of streams across the 11-disc corpus,
   most dramatic on disc-02 (HDMV-only) at 40% CLPI-only.

Orphan numbering: each appended orphan gets
`stream_number = max(existing per type) + N` so playlist-reachable
streams keep their original positions and orphans sort cleanly at
the tail.

Orphan dedup: (stream_type, language, codec_hint) tuple — fuzzier
than PID matching (PIDs aren't carried on StreamLabel) but it's the
only signal available downstream of the gap-fill. False positives
(genuine orphan that happens to share lang+codec with an existing
entry) silently drop, which is the conservative failure mode — the
user-facing display would just see a confusing duplicate otherwise.

`mpls_universal::language_display_name` and `::codec_name` promoted
from private fn to pub(crate) so this module can build orphan labels
with consistent naming.

Tests: 2 new in gap_fill_tests — synthetic-input verification of the
dedup tuple logic and the stream_number assignment. 6/6 tests in the
gap-fill module now passing.
2026-05-10 22:08:14 -07:00
MattJackson c32acff3e4 disc/read_error: rustfmt fixup for wedge-prevention commit 2026-05-10 22:00:08 -07:00
MattJackson b58e2d9873 disc/read_error: wedge PREVENTION — jump on first error + 30s cooldown
Rewrites the Pass 1 wedge handling from "slow skip after the drive
has already wedged" to "prevent the wedge transition in the first
place." Driven by 2026-05-11 empirical data: the BU40N transitioned
into IllegalRequest fast-fail mode at exactly 7 medium errors in
6.5 seconds (~1 read/sec retry cadence). Once there, only physical
eject + reload clears it — 30s pauses + 1 GB jumps do not.

The fix is the user's mental model from that session:

  "We can detect bad reads, failed reads, and asking to read again
   fast after causes a wedge. We need to prevent the wedge in the
   first place."

Two changes to the centralized error handler:

1. **`for_sweep().fast_jump_threshold = 1`** (was 4). Pass 1 now
   JumpAheads on the FIRST outer-batch failure, not the 4th. The
   drive never gets back-to-back retries at the same LBA in Pass 1
   — every error → jump 64 MB forward + long cooldown. Pass N keeps
   `fast_jump_threshold = u64::MAX` because retries on already-known-
   bad LBAs are its whole job.

2. **`ZONE_ENTRY_COOLDOWN_SECS = 30`**. The FIRST error after a
   clean run (when `consecutive_outer_failures == 1` and we're not
   bisecting) uses this long pause instead of the standard 5 s
   FAIL_PAUSE_SECS. Gives the BU40N's firmware / bridge internal
   retry counters 30 s of breathing room before the next read,
   preventing the "7 errors in 6.5 s" cascade. Subsequent errors
   in the same zone use the standard 5 s pause (we've already
   jumped past the initial damage; further errors mean we landed
   in another bad cluster).

Pass N exempt from the zone-entry cooldown — `bisect_on_marginal=
true` skips the long-pause arm. Pass N's per-sector retries on
known-bad LBAs would multiply uselessly with 30 s/error.

Test updates: 4 tests' expected behavior changed under the new
policy. Renamed `pass_1_marginal_skips_instead_of_bisecting` →
`pass_1_marginal_jumps_immediately_not_bisecting`. Renamed
`pass_1_jumps_after_4_consecutive_outer_failures` →
`pass_1_jumps_immediately_on_first_outer_failure`. Updated
`both_passes_pause_on_failed_read_for_wedge_avoidance` (now
`pass_1_zone_entry_uses_long_cooldown` + `pass_n_pauses_uniformly_on_failed_read`).

Cost analysis:
- Clean disc (no errors): unchanged. 0% overhead.
- Lightly damaged (1-2 zones): +30 s per zone = ~1 min total. Fine.
- Heavily damaged (10+ zones): +5+ min total. The trade for never
  wedging the drive and getting a usable Pass N afterwards.

Expected behavior on the next damaged-disc rip:
- Pass 1 hits damage at LBA X → jumps 64 MB forward immediately,
  pauses 30 s
- Drive's firmware never accumulates the retry pressure that triggers
  IllegalRequest fast-fail
- bytes_maybe accumulates faster (we skip more), but Pass N picks up
  the slack with proper per-sector recovery — and Pass N can actually
  RUN because the drive isn't wedged
2026-05-10 21:56:27 -07:00
MattJackson a9e802c1c2 clpi+labels: extract program_info stream table + CLPI vs MPLS audit
Two layered changes, in service of the empirical question "is CLPI
truly redundant with MPLS for label data?":

1. **clpi.rs ProgramInfo parser**. The existing CLPI parser only
   walked the EP map (for sector-range lookups). Added a parser for
   the ProgramInfo section's per-stream stream_coding_info table:
   pid, coding_type, audio_format/rate, video_format/rate, ISO 639-2
   language. Spec layout per libbluray clpi_parse.c. Best-effort —
   malformed program_info leaves `streams: vec![]`, EP map keeps
   working. `ClipInfo` gains a `streams: Vec<ClpiStream>` field.

2. **labels/clpi_audit.rs**. Diagnostic that walks both
   `/BDMV/CLIPINF/*.clpi` (via the new program_info parser) and
   `/BDMV/PLAYLIST/*.mpls`, builds a (PID → fields) merged view, and
   classifies each row:
   - `Match`: both sources agree (same coding_type + language)
   - `ClpiOnly`: PID in CLPI but no MPLS playlist references it
     (orphan stream on disc — reachable via low-level access, not via menu)
   - `MplsOnly`: PID in MPLS but no CLPI lists it (would indicate a
     parser bug; verified empirically that this NEVER happens)
   - `Divergent`: same PID, different coding_type or language between
     sources (playlist re-tagged or attribute encoding mismatch)
   Surfaced via `labels-analyze` as `clpi_vs_mpls_audit: {matches,
   clpi_only, mpls_only, divergent, total_pids}`. Doesn't affect the
   label output — pure diagnostic.

Empirical findings on the 11-disc corpus (excl. disc-04 truncated):
- 226 matches / 0 mpls_only / 8 clpi_only / 5 divergent across 239 PIDs
- 6 of 10 non-truncated discs have CLPI-only streams (orphans)
- disc-02 (HDMV-only) is the most dramatic: 40% of its 5 streams are
  CLPI-only — MPLS sees 3, CLPI sees 5
- Conclusion: CLPI is NOT truly redundant. ~5% of streams disc-wide
  are CLPI-exclusive. Future work: layer CLPI as a tertiary source
  below MPLS in the labels pipeline (orphan streams marked with even
  lower confidence than MPLS).
2026-05-10 21:50:39 -07:00
MattJackson a876ce846b labels: surface MPLS chapter summary in LabelAnalysis
LabelAnalysis gains `chapter_summary: Vec<ChapterSummary>` — one row
per .mpls file in /BDMV/PLAYLIST/, with chapter count (PlaylistMark
entries with mark_type ≤ 1) and approximate playlist duration in
seconds. Sorted by playlist filename.

Sourced from the existing crate::mpls parser (no new format work).
Useful for identifying the main feature playlist at a glance — it's
the one with the longest duration. Verified on disc-11 (Dune Pt 2):
00800.mpls correctly identified as 2h 45m 49s with 18 chapters
amid 30+ shorter playlists.

Doesn't touch the per-title `disc::DiscTitle::chapters` field which
disc::bluray.rs already populates from the same marks during disc
init — this is purely the diagnostic surface for labels-analyze.
2026-05-10 21:32:54 -07:00
MattJackson 058fd8396f labels: gap-fill MPLS streams when framework parser under-yields
When a framework parser (paramount, criterion, pixelogic, ctrm, dbp,
deluxe) is chosen but its label list covers only a subset of the
stream slots MPLS knows about, merge MPLS-derived entries for the
uncovered (stream_type, stream_number) slots. Framework labels keep
their richer fields (purpose=Commentary, codec_hint with "Atmos",
qualifier=Sdh); MPLS only fills slots the framework left unnamed.

Implementation:
- `fn fill_gaps_from_mpls` walks the MPLS label list, pushing any
  entry whose (type, number) tuple isn't already in the framework
  output. Stable sort by (type, number) groups audios before
  subtitles in the merged result.
- Called from both `extract()` and `analyze()`. Skipped when the
  chosen parser is itself `mpls_universal` (no gaps possible).
- `LabelAnalysis::gap_fill_added` field reports how many slots got
  filled — useful diagnostic from `labels-analyze`.
- `StreamLabelType` gains `Eq + Hash` so the dedup HashSet works.

Tested via 4 new unit tests (155 of 155 labels tests passing, was
151). End-to-end on partial-yield corpus discs:
- disc-05 (Oppenheimer): pixelogic 4/5 already covered, gap_fill_added=0
- disc-11 (Dune Pt 2):   pixelogic 8/11 already covered, gap_fill_added=0

(Real-world gap-fill activations are rare in the current corpus because
pixelogic already incorporates MPLS-equivalent data when matching;
the merge is defensive for less-thorough frameworks.)
2026-05-10 21:28:01 -07:00
MattJackson 5ee28c08b9 labels/mpls_universal: dense stream numbering across playlists
Per-playlist stream_number counters were resetting between MPLS
files, so a disc with 2 MPLS files each listing the same 8 audio
streams produced labels with stream_number 1..8 then 1..8 again
(dedup kept whichever PID was different, leaving the numbering
visibly broken — multiple "audio1: eng" rows).

Move the counters outside the per-file loop and increment only
when an entry survives dedup. Surviving entries now get dense
1..N numbering across the whole disc per stream_type.

Verified on corpus disc-02 (HDMV-only): was `audio1, audio1, audio1`
for the 3 distinct audio codecs (TrueHD/AC-3/DTS-HD MA), now
`audio1, audio2, audio3`. Same fix applies to disc-01 (12 audio
streams across multiple MPLS) and disc-09 (14 audio streams).
2026-05-10 20:58:33 -07:00
MattJackson 764230b1eb labels: universal MPLS fallback + bdmt disc metadata + png stub
Three new modules in the labels platform, all layered so framework-
specific parsers (paramount, criterion, pixelogic, ctrm, dbp, deluxe)
always take precedence over the universal layer.

**mpls_universal.rs** (~600 LOC, 9 tests): consumes the already-parsed
`crate::mpls::Playlist::streams` and emits StreamLabel entries with
language + codec_hint per stream. Returns `Confidence::Low` (new
variant) so framework parsers' Medium/High always win the registry's
max-by-confidence tiebreaker; MPLS only gets picked when no framework
matched. Closes the "no BD-J disc" case (HDMV-only navigation) that
previously produced zero labels — language and base codec are
spec-mandated in MPLS STN tables on every Blu-ray ever made.

**bdmt.rs** (~350 LOC, 10 tests): reads `/BDMV/META/DL/bdmt_<lang>.xml`
files into a new `DiscMetadata` struct (localized title names per
ISO 639-2 code, descriptions, optional box-set position). Runs
independently of the parser registry — disc-level metadata, not
per-stream, so the registry's confidence selection doesn't apply.
Surfaced on a new `LabelAnalysis::disc_metadata` field.

**png_filenames.rs** (noop stub): pattern documentation + dead-code
detect/parse for future reactivation. Deferred because MPLS already
delivers per-stream lang/codec/type on every disc; PNG filename
language tokens only add studio variant disambiguation (FRC vs FRP,
LAS vs CSP) — niche enough to not justify the implementation cost
right now.

Wiring changes in `mod.rs`:
- New `Confidence::Low` variant (PartialOrd places it below Medium/High)
- New `ParseResult::low()` constructor
- `mpls_universal` appended last to `PARSERS` registry
- `LabelAnalysis::disc_metadata: Option<DiscMetadata>` field
- `analyze()` runs `bdmt::parse` independently and surfaces result
- `pub use bdmt::DiscMetadata` re-export so the labels-analyze tool
  in freemkv-tools can construct the JSON payload

Total: 151 of 151 labels tests passing (was 132 — added 19 new).
2026-05-10 20:53:56 -07:00
MattJackson 7fb2e07aed v0.18.13: bump version 2026-05-10 19:57:44 -07:00
MattJackson 376aadb335 disc/patch: adaptive batching — 32 sectors, drop to 1 on failure
Pass N now reads at 32 sectors per attempt and drops to 1 only on
batch-read failure to probe each sector individually. After 16
consecutive clean single-sector reads it climbs back to 32. Net
effect: NonTrimmed regions walk ~32x faster in clean stretches
without sacrificing per-sector recovery quality — the drop-to-1
retry from the same cursor position guarantees every sector in a
failed batch is individually attempted.

Design contract:
- A batch-read failure (count > 1) is NOT a recorded failure: no
  NonTrimmed mark, no consecutive_failures bump, no damage_window
  push, cursor stays put. We just drop current_batch to 1 and the
  loop re-attempts the same position at single-sector granularity.
- A single-sector failure (count == 1) follows the existing path:
  NonTrimmed mark, consecutive_failures++, damage_window.push(false),
  post-failure pause, wedge probes.
- Backtrack always at count=1: this path fills a gap that the main
  loop's damage-window skip jumped over. Using batched reads there
  would lump good sectors into NonTrimmed marks when the gap
  contains even one bad sector.

State machine adds:
- `initial_batch` (from opts.block_sectors, default 32 in patch_internal)
- `current_batch` (mutable, starts at initial_batch, drops to 1 on
  batch failure)
- `consecutive_singles_ok` (counter, resets on upscale + failure)
- `ADAPTIVE_UPSCALE_THRESHOLD = 16` (matches sweep's pattern for
  "16 consecutive good = back to fast mode")

Tests:
- pass_n_size_aware_skip.rs PatternedSectorReader now fills each
  sector with its OWN LBA byte (not the starting LBA's byte). This
  matches real drive behavior — the pre-0.18.13 fixture's
  "fill whole batch with one byte" was a shortcut that only worked
  when patch read 1 sector at a time. Existing recovery-quality
  assertions all still pass under adaptive batching.

User spec: "try 32, pass, great, fail -> do 1 sector"
2026-05-10 19:57:27 -07:00
MattJackson 586495b2ee v0.18.12: bump version 2026-05-10 19:44:57 -07:00
MattJackson 72f2224efe disc/patch: leave failed reads NonTrimmed mid-multipass, not Unreadable
User design call after watching Pass 2 mark ~20 KB as "Cosmetic"
(permanently Unreadable) after just 10 retries within a single pass:
"i think it's good or maybe until all passes are done. then it's
gone."

That contradicts what the multi-pass design promises a user. The
project goal in project docs is "recover 100% of readable data from any
optical disc, automatically." Marking sectors Unreadable after a
SINGLE pass's per-range retry budget gives up on sectors that
subsequent passes might recover — drive reads are stochastic, the
sector that fails 10 times in Pass 2 may succeed on attempt 1 in
Pass 3 after temperature / bus state / prior-read patterns shift.
The patch.rs doc comment already noted ~36% of patch-marked
Unreadable sectors turned out to be readable in re-rip experiments.

Three sites in `Disc::patch` were emitting `PatchItem::Unreadable`
mid-pass:
  - backtrack hit damage (line ~2659)
  - all-retries-exhausted on a single LBA (line ~2846)
  - redundant second mark after the wedge-suspicion log (line ~2970)

All three now emit `PatchItem::NonTrimmed` instead. Failed bytes
stay "maybe" (NonTrimmed) so the next pass gets another shot. The
per-range skip-limit (10) and per-pass wedge-threshold (50) still
bound time-per-pass; they just no longer turn the bytes terminal.

The `PatchItem::Unreadable` variant stays in the enum (with
#[allow(dead_code)]) because the orchestrator-side end-of-recovery
promotion will use it: autorip, after the final retry pass
completes, scans the mapfile and promotes still-NonTrimmed →
Unreadable. That promotion lands in a follow-up commit on the
autorip side — separable from this libfreemkv change.

Loss accounting unchanged: `bytes_pending + bytes_unreadable` is
the "lost or pending" total that `abort_on_lost_secs` consults
(disc/mod.rs:1327). Moving bytes from one bucket to the other
mid-pass doesn't affect whether the rip would abort; it only
affects display (UI shows "Maybe" vs "Cosmetic") and whether
subsequent passes retry the bytes (the actual fix).

Test update: `test_pass_progress_separates_unreadable_from_pending`
was renamed to `test_pass2_leaves_failed_reads_as_pending_not_unreadable`
and rewritten to assert the new invariant — Pass 2 leaves all
failed bytes as bytes_pending (no mid-pass Unreadable promotion).
Original assertions were checking the pre-design-call behavior.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 18:47:05 -07:00
MattJackson 0bca7a11bd disc/read_error: unify Pass 1 and Pass N error handling
User's design call after watching the avoidance work prevent a wedge
on the live rip (no wedge events across 6 read errors): "Pass N
and 1 should both be very very similar in recovery. almost identical
just smaller sectors imo in pass n. pause times the same imo as a
failed read is a failed read."

The error-handling code path was already centralized (one
handle_read_error fn, called by both Disc::sweep and Disc::patch).
The TUNING was split — Pass 1 used 5 s inter-error pauses + a
wedge-skip-and-continue policy; Pass N used 1 s pauses + immediate
AbortPass on HARDWARE_ERROR / ILLEGAL_REQUEST. That asymmetry made
Pass N vulnerable to the same wedge that Pass 1's avoidance fixed.

Changes:

1. FAIL_PAUSE_SECS = 5 — single constant, applied uniformly to both
   passes. Dropped PASS_1_FAIL_PAUSE_SECS and POST_FAILURE_PAUSE_SECS
   in favor of one value. CONSECUTIVE_FAIL_LONG_PAUSE_SECS kept as a
   distinct (but currently equal) value for future tuning escalation.

2. HARDWARE_ERROR / ILLEGAL_REQUEST path is now symmetric:
   - Pass 1: JumpAhead WEDGE_JUMP_SECTORS (1 GB) + WEDGE_PAUSE_SECS
     cooldown, mark skipped region NonTrimmed.
   - Pass N: JumpAhead WEDGE_PASS_N_SKIP_SECTORS (64 sectors / 128 KB)
     + WEDGE_PAUSE_SECS cooldown. Pass N's batch=1 means a 1 GB skip
     would abandon the entire current NonTrimmed range; small skip
     moves past the bricked LBA + buffer, outer patch loop picks up
     the next sector.
   - Both share WEDGE_ABORT_THRESHOLD — same 16-skip budget before
     real AbortPass on a permanently stuck drive.

3. wedge_skip / wedge_abort tracing logs now include `pass=1|N`
   so post-mortems can see which pass hit the wedge condition.

Cost analysis:

Pre-reframe worry was "5 s × 5500 NonTrimmed sectors per Pass N
pass × 7 passes = 53 hours." Reality: most NonTrimmed sectors
recover on first or second retry, so most reads are successful and
pay 0 pause. The few that DON'T recover hit the 10-skip budget and
get marked Unreadable — bounded at 10 × 5 s = 50 s per truly-bad
sector. Worst-case Pass N pause overhead on a typical damaged disc
is single-digit minutes, not hours. And it's strictly cheaper than
the alternative (wedge kills the entire multi-pass recovery).

Tests:

- `both_passes_pause_on_failed_read_for_wedge_avoidance` — locks the
  unified pause-tuning policy (was pass_1_pauses_briefly).
- `pass_n_hardware_error_also_skips_not_aborts` — was
  `pass_n_hardware_error_still_aborts`. New behavior verified:
  JumpAhead with WEDGE_PASS_N_SKIP_SECTORS + WEDGE_PAUSE_SECS.
- `pass_n_hardware_error_aborts_after_threshold` — new. Confirms
  Pass N respects the same WEDGE_ABORT_THRESHOLD as Pass 1.
- pass_1_does_not_pause_on_skip is gone (it was the old "Pass 1
  pause=0" assertion, irrelevant after the avoidance work).

Empirical validation: avoidance was already proven on a live rip
tonight — 6 read errors on a damaged disc, sense_family=Medium
throughout, wedge_count=0, Pass 1 continued cleanly past 40%
where it previously died at 48%. This commit extends the same
discipline to Pass N's recovery loop.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 18:04:01 -07:00
MattJackson 72ab714c1e labels/deluxe: Phase D rewrite against ground-truth binding pattern
Replaces the speculative arg-position heuristic with type-presence
detection driven by real disc bytecode. Ground truth captured in
(internal)/research/deluxe-poc/data/ via POC v0.3 binding-
bytecode dumps against disc-01 (Disney) and disc-09 (Warner).

What changed:

1. StackVal::CodingType(String) — new variant. getstatic against
   org/bluray/ti/CodingType (the BD-J spec codec enum) now pushes
   this, carrying the field name (e.g. DOLBY_LOSSLESS_AUDIO). The
   pre-fix code was treating codecs as a Deluxe-internal enum
   subclass walk (Phase B), which is the wrong model — codecs are
   standard BD-J API references.

2. coding_type_to_codec_hint(field) — new function. Maps
   org.bluray.ti.CodingType field names to human-readable codec
   strings (DOLBY_LOSSLESS_AUDIO -> "Dolby TrueHD", DOLBY_AC3_AUDIO
   -> "Dolby Digital", etc.). Unknown field names pass through
   verbatim so future codec values still surface something.

3. find_binding_classes — multi-class variant. Some Deluxe discs
   split per-stream tables across two binding classes (audio +
   subtitle). Returns top-K candidates by getstatic count, filtered
   to >=40% of the top count and capped at 4. Replaces the old
   single-class find_binding_class (which was unused after this
   change).

4. interpret_streams — rewritten. Args identified by TYPE not
   position:
   - First EnumRef{kind:"Language"} -> language
   - First EnumRef{kind:"Purpose"}  -> purpose
   - First CodingType(name)         -> codec_hint
   - First Int(n)                   -> stream index hint (traced
     only; per-type sequential stream_number still wins because BD
     spec stream-numbering is anchored on MPLS)
   - Construction has CodingType -> Audio stream; otherwise Subtitle
   - No Language -> skip (not a stream construction)

   This handles BOTH the Disney 5-arg pattern (I, Lbe, Llp, I,
   LCodingType) and the Warner 4-arg pattern (I, Law, Lgp,
   LCodingType) automatically — same code path because args are
   identified by type rather than constructor-signature shape.

5. parse() now walks all binding-class candidates and unions
   their constructions before calling interpret_streams. Logs each
   candidate at INFO with getstatic_count for diagnosis.

Tests:
- 2 new tests verify the CodingType -> codec_hint mapping for
  known + unknown field names.
- Existing interpret_streams tests updated to use the new
  signature (dropped CodecTable arg).
- Audio-emission test rewritten to use CodingType arg instead of
  the old binding_type substring-match approach.

Confidence is still Medium for now (single-corpus verification);
ready to promote to High once tested against a third Deluxe disc.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 17:32:29 -07:00
MattJackson d7fb1b35ed disc: emit Pass 1 summary INFO log at sweep exit
Wires the existing PassSummary infrastructure (in read_error.rs as
of a832bad) into the sweep loop's exit path. One INFO log line per
Pass 1 completion gives operators an at-a-glance damage profile
without grepping per-error WARN lines:

  INFO pass1_summary  total_reads_ok=384521 total_errors=5
                      zones_entered=1 jumps_taken=2
                      bytes_good=38_725_644_288 bytes_pending=46_GB
                      copy_elapsed_ms=1751650

Particularly useful for post-mortem analysis when combined with
the per-error structured WARN logs (ms_since_last_error /
ms_since_last_success / sense_family / wedge_transition) shipped
in 0.18.10. Single line tells you the pass shape; preceding WARN
lines tell you the per-error detail.

Pass N (Disc::patch) intentionally NOT covered in this commit —
Pass N has its own retry-budget summary semantics that warrant a
separate design pass. Pass 1 sweep is where wedge incidents
originate, so it gets the diagnostic surface first.

Staged for 0.18.11. 0.18.10 already shipped the per-error WARN
layer; this is the finishing companion log.
2026-05-10 17:21:37 -07:00
MattJackson 06be4defd2 v0.18.10: bump version 2026-05-10 17:14:52 -07:00
MattJackson a832bad697 disc: structured timing + transition diagnostics for read errors
Adds the observability we need to debug wedge incidents from logs
alone — without needing to enable verbose TRACE-level SCSI tracing.
Goal stated by user: "when error occurs we can debug and code
correctly."

Pre-fix the WARN log on each read error showed only sense codes
and consecutive_failures. Missing: timing context (was the failed
read fast or slow?), gap to previous events (cumulative vs.
immediate failure?), and family transitions (did the drive just
flip into wedge mode, or has it been there?).

New fields on ReadCtx (no caller signature change):

  last_success_at: Option<Instant>
  last_error_at: Option<Instant>
  last_error_family: Option<SenseFamily>
  total_errors: u64
  total_reads_ok: u64
  zones_entered: u64
  jumps_taken: u64
  in_damage_zone: bool

New SenseFamily enum (NotReady / Medium / Hardware / IllegalRequest
/ Other) with is_wedge_family predicate.

handle_read_error WARN log now carries:
  consecutive_failures
  consecutive_outer_failures
  ms_since_last_error    NEW gap between this and previous error
  ms_since_last_success  NEW gap to last good read
  total_errors           NEW aggregate this pass
  total_reads_ok         NEW
  wedge_count
  sense_family           NEW typed category, easier to filter
  sense_key / asc / ascq (existing)

NEW WARN log "wedge_transition" fires once when the sense family
changes from non-wedge to wedge (Medium to Hardware/IllegalRequest).
That's the moment the drive's firmware flipped into fast-fail
mode. Single timestamped event in the log so post-mortems can
pinpoint the transition without scanning thousands of TRACE lines.

Worked example: if the next wedge incident shows

  read_error  ms_since_last_success=18234  ms_since_last_error=null
  read_error  ms_since_last_success=28000  ms_since_last_error=10000
  read_error  ms_since_last_success=43000  ms_since_last_error=68
                                          (drive returned <100ms = wedge symptom)
  wedge_transition  errors_in_zone=5  ms_since_last_success=43000

we can immediately tell cumulative damage, 5 errors over 43 s,
drive went into fast-fail mode at the 5th. If instead we see

  read_error  ms_since_last_success=200  ms_since_last_error=null  sense_family=Hardware
  wedge_transition  errors_in_zone=1

the wedge was triggered by ONE read at a physically-bricked LBA
(immediate fast-fail, no warm-up).

These two patterns demand different tuning responses (longer
pause vs. larger initial jump), and now we can distinguish them
from a single WARN log line each instead of needing TRACE
verbose for the whole rip.

Plus jumps_taken / zones_entered counters that feed an end-of-pass
INFO summary (PassSummary). Caller invokes pass_summary at sweep
end and logs structured stats: "Pass 1 saw N errors / M ok reads
/ K zones / J jumps". Single-line post-mortem for any rip.

No caller signature change (timing is internal to the handler;
end-of-pass summary is a new method callers opt into). Precommit
green; 433+ tests pass. Staged for the 0.18.10 release once we
have user-validation data on 0.18.9's avoidance tuning.
2026-05-10 17:10:52 -07:00
MattJackson 26e2d847d5 v0.18.9: bump version 2026-05-10 17:02:24 -07:00
MattJackson 4442fa2df6 disc: wedge AVOIDANCE on Pass 1 — inter-error pause + larger jumps
Complements the wedge-skip backstop (fbdb50c) with proactive
avoidance so we don't HIT the wedge in the first place. User's
take after seeing the Dune Pt 2 rip wedge at 48%: 'we shouldn't be
wedging.'

Empirical observations from the 23:09:12-23:09:55 wedge timeline:

  5 read errors over 43 s, ~8 s apart (drive's own ECC recovery
  takes 5-10 s per failure). Not 'hammering' in any usual sense,
  but cumulative firmware-state buildup over 5 in-cluster errors
  was enough to tip the BU40N into wedge mode at the 5th error.

  Damage cluster spanned ~140 MB (LBAs 19.898M-19.965M). Current
  damage-jump base of 256 sectors × batch=32 = 16 MB first jump,
  doubling to 32 MB, 64 MB... Each jump landed BACK INSIDE the
  140 MB cluster, exposing the drive to MORE in-cluster errors.

Two avoidance levers:

1. Inter-error pause on Pass 1 (PASS_1_FAIL_PAUSE_SECS = 5 s):
   pre-fix Pass 1 ran pause_secs=0 on all errors to 'zoom past'
   damage zones. Successful reads still zoom at zero pause — the
   pause applies only to FAILED reads, giving the drive's firmware
   cool-down between cluster exposures. Cost: ~5 s per scattered
   failure (~30-60 s total on a damage cluster); trivial vs.
   crashing the rip at 48%.

2. Larger damage-jump base (JUMP_BASE_SECTORS = 1024, up from
   256): first jump at batch=32 now covers 64 MB instead of 16 MB,
   second jump 128 MB instead of 32 MB. Two jumps clear 192 MB —
   well past most single-cluster damage patterns. Smaller jumps
   were landing inside the cluster and adding to the wedge counter.

Plus a halt-aware sleep helper (sleep_secs_or_halt) so the new
inter-error pause doesn't degrade halt response time. Halt poll
granularity 100 ms — halt fires within ~100 ms regardless of
remaining pause time. Updated three sleep call sites in disc/mod.rs
(SkipBlock pause, JumpAhead post-pause, Retry pause).

The wedge-SKIP backstop (fbdb50c) stays — combined with this
avoidance work, the flow becomes:
  damage cluster encountered →
    pause 5 s, mark NonTrimmed →
    second failure →
    pause 5 s, mark NonTrimmed →
    ...
    threshold hit →
    damage-jump 64 MB (clears 95% of clusters) →
    if jump lands in another cluster: 128 MB next jump →
    only if drive STILL wedges after all this:
      wedge-skip kicks in (1 GB jump + 30 s cooldown × 16 budget)

Tests:
  pass_1_pauses_briefly_on_skip_for_wedge_avoidance — locks the
    new 5 s pause behavior in place (replaces the old pause=0 test).
  integration test threshold bumped from 5 s to 60 s with comment
    explaining the new bound is 'not infinite' rather than
    'milliseconds-fast'.
  All 433+ tests green on cargo +1.86 fmt + clippy + test.

Precommit green.
2026-05-10 16:55:02 -07:00
MattJackson 23dc55661c labels: apply_labels integration tests + class_reader robustness fuzz tests
Closes the final two audit items from this session.

labels::apply_labels: factored out of apply() so the matching logic
is unit-testable without needing a SectorReader / UdfFs. 11 new
tests in apply_tests cover:
  - codec_hint + variant flow through to AudioStream.label
  - purpose set on audio with no label English text
  - name fallback only when purpose=Normal (CLI owns purpose i18n)
  - subtitle SDH qualifier set; forced flag flipped on Forced
  - per-type 1-based indexing (audio #2 maps to 2nd audio stream,
    not 2nd stream overall)
  - labels for nonexistent streams are no-ops
  - empty labels list leaves streams untouched
  - fill_defaults generates audio + video labels; preserves existing

class_reader: robustness smoke tests. ClassFile::parse must NEVER
panic on adversarial input — only return Err. 9 new tests:
  - empty input
  - short magic (0..4 bytes)
  - wrong magic
  - truncated after magic
  - bad CP tag
  - truncated UTF-8 in CP
  - 200 random byte buffers (deterministic xorshift)
  - 100 magic + random tail (most adversarial — magic check passes,
    everything else garbage)
  - instructions iter on random code (200 buffers)
  - instruction_size on every opcode 0..255 with varied tail buffers
  - modified_utf8 on random byte buffers (500)

The xorshift PRNG keeps the tests deterministic (no rand dep) and
reproducible — failures will be the same buffer every time. This is
the lightweight alternative to a cargo-fuzz setup; if/when we adopt
cargo-fuzz, these tests stay as regression cases.

All 451 tests passing on cargo +1.86 fmt + clippy + test.
2026-05-10 16:38:19 -07:00
MattJackson fbdb50c79f disc: Pass 1 wedge-skip instead of abort-on-first-wedge
Pre-fix: when the drive returned HARDWARE_ERROR or ILLEGAL_REQUEST
during Pass 1 sweep, libfreemkv immediately returned ReadAction::
AbortPass. Autorip surfaced this as a fatal error and stopped the
rip at whatever progress percentage Pass 1 had reached — typically
40-50%. On a disc with one physical-damage cluster, the user would
see Pass 1 die at ~48% with the cryptic message 'E6000: <lba>
0x02/0x04/0x3e' and have no rip output to work with.

Root cause analysis: BU40N firmware transitions into a fast-fail
state when it hits cumulative read failures in a small LBA range —
returns HARDWARE_ERROR for every subsequent read near that LBA, even
sectors that aren't physically damaged. Per project docs 'Bad-sector
handling' rule #2, 'Recovery requires eject+reload OR significant
cool-down.' Aborting on first wedge throws away the rest of the
disc; the right response is to SKIP the wedged region (mark as
NonTrimmed for Pass N), pause for drive cooldown, and continue.

Fix: in handle_read_error, the HARDWARE_ERROR / ILLEGAL_REQUEST arm
now branches on bisect_on_marginal:

  Pass 1 (bisect_on_marginal=false): JumpAhead with WEDGE_JUMP_SECTORS
    (1 GB at 2048 bytes/sector) and WEDGE_PAUSE_SECS (30 s cooldown).
    Tracks wedge_count in ReadCtx; resets on any successful read.
    Truly aborts only after WEDGE_ABORT_THRESHOLD (16) consecutive
    wedges with no good read in between — generous enough to clear
    most physical-damage clusters, bounded enough to not loop forever
    on a permanently bricked drive.

  Pass N (bisect_on_marginal=true): unchanged AbortPass. Pass N's
    job is single-sector recovery; if the drive won't talk near a
    specific LBA, skipping doesn't help. Pass N exits and lets the
    outer layer decide retry/eject/surface.

5 unit tests cover the new policy:
  pass_1_hardware_error_jumps_ahead_not_aborts — JumpAhead emitted
    with correct sectors+pause, wedge_count incremented.
  pass_1_hardware_error_aborts_after_threshold — AbortPass kicks in
    on the WEDGE_ABORT_THRESHOLD-th consecutive wedge.
  pass_1_good_read_resets_wedge_count — on_success clears
    wedge_count; subsequent wedge gets fresh skip budget.
  pass_n_hardware_error_still_aborts — Pass N's AbortPass behavior
    intact.
  pass_1_illegal_request_also_routes_to_wedge_skip — both wedge
    sense families get the skip treatment.

Impact: on the Dune Pt 2 disc that consistently wedged at 48%
(physical damage at LBA ~19.9M), Pass 1 will now jump ahead 1 GB
on the wedge, give the drive 30 s cooldown, and continue scanning
the rest of the disc. The damaged region becomes Pass N's job to
revisit. Worst case if the drive stays wedged: 16 GB of NonTrimmed
disc area before honest AbortPass.

Precommit (cargo +1.86 fmt + clippy + test) green; 430 passing.
2026-05-10 16:37:59 -07:00
MattJackson 7cc74f0087 labels/xml: shared tolerant XML helper, paramount + criterion onto it
Replaces two near-duplicate hand-rolled XML scrapers in paramount.rs
and criterion.rs with a single labels::xml module that's robust to:

- Case-insensitive tag / attribute names ('<Playlist>' matches the
  same as '<playlist>'; 'Name=...' matches 'name=...').
- XML namespace prefixes (matches '<ns:tag>' for tag='tag').
- Arbitrary whitespace inside open tags and around '=' separators
  ('<tag  name = "X">' works).
- Both quote styles for attribute values (" and ').
- Self-closing tag forms ('<tag/>' and '<tag />').
- '>' chars inside quoted attribute values (no premature end-of-tag).

Three functions:
  xml::attr(element, name) -> Option<String>
      Extract attribute value from an open-tag fragment.
  xml::text(xml, tag) -> Option<String>
      Trimmed text content of first <tag>...</tag>.
  xml::find_element(xml, tag, from) -> Option<(start, end)>
      Locate next <tag>...</tag> for iteration; handles self-closing.

22 unit tests cover the robustness properties: case-insensitivity,
namespace stripping, whitespace tolerance, quote styles, self-close
forms, no-substring-false-positive (looking for 'lang' must NOT
match 'lang_id' or 'language'), '>' inside quoted attrs, iteration
across repeated elements.

paramount.rs: drops local extract_attr; find_feature_playlist now
walks xml::find_element('playlist', ...) so it works regardless of
case and self-closing style. Pre-refactor: required exactly
'<playlist ' (single space, exact case) and '/>' for self-close.

criterion.rs: drops local extract_tag; parse_stream_infos and
parse_playback_config iterate via xml::find_element. Same case-
sensitivity + namespace gains. The 'COMMENTARY' / 'SDH' / 'DS'
content-value match is now case-insensitive too (previously a disc
authored with 'commentary' would have been miscategorized as Normal).

Pre-refactor known failure modes (none observed yet, but trivial
to trip on a future disc): vendor switches whitespace around '=',
uses single quotes, capitalizes a tag, prefixes a namespace. All
now handled.

Out of scope by design: XML entity decoding (&amp;, &lt;), CDATA
sections, comments, processing instructions. None observed in BD-J
authored label data. If a future disc trips them, the entity
decoder is a localized addition.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:21:51 -07:00
MattJackson 3aa1e528c5 labels/deluxe: full Phase B/C/D buildout — codec walk, binding decode
Completes the Deluxe parser pipeline. Phase A (master enums) was
already shipping; this commit lands Phases B/C/D so the parser now
emits per-stream StreamLabel records on Deluxe-authored discs.

Phase B (decode_codec_enum): walks the codec enum's subclass
references (one .class per codec ordinal) and extracts the codec
name string from each subclass's constant pool. Heuristic: pick the
first Utf8 entry that's uppercase + underscored + >=4 chars, or one
of the known codec roots (ATMOS/DOLBY/DTS/TRUEHD/MLP/AC3/EAC3/PCM)
when no underscored candidate is found. CodecTable maps ordinal ->
codec string; empty string for ordinals where extraction failed
(logged via tracing, not fatal).

Phase C (find_binding_class): identifies the class that builds the
per-stream label table by counting getstatic operations targeting
any of the master enum classes from Phase A. Class with the highest
count >= 4 wins. Threshold is empirical (real binding classes have
50+ matches; floor of 4 admits small discs while rejecting incidental
single-reference classes).

Phase D (decode_binding + BindingDecoder): symbolic stack machine
that walks the binding class's <clinit> bytecode. Handles:
  - constant pushes: iconst_<n>/bipush/sipush/ldc(Integer)
  - new <X>: pushes uninit-object marker
  - dup: stack copy
  - getstatic <Y.Z>: pushes EnumRef when Y is in MasterEnumTable,
    else Unknown
  - invokespecial X.<init>(...)V: pops args per descriptor; when the
    receiver is NewObj(X), emits a Construction { binding_type: X,
    args: [...] }
  - invokevirtual/invokestatic/invokeinterface: pop args per
    descriptor, push return placeholder unless void
  - pop/pop2/aastore/putstatic/putfield: standard stack effects
  - branches/returns: clear stack (conservative resync — binding
    <clinit> is straight-line in practice)
  parse_method_arg_count: JVMS field-descriptor parser, handles
  primitives, references (L...;), arrays ([...).

interpret_streams: converts Constructions to StreamLabels using
the master enum table + CodecTable. Each construction with a
Language ref becomes a stream. Audio when codec_hint resolves via
binding_type substring match against CodecTable; subtitle otherwise.
Purpose ordinal -> LabelPurpose via the verified Deluxe Purpose enum
order (Normal/Commentary/PiP/Trivia/Descriptive/Score/NoForced/
NoForcedDescriptive). Stream index = sequential per type. Language
goes through vocab::lang for ISO code + variant.

deluxe::parse now returns Some(ParseResult::medium(labels)) when
all four phases produce labels. Medium confidence — the bytecode
mechanism is rigorously tested but the signal-to-StreamLabel
mapping (which arg is which, audio vs subtitle classification) is
heuristic until corpus binding-class bytecode confirms the exact
pattern.

Test coverage: 13 new unit tests in deluxe.rs
  parse_method_arg_count: 3 tests (basic types, references, malformed)
  BindingDecoder: 4 tests (simple construction, with int pushes,
    skips unmatched invokespecial, resolves master-enum ordinal)
  interpret_streams: 4 tests (subtitle on no codec, audio on codec
    match, purpose routing, skips no-language)
  MasterEnumTable: 3 tests (resolve, value, class_name_set)
  extract_codec_name: 1 test (uppercase+underscore matching)

class_reader.rs gained a #[cfg(test)] ConstantPool::from_entries
test-only constructor so Phase D tests can build synthetic CP
fixtures without writing raw .class bytes.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:15:39 -07:00
MattJackson 7c6b0f82ab labels: per-parser confidence + highest-confidence-wins registry
Replaces 'first-match-wins by array order' with 'highest-confidence-
wins, array order tiebreaker'. Removes the arbitrariness when more
than one parser can claim a disc (e.g. one with both
bluray_project.bin and playlists.xml).

New types in labels::mod:
  pub enum Confidence { Medium, High }
  pub struct ParseResult { labels: Vec<StreamLabel>, confidence }
  ParseResult::high(labels) / ::medium(labels) constructors

Parser signature change: every parse() now returns
Option<ParseResult> instead of Option<Vec<StreamLabel>>. Updated all
six parsers in lockstep:
  paramount: High (fully structured XML)
  criterion: High (fully structured XML)
  pixelogic: High by default, Medium when an unknown token component
             is encountered (the skip-unknown path now propagates the
             coverage gap to the caller instead of silently degrading)
  ctrm:      High (structured key-value)
  dbp:       High (anchor scan with vocab routing)
  deluxe:    still returns None pending Phase D — signature aligned

Registry behavior:
  extract() iterates all detect-positive parsers, picks highest
  Confidence with non-empty labels. Equal confidence falls to array
  order (deterministic). Same selection logic in analyze().

LabelAnalysis grew a confidence: Option<Confidence> field so the
diagnostic surface (freemkv-tools labels-analyze) exposes which
confidence tier the selected parser claimed. labels-analyze JSON
and labels-corpus-check structural diff both gained the field.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:01:20 -07:00
MattJackson 4226a53e73 labels: fresh-eyes audit — capture variant, dedupe detect, lock registry
Three targeted fixes from a second-pass audit of the labels module.

1. vocab::lang now returns Option<LangInfo> with both code AND a
   human-readable variant string. Pre-fix: 'Brazilian Portuguese 5.1'
   became language=por, variant='', dropping the dialect info the
   disc had explicitly authored. Post-fix: language=por,
   variant='Brazilian' — matches the convention pixelogic / ctrm /
   criterion already use for their region variants. dbp now
   populates StreamLabel::variant from this. Compound table grew a
   3-tuple (needle, code, variant); bare matches still return
   variant=''.

2. dbp and deluxe had duplicated detect() boilerplate (any top-level
   .jar in /BDMV/JAR/). Both now call jar::has_any_top_level_jar.
   The trait-level detect contract — see super::PARSERS — can't peek
   inside a jar without a SectorReader, so loose-detect-plus-real-
   check-in-parse is the unavoidable pattern for jar-content parsers.
   Consolidating in jar.rs at least makes the duplication visible.

3. mod.rs comment about parser ordering said 'dbp last'; deluxe is
   actually now last. Updated to explain the dbp-before-deluxe order
   is by cost (cp-iteration cheaper than bytecode walking when Phase
   D lands).

Plus a registry-level lock test in mod.rs::registry_tests — asserts
the PARSERS array order is exactly [paramount, criterion, pixelogic,
ctrm, dbp, deluxe]. This was previously implicit; if someone reorders
the array (which changes which parser wins on overlapping signals),
unit tests would have stayed green. Now they fail with an explanatory
message about why the order matters.

Audit findings deferred to follow-ups (each its own commit + design
discussion):
- Stronger detect contract — current loose-detect-real-check pattern
  is forced by SectorReader-not-in-detect-signature; could be fixed
  by changing the trait to take an Option<&mut dyn SectorReader> or
  similar.
- Per-parser confidence scoring — registry currently first-match-wins.
  A high-confidence parser ought to beat a low-confidence one
  regardless of array order.
- class_reader fuzzing — handles malformed input via Result but no
  adversarial corpus yet.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 15:50:32 -07:00
MattJackson 92f34a289e labels: refactor pixelogic + ctrm onto shared platform + hardening
Closes the platform unification: every label parser now routes
purpose/qualifier/codec classification through one source of truth
(vocab.rs) instead of N hand-rolls, and every binary-blob byte
scanner goes through one helper (text::extract_ascii_strings).

pixelogic.rs:
- Drop local extract_strings (~20 lines) — use text::extract_ascii_strings.
- HARDENING: replace  with skip-unknown-component +
  trace log. Pre-refactor behavior: any single uncatalogued token part
  (e.g. a future codec ID, new framework variant) silently dropped the
  entire stream record. New behavior: skip just the unknown part,
  surface what we know about the stream.
- 8 new unit tests cover basic audio/subtitle paths, commentary,
  descriptive, region variant, the new skip-unknown-component
  regression, and the non-audio/non-subtitle early-out.

ctrm.rs:
- Replace  with
  vocab::purpose(&name). Now word-boundary matched — 'Commenter Pro
  Track' no longer false-matches Commentary.
- Replace  with vocab::qualifier(&name).
  Same word-boundary tightening, plus picks up Forced and
  DescriptiveService for free.
- Preserved structural commentary signal via
  as a fallback when name is silent (e.g. 'audio_commentary_1.name=Track 2').
- 6 new unit tests including the 'Commenter' false-positive regression
  and the SDH-only-on-subtitles boundary.

text.rs:
- Drop module-level dead_code allow now that pixelogic uses
  extract_ascii_strings.

Net: all 5 framework parsers now on the unified platform. Future work
(deluxe Phase D, paramount/criterion XML hardening) builds on the
same scaffolding.

Precommit green.
2026-05-10 15:33:22 -07:00
MattJackson fdbe469d50 labels: add deluxe parser (Phase A — master enum identification)
Closes the 'detected but no parser' gap on Deluxe-authored BD-J discs
(com/bydeluxe/ package signature; ~20% of UHD discs in our corpus per
the 2026-05-10 11-disc capture session).

What ships in this commit:

- detect() — registers the parser in the chain (loose pre-check at the
  /BDMV/JAR/ level; real signal in parse via has_path_prefix on
  'com/bydeluxe/').
- Phase A master enum identification — walks every .class's <clinit>
  ldc sequence and matches against framework-stable fingerprints for:
    Language    (70 ldcs starting English/French/Spanish/Dutch)
    Purpose     (8  ldcs starting Normal/Commentary/PiP/Trivia)
    VideoFormat (7  ldcs starting HD/HDR10 Plus/HD Dolby)
    Region      (22 ldcs starting USA_D1/LIC1/LIC2/LIC3 — Disney-only)
    Studio      (6  ldcs starting Disney/Marvel/Pixar — Disney-only)
  All identifications verified out-of-tree on corpus disc-01 (Disney,
  The Amateur) and disc-09 (Warner, Dune Part 1) via the standalone
  POC.
- Phase B structural skeleton (find_codec_enum) — identifies the
  codec enum class by structural signature (>=20 'new' ops, 0 ldcs),
  returns the ordered subclass list. Codec string extraction from
  subclasses is dead-coded pending the follow-up commit.

What does NOT ship yet:

- Phase D (per-stream binding-class decoder). parse() returns None
  intentionally — the master enums alone don't yield StreamLabels
  without the streamTable.put(...) bytecode walker. analyze() will
  show 'deluxe' in parsers_detected with the enum identification in
  tracing logs, so the analyzer reports honestly: 'detected, can't
  emit labels yet' rather than silent failure.

Why ship A without D: A is proven on real corpus discs; D's design
needs ground-truth binding bytecode from at least 2 corpus discs
side-by-side to verify the stack-machine pattern. The Phase A
infrastructure (master enum identification + ordinal->name table)
is what D will consume — landing it now unblocks D's design without
holding back the parser registration.

5 unit tests cover the fingerprint matcher logic + a roster lock that
forces explicit consideration when adding/removing fingerprints.

Precommit green.
2026-05-10 15:22:11 -07:00
MattJackson 4ec75a03f2 labels: shared platform (vocab/text/jar) + dbp refactor
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.

New modules:

  vocab.rs       expanded from 27 -> ~370 lines
                 + lang(text) -> Option<&'static str>      (English/multi-word
                                                            -> ISO 639-2; ~45
                                                            languages, compound
                                                            phrases like
                                                            'Brazilian Portuguese'
                                                            and 'Castilian Spanish')
                 + purpose(text) -> LabelPurpose            (Commentary,
                                                            Descriptive, Score,
                                                            Ime; word-boundary
                                                            matched)
                 + qualifier(text) -> LabelQualifier        (SDH, Forced,
                                                            DescriptiveService)
                 + has_word internal primitive — enforces word-boundary
                   matching so 'Commenter' no longer matches 'commentary' and
                   'engineering' no longer matches 'english'. Existing parsers
                   used .contains() and got lucky on the corpus; vocab now
                   guarantees the boundary in one place. 20+ unit tests.

  text.rs        NEW (~85 lines)
                 + extract_ascii_strings(data, min_len) — promoted from two
                   near-duplicate copies (pixelogic min=4, dbp min=5);
                   threshold passed in. 7 unit tests including
                   trailing-without-terminator + high-bit-byte handling.

  jar.rs         NEW (~120 lines)
                 + for_each_jar(reader, udf, fn)  — walk every top-level
                                                    .jar under /BDMV/JAR/,
                                                    yield to callback.
                 + has_path_prefix(archive, prefix) — cheap 'is this MY
                                                      framework's jar?' check
                                                      via central-dir filenames.
                 + for_each_class(archive, fn)    — parse every .class entry
                                                    through class_reader,
                                                    yield (name, &ClassFile).
                 + try_each_class(archive, fn)    — same with early-return on
                                                    first Some(R) match.

Refactored:

  dbp.rs         v2 on the new platform:
                 - dropped extract_printable raw byte scan
                 - dropped its own English -> ISO 639-2 map
                 - dropped its own parse_attributes hand-roll
                 + iterates CpInfo::Utf8 via class_reader (structurally clean,
                   no false-positive risk from method bytecode bytes)
                 + routes language/purpose/qualifier through vocab
                 All 7 prior dbp tests still pass; +2 new ones cover
                 vocab routing.

dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.

Precommit green (cargo +1.86 fmt + clippy + test).
2026-05-10 15:16:25 -07:00
MattJackson 99236ffd55 labels: add class_reader, hand-rolled JVM .class file parser
Foundation for label parsers that need structured access to .class
files inside /BDMV/JAR/<x>.jar. Replaces noak (~3KLOC dep) with a
~1000-line std-only reader.

Public API:
- ClassFile::parse(&[u8]) -> Result<ClassFile>
- ConstantPool::{get, utf8, class_name, string, integer, member_ref, iter}
- Member::code(&pool) -> Option<CodeAttribute>
- CodeAttribute::instructions() -> Instructions iterator
- Instruction::{name, operand_u8, operand_u16, cp_index}
- Opcode constants (LDC, AASTORE, NEW, GETSTATIC, INVOKESPECIAL, ...)

Spec coverage:
- Constant pool: all 17 tag types incl. Long/Double 2-slot quirk
- Modified UTF-8 incl. 0xC0 0x80 -> U+0000 special case
- Bytecode iteration with full opcode size table
- Variable-length tableswitch / lookupswitch / wide

12 unit tests cover the opcode table edge cases (padded switch tables,
wide-iinc 6-byte form), modified-UTF-8 decoder, and iterator
stop-on-truncated behaviour.

Module is currently #![allow(dead_code)] — the public API is staged
for labels::deluxe (Phases A-E bytecode walker) and a labels::dbp
refactor onto the constant-pool iterator. Tests exercise the API
in isolation. The allow comes off as those callers land.

Also fixes two pre-existing clippy lints that 1.86's stricter checks
flagged after I touched the labels module:
- src/mux/disc.rs: while-let-loop in test fixture
- tests/pass_n_size_aware_skip.rs: type_complexity in helper signature

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 15:06:34 -07:00
MattJackson 9e3327a569 v0.18.8: bump version (Pass-1 fast-skip never reached autorip 0.18.7 — Cargo.lock mis-resolved) 2026-05-10 14:01:23 -07:00
MattJackson 0081955686 labels: add dbp parser (Magnolia Pictures BD-J framework)
5th BD-J authoring framework recognized. Discriminator: any top-
level .jar in /BDMV/JAR/ that contains com/dbp/ package paths.
Identified during the 2026-05-10 corpus session via string-mining
disc-07's BD-J jar — perm files reference bd-live.magpictures.com
(Magnolia / Magnet Releasing).

Stream labels live as plain ASCII strings inside compiled .class
files in the form

    LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,...
    HTextField,Subtitle1,English SDH,Fontstrip_Composite,...
    ATextField,Subtitle0,None,Fontstrip_Composite,...

— a quirk of the menu-rendering layer encoding TextField positions
and content as constant strings the Java compiler retained in the
class string pool. The leading single-letter prefix is string-pool
ordering noise; parser anchors on `TextField,`. Subtitle0 is the
disable-subtitles button and is skipped.

The parser:
  - reads each top-level .jar via udf.read_file
  - opens it with the existing zip dependency
  - confirms com/dbp/ presence in the central directory
  - walks .class entries, extracts printable strings, matches the
    `TextField,(Audio|Subtitle)<N>,<label>,...` pattern
  - maps human-readable language names ("English", "Castilian
    Spanish", "Brazilian Portuguese", "Canadian French", ...) to
    ISO 639-2 codes via a parser-local table (per the rules-of-
    engagement memo, each parser knows its own format)
  - preserves the full disc-authored label string in `name` so
    consumers display it raw without the lib guessing further
    structure
  - detects SDH / Forced qualifiers and Commentary / Descriptive
    purposes from substring matches; everything else falls through
    to fill_defaults using BD-spec MPLS data

Verified live on the corpus: disc-07 (Civil War UHD) now matches
parser=dbp with 3 audio + 2 subtitle labels, exactly the count
visible in the disc's authored TextField definitions and what BD
spec MPLS reports.

Limitation: dbp's detect() returns true for ANY top-level .jar in
/BDMV/JAR/ (every BD-J disc has one), since the discriminator
trait function takes only `&UdfFs` and can't read jar contents.
parse() does the real com/dbp/ check — a non-dbp disc gets
parsed-as-dbp, archive_has_dbp returns false, parse() returns
None, and we fall through. Diagnostic noise: parsers_detected
includes "dbp" on non-dbp BD-J discs. Real fix is refactoring the
DetectFn signature to take a SectorReader; deferred.

7 unit tests cover the TextField extraction, language detection
(simple + compound: "Brazilian Portuguese", "Castilian Spanish",
"Canadian French", "Latin American Spanish", "Australian English"
plus the disc-corpus typo "Austrailian English"), SDH/Forced/RNIB
qualifier detection, and Commentary/Descriptive purpose detection.
Don't-guess discipline preserved: unknown languages return ""
(consumer falls back to MPLS spec data via fill_defaults).
2026-05-10 13:15:07 -07:00
MattJackson 3244fdd683 v0.18.7: Pass 1 fast-skip, defer recovery to Pass N
Pass 1 sweep was grinding through damage zones because the marginal-
media handler returned `Bisect` for every failed 32-sector batch —
forcing 32 single-sector reads per bad block at ~5s each on a real
BU40N-vs-Dune-Pt-2 trace. AND the JumpAhead trigger required a 16-
block damage window to fill before firing, so entry into a
contiguous damage zone took ~40 minutes of grinding before the
first jump fired. Architecturally wrong: Pass 1's job is "fast and
accurate, get the most data in the shortest time." Bisection +
recovery is Pass N's purpose-built role.

ReadCtx now carries two new fields:
  - `consecutive_outer_failures: u64` — outer-batch failures since
    last outer success. Bisect inner failures don't count.
  - `bisect_on_marginal: bool` — whether to return Bisect on a
    marginal-media batch failure.
  - `fast_jump_threshold: u64` — outer-failures count that triggers
    JumpAhead before the damage window has filled.

`for_sweep` (Pass 1) sets `bisect_on_marginal=false`,
`fast_jump_threshold=4`, and zeroes the post-failure pause. Failed
batches become SkipBlock → whole block NonTrimmed → advance, no
sleep. After 4 consecutive outer failures: JumpAhead with the
existing escalating multiplier.

`for_patch` (Pass N) sets `bisect_on_marginal=true`,
`fast_jump_threshold=u64::MAX`, keeps the original cooldown pauses.
Pass N's whole reason to exist is to grind on bad ranges with
proper recovery semantics — single-sector reads, 60s recovery
timeout, retry budget, escalating skip — and that's unchanged.

`on_success` resets `consecutive_outer_failures` only when not
bisecting, so a good single-sector read inside Pass N's bisect
doesn't pretend we've escaped the damaged batch.

Tests:
  - `pass_n_marginal_with_batch_gt_1_bisects` — Pass N still bisects.
  - `pass_1_marginal_skips_instead_of_bisecting` — Pass 1 doesn't.
  - `pass_1_jumps_after_4_consecutive_outer_failures` — fast-entry.
  - `pass_n_does_not_fast_jump` — fast-entry is Pass-1-only.
  - `outer_success_resets_consecutive_outer_failures` — counter reset.
  - `bisect_inner_success_does_not_reset_outer_counter` — semantics.
  - `pass_1_does_not_pause_on_skip` — explicit zero-pause contract.
  - `long_failure_streak_extends_pause_on_pass_n` — Pass N still
    extends pauses on long failure streaks (renamed from the old
    sweep-based test).

Integration test `test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed`
updated: it used to assert Pass 1 recovers all sectors via bisect
(bytes_good=total). New contract: Pass 1 marks NonTrimmed; Pass N
recovers. Test now asserts Pass-1-only outcome (bytes_pending=total,
complete=false) consistent with the redesign.

Real-world impact on the user's BU40N + Dune Pt 2 trace from this
session: a damage zone that was on track to take ~40 minutes of
Pass-1 grinding will now jump in ~20 seconds. Pass N still has the
full 7-pass recovery budget to revisit those NonTrimmed ranges.
2026-05-10 12:50:12 -07:00
MattJackson ef5487e5a5 labels: distinguish "parser detected" from "parser succeeded"
Adds `parsers_detected: Vec<&'static str>` to `LabelAnalysis`. Records
every parser whose discriminator matched, regardless of whether its
parse() then returned Some/None.

Why: when `parser=None` we currently can't tell apart:
  (a) no parser recognized this disc — missing parser, candidate for a
      new module
  (b) a parser recognized it but parse() returned None / empty —
      capture truncated, or genuine empty authoring data, or a parser
      bug

Surfaced concretely on the 11-disc capture session 2026-05-10:
disc-04 had `bluray_project.bin` in jar_inventory (pixelogic detect()
returned true) but parse() returned None because file content was
past the 1 GB capture window. Old API: parser=None — looked like
"missing parser." New API: detected=[pixelogic], parser=None — clearly
"capture problem, not a parser gap."

The tracing log line on the no-parser-emitted-labels path now
distinguishes the two cases too.

No behavior change to production rip path. extract() is unchanged;
only the diagnostic analyze() returns the richer result.
2026-05-10 12:36:05 -07:00
MattJackson e7dffa63d2 v0.18.6: bump version (unified release with bdemu/freemkv/autorip) 2026-05-10 10:03:03 -07:00
MattJackson e5989c8270 labels: expose analyze() for corpus regression tooling
Promotes `mod labels` to `pub mod labels` and adds `analyze()` plus
`LabelAnalysis` (both `#[doc(hidden)]`) so an out-of-tree diagnostic
binary (freemkv-tools labels-analyze) can introspect which BD-J parser
matched a given disc, what JAR files the discriminators saw, and what
labels came out — without going through the production `apply()` path
that mutates DiscTitles.

Also adds `tracing::info!(parser = name, "label parser matched")` /
"no label parser matched" inside `extract()`. Dev-only signal: users
get the same seamless behavior; developers can finally tell whether a
disc hit a real parser or fell through to the codec-name fallback in
fill_defaults().

The new `jar_inventory()` helper deduplicates and sorts filenames
under any `/BDMV/JAR/<x>/` subdirectory — same plumbing the existing
`jar_file_exists()` discriminators use, just enumerated rather than
predicate-tested. Used by `analyze()` to surface unrecognized
parser-source files when no parser matches, which is the input to
"do we need a new parser?" triage.

No behavior change to the production label path. `apply()` and
`extract()` remain functionally identical; the new public surface
exists alongside.
2026-05-10 07:42:45 -07:00
MattJackson e1c938b82a ci: drop --locked from libfreemkv workflows
libfreemkv is a library — Cargo.lock is gitignored (standard for
libs). --locked refuses to create a lockfile on a fresh runner,
so it always fails CI. --locked stays in the binary crates
(freemkv, autorip, bdemu) which DO track Cargo.lock and benefit
from the dependency-race hard-fail behaviour.
2026-05-09 20:40:31 -07:00
MattJackson ed801a708b fmt: rustfmt-mandated reflow of canonical_title_order tests
The 0.18.4 commit landed with rustfmt diffs in the new
canonical_order tests because my local validation script piped
'cargo fmt --check' to 'tail -1', masking the diff output and
reporting green when fmt was actually unhappy. CI's lint job
caught it immediately. No code change — pure formatting.
2026-05-09 20:34:40 -07:00
MattJackson d94451954c v0.18.4: cargo --locked everywhere — hard-fail dependency races 2026-05-09 20:30:34 -07:00
MattJackson 385c9f094c v0.18.3: canonical_title_order — main feature first on branching UHDs 2026-05-09 20:08:56 -07:00
MattJackson 9f2a13739d Disc title order: main feature first on branching UHDs
Disc::titles previously sorted purely by duration_secs descending,
which puts a play-all virtual playlist at index 0 on UHDs that ship
one. Such playlists reference the same source clips multiple times
for seamless alternate-angle / alternate-ending playback and report
inflated duration AND inflated size_bytes that exceeds the disc's
physical capacity.

Concrete observed case (The Amateur 2025 4K UHD, 58.5 GB BD-100):
  Title 1 — 00020.mpls — 4h13m — 92.4 GB — 253 clips  ← impossible
  Title 2 — 00800.mpls — 2h02m — 57.2 GB — 1 clip      ← the movie

92.4 GB > 58.5 GB capacity is proof of clip double-counting. With
the duration-only sort, freemkv -t 1 / disc.titles.first() / autorip's
main-feature picker all selected the 4-hour composite instead of the
2-hour movie.

New canonical_title_order:
  1. Real titles (size_bytes <= capacity_bytes) before virtual
     composites — capacity gate is hard physical truth.
  2. Among real titles, fewer clips first (1-clip wins as the
     canonical main feature; multi-clip is either chapter-stitched
     or composite).
  3. Tiebreak on longer duration first.

Behaviour:
- Non-branching discs: unchanged. The longest 1-clip title is
  already the movie.
- Branching UHDs: virtual composite drops to the back, the real
  movie surfaces at index 0.

Comparator exposed as Disc::canonical_title_order for downstream
consumers that need the same logic on custom title sets.

Three regression tests (disc::tests::canonical_order_*):
- pushes_oversize_play_all_behind_real_main (The Amateur)
- preserves_natural_ranking_on_normal_disc
- fewer_clips_wins_tiebreak
2026-05-09 19:57:07 -07:00
MattJackson d9ce69bc9d v0.18.2: fix AACS nav-file scramble + sweep progress non-regression
decrypt::decrypt_sectors now restores chunks when decrypt_unit_full's
TS-sync verification fails, preventing 0.18.1's silent corruption of
MPLS/CLPI navigation files when DecryptingSectorSource decorates the
sweep reader. Fixes E6009 NoStreams on info iso:// for AACS-encrypted
UHDs ripped without --raw.

Disc::sweep progress takes max(snapshot.bytes_good, bytes_done) so
the user-visible counter never regresses below what the producer has
already sent.
2026-05-09 17:19:47 -07:00
MattJackson 7cd2c937ed 0.18.1 docs: refresh README, CHANGELOG, and docs/ for the trait split
The library's public-facing docs were sitting on the 0.17 trait
surface — Disc::copy, pes::Stream, SectorReader, etc. — even though
all in-tree callers migrated in 0.18 rounds 1-3. With 0.18.1 about
to ship, a user copy-pasting the README sample from crates.io would
have hit a compile error.

This commit is purely doc-side:

- README.md: Quick Start rewritten onto Disc::sweep + Disc::patch
  with caller-orchestrated multipass; Streams table footnote and
  Architecture row reference FrameSource / FrameSink.
- CHANGELOG.md: 0.18.1 entry describing the redesign — primitives,
  trait splits, deprecations (kept alive through 0.18.x, deletion
  target 0.18.2), throughput numbers.
- docs/{rip-recovery,api-design,architecture,disc-to-rip,
  drive-access,udf}.md: every Disc::copy / pes::Stream /
  SectorReader reference updated to the 0.18 trait surface.
- FEATURES.md: deleted (8+ versions stale; capabilities live in
  README.md and CHANGELOG.md now, matching the workspace-top
  FEATURES.md removal in 84acd65).
- examples/iso_dump.rs: verified compiles against 0.18.1.

No code changes.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 12:13:51 -07:00
MattJackson 107524da5e 0.18.1: bump version 2026-05-09 11:38:24 -07:00
MattJackson 055a3c5276 0.18 round 3: make Disc::sweep + Disc::patch pub (was pub(crate))
Round 3 step 1: lift the visibility on the two flat rip-phase verbs
so consumers (autorip + freemkv CLI) can call them directly instead
of going through Disc::copy's multipass dispatcher. Also lift their
option/outcome types and re-export at crate root.

- fn sweep -> pub fn sweep (with rustdoc explaining its role)
- fn patch -> pub fn patch (ditto)
- pub(crate) struct SweepOptions -> pub struct SweepOptions
- pub(crate) struct PatchOpts -> pub struct PatchOptions (renamed
  for consistency — both are 'Options')
- pub(crate) struct PatchOutcome -> pub struct PatchOutcome
- libfreemkv::{SweepOptions, PatchOptions, PatchOutcome} re-exports
  at crate root.

Disc::copy still exists and still calls Disc::sweep / Disc::patch
through the now-private sweep_internal / patch_internal wrappers.
Migration of the two autorip callers + the freemkv CLI's
disc_to_iso to direct sweep/patch is a follow-up; once those land
Disc::copy + CopyOptions + CopyResult delete in the same commit.

See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.

Single contributor: MattJackson.
2026-05-09 11:19:46 -07:00
MattJackson 50b04790ce 0.18 round 2: thread Halt through DiscStream construction
# Conflicts:
#	src/mux/disc.rs
2026-05-09 11:04:07 -07:00
MattJackson a00b2c884b 0.18 round 2: adopt DecryptingSectorSource decorator at sweep + patch + DiscStream 2026-05-09 11:03:43 -07:00
MattJackson 44e2c64d7e 0.18 round 2 (Halt threading): DiscStream accepts Halt at construction
Adds a Halt field to DiscStream, populated via the new
`with_halt(halt)` builder. The internal recovery / fill_extents
loops check `halt.is_cancelled()` directly. The existing
`set_halt(Arc<AtomicBool>)` method stays through the deprecation
window for callers (autorip mux) that haven't migrated; marked
#[deprecated] with a pointer to the constructor-time path.

Both signals are unified inside DiscStream: either Halt or the
legacy Arc<AtomicBool> triggers cancellation, so callers can mix
during the deprecation window without breaking stop behaviour.

See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.

Single contributor: MattJackson.
2026-05-09 10:58:54 -07:00
MattJackson d6535b8f57 0.18 round 2 (decrypt dedup): adopt DecryptingSectorSource at the two
existing call sites — sweep producer and DiscStream demux

Round 1 shipped the DecryptingSectorSource decorator
(libfreemkv/src/sector/decrypting.rs) but the existing decrypt
sites kept calling crate::decrypt::decrypt_sectors inline. This
commit migrates both:

- Disc::sweep (disc/mod.rs): producer wraps the input reader
  in DecryptingSectorSource::new(reader, keys) before the read loop.
  The inline decrypt_sectors call goes away — read_sectors yields
  plaintext directly.

- DiscStream (mux/disc.rs): constructor wraps the underlying
  Box<dyn SectorReader> in DecryptingSectorSource so the internal
  fill_extents / read path sees plaintext bytes. The DecryptKeys
  field stays on DiscStream for metadata-side use; it just no
  longer drives decryption.

Disc::patch carried the same inline decrypt step at three call
sites (main read, backtrack read, non-NOT_READY retry read). All
three migrated onto the same wrapping for a single audit surface.

Two small support changes carry the migration without touching
the round-1 decorator shape:
- sector/mod.rs gains specific SectorSource impls for
  &mut dyn SectorReader and Box<dyn SectorReader>, mirroring
  std's Read forwarding pattern. Generic blankets would conflict
  with the existing SectorReader → SectorSource blanket under the
  orphan rule (downstream could impl SectorReader for &mut U), so
  the impls are scoped to the dyn-trait shape we actually consume.
- sector/decrypting.rs gains DecryptingSectorSource::set_keys so
  DiscStream::set_raw() can flip the wrapped reader to a
  DecryptKeys::None pass-through without rebuilding the decorator
  (which would require moving the inner Box out from behind &mut self).

After this commit, grep `decrypt_sectors` in src/ shows the
function definition, its single use inside DecryptingSectorSource,
plus comments only. One audit surface for AACS / CSS / passthrough
correctness.

Behaviour-preserving: same plaintext bytes flow through; the only
difference is which type owns the decrypt step.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:49:53 -07:00
MattJackson 90ab00ed45 0.18 round 2: cargo fmt after sweep+patch+DiscStream merge
Auto-fmt nit on the multi-line map.flush() expression that landed
when the sweep + patch + DiscStream-FrameSource branches were merged
together. No semantic change.
2026-05-09 10:36:40 -07:00
MattJackson 6bfd7dfd13 0.18 round 2: confirm DiscStream as FrameSource via blanket impl 2026-05-09 10:32:56 -07:00
MattJackson b9a7f601d4 0.18 round 2: refactor Disc::patch onto Pipeline + PatchSink
# Conflicts:
#	src/io/mod.rs
#	src/io/pipeline.rs
2026-05-09 10:32:51 -07:00
MattJackson cf0a61f8b4 0.18 round 2: refactor Disc::sweep onto Pipeline + SweepSink (delete sweep_pipeline.rs) 2026-05-09 10:32:05 -07:00
MattJackson f98f07b2d3 0.18 round 2: refactor Disc::sweep onto Pipeline + SweepSink
Sweep was the original producer/consumer split that motivated the
generic Pipeline primitive (round 1, commit 198268b). Now that
Pipeline + Sink exist, sweep stops shipping its own bespoke
threading.

- New SweepSink: Sink<WorkItem> impl in src/disc/sweep.rs. Owns
  WritebackFile + Mapfile + ProgressSnapshot back-channel. apply()
  carries the file-write + mapfile.record per WorkItem; close()
  drains writeback, fsyncs, flushes mapfile.
- Disc::sweep: constructs SweepSink, calls Pipeline::spawn_named
  (so the consumer thread keeps showing up as
  freemkv-sweep-consumer), sends WorkItems, calls pipe.finish().
  The producer-side ReadCtx state machine, decrypt, set_speed,
  halt — all unchanged.
- Pipeline gains spawn_named(name, depth, sink) so callers can
  preserve identifiable thread names without the primitive baking
  one in. Also adds Pipeline::try_send for the throttled
  StatsRequest path that must not block the producer.
- Deleted src/disc/sweep_pipeline.rs entirely. WorkItem,
  ProgressSnapshot, ConsumerSummary moved into disc/sweep.rs as
  module-private types. WorkItem::Finish dropped — dropping the
  channel is the end-of-stream signal Pipeline already uses.

Behaviour-preserving: the sweep algorithm, mapfile invariants,
back-pressure via channel depth (DEFAULT_PIPELINE_DEPTH = 4) all
match the 0.17.13 implementation. New synthetic regression test
(sweep_pipeline_full_good_100_batches) exercises ~100 batches of
clean reads end-to-end through the new Pipeline path and verifies
bytes_good and ISO file size.

See (internal)/memory/0_18_redesign.md.
2026-05-09 10:31:05 -07:00
MattJackson b53454fa09 0.18 round 2: refactor Disc::patch onto Pipeline + PatchSink
Patch was strictly serial (per-sector recovery: read → seek+write
→ mapfile.record → next). Lifting the write+record onto a consumer
thread lets the drive issue the next per-sector retry while the
previous block's recovered bytes are being committed — small but
real win on damaged discs with many bad sectors, and uniform with
sweep's threading model.

- New PatchSink: Sink<PatchItem> impl in src/disc/patch.rs. Owns
  WritebackFile + Mapfile. apply() seeks+writes recovered bytes
  and records mapfile state per item; close() runs sync_all and
  mapfile.flush.
- Channel depth: WRITE_THROUGH_DEPTH (1). Patch wants minimum
  buffering — back-pressure should kick in immediately so the
  drive's per-sector retry budget isn't ahead of the writer.
- Disc::patch: keeps every existing recovery decision on the
  producer (reverse walk, damage-window skip, NOT_READY pauses,
  bridge-degradation handling, wedge exit, range watchdog).
  WritebackFile ownership moves to the sink.

Behaviour-preserving: per-sector single-shot read budget unchanged
(BU40N+Initio bridge wedge concern still respected); recovery
algorithm bit-identical.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:28:14 -07:00
MattJackson f28b6ee6d9 0.18 round 2: re-export Pipeline + Sink + Flow at the crate root
Round 2 #1 landed `Pipeline` / `Sink` / `Flow` / `DEFAULT_PIPELINE_DEPTH`
in `crate::io::pipeline` but only re-exported them through
`crate::io` (which is `pub(crate)`), so no out-of-tree consumer could
reach them. autorip's round 2 #2 (lifting the mux loop onto Pipeline +
MuxSink) is the first such consumer; surface the primitives at the
crate root for ergonomic access.

No behaviour change — the items themselves are unchanged from round
2 #1; this is just `pub use` plumbing.

Single contributor: MattJackson.
2026-05-09 10:22:44 -07:00
MattJackson d9d778fa64 0.18 round 2: confirm DiscStream as FrameSource
Mirror of the FrameSink concrete migrations slice (f52e4c5) on the
read side. DiscStream is the only meaningful source impl in tree;
all the mux/* impls are sinks.

The round-1 blanket impl<T: Stream + Send> FrameSource for T
already covers DiscStream if it's Send. This slice:

- Audits DiscStream's interior types for Send (its Box<dyn
  SectorReader> already requires Send via the trait's super-bound;
  verify nothing else interior breaks Send).
- Adds a synthetic-input test that constructs Box<dyn FrameSource>
  over a DiscStream, reads frames through the trait object, and
  exercises info() / headers_ready() / codec_private().
- (Conditional) Adds a direct FrameSource impl on DiscStream only
  if call-site ergonomics demand it; otherwise relies on the
  blanket.

No caller migrated. mux::resolve::input still returns
Box<dyn Stream>; autorip / CLI consumers still call Stream::read.
Per-caller migration is a later slice.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:13:14 -07:00
MattJackson bbfb887a35 0.18 round 1+2 integration fixes
Two clippy issues surfaced when round 1 polish + round 2 FrameSink
migrations both landed on libfreemkv main:

- src/halt.rs: clippy::new_without_default fires when a public new()
  exists without Default. The polish pass dropped the derive thinking
  it was redundant — clippy disagrees, so add a manual impl that
  forwards to new(). Doc-comment notes why both exist.

- src/disc/read_error.rs:372: pre-existing
  assert_eq!(.., true) trips clippy::bool_assert_comparison. Pre-0.18
  precommits passed because that lint sat outside the gate; the
  round-2 commits brought enough new clippy surface that it now
  shows up. Trivial cleanup: assert!(...) instead of assert_eq!.

Single contributor: MattJackson.
2026-05-09 10:00:06 -07:00
MattJackson 9884346c24 0.18 round 2: FrameSink concrete migrations for mux/* sinks 2026-05-09 09:53:12 -07:00
MattJackson 766b7c6636 0.18 round 1 polish: address libfreemkv code-review findings 2026-05-09 09:53:06 -07:00
MattJackson 925c30686b 0.18 round 1 polish: address libfreemkv code-review findings
Applies must-fix + in-scope should-fix items from the round-1 code
review:

- M1: FileSectorSource::open takes &Path (was &str — non-UTF-8 panic)
- M2: drop FileSectorSource's BufReader (defeated by absolute seeks)
- M3: WritebackFile Drop impl finalises the writeback pipeline
- M4: Pipeline::finish preserves panic payload in error message
- M5: pes::Stream is left without a : Send supertrait — concrete
  in-tree impls (MkvStream, M2tsStream) hold Box<dyn Read> /
  Box<dyn Write> trait objects that aren't Send, so the simple
  trait tightening would cascade into a wider Send audit. Per the
  review's escape clause the FrameSource blanket impl keeps its
  T: Send bound and the constraint is documented loudly there.
- S6: document Pipeline::send post-Flow::Stop semantics
- S9: truncate stale Stream docs (E9001/E9000 was runtime-only)
- S10: document WritebackPipeline.fd lifetime invariant
- S11: pub use pes::Stream as PesStream to disambiguate from
  disc::Stream codec enum at crate root
- S12: rename DEFAULT_DEPTH → DEFAULT_PIPELINE_DEPTH; add
  WRITE_THROUGH_DEPTH constant
- N14: drop Halt's Default derive (redundant with Halt::new)
- N17: Pipeline::spawn propagates thread-spawn error instead of expect
- N19: deprecation since = "0.18.0" (was "0.18.0-dev", non-conventional)
- N21: rename Apply enum to Flow

Deferred to follow-up commits: SectorReader/SectorSource competition
(migration commit), WritebackFile::create/open orphans (migration
commit), AACS round-trip test (design doc defers), various nits.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:52:25 -07:00
MattJackson f52e4c5d22 0.18 round 2: add FrameSink impls to concrete mux sinks
Per-impl migration of MkvStream / M2tsStream / NetworkStream /
NullStream / StdioStream from the deprecated pes::Stream trait
to the typed pes::FrameSink trait. Both impls coexist during
the 0.18 deprecation window — the existing Stream impls are
unchanged.

The FrameSink::finish signature differs (Box<Self> vs &mut self),
which is why this couldn't be a blanket impl. Each migration
re-borrows the box and delegates to the underlying Stream::finish
body.

FrameSink: Send forced two struct fields (M2tsStream's boxed
Write/Read, MkvStream's boxed WriteSeek/Read) to gain `+ Send`
bounds — minimum surface needed to make the Send-bounded trait
impl-able. mux::resolve::output's local Box<dyn WriteSeek>
construction picks up the same `+ Send`. tests/streams.rs's
shared `stream.write/.finish/.info/.read` calls were
disambiguated to `PesStream::*` to resolve the now-multiple
candidates from coexisting trait impls.

Caller migration (mux::resolve::output return type, autorip,
CLI) is a later slice. This commit only adds new impls; nothing
removed.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:49:18 -07:00
MattJackson 8d67790d2e 0.18: thread WritebackFile rename through FileSectorSink
The SectorSource/Sink agent and the WritebackFile-rename agent both
branched from main concurrently; the sector branch wrote against the
0.17 Writer name and only the rename branch knew about WritebackFile.
This integration commit reconciles the two: FileSectorSink::create /
::open / the inner-field type all use WritebackFile directly, and the
module-level + struct-level docs are corrected.
2026-05-09 09:17:39 -07:00
MattJackson 246a439990 0.18: FrameSource/FrameSink trait split (deprecate Stream) 2026-05-09 09:13:39 -07:00
MattJackson bd19041648 0.18: SectorSource/SectorSink trait split + DecryptingSectorSource decorator 2026-05-09 09:13:34 -07:00
MattJackson 198268b725 0.18: add crate::io::Pipeline + Sink trait
# Conflicts:
#	src/io/mod.rs
2026-05-09 09:13:30 -07:00
MattJackson 31424c8203 0.18: add crate::halt::Halt cancellation token 2026-05-09 09:13:07 -07:00
MattJackson c747ecc589 0.18: rename crate::io::Writer → WritebackFile 2026-05-09 09:12:59 -07:00
MattJackson 283a561c12 0.18 primitive: SectorSource/SectorSink trait split + DecryptingSectorSource
Splits the unidirectional read trait from a (planned) write trait at
the sector level, eliminating runtime "wrong direction" potential.
Keeps SectorReader alive as a pre-deprecation alias via blanket impl
so existing callers compile unchanged through the migration window.

Adds DecryptingSectorSource decorator: wrap any SectorSource in this
to get plaintext sectors out. Replaces the duplicate decrypt code
paths in sweep_pipeline and DiscStream (those migrations are
follow-up commits).

The formal #[deprecated] attribute on SectorReader is held back to a
follow-up commit because internal call sites in disc/, udf/, mux/,
and verify/ still go through the legacy trait, and the CI gauntlet
treats deprecation lints as errors. Behavioural intent — "this trait
is going away" — is documented on the trait itself.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:04:16 -07:00
MattJackson 2667d68675 0.18 primitive: FrameSource/FrameSink trait split (deprecate Stream)
Splits the bidirectional pes::Stream into one-direction traits so
calling read() on a write-only sink is a compile error instead of
runtime E9001. Keeps Stream alive as deprecated through 0.18 with a
blanket FrameSource impl so existing concrete types compile unchanged.

FrameSink can't be blanket-impl'd from Stream (different finish
signature), so concrete impls migrate per-type in a follow-up.

Concrete `impl pes::Stream for X` blocks in mux/* and the existing
tests gain a one-line `#[allow(deprecated)]` to keep `-D warnings`
clean during the deprecation window — no behavior changes.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 08:57:48 -07:00
MattJackson 57f2c22e30 0.18 primitive: crate::io::Pipeline + Sink trait
Generic bounded producer/consumer pipeline. The same shape applies to
sweep, patch, and mux today via three near-duplicate implementations
(or, in patch's and mux's case, no implementation at all). 0.18
collapses them onto one primitive. See
(internal)/memory/0_18_redesign.md for full context.

Single contributor: MattJackson.
2026-05-09 08:56:31 -07:00
MattJackson 5f3545d244 0.18 primitive: rename crate::io::Writer → WritebackFile
The type's job is the bounded-cache writeback pipeline (sync_file_range
+ posix_fadvise(DONTNEED)) — not generic writing. The 0.17 name was
ambiguous; reading `Writer::new(file)` gave no hint about what was
special. New name makes the role obvious at every call site.

Adds `WritebackFile::create(path)` and `WritebackFile::open(path)`
constructors so callers don't have to assemble a `File` first.

No alias kept; this is a clean 0.18 rename. See
(internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 08:53:17 -07:00
MattJackson 6e17ef0859 0.18 primitive: crate::halt::Halt cancellation token
One-bit cooperative cancellation flag. Replaces ad-hoc Arc<AtomicBool>
patterns scattered across libfreemkv (DiscStream::set_halt) and the
HALT_FLAGS global registry in autorip. See
(internal)/memory/0_18_redesign.md.

Single contributor: MattJackson (no attribution trailers).
2026-05-09 08:50:13 -07:00
MattJackson 6ec97af104 v0.17.13: thread Writer through patch + mux for big-write consistency
The bounded-cache writeback wrapper (crate::io::Writer) was added in
0.17.10 and wired into Disc::sweep in 0.17.11, but the other two
paths in the crate that write large amounts of data sequentially —
Disc::patch and the MKV/M2TS mux — were still operating on raw
std::fs::File. That meant the dirty-page burst pathology the wrapper
exists to prevent could still bite on slow / network-attached staging
during recovery and mux phases.

This release plugs those gaps:

- Disc::patch (disc/mod.rs:1981) now wraps the reopened ISO in
  Writer before any seek / write. sync_all on Writer cleanly drains
  the in-flight chunk before the existing fsync.
- mux/resolve.rs MKV and M2TS branches wrap the output File in
  Writer underneath BufWriter. UHD MKV mux routinely produces 70+ GB
  of sequential output; the page cache no longer absorbs that as a
  single hot blast on slow targets.

Mapfile, log, settings, history, and stream-pipeline byte buffers
remain unchanged: those are either small one-shot writes (where
the wrapper has zero benefit and adds a stream_position syscall) or
already use bounded persistence (mapfile time-batched in 0.17.12).
The principle: any path that writes substantial sequential data to
a single file uses Writer; trivial writes don't.
2026-05-09 06:32:08 -07:00
MattJackson 3a6c1aa5a3 v0.17.12: mapfile time-batched persistence — unblock NFS staging
Pre-0.17.12 every Mapfile::record() persisted the full mapfile via
tempfile-create + write + atomic-rename. On local LVM that's
microseconds; on NFS each rename is multiple RPCs through the
unraid user-share fuse layer, dragging a Black Mass UHD rip from
~11 MB/s on local to ~1.5 MB/s on NFS — the mapfile path alone burned
multiple seconds of wall time per real-world second of work.

Mapfile now batches the rename to once per second:

- record() always updates in-memory state and stats; only fires
  write_to_disk when last_flushed.elapsed() >= FLUSH_INTERVAL (1 s).
- New flush() API forces a persist; called by sweep_pipeline's
  consumer at end-of-sweep and by Disc::patch at end-of-patch,
  after the file's sync_all.
- Drop impl best-effort flushes so an early-return / unwind doesn't
  silently lose pending state.

Crash-safety changes from "lose at most one block" to "lose at most
1 s of recorded progress" — the ISO file's payload bytes are unaffected;
only the mapfile's authority over which sectors are already-good is at
risk, and a resume re-reads anything Pass 1 had already covered.

Measured on the BU40N test bed against Black Mass UHD inner zone:
- NFS staging: 1.5 MB/s → 16.48 MB/s (10.9× recovery)
- Local LVM staging: 11.09 MB/s → 11.83 MB/s (+6.7 % bonus)

Internal round_trip_load test now flushes before reading back from
disk. External patch / copy tests are unaffected: patch and
sweep_pipeline flush at completion before returning.
2026-05-08 23:00:52 -07:00
MattJackson 1ba3264747 v0.17.11: sweep producer/consumer split — overlap drive read with file write
Pre-0.17.11 sweep ran strictly serialised: SCSI read → decrypt → seek
+ write → mapfile.record → next read. Drive idled for the post-read
work; throughput capped at the sum of both costs. On a healthy disc
that's ~7-12 ms read + ~5-15 ms write/record per 64 KB batch, limiting
sustained throughput to ~10-12 MB/s on the test bed (BU40N + UHD inner
zone), well below the ~14-16 MB/s drive ceiling.

Decouples them: producer thread (caller's) owns SectorReader +
read_error state + decrypt + set_speed + halt; consumer thread (one
spawn) owns Writer + Mapfile, receives WorkItem messages, applies
file write + mapfile record. Bounded mpsc::sync_channel(4) gives
natural back-pressure. While the consumer writes batch N, the
producer is already reading batch N+1 — steady-state throughput is
now bound by the slower of the two pipelines (drive on healthy
discs), not their sum.

Side effects:
- Bisect path now decrypts. Pre-0.17.11 the bisect inner loop wrote
  raw cyphertext for single-sector recoveries on encrypted discs —
  quiet correctness bug exercised only by batch-fail-then-
  bisect-succeed on encrypted media. New producer-side decrypt
  covers main + bisect success paths uniformly.
- All read_ctx state stays single-threaded on producer (damage
  window, jump multiplier, etc.). No locking added.
- Mapfile remains single-writer on consumer. No locking.
- Halt latency: producer breaks loop, sends Finish, consumer drains
  ≤4 in-flight items + sync_all. ~1 batch (~12 ms) typical.
- BU40N + Initio bridge wedge concern unchanged: still single SCSI
  command in flight, error-path timing identical, no new retries.

New module: src/disc/sweep_pipeline.rs (WorkItem, ProgressSnapshot,
ConsumerInputs, spawn_consumer, consumer_loop, helpers). Public API
unchanged — Disc::copy / CopyOptions / CopyResult identical.

Patch (Pass N) is NOT changed; it's bound by drive recovery time, not
the read/write serialisation.
2026-05-08 21:32:11 -07:00
MattJackson ae2909fe8d v0.17.10: bounded-cache writeback pipeline for big sequential writes
Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
settings. Empirical cause: the kernel's vm.dirty_ratio (~20% of RAM)
lets hundreds of MB of dirty pages accumulate, then bursts a flush at
99% disk utilisation that blocks app writes for ~1 s. Confirmed on
the BU40N test bed — dirty pages grew 112 → 563 MB between bursts;
lowering vm.dirty_bytes to 64 MB at the host sysctl level eliminated
the dips. Shipping the equivalent inside libfreemkv so users do not
need to tune the host kernel.

- New crate::io::Writer: drop-in File wrapper (impl Write + Seek).
  Wraps a per-platform WritebackPipeline that on Linux schedules
  sync_file_range(WRITE) + lagging sync_file_range(WAIT_AFTER) +
  posix_fadvise(DONTNEED) in 32 MB chunks, bounding dirty cache at
  ~64 MB. macOS and Windows ship a no-op stub.
- Disc::sweep wraps its output File in Writer. Loop body unchanged.
- Module is purpose-built so any large sequential output (patch,
  mux) can adopt the same wrapper as a one-line change later.
2026-05-08 19:54:25 -07:00
MattJackson 6a3d19a453 v0.17.7: sync release — no functional changes
Version bump to keep the four freemkv crates at unified versioning
after autorip's v0.17.6 + v0.17.7 work today. No libfreemkv code
changes; republished to crates.io so downstream consumers stay
aligned on the latest patch version.
2026-05-08 16:20:28 -07:00
MattJackson a35596d2d1 v0.17.5: Pass N kernel block-device fallback + per-range fixes
Direct-SATA BU40N + Dune Part Two UHD live testing exposed that the
v0.17.3 single-shot SCSI READ path matched 0/22 of the small bad-
sector LBAs that dd if=/dev/sr0 recovers on the same drive. This
release closes that gap and fixes adjacent bugs silently capping
recovery.

- /dev/sr0 pread fallback in Drive::read (Linux only): on SCSI READ
  Err, fall back to posix_fadvise(DONTNEED) + pread() against the
  corresponding block device. Kernel sr_mod runs ~5 internal retries
  with no per-attempt mid-layer escalation overhead — the mechanism
  behind dd's recovery advantage. End-to-end byte verification
  confirms the fallback path returns real disc data.

- Disc::patch per-range watchdog fix: MAX_RANGE_SECS was breaking
  'outer (one slow range killed the entire patch). Now skips to the
  next range. Pre-fix patch died after 4 sectors of range 1 of 47.

- Per-sector range budget: range_budget = sectors × 25 s, capped at
  1800 s. Replaces the flat 180 s/range that was unfair to medium
  ranges and pointlessly generous to single-sector ones.

- consecutive_failures resets per range. The wedge-exit detector is
  for stuck-on-one-range, not many-small-ranges-with-one-fail-each.

- Reverted inline 5× retry experiment (was hurting: each retry paid
  kernel SCSI escalation overhead). Restored READ_RECOVERY_TIMEOUT_MS
  to 60 s. The kernel-auto-retry pattern is now provided by sr0
  fallback.

Empirical: pass 1 recovered 94.6 MB / 11 s of main title (33 sr0
saves). Pass 2 added 0.6 MB. Remaining ~233 MB on the test disc
appears physically unrecoverable on this hardware.
2026-05-08 12:58:48 -07:00
MattJackson 318f654fed v0.17.3: fix scsi_recovery.rs Linux compile (missing Duration import)
tests/scsi_recovery.rs:
- Add `use std::time::Duration` inside both `#[cfg(target_os = "linux")]`
  blocks. Locally on macOS the linux blocks are cfg-out so the missing
  import was invisible to precommit on macOS.
- Bug pre-dated this branch but only surfaced when v0.17.2 release CI
  ran the test compile on Linux.

Cargo.toml: 0.17.2 -> 0.17.3.
2026-05-07 19:16:21 -07:00
MattJackson 5e82ef65de v0.17.2: bump version (skip 0.17.1 — stray tag exists)
Cargo.toml: 0.17.1 -> 0.17.2. Functionally identical to the prior
commit; 0.17.1 was never published to crates.io but a tag exists on
the remote pointing at an unrelated commit. Bumping past it.
2026-05-07 19:10:39 -07:00
MattJackson 8534607329 v0.17.1: cache priming, NonTrimmed marking, decrypt regression test
src/disc/mod.rs:
- Cache priming (3-sector lookback) before patch's single-sector reads.
  Drive read-ahead pulls in adjacent pages so the target may already be
  cached when we ask for it. Throwaway reads — failures here don't
  update mapfile state.
- When patch hits skip-limit on a range, leave remaining sectors
  NonTrimmed instead of marking Unreadable. We never tried to read those
  sectors, so don't give them terminal status — drive state evolves
  between passes (cache, mechanical settle), and a later pass may
  succeed.

tests/pass_n_patch_fix.rs:
- New regression test for the decrypt key inversion bug at
  src/disc/mod.rs:1938-1942. Asserts decrypt_sectors is invoked with
  the correct key when opts.decrypt=true.

tests/pass_n_size_aware_skip.rs:
- rustfmt-only changes.

Cargo.toml: 0.17.0 -> 0.17.1.
2026-05-07 19:09:17 -07:00
MattJackson 97e1a4cad3 unified read-error handler + pass N size-aware skip
New disc/read_error.rs as the single entry point all read failures flow
through. Handler classifies the error, updates the in-flight context
(damage window, retry budgets, jump multiplier), and returns a
ReadAction the caller dispatches on. Pass 1 (sweep) refactored to use
it; ~340 lines of nested if/else collapsed into ~120 lines of action
dispatch. Adding a new error class = one match arm. Logging is in one
place. Bisect inner failures don't poison the damage window. Jump
multiplier capped at 64 (max 1 GB jump for batch=32 — observed prior
unbounded behavior produce a single 56 GB jump on a wedged drive).

Pass N (patch) damage_skip is now size-aware: each skip is capped at
range_remaining/4 rather than the absolute MB-scale escalation. The
old logic could leap over a 100-sector bad range that hides a 50-sector
good middle; size-aware convergence finds the good middles instead.

Tests in tests/pass_n_size_aware_skip.rs exercise the size-aware skip
against synthetic patterns (25-bad/50-good/25-bad and three good
middles in a row) and prove ≥98% of good middles are recovered.
Existing test test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed
updated to reflect that MEDIUM_ERROR now triggers single-sector
bisect (which the BlockSizeFailingReader succeeds at).
2026-05-07 09:08:43 -07:00
MattJackson 3f6bf33de5 Format code 2026-05-04 20:06:10 -07:00
MattJackson 359301e8fc Pass 1 transport failure recovery without user intervention 2026-05-04 20:02:01 -07:00
MattJackson bfa527162a 0.17.0: unwrap safety fix, patch pass algorithm, clippy compliance
- Fix unwrap in disc/mod.rs sweep() hot path using pattern matching
- Patch pass excludes Unreadable sectors from work list
- Expose bytes_bad_in_title for accurate UI reporting
- All 256 tests pass, cargo clippy clean with -D warnings
2026-05-04 09:42:07 -07:00
MattJackson 45defd47f3 fix patch pass: exclude Unreadable from work list; expose bytes_bad_in_title; clippy 1.86 fixes
- patch(): only process NonTrimmed + NonScraped ranges (Unreadable=terminal, NonTried=not-yet-swept)
- bytes_bad_in_title: pub fn for autorip main-movie lost_ms computation
- Clippy 1.86: saturating_sub, unused assignments, unused variable
- fmt: rustfmt formatting
2026-05-03 16:35:06 -07:00
MattJackson fe8933dad2 v0.16.3: i18n damage display
- Add rip.damage_lost, rip.damage_lost_movie, rip.damage_lost_simple, rip.damage_none keys
- Update all 6 locale files (en, es, fr, de, it, pt, nl)
- CLI progress shows 'Xs lost (Y in movie)' instead of 'Xs unreadable'
- Perfect rips show '0s' instead of '0s unreadable'
2026-04-30 20:51:02 -07:00
MattJackson 0a3c1b7b70 fix: scsi_recovery test variable shadowing and missing imports for Linux CI 2026-04-30 19:57:16 -07:00
MattJackson 2bcdc97341 fix: add missing Duration import in scsi_recovery test 2026-04-30 19:52:53 -07:00
MattJackson 4863e9c545 v0.16.2: sticky escalation in patch, title-aware damage display, PASS1/PASSN constant naming 2026-04-30 19:50:08 -07:00
MattJackson 03db038dd9 docs: update project docs, CHANGELOG, drive-access.md, rip-recovery.md for v0.16.x 2026-04-30 15:43:58 -07:00
MattJackson 95001ba2c3 v0.16.1: bump version 2026-04-30 15:23:44 -07:00
MattJackson a1f4dff6f6 v0.16.0: IOKit registry-based drive enumeration, BSD name → IOBDServices matching, reverse patch default 2026-04-30 15:17:20 -07:00
MattJackson 7dd5001d45 v0.15.1: fix damage-jump detection (window=16, threshold=12%), fix dispatch covers_disc check, sweep resume from mapfile
- DAMAGE_WINDOW 50→16, DAMAGE_THRESHOLD_PCT 25→12: triggers on 2nd scattered failure
- Previous 50/25% was too diluted by good reads between sparse failures
- copy() dispatch checks mapfile total_size == disc capacity_bytes (covers_disc)
- NonTried regions → sweep with resume=true (preserves mapfile)
- Only NonTrimmed/NonScraped/Unreadable → patch
- sweep_internal takes resume param: true when dispatching from existing mapfile
- Verified: Pass 1 from 30-100% completes in ~20 min with correct jumps through 3 damage zones
2026-04-30 13:15:03 -07:00
MattJackson 67fe93c0b8 v0.15.0: multipass CopyOptions, auto-detect sweep vs patch, speed control on damage zone entry/exit
- CopyOptions: replace resume/skip_on_error/batch_sectors with single multipass bool
- Disc::copy() auto-detects pass from mapfile state: no mapfile or NonTried → sweep, only NonTrimmed/Unreadable → patch
- Fix bug where mapfile with NonTried regions incorrectly dispatched to patch mode
- SectorReader::set_speed() default method, Drive impl sends SET CD SPEED
- On damage zone entry: set_speed(0x0000) for better error recovery
- On damage zone exit (50 consecutive good): set_speed(0xFFFF) to restore max speed
- Disc::mapfile_for() returns /tmp/<name>.mapfile for null:// output
- patch_internal/sweep_internal as private helpers, CopyResult gains recovered_this_pass
2026-04-30 11:31:31 -07:00
MattJackson 5c4f575c46 v0.14.0: bump version 2026-04-30 08:45:32 -07:00
MattJackson e2cb7a78bb v0.13.46: damage-jump algorithm replaces probe, bridge degradation detection, ecc_sectors() 2026-04-29 22:08:52 -07:00
MattJackson a6f1bd19bc v0.13.45: multipass adaptive probe, NOT_READY retry, progress bytes_bad_total
- Adaptive probe algorithm in Disc::copy skip_on_error mode: after 4
  consecutive errors, probe 1 sector at 256x batch (8 MB) ahead. If
  good, zero-fill gap, mark NonTrimmed, jump. Clears bad zones in
  seconds instead of hours.
- NOT_READY sense key (0x02) now retries up to 3x with 3s pause before
  marking NonTrimmed. BU40N returns NOT READY for bad sectors, not
  MEDIUM ERROR.
- PassProgress struct gains bytes_bad_total field for consumer-side
  bad/retryable byte counts.
- Mapfile header version string fix: no longer duplicates 'libfreemkv v'
  prefix on each write.
- Structured sense_key/asc/ascq logging in copy error path.
2026-04-29 19:41:34 -07:00
MattJackson 9e3d0f1383 v0.13.44: macOS raw CDB transport via IOKit exclusive access
macOS SCSI transport rewritten from hybrid MMC+pread to single-path
raw CDB dispatch through SCSITaskDeviceInterface. All CDBs (INQUIRY,
READ, REPORT KEY, etc.) now go through ExecuteTaskSync — 1:1 with
the Linux SG_IO backend.

Key changes:
- New macos_shim.c: diskutil unmount → find IOBDServices →
  ObtainExclusiveAccess → raw CDB dispatch. Eliminates Rust-side
  IOKit COM vtable complexity.
- build.rs compiles macos_shim.c via cc into static lib
- macos.rs simplified to three FFI calls (open/close/execute)
- disc/mod.rs: graduated batch restore after errors, skip-ahead
  through bad zones, configurable error pause
2026-04-29 15:59:35 -07:00
MattJackson 1e6eb0698d v0.13.43: Pass 1 transport-failure recovery loop 2026-04-29 08:53:56 -07:00
MattJackson bd744171e4 v0.13.42: transport failure skips instead of aborting copy 2026-04-29 07:46:42 -07:00
MattJackson 7de9d4d42c v0.13.41: debug logging for sector-0 regression 2026-04-29 07:09:46 -07:00
MattJackson 9ac0c2fcdf v0.13.40 2026-04-28 21:50:32 -07:00
MattJackson 3aa27d5d1b fix CI: Rust 2024 match ergonomics - remove ref binding modifier 2026-04-28 21:47:19 -07:00
MattJackson 5083f70cff fix CI: import ScsiTransport trait in integration test 2026-04-28 21:45:34 -07:00
MattJackson 477cf7ee0e fix CI: pub scsi::linux module and SgIoTransport fields for integration tests 2026-04-28 21:43:14 -07:00
MattJackson 0b079e27ce fix CI: make scsi::linux pub(crate) for integration test access 2026-04-28 21:39:50 -07:00
MattJackson 5f69fd2a44 update CHANGELOG and README for v0.13.39 2026-04-28 21:36:37 -07:00
MattJackson befe6ef69b remove (internal) references from public code 2026-04-28 21:32:15 -07:00
MattJackson 29491d6eaf v0.13.39 2026-04-28 21:19:51 -07:00
MattJackson 2e602f7e37 fix scsi_recovery test: dereference status in pattern match 2026-04-28 21:17:11 -07:00
MattJackson a126f31b70 v0.13.38 2026-04-28 21:12:05 -07:00
MattJackson 6ff10df83a fix scsi_recovery test compilation (old Drive::read signature) 2026-04-28 21:11:37 -07:00
MattJackson 52b8522a75 v0.13.37: Pass 1 is pure ECC-block sweep — read 32 sectors, fail → skip, no single-sector reads 2026-04-28 21:08:58 -07:00
MattJackson 13eb4331b6 v0.13.36: remove unused reset() 2026-04-28 17:57:25 -07:00
MattJackson 1a3f7ce676 remove unused reset() 2026-04-28 17:56:42 -07:00
MattJackson e1a1f730be v0.13.35: pause 3s after first MEDIUM ERROR before retry to prevent USB bridge crash 2026-04-28 17:54:12 -07:00
MattJackson 646c22ae93 v0.13.34: open() just opens, drive_has_disc() is side-effect-free direct TUR, enumerate_sg_names skips unreadable type files, Disc::copy read_err_count fix 2026-04-28 16:00:00 -07:00
MattJackson 2f6092a686 scsi/linux: fix as_bytes() → as_encoded_bytes() for OsStrExt
- as_bytes() requires OsStrExt import which is platform-specific
- as_encoded_bytes() is the portable API available on all platforms
- Fixes CI failure on Linux (Ubuntu) in GitHub Actions
2026-04-28 14:59:55 -07:00
MattJackson e5a90a6567 disc: fix hysteresis, SgIoTransport recovery, wallclock budget, patch instrumentation
- Fix hysteresis: use_single now correctly forces block_count=1 (was computed before the check)
- Fix SgIoTransport: spawn close+reopen in background thread on transport error
- Fix autorip: rip_disc() spawns wallclock watcher thread, caps entire rip at max(disc_runtime, 1h)
- Fix Disc::patch: add unreadable_count counter + tracing instrumentation
- Simplify Disc::copy() error handling per RIP_DESIGN.md §2.1
2026-04-28 14:48:16 -07:00
MattJackson afce1031d5 disc: add eprintln debug 2026-04-27 23:46:34 -07:00
MattJackson 2c0bff1303 disc: add debug logging for error type 2026-04-27 22:39:03 -07:00
MattJackson 2067ebe485 disc: add warn trace for MEDIUM ERROR skip 2026-04-27 21:38:03 -07:00
MattJackson 2ce4c20221 disc: skip MEDIUM ERROR sectors instead of bailing
0.13.28 - when drive returns MEDIUM ERROR (bad sector), skip the sector
and continue instead of retrying or bailing. Write zero-fill, mark as
Unreadable for pass 2+ recovery.

Closes: (internal)#20260427
2026-04-27 20:44:59 -07:00
MattJackson da3004b120 disc: DiscRead carries SCSI status/sense
Fix extract_scsi_context() and Error::scsi_sense() to handle Error::DiscRead
in addition to Error::ScsiError, so is_marginal_read() works for DiscRead
errors and disc::copy() can properly route MEDIUM ERROR as a marginal
(bad sector) instead of bailing.

Closes: (internal)#20260427
2026-04-27 18:38:57 -07:00
MattJackson b5ed97801b libfreemkv: extend DiscRead with SCSI status/sense for 30% wedge diagnostics 2026-04-27 16:47:27 -07:00
MattJackson ae76aaf0fa v0.13.24 — MapStats: split bytes_pending into nontried / retryable
bytes_pending was an opaque aggregate of NonTried + NonTrimmed +
NonScraped. UIs that wanted a "will retry in Pass 2-N" bucket were
stuck showing the entire unread disc as Maybe at pct=0.

Adds two granular fields to MapStats:

  bytes_nontried   — Pass 1 hasn't read these yet
  bytes_retryable  — NonTrimmed + NonScraped, Pass 2-N will retry

bytes_pending stays for back-compat (= bytes_nontried + bytes_retryable).

Also picks up the cargo fmt --check lint that's been red on main CI
since v0.13.18 (rustfmt fold differences on a few long format-string
layouts; functional no-op).
2026-04-26 19:28:35 -07:00
MattJackson 2cd4fbead7 v0.13.23 — stop discarding the drive's SCSI sense data
Through the entire 0.13.x line, every CHECK CONDITION reply from the
drive (the standard way SCSI tells you why a sector failed) was being
collapsed into a synthetic status=0xFF, sense_key=0 transport-wedge
sentinel and the actual sense data was thrown away. Confirmed live on
the BU40N reading Dune 2 on 2026-04-27: drive returned host_status=0,
driver_status=8, status=2, exec_elapsed_ms=1416 on every bad sector
— a clean CHECK CONDITION carrying full sense data — and Disc::copy
was bailing on it as if the bridge had wedged.

Root cause: scsi/linux.rs's wedge check was
  `host_status != 0 || driver_status != 0`
SG's DRIVER_SENSE bit (0x08) is set on every CHECK CONDITION reply
just to flag "sense buffer is populated" — it's not a transport
failure on its own. Pre-fix we conflated the two and silently lost
every drive-reported error reason. macOS and Windows backends had
the same shape: they extracted sense_key only, dropping ASC/ASCQ.

API restructure (clean separation):

  Error::ScsiError {
      opcode: u8,
      status: u8,                  // 0xFF = synthetic transport-failure
      sense: Option<ScsiSense>,    // None ⇔ no sense delivered
  }

  pub struct ScsiSense { sense_key: u8, asc: u8, ascq: u8 }
  impl ScsiSense {
      pub fn is_marginal(&self) -> bool       // keys 0/1/3/B
      pub fn is_medium_error(&self) -> bool
      pub fn is_hardware_error(&self) -> bool
      pub fn is_unit_attention(&self) -> bool
      pub fn is_data_protect(&self) -> bool
      pub fn is_not_ready(&self) -> bool
      pub fn is_illegal_request(&self) -> bool
      pub fn is_aborted_command(&self) -> bool
  }

  impl Error {
      pub fn scsi_sense(&self) -> Option<&ScsiSense>
      pub fn is_scsi_transport_failure(&self) -> bool
      pub fn is_marginal_read(&self) -> bool
  }

SCSI protocol constants (SCSI_STATUS_*, SENSE_KEY_*) moved from
error.rs to scsi/mod.rs where they belong alongside SCSI_INQUIRY,
SCSI_READ_10, etc. parse_sense replaces parse_sense_key (returns the
full triple, not just the key); inline tests now exercise ASC/ASCQ
extraction at the right offsets for both descriptor (0x72/0x73) and
fixed (0x70/0x71) sense formats.

Disc::copy + Disc::patch sense-aware dispatch:
  - marginal sense (MEDIUM ERROR / ABORTED COMMAND / RECOVERED ERROR
    / NO SENSE) → engage hysteresis (Block→Single, bpt=1)
  - non-marginal sense (HARDWARE / DATA PROTECT / UNIT ATTENTION /
    NOT READY / ILLEGAL REQUEST / transport failure / kernel
    IoError) → bail with full sense info preserved; caller (autorip)
    surfaces "physical replug" / "drive failing" / "media changed"

  Pre-fix: every CHECK CONDITION → 0xFF synthetic → Disc::copy bailed
  → bytes_good froze at the bad zone. The hysteresis from v0.13.22
  was correct but never got to run. This release unblocks it.

  Disc::patch's wedged_threshold (50 consecutive failures) stays as
  defense-in-depth for chains of marginal failures; a single
  non-marginal sense now short-circuits it.

New phase=bail trace event records the bail reason with the sense
triple. phase=transport_err remains for genuine bridge wedges /
kernel timeouts; phase=scsi_err carries the parsed sense_key, asc,
ascq for drive-reported errors.

All 350 tests pass. Clippy clean across all targets.
2026-04-26 19:06:19 -07:00
MattJackson ebffc6eb88 v0.13.22 — replace bisect-on-fail with hysteresis Block↔Single
The v0.13.21 bisect-on-fail recovery was correct (100% of recoverable
sectors picked up) but slow on dense damage clusters. Live test on
Dune 2 v0.13.21 burned ~30 s per damaged 60-block — paying a ~5 s
kernel ABORT/timeout at every level of a log₂(60) ≈ 6 deep DFS, on
the failing branch each time.

Replaced with a two-state hysteresis machine in Disc::copy:

  Block(batch):
    read(batch) ok    → write, advance, stay Block
    read(batch) fail  → switch to Single, retry SAME range at bpt=1

  Single:
    read(1) ok    → write, consecutive_good++
                    if consecutive_good >= BPT1_EXIT_THRESHOLD:
                      switch to Block, reset counter
    read(1) fail  → mark NonTrimmed, consecutive_good = 0

BPT1_EXIT_THRESHOLD = 10_000 sectors (= 20 MB clean run). Calibrated
from the 2026-04-26 BU40N empirical probe data; tunable.

Per-block math on a damaged 60-block with 1 truly bad sector:

  Bisect      (v0.13.21): ~30 s  (5 s × 6 levels)
  Hysteresis  (v0.13.22): ~10 s  (5 s bpt=batch fail
                                  + 59 × 1 ms good
                                  + 1 × 5 s bad)

Inside a damaged cluster spanning many 60-blocks the win compounds:
hysteresis pays the bpt=batch fail cost ONCE on entry, then stays at
bpt=1 across the cluster; bisection re-paid it every 60 sectors. For
Dune 2's ~1248-sector boundary cluster that's ~21 fewer 5-sec
kernel timeouts ≈ 100 s saved per pass.

Telemetry: new phase=mode_change trace event with from, to, lba, and
consecutive_good. Replaces v0.13.21's phase=bisect. Worklist DFS is
gone — single iterative for s in 0..count on the failure path.

Test rename, same fixture and same 100% recovery expectation:
  test_disc_copy_bisect_recovers_via_single_sector_reads
  → test_disc_copy_hysteresis_recovers_via_single_sector_reads

Also adds DamageSeverity (Clean / Cosmetic / Moderate / Serious) +
classify_damage(bad_sectors, lost_ms), re-exported from libfreemkv,
so applications can render structured severity instead of formatting
their own from raw counters.
2026-04-26 17:27:57 -07:00
MattJackson 424d3cd4f2 v0.13.21 — bisect-on-fail in Disc::copy + 10s caller READ timeout
Fixes the BU40N wedge cycle that has been chasing us through
v0.13.18-20. Two changes, both backed by empirical live-hardware
probes recorded in (internal)/docs/TEST_PLAN.md:

1. scsi/mod.rs: READ_TIMEOUT_MS 1500 → 10000 ms.
   Cold-start seek on the BU40N takes ~1.5 s. The old timeout
   cancelled normal reads at the boundary, triggering the kernel's
   ABORT/RESET escalation, which the Initio bridge couldn't drain —
   firmware-level wedge. 10 s catches every legitimate slow read
   (max successful ECC recovery: 2.6 s; cold-start: 1.5 s) with
   margin and short-circuits truly bad sectors at ~10 s.

2. disc/mod.rs: Disc::copy bisect-on-fail (replaces skip-forward).
   Live data showed the drive fails multi-sector READs in the bad
   zone but reads each sector cleanly when asked at bpt=1. Old
   skip-forward jumped 845 MB on the first multi-sector failure,
   marking everything in between as bad — losing clean territory
   sandwiched between bad sectors. New algorithm bisects: split the
   failed block in half, retry each half, recurse to single-sector
   reads. Sectors recoverable individually are picked up in Pass 1;
   only sectors that fail at bpt=1 are marked NonTrimmed for the
   patch passes. Stack-based DFS, log2(batch) = 6 levels for the
   default 60-sector batch.

Multi-pass machinery is untouched. Pass 2..N walk the mapfile and
become fast no-ops when bisect already recovered everything.
Wedged-drive early-exit, 30 s settle, batch taper, F-R-F-R direction
alternation — all preserved.

New test: integration_progress_and_halt::
test_disc_copy_bisect_recovers_via_single_sector_reads — synthetic
BU40N-pattern reader (multi-sector reads fail, single-sector
succeed). Pre-patch: lost everything to skip-forward. Post-patch:
100 % bytes_good. Plus the 10 sense-key parser tests from the
0.13.20 test-coverage pass.

Empirical recovery on Dune 2 UHD on the BU40N (per TEST_PLAN.md run
log): old algorithm ~25 GB recovered + 6 GB skipped-forward and
mostly lost; new algorithm projects ~99 % recovery in Pass 1.

Audits + raw probe data:
- (internal)/docs/TEST_PLAN.md (run log)
- (internal)/docs/audits/2026-04-26-scsi-architecture-research.md
2026-04-26 15:57:44 -07:00
MattJackson d2905ba7bb v0.13.20 — sync blocking SG_IO + cross-platform parity strip
- scsi/linux.rs: full rewrite from async write/poll/read+1.5s timeout+
  close-on-timeout to one synchronous ioctl(fd, SG_IO, &hdr). Kernel
  honors hdr.timeout and runs its own ABORT/RESET escalation. Errors
  check host_status and driver_status (both 0xFF-synthesised) plus
  status. Sense-key parser handles descriptor (0x72/0x73) + fixed
  (0x70/0x71) formats. Deleted fd_recovery, bg close+open thread, fd
  swap dance. -331/+155 lines.

- scsi/macos.rs: try_recover() removed (userspace handle-recovery on
  task failure was the same anti-pattern stripped from Linux). bsd_name
  field deleted. Errors bubble up directly.

- scsi/windows.rs: try_recover() removed, wide_path field deleted,
  INVALID_HANDLE guard removed.

- scsi/mod.rs: parse_sense_key() helper extracted (used by all three
  platforms now — single canonical sense-key parse rather than three
  inlined copies). +10 unit tests covering descriptor format, fixed
  format, truncated buffers, unknown response codes.

- drive/mod.rs: Drive::reset() deleted (escalating eject + STOP/START +
  reinit recovery — per audit, kernel handles its own escalation;
  userspace shouldn't).
  pub fn find_drives() -> Vec<Drive> deleted (opened N drives just to
  throw most away). find_drive() now uses discover_drives() directly.
  wait_ready() simplified — drops the reset path on sense_key=5,
  just keeps polling TUR for 60 iterations.

- lib.rs: find_drives re-export removed.

- benches/sgio_read.rs: switched to find_drive() (no longer iterates a
  drive list).

Net: 9 files changed, 226 insertions(+), 473 deletions(-). 329 tests
pass, clippy -D warnings clean. No consumer breakage (CLI, autorip,
bdemu compile + test green).

Architecture decision documented in
(internal)/docs/audits/2026-04-26-scsi-architecture-research.md
(primary-source survey of MakeMKV, sg_dd, ddrescue, and the kernel
mid-layer's own scsi_eh.rst escalation ladder).
2026-04-26 09:51:46 -07:00
MattJackson b4de5d343d v0.13.18 — sync release (no functional changes; autorip two-bar UI fix) 2026-04-26 07:47:37 -07:00
MattJackson 8beeac7df9 v0.13.17 — sync release (no functional changes; actual fix is autorip hot-plug) 2026-04-26 07:27:07 -07:00
MattJackson b33f41e219 v0.13.16 — single Progress trait + PassProgress (RIP_DESIGN.md §16)
Pre-0.13.16 the rip API leaked internal mapfile concepts (pos,
bytes_good, work_done, bytes_pending, Finished/NonTrimmed) into per-pass
positional callbacks Fn(u64, u64, u64). Consumers reinvented the math
each time, and v0.13.15's UI bug surfaced exactly because of this —
autorip's web JS computed pct from bytes_good while the backend
computed from pos, silent drift, frozen UI bar.

This release replaces both Disc::copy::on_progress and
Disc::patch::on_progress callbacks with a single Progress trait +
PassProgress struct (new progress module).

  pub struct PassProgress {
      pub kind: PassKind,            // Sweep | Trim {reverse} | Scrape {reverse} | Mux
      pub work_done: u64,
      pub work_total: u64,
      pub bytes_good_total: u64,
      pub bytes_total_disc: u64,
  }

  pub trait Progress {
      fn report(&self, p: &PassProgress);
  }

  impl<F: Fn(&PassProgress)> Progress for F { ... }   // closures work directly

CopyOptions::on_progress and PatchOptions::on_progress are renamed to
progress: Option<&dyn Progress>. Closure callers update trivially via
the blanket impl; struct callers gain a clean named-field shape with no
positional-arg confusion.

PassKind carries the semantic (sweep vs trim vs scrape vs mux) so
consumers can label phases without reinventing detection logic.
Disc::patch reports Trim {reverse} for retry passes with block_sectors
>= 2 and Scrape {reverse} when block_sectors == 1. Direction comes
through reverse: bool. Mux variant is reserved for v0.13.17 when the
mux pipeline emits progress.

Tests + clippy clean across all 4 crates.
2026-04-26 07:15:26 -07:00
MattJackson e85e20f436 v0.13.15 — pos in on_progress, PatchOptions::reverse, wedged_threshold
Breaking: CopyOptions::on_progress + PatchOptions::on_progress now take
Fn(bytes_good, pos, total). Consumers display `pos` for "% swept" — the
true Pass 1 progress that advances through skip-forward bad zones, where
bytes_good (Finished sectors only) freezes. v0.13.14 live trace proved
the existing UI was lying for ~14 minutes about Dune 2 being "stuck at
30%" while Pass 1 was actually 83% through the disc via skip-forward.

PatchOptions::reverse: walk bad ranges from highest LBA to lowest. For
drives that wedge after a forward read of a bad sector, approaching the
post-bad-zone NonTrimmed range from end-of-disc reads good sectors before
the drive sees a bad one. Hypothesis informed by the BU40N + Initio
bridge live data — Pass 2 forward saw zero successful reads in 7 min
while Pass 1's pos walked all the way to end-of-disc.

PatchOptions::wedged_threshold: > 0 → exit early after that many
consecutive failures with zero successes in the same pass. Saves the
wallclock budget for productive grinding when the drive has wedged on
the bad zone for THIS pass; a different direction or block size in the
next pass may still recover. New PatchResult::wedged_exit reports it.

Trace: patch_start (block_sectors, recovery, reverse, wedged_threshold,
num_ranges) and patch_done (blocks_attempted, blocks_read_ok,
blocks_read_failed, wedged_exit, halted, bytes_recovered) at the
freemkv::disc target.
2026-04-25 20:06:03 -07:00
MattJackson 7bc54be8d3 v0.13.14 — sync release, no functional changes (autorip subscriber filter fix in autorip 0.13.14) 2026-04-25 18:38:17 -07:00
MattJackson 6d9083743b v0.13.13 — telemetry: tracing instrumentation in SgIoTransport + Disc::copy
v0.13.12 shipped the async fd_recovery design but a live test on Dune 2
showed Pass 1 sat for 14 minutes with bytes_good=0 — the inner loop iterates
(throttled on_progress log fires every 78s) but each iteration evidently
takes ~60s instead of the microseconds the design promises on fast-fail.
Without trace-level telemetry at the SCSI + Disc::copy boundaries we
can't tell where the time goes.

This release is instrumentation only — no behavior change.

- New dep: tracing 0.1. Per project docs, debug/trace logging is allowed in
  libfreemkv (the no-English rule applies to errors). Consumers wire a
  tracing subscriber.
- SgIoTransport::execute (Linux): trace at every state transition (entry,
  recovery_swap_ok, recovery_pending, write_ok / write_err, poll_done,
  timeout_spawn_recovery, scsi_err, read_err, ok). Each event includes
  opcode + elapsed timing. The bg recovery thread also traces close_ms +
  open_ms so we can see if the kernel really takes 60s to close+open on a
  wedged Initio bridge.
- Disc::copy: trace at copy_start, outer_loop, region_enter, every 100
  inner-loop iterations (iter_progress with pos/region_end/skip_size/
  bytes_good/read_ok_count/read_err_count/last_read_ms/copy_elapsed_ms),
  copy_done.
- All trace events use targets `freemkv::scsi` and `freemkv::disc` so
  consumers can filter by subsystem (e.g. autorip /api/debug?q=freemkv::scsi).

Next: run the live test on Dune 2 again, read the autorip JSONL log,
diagnose why each iter is slow, fix the actual bug.
2026-04-25 18:32:05 -07:00
MattJackson c49a68054f style: cargo fmt on integration test 2026-04-25 17:36:06 -07:00
MattJackson 4bd38787c6 test(integration): make halt-on-skip-forward test deterministic
The wallclock-based halt timing failed on fast CI runners where a 2 GB
synthetic-disc skip-forward sweep finishes in <100 ms — well under the
200 ms halt fire delay. Reader now signals halt on first read; the
inner-loop halt check on iteration 2 breaks 'outer. No wallclock race.
2026-04-25 17:33:52 -07:00
MattJackson b4951b1c5b v0.13.12 — Fix 1+2+4 + cross-platform SCSI parity (RIP_DESIGN.md §6, §7, §15.1)
Fix 1: delete stall guard from Disc::copy. Pass 1 must sweep end-to-end
per ddrescue model (RIP_DESIGN.md §2.1, §3, §9). The v0.13.9 guard at
disc/mod.rs broke Pass 1 at 30% on Dune 2 with 56 GB still NonTried.
Removed stall_secs field, narrative comment in scsi/linux.rs, and the
broken regression test. Replaced with test_disc_copy_completes_full_disc_
with_failing_reader and test_disc_copy_halts_promptly_on_failing_reader.

Fix 2: async SCSI transport recovery. Added Arc<AtomicI32> fd_recovery
on SgIoTransport. On poll timeout: spawn close + spawn open in
background, return Err immediately. Top of execute() swaps fd from
recovery atomic. Main thread never blocked beyond ~1.5s poll budget
(was up to ~60s per timeout because kernel serialized main-thread
open() against in-flight close()). Drop drains pending recovery fd.

§15.1 cross-platform parity: Windows + macOS now have the same
observable recovery contract. SptiTransport gets try_recover()
(synchronous CloseHandle + CreateFileW; Windows close is fast, no
in-flight CDB drain like Linux). MacScsiTransport gets try_recover()
(release IOKit interface + reacquire via new acquire_device_iface()
helper); stores bsd_name for re-resolution. Drop guards null'd-out
interfaces. Stripped English error strings ("try as root" / "run as
administrator") on Linux + Windows. Fixed Windows TimeOutValue
ms→s ceiling so 1500ms gets 2s (was 1s; broke Drive::read fast path).

Fix 4: instrument Disc::patch arms. PatchResult exposes
blocks_attempted, blocks_read_ok, blocks_read_failed so the v0.13.11
mystery (Dune 2 Pass 2 recovered 0 bytes in 100 min) is diagnosable
from the live device log without re-instrumenting from outside.

Cleanup: honor PatchOptions::full_recovery (was read into _ and
ignored; now routed to read_sectors recovery arg). Updated
CopyOptions::batch_sectors doc to describe the actual production
path (sysfs detect_max_batch_sectors, typically 60 sectors / ~120 KB
on BU40N) rather than the test-only 32-sector internal default.

All four crates clippy-clean and tests green on the host targets
(macOS native + cargo check on Linux). Cross-platform CI watches
Linux + Windows + macOS builds + tests.
2026-04-25 17:30:25 -07:00
MattJackson 92dc145864 v0.13.11: revert SgIoTransport timeout path — keep transport alive
v0.13.10's 'fd=-1 on first poll timeout' was too aggressive: a single
transient killed the entire transport, Pass 1 finished in 45ms with
0 GB good on Dune 2.

Revert to spawn-close + main-thread-reopen (the v0.13.5/8 pattern).
Per-timeout cost is up to ~60s while the kernel completes the
abandoned command, but the v0.13.9 Disc::copy stall guard caps
catastrophic stalls at 120s of bytes_good non-advance. Pass 1 bails
cleanly with NonTrimmed ranges; Pass 2 has a working Drive for
retries with recovery=true + 30s timeouts.
2026-04-25 08:47:10 -07:00
MattJackson 36b65f526a v0.13.10: version sync 2026-04-25 08:25:08 -07:00
MattJackson ca8ebf418f v0.13.9: Disc::copy stall guard + SgIoTransport no-reopen-on-timeout
Fixes the silent Pass 1 hang observed on Dune 2 with v0.13.8 (drive
grinding through bad sectors at 0 KB/s, errs=0, no error surfaced).

Root cause: SgIoTransport::execute's reopen-after-poll-timeout opened
a fresh /dev/sg* fd on the main thread, which serialized against the
spawned close() of the old fd via the kernel's per-device state lock.
The userspace 1.5s timeout still fired, but the abandon-and-reopen
recovery itself blocked the main thread for as long as close() took.
Net: reads returned slowly, skip-forward fired on every iteration,
bytes_good never advanced.

- SgIoTransport::execute: on poll timeout, spawn close, set fd=-1,
  return Err. No reopen on the main thread. The transport is now
  invalidated until the consumer creates a fresh Drive.
- Disc::copy: add stall guard. CopyOptions.stall_secs (default 120s).
  If bytes_good doesn't advance for the threshold, break 'outer
  cleanly with complete=false, bytes_pending > 0 so Pass 2 retries
  pick up the NonTrimmed ranges with recovery=true 30s timeouts.
- New regression test: test_disc_copy_stall_detection_triggers_
  skip_forward in tests/integration_progress_and_halt.rs.
2026-04-25 08:12:01 -07:00
MattJackson 34182d956a v0.13.8: version sync 2026-04-25 07:11:47 -07:00
MattJackson 0f967a2084 v0.13.7: version sync 2026-04-25 06:58:43 -07:00
MattJackson 0341995c0d fix: allow dead_code on Drive::emit (DiscStream owns BytesRead emission post-0.13.6 strip) 2026-04-24 21:43:30 -07:00
MattJackson fe0ec0c5cf fix: drop unused INQUIRY_* constants (clippy -D warnings) 2026-04-24 21:42:30 -07:00
MattJackson 022657d481 docs: complete 0.13.6 docs sweep (architecture, api-design, disc-to-rip, README)
Follow-up to the rip-recovery + drive-access updates: aligns the
remaining docs with the v0.13.6 single-shot read model and the
three-layer recovery architecture.

- architecture.md: module map says single-shot read; new paragraph on
  layered recovery with postmortem pointer.
- api-design.md: EventKind enum example expanded; emission notes
  document that BytesRead now fires from DiscStream::fill_extents and
  Retry/SectorRecovered are no longer emitted in 0.13.6+.
- disc-to-rip.md: Step 10 of the pipeline diagram + module table
  reflect single-shot read.
- docs/README.md: added rip-recovery.md to the TOC.
2026-04-24 21:36:49 -07:00
MattJackson fc8eca44e1 docs: rewrite rip-recovery + drive-access for 0.13.6 single-shot model
Updates docs/ to reflect the recovery-loop strip:
- rip-recovery.md: drops Phase 1/2/3 description, replaces with three-layer
  model (Disc::patch multi-pass / DiscStream batch halving / Drive::read
  single-shot). Notes that no SCSI resets fire from any retry path.
- drive-access.md: removes SG_SCSI_RESET + STOP/START UNIT escalation
  references; SgIoTransport::reset is now kernel SG_IO flush + ALLOW
  MEDIUM REMOVAL only.
- src/mux/disc.rs + tests/: cargo fmt cleanup.
2026-04-24 21:35:33 -07:00
MattJackson 43836865be v0.13.6: strip Drive::read inline recovery + reset escalation; emit BytesRead
Drive::read is now single-shot. Phase 1/2/3 retries + scsi::reset+reopen
removed (~80 lines). recovery=true bumps timeout to 30s; recovery=false
stays at 1.5s. On any failure returns Err(DiscRead) immediately — caller
(Disc::patch outer loop, DiscStream batch halver) handles retries.

Inline reset+reopen WAS the wedge primitive on the LG BU40N. Per prior
post-mortem, every USB/SCSI reset path tested fails to recover the
wedged Initio bridge — the inline retry was pure cost.

SgIoTransport::reset (Linux) trimmed to kernel SG_IO state flush +
ALLOW MEDIUM REMOVAL. SG_SCSI_RESET ioctl + STOP/START UNIT escalation
removed. macOS reset removed (no-op). scsi::reset() top-level family
removed (no callers).

EventKind::BytesRead { bytes, total } now actually emitted from
DiscStream::fill_extents after each successful sector read. Was
declared in 0.13.0, never fired. Drives autorip per-device progress
in direct mode.

EventKind::Retry / SectorRecovered no longer emitted (variants kept
for forward compat). SpeedChange still emitted via Drive::set_speed
public path.

Tests: new tests/integration_progress_and_halt.rs (5 tests). 233 unit
tests + 5 integration green.
2026-04-24 21:32:45 -07:00
MattJackson 5e22199441 v0.13.5: version sync (autorip-side fixes) 2026-04-24 20:13:07 -07:00
MattJackson 761b77bbd9 v0.13.4: roll back wedge recovery + add sysfs identity fallback
USB/SCSI recovery escalation in drive_has_disc (0.13.1-0.13.3) tested
on LG BU40N USB BD-RE: USBDEVFS_RESET, authorized toggle, driver
unbind/rebind, SCSI host rescan — all succeed at the USB transport
layer but the drive firmware below the bridge stays locked. Only
physical unplug-replug clears it. Rolled back so consumers can
surface the real failure to the user.

New: list_drives falls back to sysfs-cached vendor/model/rev from
/sys/class/scsi_generic/sgN/device/ when live INQUIRY returns empty,
so wedged drives still show their identity in UIs.

Removed: scsi::usb_reset, usb_reset_with_timeout, per-platform
usb_reset methods, recover_then_probe, is_wedge_signature. Breadcrumb
comment in scsi/linux.rs::drive_has_disc points at v0.13.3 tag for
the full implementation if future hardware needs it back.

Linux/macOS/Windows pass-through symmetric; 233 tests passing.
2026-04-24 19:53:44 -07:00
MattJackson cc05ef2a3a v0.13.3: broaden is_wedge_signature — fix dead-code wedge recovery
0.13.2's is_wedge_signature gated on opcode=SCSI_INQUIRY (0x12), but
drive_has_disc issues TEST UNIT READY (0x00). Production wedge errors
(E4000: 0x00/0xff/0x00) never matched → SCSI reset + USB reset
escalation never fired.

Drop the opcode gate. Status byte 0xFF is synthesised by our own
execute() path on poll() timeout — it's the ground-truth wedge
marker for any opcode.

Linux-only; macOS/Windows use sense-key-based wedge detection.
2026-04-24 19:26:11 -07:00
MattJackson e27b82ce5b v0.13.2: list_drives + drive_has_disc; SCSI primitives pub(crate)
Architectural cleanup. autorip + freemkv CLI were reimplementing drive
discovery (sysfs walking, type-5 filtering, sg-path construction) and
calling SCSI reset primitives directly. All of that hardware-aware code
moves into libfreemkv with two cheap public probes:

- DriveInfo + list_drives() — multi-OS enumeration (Linux/macOS/Windows)
  with peripheral-type-5 filtering and INQUIRY identity. Cheap.
- drive_has_disc(path) — single TUR with internal wedge recovery
  escalation (SCSI reset → USB reset → retry) hidden from callers.

USB-layer reset (USBDEVFS_RESET / IOUSBDeviceInterface::ResetDevice /
storport's combined reset) wired across all three platforms.

Visibility tightening — scsi::reset, scsi::usb_reset, and the timeout
constants are now pub(crate). Compile-time guarantee that no consumer
crate can issue SCSI commands directly.

233 lib tests pass; clippy clean.
2026-04-24 17:31:15 -07:00
MattJackson 8af47e4c19 v0.13.1: scsi::reset() bounded by wallclock timeout
Production incident: autorip's poll loop called scsi::reset() on a
wedged BU40N USB drive. The Linux SG_SCSI_RESET ioctl blocked
indefinitely (kernel SCSI subsystem waiting for a bus-wedged device to
ack a reset that will never come). Caller's poll loop hung for 60+
seconds before manual intervention.

scsi::reset() now spawns a detached worker for the platform-specific
reset and bounds the caller's wait via mpsc::recv_timeout. Default
30 s (DEFAULT_RESET_TIMEOUT_SECS); reset_with_timeout(device, dur)
exposes the bound for callers that want a different value. Returns
DeviceResetFailed on timeout. Worker thread keeps running until the
kernel eventually unblocks — leaks one OS thread per hard wedge, but
the daemon stays responsive instead of hanging forever.

Follow-up flagged for 0.13.2: USB-attached drives wedge at the USB
Mass Storage layer below SCSI; SG_SCSI_RESET doesn't help. A
scsi::usb_reset(path) using USBDEVFS_RESET is the proper escalation.
2026-04-24 16:58:13 -07:00
MattJackson d1f09439a5 v0.13.0: zero English in library + API hygiene + dead-code sweep
Audit pass against the project docs "no English text in library code" rule.
Found 9 call sites that violated the contract by stuffing English into
io::Error::new(kind, "…") or by abusing Error::DeviceNotFound { path }
as a free-form description field. Each is now a typed Error variant.

New variants and codes: ScsiInterfaceUnavailable (E1004), DeviceLocked
(E1005), IoKitPluginFailed (E1006), UnsupportedPlatform (E2003),
PlatformNotImplemented (E2004), MapfileInvalid (E6011), DiscUrlNotDirect
(E9009).

labels::apply() previously pushed Commentary/Descriptive/Score/IME and
" (Secondary)" English literals into AudioStream.label, leaking into
MKV titles + autorip UI. AudioStream now exposes structured `purpose:
LabelPurpose`, SubtitleStream `qualifier: LabelQualifier`. Callers
translate to localized text. label keeps codec-formatting only.

API hygiene: 11 mux/* modules dropped from `pub` to `pub(crate)` —
their *types* are still re-exported from lib.rs, but the modules were
leaking low-level EBML/TS/network primitives. Stream trait gets a real
rustdoc explaining read-vs-write split. lib.rs grouped re-exports into
documented sections. ScanOptions::with_keydb() removed (one-method-per-
action rule); use struct literal.

Dead-code sweep: removed lookahead.rs (orphan, never declared as mod),
tsreader.rs (TsDemuxReader unused), ebml::{write_int,read_vint,SEEK_*},
ts::{scan_first/last_pts,scan_duration,SCAN_HEAD/TAIL_SIZE,take/set_
remainder}, MkvMuxer codec_private_slots/filled fields and
fill_codec_private method (deferred-codecPrivate path never used since
the v0.10 PES rewrite). cargo clippy --all-targets -D warnings clean.

Tests: new error::tests for variant codes + Display "no English" guard +
io::ErrorKind mapping. 233 lib tests, all green (was 230).

Breaking: ScanOptions::with_keydb removed; mux/* modules pub(crate);
AudioStream and SubtitleStream gained required fields; UnsupportedDrive
{ product_revision: "Renesas not yet implemented" } no longer produced
(use PlatformNotImplemented).
2026-04-24 16:41:02 -07:00
MattJackson 37e721ee7e v0.12.2: hide halt behind checked_sleep / checked_exec primitives
Drive::read is now halt-check-free in its body. Previously, the halt
flag was checked in 4 places and the sleep logic was scattered across
4 "if halt_aware_sleep { return Halted }" call sites — correct, but
the ugliness invited drift: a new sleep added by someone unfamiliar
with the pattern would silently swallow Stop requests.

Two private primitives now own halt awareness:

  checked_sleep(Duration) -> Result<()>
  checked_exec(cdb, dir, buf, timeout_ms) -> Result<ScsiResult>

Both return Err(Halted) instead of a bool. The ? operator in read()
then propagates halts for free. The recovery path reads top-to-bottom
with no halt vocabulary.

sleep_until_halted lives as a free function so it's unit-testable
without a live Drive. 4 new tests: completes normally, bails on
pre-set flag within one slice, wakes mid-sleep, zero-duration no-op.

Public API unchanged — halt_flag/halt/clear_halt still exposed, the
refactor is entirely internal.
2026-04-24 13:12:24 -07:00
MattJackson 481b47d90d v0.12.1: halt-aware sleeps in Drive::read recovery
Stop was waiting up to 30 s to register when the drive hit L-EC
recovery mid-read — the recovery phase does 30 s sleeps between
retries and the halt flag was only checked at the start of each sleep.
UI feels broken ("stop isnt working") even though the halt was set.

halt_aware_sleep breaks each wait into 100 ms slices and returns
early on halt. Applied to all 4 sleeps in the recovery path (both
30 s retry delays, both 5 s reset-phase delays).
2026-04-24 12:59:36 -07:00
MattJackson dd7cbd70a9 v0.12.1: cut non-recovery read timeout 5s → 1500ms
Disc::copy fast pass (skip_on_error=true, recovery=false) was giving
the drive 5 s per 64 KB block. On structure-protected / marginal UHD
sectors the drive grinds L-EC for nearly the full budget per block,
pinning throughput at ~13 KB/s even though skip_forward would happily
skip past the region.

1500ms bounds the floor at ~43 KB/s/block. Recoverable sectors that
would have succeeded at 3-5 s get picked up on Disc::patch (pass 2+)
where recovery=true and the per-read budget is 30 s.
2026-04-24 12:57:53 -07:00
MattJackson 3dea679dac style: cargo fmt 2026-04-24 12:23:42 -07:00
MattJackson 64ecca0d12 fix(windows): Rust 2024 requires unsafe extern blocks
Missed in the prior 2024 edition sweep because local builds don't
exercise the cfg(target_os = "windows") path. check-windows CI caught it.
2026-04-24 12:21:43 -07:00
MattJackson 84f6baba38 v0.12.0: Rust 2024 edition migration
- edition = "2024" bump.
- FFI block in src/scsi/macos.rs wrapped in `unsafe extern "C" { }`.
- vtable_fn body gets an explicit unsafe block (unsafe_op_in_unsafe_fn).
- Match-ergonomics cleanup in mux/meta.rs, mkvstream.rs, network.rs,
  stdio.rs — removed redundant `ref` / `ref mut` bindings.

MSRV unchanged at 1.86. 226 tests pass. No behavior change.
2026-04-24 12:07:05 -07:00
MattJackson bd5b7795bd docs: multi-pass recovery — README snippet + new rip-recovery.md
- README quick-start gains a multi-pass example using the new
  Disc::copy + Disc::patch primitives.
- New docs/rip-recovery.md documents the two-stage rip model: mapfile
  format (ddrescue-compatible), CopyOptions/PatchOptions surface, the
  pass-1/pass-2 algorithm, and the design decisions (why no MODE
  SELECT, why ISO intermediate, why ddrescue mapfile).

No code change.
2026-04-24 11:53:28 -07:00
MattJackson a57506c1ce v0.11.22: version sync — no functional changes
autorip 0.11.22 ships the full multi-pass UI (live mapfile stats,
bad-range viz, Recovery settings). libfreemkv API is unchanged from
0.11.21. Part of the 0.11.22 ecosystem sync.
2026-04-24 11:43:32 -07:00
MattJackson c33f3e9557 v0.11.21: multi-pass rip — Disc::copy + Disc::patch + mapfile module
New primitives for two-stage rip workflows: fast forward pass with
zero-fill on failures, then targeted retries of bad ranges via a
ddrescue-compatible mapfile.

- Disc::copy now takes &CopyOptions (breaking change from positional
  args). Always writes a sidecar .mapfile. Opt-in skip_on_error +
  skip_forward give ddrescue-style fast sweep: 64 KB blocks,
  exponential skip-forward on failure, zero-fill bad blocks. Defaults
  preserve pre-0.11.21 behavior (recovery reads, abort on bad sector).

- Disc::patch is new and idempotent. Reads the mapfile, re-reads every
  non-finished range with full drive recovery, patches good bytes back
  into the ISO at exact offsets. Call N times for N retry attempts.

- disc::mapfile is a new module. ddrescue text format, crash-safe
  (flushed on every record()), greppable, human-editable, tool-compatible.
  Status chars match ddrescue: ? / * / / / - / +.

- Re-exports FileSectorReader from the crate root.

- freemkv CLI caller (pipe.rs) updated to the new Disc::copy signature
  in lockstep — shipped in the 0.11.21 freemkv CLI release.

Part of the 0.11.21 ecosystem sync (libfreemkv + freemkv + bdemu +
autorip all on 0.11.21).
2026-04-24 09:24:32 -07:00
MattJackson 166fc4bf8c v0.11.18: DiscStream halt flag — Stop works in dense bad-sector regions
DiscStream::fill_extents loops internally while the demuxer waits for
enough clean data to emit a PES frame. In a dense bad zone that loop
can run for minutes without returning to the outer read() call, so
the caller's Stop signal never gets serviced until a frame is finally
emitted — which may be very far away.

Add DiscStream::set_halt(Arc<AtomicBool>) — typically wired to
Drive::halt_flag() for unified Stop across drive recovery phases and
stream sector processing. fill_extents checks the flag at the top of
every retry iteration; raising it returns Err(Error::Halted) within
one SCSI round-trip.

No behavior change for callers that don't call set_halt. Unblocks the
architectural fix for the "Stop doesn't stop" bug observed on a
damaged UHD disc.
2026-04-24 07:41:13 -07:00
MattJackson 9aaddaa9b8 v0.11.17: adaptive batch sizer — no per-sector descent
Replace read_with_binary_search + 3×5s light recovery with an adaptive
sizer that shrinks on failure (halve, 3-aligned ≥6) and probes back up
after 100 MiB (51,200 sectors) of clean reads. Descent cost is paid
once per bad region, not once per bad sector.

Emit BatchSizeChanged { new_size, reason } on shrink and probe-up.
Remove BinarySearch event — no longer produced.

Side fix: scsi/macos.rs one-liner for manual_c_str_literals clippy
lint that surfaced on a newer toolchain.
2026-04-23 20:24:40 -07:00
Matt Jackson 40ec1e1eba v0.11.16: API cleanup — one method per action 2026-04-21 19:17:50 +00:00
Matt Jackson 8843833ea7 v0.11.15: lint cleanup — fmt + clippy clean 2026-04-21 18:52:55 +00:00
Matt Jackson afae37c8c3 fix: iso_dump example missing recovery arg 2026-04-21 18:41:41 +00:00
Matt Jackson d8089ec491 v0.11.14: audit fixes — trailing sectors, verify stop, SCSI sense, O_CLOEXEC
Fix trailing sectors dropped at extent boundaries when sector_count % 3 != 0.
Add verify_title stop support via progress callback returning bool.
Add O_CLOEXEC on all SCSI fd opens to prevent leak to child processes.
Fix SCSI sense descriptor format detection (0x72/0x73 vs 0x70/0x71).
2026-04-21 18:40:04 +00:00
Matt Jackson ce8132d181 Update docs: async sg transport, Drive::read recovery phases 2026-04-21 18:01:48 +00:00
Matt Jackson 71e1f57364 Async SG_IO: enforceable timeouts via write/poll/read
Replace blocking ioctl(SG_IO) with the sg driver's async interface.
Commands are submitted via write(), waited on via poll() with a hard
wall-clock timeout, and completed via read(). If poll() times out,
the fd is abandoned and a fresh one opened — the kernel can no longer
hold us hostage during USB error recovery.

- write() submits command, returns immediately
- poll() enforces exact timeout (EINTR-safe with deadline tracking)
- read() retrieves result + copies data to caller's buffer
- On timeout: old fd closed in background thread, new fd opened
- No SG_FLAG_DIRECT_IO — kernel buffers for safe timeout abandonment
- Store device_path for fd reopen after timeout
- Drop guards fd=-1 (abandoned fd)
2026-04-21 17:57:58 +00:00
Matt Jackson d8092ee093 Clean API: merge read() and read_fast() into read(recovery: bool) 2026-04-21 03:27:17 +00:00
Matt Jackson dd49d55b11 v0.11.13: all rip reads use fast timeout, no full recovery in read path 2026-04-21 02:16:49 +00:00
Matt Jackson 6ba958fe8a Fix: initial batch read uses fast read, not full recovery 2026-04-21 02:02:13 +00:00
Matt Jackson f30b958408 v0.11.12: halt, events, light recovery 2026-04-21 00:25:44 +00:00
Matt Jackson 6971f00be2 Drive halt flag, sector events, binary search light recovery (3x5s) 2026-04-21 00:18:46 +00:00
Matt Jackson ad86f92d85 v0.11.11: binary search error recovery in fill_extents 2026-04-20 21:51:46 +00:00
Matt Jackson cce1be29b2 v0.11.10: skip_errors, read_sectors_recover API 2026-04-20 15:27:56 +00:00
Matt Jackson 081ad4f619 DiscStream skip_errors, read_sectors_recover(recovery) API 2026-04-20 03:38:32 +00:00
Matt Jackson 43ecad2d43 Clean API: read_sectors_recover(recovery: bool) replaces read_sectors_fast 2026-04-20 01:12:59 +00:00
Matt Jackson 8870efa77b v0.11.9: fast verify reads — 5s timeout, no recovery loop 2026-04-20 01:01:44 +00:00
Matt Jackson a693ef965b v0.11.8: disc verify module 2026-04-20 00:03:26 +00:00
Matt Jackson 0596307c19 Add verify module: sector-by-sector disc health check 2026-04-20 00:01:02 +00:00
Matt Jackson 18c6365e08 Improve label generator: proper video labels, all resolutions, no false Atmos claims 2026-04-19 17:50:17 +00:00
Matt Jackson 981f30b1b0 Move label generation to labels system — fill_defaults() for all stream types 2026-04-19 17:23:50 +00:00
Matt Jackson 04086d73a0 Generate audio track labels when disc doesn't provide them 2026-04-19 16:34:35 +00:00
Matt Jackson 20e404e59a README: fix dep version 0.10 → 0.11, update share command 2026-04-19 04:57:55 +00:00
Matt Jackson 1bea8eb650 v0.11.7: TrueHD parser rewrite — 12-bit length, AC-3 skip, cross-PES buffering 2026-04-19 01:36:56 +00:00
Matt Jackson 0165f18fad v0.11.6: fix TrueHD BD-TS header corruption 2026-04-18 19:27:12 +00:00
Matt Jackson ca3914d537 Fix TrueHD: strip BD-TS access unit header before muxing to MKV 2026-04-18 19:26:14 +00:00
Matt Jackson 80fcffd190 v0.11.5: MKV container fixes — timestamps, frame rate, HDR, chapters, disposition 2026-04-18 16:29:21 +00:00
Matt Jackson 7437af39c8 v0.11.3: unified versioning across all repos 2026-04-18 15:01:53 +00:00
Matt Jackson bcd6925170 v0.10.10: fix dual-layer disc rips, propagate read errors
Bump version, update CHANGELOG and FEATURES for 0.10.8-0.10.10.
2026-04-18 02:10:20 +00:00
Matt Jackson fba1eb189c Fix truncated rips on dual-layer discs, propagate read errors
Use UDF file_extents() to read actual allocation descriptors instead
of assuming m2ts files are contiguous from file_start_lba. Dual-layer
UHD discs split large files across 70+ extents (~1 GB each) — the old
code created one extent from packet count which only covered the first
chunk, causing silent truncation at ~37%.

Also changed fill_extents() to return io::Result<bool> so read errors
propagate instead of being silently treated as EOF.
2026-04-18 02:08:34 +00:00
Matt Jackson a328ad3cd8 Add freemkv keydb path to KEYDB search, fail on encrypted disc without keys 2026-04-17 22:10:56 +00:00
Matt Jackson 9791e60c65 v0.10.8: prefetch all metadata file sectors — scan 2min to 18s on USB 2026-04-17 19:51:32 +00:00
Matt Jackson 8b1ded84cd Document buffered sector reads in UDF docs 2026-04-17 18:41:53 +00:00
Matt Jackson 3e4febe7f5 Remove debug timing from scan pipeline 2026-04-17 18:26:04 +00:00
Matt Jackson aa7afc0c98 Buffer UDF sector reads — eliminates scan hang on USB drives 2026-04-17 18:24:29 +00:00
Matt Jackson 29f9e916d6 DiscStream::new() replaces open_drive/open_iso/from_reader
A stream is a stream. DiscStream::new() takes reader + title + keys +
batch + format — same pattern as every other stream constructor.

Deleted: open_drive(), open_iso(), from_reader() — these were helper
functions that chained multiple operations. Library provides primitives,
callers decide the sequence.

Removed disc:// case from input() — callers use Drive::open() +
Disc::scan() + DiscStream::new() directly for disc sources.
2026-04-17 15:32:48 +00:00
Matt Jackson 30d59063d5 Fix drive discovery in Docker — remove sysfs check 2026-04-16 22:05:25 +00:00
MattJackson 6e958b0f71 v0.10.5: Buffer audio parsers across PES boundaries 2026-04-16 19:55:57 +00:00
MattJackson 911d260695 Buffer audio parsers across PES boundaries
- DTS: buffer with core sync detection + frame size from header
- TrueHD: buffer with unit length field parsing
- Same pattern as AC3 fix: incomplete frames held for next PES
- When PES boundaries align (normal case), buffering is a no-op
2026-04-16 19:02:23 +00:00
MattJackson 59d4eb8854 Fix AC3 parser: buffer across PES boundaries, proper frame sizing
- Add state to Ac3Parser (was stateless, split frames at PES boundaries)
- Buffer leftover bytes from incomplete frames for next PES packet
- Calculate exact AC3 frame size from fscod/frmsizecod table
- Calculate EAC3 frame size from frmsiz field
- Skip invalid frame sizes (0 or >8192)
- Eliminates all AC3 decode errors on BD and UHD output
2026-04-16 18:49:30 +00:00
MattJackson 8820f7a460 Fix cargo fmt formatting 2026-04-16 17:51:45 +00:00
MattJackson 8de99bb8e8 Update README and FEATURES.md for v0.10.4 DVD support 2026-04-16 17:36:41 +00:00
MattJackson 3e5e570573 v0.10.4: CSS decryption, MPEG-2 PS demuxer fixes
Full CSS key hierarchy (bus auth, disc key, title key), correct
descramble cipher, MPEG-2 codec parser routing for DVD PS path,
sequence header extraction with quantizer matrices.
2026-04-16 17:33:29 +00:00
MattJackson f95894bba3 Expand CSS Stevenson crack patterns, scan 50K sectors
- Add padding stream (0xBE) with 0xFF payload patterns
- Add video/audio PES with multiple flag/header combinations
- Add navigation pack system header pattern
- Scan up to 50K consecutive scrambled sectors (was 500 sampled)
2026-04-16 15:15:17 +00:00
MattJackson 597f512eca Fix MPEG-2 PS demuxer: codec parsers, sequence header extraction, descramble
- Route DVD PS packets through codec parsers (was bypassing them)
- Extract MPEG-2 sequence header with quantizer matrices for MKV codec_private
- Calculate exact sequence header size from intra/non-intra matrix flags
- Capture sequence extension (B5) from subsequent PES packets
- Revert TAB1 permutation in descramble (XOR-only is correct)
- Fix CSS roundtrip tests for new descramble behavior
2026-04-16 15:06:35 +00:00
MattJackson 8e907393a3 Route DVD PS demuxer through codec parsers, fix CSS test expectations
- DVD PS path now calls parser.parse() like BD-TS path does
- MPEG-2 sequence headers extracted for codec_private
- Keyframe detection from parser instead of always-true
- Fix CSS roundtrip tests: descramble uses TAB1 permutation, not pure XOR
2026-04-16 04:49:46 +00:00
MattJackson c47b9a9c3c Fix CSS decryption: full key hierarchy, correct cipher tables
- Implement complete CSS key chain: bus auth → disc key → title key
- Add 31 player keys for disc key decryption
- Read disc key via READ DVD STRUCTURE format 0x02
- Read title key via REPORT KEY format 0x04
- Fix CryptKey round 1: use original scratch for term, not modified tmp1
- Fix decrypt_key: use TAB5 for LFSR1 output, TAB4 for LFSR0^invert
- Fix descramble_sector: use TAB5 for LFSR1, TAB4 for LFSR0 (no invert),
  and apply TAB1 permutation to ciphertext before XOR
- Fix title key bus XOR: forward order (bus_key[i]), not reversed
- Two-session auth: disc key and title key need separate AGID sessions
- Fix crack_key: scan across extents for scrambled sectors
- Fix TsDemuxer: dynamic PID table size for DVD PIDs
- Set max read speed after scan for DVD riplock removal
2026-04-16 04:42:42 +00:00
MattJackson d7b9d87075 v0.10.3: CSS drive authentication for DVD ripping 2026-04-16 00:01:48 +00:00
MattJackson e0f40583c4 Fix cargo fmt formatting 2026-04-15 22:32:53 +00:00
MattJackson d06a37d3dc Add 0.10.2 changelog entry 2026-04-15 22:29:33 +00:00
MattJackson add9a929a5 Bump to v0.10.2 2026-04-15 22:26:51 +00:00
MattJackson 9bb1d02245 Fix batch overflow in Disc::copy(), DVD PGC parsing, demuxer flush at EOF
- Disc::copy() hardcoded batch=64 sectors, exceeding BU40N's 60-sector
  hw limit. Now accepts batch_sectors param, defaults to 60.
- IFO PGC: playback time at offset 0x04 not 0x02, cell time at cell+4
- DiscStream: set demuxer from content_format (TS for BD, PS for DVD)
- Flush TS/PS demuxers at EOF to avoid losing last PES frame
- M2tsStream: flush demuxer at EOF
- StdioStream: FMKV metadata header for roundtrip compatibility
2026-04-15 22:26:07 +00:00
MattJackson dc706e8153 Fix integration tests for new PES-only API
Update tests/streams.rs: IOStream → PES Stream, MkvStream::new → create,
M2tsStream::new → create, NullStream::new takes title, open_input → input,
open_output → output. All 317 tests pass.
2026-04-15 19:54:09 +00:00
MattJackson 0f18906ede v0.10.1: Streams are PES, Disc::copy() for sector dumps, zero English
Architecture:
- One stream per format, bidirectional PES (read/write on same type)
- IsoStream merged into DiscStream (one type, any SectorReader)
- Disc::copy() for disc→ISO raw sector dump
- IOStream trait deleted, all byte-level Read/Write removed
- ContentReader/OpenDisc/open_title/open_input/open_output deleted
- CountingStream wrapper for progress tracking

Error codes:
- All io::Error English strings replaced with Error enum variants
- From<Error> for io::Error conversion
- Unused variants removed, new stream/mux variants added

Deleted: mkvout.rs, pesout.rs, isowriter.rs, mkv-muxer-plan.md
Updated: all docs, README stream table, CHANGELOG

238 tests, 0 clippy warnings.
2026-04-15 19:46:01 +00:00
MattJackson bd7220c62b v0.10.0: PES pipeline audit, codec_privates on DiscTitle, streams not files 2026-04-15 17:22:52 +00:00
MattJackson 404c005e0b Remove Seek/File dependencies from stream readers
Streams are streams — they take impl Read, not Read+Seek or File.

- MkvStream::open takes impl Read (was Read+Seek)
  - EBML element skipping uses skip_bytes() instead of seek(Current)
  - Byte position tracking uses remaining-bytes counter, not stream_position()
  - Removed file_size from open (progress is CLI concern)
- M2tsStream::open takes impl Read (was Read+Seek)
  - Buffers first 1MB for FMKV header / PMT scan
  - Uses chain reader (buffered head + rest) for sequential reading
  - Duration unknown without seeking (0.0) — CLI can set from metadata
- Removed ReadSeek trait (no longer needed)
- WriteSeek kept (MKV muxer container format requires seeking internally)
2026-04-15 17:05:50 +00:00
MattJackson 87342290c5 Move codec_privates onto DiscTitle, eliminate duplicate methods
Design fix: codec_privates are now a field on DiscTitle, not a separate
parameter passed through the pipeline. This eliminates the root cause of
the network codec_private bug (forgot to pass the separate param).

API changes:
- output() takes (url, &DiscTitle) — no separate codec_privates param
- MkvOutputStream::create, M2tsOutputStream::create, NetworkOutputStream::connect
  all read codec_privates from title.codec_privates
- M2tsMeta::from_title() takes only &DiscTitle — reads privates from title
- Deleted from_title_with_privates (was the wrong-name duplicate)
- Merged read_header + read_header_from_stream into one read_header(impl Read)
- Deleted finish(self) from TsMuxer, keep only finish(&mut self)

Rule: ONE public method per action. No _with_X, _from_Y, _ref variants.
2026-04-15 16:52:06 +00:00
MattJackson 8bbf630d82 Fix all PES pipeline audit findings (20 issues)
Critical:
- C1: PES serialize validates track < 256 and data < 4GB
- C2: PES deserialize caps frame size at 256MB (OOM protection)
- C3: TsMuxer stuffing uses static buffer, no per-packet alloc
- C4: PES length uses unbounded (0x0000) for audio >65535 bytes
- C6: TsDemuxer validates AF length <= 183

Warning:
- W1: parse_timestamp validates marker bits, returns Option
- W2: PES header data_start clamped to data.len()
- W3: TsMuxer PTS conversion uses saturating_mul, rejects negative
- W4: AC3/DTS replace debug_assert with runtime bounds check
- W6: MKV block_vint handles 3-4 byte VINTs
- W7: meta.rs to_title() uses unwrap_or fallbacks instead of panic
- W8: MKV reader skips frames for non-existent tracks
- W9: DVD PTS uses higher-precision conversion (1e9/90000)
- FMKV read_header caps JSON at 10MB
- PAT section_len underflow guard

Suggestion:
- S2: TsMuxer uses static STUFF_FF buffer
- S3: HEVC parser single-pass NAL scan (was duplicated)
- S4: TsDemuxer caps remainder at one packet
- S5: PTS 90kHz→ns uses round-to-nearest
2026-04-15 16:22:28 +00:00
MattJackson 45dddc1810 Fix M2TS/MKV roundtrip: codec_private in FMKV header, Annex B conversion
Bug 1: M2TS roundtrip dropped frames — TsMuxer converts length-prefixed
NALs to Annex B, prepends VPS/SPS/PPS from HEVCDecoderConfigurationRecord.

Bug 2: MKV remux lost codec_private — MkvStream.codec_private() now returns
data from EBML header.

FMKV header carries codec_private (base64) per video stream for lossless
M2TS roundtrip.
2026-04-15 16:09:34 +00:00
MattJackson cfa80cb881 Fix test for network open_input error message change 2026-04-15 04:54:19 +00:00
MattJackson b510ae9b29 Fix M2TS/Network output: write FMKV header for roundtrip compatibility
- M2tsOutputStream writes FMKV metadata header before TS data
- NetworkOutputStream sends FMKV header on connect
- Enables M2TS→MKV and network roundtrip to work correctly
2026-04-15 04:50:19 +00:00
MattJackson 9a1c3cc218 NetworkStream PES-only: remove old IOStream/Read/Write interface
- Remove IOStream, Read, Write impls from NetworkStream
- PES write sends FMKV header before first frame (protocol fix)
- PES finish sends TCP shutdown for clean EOF
- Update tests to use PES roundtrip instead of byte-level
- Remove NetworkStream from open_input/open_output (use input/output)
2026-04-15 04:36:47 +00:00
MattJackson 9ae7d6b38a Fix audit findings: SCSI constants, sg_io_hdr assert, handshake cap, sector overflow check
- Replace magic SCSI opcodes with named constants (S2)
- Add compile-time sg_io_hdr size assertion — 88 bytes on 64-bit (W2)
- Cap handshake cert attempts at 16 (W8)
- Validate IsoSectorReader/FileSectorReader against u32 overflow for >8TB (S8)
- encrypt.rs: limit host cert loop iterations
2026-04-15 04:29:45 +00:00
MattJackson d983985faa Fix all clippy warnings: dead code, match patterns, type complexity, docs
- Remove unused pes_buf field from M2tsStream and unused TS_PACKET/BD_TS_PACKET constants
- Replace match-with-single-pattern with if let (3 instances in drive/mod.rs)
- Replace match-can-be-? with ? operator for scsi::open call
- Add type aliases PesSetup and MkvHeaderResult to reduce type complexity
- Collapse identical if/else branches in tsmux.rs build_pes_header
- Use RangeInclusive::contains instead of manual range checks
- Make WriteSeek trait pub (was pub(crate) but leaked through pub fn)
- Remove empty line after doc comment in disc.rs
- Fix doc list item indentation in scsi/linux.rs (12 instances)
2026-04-15 04:09:56 +00:00
MattJackson 56fe26c9b8 All streams complete — DVD PS demux, network/stdio PES, MKV input
- DiscStream: BD (TsDemuxer) or DVD (PsDemuxer) auto-detected
- NetworkStream: Stream impl with PES serialize/deserialize
- StdioStream: Stream impl with PES serialize/deserialize
- MkvStream: Stream read returns PesFrame from EBML blocks
- M2tsStream: Stream read via TsDemuxReader
- PesFrame: serialize/deserialize for wire format
- TsDemuxReader: shared BD-TS demux helper
- All inputs and outputs support PES
2026-04-15 04:03:56 +00:00
MattJackson 7a83f3244d Complete PES pipeline — all streams, clean API
- Unified Stream trait: read() and write() on one type
- PesFrame serialize/deserialize for wire format
- TsDemuxReader: shared BD-TS demux for any Read source
- MkvStream: PES read from MKV (EBML → PesFrame)
- M2tsStream: PES read via TsDemuxReader
- Network/Stdio output: PES serialization directly (no BD-TS wrap)
- Network/Stdio input: deferred (needs PES deserialization protocol)
- TsMuxer for M2TS output from PES frames
- input() and output() functions return Box<dyn Stream>
2026-04-15 03:56:15 +00:00
MattJackson 15f5b49fc3 M2TS implements Stream (read PES frames via TsDemux)
M2tsStream::read() → TsDemux → CodecParser → PesFrame.
M2TS input now works through PES pipeline.
MKV input deferred (needs EBML → PES extraction).
2026-04-15 03:41:03 +00:00
MattJackson ff6004a567 Unified Stream trait: read() and write() on one type
Stream trait: read() returns PesFrame, write() accepts PesFrame.
A stream is a stream — you read from it or write to it.
No separate Input/Output traits.

API: libfreemkv::input(url) and libfreemkv::output(url, title, codecs)
Returns Box<dyn Stream>.
2026-04-15 03:33:29 +00:00
MattJackson bd644d2f60 100% PES pipeline — all streams produce/consume PES frames
- TsMuxer: PES frames → BD-TS packets (new, reverse of TsDemuxer)
- M2tsOutputStream: PES → TsMuxer → file
- NetworkOutputStream: PES → TsMuxer → TCP
- StdioOutputStream: PES frames → stdout
- NullOutputStream: discard
- MkvOutputStream: PES → MKV mux
- All outputs via open_pes_output()
- All inputs via open_pes_input() (ISO, disc)
- No byte-level fallback — everything is PES
2026-04-15 03:19:03 +00:00
MattJackson ab63950ef1 PES pipeline: InputStream on DiscStream + IsoStream, MkvOutputStream
- DiscStream: next_frame() reads sectors → decrypts → demuxes → returns PesFrame
- IsoStream: same pattern, reads from ISO file
- MkvOutputStream: accepts PesFrame, writes MKV via MkvMuxer
- InputStream trait: next_frame(), info(), codec_private(), headers_ready()
- OutputStream trait: write_frame(), finish()
- FileSectorReader: SectorReader backed by a file
- PesFrame: track + pts + keyframe + data

Old Read/Write IOStream impls preserved for backward compatibility.
Next: replace pipe() in CLI to use PES pipeline.
2026-04-15 03:03:45 +00:00
MattJackson cab87ffdf4 DiscStream implements InputStream — produces PES frames
DiscStream now has TsDemuxer + CodecParsers internally.
next_frame() reads sectors → decrypts → demuxes → parses → returns PesFrame.
Old Read/Write impls preserved alongside for backward compatibility.
2026-04-15 02:54:08 +00:00
MattJackson ccb1fadedf Add PES frame types and FileSectorReader — foundation for stream refactor
- pes.rs: PesFrame, InputStream, OutputStream traits
- sector.rs: SectorReader now public, added FileSectorReader (ISO = file)
- Foundation for unified DiscStream that handles both disc and ISO
2026-04-15 02:37:54 +00:00
MattJackson 610f1b62b6 Fix ISO → MKV producing empty files (TsDemuxer remainder lost)
When the MKV muxer transitions from Scanning to Streaming phase,
it creates a fresh TsDemuxer. The old demuxer's remainder bytes
(partial 192-byte BD-TS packets) were lost, causing the new demuxer
to lose sync. All subsequent feed() calls found 0 packets.

Fix: transfer remainder via take_remainder/set_remainder so the
new demuxer maintains packet alignment.
2026-04-15 02:17:34 +00:00
MattJackson f400e4fc72 Remove eprintln from library — library code should not print to stderr
Drive recovery is silent. Results communicated through return values.
2026-04-15 02:00:45 +00:00
MattJackson 5a9980cb6b Fix unused variable warning from handshake fix 2026-04-15 01:56:57 +00:00
MattJackson 2f48877992 Fix handshake returning fake success on failure
Previously returned HandshakeResult with zeros when all host certs
failed. Now returns None. Also propagates volume_id read failure
instead of silently using zeros.
2026-04-15 01:54:58 +00:00
MattJackson 7c59d063ee decrypt_sectors returns Result — fail instead of silent corruption
Previously used all-zeros AES key when unit_key_idx was out of range,
producing silently corrupted output. Now returns DecryptFailed error.
Also wired --raw flag through InputOptions → set_raw() on streams.
2026-04-15 01:52:36 +00:00
MattJackson e4ffe7cc91 Wire --raw through InputOptions to streams
set_raw() on IsoStream and DiscStream sets keys to None.
open_input passes raw flag to streams via InputOptions.
Streams skip decrypt when raw=true.
2026-04-15 01:38:18 +00:00
MattJackson 2e23e848df Decrypt back in streams — streams handle their own decryption
IsoStream decrypts in its read path using keys from scan.
IOStream trait gets keys() method with default DecryptKeys::None.
Pipeline no longer handles decrypt — just reads decrypted bytes.
2026-04-15 01:34:30 +00:00
MattJackson fbdfccdd83 Add keys() to IOStream trait — streams know their own decrypt keys
- IOStream::keys() default returns DecryptKeys::None
- IsoStream and DiscStream override with real AACS/CSS keys
- Pipeline calls input.keys() instead of separate scan_keys()
- Streams are self-contained: read bytes + provide keys
2026-04-14 23:42:14 +00:00
MattJackson 2f52188f77 Drive recovery, reset on open, simplified DiscStream
- SgIoTransport::reset() — open/close/TUR/escalate on every open
- Drive::read() — single read method with error recovery (min speed,
  sleep 30s, retry, phase 1/2/3 escalation)
- Removed read_timeout, read_sectors, read_range — one read() method
- DiscStream simplified — no on_error/on_success/Recovery, delegates
  all error handling to Drive::read()
- IsoStream no longer decrypts — streams return raw bytes, pipeline
  handles decryption
- reset() on all platforms (Linux real, Windows/macOS stubs)
- Watchdog thread removed — kernel handles USB timeouts
2026-04-14 23:32:22 +00:00
MattJackson 252e58cab0 Bump to 0.9.0 2026-04-13 02:13:15 +00:00
MattJackson 3b96976a7a Add decrypt module, merge to one drive.read(), Disc::decrypt_keys()
- New decrypt.rs: DecryptKeys enum (AACS/CSS/None) + decrypt_sectors()
- Single drive.read() replaces read_disc/read_content (same SCSI READ(10))
- ContentReader and DiscStream use decrypt_sectors() (no duplicated crypto)
- Disc::decrypt_keys() exposes resolved keys for disc-to-ISO
2026-04-13 02:08:58 +00:00
MattJackson 3385de5704 Add Drive::read_capacity() for raw sector dump 2026-04-13 01:45:32 +00:00
MattJackson e9c907a2ca Pin Rust 1.86 MSRV in Cargo.toml and CI workflows 2026-04-13 00:35:35 +00:00
MattJackson db1f6bc446 Fix is_multiple_of nightly API, bump to 0.8.3
Replace s.len().is_multiple_of(2) with s.len() % 2 != 0 for stable Rust.
This was fixed previously but regressed.
2026-04-13 00:31:54 +00:00
MattJackson c43ba06b5b Fix macOS build: mark MacScsiTransport as Send
IOKit COM interface pointers are Mach port references, safe to send
between threads. The Send bound added in 0.8.1 broke macOS builds.
2026-04-13 00:22:19 +00:00
MattJackson 077e4d018f Bump to 0.8.1, make profile module public, fix unused import 2026-04-13 00:17:57 +00:00
MattJackson f8b5a1eaf1 API: Drive object, typed StreamUrl, tray lock/unlock, Send traits
- Rename DriveSession → Drive across entire codebase
- find_drives() returns Vec<Drive>, find_drive() returns Option<Drive>
- resolve_device() now pub(crate) — internal only
- StreamUrl is now a typed enum (Disc, Mkv, M2ts, Iso, Network, Stdio, Null)
  with scheme() and path_str() accessors, replacing struct of Strings
- Add lock_tray() / unlock_tray() for safe disc access during rips
- Improve reset() with eject cycle that clears LibreDrive stuck state
- Add Send bounds to ScsiTransport and PlatformDriver traits
- DiscOptions uses PathBuf instead of String for device/keydb paths
- Update doc example to use new Drive API
2026-04-13 00:13:41 +00:00
MattJackson fb1c35e653 DriveStatus API + reset() + wait_ready with fallback
- DriveStatus enum: TrayOpen, NoDisc, DiscPresent, NotReady, Unknown
- drive_status(): GET EVENT STATUS NOTIFICATION with TUR fallback
- reset(): PREVENT ALLOW → START STOP → init() escalation
- wait_ready(): tries reset on Illegal Request, falls back to drive_status
- Remaining: standard READ(10) still fails in LibreDrive stuck state
2026-04-12 23:36:16 +00:00
MattJackson 9268ea2fdb WIP: LibreDrive stuck state detection in wait_ready + scan
- wait_ready: detects MMkv vendor probe when TUR returns Illegal Request
- scan: capacity fallback to 0 when READ CAPACITY fails
- Still needs: re-init to restore standard SCSI commands, or use raw reads for UDF
2026-04-11 22:09:26 +00:00
MattJackson 9078aa8718 Fix UHD remux: skip DV EL from video_pending, dynamic lookahead
- Secondary video streams (Dolby Vision EL) no longer block codec detection
- Lookahead buffer: 10 MB default, 100 MB for UHD (>15 streams)
- Verified: Dune UHD 84.6 GB remux completes at 101 MB/s
- Verified: V for Vendetta BD 21.3 GB remux completes at 131 MB/s
2026-04-11 21:43:51 +00:00
MattJackson c55e6991b8 Doc comments, format string inlining, long literal separators
- Doc comments on DriveSession, find_drives, all Error variants, Result type
- 24 format! strings inlined (clippy pedantic)
- 25 long hex literals with separators (0xFFFFFFFF → 0xFFFF_FFFF)
- README install example updated to 0.8
2026-04-11 21:04:44 +00:00
MattJackson ca931b6522 Doc comments on public API, README version fix, format string cleanup 2026-04-11 21:01:16 +00:00
MattJackson d4d98ce593 Granular SCSI query methods on DriveSession, capture uses them
- get_config_feature(code) → Option<Vec<u8>>
- report_key_rpc_state() → Option<Vec<u8>>
- mode_sense_page(page) → Option<Vec<u8>>
- read_buffer(mode, buf_id, length) → Option<Vec<u8>>

capture.rs now uses these methods — zero raw CDB construction.
CLI info.rs has zero SCSI references.
autorip ejects via library, not shell command.
2026-04-11 20:54:04 +00:00
MattJackson 4d92e15224 v0.8.0: DVD support, 100% codecs, 327 tests, 4 audit rounds clean 2026-04-11 20:35:33 +00:00
MattJackson 113d6b9e9e CSS + AACS cross-validation test vectors
AACS: encrypt with aes crate independently, decrypt with our code, verify match
- 3 tests: unit decrypt, alternate key, bus decrypt
- Uses independent AACS IV constant (not imported from library)

CSS: roundtrip snapshots + Stevenson attack validation
- 4 tests: snapshot regression, multi-key roundtrip, attack validation
- Documents limitation: synthetic sectors may not converge on attack

320+ tests total.
2026-04-11 20:31:12 +00:00
MattJackson 75dfd06a02 Fix all v4 audit findings (22 items)
HIGH: ISO writer multi-extent for >4GB, end-to-end MKV mux test
MEDIUM: AACS cvalue bounds, UV offset, macOS discovery, VC-1 resolution
  from sequence header, HEVC profile flags from SPS, ISO CRC + reserve AVDP
LOW: PS AC3 sub-header, CSS crack first-match break, TrackUID unique,
  AC3 no-sync empty return, --all for iso://, --min warning, dead code removed

320 tests, all passing.
2026-04-11 20:29:00 +00:00
MattJackson f48b4925c1 Zero clippy warnings: fix all 32 remaining
- Iterator::find() replaces manual loops (6 sites)
- Index-only loops → iterators (4 sites)
- Identical if-blocks merged
- Box large MkvStream WriteState enum variant
- Vec macro initializers, late init fixes
- Unused fields prefixed with underscore (format spec fields)
- Dead code removed or documented

0 clippy warnings. 319 tests passing.
2026-04-11 19:33:13 +00:00
MattJackson 75f15cae62 Audit v3 fixes: all 3 tiers (19 findings)
Tier 1 (compilation + correctness):
- Fix nightly-only is_multiple_of → % 2 != 0 (stable Rust compat)
- Fix parse_sample_rate: check 192 before 96 (was returning wrong rate)
- macOS drive discovery: split unix.rs → linux.rs + macos.rs
- Linux: EACCES returns DevicePermission not DeviceNotFound
- CLI pipe.rs: Ctrl+C signal handler added

Tier 2 (correctness + security):
- MkvStream: reset demuxer after scanning→streaming transition
- Windows SPTI: zero data buffer before ioctl
- AACS cert verification: documented why silently skipped
- KEYDB: HOME + USERPROFILE fallback for Windows
- Library modules: pub(crate) for internal modules
- AACS: explicit re-exports, AES primitives pub(crate)

Tier 3 (performance + polish):
- IsoStream: batch 64-sector reads (was 1 sector at a time)
- DiscStream: buffer swap instead of copy in decrypt_and_buffer
- Vec capacity hints in TS/PS demuxer hot paths
- NetworkStream: TLS warning documented
- Batch rip: per-title progress display
- cargo fmt: 0 violations

319 tests, 0 fmt violations.
2026-04-11 19:24:25 +00:00
MattJackson e25035fe8c Fix audit v2 criticals: DiscStream read loop, CSS crack, ISO writer
Critical fixes:
- DiscStream: persistent read state (was creating new ContentReader per call)
  Full error recovery, AACS/CSS decryption, extent tracking across reads
- CSS crack: labeled 'outer continue (was targeting wrong loop)
- CSS crack: LFSR0 polynomial fixed to match cipher (shifts 8,1,3,7)
- CSS lfsr: operator precedence clarified in LFSR0 init

Warning fixes:
- ISO writer: UDF tag checksums computed (was zeros)
- DVD extents: saturating_add for overflow safety
- Removed dead fields: HandshakeResult.error, ContentReader.content_format
- Added TODO for ISO long_ad >4GB support

319 tests, 20 clippy warnings remaining.
2026-04-11 19:11:25 +00:00
MattJackson ffa0eaba4d cargo fmt + clippy --fix: 104 format violations fixed, 8 clippy auto-fixes 2026-04-11 19:10:20 +00:00
MattJackson 96a65de3ff Chapters, DVD subtitle palette, MKV track flags, progress total_bytes
Chapters:
- MPLS PlayList marks parsed (mark_type 1 = chapter)
- Chapter struct on DiscTitle (time_secs, name)
- MKV Chapters element with EditionEntry/ChapterAtom per mark
- 3 MPLS mark tests + 2 MKV chapter tests

DVD subtitle palette:
- IFO palette extraction (PGC offset 0xA4, 16 × YCbCr colors)
- YCbCr→RGB conversion for VobSub .idx format
- DvdSubParser codec_private returns formatted palette
- codec_data field on SubtitleStream flows through pipeline
- 5 palette tests (YCbCr conversion, formatting, overflow)

MKV track flags:
- FlagDefault: primary video/audio = 1, secondary = 0
- FlagForced: forced subtitles = 1
- Language: set from stream language code
- Already implemented, verified with 4 new tests

Progress total_bytes:
- IOStream trait: total_bytes() -> Option<u64>
- DiscStream, IsoStream: from disc_title.size_bytes
- M2tsStream, MkvStream: from file metadata on open
- NetworkStream, StdioStream, NullStream: None

316 tests total, all passing.
2026-04-11 17:43:47 +00:00
MattJackson cd575b7221 Add MKV muxer, IsoWriter, disc pipeline, and network tests
- MKV muxer: EBML header, segment, cluster, cues, multi-track, keyframe flags — 6 tests
- MkvStream: BD-TS roundtrip, metadata preservation — 2 tests
- IsoWriter: valid UDF, file size update, custom names, empty content — 4 tests
- Disc pipeline: format detection (UHD/BD/DVD), content format, capacity, duration — 5 tests
- Network: listen/connect roundtrip, metadata flow — 2 tests (ignored for CI)
- Encryption: no AACS dir, no keydb — 2 tests
- 297 tests total, all passing
2026-04-11 17:27:36 +00:00
MattJackson 515f2f6bc2 100% codec coverage + disc/ and aacs/ module refactors
Codec coverage (DVD + BD + UHD):
- E-AC-3 (Dolby Digital Plus): bsid detection, frame size calc — 8 tests
- DTS-HD MA/HR: extension substream (0x64582025) detection — 8 tests
- LPCM: BD header skip, raw PCM extraction — 6 tests
- DVD VobSub subtitles: passthrough parser — 5 tests
- Dolby Vision: verified RPU NAL type 62 preserved in HEVC — 1 test

Module refactors:
- disc.rs → disc/mod.rs + bluray.rs + dvd.rs + encrypt.rs
- aacs/mod.rs (1661 lines) → mod.rs (21) + keydb.rs + keys.rs + decrypt.rs
- All public APIs preserved, all tests pass

270 tests total, 0 failures.
2026-04-11 17:22:18 +00:00
MattJackson fe723a7759 Add crypto roundtrip tests: CSS + AACS validation
- CSS: decrypt_key determinism, descramble XOR roundtrip, TAB1 permutation,
  TAB4 bit-reversal involution, Stevenson attack on scrambled sector
- AACS: decrypt_unit roundtrip, disc hash deterministic, VUK derivation,
  unit key parsing, EC point-on-curve and ECDSA already covered
- 239 tests total
2026-04-11 17:17:29 +00:00
MattJackson e4c5c88909 CSS crypto tests + DVD pipeline fully wired
- CSS roundtrip tests: decrypt_key determinism, descramble XOR roundtrip
- CSS table verification: TAB1 is permutation, TAB4 is bit-reversal involution
- DVD scan pipeline confirmed: scan_dvd_titles, CSS crack, ContentReader descramble
- 229 tests, all passing
2026-04-11 17:14:58 +00:00
MattJackson 648ce28ac6 Rewrite CSS from Stevenson 1999 paper — proper table-driven cipher
- tables.rs: 5 CSS specification tables (TAB1-TAB5, mathematical constants)
- lfsr.rs: Table-driven LFSR1 (TAB2/TAB3) + LFSR0 (TAB4), sector seed XOR,
  decrypt_key() mangling function, descramble_sector() with proper feedback
- crack.rs: Stevenson divide-and-conquer attack (2^16 LFSR1 iteration,
  LFSR0 deduction from known plaintext, 10-byte validation)
- No external code copied — original Rust implementation from the 1999 paper
- 225 tests, 0 ignored
2026-04-11 17:07:21 +00:00
MattJackson ff5547363b Audit fixes + DVD support foundation (IFO, PS demux, MPEG-2, CSS crack)
Audit fixes (14 critical, 22 warnings):
- UDF: bounds checks on all ICB/FID parsing from disc data
- SCSI Linux: saturating_sub on residual, CDB length guard, buffer size guard
- SCSI macOS: SCSITaskStatus u32 (was u8 — stack corruption)
- AACS: EC mod_inv returns infinity instead of panic, key reduced mod n
- AACS: do_handshake tries all host certs (was returning on first failure)
- H.264: bounds check on SPS < 4 bytes
- ContentReader: error on missing unit key (was zero-fill)
- KEYDB: flat redirect loop (was recursive), 100MB response limit, Windows HOME fallback
- ISO writer: AVDP extent order, partition length, allocation cap
- Network: removed TCP_NODELAY on bulk stream
- MKV: guard on u64::MAX seek
- disc.rs: saturating_sub on extent offset, simplified dead region code
- cargo fmt (610 violations), cargo clippy --fix (55 auto-fixes)

DVD support (new files):
- src/ifo.rs — IFO parser (VIDEO_TS.IFO, VTS_XX_0.IFO, PGC chains, cells, streams) — 13 tests
- src/mux/ps.rs — MPEG-2 Program Stream demuxer (pack headers, PES, private stream 1) — 12 tests
- src/mux/codec/mpeg2.rs — MPEG-2 video parser (sequence headers, I-frame detection) — 15 tests
- src/css/crack.rs — split-attack algorithm (LFSR cipher needs verification — test ignored)

226 tests total (was 186), 1 ignored (CSS crack needs cipher verification).
2026-04-11 16:52:22 +00:00
MattJackson 6e771a1867 v0.7.2: Windows SPTI, 177 tests, platform file separation 2026-04-11 16:15:14 +00:00
MattJackson 8999db68ea Refactor: drive discovery into platform files, no inline cfg
- drive/unix.rs: find_drives() + resolve_device() for Linux/macOS
- drive/windows.rs: find_drives() + resolve_device() + normalize_path() for Windows
- drive/mod.rs: clean delegation, no cfg branches
- scsi/windows.rs: SPTI transport only, no drive discovery
2026-04-11 16:12:53 +00:00
MattJackson 2d67b11a2c Add Windows SPTI backend, CI check, platform support complete
- scsi/windows.rs: SCSI_PASS_THROUGH_DIRECT via DeviceIoControl
- Device path normalization (D:, \\.\CdRom0, \\.\D:)
- Windows drive discovery (CdRom0-15 + drive letters)
- CI: cargo check on windows-latest
- Platform table: Linux + macOS + Windows all supported
2026-04-11 16:08:30 +00:00
MattJackson 995525d3ff Add 113 tests, update CI to checkout@v5, add FEATURES.md
Test suite: 64 → 177 tests
- MPLS parser: 6 tests (synthetic binary, streams, errors)
- CLPI parser: 6 tests (EP map, PTS/SPN math, errors)
- H.264: 12 tests (NAL parsing, SPS/PPS, keyframes)
- HEVC: 13 tests (VPS/SPS/PPS, IRAP range, codec private)
- AC3: 12 tests (syncword, frame extraction)
- VC1: 15 tests (BITMAPINFOHEADER, start codes)
- DTS: 5, TrueHD: 4, PGS: 4 tests
- EBML: 6 tests (size/ID/string/float roundtrips)
- UDF: 10 tests (MockSectorReader, filesystem parsing, error paths)
- Disc: 8 tests (scan_image, DiscTitle helpers)
- Streams: 5 new (meta roundtrip, MkvStream)
- NullStream: 4, StdioStream: 2, IsoSectorReader: 2

CI: actions/checkout@v4 → v5 (all workflows)
FEATURES.md: created for v0.7.1
2026-04-11 16:02:49 +00:00
MattJackson dc4ebd7d9b v0.7.1: SectorReader trait, IsoStream, StdioStream, resolve_encryption
- SectorReader trait decouples disc scanning from SCSI
- Disc::scan_image() for ISO and any sector source
- resolve_encryption() handles AACS 1.0/2.0/none in one path
- IsoStream: full UDF/MPLS/CLPI/labels pipeline from ISO files
- StdioStream: stdin/stdout pipe
- Strict scheme:// URL format with validation
- Labels module refactored to SectorReader
- 7 stream types total
2026-04-11 15:49:29 +00:00
MattJackson 168005ef34 Fix update-readme: use ORG_DISPATCH_TOKEN to bypass branch protection 2026-04-11 15:04:22 +00:00
MattJackson 6dc308f245 Bump to v0.7.0 2026-04-11 14:55:52 +00:00
MattJackson cccb6b4631 Add StdioStream, IsoStream; enforce scheme:// URL format
- StdioStream: stdin/stdout pipe, format-agnostic
- IsoStream: read BD-TS from Blu-ray ISO images
- URL resolver: bare paths rejected, all URLs require scheme:// prefix
- Validation: empty paths, missing ports, read-only/write-only errors
- Tests: 22 passing (URL parsing, validation, metadata roundtrip)
- Docs: full stream table with 7 stream types
2026-04-11 14:54:28 +00:00
MattJackson 8c2f3898b8 Add IOStream trait and stream-based I/O architecture
Introduce IOStream trait for uniform read/write across disc, file,
network, and null streams. Rename Title→DiscTitle, add stream URL
resolver, split old stream.rs into focused modules (m2ts, mkvstream,
network, disc, null, resolve, meta).
2026-04-10 19:13:53 -07:00
MattJackson 37c98e8826 Support multiple host certs, expose handshake_error on AacsState
- host_cert: Option → host_certs: Vec (try all until one succeeds)
- handshake_error field shows why auth failed (e.g. cert rejected/revoked)
2026-04-10 09:48:29 -07:00
MattJackson 891dee3db9 Bump to v0.6.1 — republish with open() all drives, AACS 2.0, has_profile() 2026-04-10 09:30:15 -07:00
MattJackson e34b41967d v0.6.0: changelog, open() all drives, AACS 2.0, MKV muxer 2026-04-10 09:26:45 -07:00
MattJackson c063dbd76e Update README for v0.6.0: AACS 2.0 done, open() all drives, .cargo gitignore 2026-04-10 09:24:01 -07:00
MattJackson d1f8db93d6 Remove orphaned jar.rs — replaced by labels/ module 2026-04-10 09:17:11 -07:00
MattJackson 926939f24b open() works on all drives, AACS 2.0 handshake wired, raw_gc_010c on DriveId
- DriveSession::open() no longer requires profile match — works on any optical drive
- init()/probe_disc() return error gracefully for unknown drives
- find_drives() returns all optical drives (PDT 0x05), not just profile-matched
- has_profile() check for callers
- AACS 2.0: handshake wired into resolve_aacs() — real VID + read_data_key
- DriveId: added raw_gc_010c field for GET_CONFIG 010C response bytes
2026-04-10 08:48:41 -07:00
MattJackson 48d70c092c Fix test failures: correct EBML size assertion, fix doc tests
- Fix write_size test: 127 encodes as [0x40, 0x7F] not [0xC0, 0x7F]
- Mark doc examples as ignore (use incomplete pseudo-code)
2026-04-10 08:23:32 -07:00
MattJackson c14b319212 Fix compiler warnings: remove unused imports and dead code
- Remove unused NAL_SLICE and NAL_SEI constants from h264 parser
- Remove unused aacs import from Disc::scan()
2026-04-10 08:21:55 -07:00
MattJackson 55b22bfb9d Add design docs for API and MKV muxer architecture 2026-04-10 08:19:53 -07:00
MattJackson 48b212c9d2 Add MKV muxer and event system
- Add event.rs: structured event system for progress reporting
- Add mux/: MKV muxer pipeline (EBML writer, TS demuxer, stream assembly)
  - Codec parsers: H.264, HEVC, AC-3, DTS, TrueHD, PGS, VC-1
  - Lookahead buffer for codec private data extraction
  - Direct m2ts-to-MKV streaming without intermediate files
2026-04-10 08:19:40 -07:00
MattJackson 074f21ba58 Refactor error types: replace generic AacsError/DiscError with typed variants
- Split AacsError { detail } into 13 specific error variants (AacsCertShort,
  AacsAgidAlloc, AacsCertRejected, etc.) with unique error codes E7001-E7012
- Split DiscError { detail } into 7 specific variants (DiscRead, MplsParse,
  ClpiParse, UdfNotFound, DiscNoTitles, DiscTitleRange, DiscNoExtents)
- Add WriteError (E5001), KeydbLoad (E8005), MuxLookahead (E9000), MuxWrite (E9001)
- Add OpenDisc API for single-call open+scan+rip workflow
- Remove all English text from error Display impl (code-only output)
- Normalize doc comments to use -- instead of em dash for ASCII consistency
2026-04-10 08:19:28 -07:00
MattJackson 9241d767ee Code audit: fix unused variables, update docs
- Fix unused variable warnings in profile parsing (|e| -> |_|)
- Fix outdated 'freemkv info --share' reference in README (now drive-info)
2026-04-10 08:19:01 -07:00
MattJackson 65996ade59 Remove SpeedTable, add probe_disc(), named constants, clean architecture
- Removed SpeedTable entirely — drive manages speeds after probe
- Renamed read_speed_table() → probe_disc()
- Named all SCSI constants: SUB_CMD_UNLOCK, SUB_CMD_INIT, SUB_CMD_PROBE,
  INIT_ADDR_BD, INIT_ADDR_UHD, PROBE_COARSE_END, PROBE_FINE_END, etc.
- Auto-detect BD vs UHD from disc capacity for correct probe init address
- Fixed NOMINAL_SPEED_B (was invalid CDB, removed — single max instead)
- Added session.set_speed() for simple speed control
- Error recovery: re-init on first error, BD2x on repeated errors
- Batch size uses full kernel limit (was 80%, now 100%)
- Clean variant_a/variant_b with named constants

API: open() → wait_ready() → init() → probe_disc() → scan() → read
2026-04-09 15:17:25 -07:00
MattJackson 09f5bb7816 Speed table: generic zone-based speed management
- SpeedTable: maps disc positions to optimal speeds
- Default: max speed everywhere (drive manages itself)
- After read_speed_table(): calibrated per-zone speeds
- One u32 comparison per read on hot path
- Error recovery: reduce() / resume() override table temporarily
- Replaces old tier-based speed management in ContentReader
- MT1959 split into mod.rs + variant_a.rs + variant_b.rs
- PlatformDriver: init() + read_speed_table() + is_ready()
2026-04-09 13:33:02 -07:00
MattJackson 12a5985a58 Update README: v0.6.0 API, real sample output 2026-04-09 13:03:00 -07:00
MattJackson d23632931b Profiles v2: chipset+variant top-level keys, minimal per-drive data
profiles.json: { "mt1959_a": [...], "mt1959_b": [...], "renesas": [] }
Each profile: identity + signature + firmware (3 fields)
Platform enum replaces Chipset — section determines variant
2026-04-09 12:50:44 -07:00
MattJackson 92e7692779 Bump to v0.6.0 2026-04-09 12:39:50 -07:00
MattJackson 0c11623666 v0.6.0: Clean API, chipset-keyed profiles, streamlined platform driver
- API: open() is OEM-only, wait_ready() separate, init() optional
- Profiles: chipset-keyed JSON ({ "mt1959": [...], "renesas": [] })
- Profiles: identity group, variant + signature + firmware per drive
- Platform constants: mode, buffer_id, nominal speed, verify commands
  moved from profiles to code (variant-determined, not per-drive)
- Removed unused fields: register CDBs, speed tables, status data
- Platform driver: unlock + firmware upload + calibrate + speed only
- Cross-compile fix: build.rs uses CARGO_CFG_TARGET_OS for framework linking
2026-04-09 12:38:02 -07:00
MattJackson b454d100c6 Update profiles: fix data extraction offset
Firmware payload was extracted 12 bytes too early — included handler
table pointers instead of actual microcode. Fixed by scanning for
function table position (VM address pattern) instead of magic offsets.

All 206 profiles regenerated. Cold boot firmware upload should now work.
2026-04-09 08:37:23 -07:00
MattJackson 097473d371 Update README and CHANGELOG for v0.5.0
- 12-23 MB/s read speeds, kernel transfer limit auto-detection
- Full custom firmware init pipeline (all 10 handlers)
- MPLS parser fixes (PGS language offset, secondary streams)
- 206 profiles with full per-drive data
2026-04-09 07:27:10 -07:00
MattJackson 5e3157b18d Fix subtitle language parsing: PGS in audio slots, secondary PG entries
Two MPLS parsing issues:

1. PGS subtitle (0x90) appearing in audio stream slot: language was read
 from audio offset sa[2..5] instead of PG offset sa[1..4], causing
 truncation ("ng " instead of "eng"). Fixed by detecting PGS coding
 type in audio slots and using correct offset.

2. Secondary PG (PiP subtitle) entries were not consumed, causing
 position tracking to drift. Added missing n_pip_pg loop.

3. Added stream_type 5/6/7 attribute parsing for secondary audio/video.
2026-04-08 22:41:07 -07:00
MattJackson d9c5f7a0d2 Fix read speed: detect kernel max transfer size, fix phantom streams
Root cause: READ_10 requested 510 sectors (1MB) but kernel sg driver
limits to max_hw_sectors_kb=120 (60 sectors). All bulk reads errored,
error handler fell to 3-sector reads → 4.8 MB/s instead of 12+ MB/s.

- detect_max_batch_sectors() now resolves sg→block device via sysfs
 and uses 80% of kernel limit for safety margin
- Default fallback reduced from 510 to 48 sectors
- ContentReader uses per-device detected max, not hardcoded constant
- Filter out coding_type 0x00 (empty/padding) stream entries
- Add MPLS stream_type 5/6/7 attribute parsing (secondary audio/video)

Tested on BU40N: 4.8 → 12.5 MB/s sustained, 23.7 MB/s peak.
2026-04-08 22:30:08 -07:00
MattJackson 6456e24bb5 Lock down Platform trait: pub(crate), only init/set_read_speed/is_ready
Platform trait is no longer publicly exported. External code uses
DriveSession only — cannot call unlock, load_firmware, calibrate directly.

- Platform trait: pub(crate) with 3 methods only
- All handlers are private methods on Mt1959
- DriveStatus moved to mt1959 internal struct
- init() has guard: no re-init if already ready
- set_read_speed() has guard: no-op if not calibrated
- Removed open_unlocked() — open() is the only entry
- Removed Platform and DriveStatus from public exports

Prevents: out-of-sequence SCSI commands, double-init, wrong firmware writes.
2026-04-08 21:41:57 -07:00
MattJackson 0b19154bd3 Clean pipeline: one open, one init, no double-init
- Removed open_unlocked() — open() is the only entry
- Removed redundant init() call from open_title()
- init() called once in open(), handles everything
- Each function does one thing: open→init→scan→read
2026-04-08 21:35:48 -07:00
MattJackson ecc6cd9f6b Fix init() hang: make status() non-fatal, single attempt
status() (sub_cmd 0x13) returns ILLEGAL REQUEST on some drives.
Was retrying 6× with 30s timeouts = 180s hang during init().

Steps 1-10 all pass on hardware:
 unlock: OK, load_firmware: OK, calibrate: OK,
 register_a: OK, register_b: OK
Only status fails — not needed for reads.
2026-04-08 21:27:34 -07:00
MattJackson 71a21b5c6f Fix PGS subtitle misclassified as audio in STN parsing
When MPLS STN table parsing drifts (disc-specific alignment issue),
a PGS subtitle entry (coding_type 0x90/0x91) can appear in the audio
stream section. Previously this showed as garbled "ng PGS 5.1" audio.

Fix: guard in stream builder checks if audio-typed streams have
subtitle codecs and reclassifies them as subtitles.

Also: unknown stream types now filtered out (filter_map) instead of
creating fake Video entries that showed as "?" in output.

Tested on V for Vendetta BD — "ng PGS 5.1" gone, clean output.
2026-04-08 21:14:39 -07:00
MattJackson 79b1b4d5b5 Bump to v0.5.0 2026-04-08 20:53:54 -07:00
MattJackson 85b38de4a7 Update profiles.json: 206 drives with complete data
Generated by: profile generator --profiles sdf0.bin keys.json --drive-db drive_profiles.json
206 profiles (66 A + 140 B), all with identity from brute-force dictionary.
No manual merging. One tool, three inputs, complete output.
2026-04-08 20:51:15 -07:00
MattJackson 686bf91bb5 Update profiles.json: 206 drives
New profile format includes all -verified fields:
- ld_microcode (base64): firmware payload for WRITE_BUFFER/MODE_SELECT
- hardware_register_a/b_cdb: pre-built 10-byte SCSI CDBs
- drive_nominal_speed_cdb: calibration SET_CD_SPEED
- speed_zone_table, speed_calc_table: operation speed constants
- unlock_init_value, unlock_response_size_minus_init
- drive_signature: per-drive unlock check

206 profiles (66 mt1959_a + 140 mt1959_b), 661KB.
Identity fields merged from drive database.
All 32 tests pass.
2026-04-08 20:46:40 -07:00
MattJackson f07f8210c3 Fix variant B firmware upload: MODE SELECT not WRITE_BUFFER
B firmware upload () byte-level verification reveals:
- Step 1: MODE SELECT (0x55), NOT WRITE_BUFFER — sends 2496 bytes (0x9C0)
- Step 2: Check result == 2
- Step 3: READ_BUFFER mode=6 offset=0x3000 (16B firmware metadata)
- Step 4: WRITE_BUFFER mode=6 (16B from fw_write_data)
- Step 5: Vendor verify CDB (0xF1 opcode from blob)
- Step 6: do_unlock × 5 retries + 1 confirmation

Key differences from A:
- A uses WRITE_BUFFER (0x3B), B uses MODE SELECT (0x55)
- A sends 1888 bytes, B sends 2496 bytes
- B has extra READ metadata + WRITE 16B + vendor verify steps
- B retries unlock 5 times (A does 2)

Added profile fields: fw_write_data (16B), verify_cdb (10B) for B-only.
2026-04-08 20:40:37 -07:00
MattJackson 4153d23652 mt1959: separate A/B firmware upload paths
A (): single WRITE_BUFFER → verify 0x45 → unlock×2
B (): WRITE handshake → READ 0x3000 → WRITE 16B → verify → unlock×5

9/10 handlers are identical A/B. Only load_firmware has different logic.
Both paths end with do_unlock() — firmware upload is a prerequisite for
unlock, not a substitute. init() tries unlock first, falls back to
load_firmware only on failure (cold boot).
2026-04-08 20:35:43 -07:00
MattJackson 760bab0893 mt1959.rs: complete platform implementation
Every handler traced instruction-by-instruction from operation: do_unlock with configurable response size
operation: WRITE_BUFFER + verify buf=0x45 + unlock×2
operation: do_unlock → validate → send pre-built CDB → [4:20]
operation: same with CDB B
operation: init → scan 0x0000-0x5800 → build table → triple speed
operation: ↔x86 VM only (host_write 16B), no SCSI
operation: do_unlock → validate → probe 0x13 → check sig → features
operation: 3 paths by param count (1/5/9), dynamic READ_BUFFER
operation: search 64-entry table → position probe →
 set_cd_speed_max → custom SET_CD_SPEED with matched value
operation: ↔x86 VM only (host_read 8B), no SCSI

init() matches x86 dispatch exactly:
 Phase 1: unlock → [load_fw] × 6
 Phase 2: calibrate × 6
 Phase 3: probe (drive info)
 Phase 4: register A + B × 5
 Phase 5: status × 6

Handlers 5/9 are VM communication (no SCSI equivalent in Rust).
All other handlers send real SCSI commands.
2026-04-08 20:30:46 -07:00
MattJackson ce8ddb48bb DriveProfile with all per-drive fields, drive.rs uses init() as single entry
DriveProfile now has every field traced from firmware:
- drive_signature, unlock_init_value, unlock_response_size_minus_init
- ld_microcode (base64, ~1888B firmware payload)
- hardware_register_a_cdb, hardware_register_b_cdb (10B pre-built CDBs)
- drive_nominal_speed_cdb (12B calibration speed)
- speed_zone_table (28B), speed_calc_table (25B)

drive.rs simplified:
- open() calls init() instead of unlock()
- init() is the ONLY entry point — handles full dispatch sequence internally
- Removed read_config, read_register, maintain_speed, read_sectors from public API
- Added set_read_speed() for per-zone speed during content reads
- disc.rs updated to call init() instead of unlock()

Compiles clean, all tests pass.
2026-04-08 20:15:25 -07:00
MattJackson 377cbe0aec Rewrite mt1959.rs: complete platform driver with full profile support
Complete rewrite of MT1959 platform driver:
- All 10 handlers implemented matching firmware logic 1:1
- load_firmware(): WRITE_BUFFER ld_microcode on cold boot
- calibrate(): full zone probe + speed table + triple SET_CD_SPEED
- init(): x86 dispatch sequence (unlock → fw × 6, calibrate × 6)
- read_register_a/b(): use pre-built CDBs from profile
- set_read_speed(): speed table lookup per zone
- status(), probe(), keepalive(), timing()

Platform trait updated:
- Renamed read_config → load_firmware (matches actual function)
- Added init() for full x86 dispatch sequence
- Renamed read_sectors → set_read_speed (operation sets speed, not reads)
- Split read_register into read_register_a/b (separate CDBs)

Profile fields used:
- drive_signature, unlock_init_value, unlock_response_size_minus_init
- ld_microcode (1888B firmware payload)
- hardware_register_a_cdb, hardware_register_b_cdb (pre-built CDBs)
- drive_nominal_speed_cdb (calibration triple-play)
- speed_zone_table, speed_calc_table (operation lookups)
2026-04-08 20:11:26 -07:00
MattJackson a4c10887cf Make unlock non-fatal in open_title — BD discs work without it
BD drives reject the MediaTek unlock command (sense 0x05).
Unlock is only needed for UHD raw access. Standard BD reads
work with standard READ(10) without vendor unlock.
2026-04-08 17:30:09 -07:00
MattJackson 3fcab4d8d9 Strip to bare minimum for speed test: no calibration, no maintain_speed
Back to basics: open, unlock, SET CD SPEED max, read.
Remove all calibration probes, register reads, maintain_speed calls.
This is closest to the build that hit 17 MB/s earlier.

Also: drive discovery moved to libfreemkv (find_drive, resolve_device),
AACS via UDF only, clean pipeline, sg device support.
2026-04-08 15:46:42 -07:00
MattJackson f46d9706eb Fix rip: extent LBA offset, u16 truncation, batch reads, read_content
- Fixed: extents were relative to m2ts file, not absolute disc LBAs
- Fixed: u16 truncation of remaining sector count (13M → 36!)
- Added: UdfFs::file_start_lba() for m2ts LBA lookup
- Added: DriveSession::read_content() with 30s timeout for bulk reads
- Added: SET CD SPEED 0xFFFF on title open
- Added: adaptive batch reading (96→48→3 on error, ramp back up)
- Rip working end-to-end: scan → AACS → decrypt → write
2026-04-08 10:12:07 -07:00
MattJackson 9ecbf0ada5 Clean warning, AACS 2.0 full handshake + P-256 2026-04-08 08:27:13 -07:00
MattJackson 82746ed592 AACS 2.0: full P-256 handshake, HC2 parsing, stubbed for credentials
- Full aacs2_authenticate_p256(): AGID → P-256 cert exchange → ECDSA
 signatures → ECDH bus key. Complete SCSI payload format (132-byte
 certs, 128-byte key+sig).
- Falls back: tries AACS 1.0 first, P-256 only if drive rejects v1.
- HC2 KEYDB parsing: | HC2 | HOST_PRIV_KEY 0x... | HOST_CERT 0x...
 (32-byte private key, 132-byte certificate)
- P-256 key pair generation for ephemeral handshake
- AACS 2.0 cert verification with LA public key
- Waiting on: AACS 2.0 host credentials (HC2 entry in KEYDB)
- SKB: detected (use_skb_mkb flag) but not processed — VUK from KEYDB
 still works for all discs

32 tests passing.
2026-04-08 08:03:40 -07:00
MattJackson da0eeae6a7 AACS 2.0: P-256 curve, SHA-256 ECDSA, bus decryption pipeline
- Added P-256/secp256r1 curve constants
- Added AACS 2.0 LA public key for drive cert verification
- Implemented P-256 ECDSA sign/verify (SHA-256)
- Implemented P-256 ECDH for bus key derivation
- Added aacs2_authenticate() entry point (falls back to AACS 1.0)
- Extended HostCert with optional v2 credentials
- Added sha2 dependency
- Added keydb updater (HTTP GET via raw TCP, zip/gzip extraction)
- 4 new P-256 tests: generator, order, ECDSA, ECDH — all passing
- 32 total tests passing
2026-04-07 21:30:09 -07:00
MattJackson 262f9a7f8d Add keydb updater, update README with labels + multi-lingual + real output 2026-04-07 21:21:18 -07:00
MattJackson ca732d57e7 Bump to v0.4.1 2026-04-07 21:01:31 -07:00
MattJackson f357e0fe26 README: update dep version to 0.4, update architecture 2026-04-07 20:57:32 -07:00
MattJackson 976aeb4848 Labels: catch panics — never break disc scan 2026-04-07 20:36:26 -07:00
MattJackson 2bea69c7d9 Release v0.4.0: labels rewrite, eject, capture fix 2026-04-07 20:35:11 -07:00
MattJackson 0bb815ca22 Restructure labels: detect-then-parse, named parsers, raw disc data
Architecture:
- Each BD-J format in own file: paramount.rs, criterion.rs, pixelogic.rs, ctrm.rs
- Standard interface: detect() → bool, parse() → Option<Vec<StreamLabel>>
- PARSERS array in mod.rs — drop in a new parser with one line
- Shared vocab.rs for BD spec codec names only (MLP→TrueHD, AC3→Dolby Digital)
- All other label data passes through raw from disc — no guessing

Changes:
- New: paramount.rs (playlists.xml — Paramount/onQ format)
- Renamed: bluray_project.rs → pixelogic.rs
- Renamed: stream_properties.rs → criterion.rs
- Merged: language_streams.rs + menu_base.rs → ctrm.rs
- Removed: jar module (superseded by labels), dead apply functions
- Added: DriveSession::eject() with PREVENT ALLOW MEDIUM REMOVAL
- Added: DiscRegion enum (Free/BluRay/Dvd)
- Fixed: capture sector ranges now include all files (only skip STREAM/)
- Renamed: StreamLabel.region → variant (not a BD spec field)
2026-04-07 20:29:44 -07:00
MattJackson aa188751e4 Simplify labels API: one call, labels on streams or nothing
labels::apply(session, udf, titles) does everything internally.
Disc::scan() is one line: crate::labels::apply().
No intermediate variables, no fallback logic in disc.rs.

App reads disc.titles[].streams — labels already applied if
disc had config files, empty otherwise. MPLS data always there.
2026-04-07 18:46:29 -07:00
MattJackson ba44f7d928 Add labels module: 4 disc file parsers for stream labels
src/labels/ with 4 parsers tried in order:
1. language_streams.txt (Warner CTRM CSV)
2. menu_base.prop (Warner CTRM properties)
3. streamproperties.xml + playbackconfig.xml (Criterion XML)
4. bluray_project.bin (Pixelogic binary tokens)

Disc::scan() calls labels::extract() → apply_disc_labels().
If no disc files found, streams keep MPLS data as-is.
No JAR fallback — disc files or nothing.

Covers 4/8 discs with JARs (Dunkirk UHD, V for Vendetta BD,
Being There, Barbie). Remaining 3 (Civil War, Dune, V for
Vendetta UHD) have no disc config files.
2026-04-07 18:42:33 -07:00
MattJackson 8a95787426 JAR bytecode tracer POC — extract display names from BD-J enum classes
Traces <clinit> bytecode to find ldc/putstatic pairs that map
enum field names to display name strings. Pattern: new X, dup,
ldc "English", invokespecial X.<init>, putstatic X.a.

Proven on Dune UHD JAR: aw.a="English", aw.b="French", etc.
Next: trace mapping class (ISO code → enum field) to complete the chain.
2026-04-07 16:54:30 -07:00
MattJackson aebe6a256b Fix JAR label matching: by language+codec for label format, by index for TextField
Labels like eng_MLP_ now match to the correct stream by checking language
and codec hint against the stream's properties. Fixes Barbie where labels
were swapped (TrueHD label was on DD stream). TextField format (Civil War)
still uses index matching.

TrackLabel now carries language + codec_hint for structured matching.
2026-04-07 16:27:47 -07:00
MattJackson 791b832f25 Rewrite JAR parser: multi-format with TextField support
- Extract all class strings once, try format parsers in chain
- Format 1: TextField,Audio{N} (A24/Lionsgate) — fixes Civil War
- Format 2: eng_MLP_ label strings (Warner UHD) — Barbie, Dune
- Format 3: playlist-only (MAIN_FEATURE etc.)
- Simplified TrackLabel to just description + raw
- 5 unit tests for both formats
- Tested on 12 disc captures: Civil War now gets 3 audio + 2 sub labels

Known issues: Barbie labels swapped, Dune returns 0 labels (different format?)
2026-04-07 16:23:14 -07:00
MattJackson 580ce2277d Add serial_number to DriveId from GET_CONFIG 0x0108 2026-04-07 16:14:55 -07:00
MattJackson 8a556c5c63 Merge JAR labels into streams during Disc::scan()
apply_jar_labels() matches JAR audio/subtitle labels to streams by
position and sets AudioStream.label. Apps read labels directly from
streams instead of doing their own JAR matching.
2026-04-07 16:10:43 -07:00
MattJackson 520a3f912c Refactor Stream to enum with typed variants (Video, Audio, Subtitle)
Each stream type has only its relevant fields. No more HDR on audio
or channels on video. Added SubtitleStream.forced field (TODO: parse).
Removed display helpers from lib (belongs in CLI).
2026-04-07 16:08:45 -07:00
MattJackson 6f0eea48fa Add Clip struct to Title — expose clip references for playlist analysis
Title.clips[] contains clip_id, in/out times, duration, source_packets.
Apps can detect fake/scrambled playlists by checking unique clip count
vs total (253 clips referencing 2 unique = fake). Removed clip_count
field (use clips.len()).
2026-04-07 16:03:56 -07:00
MattJackson b01010f2d1 Fix title sizes: use pkt_count*192, fix metadata partition range
- Size from source_packet_count * 192 (matches libbluray approach)
- metadata_sector_ranges() uses actual metadata partition size from ICB
 instead of arbitrary +256 margin (fixes Top Gun missing CLIPINF)
- Store metadata_sectors in UdfFs
- Fix CLPI CPI bit-packed field parsing (partial, EP map still needs work)
2026-04-07 15:49:53 -07:00
MattJackson fd443e3a2f Add disc title, format, streams to Disc::scan() — move logic from CLI to lib
- Disc.volume_id: UDF Volume ID from PVD (always present)
- Disc.meta_title: from META/DL/bdmt_eng.xml (falls back to other languages)
- Disc.format: UHD/BluRay/DVD detected from video codec
- Disc.capacity_bytes, Disc.layers
- Disc.jar_labels: extracted from BDMV/JAR
- Fixed MPLS STN parsing: 16-byte header (was 8), proper stream entry offsets
- Streams now include: HDR, color space, Dolby Vision EL, secondary audio/video
- parse_dstring() for UDF d-string fields
- wait_ready() polls TEST UNIT READY before unlock

Tested on 12 disc captures — all return correct titles, streams, format.
2026-04-07 15:36:05 -07:00
MattJackson 8d85a9d778 Add metadata_sector_ranges() and max_metadata_sector() to UdfFs
Walk UDF tree to discover sector ranges for all metadata files
(MPLS, CLPI, JAR, AACS certs, etc). Skips STREAM/, BACKUP/,
DUPLICATE/, MKB_RO.inf, ContentHash. Exposes partition_start()
and metadata_start() getters. Used by bdemu smart capture.
2026-04-07 14:43:42 -07:00
MattJackson 237528b385 Add macOS SCSI support via IOKit SCSITaskDeviceInterface
IOKit backend for macOS optical drives. Accepts BSD device paths
(/dev/disk2), walks IORegistry to find authoring device, sends
SCSI commands through SCSITaskDeviceInterface COM vtable.
2026-04-07 13:15:06 -07:00
MattJackson 492a2966c0 Add docs/README.md — reading order and index for all documentation
Start with disc-to-rip.md (the big picture), then architecture,
drive access, disc formats (UDF/MPLS/CLPI), and AACS encryption.
2026-04-07 12:36:29 -07:00
MattJackson 5fa6b064da Remove standalone binaries — functionality lives in freemkv CLI and tests
Deleted:
- freemkv_info.rs — duplicate of freemkv drive-info CLI command
- freemkv_test.rs — duplicate of bdemu capture-disc
- aacs_test.rs — covered by inline #[test] functions (31 tests)

libfreemkv is a library. CLI tools belong in the freemkv repo.
Dev/debug tools belong in (internal).
2026-04-07 12:34:10 -07:00
190 changed files with 75444 additions and 19625 deletions
Submodule .claude/worktrees/agent-a1b258ca2bea02faf deleted from aa536f6ab3
Submodule .claude/worktrees/agent-afd076fb7145099f7 deleted from 5950156cf0
Submodule .claude/worktrees/halt-token deleted from 8693add66c
Submodule .claude/worktrees/pes-source-sink deleted from 919096b67f
Submodule .claude/worktrees/pipeline deleted from 8e7d1bea96
Submodule .claude/worktrees/round1-fixes deleted from 78b50dd5a6
Submodule .claude/worktrees/round2-decrypt-decorator deleted from 9e51ee9946
Submodule .claude/worktrees/round2-discstream-source deleted from f10c83ffe4
Submodule .claude/worktrees/round2-framesink deleted from eb04fdaffa
Submodule .claude/worktrees/round2-halt deleted from 3ca63235e0
Submodule .claude/worktrees/sector-source-sink deleted from 8c592d08e9
Submodule .claude/worktrees/writeback-file-rename deleted from e5a32a8f16
+10 -1
View File
@@ -13,6 +13,7 @@ jobs:
- uses: dtolnay/rust-toolchain@1.86.0
with:
components: clippy, rustfmt
- uses: Swatinem/rust-cache@v2
- run: cargo fmt --check
# libfreemkv is a library — Cargo.lock is gitignored. --locked
# would always fail on a fresh runner because there's no committed
@@ -25,6 +26,7 @@ jobs:
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
- run: cargo test --tests
check-macos:
@@ -32,6 +34,7 @@ jobs:
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
- run: cargo check
check-windows:
@@ -39,4 +42,10 @@ jobs:
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- run: cargo check
- uses: Swatinem/rust-cache@v2
# Build the tests (not just `cargo check`): catches errors in test
# code and forces full codegen of the Windows-only SPTI transport
# (src/scsi/windows.rs), which never compiles on the Linux/macOS dev
# hosts. We don't `cargo test` here — the suite needs no drive but the
# extra build is the value; running tests is covered by the Linux job.
- run: cargo build --tests
+35
View File
@@ -0,0 +1,35 @@
name: leak-guard
# Self-contained public-repo leak gate. Public CI cannot reach the private
# tooling, so this encodes only the generic net: internal-infra references,
# tracked CLAUDE.md/.claude paths, and AI-attribution in commit messages.
# No project-specific reverse-engineering vocabulary lives here.
on: [push, pull_request]
jobs:
leak-guard:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Compute commit range
id: range
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
base="${{ github.event.pull_request.base.sha }}"
head="${{ github.event.pull_request.head.sha }}"
echo "range=$base..$head" >> "$GITHUB_OUTPUT"
else
before="${{ github.event.before }}"
after="${{ github.sha }}"
# New branch / first push: github.event.before is all-zeros.
if [ -z "$before" ] || [ "$before" = "0000000000000000000000000000000000000000" ]; then
echo "range=$after" >> "$GITHUB_OUTPUT"
else
echo "range=$before..$after" >> "$GITHUB_OUTPUT"
fi
fi
- name: Run leak-guard
run: bash ci/leak-guard.sh "${{ steps.range.outputs.range }}"
+25 -3
View File
@@ -22,29 +22,51 @@ jobs:
fi
echo "Version match: $CARGO_VER"
# Tests run as a PARALLEL TRIPWIRE: they fail the run if they fail, but the
# publish/release jobs do NOT `needs:` this job. The tag decision was already
# gated by the local precommit (same Rust 1.86, same commit). Binary consumers
# (freemkv/autorip/bdemu) git-tag-pin libfreemkv and therefore start building
# the instant this tag exists — so this test job and the crates.io publish
# below must NOT sit on their critical path.
test:
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
# libfreemkv is a library — Cargo.lock isn't tracked, so --locked
# would always fail (no lockfile to lock against on a fresh runner).
- run: cargo test
# crates.io publish is an INDEPENDENT job: it serves EXTERNAL consumers only.
# The freemkv binaries no longer depend on it (they git-tag-pin libfreemkv via
# a committed [patch.crates-io]), so this publish runs in parallel with their
# release builds rather than gating them. It `needs: [verify, test]` so a
# failing test suite still blocks publication to crates.io — external
# consumers who `cargo add libfreemkv` must never receive a release whose
# tests were failing. (The two upstream jobs run in parallel, so this gate
# does not serialize publish behind test beyond their own completion.)
publish:
needs: test
needs: [verify, test]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
# --no-verify: CI already compiled this exact commit (in the `test` job
# and on every push via ci.yml). cargo publish's default re-verify does a
# full cold release build of the packaged tarball, which here is pure
# redundant work (~a cold lib build). Skip it.
- name: Publish to crates.io
run: cargo publish
run: cargo publish --no-verify
env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
release:
needs: test
# Only needs `verify`; the GitHub Release can be cut as soon as the version
# check passes, in parallel with test + publish.
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
+10 -1
View File
@@ -4,4 +4,13 @@ Cargo.lock
*.swo
.DS_Store
.cargo/
.claude/worktrees/
# session scratch — never track (may contain RE breadcrumbs)
scratch/
# stray local build artifact
/rust_out
# internal agent context — never publish (path AND dir; leak-guard blocks both)
CLAUDE.md
.claude/
+624 -2364
View File
File diff suppressed because it is too large Load Diff
-94
View File
@@ -1,94 +0,0 @@
# libfreemkv — Rules
## No English in library code
The library contains ZERO user-facing English text. All errors use numeric codes from `error.rs`. Applications (CLI, GUI, server) handle i18n.
- `io::Error::new(kind, "english string")` — NEVER. Use `Error::VariantName.into()`.
- If you need a new error, add a variant to `error.rs` with a code, not a string.
- Acceptable strings: debug/trace logging, test assertions, comments, data format strings (paths, codec IDs).
- `Error` implements `From<Error> for io::Error` — use `?` or `.into()` anywhere an `io::Error` is expected.
## Architecture
- **Streams are PES.** Every stream reads its format → PES frames out, or PES frames in → writes its format. One type per format.
- **Disc::copy() for sector dumps.** disc→ISO is NOT a stream. It's `Disc::copy()`.
- **DiscStream = any disc.** Physical drive or ISO file. Same type, different SectorReader.
- **No IOStream.** Deleted. No byte-level Read/Write on streams.
- **Streams don't know their size.** Progress/file_size is a CLI concern.
- **One method per action.** No `foo_with_X` variants. Use `Option<T>` params.
- **Streams impl Read only (conceptually).** No Seek, no File backing.
- **Functions return errors, only main() exits.** No `process::exit` in library code.
## Device rules
- Always use `/dev/sg*` not `/dev/sr*` for SCSI.
- `--raw` only skips decryption. Init/probe/speed still run.
- Each function does one thing. One runner orchestrates the sequence.
## AACS key sources
Single source: `keydb.cfg`. Located at `~/.config/freemkv/keydb.cfg` by
default, or pointed at via `ScanOptions::keydb_path`. The file holds
all DKs, PKs, host certs, and per-disc VUK entries. No keys are
compiled into the binary.
CSS player keys (DVD) remain compiled in — they're 1999-era public
inputs separate from the AACS key pipeline and have always lived in
`src/css/auth.rs`.
The library treats a missing `keydb.cfg` for an AACS-encrypted disc as
`Error::KeydbLoad` with the sentinel path `<no keydb in search paths>`.
CLIs render this as "no KEYDB.cfg found"; consumers can disambiguate
on the sentinel string.
## macOS IOKit transport
The macOS SCSI transport uses exclusive IOKit access, not hybrid MMC+pread.
- **C shim** (`src/scsi/macos_shim.c`):
- `shim_open_exclusive(bsd_name)`: `diskutil unmountDisk force` on target device only → find `IOBDServices` matching BSD name via IOKit registry walk → MMCDeviceInterface → SCSITaskDeviceInterface → `ObtainExclusiveAccess` → raw CDB dispatch.
- `shim_list_drives()`: registry-based enumeration. Walks all `IOBDServices` entries, reads `"Device Characteristics"` for vendor/model/firmware, walks child chain to `IOMedia` for BSD name. Zero SCSI, zero exclusive access, zero unmounts.
- `shim_execute()` / `shim_close()`: raw CDB dispatch and cleanup.
- **Build** (`build.rs`): compiles shim via `cc` into static lib, linked by Cargo. NOT the `cc` crate (produces object code that breaks IOKit exclusive access).
- **Rust** (`src/scsi/macos.rs`): FFI to `shim_open_exclusive`, `shim_close`, `shim_execute`, `shim_list_drives`. `list_drives()` uses registry-based enumeration. `MacScsiTransport::open()` uses exclusive access only when ripping a specific device.
- **IOBDServices parent chain**: IOSCSIPeripheralDeviceType05 → IOBDServices → IOBDBlockStorageDriver → IOMedia (has `"BSD Name"`). The shim walks this chain to match BSD name to IOBDServices.
- **IOKit lookup order**: (1) iterate all IOBDServices → match child IOMedia BSD name, (2) fallback: find IOMedia by BSD name → walk parent chain to IOBDServices, (3) fallback: first IOBDServices (single-drive systems).
- **Test disc**: DUNE_PART_TWO UHD, `/dev/disk6`, ~84.6 GB.
## Bad-sector handling (BU40N + Initio INIC-1618L)
Three failure modes on this USB bridge:
1. **NOT READY** (sense_key=2, ASC=0x04, ASCQ=0x3E) — most common on BU40N for bad sectors. Pause 3s, retry up to 3x, then mark NonTrimmed.
2. **Transport failure** (status=0xFF) — bridge crash, auto-recovers ~15s. Aborts copy.
3. **INCOMPATIBLE FORMAT** (ASC=0x30) wedge — ALL sectors fail, requires power cycle.
### Damage-jump algorithm (Pass 1 sweep)
When `skip_on_error=true` (multipass mode):
- Read each ECC block sequentially. Track a sliding window of the last 16 ECC block results.
- On error: zero-fill, mark NonTrimmed, push `false` to window.
- On success: write data, mark Finished, push `true` to window. Track consecutive good count.
- When ≥12% of the 16-block window are failures → **jump** ahead by `JUMP_BASE_SECTORS (1024) × batch × multiplier` sectors. For UHD encrypted ECC (batch=32) that's a 64 MiB base jump. Zero-fill the gap as NonTrimmed. Double the multiplier (64→128→256→512 MiB...) up to `MAX_JUMP_MULTIPLIER=64` (4 GiB cap). Plus a separate wedge-skip path of `WEDGE_JUMP_SECTORS=524288` (1 GiB) for HARDWARE_ERROR / ILLEGAL_REQUEST senses, capped at 16 consecutive wedges.
- When 16 consecutive good reads → reset multiplier to 1, restore max read speed.
- Only transport failures (bridge crash) abort the pass.
Tuning knobs: `DAMAGE_WINDOW=16` and `DAMAGE_THRESHOLD_PCT=12%`. Calibrated from live BU40N data: old 50/25% was too diluted by good reads between sparse failures; 16/12% triggers on the 2nd scattered failure (2/16 = 12.5% ≥ 12%).
### Patch (Pass N) — `disc/mod.rs:1910`
- Default: **reverse** mode. Walks bad ranges from highest LBA to lowest, and within each range from end to start. Rationale: sweep jumps forward with escalating gaps, so NonTrimmed ranges have good data at their tail (where the jump landed). Reverse hits good data first, converges on actual bad block boundaries.
- Single-sector reads with 60 s timeout (`READ_RECOVERY_TIMEOUT_MS`).
- NOT_READY (sense=2, ASC ∈ {0x02, 0x03, 0x04}): 15 s pause, retry without immediate Unreadable mark.
- Non-marginal SCSI sense → mark Unreadable and continue.
- Skip escalation: damage window 16, `PASSN_DAMAGE_THRESHOLD_PCT=6`, skip `PASSN_SKIP_SECTORS_BASE (32) << escalation` sectors capped at `PASSN_SKIP_SECTORS_CAP=4096`; `MAX_SKIPS_PER_RANGE=10`, then mark range Unreadable.
- Wedge exit: 50 consecutive failures **and** ≥ 2 ranges attempted (single-range stalls don't kill the pass).
- Whole-pass watchdog: `STALL_SECS = 3600` on `bytes_good`. Per-range watchdog: proportional `range_sectors × SECONDS_PER_SECTOR(25)`, capped at `RANGE_BUDGET_CAP_SECS=1800` (replaces the old flat 180s/range — tiny ranges got starved).
Constants live in `disc/patch.rs::Disc::patch` (PASSN_*, STALL_SECS, SECONDS_PER_SECTOR, RANGE_BUDGET_CAP_SECS, MAX_SKIPS_PER_RANGE). The full algorithm is documented in `freemkv-private/memory/project_recovery_v0_16.md`.
## Public repo rules
- **No internal docs.** Audit reports, test plans, roadmaps, TODOs go in freemkv-private, never here.
- **No Co-Authored-By** in commit messages. One contributor: MattJackson.
- **No private references.** No Gitea URLs, no /data/code paths, no internal IPs in code.
+83
View File
@@ -0,0 +1,83 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at matthew@pq.io. All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of actions.
**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
+15 -4
View File
@@ -1,6 +1,6 @@
[package]
name = "libfreemkv"
version = "0.26.0"
version = "1.2.1"
edition = "2024"
rust-version = "1.86"
license = "AGPL-3.0-only"
@@ -8,6 +8,15 @@ description = "Open source raw disc access library for optical drives"
repository = "https://github.com/freemkv/libfreemkv"
keywords = ["bluray", "uhd", "optical", "scsi", "disc"]
categories = ["hardware-support", "multimedia"]
# Keep internal AI-instruction / private notes out of the published crate.
exclude = ["CLAUDE.md"]
# OFF crates.io: libfreemkv git-deps freemkv-unlock (firmware, never published),
# so libfreemkv itself can only be consumed by git tag. Clients git-tag-pin it.
publish = false
[profile.release]
lto = "thin"
codegen-units = 1
[dependencies]
serde = { version = "1", features = ["derive"] }
@@ -16,7 +25,10 @@ sha1 = "0.10"
sha2 = "0.10"
aes = "0.8"
cbc = "0.1"
flate2 = "1"
# Interim path dep for local cross-repo dev; the release script re-pins this to
# `{ git = ".../freemkv-unlock", tag = "vX.Y.Z" }` before tagging libfreemkv (so
# the released tag resolves freemkv-unlock from git, not a sibling path).
freemkv-unlock = { path = "../freemkv-unlock" }
num-bigint = "0.4"
num-traits = "0.2"
num-integer = "0.1"
@@ -31,8 +43,7 @@ tracing = "0.1"
# Bounded MPSC channel with kernel-wakeup send_timeout. Used by `io::pipeline`
# so the halt-aware send/finish loops can BLOCK on consumer drain instead of
# polling — the 50 ms poll cadence of the previous mpsc-based impl capped mux
# throughput at ~1 MB/s (see freemkv-private/memory/
# feedback_send_with_halt_poll_throttle.md, 0.21.7).
# throughput at ~1 MB/s (0.21.7).
crossbeam-channel = "0.5"
# Persistent work-stealing thread pool for parallel AACS unit
# decryption. Per-call std::thread::scope spawned fresh OS threads
+1 -2
View File
@@ -1,6 +1,5 @@
# libfreemkv — local dev helper.
# Mirrors the cross-crate scripts in freemkv-private/scripts/test-all.sh
# but scoped to this single crate.
# Mirrors the workspace-wide CI checks but scoped to this single crate.
.PHONY: test build check ci clean
+20 -20
View File
@@ -1,26 +1,26 @@
[![Crates.io](https://img.shields.io/crates/v/libfreemkv)](https://crates.io/crates/libfreemkv)
[![docs.rs](https://img.shields.io/docsrs/libfreemkv)](https://docs.rs/libfreemkv)
[![License: AGPL-3.0](https://img.shields.io/badge/license-AGPL--3.0-blue)](LICENSE)
# libfreemkv
Rust library for 4K UHD / Blu-ray / DVD optical drives. Drive access, disc scanning, stream labels, AACS decryption, CSS decryption, KEYDB updates, and content reading in one crate. Bundled drive profiles — no external files needed.
Rust library for 4K UHD / Blu-ray / DVD optical drives. Drive access, disc scanning, stream labels, AACS decryption, CSS decryption, KEYDB updates, and content reading in one crate. Drive-level unlocking is handled internally; consumers work with disc access and decryption only.
Built-in keys cover DVDs and Blu-rays (AACS 1.0). For UHD (AACS 2.0 / 2.1) discs, an optional `keydb.cfg` supplies disc-specific volume unique keys.
DVDs (CSS) decrypt out of the box. Blu-ray and UHD (AACS) require a `keydb.cfg` (default `~/.config/freemkv/keydb.cfg`) supplying disc-specific volume unique keys; no AACS key material is compiled in.
**12+ MB/s** sustained read speeds on BD. Full init: unlock, firmware upload, speed calibration — all from pure Rust.
**12+ MB/s** sustained read speeds on BD. Drive prep (`init()`) handles unlocking internally via the `freemkv-unlock` crate — clients never see it; when no drive unlock applies, the library rips via the host-certificate AACS handshake.
Multi-lingual by design — the library outputs structured data and numeric error codes, never English text. Build any UI or localization on top.
**[API Documentation](https://docs.rs/libfreemkv)** · **[Technical Docs](docs/)**
**[Source & API](https://github.com/freemkv/libfreemkv)** · **[Technical Docs](docs/)**
Part of the [freemkv](https://github.com/freemkv) project.
## Install
Consumed by git tag (not published to crates.io):
```toml
[dependencies]
libfreemkv = "0.25"
libfreemkv = { git = "https://github.com/freemkv/libfreemkv", tag = "vX.Y.Z" }
```
## Quick Start
@@ -29,10 +29,10 @@ libfreemkv = "0.25"
use libfreemkv::{Drive, Disc, ScanOptions};
use std::path::Path;
// Open drive — profiles are bundled, auto-identified
// Open drive — identified via INQUIRY
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?; // wait for disc
drive.init()?; // unlock + firmware upload
drive.init()?; // unlock + prep (handled internally)
drive.probe_disc()?; // probe disc surface for optimal speeds
// Scan disc — UDF, playlists, streams, AACS (all automatic)
@@ -100,7 +100,7 @@ loop {
## What It Does
- **Drive access** — open, identify, unlock, firmware upload, speed calibration, eject
- **Drive access** — open, identify, internal unlock + prep, speed control, eject
- **12+ MB/s reads** — auto-detects kernel transfer limits, sustained full speed
- **Disc scanning** — UDF 2.50 filesystem, MPLS playlists, CLPI clip info
- **Stream labels** — 5 BD-J format parsers (Paramount, Criterion, Pixelogic, CTRM, Deluxe)
@@ -121,21 +121,21 @@ loop {
| StdioStream | Yes (stdin) | Yes (stdout) | Raw byte pipe |
| NullStream | -- | Yes | Discard sink (byte counter for benchmarks) |
Streams implement `FrameSource` (read) and/or `FrameSink` (write); direction is type-checked. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
Streams implement a single unified `pes::Stream` trait (re-exported as `PesStream`) exposing `read()` and `write()` on one type. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
### Keys
DVDs (CSS) decrypt out of the box — the 1999-era public player keys are compiled into the library.
DVDs (CSS) decrypt out of the box, with no external key file needed.
Blu-rays and UHD (AACS) require a `keydb.cfg` at `~/.config/freemkv/keydb.cfg` (or passed via `ScanOptions`). The file holds all DKs, PKs, host certs, and per-disc VUKs. No AACS key material is compiled into the binary.
Blu-rays and UHD (AACS) require a `keydb.cfg` at `~/.config/freemkv/keydb.cfg` (or passed via `ScanOptions`). No AACS key material is compiled into the binary.
## Architecture
```text
Drive — open, identify, init, unlock, single-shot read
Drive — open, identify, init, single-shot read
├── ScsiTransport — SG_IO (Linux), IOKit (macOS), SPTI (Windows)
── DriveProfile — per-drive unlock parameters (bundled)
└── PlatformDriver — MediaTek (supported), Renesas (planned)
── unlock_bridge — private seam to the freemkv-unlock crate
(firmware / AACS cert / CSS bus-auth unlockers)
Disc — scan titles, streams, AACS/CSS state
├── UDF reader — Blu-ray UDF 2.50 with metadata partitions
@@ -144,12 +144,11 @@ Disc — scan titles, streams, AACS/CSS state
├── IFO parser — DVD title sets, PGC chains, cell addresses
├── Labels — 5 BD-J format parsers (detect + parse)
├── AACS — key resolution + content decryption
├── CSS — DVD CSS cipher (table-driven, no keys needed)
├── CSS — DVD CSS (bus auth → player-key disc crack → known-plaintext title-key attack)
└── KEYDB — download + verify + save
Streams — unified PES pipeline
├── FrameSource — read() PES frames (direction-typed)
├── FrameSink — write() PES frames (direction-typed)
├── PesStream — pes::Stream: one trait, read()/write() PES frames
├── DiscStream — sectors → decrypt → TS demux → PES
├── IsoStream — ISO file → decrypt → TS demux → PES
├── MkvStream — MKV mux/demux
@@ -175,6 +174,7 @@ All errors are structured with numeric codes. No user-facing English text — ap
| E6xxx | Disc format errors |
| E7xxx | AACS errors |
| E8xxx | KEYDB update errors |
| E9xxx | Stream / mux errors (URL, PES, ISO, pipeline, demux) |
## Platform Support
@@ -186,7 +186,7 @@ All errors are structured with numeric codes. No user-facing English text — ap
## Contributing
Run `freemkv info disc:// --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to contribute your drive's profile.
Run `freemkv info disc:// --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to capture your drive's identity for contribution. Drive-unlock profiles are maintained in the [freemkv-unlock](https://github.com/freemkv/freemkv-unlock) repository.
## License
+61
View File
@@ -1,4 +1,6 @@
fn main() {
emit_git_suffix();
let target = std::env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
if target == "macos" {
println!("cargo:rustc-link-lib=framework=IOKit");
@@ -8,8 +10,22 @@ fn main() {
let obj = format!("{out_dir}/macos_shim.o");
let lib = format!("{out_dir}/libmacos_scsi.a");
// Build the shim for the TARGET arch, not the host's. A bare `cc` on an
// Apple-Silicon CI runner defaults to arm64, so cross-building to
// x86_64-apple-darwin would link a host-arch object against x86_64 Rust
// code → "Undefined symbols for architecture x86_64". (Still raw `cc`,
// not the `cc` crate, which breaks IOKit exclusive access.)
let target_arch = std::env::var("CARGO_CFG_TARGET_ARCH").unwrap_or_default();
let clang_arch: &str = if target_arch == "aarch64" {
"arm64"
} else {
&target_arch // x86_64 → x86_64
};
std::process::Command::new("cc")
.args([
"-arch",
clang_arch,
"-c",
"src/scsi/macos_shim.c",
"-o",
@@ -34,3 +50,48 @@ fn main() {
println!("cargo:rerun-if-changed=src/scsi/macos_shim.c");
}
}
/// Bake the git short hash into the build as `GIT_SUFFIX` so any muxed MKV or
/// FVI index is traceable to the exact source revision (e.g. ` (g835cc99)`).
/// Empty when git or the repo is unavailable (e.g. a crates.io tarball build),
/// leaving just the package version. Always emitted so `env!("GIT_SUFFIX")`
/// resolves on every target.
fn emit_git_suffix() {
// Version label for the muxing-app / FVI generator tag. `FREEMKV_BUILD_LABEL`
// overrides the Cargo package version when set (non-empty) — used to stamp a
// pre-release/test build without bumping Cargo.toml and disturbing the
// tag-pinned [patch] version matching. Unset → the package version.
let version = std::env::var("FREEMKV_BUILD_LABEL")
.ok()
.filter(|s| !s.trim().is_empty())
.or_else(|| std::env::var("CARGO_PKG_VERSION").ok())
.unwrap_or_default();
println!("cargo:rustc-env=FREEMKV_VERSION={version}");
println!("cargo:rerun-if-env-changed=FREEMKV_BUILD_LABEL");
let suffix = git_short_hash()
.map(|h| format!(" (g{h})"))
.unwrap_or_default();
println!("cargo:rustc-env=GIT_SUFFIX={suffix}");
// Re-run when HEAD (or the branch it points at) moves so the stamp stays
// current without a clean rebuild.
println!("cargo:rerun-if-changed=.git/HEAD");
if let Ok(head) = std::fs::read_to_string(".git/HEAD") {
if let Some(ref_path) = head.strip_prefix("ref: ") {
println!("cargo:rerun-if-changed=.git/{}", ref_path.trim());
}
}
}
fn git_short_hash() -> Option<String> {
let out = std::process::Command::new("git")
.args(["rev-parse", "--short=7", "HEAD"])
.output()
.ok()?;
if !out.status.success() {
return None;
}
let h = String::from_utf8(out.stdout).ok()?.trim().to_string();
if h.is_empty() { None } else { Some(h) }
}
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
#
# leak-guard.sh — self-contained public-repo leak gate.
#
# This is the LAST line of defense in CI. It is intentionally self-contained:
# public CI cannot reach the private tooling, so this script encodes ONLY the
# generic net — internal infrastructure references, agent-context files, and
# AI-attribution in commit messages. It deliberately contains NO project-
# specific reverse-engineering vocabulary (those words would themselves be a
# leak). The richer private scanner stays private.
#
# Fails (exit 1) if any of the following appear in the repo:
# 1. a tracked CLAUDE.md or .claude/ path (agent context — never public),
# 2. tracked file content matching the internal-infra net,
# 3. a commit message (in the given range) with AI attribution.
#
# Usage:
# leak-guard.sh [<commit-range>]
# <commit-range> optional git rev-list range to scan commit messages
# (e.g. "abc..def"). If omitted, commit-message scan is
# skipped (path + content checks always run).
set -euo pipefail
# Absolute path to this script, resolved before any cd, so we can exclude it
# from the content scan (it necessarily contains the detection patterns).
SELF_ABS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
REPO="$(git rev-parse --show-toplevel)"
cd "$REPO"
fail=0
note() { printf ' ✗ %s\n' "$1"; fail=1; }
# Internal-infra net — GENERIC ONLY. This script ships in the public repo, so
# the patterns themselves must not name any org-specific identifier (doing so
# would itself leak the infra they guard). We catch the leak *class*:
# - RFC1918 private IPv4 ranges (10/8, 172.16/12, 192.168/16),
# - private/internal/non-routable TLDs (.internal/.local/.lan/.corp/.invalid),
# - docker.internal.
# The full org-specific net (literal hostnames, service names, repo paths,
# vendor tooling, …) lives ONLY in the private scanner and never ships here.
INFRA_RE='\b10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|\b172\.(1[6-9]|2[0-9]|3[01])\.[0-9]{1,3}\.[0-9]{1,3}|\b192\.168\.[0-9]{1,3}\.[0-9]{1,3}|\.internal\b|\.local\b|\.lan\b|\.corp\b|\.invalid\b|docker\.internal'
# Home-path net — GENERIC ONLY. Catches an absolute developer home path
# committed into a tracked file (a macOS /Users/<user>/… or Linux /home/<user>/…
# path). This names NO specific user — it matches the leak *class* (any home
# path), so the pattern itself reveals nothing org- or person-specific. A real
# leak (e.g. /Users/alice/Developer/x slipping into a public RELEASE.md) trips
# this regardless of whose machine it came from. The username segment is a
# literal-username class ([A-Za-z0-9._-]) so dynamic/templated paths that build
# the user at runtime — shell `/home/$USER/`, doc `/home/<rip>/`, Rust
# `/home/{user}/` — do NOT false-positive; only a baked-in literal home leaks.
HOMEPATH_RE='/Users/[A-Za-z0-9._-]+/|/home/[A-Za-z0-9._-]+/'
# AI-attribution net (case-insensitive). "claude" matches only as a standalone
# word — NOT preceded by a dot/slash/alnum and NOT followed by .md — so legit
# mentions of CLAUDE.md / .claude/ in a commit message don't false-positive.
ATTR_RE='co-authored-by|generated with|🤖|(?<![.\/A-Za-z0-9])claude(?!\.md)'
echo "── leak-guard: tracked agent-context paths ──"
while IFS= read -r f; do
case "$f" in
CLAUDE.md|*/CLAUDE.md|.claude|.claude/*|*/.claude|*/.claude/*)
note "tracked agent-context file: $f (CLAUDE.md/.claude must never be tracked in a public repo)" ;;
esac
done < <(git ls-files)
# Match a PCRE against a file, emitting "LINE: MATCH". The pattern is passed as
# an argument (not interpolated into a //) so metacharacters like the "/" in a
# path-style token can't break the regex. Reads raw bytes so non-UTF-8 blobs
# don't abort the scan.
pcre_matches() {
perl -e '
my ($file, $re) = @ARGV;
open(my $fh, "<:raw", $file) or exit 0;
my $rx; eval { $rx = qr/$re/i }; exit 0 if $@;
while (my $l = <$fh>) { if ($l =~ /$rx/) { print "$.: $&\n"; } }
' "$1" "$2" 2>/dev/null
}
# This script's own source necessarily contains the detection patterns (e.g.
# the regex tokens in INFRA_RE), so scanning it would always self-flag. Skip it.
SELF="$(git ls-files --full-name -- "$SELF_ABS" 2>/dev/null | head -1)"
echo "── leak-guard: internal-infra references in tracked files ──"
while IFS= read -r f; do
case "$f" in *.png|*.jpg|*.jpeg|*.ico|*.gif|*.bin|*.crate|*.gz|*.zip|*.pdf) continue ;; esac
[ -n "$SELF" ] && [ "$f" = "$SELF" ] && continue
[ -f "$f" ] || continue
while IFS= read -r hit; do
[ -z "$hit" ] && continue
note "internal-infra reference: $f:$hit"
done < <(pcre_matches "$f" "$INFRA_RE")
while IFS= read -r hit; do
[ -z "$hit" ] && continue
note "[HOME-PATH] absolute home path: $f:$hit (no local home path may be committed to a public repo)"
done < <(pcre_matches "$f" "$HOMEPATH_RE")
done < <(git ls-files)
RANGE="${1:-}"
if [ -n "$RANGE" ]; then
echo "── leak-guard: AI-attribution in commit messages ($RANGE) ──"
while IFS= read -r sha; do
[ -z "$sha" ] && continue
msg="$(git log -1 --format='%B' "$sha" 2>/dev/null || true)"
# Pass the pattern as an argument (not interpolated into a //) so the
# lookbehind char class and "/" don't break the regex.
hit="$(printf '%s' "$msg" | perl -e '
my $re = $ARGV[0]; my $rx = qr/$re/i;
while (my $l = <STDIN>) { if ($l =~ /($rx)/) { print "$1\n"; last; } }
' "$ATTR_RE" | head -1 || true)"
[ -n "$hit" ] && note "commit ${sha:0:12}: message contains \"$hit\" (owner rule: zero AI attribution, ever)"
done < <(git rev-list "$RANGE" 2>/dev/null || true)
fi
echo
if [ "$fail" -ne 0 ]; then
echo "✗ leak-guard: blocking finding(s) above — DO NOT MERGE/PUBLISH"
exit 1
fi
echo "✓ leak-guard: clean"
+302
View File
@@ -0,0 +1,302 @@
# FVI — Freemkv Video Index Format
**Specification version:** 1.0 (DRAFT)\
**File extension:** `.fvi`\
**Media type:** `application/vnd.freemkv.fvi+jsonl`\
**Status:** Draft for review. This document is the normative reference for the FVI
format; implementations and downstream tools cite it by section.
---
## 1. Scope and purpose
FVI is an open, codec-agnostic, byte-exact **index of the coded pictures** in a
video bitstream, together with **provenance** back to the source medium.
An FVI document answers, for every picture in a stream, three questions:
1. **Where is it?** — the byte-exact offset of its first byte in the *source*
(the disc/ISO/file), so a reader can extract or seek to any picture without
re-parsing the whole bitstream.
2. **What is it?** — coding type, random-access capability, GOP boundary, and
(where the codec defines them) field/pulldown attributes.
3. **When is it?** — decode and presentation timestamps on a declared timescale.
FVI is **not** a container, a codec, or a copy of the bitstream. It indexes; it
never stores coded samples. It is the serialized form of an indexer's per-picture
truth — carried from the demuxer, **never reconstructed** (§9).
## 2. Conformance
The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**,
**SHOULD**, **SHOULD NOT**, **MAY**, and **OPTIONAL** are to be interpreted as
described in BCP 14 (RFC 2119, RFC 8174) when, and only when, they appear in all
capitals.
A **conformant writer** MUST emit a document that satisfies §4–§10. A
**conformant reader** MUST accept any such document and MUST ignore unknown
object members (§11) so that forward-compatible extensions do not break it.
## 3. Terminology
- **Picture** — one coded video frame (or pair of fields coded as a frame). The
unit FVI indexes.
- **Access unit (AU)** — the set of bitstream bytes that decode to exactly one
picture (ISO/IEC 14496-10 §3; ISO/IEC 23008-2 §3).
- **Coded order** — the order pictures appear in the bitstream. FVI records are
emitted in coded order.
- **GOP / coded video sequence** — a self-contained run beginning at a
random-access point.
- **Provenance** — the mapping from an AU back to the exact bytes of the physical
source it was read from (§9).
- **Source position (`src`)** — `{ file, sector, byte }`, the provenance anchor of
an AU.
## 4. Encoding
An FVI document is a sequence of **UTF-8** text lines separated by a single LF
(`U+000A`). Each non-empty line is exactly one JSON value (RFC 8259), forming a
**JSON Lines / NDJSON** stream. A writer MUST NOT emit a UTF-8 BOM. A writer MUST
NOT pretty-print: each JSON value occupies exactly one line.
The first line MUST be the **Header** object (§6). Each subsequent line is one
**Picture record** (§7), in coded order.
Rationale: line-delimited JSON is streamable (a writer appends as it indexes; a
reader processes without loading the whole file), line-addressable (picture *n*
is near line *n+1*), append-safe, and parseable by every language without a
custom grammar — while remaining a precisely specified format, not an ad-hoc dump.
A document MAY be concatenated for multiple elementary streams: each stream is its
own header line followed by its records. Readers MUST treat a Header line as the
start of a new stream section.
## 5. Document structure
```
<header> line 1 (exactly one Header object)
<record> line 2 .. N (one Picture record per picture, coded order)
[<header> <record>…] (OPTIONAL further stream sections)
```
## 6. Header object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `format` | string | MUST | Constant `"freemkv/video-index"`. Signature: a document begins with these bytes. |
| `fvi_version` | integer | MUST | Document format version. This spec defines `1`. |
| `generator` | string | SHOULD | Producing tool + version, e.g. `"freemkv/1.0.0-rc.6"`. |
| `stream` | object | MUST | The indexed elementary stream (§6.1). |
| `source` | object | MUST | Provenance root (§6.2). |
| `timescale` | integer | MUST | Ticks per second for all `pts`/`dts` (§10). E.g. `90000`. |
| `picture_count` | integer | MAY | Total pictures, if known at header time; OMITTED when streaming. |
### 6.1 `stream` object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `codec` | string | MUST | Registered codec id (Appendix B), e.g. `"mpeg2video"`, `"hevc"`. |
| `width`,`height` | integer | MUST | Coded luma dimensions in pixels. |
| `dar` | `[int,int]` | SHOULD | Display aspect ratio as `[num,den]`. |
| `frame_rate` | `[int,int]` | SHOULD | Nominal rate as exact rational `[num,den]` (e.g. `[24000,1001]`). |
| `scan` | string | MUST | `"progressive"`<br>`"interlaced"`<br>`"mbaff"` |
| `colour` | object | SHOULD | CICP per ITU-T H.273: `primaries`, `transfer`, `matrix` (integer CICP codes or registered names)<br>`range`: `"limited"` \| `"full"`<br>HDR: `mastering_display`, `max_cll`, `max_fall` per ITU-T H.273 / SMPTE ST 2086. |
| `language` | string | MAY | BCP 47 tag, if known. |
### 6.2 `source` object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `medium` | string | MUST | `"disc"`<br>`"iso"`<br>`"file"`<br>`"stream"` |
| `path` | string | MAY | Source path/label. |
| `title` | integer | MAY | Title/program number. |
| `playlist` | string | MAY | Playlist/PGC identifier. |
| `volume_id` | string | MAY | Disc volume identifier, if read. |
| `sector_size` | integer | SHOULD | Bytes per `src.sector` unit (e.g. `2048`). Lets readers convert `src` to an absolute byte offset. |
## 7. Picture record
One JSON object per coded picture, in coded order.
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `n` | integer | MUST | Coded-order index, 0-based, contiguous. |
| `src` | object | MUST | Provenance: `{ "file": int?, "sector": uint, "byte": uint }` — the offset of this AU's **first byte** in the source (§9). MUST be carried from demux, never reconstructed. |
| `type` | string | MUST | Coding type:<br>`"I"`<br>`"P"`<br>`"B"`<br>_ISO/IEC 13818-2 §6.3.9; H.264/H.265 slice types collapsed to frame type._ |
| `key` | boolean | MUST | `true` iff this picture is an intra (I) picture / parser-flagged decode-restart point (IDR / IRAP / I-picture).<br>_MPEG-2 open-GOP clean-RAP precision (`closed_gop`) is not currently distinguished — see note below._ |
| `gop` | boolean | SHOULD | `true` iff this picture begins a GOP / coded video sequence.<br>_Omitted when the implementation does not carry a distinct GOP-boundary signal._ |
| `pts` | integer\|null | SHOULD | Presentation timestamp in `timescale` ticks; `null` if unknown. |
| `dts` | integer\|null | MAY | Decode timestamp in `timescale` ticks. |
| `size` | integer | MAY | AU length in bytes; enables byte-range extraction with `src`. |
| `recovered` | boolean | MAY | `true` iff any byte of this AU came from a retried/marginal read (§9.1).<br>_Default `false`._ |
| codec ext | object | MAY | Codec-specific members under the codec's namespace (§8). |
The `type` and `key` members are **codec-agnostic** and MUST be populated for
every codec. `type` is the I/P/B coding type the parser decoded (collapsing
H.264/H.265 slice types to a frame type); where no per-picture coding is carried
(audio / synthetic frames), `type` is `"I"` for a key picture else `"P"`. `key`
is the picture's random-access flag as the codec parser sets it (IDR / IRAP /
I-picture). A writer MUST NOT emit a degraded record (`type:"?"` or `src:null`)
merely because a codec lacks per-picture coding info — those fallbacks are
reserved for a field that is genuinely unavailable (e.g. provenance absent on a
synthetic source).
> **Limitation (honest random-access).** `key` is set from the picture's
> intra / decode-restart flag. The per-picture coding model this index carries
> does **not** distinguish MPEG-2 open-GOP clean random-access points
> (`closed_gop`) from any other I-picture, so `key` is the parser-flagged
> decode-restart point, not a verified clean-RAP claim. A future revision MAY
> tighten `key` for codecs/profiles that carry that signal; readers MUST NOT
> assume present `key` precision beyond "intra / decode-restart point".
### 7.1 Interlace / pulldown fields
Codec-agnostic interlace/pulldown attributes, derived through the indexer's
per-picture coding accessors (MPEG-2: ISO/IEC 13818-2 §6.3.10). Emitted as
top-level members of the record, and ONLY when the codec actually measured the
signal — an OPTIONAL member that is omitted (not defaulted) when unknown:
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `field_order` | string | MAY | Display field order:<br>`"tff"` — top field first<br>`"bff"` — bottom field first<br>`"progressive"` — no field order applies<br>_Omitted when the codec did not signal it._ |
| `progressive` | boolean | MAY | `true` iff the picture is progressive.<br>_Omitted when the codec did not signal it._ |
| `nb_fields` | integer | MAY | Number of displayed field periods this picture occupies (the soft-telecine / 2:3 pulldown basis):<br>`1` for a single field picture<br>`2` for a normal frame<br>`3`/`4`/`6` for `repeat_first_field` pulldown per §6.3.10 |
Codecs that carry only a coding type (e.g. H.264 / HEVC / VC-1 through this
pipeline) omit `field_order` and `progressive` rather than guessing a default.
## 8. Codec model and extensibility
Core record members (§7) are codec-agnostic and present for every codec.
Codec-specific data is either (a) promoted to top-level members for a small,
registered set per codec profile (e.g. MPEG-2 §7.1), or (b) placed under an
`ext` object keyed by codec id for richer/optional data:
```json
{
"n": 42,
"type": "P",
"key": false,
"src": {
"sector": 17,
"byte": 924
},
"ext": {
"hevc": {
"temporal_id": 0,
"nal_type": 1
}
}
}
```
New codecs and members are added through Appendix B (codec registry) without a
breaking version bump, provided readers continue to ignore unknown members (§11).
## 9. Provenance and recovery semantics
`src` is **byte-exact** to the source as read. `src.sector` counts in
`source.sector_size`-byte units; `src.byte` is the offset within that sector of
the AU's first byte. For multi-file sources, `src.file` indexes a writer-declared
file list. Provenance MUST be the value observed at demux time; an implementation
MUST NOT recompute `src` by re-parsing — the point of FVI is to *carry* the truth.
### 9.1 Recovery
Because FVI is provenance-native, it can record reliability. A record with
`"recovered":true` indicates the AU's source bytes required retry/marginal-read
recovery. This lets downstream tools surface or quarantine pictures whose bytes
are not byte-identical to a clean read — a capability legacy index formats lack.
## 10. Time model
All `pts`/`dts` are integers in units of `1/timescale` seconds. `pts` is
presentation (display) time; `dts` is decode time. Records are in **coded**
(decode) order, so `pts` is not necessarily monotonic across records (B-pictures
reorder); `dts` is non-decreasing. Readers needing display order sort by `pts`.
## 11. Versioning and forward compatibility
- `fvi_version` is the document version; this spec defines `1`.
- **Additive** changes (new OPTIONAL members, new registered codecs) do NOT bump
`fvi_version`. Readers MUST ignore members they do not recognize.
- A change that alters the meaning of an existing member or makes a new member
REQUIRED bumps `fvi_version`.
- A reader encountering a higher `fvi_version` than it implements SHOULD process
the members it understands and MUST NOT reject the document solely for the
version being higher, unless a member it relies on is absent.
## 12. Conformance requirements (summary)
A conformant **writer** MUST: emit a Header first; emit records in coded order
with contiguous `n`; populate `src` from demux; use named/registered codec ids;
encode one JSON value per UTF-8 LF-terminated line.
A conformant **reader** MUST: accept any §4–§10 document; ignore unknown members;
not assume `picture_count`, `pts`, or `size` are present unless required above.
---
## Appendix A — JSON Schema (informative)
Header:
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["format", "fvi_version", "stream", "source", "timescale"],
"properties": {
"format": { "const": "freemkv/video-index" },
"fvi_version": { "type": "integer", "minimum": 1 },
"timescale": { "type": "integer", "minimum": 1 },
"stream": { "type": "object", "required": ["codec", "width", "height", "scan"] },
"source": { "type": "object", "required": ["medium"] }
}
}
```
Record:
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["n", "src", "type", "key"],
"properties": {
"n": { "type": "integer", "minimum": 0 },
"type": { "enum": ["I", "P", "B"] },
"key": { "type": "boolean" },
"src": {
"type": "object",
"required": ["sector", "byte"],
"properties": {
"file": { "type": "integer" },
"sector": { "type": "integer", "minimum": 0 },
"byte": { "type": "integer", "minimum": 0 }
}
}
}
}
```
## Appendix B — Registered codec identifiers
| `codec` | Bitstream | Field profile |
|---|---|---|
| `mpeg2video` | ISO/IEC 13818-2 | §7.1 (field_order/progressive/nb_fields) |
| `mpeg1video` | ISO/IEC 11172-2 | §7.1 |
| `h264` | ISO/IEC 14496-10 | core + `ext.h264` |
| `hevc` | ISO/IEC 23008-2 | core + `ext.hevc` |
| `vc1` | SMPTE 421M | core |
## Appendix C — Normative references
- RFC 2119, RFC 8174 — Requirement keywords (BCP 14).
- RFC 8259 — JSON.
- ISO/IEC 13818-2 — MPEG-2 video (picture coding, §6.3.96.3.10).
- ISO/IEC 14496-10 — H.264/AVC. ISO/IEC 23008-2 — H.265/HEVC.
- ITU-T H.273 — Coding-independent code points (colour primaries/transfer/matrix).
- SMPTE ST 2086 — Mastering display colour volume (HDR).
- BCP 47 — Language tags.
- RFC 9559 — Matroska (alignment of colour/field-order semantics).
+50 -264
View File
@@ -2,191 +2,45 @@
## Overview
AACS (Advanced Access Content System) is the encryption layer used by Blu-ray and UHD 4K discs to protect content. libfreemkv implements AACS decryption to enable transparent disc access.
AACS (Advanced Access Content System) is the encryption layer used by Blu-ray
and UHD 4K discs to protect content. libfreemkv implements AACS decryption so
disc access is transparent to the application.
There are two major versions:
- **AACS 1.0** -- Used by standard Blu-ray discs. Relies on a custom 160-bit elliptic curve for bus authentication and AES-128 for content encryption. Processing keys and device keys can derive the media key from the disc's Media Key Block (MKB).
- **AACS 2.0** -- Used by UHD 4K Blu-ray discs. Adds a per-sector bus encryption layer (read_data_key) on top of the standard content encryption. Uses P-256/SHA-256 for its native handshake, though drives accept AACS 1.0 host certificates for backward compatibility.
Both versions use AES-128-CBC for content decryption with a fixed initialization vector. The fundamental key hierarchy is the same: a Volume Unique Key (VUK) decrypts per-title unit keys, which in turn decrypt the content stream.
## Architecture
AACS support is split across two modules:
### `aacs.rs` -- Keys and Decryption
Handles everything related to key resolution and content decryption:
- KEYDB.cfg parsing (device keys, processing keys, host certificates, per-disc entries)
- Disc hash computation (SHA-1 of `Unit_Key_RO.inf`)
- VUK resolution chain (4 paths, described below)
- MKB record parsing and media key derivation
- Subset-difference tree traversal (AACS-G3 key derivation)
- Unit_Key_RO.inf parsing and unit key decryption
- Content Certificate parsing (AACS version detection)
- Aligned unit decryption (AES-128-CBC)
- Bus decryption (AACS 2.0 read_data_key layer)
### `aacs_handshake.rs` -- SCSI Authentication
Handles the drive-level SCSI authentication protocol:
- ECDH key agreement on the AACS 160-bit curve
- ECDSA signing and verification
- Bus key derivation
- AGID management (allocate/invalidate)
- Volume ID retrieval (encrypted with bus key, verified by AES-CMAC)
- Read Data Key retrieval (for AACS 2.0 bus decryption)
- AACS LA public key certificate verification
## Key Resolution Chain
When a disc is scanned, `resolve_keys()` attempts four paths in priority order. The first path that succeeds is used.
### Path 1: KEYDB VUK Lookup (fastest)
```
Unit_Key_RO.inf --> SHA-1 --> disc_hash --> KEYDB lookup --> VUK
```
The disc hash is computed as the SHA-1 digest of the raw `Unit_Key_RO.inf` file from the disc's `/AACS/` directory. This hash is used as the lookup key in `KEYDB.cfg`. If a matching entry contains a VUK (`V` field), it is used directly.
This is the fast path and resolves the vast majority of discs in a well-maintained KEYDB.
### Path 2: KEYDB Media Key + Volume ID
```
KEYDB media_key + Volume ID (from SCSI handshake) --> VUK derivation
```
If the disc hash is not in the KEYDB but a KEYDB entry has a matching Volume ID (`I` field) and a media key (`M` field), the VUK is derived:
```
VUK = AES-128-ECB-DECRYPT(media_key, volume_id) XOR volume_id
```
Requires a successful SCSI handshake to obtain the Volume ID.
### Path 3: MKB + Processing Keys
```
MKB (from disc) + processing_keys (from KEYDB) --> media_key --> VUK
```
Processing keys are pre-computed keys that work against specific MKB versions. For each processing key, the library:
1. Parses the MKB to extract the Verify Media Key Record (`mk_dv`), subset-difference index, and conditional values (cvalues).
2. Tries each processing key against each UV/cvalue pair: `mk = AES-DEC(pk, cvalue) XOR cvalue`.
3. Validates the derived media key: `AES-ECB(mk, mk_dv)` must produce 12 leading zero bytes.
4. Derives VUK from the validated media key and Volume ID.
### Path 4: MKB + Device Keys (Subset-Difference Tree)
```
MKB + device_keys --> subset-difference tree traversal --> processing_key --> media_key --> VUK
```
The most complex path. Each device key has an associated node number, UV value, and mask parameters that position it in the AACS subset-difference tree. The library:
1. Finds the subset-difference entry in the MKB that applies to the device key's node.
2. Traverses the tree using AACS-G3 key derivation: `aesg3(key, inc) = AES-DEC(key, seed) XOR seed`, where `seed[15]` is incremented by `inc`. Each tree node produces a left child (inc=0), a processing key (inc=1), and a right child (inc=2).
3. At each level, selects left or right based on the UV bit at the current position.
4. The resulting processing key is validated against the MKB cvalue to derive the media key.
5. VUK is derived from the media key and Volume ID.
## Content Decryption
### Aligned Units
AACS encrypts content in aligned units of 6144 bytes (3 sectors of 2048 bytes each). The encryption flag is signaled by the copy_permission_indicator bits in byte 0 of the unit (`unit[0] & 0xC0 != 0`).
### Per-Unit Key Derivation
Each aligned unit has its own decryption key derived from the CPS unit key:
1. **Derive**: AES-128-ECB encrypt the first 16 bytes of the unit (plaintext TP_extra_header) with the unit key.
2. **XOR**: XOR the encrypted result with the original 16 bytes to produce the per-unit decryption key.
3. **Decrypt**: AES-128-CBC decrypt bytes 16 through 6143 using the per-unit key and the fixed AACS IV.
4. **Clear flag**: Clear the encryption indicator bits (`unit[0] &= !0xC0`).
### Fixed IV
All AES-CBC operations in AACS use the same fixed initialization vector, defined in the AACS specification.
### Verification
After decryption, the library verifies correctness by checking for MPEG-TS sync bytes (0x47) at the expected 192-byte packet boundaries within the unit. Blu-ray transport stream packets are 192 bytes: 4-byte TP_extra_header followed by a 188-byte TS packet.
## Bus Encryption
### AACS 1.0
Standard Blu-ray discs do not use bus encryption. Content is read directly from the disc and decrypted using the unit key.
### AACS 2.0
UHD 4K discs add a per-sector bus encryption layer. The drive encrypts data as it is read from the disc, and the host must decrypt it before applying AACS content decryption.
Bus encryption uses a **read_data_key** obtained during the SCSI handshake. For each 2048-byte sector within an aligned unit, bytes 16 through 2047 are AES-128-CBC encrypted with the read_data_key and the fixed AACS IV. The first 16 bytes of each sector remain plaintext.
The full decryption pipeline for AACS 2.0:
1. **Bus decrypt**: For each sector, AES-128-CBC decrypt bytes 16..2047 with the read_data_key.
2. **Content decrypt**: Standard per-unit key derivation and AES-128-CBC decryption as described above.
## SCSI Handshake
The AACS SCSI authentication handshake establishes a shared bus key between host and drive, then uses it to securely transfer the Volume ID and read data keys.
### Protocol Flow
1. **Invalidate AGIDs**: Send REPORT KEY with format 0x3F for AGIDs 0-3 to clear stale sessions.
2. **Allocate AGID**: REPORT KEY format 0x00 returns a fresh Authentication Grant ID.
3. **Send host credentials**: SEND KEY format 0x01 transmits the host nonce (20 random bytes) and host certificate (92 bytes).
4. **Receive drive credentials**: REPORT KEY format 0x01 returns the drive nonce and drive certificate.
5. **Receive drive key**: REPORT KEY format 0x02 returns the drive's ephemeral EC key point and ECDSA signature over `host_nonce || drive_key_point`.
6. **Verify drive key**: The signature is verified against the drive's public key (extracted from its certificate). AACS 1.0 certificates are verified against the AACS LA public key.
7. **Send host key**: The host generates an ephemeral key pair, signs `drive_nonce || host_key_point` with the host private key, and sends via SEND KEY format 0x02.
8. **Compute bus key**: ECDH shared secret = `host_private_key * drive_key_point`. The bus key is the low 128 bits of the shared point's x-coordinate.
### Post-Authentication Reads
- **Volume ID**: REPORT DISC STRUCTURE format 0x80. Returns 16-byte VID encrypted with the bus key, plus an AES-CMAC MAC for integrity verification.
- **Read Data Keys**: REPORT DISC STRUCTURE format 0x84. Returns the read_data_key and write_data_key, each AES-ECB encrypted with the bus key.
### Elliptic Curve
AACS 1.0 uses a custom 160-bit Weierstrass curve (`y^2 = x^3 + ax + b mod p`) with 20-byte field elements. The library implements full EC arithmetic: point addition, doubling, scalar multiplication, modular inverse, ECDSA sign/verify, and ECDH key agreement.
## AACS 2.0 Status
AACS 2.0 discs are detected via the Content Certificate file (`Content000.cer` or `Content001.cer`). A certificate type byte of 0x01 indicates AACS 2.0.
AACS 2.0 drives are identified by their drive certificate type (0x11). These drives natively use P-256/SHA-256, but accept AACS 1.0 host certificates for backward compatibility.
Current implementation status:
- AACS 2.0 detection: **implemented** (Content Certificate parsing, drive cert type check)
- AACS 1.0 handshake with AACS 2.0 drives: **implemented** (backward compatibility mode)
- Full P-256 AACS 2.0 handshake: **not yet implemented** (prepared but rarely needed since drives accept AACS 1.0 host certs)
- Bus decryption with read_data_key: **implemented**
- Content decryption: **implemented** (same as AACS 1.0)
In practice, AACS 2.0 UHD discs work through the backward-compatible AACS 1.0 handshake path, with the addition of read_data_key bus decryption.
- **AACS 1.0** -- Used by standard Blu-ray discs.
- **AACS 2.0 / 2.1** -- Used by UHD 4K Blu-ray discs. Adds a per-sector bus
encryption layer on top of the standard content encryption. UHD drives accept
AACS 1.0 host credentials for backward compatibility.
All versions use AES-128 for content decryption. The library reads the keys it
needs from `keydb.cfg`, walks the disc's Media Key Block (MKB) to resolve the
disc's key, and decrypts the content stream. AACS-encrypted discs therefore
require a `keydb.cfg`; CSS-protected DVDs do not (see the CSS notes in the
library docs).
## How it works (feature level)
When a disc is scanned, the library:
1. Reads the disc's AACS key-input files from the `/AACS/` directory.
2. Resolves the disc's key from `keydb.cfg` — either directly from a per-disc
entry, or by walking the MKB with the keys present in the keydb.
3. Performs the drive-level SCSI authentication handshake needed to obtain the
Volume ID and, for UHD, the bus-decryption key.
4. Decrypts the content stream as titles are read.
A resolved key is verified against actual disc content before it is applied, so
a stale or wrong key fails loudly rather than producing silent garbage. If no
usable key is available for an AACS-encrypted disc, the library surfaces a
specific error (the E70xx family) describing which part of the chain was
missing, and a missing `keydb.cfg` surfaces as `Error::KeydbLoad` with the
sentinel path `<no keydb in search paths>`.
## API Usage
AACS decryption is transparent to the application. The `Disc::scan()` method handles everything automatically:
AACS decryption is transparent to the application. `Disc::scan()` handles
everything automatically:
```rust
use libfreemkv::{Drive, Disc};
@@ -203,7 +57,6 @@ if disc.encrypted {
if let Some(ref aacs) = disc.aacs {
println!("AACS {}.0", aacs.version);
println!("Key source: {}", aacs.key_source.name());
println!("Disc hash: {}", aacs.disc_hash);
if let Some(mkb_ver) = aacs.mkb_version {
println!("MKB version: {}", mkb_ver);
}
@@ -215,108 +68,41 @@ if disc.encrypted {
// Read content -- decryption is automatic
let mut reader = disc.open_title(&mut session, 0).unwrap();
while let Some(unit) = reader.read_unit().unwrap() {
// unit is 6144 bytes of decrypted content
// decrypted content
}
```
The application never touches keys, never calls decryption functions, and never manages handshakes. All of that is internal to `Disc::scan()` and `ContentReader::read_unit()`.
The application never touches keys, never calls decryption functions, and never
manages handshakes. All of that is internal to `Disc::scan()` and the content
reader.
### KEYDB Location
`ScanOptions` controls where the KEYDB is loaded from. If no explicit path is set, the library checks:
1. `~/.config/aacs/KEYDB.cfg`
2. `/etc/aacs/KEYDB.cfg`
To specify an explicit path:
`ScanOptions` controls where the keydb is loaded from. If no explicit path is
set, the library checks the standard config locations. To specify an explicit
path:
```rust
let opts = ScanOptions::with_keydb("/path/to/KEYDB.cfg");
let opts = ScanOptions::with_keydb("/path/to/keydb.cfg");
let disc = Disc::scan(&mut session, &opts).unwrap();
```
### AacsState
After a successful scan, `disc.aacs` contains an `AacsState` with:
After a successful scan, `disc.aacs` contains an `AacsState`:
| Field | Type | Description |
|-------|------|-------------|
| `version` | `u8` | AACS version (1 or 2) |
| `bus_encryption` | `bool` | Whether bus encryption is active |
| `mkb_version` | `Option<u32>` | MKB version from disc |
| `disc_hash` | `String` | SHA-1 of Unit_Key_RO.inf (hex with 0x prefix) |
| `key_source` | `KeySource` | How keys were resolved |
| `vuk` | `[u8; 16]` | Volume Unique Key |
| `unit_keys` | `Vec<(u32, [u8; 16])>` | Decrypted unit keys (CPS unit number, key) |
| `read_data_key` | `Option<[u8; 16]>` | AACS 2.0 bus decryption key |
| `volume_id` | `[u8; 16]` | Volume ID from SCSI handshake |
| `disc_hash` | `String` | Identifier for the disc's key-input files |
| `key_source` | `KeySource` | How the disc's key was resolved |
### KeySource
## keydb.cfg
| Variant | Description |
|---------|-------------|
| `KeyDb` | VUK found directly in KEYDB by disc hash |
| `KeyDbDerived` | Media key + Volume ID from KEYDB, VUK derived |
| `ProcessingKey` | MKB + processing keys from KEYDB |
| `DeviceKey` | MKB + device keys, subset-difference tree traversal |
## KEYDB.cfg Format Reference
The KEYDB.cfg file contains all cryptographic material needed for AACS decryption. Lines starting with `;` or `#` are comments.
### Device Keys
```
| DK | DEVICE_KEY 0x<key> | DEVICE_NODE 0x<node> | KEY_UV 0x<uv> | KEY_U_MASK_SHIFT 0x<shift>
```
- `key`: 16-byte AES device key (hex)
- `node`: Device node number in the subset-difference tree (hex)
- `uv`: UV value for tree positioning (hex)
- `shift`: U mask shift value (hex)
### Processing Keys
```
| PK | 0x<key>
```
- `key`: 16-byte pre-computed processing key (hex)
### Host Certificate
```
| HC | HOST_PRIV_KEY 0x<privkey> | HOST_CERT 0x<cert>
```
- `privkey`: 20-byte ECDSA private key (hex)
- `cert`: 92-byte AACS host certificate (hex)
The host certificate is used for SCSI authentication. It contains the host's public key and is signed by the AACS Licensing Administrator.
### Disc Entries
```
0x<disc_hash> = <title> | D | <date> | M | 0x<media_key> | I | 0x<disc_id> | V | 0x<vuk> | U | <unit_keys>
```
- `disc_hash`: 20-byte SHA-1 of Unit_Key_RO.inf (hex)
- `title`: Human-readable disc title
- `D`: Date tag, followed by release/rip date
- `M`: Media key tag, followed by 16-byte media key (hex)
- `I`: Disc ID tag, followed by 16-byte Volume ID (hex)
- `V`: VUK tag, followed by 16-byte Volume Unique Key (hex)
- `U`: Unit keys tag, followed by space-separated `<unit_num>-0x<key>` pairs
All fields after the title are optional. A minimal entry needs only the disc hash and VUK:
```
0x<disc_hash> = <title> | V | 0x<vuk>
```
Inline comments are supported with `;`:
```
0x<disc_hash> = <title> | V | 0x<vuk> ; MKBv77
```
`keydb.cfg` is the single source of AACS key material. It is a text file (lines
starting with `;` or `#` are comments) holding the host credentials and per-disc
entries the library uses to resolve a disc. autorip can auto-download and
refresh it from a configured URL. The library does not ship any AACS keys
compiled into the binary.
+9 -10
View File
@@ -85,10 +85,10 @@ All URLs require a `scheme://path` format. Bare paths are rejected.
// PES pipeline (frame-level) — input() returns Box<dyn FrameSource>,
// output() returns Box<dyn FrameSink>.
let input = libfreemkv::input("disc:///dev/sg4", &opts)?; // DiscStream
let input = libfreemkv::input("iso://Dune.iso", &opts)?; // IsoStream
let output = libfreemkv::output("mkv://Dune.mkv", &title)?; // MkvOutputStream
let output = libfreemkv::output("m2ts://Dune.m2ts", &title)?; // M2tsOutputStream
let output = libfreemkv::output("network://10.1.7.11:9000", &title)?; // NetworkOutputStream
let input = libfreemkv::input("iso://Movie.iso", &opts)?; // IsoStream
let output = libfreemkv::output("mkv://Movie.mkv", &title)?; // MkvOutputStream
let output = libfreemkv::output("m2ts://Movie.m2ts", &title)?; // M2tsOutputStream
let output = libfreemkv::output("network://192.0.2.10:9000", &title)?; // NetworkOutputStream
let output = libfreemkv::output("null://", &title)?; // NullOutputStream
```
@@ -171,23 +171,23 @@ libfreemkv/src/
│ └── writeback.rs sync_file_range pipeline
├── drive/ Drive (open, init, single-shot read)
│ ├── mod.rs Drive struct, init, read (single-shot), reset, eject
│ ├── capture.rs Drive profile capture for contribution
│ ├── capture.rs Raw drive SCSI capture (INQUIRY/GET_CONFIG) for contribution
│ ├── linux.rs Linux drive discovery
│ ├── macos.rs macOS drive discovery
│ └── windows.rs Windows drive discovery
├── disc/ Disc (scan, titles, AACS setup, sweep, patch)
│ ├── mod.rs Disc struct, scan, titles, formats
│ ├── sweep.rs Disc::sweep (Pass 1 forward sweep)
│ ├── mod.rs Disc struct, scan, titles, formats; Disc::copy + Disc::sweep (Pass 1)
│ ├── sweep.rs Pass 1 internal helpers (pub(super))
│ ├── patch.rs Disc::patch (Pass N retry over mapfile)
│ ├── mapfile.rs ddrescue-format mapfile
│ └── read_error.rs ReadCtx / ReadAction state machine
├── scsi/ SCSI transport (Linux SG_IO, macOS IOKit, Windows SPTI)
├── platform/ Drive unlock (MT1959 A/B)
├── unlock.rs Unlocker trait + registry (pluggable unlock seam)
├── aacs/ AACS decryption (handshake, keys, keydb, decrypt)
├── css/ DVD CSS cipher
├── decrypt.rs Unified decrypt dispatcher (AACS/CSS/None)
├── pes.rs PES frame types, FrameSource / FrameSink traits
├── sector/ Sector I/O (was sector.rs in 0.17)
├── sector/ Sector I/O
│ ├── mod.rs SectorSource, SectorSink traits
│ ├── file.rs FileSectorSource, FileSectorSink (ISO-backed)
│ └── decrypting.rs DecryptingSectorSource decorator
@@ -198,7 +198,6 @@ libfreemkv/src/
├── labels/ BD-J label extraction (5 format parsers)
├── keydb.rs KEYDB download, parse, save
├── identity.rs DriveId from INQUIRY
├── profile.rs Bundled drive profiles
├── speed.rs DriveSpeed enum
├── mux/
│ ├── mod.rs Public mux exports
+27 -25
View File
@@ -1,8 +1,10 @@
# libfreemkv Architecture
Open source optical drive access library for 4K UHD Blu-ray, Blu-ray, and DVD.
Rust library with no external dependencies at runtime -- profiles are bundled,
AACS keys are derived internally, and all SCSI communication is handled in-process.
Rust library with profiles bundled and all SCSI communication handled in-process.
AACS decryption requires an external `keydb.cfg` (default
`~/.config/freemkv/keydb.cfg`) — the derivation math is internal, but no AACS key
material is compiled in; DVD CSS player keys are the only compiled-in keys.
**Repository:** <https://github.com/freemkv/libfreemkv>
**License:** AGPL-3.0-only
@@ -15,9 +17,10 @@ AACS keys are derived internally, and all SCSI communication is handled in-proce
format handling live in the library. CLI binaries are thin wrappers that call
`Drive::open()` and `Disc::scan()`.
2. **No external files.** Bundled drive profiles are compiled into the binary via
`include_str!`. No configuration directory, no runtime file lookups for drive
support.
2. **Firmware-clean core.** libfreemkv ships no firmware, no unlock CDBs, and no
drive profiles. Drive-unlock logic is plugged in by an external crate through
the `Unlocker` trait + registry (`register_unlocker`); without one the library
still rips via the host-certificate AACS handshake.
3. **Transparent AACS.** The `ContentReader` decrypts on the fly when keys are
available. Callers read cleartext sectors without knowing whether the disc
@@ -41,11 +44,9 @@ AACS keys are derived internally, and all SCSI communication is handled in-proce
libfreemkv (lib.rs)
├── Drive Access
│ ├── drive Drive — open, identify, init, unlock, single-shot read
│ ├── drive Drive — open, identify, init, single-shot read
│ ├── scsi ScsiTransport trait + platform backends (sg async, IOKit, SPTI)
│ ├── platform/ Platform trait — per-chipset command handlers
│ │ └── mt1959 MediaTek MT1959 driver (LG, ASUS, HP)
│ ├── profile DriveProfile loading, matching, bundled JSON
│ ├── unlock Unlocker trait + registry — the pluggable unlock seam
│ ├── identity DriveId from INQUIRY + GET_CONFIG 010C
│ ├── speed DriveSpeed enum, SET CD SPEED CDB builder
│ └── event Event system for drive status callbacks
@@ -65,7 +66,7 @@ libfreemkv (lib.rs)
├── Streaming
│ ├── mux/ Stream implementations (Disc, ISO, MKV, M2TS, Network, Stdio, Null)
│ ├── pes PES frame types; FrameSource / FrameSink direction-typed traits
│ ├── pes PES frame types; the unified pes::Stream (PesStream) read/write trait
│ └── sector/ SectorSource / SectorSink traits, FileSector{Source,Sink}, DecryptingSectorSource
├── I/O Primitives
@@ -74,8 +75,7 @@ libfreemkv (lib.rs)
├── Support
│ ├── keydb KEYDB.cfg download, parse, verify, save
── error Error enum with numeric codes E1000-E8000
│ └── profile Bundled drive profiles
── error Error enum with numeric codes E1000-E8000
└── lib.rs Public API re-exports
```
@@ -89,13 +89,12 @@ Drive::open(Path::new("/dev/sg4"))
├─ scsi::open() Open /dev/sg4 (async write/poll/read)
├─ DriveId::from_drive() INQUIRY + GET_CONFIG 010C
profile::find_by_drive_id() Match against bundled profiles
├─ Platform::new() Instantiate chipset driver (Mt1959)
└─ Drive ready for init/unlock/read
Drive ready for init/read
```
After open:
- `init()` -- unlock + firmware upload + speed calibration
- `init()` -- routes to the matching registered unlocker (if any); otherwise
a no-op and the cert handshake carries the disc
- `probe_disc()` -- probe disc surface for optimal speeds
- `read(lba, count, buf, recovery)` -- single-shot read; `recovery` only selects the per-CDB timeout (1.5 s vs. 30 s)
- `wait_ready()` -- wait for disc insertion
@@ -195,17 +194,20 @@ implementing `execute()` for that OS and wiring it into `scsi::open()`.
---
## Chipset Support
## Drive Unlock
| Chipset | Drives | Status |
|---------|--------|--------|
| MediaTek MT1959 | LG, ASUS, HP | Supported (bundled profiles) |
| Renesas RS8xxx/RS9xxx | Pioneer, some HL-DT-ST | Planned |
libfreemkv carries no drive-unlock mechanism. The `Unlocker` trait + registry
(`src/unlock.rs`) is the seam: an external crate implements `Unlocker` and
registers it once via `register_unlocker(...)`. At drive-prep the registry is
walked in order and the first unlocker whose `matches()` is true is asked to
`unlock_drive()` over the raw `ScsiTransport`. If none match, the drive is left
untouched and the host-certificate AACS handshake carries the disc.
The `Platform` trait abstracts chipset-specific commands. Each chipset implements
handlers (unlock, config, register, calibrate, keepalive, status, probe,
read_sectors, timing). All handlers are accessed via SCSI READ BUFFER with
chipset-specific mode and buffer ID bytes.
The implementor owns everything firmware-specific — drive profiles, vendor CDBs,
variant logic. Concrete unlockers live in the separate
**[freemkv-unlock](https://github.com/freemkv/freemkv-unlock)** repository, never
in libfreemkv. See [`drive-access.md`](drive-access.md#drive-unlock-seam) for the
trait definition and routing.
---
+1 -1
View File
@@ -189,7 +189,7 @@ The full ripping pipeline chains three parsers:
2. **CLPI** converts those timestamps to SPN ranges, then to sector extents.
3. **UDF** provides the file's starting LBA on disc for absolute sector addressing.
The `Disc::scan()` method in `src/disc.rs` orchestrates this: for each play item in each playlist, it loads the corresponding CLPI, calls `get_extents()` with the play item's in/out times, and collects the resulting sector ranges into the title's extent list.
The `Disc::scan()` method in `src/disc/mod.rs` orchestrates this: for each play item in each playlist, it loads the corresponding CLPI, calls `get_extents()` with the play item's in/out times, and collects the resulting sector ranges into the title's extent list.
## References
+7 -7
View File
@@ -10,14 +10,14 @@ Insert disc
1. Open drive (drive/mod.rs)
│ INQUIRY → identify drive
│ Match bundled profile → chipset, unlock parameters
│ INQUIRY → identify drive (DriveId)
2. Init drive (drive/mod.rs → platform/mt1959)
Firmware upload (if needed, 10s recovery wait)
Unlock → vendor-specific command activates raw read mode
Speed calibration → probe_disc()
2. Init drive (drive/mod.rs → unlock seam)
Walk the registered-unlocker registry; first match unlocks the drive
(firmware/vendor handshakes are the unlocker's own business)
No match → drive untouched; host-cert AACS handshake carries the disc
│ Speed control → probe_disc()
3. AACS handshake (aacs/handshake.rs) — optional
@@ -98,7 +98,7 @@ drive.probe_disc()?;
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
// Stream pipeline — PES frames from any source to any output.
// 0.18: input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
// input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
// direction is type-checked, so calling .write() on an input is a compile error.
let opts = InputOptions::default();
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
+63 -89
View File
@@ -7,8 +7,9 @@ optical drives.
## Drive
`Drive` is the primary API. It owns the SCSI transport, the matched
drive profile, and the chipset-specific platform driver.
`Drive` is the primary API. It owns the SCSI transport and the drive
identity (`DriveId`); any drive-specific unlock logic lives behind the
pluggable [unlock seam](#drive-unlock-seam), not in `Drive` itself.
### Opening a Drive
@@ -16,15 +17,16 @@ drive profile, and the chipset-specific platform driver.
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
```
`open()` performs: open device → send INQUIRY → match profile → instantiate
platform driver. The drive is ready for `wait_ready()` and `init()`.
`open()` performs: open device → send INQUIRY → build `DriveId`. The drive
is ready for `wait_ready()` and `init()` (which routes through the unlock
seam).
### Drive Operations
| Method | Description |
|--------|-------------|
| `wait_ready()` | Wait for disc insertion (30s timeout, TUR polling) |
| `init()` | Firmware upload + unlock + speed calibration |
| `init()` | Route to the matching registered unlocker (if any), then prepare for reads |
| `probe_disc()` | Probe disc surface for optimal speeds |
| `read(lba, count, buf, recovery)` | Read sectors. Single-shot — no inline retries or reset. |
| `reset()` | Eject-cycle escape hatch. Caller-invoked only; not on the read path. |
@@ -32,17 +34,21 @@ platform driver. The drive is ready for `wait_ready()` and `init()`.
| `unlock_tray()` | Allow tray ejection (also runs on Drop) |
| `eject()` | Eject disc tray |
| `drive_status()` | Query physical state (disc present, tray open, etc.) |
| `has_profile()` | Whether a bundled profile matched |
| `has_profile()` | Whether a registered unlocker matches this drive |
| `close()` | Consume Drive, cleanup (also runs via Drop) |
### init() Sequence
`init()` orchestrates the full drive unlock:
`init()` routes drive preparation through the unlock seam:
1. Platform driver `run_init()` — sends vendor-specific SCSI commands
2. If firmware upload needed: upload, wait 10s for drive reset, retry
3. Speed calibration after unlock
4. Max 3 attempts before giving up
1. Walk the registered-unlocker registry; the first whose `matches()` is true
is asked to `unlock_drive()` over the raw transport.
2. Whatever that unlocker needs (firmware upload, vendor handshakes, retries)
is the unlocker's own business — libfreemkv only forwards the transport.
3. If no unlocker matches, the drive is left untouched and the library uses
the host-certificate AACS handshake.
See [Drive Unlock Seam](#drive-unlock-seam) for the trait and registry.
### read() — single-shot
@@ -164,100 +170,68 @@ date for drives where Feature 010C is unavailable.
---
## Drive Profiles
## Drive Unlock Seam
Profiles are JSON objects compiled into the binary (`profiles.json`).
Each profile contains:
| Field | Purpose |
|-------|---------|
| `vendor_id`, `product_revision`, `vendor_specific`, `firmware_date` | Matching fields |
| `chipset` | `"mediatek"` or `"renesas"` |
| `unlock_mode`, `unlock_buf_id` | READ BUFFER CDB parameters |
| `signature` | Expected 4-byte response signature |
| `unlock_cdb` | Pre-built unlock CDB (hex-encoded) |
| `register_offsets` | Offsets for hardware register reads |
| `capabilities` | Feature flags: `bd_raw_read`, `dvd_all_regions`, etc. |
Loading:
libfreemkv ships **no firmware, no unlock CDBs, and no drive profiles.** It
knows only the *seam*, never the *mechanism*. The seam is the `Unlocker`
trait plus a small process-wide registry (`src/unlock.rs`):
```rust
// Bundled (compiled-in) -- no file I/O
let profiles = profile::load_bundled()?;
pub trait Unlocker: Send + Sync {
/// Stable, language-neutral identifier (logged).
fn name(&self) -> &str;
// External file
let profiles = profile::load_all(Path::new("/path/to/profiles.json"))?;
/// True if this unlocker handles the given drive.
fn matches(&self, id: &DriveId) -> bool;
/// Put the drive into extended-access mode. The one required capability.
fn unlock_drive(&self, scsi: &mut dyn ScsiTransport, id: &DriveId) -> Result<()>;
/// Read the disc Volume ID via the drive's OEM path. Default: no-op.
fn read_volume_id(&self, _scsi: &mut dyn ScsiTransport, _id: &DriveId)
-> Result<Option<[u8; 16]>> { Ok(None) }
/// Raise the drive to its maximum read speed. Default: no-op.
fn set_max_read_speed(&self, _scsi: &mut dyn ScsiTransport, _id: &DriveId)
-> Result<()> { Ok(()) }
}
```
---
An unlocker is supplied by an **external crate** and registered once at
process start:
## Chipsets
```rust
libfreemkv::register_unlocker(Box::new(some_unlocker::Plugin::new()));
```
### MediaTek MT1959
The implementor owns everything about *how* a particular drive family is
driven — drive identification against its own profile database, firmware
upload, vendor CDBs, variant logic. libfreemkv only hands over the raw
`ScsiTransport` and the `DriveId`.
Covers all LG, ASUS, and HP optical drives. Two sub-variants share identical
logic with different SCSI parameters:
### Routing
| Variant | READ BUFFER mode | Buffer ID |
|---------|------------------|-----------|
| MT1959-A | 0x01 | 0x44 |
| MT1959-B | 0x02 | 0x77 |
At drive-prep the registry is walked in registration order; the first
unlocker whose `matches()` returns true is asked to `unlock_drive()` (and,
when needed, `read_volume_id()` / `set_max_read_speed()`). If no unlocker
matches, the drive is left untouched and the library falls back to the
standard host-certificate AACS handshake (the "OEM route"). The
`register_unlocker(...)` line is the entire plug: drop it (and the unlocker
crate) and libfreemkv still compiles and rips via the cert handshake.
The Platform trait maps to command handlers:
| Handler | Function | Description |
|---------|----------|-------------|
| 0 | `unlock()` | Send READ BUFFER, verify signature + verification bytes |
| 1 | `read_config()` | Read 1888-byte configuration block + 4-byte status |
| 2-3 | `read_register()` | Read hardware registers at profile-specified offsets |
| 4 | `calibrate()` | Probe disc surface, build 64-entry speed table |
| 5 | `keepalive()` | Periodic session maintenance |
| 6 | `status()` | Query current mode and feature flags |
| 7 | `probe()` | Generic READ BUFFER with dynamic parameters |
| 8 | `read_sectors()` | Speed lookup + SET CD SPEED + READ(10) with flag 0x08 |
| 9 | `timing()` | Timing calibration |
### Renesas (Planned)
RS8xxx/RS9xxx chipsets used in Pioneer and some HL-DT-ST drives.
Currently returns `Error::UnsupportedDrive` when a Renesas profile is matched.
---
## Why Unlock Is Needed
Optical drive firmware restricts what applications can read from disc. Without
unlock:
- **READ(10) works for unencrypted filesystem data.** UDF structures, MPLS
playlists, and CLPI clip info are readable without unlock. Standard READ(10)
works on any drive.
- **READ(10) fails for encrypted content sectors.** The drive firmware returns
SCSI errors (sense key 0x05, illegal request) when an application attempts to
read sectors containing encrypted m2ts content without prior AACS
authentication via the bus key.
- **Raw mode bypasses firmware restrictions.** After unlock, the drive accepts
READ(10) with the raw read flag (CDB byte 1 = 0x08) for all sectors,
regardless of encryption status.
### AACS Before Unlock
AACS bus authentication uses standard MMC REPORT KEY / SEND KEY commands.
On some drives these must execute before unlock. The `Disc::scan()` handles
this internally — it manages the handshake/unlock ordering automatically.
Concrete unlockers — including the firmware-unlock profile databases,
variant logic, and vendor CDBs that used to live in-tree — are maintained
in the separate **[freemkv-unlock](https://github.com/freemkv/freemkv-unlock)**
repository, never here.
---
## Speed Control
After `probe_disc()`, the platform driver maintains a speed lookup table
built by probing the disc surface. On each `read()` call, the driver:
1. Looks up the optimal speed for the target LBA.
2. Issues SET CD SPEED (0xBB) if the speed differs from current.
3. Performs the READ(10).
A matching unlocker may raise the drive to its maximum read speed via
`set_max_read_speed()` (a no-op when no unlocker matches or the unlocker
declines). The library issues SET CD SPEED (0xBB) through the generic CDB
builder; the concrete speed policy lives in the unlocker.
Available speeds:
+6 -6
View File
@@ -108,8 +108,8 @@ lines of `Mapfile::stats()` checks.
2. On success: write data to ISO, mark `+`, advance.
3. On failure (with `multipass`): zero-fill, mark `*`, advance.
4. Track a sliding window of the last 16 ECC block results. When ≥12% are failures
**damage-jump**: skip ahead by `256×batch×multiplier` sectors (8 MB base for
UHD). Double the multiplier on each jump (8→16→32→64 MB...). Zero-fill the gap as `*`.
**damage-jump**: skip ahead by `1024×batch×multiplier` sectors (64 MB base for
UHD). Double the multiplier on each jump (64→128→256→512 MB...). Zero-fill the gap as `*`.
5. On 16 consecutive good reads: reset jump multiplier to 1, restore max read speed.
6. Speed control: damage zone entry → minimum speed, exit → maximum speed.
7. Only transport failures (USB bridge crash) abort the pass.
@@ -161,8 +161,8 @@ recoveries to show for it. Recovery responsibility is now layered: layer 1
handles ranges, layer 3 handles request size, neither touches the
wedge-prone reset path.
**No `MODE SELECT` to disable drive retries.** Research showed neither ddrescue
nor MakeMKV does this. Drive firmware has access to raw analog signal, laser
**No `MODE SELECT` to disable drive retries.** Neither ddrescue
nor any consumer ripper does this. Drive firmware has access to raw analog signal, laser
power control, and drive-specific ECC tuning that userspace can't replicate —
disabling it throws away recovery headroom on marginal sectors. We fail fast
via short SG_IO timeouts in pass 1 and let the firmware work the long timeout
@@ -178,7 +178,7 @@ explicitly by callers that need an eject-cycle escape hatch — it is never
reached from a read path.
**ISO intermediate, even for single-pass.** Pass 1 always writes an ISO. The
mux stage reads the ISO via `IsoSectorReader`. For single-pass (no retries),
mux stage reads the ISO via `FileSectorSource`. For single-pass (no retries),
this adds ~2-3 min (local disk mux) but gains resumability across crashes,
re-muxability without re-ripping, and a persistent forensic artifact. Callers
who need pure speed can bypass and use `DiscStream::new(Box::new(drive), …)`
@@ -198,4 +198,4 @@ scrape vs. retry with direction reversal) if there's measured benefit.
- [ddrescue manual, Algorithm chapter](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)
- [ddrescue optical media notes](https://www.electric-spoon.com/doc/gddrescue/html/Optical-media.html)
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/sweep.rs`](../src/disc/sweep.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/mod.rs`](../src/disc/mod.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
-4325
View File
File diff suppressed because it is too large Load Diff
+500
View File
@@ -0,0 +1,500 @@
//! AACS derivation "boil-down" — one public home for the key chain.
//!
//! Thin newtypes at the API boundary and three wrapper functions over the
//! existing crypto. Nothing here re-implements a primitive: every function
//! delegates to the already-audited code in [`super::keys`] and
//! [`super::variants`], so the boil-down cannot drift from production math.
//!
//! The newtypes wrap bare `[u8; 16]` ONLY at this boundary — the crypto
//! internals continue to operate on raw arrays. They exist so a caller threads
//! the chain `DK → MK → VUK → UK` without confusing one 16-byte secret for
//! another, not to refactor the resolver.
//!
//! Chain (matches `aacs::keys::resolve_keys_classical` path 1 and
//! `aacs::keys::resolve_keys_v21` path 1 byte-for-byte):
//!
//! ```text
//! mk_from_dk(device_keys, mkb, vid) → MediaKey (Km)
//! mk_from_pk(processing_keys, mkb) → MediaKey (Km)
//! vuk_from_mk(MediaKey, Vid) → Vuk (= AES-G(Km, VID))
//! uk_from_vuk(Vuk, enc_title_keys) → [UnitKey] (decrypt_unit_key each)
//! ```
//!
//! `mk_from_dk` and `mk_from_pk` are two entry points to the SAME Media Key,
//! both via the MKB's Subset-Difference cvalue tables: the device-key path
//! recovers its Processing Key at the matching SD node and walks on to the MK;
//! the processing-key path starts from a precomputed PK. Neither needs a VID
//! (the VID enters at `vuk_from_mk`).
use super::keys::{
decrypt_unit_key, derive_media_key_and_pk_from_dk, derive_media_key_from_pk, derive_vuk,
};
use super::types::DeviceKey;
/// Volume ID (16 bytes) — read from the disc via the SCSI handshake / OEM path.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Vid(pub [u8; 16]);
/// Media Key (Km, 16 bytes) — the MKB-scoped key derived from device keys.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct MediaKey(pub [u8; 16]);
/// Volume Unique Key (VUK / Kvu, 16 bytes) — derived from `MediaKey` + `Vid`,
/// decrypts the per-disc encrypted title keys in `Unit_Key_RO.inf`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Vuk(pub [u8; 16]);
/// Processing Key (Kp, 16 bytes) — an MKB Subset-Difference key that yields the
/// Media Key. A leaked/precomputed PK in the keydb, or the intermediate PK a
/// device-key walk derives at its matching SD node.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ProcessingKey(pub [u8; 16]);
/// One decrypted per-CPS-unit AACS title key.
///
/// `idx` is the POSITIONAL index of the encrypted title key within the slice
/// handed to [`uk_from_vuk`] (i.e. its order in `Unit_Key_RO.inf`'s key-storage
/// area). The CPS-unit *number* association (the `u32` in
/// `ResolvedKeys::unit_keys`) is a higher-level concern owned by
/// [`super::keys::parse_unit_key_ro`], which pairs each positional key with its
/// declared CPS unit; this primitive only does the AES, so it surfaces position.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct UnitKey {
pub idx: u32,
pub key: [u8; 16],
}
/// Derive the Volume Unique Key from a Media Key and Volume ID.
///
/// Wraps [`derive_vuk`] verbatim: `VUK = AES-128-ECB-DECRYPT(MK, VID) XOR VID`.
/// This is byte-identical to the inline `derive_vuk(&mk, ctx.volume_id)` call in
/// every classical resolver path AND to the `Kvu = AES-G(Km, VID)` step inside
/// [`derive_media_key_variant`] (AES-G and `derive_vuk` are the same math), so
/// `vuk_from_mk(mk_from_dk(..)?, vid)` reproduces the V21 variant VUK exactly.
pub fn vuk_from_mk(mk: MediaKey, vid: Vid) -> Vuk {
Vuk(derive_vuk(&mk.0, &vid.0))
}
/// Decrypt the disc's encrypted title keys with a VUK.
///
/// Wraps [`decrypt_unit_key`] (AES-128-ECB-DECRYPT) per entry, mirroring the
/// `derive_uks` closure in `resolve_keys_classical` / `resolve_keys_v21`. The
/// returned `UnitKey::idx` is the slice position; pair with CPS-unit numbers via
/// [`super::keys::parse_unit_key_ro`] when the numbering matters.
pub fn uk_from_vuk(vuk: Vuk, enc_title_keys: &[[u8; 16]]) -> Vec<UnitKey> {
enc_title_keys
.iter()
.enumerate()
.map(|(i, enc)| UnitKey {
idx: i as u32,
key: decrypt_unit_key(&vuk.0, enc),
})
.collect()
}
/// Derive the Media Key (Km) from positioned device keys via the MKB's
/// Subset-Difference tables.
///
/// Wraps [`derive_media_key_and_pk_from_dk`] — the real SD walk the resolver
/// runs: each positioned device key is placed against the MKB's subset-diff /
/// cvalue records, recovering its Processing Key at the matching node and
/// continuing to the Media Key. Reachable for real discs whenever a device key
/// applies to the MKB. No VID is involved here — it enters at [`vuk_from_mk`].
///
/// Returns [`Error::AacsMkUnavailable`] (E7018) when no supplied device key
/// resolves the MKB — the same terminal error as [`mk_from_pk`]; no numeric
/// distinction is load-bearing at this boundary.
///
/// [`Error::AacsMkUnavailable`]: crate::error::Error::AacsMkUnavailable
pub fn mk_from_dk(device_keys: &[DeviceKey], mkb: &[u8]) -> Result<MediaKey, crate::error::Error> {
// Positioned device keys drive the real Subset-Difference MKB walk
// ([`derive_media_key_and_pk_from_dk`], the same walk the resolver runs). The
// old Media-Key-Variant path needed integrator Key Correction Data absent
// in-tree, so it Err'd for EVERY real disc (dead for both consumers —
// freemkv-keysources' DK fallback and the kdb harvester). No VID is needed
// for the Media Key; it enters only at [`vuk_from_mk`].
match derive_media_key_and_pk_from_dk(mkb, device_keys) {
Some((km, _pk)) => Ok(MediaKey(km)),
None => Err(crate::error::Error::AacsMkUnavailable),
}
}
/// Derive the Media Key (Km) from one or more Processing Keys and the disc MKB.
///
/// Wraps [`derive_media_key_from_pk`] — the Subset-Difference PK→MK walk: each
/// processing key is validated (and tree-walked) against the MKB's cvalue tables
/// (records `0x04`/`0x05`) until one yields the Media Key whose verify record
/// (`0x81`/`0x86`) matches. Unlike [`mk_from_dk`] this path is reachable for
/// real discs — a leaked/precomputed AACS Processing Key in the keydb resolves
/// the Media Key directly. No VID is involved at this step; the VID enters at
/// [`vuk_from_mk`].
///
/// Returns [`Error::AacsMkUnavailable`] (E7018) when no processing key resolves
/// the MKB — the same terminal error as [`mk_from_dk`]; no numeric distinction
/// is load-bearing at this boundary.
///
/// [`Error::AacsMkUnavailable`]: crate::error::Error::AacsMkUnavailable
pub fn mk_from_pk(
processing_keys: &[[u8; 16]],
mkb: &[u8],
) -> Result<MediaKey, crate::error::Error> {
match derive_media_key_from_pk(mkb, processing_keys) {
Some(km) => Ok(MediaKey(km)),
None => Err(crate::error::Error::AacsMkUnavailable),
}
}
/// A candidate key at any rung of the AACS ladder, handed to [`resolve_candidate`].
///
/// Each variant carries the module's existing newtype for that rung (a `Dk` is a
/// POSITIONED [`DeviceKey`] — recover an unpositioned one with
/// [`super::keys::recover_dk_position`] first).
#[derive(Debug, Clone)]
pub enum KeyCandidate {
Uk(UnitKey),
Vuk(Vuk),
Mk(MediaKey),
Pk(ProcessingKey),
Dk(DeviceKey),
}
/// The AACS key chain derived from a candidate, from [`resolve_candidate`].
///
/// PURE DERIVATION — no unit sampling, no validation. `unit_keys` holds every
/// CPS-unit key the disc's `Unit_Key_RO.inf` yields from the VUK (positional
/// order); the caller runs [`super::decrypt::unit_key_validates`] to find which
/// one actually opens the disc. Rungs above the candidate are `None` (a `Vuk`
/// candidate has no `mk`/`pk`/`dk`; a `Uk` candidate has only `unit_keys`).
#[derive(Debug, Clone)]
pub struct ResolvedChain {
/// Every unit key derived from the VUK, as `(cps_unit_number, key)` — the
/// CPS-unit numbers come from `Unit_Key_RO.inf` (via `parse_unit_key_ro`), so
/// a consumer maps `UK → CPS unit` directly. Same shape as
/// [`super::keys::ResolvedKeys::unit_keys`]. A `Uk` candidate yields exactly
/// itself, keyed by its own `idx`.
pub unit_keys: Vec<(u32, [u8; 16])>,
pub vuk: Option<Vuk>,
pub mk: Option<MediaKey>,
pub pk: Option<ProcessingKey>,
/// The positioned device key (for a `Dk` candidate).
pub dk: Option<DeviceKey>,
}
/// Derive the full AACS key chain from a candidate key of ANY ladder rung.
///
/// Runs the deterministic derivation DOWNWARD to the disc's terminal unit keys:
/// `DK → MK → VUK → UKs`, `PK → MK → VUK → UKs`, `MK → VUK → UKs`,
/// `VUK → UKs`, or `UK → itself`. Composes the module's own boil steps
/// ([`mk_from_pk`], [`vuk_from_mk`], [`uk_from_vuk`]) and parses
/// `Unit_Key_RO.inf` at the version the disc's MKB declares (48-byte stride for
/// AACS-1.0, 64 for AACS-2.x), so a multi-CPS disc yields all its unit keys from
/// the one candidate.
///
/// PURE DERIVATION: no sampling, no validation, no position recovery. Every step
/// is deterministic AES, so the returned keys are only as sound as the input
/// candidate — validate `unit_keys` against a real encrypted unit with
/// [`super::decrypt::unit_key_validates`] to prove the candidate opens the disc.
///
/// Returns `None` only when derivation itself cannot proceed: a PK its MKB
/// rejects, a `Dk` the MKB can't process, a missing VID on a path that needs
/// one, or an unparseable/empty `Unit_Key_RO.inf`.
pub fn resolve_candidate(
candidate: &KeyCandidate,
mkb: &[u8],
unit_key_ro: &[u8],
vid: Option<Vid>,
) -> Option<ResolvedChain> {
use super::keys::{AacsVersion, derive_media_key_and_pk_from_dk, mkb_type, parse_unit_key_ro};
// Boil a VUK → all unit keys, each paired with its declared CPS-unit number.
// `.inf` parsing lives here: derive the stride version from the disc's own
// MKB, then defer the VUK→unit-keys step to the shared `derive_unit_keys`
// (the one place both resolvers and this path decrypt the title keys).
let boil = |vuk: Vuk| -> Option<Vec<(u32, [u8; 16])>> {
let version = mkb_type(mkb)
.map(|t| t.generation())
.unwrap_or(AacsVersion::V10);
let ukf = parse_unit_key_ro(unit_key_ro, version)?;
if ukf.encrypted_keys.is_empty() {
return None;
}
Some(super::keys::derive_unit_keys(&ukf, &vuk.0))
};
match candidate {
KeyCandidate::Uk(uk) => Some(ResolvedChain {
unit_keys: vec![(uk.idx, uk.key)],
vuk: None,
mk: None,
pk: None,
dk: None,
}),
KeyCandidate::Vuk(v) => Some(ResolvedChain {
unit_keys: boil(*v)?,
vuk: Some(*v),
mk: None,
pk: None,
dk: None,
}),
KeyCandidate::Mk(mk) => {
let vuk = vuk_from_mk(*mk, vid?);
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(*mk),
pk: None,
dk: None,
})
}
KeyCandidate::Pk(pk) => {
let mk = mk_from_pk(std::slice::from_ref(&pk.0), mkb).ok()?;
let vuk = vuk_from_mk(mk, vid?);
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(mk),
pk: Some(*pk),
dk: None,
})
}
KeyCandidate::Dk(dk) => {
let (km, pk) = derive_media_key_and_pk_from_dk(mkb, std::slice::from_ref(dk))?;
let mk = MediaKey(km);
let vuk = vuk_from_mk(mk, vid?);
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(mk),
pk: Some(ProcessingKey(pk)),
dk: Some(dk.clone()),
})
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::aacs::decrypt::aes_ecb_encrypt;
use crate::aacs::keys::{decrypt_unit_key, derive_vuk};
/// `vuk_from_mk` must equal the inline `derive_vuk` path bit-for-bit, for
/// several known (MK, VID) vectors.
#[test]
fn vuk_from_mk_matches_inline_derive_vuk() {
let cases: [([u8; 16], [u8; 16]); 3] = [
([0x5A; 16], [0xA5; 16]),
([0x11; 16], [0x22; 16]),
(
[
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C,
0x0D, 0x0E, 0x0F,
],
[
0xF0, 0xE1, 0xD2, 0xC3, 0xB4, 0xA5, 0x96, 0x87, 0x78, 0x69, 0x5A, 0x4B, 0x3C,
0x2D, 0x1E, 0x0F,
],
),
];
for (mk, vid) in cases {
let inline = derive_vuk(&mk, &vid);
let boiled = vuk_from_mk(MediaKey(mk), Vid(vid));
assert_eq!(boiled.0, inline, "vuk_from_mk must equal derive_vuk");
}
}
/// `uk_from_vuk` must equal the inline `decrypt_unit_key` path bit-for-bit
/// and carry positional indices 0..n. Built by encrypting known plaintext
/// title keys under the VUK (the same primitive the resolver inverts).
#[test]
fn uk_from_vuk_matches_inline_decrypt_unit_key() {
let vuk = [0x5Au8; 16];
let plain_keys = [[0x11u8; 16], [0x22u8; 16], [0xCDu8; 16]];
let enc: Vec<[u8; 16]> = plain_keys
.iter()
.map(|k| aes_ecb_encrypt(&vuk, k))
.collect();
let boiled = uk_from_vuk(Vuk(vuk), &enc);
assert_eq!(boiled.len(), enc.len());
for (i, uk) in boiled.iter().enumerate() {
assert_eq!(uk.idx, i as u32, "idx must be the positional index");
// Matches the inline derive_uks closure: decrypt_unit_key(vuk, enc).
assert_eq!(uk.key, decrypt_unit_key(&vuk, &enc[i]));
// And recovers the original plaintext title key.
assert_eq!(
uk.key, plain_keys[i],
"VUK roundtrip recovers the title key"
);
}
}
/// `uk_from_vuk` on an empty slice yields no keys (no panic, no phantom idx).
#[test]
fn uk_from_vuk_empty_is_empty() {
assert!(uk_from_vuk(Vuk([0u8; 16]), &[]).is_empty());
}
/// `mk_from_dk` returns `Err(AacsMkUnavailable)` when the MKB has no
/// processable Subset-Difference tables (empty MKB, or one with no
/// mk_dv/cvalues/subdiff records) — never a wrong key, never a panic.
#[test]
fn mk_from_dk_errors_on_unprocessable_mkb() {
let dk = DeviceKey {
key: [0x11; 16],
node: 1,
uv: 1,
u_mask_shift: 0,
};
// Empty MKB → no SD records to walk → Err.
let e = mk_from_dk(std::slice::from_ref(&dk), &[]);
assert!(matches!(e, Err(crate::error::Error::AacsMkUnavailable)));
// An MKB with no complete Subset-Difference tables (mk_dv / cvalues /
// subdiff) cannot yield a Media Key, so the real walk also errors —
// never silently yields a key.
let mut mkb: Vec<u8> = Vec::new();
mkb.extend_from_slice(&[0x82, 0x00, 0x00, 0x14]); // stray data record only
mkb.extend_from_slice(&[0xAB; 16]);
let e2 = mk_from_dk(&[dk], &mkb);
assert!(matches!(e2, Err(crate::error::Error::AacsMkUnavailable)));
}
/// Build a 4-byte MKB record header (type + 3-byte big-endian total length,
/// header included) and append `body`. Mirrors the MKB record framing the
/// parser expects; no crypto.
fn mkb_record(rec_type: u8, body: &[u8]) -> Vec<u8> {
let total = 4 + body.len();
let mut rec = Vec::with_capacity(total);
rec.push(rec_type);
rec.push(((total >> 16) & 0xFF) as u8);
rec.push(((total >> 8) & 0xFF) as u8);
rec.push((total & 0xFF) as u8);
rec.extend_from_slice(body);
rec
}
/// `mk_from_pk` resolves a planted Processing Key against a synthetic MKB and
/// drives the FULL boil chain PK → MK → VUK → UK. The MKB is built with the
/// same (pk, cv, mk_dv, uv) construction the production SD walk validates, so
/// this proves a PK entry yields real Unit Keys — not just an `Ok`.
#[test]
fn mk_from_pk_drives_full_chain_to_uks() {
let pk: [u8; 16] = [
0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE,
0xFF, 0x00,
];
let mk: [u8; 16] = [
0xA0, 0xA1, 0xA2, 0xA3, 0xA4, 0xA5, 0xA6, 0xA7, 0xA8, 0xA9, 0xAA, 0xAB, 0xAC, 0xAD,
0xAE, 0xAF,
];
let uv: [u8; 4] = [0x00, 0x00, 0x04, 0x00];
// cv = AES-E(pk, mk_raw), where mk_raw is mk with the last-4-bytes-uv XOR
// pre-undone, so the validate step XORs uv back in and recovers mk.
let mut mk_raw = mk;
for a in 0..4 {
mk_raw[12 + a] ^= uv[a];
}
let cv = aes_ecb_encrypt(&pk, &mk_raw);
// mk_dv = AES-E(mk, magic||pad): AES-D(mk, mk_dv) starts with the AACS
// verify sentinel.
let mut vd = [0x11u8; 16];
vd[..8].copy_from_slice(&[0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF]);
let mk_dv = aes_ecb_encrypt(&mk, &vd);
// Synthetic MKB: type/version (0x10), verify record (0x86 = mk_dv),
// one-entry SD index (0x04 = [u_mask_shift=0][uv]), one-entry cvalue
// table (0x05 = cv).
let mut sd = vec![0u8];
sd.extend_from_slice(&uv);
let mut mkb = Vec::new();
mkb.extend_from_slice(&mkb_record(0x10, &[0, 0, 0, 0x20, 0, 0, 0, 0x52]));
mkb.extend_from_slice(&mkb_record(0x86, &mk_dv));
mkb.extend_from_slice(&mkb_record(0x04, &sd));
mkb.extend_from_slice(&mkb_record(0x05, &cv));
// PK → MK.
let got_mk = mk_from_pk(std::slice::from_ref(&pk), &mkb).expect("planted PK resolves MK");
assert_eq!(got_mk, MediaKey(mk), "mk_from_pk recovers the planted MK");
// MK → VUK → UK over an encrypted title key.
let vid = Vid([0x42u8; 16]);
let plain_uk = [0x7Eu8; 16];
let vuk = vuk_from_mk(got_mk, vid);
let enc = aes_ecb_encrypt(&vuk.0, &plain_uk);
let uks = uk_from_vuk(vuk, std::slice::from_ref(&enc));
assert_eq!(uks.len(), 1);
assert_eq!(uks[0].key, plain_uk, "PK chain recovers the title key");
// A corrupt PK resolves nothing.
let mut bad = pk;
bad[0] ^= 0xFF;
assert!(matches!(
mk_from_pk(std::slice::from_ref(&bad), &mkb),
Err(crate::error::Error::AacsMkUnavailable)
));
}
/// Minimal AACS-1.0 (48-byte stride) `Unit_Key_RO.inf` with `n` encrypted
/// unit keys — `parse_unit_key_ro` numbers CPS units 1..=n.
fn synth_inf(encs: &[[u8; 16]]) -> Vec<u8> {
let uk_pos = 32usize;
let stride = 48usize;
let n = encs.len();
let total = uk_pos + 48 + n.saturating_sub(1) * stride + 16;
let mut inf = vec![0u8; total.max(20)];
inf[..4].copy_from_slice(&(uk_pos as u32).to_be_bytes());
inf[uk_pos..uk_pos + 2].copy_from_slice(&(n as u16).to_be_bytes());
for (i, k) in encs.iter().enumerate() {
let o = uk_pos + 48 + i * stride;
inf[o..o + 16].copy_from_slice(k);
}
inf
}
/// A VUK candidate boils to ALL the disc's unit keys, each paired with its
/// declared CPS-unit number, and each key equals the VUK-decrypt of its slot.
#[test]
fn resolve_candidate_vuk_returns_all_cps_units() {
let vuk = Vuk([0x33u8; 16]);
let encs = [[0x11u8; 16], [0x22u8; 16], [0x44u8; 16]];
let inf = synth_inf(&encs);
let r = resolve_candidate(&KeyCandidate::Vuk(vuk), &[], &inf, None).expect("vuk derives");
let cps: Vec<u32> = r.unit_keys.iter().map(|(c, _)| *c).collect();
assert_eq!(
cps,
vec![1, 2, 3],
"every CPS unit surfaced, numbered from the inf"
);
for ((_, key), enc) in r.unit_keys.iter().zip(encs.iter()) {
assert_eq!(
*key,
decrypt_unit_key(&vuk.0, enc),
"key = VUK-decrypt of its slot"
);
}
assert_eq!(r.vuk, Some(vuk));
assert!(r.mk.is_none() && r.pk.is_none() && r.dk.is_none());
}
/// A bare UK candidate is terminal — it returns itself keyed by its own idx.
#[test]
fn resolve_candidate_uk_is_itself() {
let uk = UnitKey {
idx: 2,
key: [0x9u8; 16],
};
let r = resolve_candidate(&KeyCandidate::Uk(uk), &[], &[], None).expect("uk is terminal");
assert_eq!(r.unit_keys, vec![(2, uk.key)]);
assert!(r.vuk.is_none() && r.mk.is_none());
}
/// MK/PK/DK paths derive the VUK from a VID; without one, derivation stops.
#[test]
fn resolve_candidate_mk_requires_vid() {
let r = resolve_candidate(&KeyCandidate::Mk(MediaKey([1u8; 16])), &[], &[], None);
assert!(r.is_none(), "MK path returns None without a VID");
}
}
+1250 -48
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+22
View File
@@ -0,0 +1,22 @@
//! Host-certificate collection — the one libfreemkv-side concern left from the
//! old in-tree AACS handshake. The cert mutual-auth itself now lives in the
//! `freemkv-unlock` AACS unlocker; libfreemkv only gathers the certs (a
//! keysource concern) and hands them across the seam.
/// Union the host certificates a scan can offer the drive: the explicit
/// `DriveCredentials`, then each key source's `host_certs(mkb)`. Host certs are
/// keysource-served, never compiled in. `mkb` lets a source pick a
/// generation-appropriate cert (the default impl ignores it).
pub fn collect_host_certs(
opts: &crate::disc::ScanOptions,
mkb: Option<u32>,
) -> Vec<crate::aacs::HostCert> {
let mut host_certs: Vec<crate::aacs::HostCert> = Vec::new();
if let Some(c) = &opts.credentials {
host_certs.extend(c.host_certs.iter().cloned());
}
for src in &opts.key_sources {
host_certs.extend(src.host_certs(mkb));
}
host_certs
}
-436
View File
@@ -1,436 +0,0 @@
//! AACS Key Database parsing — KEYDB.cfg format.
use std::collections::HashMap;
/// Parsed AACS key database.
#[derive(Debug)]
pub struct KeyDb {
/// Device keys for MKB processing
pub device_keys: Vec<DeviceKey>,
/// Processing keys (pre-computed media keys for specific MKB versions)
pub processing_keys: Vec<[u8; 16]>,
/// Host certificate + private key for SCSI authentication
pub host_certs: Vec<HostCert>,
/// Per-disc VUK entries indexed by disc hash (hex lowercase)
pub disc_entries: HashMap<String, DiscEntry>,
}
/// A device key for MKB subset-difference tree processing.
#[derive(Debug, Clone)]
pub struct DeviceKey {
pub key: [u8; 16],
pub node: u16,
pub uv: u32,
pub u_mask_shift: u8,
}
/// Host certificate + private key for AACS SCSI authentication.
#[derive(Debug, Clone)]
pub struct HostCert {
/// AACS 1.0: 20 bytes. AACS 2.0: 32 bytes.
pub private_key: [u8; 20],
/// AACS 1.0: 92 bytes. AACS 2.0: 132 bytes.
pub certificate: Vec<u8>,
/// AACS 2.0 host private key (P-256, 32 bytes). None for AACS 1.0 only.
pub private_key_v2: Option<[u8; 32]>,
/// AACS 2.0 host certificate (type 0x11). None for AACS 1.0 only.
pub certificate_v2: Option<Vec<u8>>,
}
/// A per-disc entry from the key database.
#[derive(Debug, Clone)]
pub struct DiscEntry {
/// Disc hash (20 bytes, hex)
pub disc_hash: String,
/// Disc title
pub title: String,
/// Media Key (16 bytes) — from MKB processing
pub media_key: Option<[u8; 16]>,
/// Disc ID (16 bytes)
pub disc_id: Option<[u8; 16]>,
/// Volume Unique Key (16 bytes) — decrypts title keys
pub vuk: Option<[u8; 16]>,
/// Unit keys (title keys) indexed by CPS unit number
pub unit_keys: Vec<(u32, [u8; 16])>,
}
/// Parse a hex string like "0xABCD..." into bytes.
pub(crate) fn parse_hex(s: &str) -> Option<Vec<u8>> {
let s = s.trim().trim_start_matches("0x").trim_start_matches("0X");
if s.len() % 2 != 0 {
return None;
}
let mut out = Vec::with_capacity(s.len() / 2);
for i in (0..s.len()).step_by(2) {
out.push(u8::from_str_radix(&s[i..i + 2], 16).ok()?);
}
Some(out)
}
/// Parse hex into a fixed-size array.
pub(crate) fn parse_hex16(s: &str) -> Option<[u8; 16]> {
let v = parse_hex(s)?;
if v.len() != 16 {
return None;
}
let mut out = [0u8; 16];
out.copy_from_slice(&v);
Some(out)
}
pub(crate) fn parse_hex20(s: &str) -> Option<[u8; 20]> {
let v = parse_hex(s)?;
if v.len() != 20 {
return None;
}
let mut out = [0u8; 20];
out.copy_from_slice(&v);
Some(out)
}
impl KeyDb {
/// Construct an empty KeyDb. Used by unit tests; production code
/// reaches a populated KeyDb via [`KeyDb::load`] or [`KeyDb::parse`].
pub fn empty() -> Self {
KeyDb {
device_keys: Vec::new(),
processing_keys: Vec::new(),
host_certs: Vec::new(),
disc_entries: HashMap::new(),
}
}
/// Parse a KEYDB.cfg file from a string.
pub fn parse(data: &str) -> Self {
let mut db = KeyDb {
device_keys: Vec::new(),
processing_keys: Vec::new(),
host_certs: Vec::new(),
disc_entries: HashMap::new(),
};
for line in data.lines() {
let line = line.trim();
// Skip comments and empty lines
if line.is_empty() || line.starts_with(';') || line.starts_with('#') {
continue;
}
// Device Key
if line.starts_with("| DK") {
if let Some(dk) = Self::parse_device_key(line) {
db.device_keys.push(dk);
}
continue;
}
// Processing Key
if line.starts_with("| PK") {
if let Some(pk) = Self::parse_processing_key(line) {
db.processing_keys.push(pk);
}
continue;
}
// Host Certificate (AACS 2.0)
if line.starts_with("| HC2") {
if let Some(hc) = db.host_certs.last_mut() {
if let Some((pk, cert)) = Self::parse_host_cert_v2(line) {
hc.private_key_v2 = Some(pk);
hc.certificate_v2 = Some(cert);
}
}
continue;
}
// Host Certificate (AACS 1.0)
if line.starts_with("| HC") {
if let Some(hc) = Self::parse_host_cert(line) {
db.host_certs.push(hc);
}
continue;
}
// Disc entry: starts with 0x
if line.starts_with("0x") && line.contains(" = ") {
if let Some(entry) = Self::parse_disc_entry(line) {
db.disc_entries.insert(entry.disc_hash.clone(), entry);
}
}
}
db
}
/// Load a KEYDB.cfg from disk.
pub fn load(path: &std::path::Path) -> std::io::Result<Self> {
let data = std::fs::read_to_string(path)?;
Ok(Self::parse(&data))
}
/// Look up a disc by its hash. Returns the VUK if found.
pub fn find_vuk(&self, disc_hash: &str) -> Option<[u8; 16]> {
let hash = disc_hash
.trim()
.to_lowercase()
.trim_start_matches("0x")
.to_string();
// Try with 0x prefix and without
self.disc_entries
.get(&format!("0x{hash}"))
.or_else(|| self.disc_entries.get(&hash))
.and_then(|e| e.vuk)
}
/// Look up a disc by its hash. Returns the full entry.
pub fn find_disc(&self, disc_hash: &str) -> Option<&DiscEntry> {
let hash = disc_hash
.trim()
.to_lowercase()
.trim_start_matches("0x")
.to_string();
self.disc_entries
.get(&format!("0x{hash}"))
.or_else(|| self.disc_entries.get(&hash))
}
// ── Parsers ─────────────────────────────────────────────────────────────
fn parse_device_key(line: &str) -> Option<DeviceKey> {
// | DK | DEVICE_KEY 0x... | DEVICE_NODE 0x... | KEY_UV 0x... | KEY_U_MASK_SHIFT 0x...
let key_str = line.split("DEVICE_KEY").nth(1)?.split('|').next()?.trim();
let node_str = line.split("DEVICE_NODE").nth(1)?.split('|').next()?.trim();
let uv_str = line.split("KEY_UV").nth(1)?.split('|').next()?.trim();
let shift_str = line
.split("KEY_U_MASK_SHIFT")
.nth(1)?
.split(';')
.next()?
.split('|')
.next()?
.trim();
Some(DeviceKey {
key: parse_hex16(key_str)?,
node: u16::from_str_radix(node_str.trim_start_matches("0x"), 16).ok()?,
uv: u32::from_str_radix(uv_str.trim_start_matches("0x"), 16).ok()?,
u_mask_shift: u8::from_str_radix(shift_str.trim_start_matches("0x"), 16).ok()?,
})
}
fn parse_processing_key(line: &str) -> Option<[u8; 16]> {
// | PK | 0x...
let parts: Vec<&str> = line.split('|').collect();
if parts.len() >= 3 {
let key_str = parts[2].split(';').next()?.trim();
return parse_hex16(key_str);
}
None
}
fn parse_host_cert(line: &str) -> Option<HostCert> {
// | HC | HOST_PRIV_KEY 0x... | HOST_CERT 0x...
let priv_str = line
.split("HOST_PRIV_KEY")
.nth(1)?
.split('|')
.next()?
.trim();
let cert_str = line
.split("HOST_CERT")
.nth(1)?
.split(';')
.next()?
.split('|')
.next()?
.trim();
Some(HostCert {
private_key: parse_hex20(priv_str)?,
certificate: parse_hex(cert_str)?,
private_key_v2: None,
certificate_v2: None,
})
}
/// Parse AACS 2.0 host cert: `| HC2 | HOST_PRIV_KEY 0x... | HOST_CERT 0x...`
fn parse_host_cert_v2(line: &str) -> Option<([u8; 32], Vec<u8>)> {
let priv_str = line
.split("HOST_PRIV_KEY")
.nth(1)?
.split('|')
.next()?
.trim();
let cert_str = line
.split("HOST_CERT")
.nth(1)?
.split(';')
.next()?
.split('|')
.next()?
.trim();
let priv_bytes = parse_hex(priv_str)?;
if priv_bytes.len() != 32 {
return None;
}
let mut pk = [0u8; 32];
pk.copy_from_slice(&priv_bytes);
let cert = parse_hex(cert_str)?;
if cert.len() < 132 {
return None;
}
Some((pk, cert))
}
fn parse_disc_entry(line: &str) -> Option<DiscEntry> {
// 0x<hash> = <title> | D | <date> | M | 0x<mk> | I | 0x<id> | V | 0x<vuk> | U | <unit_keys>
let (hash_part, rest) = line.split_once(" = ")?;
let disc_hash = hash_part.trim().to_lowercase();
// Extract title (before first |)
let title_part = rest.split(" | ").next().unwrap_or("").trim();
// Clean title: "TITLE_NAME (Display Title)" → use display title if present
let title = if let Some(start) = title_part.find('(') {
if let Some(end) = title_part.rfind(')') {
title_part[start + 1..end].to_string()
} else {
title_part.to_string()
}
} else {
title_part.to_string()
};
// Parse fields by tag
let mut media_key = None;
let mut disc_id = None;
let mut vuk = None;
let mut unit_keys = Vec::new();
let parts: Vec<&str> = rest.split(" | ").collect();
let mut i = 0;
while i < parts.len() {
match parts[i].trim() {
"M" => {
if i + 1 < parts.len() {
media_key = parse_hex16(parts[i + 1].trim());
i += 1;
}
}
"I" => {
if i + 1 < parts.len() {
disc_id = parse_hex16(parts[i + 1].trim());
i += 1;
}
}
"V" => {
if i + 1 < parts.len() {
vuk = parse_hex16(parts[i + 1].trim());
i += 1;
}
}
"U" => {
if i + 1 < parts.len() {
// Unit keys: "1-0xKEY" or "1-0xKEY ; comment"
let uk_str = parts[i + 1].split(';').next().unwrap_or("").trim();
for uk in uk_str.split(' ') {
let uk = uk.trim();
if let Some((num, key)) = uk.split_once('-') {
if let Ok(n) = num.parse::<u32>() {
if let Some(k) = parse_hex16(key) {
unit_keys.push((n, k));
}
}
}
}
i += 1;
}
}
_ => {}
}
i += 1;
}
Some(DiscEntry {
disc_hash,
title,
media_key,
disc_id,
vuk,
unit_keys,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Get KEYDB path from KEYDB_PATH environment variable. Returns None if not set or not found.
fn keydb_path() -> Option<std::path::PathBuf> {
let path = std::path::PathBuf::from(std::env::var("KEYDB_PATH").ok()?);
if path.exists() { Some(path) } else { None }
}
#[test]
fn test_parse_disc_entry() {
let line = r#"***REMOVED*** = DUNE_PART_TWO (Dune: Part Two) | D | 2024-04-02 | M | ***REMOVED*** | I | ***REMOVED*** | V | ***REMOVED*** | U | 1-***REMOVED*** ; MKBv77"#;
let entry = KeyDb::parse_disc_entry(line).unwrap();
assert_eq!(entry.title, "Dune: Part Two");
assert!(entry.media_key.is_some());
assert!(entry.vuk.is_some());
assert_eq!(entry.unit_keys.len(), 1);
assert_eq!(entry.unit_keys[0].0, 1);
}
#[test]
fn test_parse_device_key() {
let line = "| DK | DEVICE_KEY ***REMOVED*** | DEVICE_NODE 0x0800 | KEY_UV 0x00000400 | KEY_U_MASK_SHIFT 0x17 ; MKBv01-MKBv48";
let dk = KeyDb::parse_device_key(line).unwrap();
assert_eq!(dk.node, 0x0800);
assert_eq!(dk.u_mask_shift, 0x17);
}
#[test]
fn test_parse_host_cert() {
let line = "| HC | HOST_PRIV_KEY ***REMOVED*** | HOST_CERT ***REMOVED*** ; Revoked";
let hc = KeyDb::parse_host_cert(line).unwrap();
assert_eq!(hc.private_key[0], 0x90);
assert_eq!(hc.certificate.len(), 92);
}
#[test]
fn test_parse_full_keydb() {
let path = match keydb_path() {
Some(p) => p,
None => return,
}; // skip if not available
let db = KeyDb::load(&path).unwrap();
assert_eq!(db.device_keys.len(), 4);
assert_eq!(db.processing_keys.len(), 3);
assert!(!db.host_certs.is_empty());
assert!(db.disc_entries.len() > 170000);
// Look up Dune: Part Two
let dune = db
.disc_entries
.values()
.find(|e| e.title.contains("Dune: Part Two") && e.vuk.is_some())
.expect("Dune: Part Two not found");
assert!(dune.media_key.is_some());
assert!(dune.vuk.is_some());
assert!(!dune.unit_keys.is_empty());
eprintln!(
"Parsed {} disc entries, {} DK, {} PK",
db.disc_entries.len(),
db.device_keys.len(),
db.processing_keys.len()
);
}
}
+2142 -532
View File
File diff suppressed because it is too large Load Diff
+94 -11
View File
@@ -8,33 +8,116 @@
//! | DK | DEVICE_KEY 0x... | DEVICE_NODE 0x... | KEY_UV 0x... | KEY_U_MASK_SHIFT 0x...
//! | PK | 0x...
//! | HC | HOST_PRIV_KEY 0x... | HOST_CERT 0x...
//! | HC2 | HOST_PRIV_KEY 0x... | HOST_CERT 0x...
//! 0x<disc_hash> = <title> | D | <date> | M | 0x<media_key> | I | 0x<disc_id> | V | 0x<vuk> | U | <unit_keys>
//!
//! The VUK decrypts title keys from AACS/Unit_Key_RO.inf on disc.
//! Title keys decrypt m2ts stream content (AES-128-CBC).
pub mod boil;
pub mod decrypt;
pub mod handshake;
pub mod keydb;
pub mod host_certs;
pub mod keys;
pub mod provider;
pub mod trace;
pub mod types;
pub mod variants;
pub mod verify_magics;
/// On-disc UDF paths to the AACS key-input files (with their fallbacks).
/// Centralised so every reader (`resolve_vid_only`, `read_aacs_inputs`,
/// `read_mkb_content`, `read_aacs_version`) walks the exact same files — adding
/// or changing a fallback in one place can then never silently diverge the
/// disc_hash / MKB / VID that another reader feeds a key service.
pub const PATH_UNIT_KEY_RO: &str = "/AACS/Unit_Key_RO.inf";
pub const PATH_UNIT_KEY_RO_DUPLICATE: &str = "/AACS/DUPLICATE/Unit_Key_RO.inf";
pub const PATH_MKB_RO: &str = "/AACS/MKB_RO.inf";
pub const PATH_MKB_RW: &str = "/AACS/MKB_RW.inf";
pub const PATH_CONTENT_CERT: &str = "/AACS/Content000.cer";
pub const PATH_CONTENT_CERT_ALT: &str = "/AACS/Content001.cer";
// Boil-down derivation primitives (thin newtypes + wrappers over the crypto).
pub use boil::{
KeyCandidate, MediaKey, ProcessingKey, ResolvedChain, UnitKey, Vid, Vuk, mk_from_dk,
mk_from_pk, resolve_candidate, uk_from_vuk, vuk_from_mk,
};
// Structured, English-free resolution trace.
pub use trace::{KeyNode, KeyOutcome, KeyStep, ResolutionTrace, UnlockOutcome, UnlockStep};
// Explicit re-exports — only items needed by external consumers and sibling crate modules.
// AES primitives (aes_ecb_encrypt, aes_ecb_decrypt, aes_cbc_decrypt) are pub(crate) in decrypt.rs.
pub use decrypt::{
ALIGNED_UNIT_LEN, decrypt_bus, decrypt_unit, decrypt_unit_full, decrypt_unit_try_keys,
is_unit_encrypted,
ALIGNED_UNIT_LEN, ALIGNED_UNIT_SECTORS, UnitKeyResult, aacs_unit_encrypted,
aacs_unit_needs_decrypt, aacs_unit_still_ciphertext, decrypt_bus, decrypt_unit,
decrypt_unit_checked, decrypt_unit_full, decrypt_unit_try_keys, fill_null_ts_unit,
is_unit_aligned, ts_packet_total, ts_sync_count, ts_sync_destroyed, unit_is_clean_ps,
unit_is_clean_ts, unit_key_validates,
};
pub use keydb::{DeviceKey, DiscEntry, HostCert, KeyDb};
// `probe` is a reproduction-harness helper (see keys.rs), not part of the
// documented 1.0 surface; keep it reachable but off the rendered docs so we
// don't commit semver stability to test primitives.
#[doc(hidden)]
pub use keys::probe;
pub use keys::{
AacsVersion, ContentCert, ResolveContext, ResolvedKeys, UnitKeyFile, decrypt_unit_key,
AACS_MAJOR_BD, AACS_MAJOR_UHD, AacsVersion, ContentCert, MKB_20_CATEGORY_C, MKB_21_CATEGORY_C,
MKB_TYPE_3_RECORDABLE, MKB_TYPE_4_PRERECORDED, MKB_TYPE_10_CLASS_II, MkbType, ResolveContext,
ResolveFailure, ResolvedKeys, UnitKeyFile, decrypt_unit_key, derive_media_key_and_pk_from_dk,
derive_media_key_from_dk, derive_media_key_from_pk, derive_vuk, disc_hash, disc_hash_hex,
mkb_version, parse_content_cert, parse_unit_key_ro, read_mkb_from_drive, resolve_keys_v1,
resolve_keys_v2, resolve_keys_v21, validate_media_key_against_mkb,
mkb_content_len, mkb_is_uhd, mkb_type, mkb_type_raw, mkb_version, parse_content_cert,
parse_unit_key_ro, read_mkb_from_drive, recover_dk_position, resolve_keys_v1, resolve_keys_v2,
resolve_keys_v21, resolve_keys_with_reason, trim_mkb,
};
pub use provider::KeyProvider;
pub use types::{DeviceKey, DiscEntry, HostCert};
pub use variants::{
KEY_CORRECTION_DATA_PLACEHOLDER, MediaKeyVariantError, MkbRecord, ProcessingKeyMatch,
derive_media_key_variant, is_variant_mkb, variant_data_record, variant_key_data, variant_nonce,
walk_mkb, walk_processing_key,
derive_media_key_variant, is_variant_mkb, variant_nonce, walk_mkb, walk_processing_key,
};
#[cfg(test)]
mod tests {
//! Re-export surface guards. The module's public API is the set of
//! `pub use` items above. A regression that drops or renames an export
//! (the class of bug that shipped in 0.31.0 by silently changing a
//! surface) breaks compilation of these references, so they act as a
//! compile-time contract for the crate's AACS surface.
use super::*;
#[test]
fn aligned_unit_len_is_three_2048_byte_sectors() {
// ALIGNED_UNIT_LEN is the AACS aligned-unit size: 3 × 2048 = 6144.
// Re-exported from decrypt; pin the value here so the public constant
// and the spec stay in lockstep.
assert_eq!(ALIGNED_UNIT_LEN, 6144);
assert_eq!(ALIGNED_UNIT_LEN, 3 * 2048);
}
#[test]
fn version_strides_are_reexported_and_distinct() {
// The three AACS generations are part of the public surface, and the
// V10 (48) vs V20/V21 (64) stride distinction is the load-bearing
// difference. Confirm the enum re-export is usable and the variants
// are distinct values.
assert_ne!(AacsVersion::V10, AacsVersion::V20);
assert_ne!(AacsVersion::V20, AacsVersion::V21);
}
#[test]
fn key_correction_data_placeholder_is_all_zero() {
// The variant chain refuses to run against this all-zero placeholder
// KCD; the public constant must therefore be exactly 16 zero bytes.
assert_eq!(KEY_CORRECTION_DATA_PLACEHOLDER, [0u8; 16]);
}
#[test]
fn public_helpers_are_callable_through_the_facade() {
// Touch a representative function from each re-export group so a
// dropped/renamed export fails to compile. These are smoke calls, not
// behavioural assertions (behaviour is covered in each module).
let _ = ts_sync_destroyed(&[0u8; ALIGNED_UNIT_LEN]);
let _ = mkb_content_len(&[]);
let _ = is_variant_mkb(&walk_mkb(&[]));
let _ = disc_hash_hex(&disc_hash(b"x"));
let _ = mk_from_pk(&[[0u8; 16]], &[]);
}
}
+409
View File
@@ -0,0 +1,409 @@
//! Key source abstraction for the AACS resolve chain.
//!
//! libfreemkv keeps all crypto (AES-G primitives, SD-tree walking,
//! validation, MK/VUK/TK derivation) but accepts key material from
//! arbitrary backends via [`KeyProvider`].
//!
//! Methods come in two flavors:
//!
//! - **Bulk material** ([`device_keys`], [`processing_keys`],
//! [`media_keys`]) — the resolver unions (and dedups) results
//! across all providers and tries each candidate.
//! - **Disc-keyed lookup** ([`lookup_disc_by_hash`],
//! [`lookup_disc_by_vid`]) — the resolver short-circuits on the
//! first hit, so providers are queried in array order with
//! fastest/closest first.
//!
//! [`host_certs`] is a sixth method but is NOT consumed by the
//! resolver chain: the SCSI handshake reads host certs directly from
//! the caller-supplied credentials, not from the provider array. A
//! provider that overrides `host_certs` today has no effect on the
//! handshake; the method is retained as a forward-looking extension
//! point only.
//!
//! Default impls return empty / `None` so backends only override
//! the methods they actually support — an external key service might
//! implement only `lookup_disc_by_hash`, while a local file might
//! implement all six.
//!
//! Calls may block (disk I/O, network round-trips). The resolver
//! invokes each method at most a handful of times per scan; for
//! per-disc memoization, implementations should cache internally.
//!
//! [`device_keys`]: KeyProvider::device_keys
//! [`processing_keys`]: KeyProvider::processing_keys
//! [`media_keys`]: KeyProvider::media_keys
//! [`host_certs`]: KeyProvider::host_certs
//! [`lookup_disc_by_hash`]: KeyProvider::lookup_disc_by_hash
//! [`lookup_disc_by_vid`]: KeyProvider::lookup_disc_by_vid
use super::types::{DeviceKey, DiscEntry, HostCert};
/// Source of AACS key material.
///
/// Implementors return raw material only — the resolver in
/// `aacs::keys` owns all the crypto (DK→PK walking, PK validation,
/// MK→VUK→TK derivation). See module docs for method semantics.
pub trait KeyProvider: Send + Sync {
/// Device keys (top-of-tree, walked by the resolver).
fn device_keys(&self) -> Vec<DeviceKey> {
Vec::new()
}
/// Processing keys — terminal PKs or walk-input PKs. The
/// resolver tries each as a terminal first (cheap validate).
fn processing_keys(&self) -> Vec<[u8; 16]> {
Vec::new()
}
/// Every Media Key this provider holds, regardless of which disc it was
/// filed under. An MK is MKB-scoped (shared across a pressing/MKB-family),
/// so the resolver can verify each against the disc's MKB (`km_verifies`)
/// and resolve a disc whose own hash/VID isn't directly keyed.
fn media_keys(&self) -> Vec<[u8; 16]> {
Vec::new()
}
/// AACS host certificates (with their private keys) for drive
/// authentication. Multiple in case some are revoked.
///
/// NOTE: not consumed by the resolver chain — the handshake reads
/// host certs from the caller-supplied credentials directly, so
/// overriding this method has no effect on drive authentication
/// today. Retained as a forward-looking extension point.
fn host_certs(&self) -> Vec<HostCert> {
Vec::new()
}
/// Direct per-disc lookup by SHA-1 of `Unit_Key_RO.inf`. Returns
/// `Some(entry)` if this provider has pre-computed material for
/// the disc (paths 4 and 5). Short-circuits the resolver.
fn lookup_disc_by_hash(&self, _disc_hash: &[u8; 20]) -> Option<DiscEntry> {
None
}
/// Lookup by Volume ID (path 3 — pre-computed MK + matching
/// VID). Short-circuits the resolver on hit.
fn lookup_disc_by_vid(&self, _volume_id: &[u8; 16]) -> Option<DiscEntry> {
None
}
}
/// Resolver-side helpers that aggregate across a provider array.
///
/// The resolver wraps `ctx.providers` (`&[&dyn KeyProvider]`) in this
/// struct; these helpers apply the union-vs-short-circuit policy per
/// method. The bulk unions dedup so overlapping providers don't make
/// the resolver re-walk/re-validate identical material.
pub(crate) struct Providers<'a>(pub &'a [&'a dyn KeyProvider]);
impl Providers<'_> {
/// Union (deduped) — gather DKs from every provider.
pub fn device_keys(&self) -> Vec<DeviceKey> {
let mut v: Vec<DeviceKey> = self.0.iter().flat_map(|p| p.device_keys()).collect();
// DeviceKey has no Ord/Hash; dedup on the value-defining tuple.
v.sort_unstable_by_key(|d| (d.key, d.node, d.uv, d.u_mask_shift));
v.dedup_by_key(|d| (d.key, d.node, d.uv, d.u_mask_shift));
v
}
/// Union (deduped) — gather PKs from every provider.
pub fn processing_keys(&self) -> Vec<[u8; 16]> {
let mut v: Vec<[u8; 16]> = self.0.iter().flat_map(|p| p.processing_keys()).collect();
v.sort_unstable();
v.dedup();
v
}
/// Union of distinct Media Keys across every provider, for the MK-pool
/// brute (`km_verifies` against the disc's MKB).
pub fn media_keys(&self) -> Vec<[u8; 16]> {
let mut v: Vec<[u8; 16]> = self.0.iter().flat_map(|p| p.media_keys()).collect();
v.sort_unstable();
v.dedup();
v
}
/// Union — gather host certs from every provider. The SCSI handshake
/// reads host certs from the caller-supplied credentials directly and
/// does not call this, so it is currently unused by the resolver chain.
#[allow(dead_code)]
pub fn host_certs(&self) -> Vec<HostCert> {
self.0.iter().flat_map(|p| p.host_certs()).collect()
}
/// Short-circuit — query providers in array order, first hit wins.
pub fn lookup_disc_by_hash(&self, disc_hash: &[u8; 20]) -> Option<DiscEntry> {
self.0.iter().find_map(|p| p.lookup_disc_by_hash(disc_hash))
}
/// Short-circuit — query providers in array order, first hit wins.
pub fn lookup_disc_by_vid(&self, volume_id: &[u8; 16]) -> Option<DiscEntry> {
self.0.iter().find_map(|p| p.lookup_disc_by_vid(volume_id))
}
}
/// A [`KeyProvider`] backed by a single caller-supplied key's raw material —
/// the bridge for [`crate::disc::Disc::decrypt_with`].
///
/// The application's key source did the lookup and handed in material at one
/// level (DK / PK / MK / VUK). This exposes exactly that material to the
/// version-dispatched resolver, which owns ALL derivation — so a source never
/// derives, and the lib remains the single home for the AACS chain across
/// 1.0 / 2.0 / 2.1 / 2.x.
///
/// Each level fills only its own field; the rest stay empty, so the resolver
/// naturally runs the matching path (DK→…, PK→…, MK-pool brute, or a
/// disc-keyed VUK hit). `decrypt_with` already knows the disc, so the
/// `lookup_disc_by_*` hash/VID arguments are irrelevant — a present
/// `disc_entry` is returned for any query.
pub(crate) struct SuppliedKey {
pub device_keys: Vec<DeviceKey>,
pub processing_keys: Vec<[u8; 16]>,
pub media_keys: Vec<[u8; 16]>,
pub disc_entry: Option<DiscEntry>,
}
impl KeyProvider for SuppliedKey {
fn device_keys(&self) -> Vec<DeviceKey> {
self.device_keys.clone()
}
fn processing_keys(&self) -> Vec<[u8; 16]> {
self.processing_keys.clone()
}
fn media_keys(&self) -> Vec<[u8; 16]> {
self.media_keys.clone()
}
fn lookup_disc_by_hash(&self, _disc_hash: &[u8; 20]) -> Option<DiscEntry> {
self.disc_entry.clone()
}
fn lookup_disc_by_vid(&self, _volume_id: &[u8; 16]) -> Option<DiscEntry> {
self.disc_entry.clone()
}
}
#[cfg(test)]
mod tests {
use super::*;
fn entry(hash: &str, vuk: u8) -> DiscEntry {
DiscEntry {
disc_hash: hash.to_string(),
title: "t".to_string(),
media_key: None,
disc_id: None,
vuk: Some([vuk; 16]),
unit_keys: Vec::new(),
}
}
fn dk(byte: u8, node: u16) -> DeviceKey {
DeviceKey {
key: [byte; 16],
node,
uv: 1,
u_mask_shift: 0,
}
}
/// A provider that returns fixed bulk material and an optional disc entry
/// keyed unconditionally (used to test array-order short-circuiting).
#[derive(Default)]
struct Fixed {
dks: Vec<DeviceKey>,
pks: Vec<[u8; 16]>,
mks: Vec<[u8; 16]>,
hash_hit: Option<DiscEntry>,
vid_hit: Option<DiscEntry>,
}
impl KeyProvider for Fixed {
fn device_keys(&self) -> Vec<DeviceKey> {
self.dks.clone()
}
fn processing_keys(&self) -> Vec<[u8; 16]> {
self.pks.clone()
}
fn media_keys(&self) -> Vec<[u8; 16]> {
self.mks.clone()
}
fn lookup_disc_by_hash(&self, _h: &[u8; 20]) -> Option<DiscEntry> {
self.hash_hit.clone()
}
fn lookup_disc_by_vid(&self, _v: &[u8; 16]) -> Option<DiscEntry> {
self.vid_hit.clone()
}
}
// ── KeyProvider default methods all return empty ───────────────────────
#[test]
fn default_provider_methods_return_empty() {
// A bare provider that overrides nothing must yield empty material so
// the resolver simply finds nothing through it (no surprise hits).
struct Empty;
impl KeyProvider for Empty {}
let e = Empty;
assert!(e.device_keys().is_empty());
assert!(e.processing_keys().is_empty());
assert!(e.media_keys().is_empty());
assert!(e.host_certs().is_empty());
assert!(e.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(e.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
// ── Providers::processing_keys: union + dedup ──────────────────────────
#[test]
fn providers_processing_keys_union_and_dedup() {
// Two providers each carrying overlapping PKs → the aggregate is the
// deduped union (the resolver must not re-validate identical material).
let a = Fixed {
pks: vec![[0x01u8; 16], [0x02u8; 16]],
..Default::default()
};
let b = Fixed {
pks: vec![[0x02u8; 16], [0x03u8; 16]],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let mut got = Providers(arr).processing_keys();
got.sort();
assert_eq!(got, vec![[0x01u8; 16], [0x02u8; 16], [0x03u8; 16]]);
}
#[test]
fn providers_media_keys_union_and_dedup() {
let a = Fixed {
mks: vec![[0xAAu8; 16]],
..Default::default()
};
let b = Fixed {
mks: vec![[0xAAu8; 16], [0xBBu8; 16]],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let mut got = Providers(arr).media_keys();
got.sort();
assert_eq!(got, vec![[0xAAu8; 16], [0xBBu8; 16]]);
}
#[test]
fn providers_device_keys_dedup_on_value_tuple() {
// DeviceKey has no Hash/Ord; dedup keys on (key,node,uv,u_mask_shift).
// Two identical DKs across providers collapse to one; a DK differing
// only in node is kept.
let a = Fixed {
dks: vec![dk(0x11, 5), dk(0x11, 5)],
..Default::default()
};
let b = Fixed {
dks: vec![dk(0x11, 5), dk(0x11, 6)],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).device_keys();
assert_eq!(got.len(), 2, "identical DKs dedup; differing node kept");
let nodes: Vec<u16> = got.iter().map(|d| d.node).collect();
assert!(nodes.contains(&5) && nodes.contains(&6));
}
// ── Disc-keyed lookups: array-order short-circuit ──────────────────────
#[test]
fn providers_lookup_by_hash_first_hit_wins() {
// Querying providers in array order, the FIRST hit wins (closest /
// fastest first). Provider 0 hits → its entry is returned even though
// provider 1 also has one.
let a = Fixed {
hash_hit: Some(entry("first", 0x01)),
..Default::default()
};
let b = Fixed {
hash_hit: Some(entry("second", 0x02)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_hash(&[0u8; 20]).unwrap();
assert_eq!(got.disc_hash, "first");
assert_eq!(got.vuk, Some([0x01u8; 16]));
}
#[test]
fn providers_lookup_by_hash_falls_through_to_later_provider() {
// Provider 0 misses, provider 1 hits → the later provider's entry is
// used (find_map continues past None).
let a = Fixed::default(); // hash_hit None
let b = Fixed {
hash_hit: Some(entry("second", 0x02)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_hash(&[0u8; 20]).unwrap();
assert_eq!(got.disc_hash, "second");
}
#[test]
fn providers_lookup_by_vid_first_hit_wins() {
let a = Fixed {
vid_hit: Some(entry("vid-a", 0x07)),
..Default::default()
};
let b = Fixed {
vid_hit: Some(entry("vid-b", 0x08)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_vid(&[0u8; 16]).unwrap();
assert_eq!(got.disc_hash, "vid-a");
}
#[test]
fn providers_empty_array_yields_nothing() {
let arr: &[&dyn KeyProvider] = &[];
let p = Providers(arr);
assert!(p.device_keys().is_empty());
assert!(p.processing_keys().is_empty());
assert!(p.media_keys().is_empty());
assert!(p.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(p.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
// ── SuppliedKey: each level exposes only its own material ──────────────
#[test]
fn supplied_key_exposes_only_populated_fields() {
// A SuppliedKey filled at the DK level exposes DKs and nothing else,
// so the resolver runs the matching (DK→…) path and no other.
let sk = SuppliedKey {
device_keys: vec![dk(0x33, 9)],
processing_keys: Vec::new(),
media_keys: Vec::new(),
disc_entry: None,
};
assert_eq!(sk.device_keys().len(), 1);
assert!(sk.processing_keys().is_empty());
assert!(sk.media_keys().is_empty());
assert!(sk.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(sk.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
#[test]
fn supplied_key_disc_entry_returned_for_any_hash_or_vid() {
// decrypt_with already knows the disc, so a present disc_entry is
// returned regardless of the hash/VID argument (the lookup args are
// irrelevant in this bridge).
let sk = SuppliedKey {
device_keys: Vec::new(),
processing_keys: Vec::new(),
media_keys: Vec::new(),
disc_entry: Some(entry("supplied", 0x44)),
};
// Two unrelated hashes both return the same entry.
let h1 = sk.lookup_disc_by_hash(&[0x01u8; 20]).unwrap();
let h2 = sk.lookup_disc_by_hash(&[0xFFu8; 20]).unwrap();
assert_eq!(h1.disc_hash, "supplied");
assert_eq!(h2.disc_hash, "supplied");
// And by VID likewise.
assert!(sk.lookup_disc_by_vid(&[0x00u8; 16]).is_some());
}
}
+144
View File
@@ -0,0 +1,144 @@
//! Structured resolution trace — what the unlock + key-resolution attempt did.
//!
//! No user-facing English. Every step's STATE is a typed enum variant;
//! applications RENDER these into localized text (the library never does). This
//! module only DEFINES the shape and is wired through the resolve/handshake
//! return path far enough to compile.
//!
//! The `who` of each step is the source's `label()` / unlocker's `name()` — a
//! stable identifier string (a NAME, like a codec id, NOT user-facing prose),
//! carried verbatim so an app renderer never has to match an enum back to a name
//! it already has. Only the OUTCOME / path enums are structured states the app
//! maps to i18n English.
/// The full trace of a resolution attempt: the unlock phase, then the
/// key-resolution phase.
#[derive(Debug, Clone, PartialEq, Default)]
pub struct ResolutionTrace {
/// One step per unlocker consulted, in consultation order.
pub unlock: Vec<UnlockStep>,
/// One step per key source consulted, in consultation order.
pub keys: Vec<KeyStep>,
}
impl ResolutionTrace {
/// An empty trace (no steps recorded).
pub fn new() -> Self {
Self::default()
}
}
// ── Unlock phase ────────────────────────────────────────────────────────────
/// One unlocker's contribution to the unlock phase. `who` is the unlocker's
/// `name()` (a stable, product-neutral identifier), carried verbatim.
#[derive(Debug, Clone, PartialEq)]
pub struct UnlockStep {
pub who: String,
pub outcome: UnlockOutcome,
}
/// What an unlocker did.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum UnlockOutcome {
/// The drive was unlocked (or already usable) and a VID is available.
Unlocked,
/// This unlocker cannot unlock this drive's firmware.
FirmwareNotUnlockable,
/// No non-revoked host cert was usable for the auth attempt. `mkb` is the
/// disc MKB generation when known.
NoUsableHostCert { mkb: Option<u32> },
/// Every available host cert was revoked on this drive's HRL. `mkb` is the
/// disc MKB generation when known.
CertRevoked { mkb: Option<u32> },
/// The drive rejected the auth handshake (non-revocation rejection / wedge).
HandshakeRejected,
/// Auth succeeded (or was skipped) but the Volume ID could not be read.
VidUnavailable,
}
// ── Key-resolution phase ────────────────────────────────────────────────────
/// One key source's contribution to the key-resolution phase, including the
/// derivation path it walked. `who` is the source's `label()` (a stable
/// identifier, e.g. `"keydb"` / `"online"`), carried verbatim.
#[derive(Debug, Clone, PartialEq)]
pub struct KeyStep {
pub who: String,
pub path: Vec<KeyNode>,
pub outcome: KeyOutcome,
}
/// A node on the derivation path a source walked. Ordered as encountered; not
/// every path hits every node.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum KeyNode {
/// The source matched this disc (by hash / VID).
MatchedDisc,
/// The source had no entry for this disc.
NoEntry,
/// Pre-decrypted unit keys were found.
FoundUnitKeys,
/// A VUK was found.
FoundVuk,
/// A Media Key was found.
FoundMediaKey,
/// A VID is required to proceed.
NeedVid,
/// The VID came from the unlock phase.
VidFromUnlock,
/// The VID came from the keydb entry.
VidFromKeydb,
/// No VID was available.
NoVid,
/// A VUK was derived (from MK + VID).
DerivedVuk,
/// Unit keys were derived (from VUK).
DerivedUnitKeys,
}
/// The terminal outcome of a source's resolution attempt.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum KeyOutcome {
/// Usable unit keys were produced.
Resolved,
/// Derivation material existed but no VID was available to finish.
MissingVid,
/// No usable key from this source.
NoKey,
}
#[cfg(test)]
mod tests {
use super::*;
/// The trace types are constructible, derive the required traits, and an
/// empty trace round-trips. Pins the structural contract apps build against.
#[test]
fn trace_is_constructible_and_comparable() {
let t = ResolutionTrace {
unlock: vec![UnlockStep {
who: "AACS cert".to_string(),
outcome: UnlockOutcome::NoUsableHostCert { mkb: Some(68) },
}],
keys: vec![KeyStep {
who: "keydb".to_string(),
path: vec![
KeyNode::MatchedDisc,
KeyNode::FoundVuk,
KeyNode::DerivedUnitKeys,
],
outcome: KeyOutcome::Resolved,
}],
};
// Clone + PartialEq (derive contract the renderers rely on).
assert_eq!(t.clone(), t);
// `who` is the source's name carried verbatim.
assert_eq!(t.keys[0].who, "keydb");
assert_eq!(t.unlock[0].who, "AACS cert");
// Default / new is empty.
assert_eq!(ResolutionTrace::new(), ResolutionTrace::default());
assert!(ResolutionTrace::new().unlock.is_empty());
assert!(ResolutionTrace::new().keys.is_empty());
}
}
+45
View File
@@ -0,0 +1,45 @@
//! AACS primitive types shared across the resolve chain.
//!
//! These structs describe AACS key material (device keys, host
//! certificates, per-disc entries). They carry no parsing logic — the
//! keydb.cfg format lives in the `freemkv-keysources` crate. libfreemkv
//! owns only the crypto and these value types that flow through it.
/// A device key for MKB subset-difference tree processing.
#[derive(Debug, Clone)]
pub struct DeviceKey {
pub key: [u8; 16],
pub node: u16,
pub uv: u32,
pub u_mask_shift: u8,
}
/// Host certificate + private key for AACS SCSI authentication.
#[derive(Debug, Clone)]
pub struct HostCert {
/// AACS 1.0: 20 bytes. AACS 2.0: 32 bytes.
pub private_key: [u8; 20],
/// AACS 1.0: 92 bytes. AACS 2.0: 132 bytes.
pub certificate: Vec<u8>,
/// AACS 2.0 host private key (P-256, 32 bytes). None for AACS 1.0 only.
pub private_key_v2: Option<[u8; 32]>,
/// AACS 2.0 host certificate (type 0x11). None for AACS 1.0 only.
pub certificate_v2: Option<Vec<u8>>,
}
/// A per-disc entry from the key database.
#[derive(Debug, Clone)]
pub struct DiscEntry {
/// Disc hash (20 bytes, hex)
pub disc_hash: String,
/// Disc title
pub title: String,
/// Media Key (16 bytes) — from MKB processing
pub media_key: Option<[u8; 16]>,
/// Disc ID (16 bytes)
pub disc_id: Option<[u8; 16]>,
/// Volume Unique Key (16 bytes) — decrypts title keys
pub vuk: Option<[u8; 16]>,
/// Unit keys (title keys) indexed by CPS unit number
pub unit_keys: Vec<(u32, [u8; 16])>,
}
+442 -76
View File
@@ -14,6 +14,19 @@
//! a disc carries neither, callers should fall back to the classical
//! single-stage derivation in [`super::keys`].
//!
//! **Status: the chain cannot yet produce a key on a real disc.** Two
//! sub-fields are unfinished:
//! - [`variants_for_uv`] (the `VARIANTS[uv]` lookup in the `0x83`
//! record) is a stub that always returns `None`, so the chain
//! short-circuits with [`MediaKeyVariantError::VariantsTableUnavailable`].
//! - The Encrypted Media Key Variant Data (C) and the Variant Key
//! Data (VKD) table are *distinct* sub-fields of the `0x82` record
//! per AACS 2.1, but [`variant_data_record`] (C) and
//! [`variant_key_data`] (VKD) both currently return the *whole*
//! first `0x82` body — so on a single-`0x82` disc they alias. The
//! `0x82` sub-field offsets must be fixed against a real Variant
//! disc before this chain is wired into `resolve_keys`.
//!
//! The chain follows the published spec:
//!
//! ```text
@@ -28,9 +41,22 @@
//! Two condition bits on `Kmp[15]` route off the hardcoded-KCD path
//! (Soft Correction and Online Challenge). The chain refuses to run in
//! either case — callers must handle those modes out of band.
//!
//! # Status: Kp verification
//!
//! On the classical path [`walk_processing_key`] gates each match on
//! the VERIFY_MAGIC relation, which authenticates the Processing Key.
//! On a variant MKB that magic check does NOT hold (the walk yields a
//! Media Key *Precursor*, not the Media Key), so the walk accepts a
//! variant match without it. The replacement gate lives at the END of
//! [`derive_media_key_variant`]: the derived final `Km` is verified
//! against the MKB's Verify-Media-Key record before any `(Km, Kvu)` is
//! returned. A future implementer wiring [`variants_for_uv`] must keep
//! that final gate — the per-match magic check no longer protects the
//! variant path.
use super::decrypt::aes_ecb_decrypt;
use super::keydb::DeviceKey;
use super::types::DeviceKey;
// ── Public constants ──────────────────────────────────────────────────────
@@ -62,29 +88,44 @@ pub struct MkbRecord {
/// INCLUDING the 4-byte header, followed by payload. The walker stops
/// at the first `(type=0, len=0)` end marker or at end of buffer.
pub fn walk_mkb(mkb: &[u8]) -> Vec<MkbRecord> {
let mut out = Vec::new();
let mut pos = 0;
while pos + 4 <= mkb.len() {
mkb_records(mkb)
.map(|(offset, rec_type, rec_len)| MkbRecord {
offset,
rec_type,
rec_len,
body: mkb[offset + 4..offset + rec_len].to_vec(),
})
.collect()
}
/// THE single MKB record-framing walker: yields `(offset, rec_type, rec_len)`
/// for each record — a 4-byte header (type byte + big-endian 24-bit length)
/// then the body — stopping at the `00 000000` end marker or a
/// malformed/out-of-bounds length. Lazy (no body clone), so a find-one-record
/// caller never materialises the multi-MB cvalue table. [`walk_mkb`] and every
/// MKB record walk in `aacs::keys` are built on this, so the framing rules — and
/// any future fix to them — live in exactly one place (they had drifted across
/// six hand-rolled copies).
pub(crate) fn mkb_records(mkb: &[u8]) -> impl Iterator<Item = (usize, u8, usize)> + '_ {
let mut pos = 0usize;
std::iter::from_fn(move || {
if pos + 4 > mkb.len() {
return None;
}
let rec_type = mkb[pos];
let rec_len = ((mkb[pos + 1] as usize) << 16)
| ((mkb[pos + 2] as usize) << 8)
| (mkb[pos + 3] as usize);
if rec_type == 0 && rec_len == 0 {
break;
return None;
}
if rec_len < 4 || pos + rec_len > mkb.len() {
break;
return None;
}
let body = mkb[pos + 4..pos + rec_len].to_vec();
out.push(MkbRecord {
offset: pos,
rec_type,
rec_len,
body,
});
let here = pos;
pos += rec_len;
}
out
Some((here, rec_type, rec_len))
})
}
/// True iff `records` contains at least one Media Key Variant record
@@ -94,7 +135,13 @@ pub fn is_variant_mkb(records: &[MkbRecord]) -> bool {
}
/// Body of the Encrypted Media Key Variant Data record (type `0x82`).
pub fn variant_data_record(records: &[MkbRecord]) -> Option<&[u8]> {
///
/// Returns the whole first `0x82` body; the internal C / VKD sub-field
/// split is not yet decoded, so this aliases [`variant_key_data`] on a
/// single-`0x82` disc. `pub(crate)` until the sub-field offsets are fixed
/// against a real variant disc — it is not part of the public surface
/// because it knowingly returns an undecoded composite.
pub(crate) fn variant_data_record(records: &[MkbRecord]) -> Option<&[u8]> {
records
.iter()
.find(|r| r.rec_type == 0x82)
@@ -115,7 +162,11 @@ pub fn variant_nonce(records: &[MkbRecord]) -> Option<[u8; 16]> {
/// Body of the Variant Key Data record. Returns the first `0x82` body
/// that is a non-empty multiple of 16 bytes.
pub fn variant_key_data(records: &[MkbRecord]) -> Option<&[u8]> {
///
/// Like [`variant_data_record`], this returns the whole `0x82` body and
/// aliases it on a single-`0x82` disc; the C / VKD sub-field split is
/// undecoded. `pub(crate)` until fixed against a real variant disc.
pub(crate) fn variant_key_data(records: &[MkbRecord]) -> Option<&[u8]> {
records
.iter()
.find(|r| r.rec_type == 0x82 && !r.body.is_empty() && r.body.len() % 16 == 0)
@@ -141,56 +192,11 @@ fn aes_g(x1: &[u8; 16], x2: &[u8; 16]) -> [u8; 16] {
// ── Subset-difference walk that exposes (Kp, uv) ──────────────────────────
/// AES-G3 seed register initial value.
const AESG3_SEED: [u8; 16] = [
0x7B, 0x10, 0x3C, 0x5D, 0xCB, 0x08, 0xC4, 0xE5, 0x1A, 0x27, 0xB0, 0x17, 0x99, 0x05, 0x3B, 0xD9,
];
/// AES-G3 single step: AES-G against the seed register at offset `inc`.
fn aesg3_step(key: &[u8; 16], inc: u8) -> [u8; 16] {
let mut seed = AESG3_SEED;
seed[15] = seed[15].wrapping_add(inc);
aes_g(key, &seed)
}
fn calc_v_mask(uv: u32) -> u32 {
let mut v_mask: u32 = 0xFFFF_FFFF;
while (uv & !v_mask) == 0 && v_mask != 0 {
v_mask <<= 1;
}
v_mask
}
fn calc_pk_from_dk(dk: &[u8; 16], uv: u32, v_mask: u32, dev_key_v_mask: u32) -> [u8; 16] {
let mut left_child = aesg3_step(dk, 0);
let mut pk = aesg3_step(dk, 1);
let mut right_child = aesg3_step(dk, 2);
let mut current_v_mask = dev_key_v_mask;
while current_v_mask != v_mask {
let mut bit_pos: i32 = -1;
for i in (0..32).rev() {
if (current_v_mask & (1u32 << i)) == 0 {
bit_pos = i;
break;
}
}
let curr_key = if bit_pos < 0 || (uv & (1u32 << bit_pos as u32)) == 0 {
left_child
} else {
right_child
};
left_child = aesg3_step(&curr_key, 0);
pk = aesg3_step(&curr_key, 1);
right_child = aesg3_step(&curr_key, 2);
current_v_mask = ((current_v_mask as i32) >> 1) as u32;
}
pk
}
// `calc_v_mask` and `calc_pk_from_dk` (and the AES-G3 seed step they ride
// on) are shared with the classical walk in [`super::keys`] — a single
// definition keeps the variant SD tree byte-identical to the classical one.
// (`aesg3` itself is imported separately in the test module.)
use super::keys::{calc_pk_from_dk, calc_v_mask};
/// Outcome of a subset-difference walk against an MKB. Carries the
/// processing key and the matching `uv` slot — both needed as inputs
@@ -225,6 +231,27 @@ fn mkb_find_mk_dv(records: &[MkbRecord]) -> Option<[u8; 16]> {
/// Walk an MKB and return the first `(Kp, uv, cvalue)` that
/// `device_keys` covers. Returns `None` if no DK walks any uv.
///
/// This is the AACS-2.1 **variant** walk; the classical walk lives in
/// [`super::keys::derive_media_key_and_pk_from_dk`]. The two are kept
/// separate on purpose and select MKB records in DELIBERATELY different
/// order:
///
/// - cvalues: this variant walk tries record `0x07`-then-`0x05`; the
/// classical walk tries `0x05`-then-`0x07`. On a variant MKB the
/// small `0x07` Explicit-Subset-Difference record carries the
/// cvalue the Precursor chain consumes, whereas a classical UHD MKB
/// keeps its 1:1 cvalue table in the large `0x05` record (see the
/// note on [`super::keys::probe::mkb_cvalues`]). They must NOT be
/// unified to one order — each is correct for its own MKB shape.
/// - finders: this walk operates on parsed [`MkbRecord`]s (needed
/// because the variant chain also reads `0x82`/`0x83`); the
/// classical walk operates on raw MKB bytes. Same framing, different
/// input type.
///
/// Consequence: do NOT route the classical DK path through this function
/// — on a classical MKB the `0x07`-first selection picks the wrong (or
/// missing) cvalue and the magic check fails, so it returns `None`.
pub fn walk_processing_key(
records: &[MkbRecord],
device_keys: &[DeviceKey],
@@ -243,11 +270,23 @@ pub fn walk_processing_key(
for uvs_idx in 0..num_uvs {
let p_uv = &uvs[1 + 5 * uvs_idx..];
// `num_uvs` was computed by `take_while(.. (c[0] & 0xC0) == 0)`, so
// every chunk in `0..num_uvs` already has its revoked-marker bits
// clear — that `take_while` is the single authoritative place the
// parse stops, no inner re-check needed.
let u_mask_shift = uvs[5 * uvs_idx];
if u_mask_shift & 0xC0 != 0 {
break;
}
// 0x20..=0x3F (32..=63) pass the 0xC0 revoked-marker check but are
// out of range for a u32 shift. `wrapping_shl` would silently
// compute shift % 32 (e.g. 32 → no shift → 0xFFFF_FFFF), matching a
// wrong uv slot and deriving a wrong key. Disc-controlled byte:
// skip the slot instead.
if u_mask_shift >= 32 {
continue;
}
let uv = u32::from_be_bytes([p_uv[0], p_uv[1], p_uv[2], p_uv[3]]);
if uv == 0 {
@@ -260,6 +299,11 @@ pub fn walk_processing_key(
if ((device_number & u_mask) == (uv & u_mask))
&& ((device_number & v_mask) != (uv & v_mask))
{
// dk.u_mask_shift is a u8 from keydb with no range check; guard
// it the same way before the wrapping_shl below.
if dk.u_mask_shift >= 32 {
continue;
}
let dev_key_v_mask = calc_v_mask(dk.uv);
let dev_key_u_mask: u32 = 0xFFFF_FFFFu32.wrapping_shl(dk.u_mask_shift as u32);
@@ -334,6 +378,10 @@ pub enum MediaKeyVariantError {
VariantsTableUnavailable,
/// VKD index resolved out of the supplied `vkd_table`.
VkdIndexOutOfRange,
/// The derived Media Key failed the MKB's Verify-Media-Key relation.
/// On the variant path this final gate replaces the per-match magic
/// check (which does not hold for a Precursor).
MediaKeyVerifyFailed,
}
impl std::fmt::Display for MediaKeyVariantError {
@@ -347,6 +395,7 @@ impl std::fmt::Display for MediaKeyVariantError {
MediaKeyVariantError::KcdNotProvided => 7105,
MediaKeyVariantError::VariantsTableUnavailable => 7106,
MediaKeyVariantError::VkdIndexOutOfRange => 7107,
MediaKeyVariantError::MediaKeyVerifyFailed => 7108,
};
write!(f, "E{code}")
}
@@ -356,11 +405,15 @@ impl std::error::Error for MediaKeyVariantError {}
// ── Chain ─────────────────────────────────────────────────────────────────
/// Look up `VARIANTS[uv]` for the matched uv. The byte layout of the
/// per-uv slot in the Variant Number record is undocumented and is
/// disc-specific; this helper returns `None` until a Variant disc is
/// available to fix the layout against.
fn variants_for_uv(_records: &[MkbRecord], _uv_index: usize) -> Option<u16> {
/// Look up the per-slot `VARIANTS` value for the matched subset-difference
/// slot. AACS 2.1 keys the VARIANTS table by the matched SD slot (the same
/// index that selected the cvalue), so the caller passes
/// [`ProcessingKeyMatch::cvalue_index`]. The byte layout of the per-slot entry
/// in the Variant Number record is undocumented and disc-specific; this helper
/// returns `None` until a Variant disc is available to fix the layout against.
///
/// `sd_slot_index` is the matched subset-difference slot (== cvalue index).
fn variants_for_uv(_records: &[MkbRecord], _sd_slot_index: usize) -> Option<u16> {
None
}
@@ -377,6 +430,12 @@ fn variants_for_uv(_records: &[MkbRecord], _uv_index: usize) -> Option<u16> {
/// the final VUK alongside the Media Key.
///
/// Returns `(Km, Kvu)` on success.
///
/// NOTE: the `VARIANTS[uv]` lookup ([`variants_for_uv`]) is not yet
/// implemented, so on a real Variant disc this always returns
/// `Err(`[`MediaKeyVariantError::VariantsTableUnavailable`]`)` before a
/// key is produced. The chain can only succeed against synthetic test
/// fixtures today.
pub fn derive_media_key_variant(
mkb_records: &[MkbRecord],
device_keys: &[DeviceKey],
@@ -446,6 +505,17 @@ pub fn derive_media_key_variant(
km[12 + i] ^= uv_bytes[i];
}
// Gate: verify the derived Media Key against the MKB's Verify-Media-Key
// record. On the variant path the per-match magic check in
// `walk_processing_key` does NOT hold (it only saw the Precursor), so this
// is the authoritative Kp/Km verification — it MUST run before returning a
// real key.
let mk_dv = mkb_find_mk_dv(mkb_records).ok_or(MediaKeyVariantError::MkbIncomplete)?;
const VERIFY_MAGIC: [u8; 8] = [0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF];
if aes_ecb_decrypt(&km, &mk_dv)[..8] != VERIFY_MAGIC {
return Err(MediaKeyVariantError::MediaKeyVerifyFailed);
}
// Step: Kvu = AES-G(Km, VID).
let kvu = aes_g(&km, vid);
@@ -455,6 +525,23 @@ pub fn derive_media_key_variant(
#[cfg(test)]
mod tests {
use super::*;
// These three live in `super::keys` now (consolidated SD-walk helpers);
// `use super::*` does not re-export the parent module's private `use`
// imports, so pull them in directly for the tests below.
use super::super::keys::{aesg3, calc_pk_from_dk};
#[test]
fn calc_pk_from_dk_terminates_on_nonconvergent_mask() {
// Regression for the unbounded-loop hang: pick a (dev_key_v_mask,
// v_mask) pair the arithmetic `>> 1` walk can never reconcile.
// dev_key_v_mask has the MSB set, so `>> 1` sign-extends and the
// mask saturates at 0xFFFF_FFFF, never reaching a coarser v_mask.
// The 32-step bound must let this return rather than spin forever.
let dk = [0x11u8; 16];
let pk = calc_pk_from_dk(&dk, 0x0000_0002, 0x0000_0000, 0xFFFF_FFFE);
// Bounded exit yields *some* key; we only assert it terminated.
let _ = pk;
}
// ── Helpers ──
@@ -575,6 +662,7 @@ mod tests {
MediaKeyVariantError::KcdNotProvided,
MediaKeyVariantError::VariantsTableUnavailable,
MediaKeyVariantError::VkdIndexOutOfRange,
MediaKeyVariantError::MediaKeyVerifyFailed,
];
for e in cases {
let s = e.to_string();
@@ -605,7 +693,7 @@ mod tests {
/// agreeing with uv on bits 3+ (the u_mask=1 region). dk.uv ==
/// MKB.uv and dk.u_mask_shift == MKB.u_mask_shift make
/// `dev_key_v_mask == v_mask`, so `calc_pk_from_dk` loops zero
/// times — Kp = aesg3_step(dk, 1).
/// times — Kp = aesg3(dk, 1).
/// - one cvalue in record 0x07 chosen so AES-D(Kp, C) ⊕ uv produces a
/// Kmp whose byte-15 is exactly `kmp15`.
/// - record 0x82 with a 16-byte body (acts as both Variant Data
@@ -626,14 +714,14 @@ mod tests {
mkb.extend_from_slice(&[0x03, 0x00, 0x00, 0x00, 0x02]);
// Pick a known DK; with dk.uv == MKB.uv (==2) and
// dk.u_mask_shift == MKB.u_mask_shift (==1), dev_key_v_mask
// dk.u_mask_shift == MKB.u_mask_shift (==3), dev_key_v_mask
// equals the MKB's v_mask and the calc_pk_from_dk loop is a
// no-op — Kp = aesg3_step(dk, 1).
// no-op — Kp = aesg3(dk, 1).
let dk_bytes: [u8; 16] = [
0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE,
0xFF, 0x00,
];
let kp = aesg3_step(&dk_bytes, 1);
let kp = aesg3(&dk_bytes, 1);
// Plant Kmp with chosen byte-15, then compute C such that
// AES-D(Kp, C) ⊕ uv == Kmp. uv=2 → low-4 bytes XOR is 00 00 00 02.
@@ -676,4 +764,282 @@ mod tests {
};
(recs, dk, kp, kmp)
}
// ════════════════════════════════════════════════════════════════════
// Hardening additions
// ════════════════════════════════════════════════════════════════════
// ── walk_mkb framing: BE24 length incl. header, end markers ────────────
#[test]
fn walk_mkb_reports_offsets_and_be24_lengths() {
// Two records; the walker must report each record's byte offset and
// its full length (header + body). rec_len is the 3-byte BE field at
// bytes 1..4, and INCLUDES the 4-byte header.
let mut mkb = vec![0x10, 0x00, 0x00, 0x06, 0xAA, 0xBB]; // len 6 (2-byte body)
mkb.extend_from_slice(&[0x05, 0x00, 0x00, 0x08, 1, 2, 3, 4]); // len 8
let recs = walk_mkb(&mkb);
assert_eq!(recs.len(), 2);
assert_eq!(recs[0].offset, 0);
assert_eq!(recs[0].rec_len, 6);
assert_eq!(recs[0].body, vec![0xAA, 0xBB]);
assert_eq!(recs[1].offset, 6);
assert_eq!(recs[1].rec_len, 8);
assert_eq!(recs[1].body, vec![1, 2, 3, 4]);
}
#[test]
fn mkb_records_matches_walk_mkb_framing() {
// The lazy `mkb_records` iterator and the owning `walk_mkb` must agree on
// (offset, type, len) for every record — they share the one framing
// walker, and every keys.rs MKB walk now relies on this equivalence.
let mut mkb = vec![0x10, 0x00, 0x00, 0x06, 0xAA, 0xBB];
mkb.extend_from_slice(&[0x05, 0x00, 0x00, 0x08, 1, 2, 3, 4]);
mkb.extend_from_slice(&[0x00, 0x00, 0x00, 0x00, 0xFF]); // terminator + trailing
let owned: Vec<(usize, u8, usize)> = walk_mkb(&mkb)
.iter()
.map(|r| (r.offset, r.rec_type, r.rec_len))
.collect();
let lazy: Vec<(usize, u8, usize)> = mkb_records(&mkb).collect();
assert_eq!(lazy, owned);
assert_eq!(lazy, vec![(0, 0x10, 6), (6, 0x05, 8)]);
}
#[test]
fn walk_mkb_be24_high_byte_is_honored() {
// A record longer than 255 bytes needs the high BE24 byte. Build a
// 0x10 record of total length 0x000110 (272) and confirm the body is
// 268 bytes (a parser that read only the low byte would see len 0x10).
let total = 0x0110usize; // 272
let mut mkb = vec![0x10, 0x00, 0x01, 0x10];
mkb.resize(total, 0xAB);
let recs = walk_mkb(&mkb);
assert_eq!(recs.len(), 1);
assert_eq!(recs[0].rec_len, total);
assert_eq!(recs[0].body.len(), total - 4);
}
#[test]
fn walk_mkb_stops_at_type0_len0_end_marker() {
// A (type=0, len=0) record ends the walk; trailing bytes after it are
// not parsed.
let mut mkb = vec![0x10, 0x00, 0x00, 0x06, 0xAA, 0xBB];
mkb.extend_from_slice(&[0x00, 0x00, 0x00, 0x00]); // end marker
mkb.extend_from_slice(&[0x05, 0x00, 0x00, 0x08, 9, 9, 9, 9]); // ignored
let recs = walk_mkb(&mkb);
assert_eq!(recs.len(), 1);
assert_eq!(recs[0].rec_type, 0x10);
}
#[test]
fn walk_mkb_stops_on_overrun_record() {
// rec_len running past the buffer ends the walk after the records that
// fit (no OOB, no partial body past the end).
let mut mkb = vec![0x10, 0x00, 0x00, 0x06, 0xAA, 0xBB];
mkb.extend_from_slice(&[0x05, 0x00, 0xFF, 0xFF]); // claims 65535 bytes
let recs = walk_mkb(&mkb);
assert_eq!(recs.len(), 1, "overrun record must be dropped");
}
#[test]
fn walk_mkb_stops_on_sub_4_length() {
// A non-zero type with rec_len < 4 (and not the 0/0 marker) breaks the
// walk — otherwise pos would not advance (infinite loop guard).
let mkb = vec![0x10, 0x00, 0x00, 0x02, 0xAA];
assert!(walk_mkb(&mkb).is_empty());
}
#[test]
fn walk_mkb_handles_trailing_partial_header() {
// Fewer than 4 bytes left → loop condition `pos + 4 <= len` stops.
let mkb = vec![0x10, 0x00, 0x00, 0x06, 0xAA, 0xBB, 0x05, 0x00]; // 2 trailing
let recs = walk_mkb(&mkb);
assert_eq!(recs.len(), 1);
}
// ── Record selectors ───────────────────────────────────────────────────
#[test]
fn is_variant_mkb_true_for_0x82_alone_and_0x83_alone() {
// Either record type alone flags the MKB as variant.
let only82 = walk_mkb(&{
let mut m = vec![0x10, 0x00, 0x00, 0x08, 0, 0, 0, 0];
m.extend_from_slice(&[0x82, 0x00, 0x00, 0x14]);
m.extend_from_slice(&[0xEE; 16]);
m
});
assert!(is_variant_mkb(&only82));
let only83 = walk_mkb(&{
let mut m = vec![0x10, 0x00, 0x00, 0x08, 0, 0, 0, 0];
m.extend_from_slice(&[0x83, 0x00, 0x00, 0x14]);
m.extend_from_slice(&[0x55; 16]);
m
});
assert!(is_variant_mkb(&only83));
}
#[test]
fn variant_nonce_requires_16_byte_body() {
// A 0x83 record with < 16-byte body → None (no panic on the copy).
let recs = walk_mkb(&{
let mut m = vec![0x83, 0x00, 0x00, 0x0C]; // 8-byte body
m.extend_from_slice(&[0x11; 8]);
m
});
assert_eq!(variant_nonce(&recs), None);
}
#[test]
fn variant_key_data_requires_nonempty_multiple_of_16() {
// A 0x82 body that is NOT a multiple of 16 is rejected by
// variant_key_data (it needs whole 16-byte VKD slots).
let recs = walk_mkb(&{
let mut m = vec![0x82, 0x00, 0x00, 0x0E]; // 10-byte body (not %16)
m.extend_from_slice(&[0x22; 10]);
m
});
assert_eq!(variant_key_data(&recs), None);
// variant_data_record returns the body regardless of length.
assert_eq!(variant_data_record(&recs), Some(&[0x22u8; 10][..]));
}
// ── derive_media_key_variant: missing-record classification ────────────
#[test]
fn chain_reports_processing_key_unavailable_with_no_dks() {
// A complete variant MKB but an empty device-key pool → no uv covered
// → ProcessingKeyUnavailable (the walk_processing_key None branch).
let (recs, _dk, _, _) = synthetic_variant_setup(0x00);
let err = derive_media_key_variant(&recs, &[], &[0xAA; 16], &[0u8; 16])
.expect_err("no DK → ProcessingKeyUnavailable");
assert_eq!(err, MediaKeyVariantError::ProcessingKeyUnavailable);
}
#[test]
fn chain_reports_mkb_incomplete_when_nonce_missing() {
// Build a variant MKB (has 0x82 so is_variant true, and a DK can walk
// it) but WITHOUT a 0x83 nonce record → MkbIncomplete at the
// variant_nonce `?`.
// Start from the full setup, then rebuild the byte stream dropping
// the 0x83 record.
let (recs, dk, _, _) = synthetic_variant_setup(0x00);
// Reconstruct bytes without the 0x83 record.
let mut mkb = Vec::new();
for r in &recs {
if r.rec_type == 0x83 {
continue;
}
mkb.push(r.rec_type);
mkb.push(((r.rec_len >> 16) & 0xFF) as u8);
mkb.push(((r.rec_len >> 8) & 0xFF) as u8);
mkb.push((r.rec_len & 0xFF) as u8);
mkb.extend_from_slice(&r.body);
}
let recs2 = walk_mkb(&mkb);
assert!(is_variant_mkb(&recs2), "still variant via 0x82");
let err = derive_media_key_variant(&recs2, &[dk], &[0xAA; 16], &[0u8; 16])
.expect_err("missing nonce → MkbIncomplete");
assert_eq!(err, MediaKeyVariantError::MkbIncomplete);
}
// ── walk_processing_key: skips out-of-range u_mask_shift ───────────────
#[test]
fn walk_processing_key_skips_shift_32_to_63_without_panic() {
// A subset-difference u_mask_shift in 0x20..=0x3F passes the 0xC0
// revoke check but is out of range for a u32 shift. The walk must skip
// the slot (continue) and not panic / not match a wrong uv. With only
// that one bad slot, no match → None.
let mut mkb = vec![
0x10, 0x00, 0x00, 0x0C, 0x48, 0x14, 0x10, 0x03, 0x00, 0x00, 0x00, 0x4D,
];
// 0x04: u_mask_shift=0x20 (32), uv=2.
mkb.extend_from_slice(&[0x04, 0x00, 0x00, 0x09]);
mkb.extend_from_slice(&[0x20, 0x00, 0x00, 0x00, 0x02]);
mkb.extend_from_slice(&[0x07, 0x00, 0x00, 0x14]);
mkb.extend_from_slice(&[0xAB; 16]);
mkb.extend_from_slice(&[0x86, 0x00, 0x00, 0x14]);
mkb.extend_from_slice(&[0xCD; 16]);
let recs = walk_mkb(&mkb);
let dk = DeviceKey {
key: [0x11; 16],
node: 4,
uv: 2,
u_mask_shift: 3,
};
assert!(
walk_processing_key(&recs, &[dk]).is_none(),
"out-of-range shift must be skipped, yielding no match"
);
}
#[test]
fn walk_processing_key_skips_uv_zero() {
// A uv == 0 slot is skipped (`if uv == 0 { continue }`). With only a
// zero-uv slot present, no DK can match → None.
let mut mkb = vec![
0x10, 0x00, 0x00, 0x0C, 0x48, 0x14, 0x10, 0x03, 0x00, 0x00, 0x00, 0x4D,
];
mkb.extend_from_slice(&[0x04, 0x00, 0x00, 0x09]);
mkb.extend_from_slice(&[0x03, 0x00, 0x00, 0x00, 0x00]); // uv = 0
mkb.extend_from_slice(&[0x07, 0x00, 0x00, 0x14]);
mkb.extend_from_slice(&[0xAB; 16]);
mkb.extend_from_slice(&[0x86, 0x00, 0x00, 0x14]);
mkb.extend_from_slice(&[0xCD; 16]);
let recs = walk_mkb(&mkb);
let dk = DeviceKey {
key: [0x11; 16],
node: 4,
uv: 2,
u_mask_shift: 3,
};
assert!(walk_processing_key(&recs, &[dk]).is_none());
}
#[test]
fn walk_processing_key_returns_match_on_variant_mkb_without_magic() {
// On a variant MKB the per-match VERIFY_MAGIC check does not hold, but
// the walk still returns the (Kp, uv) match because variant_present is
// true. The synthetic_variant_setup fixture is exactly this case.
let (recs, dk, planted_kp, _) = synthetic_variant_setup(0x00);
let m = walk_processing_key(&recs, &[dk]).expect("variant MKB yields a match");
assert_eq!(m.uv, 2, "matched the planted uv");
assert_eq!(m.kp, planted_kp, "Kp equals aesg3(dk,1) for the no-op walk");
assert_eq!(m.cvalue_index, 0);
}
#[test]
fn aes_g_matches_decrypt_xor_relation() {
// AES-G(x1,x2) = AES-128D(x1,x2) XOR x2 — the same form as derive_vuk.
// Pin it explicitly so a dropped XOR or an encrypt-instead-of-decrypt
// is caught.
let x1 = [0x31u8; 16];
let x2 = [0x9Fu8; 16];
let mut expected = aes_ecb_decrypt(&x1, &x2);
for i in 0..16 {
expected[i] ^= x2[i];
}
assert_eq!(aes_g(&x1, &x2), expected);
}
#[test]
fn error_codes_are_unique_and_in_7100_range() {
// Each MediaKeyVariantError maps to a distinct E71xx code. A
// copy-paste collision (two variants sharing a code) would break
// operator triage; assert all nine are distinct.
use std::collections::HashSet;
let cases = [
MediaKeyVariantError::NotVariantMkb,
MediaKeyVariantError::MkbIncomplete,
MediaKeyVariantError::ProcessingKeyUnavailable,
MediaKeyVariantError::SoftCorrectionRequired,
MediaKeyVariantError::OnlineChallengeRequired,
MediaKeyVariantError::KcdNotProvided,
MediaKeyVariantError::VariantsTableUnavailable,
MediaKeyVariantError::VkdIndexOutOfRange,
MediaKeyVariantError::MediaKeyVerifyFailed,
];
let codes: HashSet<String> = cases.iter().map(|e| e.to_string()).collect();
assert_eq!(codes.len(), cases.len(), "all error codes must be unique");
}
}
-172
View File
@@ -1,172 +0,0 @@
//! AACS Verify-Media-Key magic constants used to confirm Media Key
//! candidates produced during MKB walking.
//!
//! AACS MKBs contain "Verify Media Key Records" whose decrypted output
//! is a known-plaintext constant. Walking code decrypts the verify
//! record with each MK candidate and compares the result against the
//! magic; on match, the MK is correct.
//!
//! Five distinct magics are observed in the canonical reference AACS
//! engine (MakeMKV v1.18.3, file offsets in parens):
//!
//! 1. **MK\_V10** at `.rodata:0x2909c0`. The original AACS-1.0 spec
//! constant. Single 16-byte AES-128-ECB compare. Used at 3 sites in
//! that engine. We already use it in `keys.rs::validate_media_key_against_mkb`.
//!
//! 2. **MK\_AUX\_16** at `.rodata:0x290890`. A second single-block
//! 16-byte verification magic. Reverse-engineering of the call site
//! at `0x580f73` shows it after a call to the single-block AES-ECB
//! helper. Likely a per-vendor or per-record-type extended verify.
//! Use it when an MKB carries an extended verify record alongside
//! the standard one.
//!
//! 3. **MK\_SK\_32a** = `MK_SK32A_BLK0` || `MK_SK32A_BLK1`. A 32-byte
//! (2-block) verify magic at `.rodata:0x290910 / 0x290620`. Used at
//! `0x580ff0`: both blocks must match after AES-128 decrypt of a
//! 32-byte verify record. Almost certainly the AACS-2 / Sequence
//! Key Block "Verify Media Key Record for Sequence Keys" expanded
//! form — i.e. AACS-2 SKB verification.
//!
//! 4. **MK\_SK\_32b** = `MK_SK32B_BLK0` || `MK_SK32B_BLK1`. A second
//! 32-byte verify magic at `.rodata:0x290980 / 0x290a60`. Used at
//! `0x581063`. Different record type within the SKB family — likely
//! the AACS-2 SD-tree variant verification.
//!
//! All five are KNOWN PLAINTEXT compared bit-for-bit against the
//! AES-128 decrypt output. They are NOT keys. They are oracle values
//! that say "yes, the MK candidate you tried is the right one."
//!
//! Provenance: identified via static RE of MakeMKV v1.18.3 amd64
//! (binary sha256 `9970a50a97231b2d09d73f521ff1daf0609ea201040a68ecaa9f31af957d6401`)
//! on 2026-05-22 via objdump of the `pcmpeqb` callsite cluster around
//! file offset `0x580f70..0x581080`.
/// AACS-1.0 / pre-existing canonical Verify Media Key magic.
///
/// `AES-128-ECB-DECRYPT(MK, verify_record) == [VERIFY_MK_V10 || pad]`
pub const VERIFY_MK_V10: [u8; 8] = [0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF];
/// Single-block 16-byte verify magic (auxiliary). Compared full-16
/// after AES-128-ECB(MK, in) at `pcmpeqb` site `0x580f73`.
pub const VERIFY_MK_AUX_16: [u8; 16] = [
0xf9, 0x91, 0xa3, 0x60, 0x68, 0x15, 0xa6, 0xb9, 0x55, 0xbb, 0xce, 0xa3, 0xb1, 0x4b, 0xf8, 0xd8,
];
/// 32-byte SKB-style verify magic, block 0 of 2. Compared full-16
/// after AES-128 decrypt of the first 16 bytes of a 32-byte verify
/// record. `pcmpeqb` site `0x580ff0`.
pub const VERIFY_MK_SK_32A_BLK0: [u8; 16] = [
0x19, 0x0f, 0xe9, 0x7f, 0xad, 0x11, 0xa4, 0x10, 0xc6, 0x56, 0x9d, 0x1c, 0x84, 0x21, 0x1d, 0x18,
];
/// 32-byte SKB-style verify magic, block 1 of 2. Compared full-16
/// after AES-128 decrypt of bytes 16..32 of the same record.
/// `pcmpeqb` site `0x580fe8`.
pub const VERIFY_MK_SK_32A_BLK1: [u8; 16] = [
0x9b, 0x54, 0x9a, 0x25, 0x69, 0x8a, 0xa2, 0x3f, 0x9d, 0xfd, 0x2c, 0x95, 0xe2, 0x4a, 0x97, 0x02,
];
/// 32-byte SKB-style verify magic (variant B), block 0 of 2.
/// `pcmpeqb` site `0x581063`.
pub const VERIFY_MK_SK_32B_BLK0: [u8; 16] = [
0x8d, 0xee, 0xe0, 0x1e, 0xc7, 0x0c, 0xea, 0xb3, 0xdb, 0xd2, 0xfb, 0x82, 0x16, 0x3c, 0x26, 0x80,
];
/// 32-byte SKB-style verify magic (variant B), block 1 of 2.
/// `pcmpeqb` site `0x58105b`.
pub const VERIFY_MK_SK_32B_BLK1: [u8; 16] = [
0xaf, 0x93, 0x7a, 0x74, 0x8a, 0xce, 0xd3, 0x69, 0x36, 0x84, 0xe6, 0xea, 0xf8, 0x54, 0xe8, 0xa2,
];
/// Tag for a candidate-Media-Key check. Tells the verifier which
/// known-plaintext to compare against; the verifier chooses the
/// magic that matches the MKB record type at hand.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum VerifyMagic {
/// AACS-1.0 / canonical.
V10,
/// Auxiliary single-block (16-byte) verification.
Aux16,
/// SKB-style 32-byte verification, variant A.
Sk32A,
/// SKB-style 32-byte verification, variant B.
Sk32B,
}
/// Verify a candidate Media Key against a `dec_vd` (AES-128 decrypt
/// of the MKB Verify Media Key Record under the candidate MK).
///
/// Returns `true` if `dec_vd` matches the magic identified by `tag`.
///
/// - `V10`: compares the first 8 bytes against `VERIFY_MK_V10`.
/// - `Aux16`: compares the full 16 bytes against `VERIFY_MK_AUX_16`.
/// - `Sk32A` / `Sk32B`: `dec_vd` must be exactly 32 bytes (`block0 ||
/// block1`); compares each block against the corresponding constant.
pub fn check_verify(tag: VerifyMagic, dec_vd: &[u8]) -> bool {
match tag {
VerifyMagic::V10 => dec_vd.len() >= 8 && dec_vd[..8] == VERIFY_MK_V10,
VerifyMagic::Aux16 => dec_vd.len() >= 16 && dec_vd[..16] == VERIFY_MK_AUX_16,
VerifyMagic::Sk32A => {
dec_vd.len() >= 32
&& dec_vd[..16] == VERIFY_MK_SK_32A_BLK0
&& dec_vd[16..32] == VERIFY_MK_SK_32A_BLK1
}
VerifyMagic::Sk32B => {
dec_vd.len() >= 32
&& dec_vd[..16] == VERIFY_MK_SK_32B_BLK0
&& dec_vd[16..32] == VERIFY_MK_SK_32B_BLK1
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn v10_matches_canonical_prefix() {
let mut dec = [0u8; 16];
dec[..8].copy_from_slice(&VERIFY_MK_V10);
assert!(check_verify(VerifyMagic::V10, &dec));
}
#[test]
fn aux16_matches_full_block() {
assert!(check_verify(VerifyMagic::Aux16, &VERIFY_MK_AUX_16));
}
#[test]
fn sk32a_requires_both_blocks() {
let mut dec = [0u8; 32];
dec[..16].copy_from_slice(&VERIFY_MK_SK_32A_BLK0);
dec[16..].copy_from_slice(&VERIFY_MK_SK_32A_BLK1);
assert!(check_verify(VerifyMagic::Sk32A, &dec));
// Mutate block 1, must fail.
dec[20] ^= 0x80;
assert!(!check_verify(VerifyMagic::Sk32A, &dec));
}
#[test]
fn sk32b_distinct_from_sk32a() {
let mut dec = [0u8; 32];
dec[..16].copy_from_slice(&VERIFY_MK_SK_32B_BLK0);
dec[16..].copy_from_slice(&VERIFY_MK_SK_32B_BLK1);
assert!(check_verify(VerifyMagic::Sk32B, &dec));
// Same plaintext must NOT validate as Sk32A.
assert!(!check_verify(VerifyMagic::Sk32A, &dec));
}
#[test]
fn short_input_never_matches() {
let dec = [0u8; 4];
for tag in [
VerifyMagic::V10,
VerifyMagic::Aux16,
VerifyMagic::Sk32A,
VerifyMagic::Sk32B,
] {
assert!(!check_verify(tag, &dec));
}
}
}
+609 -39
View File
@@ -6,22 +6,28 @@
//!
//! Reference: https://github.com/lw/BluRay/wiki/CLPI
use crate::consts::{BD_SOURCE_PACKET_BYTES, SECTOR_BYTES_U64};
use crate::disc::Extent;
use crate::error::{Error, Result};
/// Parsed CLPI clip info.
#[derive(Debug)]
pub(crate) struct ClipInfo {
/// CLPI version string. Parsed for completeness; not yet consumed.
#[allow(dead_code)]
pub struct ClipInfo {
pub version: String,
/// Total source packets in the m2ts (each 192 bytes)
pub source_packet_count: u32,
/// Coarse EP entries for the primary video stream
/// Coarse EP entries for the primary video stream. Populated for the
/// EP-map → sector-extent lookup (`get_extents`), which is exercised by
/// tests and reserved for the timestamp-range read path.
#[allow(dead_code)]
pub ep_coarse: Vec<EpCoarse>,
/// Fine EP entries for the primary video stream
/// Fine EP entries for the primary video stream (see `ep_coarse`).
#[allow(dead_code)]
pub ep_fine: Vec<EpFine>,
/// Per-stream metadata from the ProgramInfo section (BD spec).
/// Cross-validates the MPLS STN view — see `labels/clpi.rs`.
/// Cross-validates the MPLS STN view — see `labels/clpi_audit.rs`.
/// Empty when program_info is missing or malformed.
pub streams: Vec<ClpiStream>,
}
@@ -30,57 +36,82 @@ pub struct ClipInfo {
/// table. Mirrors the same fields the MPLS STN table carries — see
/// `mpls::StreamEntry` for the playlist-side equivalent.
#[derive(Debug, Clone)]
#[allow(dead_code)]
pub struct ClpiStream {
pub(crate) struct ClpiStream {
/// PID of the stream in the MPEG-TS (matches MPLS).
pub pid: u16,
/// SCSI/BD coding type byte (0x80 LPCM, 0x83 TrueHD, 0x86 DTS-HD MA,
/// BD stream coding type byte (0x80 LPCM, 0x83 TrueHD, 0x86 DTS-HD MA,
/// 0x90 PG, etc.). See `labels::mpls_universal::coding_type_to_codec_hint`.
pub coding_type: u8,
/// ISO 639-2 3-char language code. Empty for video streams.
pub language: String,
// The CLPI cross-validation consumer (labels/clpi_audit.rs) reads only
// pid/coding_type/language. The codec sub-fields below are parsed from
// the BD stream_coding_info for completeness but have no reader yet.
/// Audio format byte (1=mono, 3=stereo, 6=5.1, 12=7.1).
/// Zero for non-audio streams.
#[allow(dead_code)]
pub audio_format: u8,
/// Audio sample rate (1=48kHz, 4=96kHz, 5=192kHz). Zero for non-audio.
#[allow(dead_code)]
pub audio_rate: u8,
/// Video format byte (1=480i, 4=1080i, 5=720p, 6=1080p, 8=2160p).
/// Zero for non-video.
#[allow(dead_code)]
pub video_format: u8,
/// Video rate (1=23.976, 2=24, 3=25, 4=29.97, 6=50, 7=59.94).
#[allow(dead_code)]
pub video_rate: u8,
}
/// Coarse EP-map entry. Fields feed the EP-map resolution used by
/// `get_extents` (test-exercised; reserved for the timestamp-range path).
#[derive(Debug, Clone)]
#[allow(dead_code)]
pub struct EpCoarse {
pub(crate) struct EpCoarse {
pub ref_to_fine_id: u32,
pub pts_coarse: u32,
pub spn_coarse: u32,
}
/// Fine EP-map entry (see `EpCoarse`).
#[derive(Debug, Clone)]
#[allow(dead_code)]
pub struct EpFine {
pub(crate) struct EpFine {
pub pts_fine: u32,
pub spn_fine: u32,
}
// EP-map → sector-extent resolution. Exercised by the unit tests and
// reserved for the timestamp-range read path; no production caller yet.
#[allow(dead_code)]
impl ClipInfo {
/// Reconstruct full PTS from coarse + fine entry.
pub fn full_pts(coarse: &EpCoarse, fine: &EpFine) -> u32 {
(coarse.pts_coarse << 19) + (fine.pts_fine << 8)
///
/// The BD spec PTS is 33-bit: `pts_coarse` is 14 bits (max 16383) and
/// `16383 << 19` exceeds `u32::MAX`, so the result must be `u64` to
/// avoid overflow (panic in debug, silent wrap in release).
pub fn full_pts(coarse: &EpCoarse, fine: &EpFine) -> u64 {
((coarse.pts_coarse as u64) << 19) + ((fine.pts_fine as u64) << 8)
}
/// Reconstruct full SPN from coarse + fine entry.
pub fn full_spn(coarse: &EpCoarse, fine: &EpFine) -> u32 {
(coarse.spn_coarse & 0xFFFE_0000) + fine.spn_fine
// The two operands occupy non-overlapping bit ranges (coarse holds
// the high bits, fine the low 17), so OR expresses intent and is
// robust to a hand-constructed EpFine.
debug_assert!(fine.spn_fine <= 0x1_FFFF);
(coarse.spn_coarse & 0xFFFE_0000) | fine.spn_fine
}
/// Get all EP entries as (PTS, SPN) pairs, fully resolved.
pub fn resolved_ep_map(&self) -> Vec<(u32, u32)> {
let mut entries = Vec::new();
///
/// PTS resets at each coarse-group boundary on disc, so the raw
/// concatenation is not globally monotonic. The returned vector is
/// sorted by PTS so callers (e.g. [`get_extents`]) can binary-search it.
///
/// [`get_extents`]: ClipInfo::get_extents
pub fn resolved_ep_map(&self) -> Vec<(u64, u32)> {
let mut entries = Vec::with_capacity(self.ep_fine.len());
for (ci, coarse) in self.ep_coarse.iter().enumerate() {
let fine_start = coarse.ref_to_fine_id as usize;
@@ -98,6 +129,12 @@ impl ClipInfo {
}
}
// get_extents binary-searches by PTS, so the map must be ordered.
// Real discs have globally increasing PTS in coarse order; sort by
// (pts, spn) so a cross-group PTS collision can't leave the search
// landing on the wrong group's SPN.
entries.sort_by_key(|&(pts, spn)| (pts, spn));
entries
}
@@ -105,7 +142,9 @@ impl ClipInfo {
///
/// Converts PTS timestamps to SPN ranges, then SPN to LBA
/// using the file's starting LBA on disc.
pub fn get_extents(&self, in_time: u32, out_time: u32) -> Vec<Extent> {
pub fn get_extents(&self, in_time: u64, out_time: u64) -> Vec<Extent> {
// resolved_ep_map() returns entries sorted by PTS, so binary search
// is valid here.
let ep_map = self.resolved_ep_map();
if ep_map.is_empty() {
return Vec::new();
@@ -122,20 +161,22 @@ impl ClipInfo {
let end_spn = match ep_map.binary_search_by_key(&out_time, |(pts, _)| *pts) {
Ok(i) => ep_map[i].1,
Err(i) if i < ep_map.len() => ep_map[i].1,
_ => ep_map.last().unwrap().1 + 1,
_ => ep_map.last().unwrap().1.saturating_add(1),
};
if end_spn <= start_spn {
return Vec::new();
}
// SPN → byte offset: spn × 192
// Byte offset → sectors: offset / 2048
// Note: the caller needs to add the file's starting LBA from UDF
let start_byte = start_spn as u64 * 192;
let end_byte = end_spn as u64 * 192;
let start_sector = (start_byte / 2048) as u32;
let end_sector = end_byte.div_ceil(2048) as u32;
// SPN → byte offset → sector range. Note: the caller adds the file's
// starting LBA from UDF. The start sector FLOORS (the extent begins in
// whichever sector contains its first byte) and the end sector CEILS
// (the extent must cover through the sector holding its last byte), so
// a sub-sector-aligned range still spans every sector it touches.
let start_byte = start_spn as u64 * BD_SOURCE_PACKET_BYTES as u64;
let end_byte = end_spn as u64 * BD_SOURCE_PACKET_BYTES as u64;
let start_sector = (start_byte / SECTOR_BYTES_U64) as u32;
let end_sector = end_byte.div_ceil(SECTOR_BYTES_U64) as u32;
vec![Extent {
start_lba: start_sector, // relative to m2ts file start
@@ -162,7 +203,7 @@ pub fn parse(data: &[u8]) -> Result<ClipInfo> {
// ClipInfo section at offset 40
// source_packet_count at offset 40 + 4(len) + 2(reserved) + 1(stream_type) + 1(app_type) + 4(reserved) + 4(ts_rate)
let source_packet_count = if data.len() > 56 {
let source_packet_count = if data.len() >= 60 {
u32::from_be_bytes([data[56], data[57], data[58], data[59]])
} else {
0
@@ -220,6 +261,7 @@ pub fn parse(data: &[u8]) -> Result<ClipInfo> {
/// errors because the EP map is the primary CLPI output, and a corrupt
/// program_info shouldn't break sector-range lookups.
fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
use crate::consts::coding_type as c;
let mut out = Vec::new();
if data.len() < 6 {
return out;
@@ -258,16 +300,15 @@ fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
let mut language = String::new();
match coding_type {
// Video — MPEG-2 (0x02), H.264 (0x1B), HEVC (0x24)
0x02 | 0x1B | 0x24 => {
// Video — MPEG-2, H.264, HEVC
c::MPEG2_VIDEO | c::H264 | c::HEVC => {
if sci.len() >= 2 {
video_format = (sci[1] >> 4) & 0x0F;
video_rate = sci[1] & 0x0F;
}
}
// Primary audio — LPCM(0x80), AC-3(0x81), DTS(0x82),
// TrueHD(0x83), AC-3+(0x84), DTS-HD(0x85), DTS-HD MA(0x86)
0x80..=0x86 => {
// Primary audio — LPCM, AC-3, DTS, TrueHD, AC-3+, DTS-HD HR, DTS-HD MA
c::LPCM..=c::DTS_HD_MA => {
if sci.len() >= 2 {
audio_format = (sci[1] >> 4) & 0x0F;
audio_rate = sci[1] & 0x0F;
@@ -276,8 +317,8 @@ fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
language = String::from_utf8_lossy(&sci[2..5]).to_string();
}
}
// Secondary audio (0xA1 AC-3+, 0xA2 DTS-HD)
0xA1 | 0xA2 => {
// Secondary audio (AC-3+ secondary, DTS-HD secondary)
c::AC3_PLUS_SECONDARY | c::DTS_HD_SECONDARY => {
if sci.len() >= 2 {
audio_format = (sci[1] >> 4) & 0x0F;
audio_rate = sci[1] & 0x0F;
@@ -286,8 +327,8 @@ fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
language = String::from_utf8_lossy(&sci[2..5]).to_string();
}
}
// PG (0x90), IG (0x91): coding_type + 3-byte language [+ char_code for PG]
0x90 | 0x91 => {
// PG, IG: coding_type + 3-byte language [+ char_code for PG]
c::PG | c::IG => {
if sci.len() >= 4 {
language = String::from_utf8_lossy(&sci[1..4]).to_string();
}
@@ -322,8 +363,17 @@ fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
return Ok((Vec::new(), Vec::new()));
}
// Bound all EP-map reads to this CPI section. The length field counts
// bytes after itself, so the section spans data[..cpi_length + 4]. A
// bogus ep_map_offset within data.len() but past the CPI section would
// otherwise read into an adjacent CLPI section; clamp first.
let data = &data[..(cpi_length + 4).min(data.len())];
// CPI type at bits 44-47 (byte 5, lower 4 bits)
// Skip to EP map: offset 4 (after length) + 2 (reserved/type)
if data.len() < 6 {
return Ok((Vec::new(), Vec::new()));
}
let ep_map = &data[6..];
if ep_map.len() < 4 {
return Ok((Vec::new(), Vec::new()));
@@ -351,9 +401,6 @@ fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
// num_EP_coarse: 16 bits │ (10+4+16+18+32 = 80)
// num_EP_fine: 18 bits │
// EP_map_start_address: 32 bits ┘
if ep_map.len() < 16 {
return Ok((Vec::new(), Vec::new()));
}
let _stream_pid = u16::from_be_bytes([ep_map[2], ep_map[3]]);
// Read 10 bytes (80 bits) from ep_map[4..14] for bit extraction
@@ -389,7 +436,10 @@ fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
// Coarse entries start at offset 4, 8 bytes each
let coarse_data = &stream_ep[4..];
let mut ep_coarse = Vec::with_capacity(num_coarse);
// Cap the pre-reservation by what the slice can actually hold:
// num_coarse is a 16-bit disc field, so a hostile value would
// otherwise reserve up to ~0.5 MB for an entry table that doesn't exist.
let mut ep_coarse = Vec::with_capacity(num_coarse.min(coarse_data.len() / 8));
for i in 0..num_coarse {
let off = i * 8;
if off + 8 > coarse_data.len() {
@@ -419,7 +469,13 @@ fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
}
// Fine entries at fine_start, 4 bytes each
let mut ep_fine = Vec::with_capacity(num_fine);
// Cap the pre-reservation: num_fine is an 18-bit disc field (max
// 262143), so reserve only what the slice can actually hold.
let mut ep_fine = if fine_start < stream_ep.len() {
Vec::with_capacity(num_fine.min((stream_ep.len() - fine_start) / 4))
} else {
Vec::new()
};
if fine_start < stream_ep.len() {
let fine_data = &stream_ep[fine_start..];
for i in 0..num_fine {
@@ -643,10 +699,49 @@ mod tests {
};
// full_pts = (100 << 19) + (50 << 8) = 52_428_800 + 12_800 = 52_441_600
let pts = ClipInfo::full_pts(&coarse, &fine);
assert_eq!(pts, (100 << 19) + (50 << 8));
assert_eq!(pts, (100u64 << 19) + (50u64 << 8));
assert_eq!(pts, 52_441_600);
}
#[test]
fn full_pts_no_u32_overflow() {
// pts_coarse is a 14-bit field (max 0x3FFF = 16383); 16383 << 19
// overflows u32, so full_pts must use u64.
let coarse = EpCoarse {
ref_to_fine_id: 0,
pts_coarse: 0x3FFF,
spn_coarse: 0,
};
let fine = EpFine {
pts_fine: 0x7FF,
spn_fine: 0,
};
let pts = ClipInfo::full_pts(&coarse, &fine);
assert_eq!(pts, (0x3FFFu64 << 19) + (0x7FFu64 << 8));
assert!(pts > u32::MAX as u64);
}
#[test]
fn resolved_ep_map_sorted_for_binary_search() {
// Two coarse groups whose fine PTS reset across the boundary
// (50,100 then 25,75) produce a non-monotonic raw concatenation.
// resolved_ep_map must sort so get_extents' binary search is valid.
let cpi = build_cpi(
0x1011,
&[(0, 0, 0x00020000), (2, 0, 0x00040000)],
&[(50, 1024), (100, 2048), (25, 512), (75, 1536)],
);
let data = build_clpi(1_000_000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let resolved = clip.resolved_ep_map();
assert_eq!(resolved.len(), 4);
// Strictly sorted by PTS.
for w in resolved.windows(2) {
assert!(w[0].0 <= w[1].0, "ep_map not sorted: {resolved:?}");
}
}
#[test]
fn full_spn_calculation() {
let coarse = EpCoarse {
@@ -674,6 +769,22 @@ mod tests {
assert_eq!(spn2, 0x00FE0000 + 0x1234);
}
#[test]
fn parse_truncated_clipinfo_no_panic() {
// 57/58/59-byte CLPI with valid magic: passes the data.len() < 40
// guard but data[56..60] needs 60 bytes. Must not panic.
for len in 40..60usize {
let mut data = vec![0u8; len];
data[0..4].copy_from_slice(b"HDMV");
if len >= 8 {
data[4..8].copy_from_slice(b"0200");
}
let clip = parse(&data).expect("short CLPI should parse, not panic");
// source_packet_count is unreadable below 60 bytes → 0.
assert_eq!(clip.source_packet_count, 0);
}
}
#[test]
fn parse_invalid_magic() {
let mut data = build_clpi(1000, None);
@@ -707,4 +818,463 @@ mod tests {
assert!(clip2.ep_coarse.is_empty());
assert!(clip2.ep_fine.is_empty());
}
// ─────────────────────────────────────────────────────────────────────
// Added hardening tests. Grounded in the BD-ROM CLPI spec
// (https://github.com/lw/BluRay/wiki/CLPI) and libbluray clpi_parse.c.
// ─────────────────────────────────────────────────────────────────────
/// Build a ProgramInfo section. `streams` = Vec<(pid, sci_bytes)>.
/// Layout per source doc: length(4)+reserved(1)+num_programs(1)+
/// per program [spn(4)+pmt_pid(2)+num_streams(1)+num_groups(1)] then
/// per stream [pid(2)+sci_len(1)+sci].
fn build_program_info(streams: &[(u16, Vec<u8>)]) -> Vec<u8> {
let mut body = Vec::new();
body.push(0); // reserved (offset 4)
body.push(1); // num_programs = 1 (offset 5)
// program 0 header (8 bytes)
body.extend_from_slice(&0u32.to_be_bytes()); // spn_program_sequence_start
body.extend_from_slice(&0u16.to_be_bytes()); // program_map_pid
body.push(streams.len() as u8); // num_streams
body.push(0); // num_groups
for (pid, sci) in streams {
body.extend_from_slice(&pid.to_be_bytes());
body.push(sci.len() as u8);
body.extend_from_slice(sci);
}
// Prepend length(4) = bytes after the length field.
let mut out = Vec::new();
out.extend_from_slice(&(body.len() as u32).to_be_bytes());
out.extend_from_slice(&body);
out
}
/// Build a CLPI with a ProgramInfo section. prog_info_start is placed
/// right after the 60-byte header; cpi (if any) follows program_info.
fn build_clpi_with_proginfo(
source_packet_count: u32,
prog_info: &[u8],
cpi_data: Option<&[u8]>,
) -> Vec<u8> {
let mut buf = vec![0u8; 60];
buf[0..4].copy_from_slice(b"HDMV");
buf[4..8].copy_from_slice(b"0200");
let prog_info_start: u32 = 60;
buf[12..16].copy_from_slice(&prog_info_start.to_be_bytes());
let cpi_start: u32 = if cpi_data.is_some() {
(60 + prog_info.len()) as u32
} else {
0
};
buf[16..20].copy_from_slice(&cpi_start.to_be_bytes());
buf[56..60].copy_from_slice(&source_packet_count.to_be_bytes());
buf.extend_from_slice(prog_info);
if let Some(cpi) = cpi_data {
buf.extend_from_slice(cpi);
}
buf
}
/// source_packet_count is a big-endian u32 at offset [56..60]. Verify
/// BE decode of a value with all four bytes distinct (not LE / wrong
/// offset).
#[test]
fn source_packet_count_big_endian_offset_56() {
let data = build_clpi(0x01020304, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.source_packet_count, 0x01020304);
}
/// Magic must be exactly "HDMV" at [0..4]. Anything else → ClpiParse.
/// Spec: CLPI files begin with the type_indicator "HDMV".
#[test]
fn wrong_magic_rejected() {
let mut data = build_clpi(1000, None);
data[0..4].copy_from_slice(b"INDX");
assert!(parse(&data).is_err());
}
/// Under-40-byte input is rejected before any field read
/// (`data.len() < 40` guard).
#[test]
fn under_40_bytes_rejected() {
assert!(parse(&[0u8; 39]).is_err());
assert!(parse(b"HDMV0200").is_err());
assert!(parse(&[]).is_err());
}
/// ProgramInfo: a video stream (coding 0x1B = H.264) carries
/// format/rate in sci[1] nibbles and NO language. Verify the video
/// arm: format hi-nibble, rate lo-nibble, language stays empty.
#[test]
fn program_info_video_stream() {
// sci = coding_type(0x1B) + format_rate(0x61 → fmt 6, rate 1)
let sci = vec![0x1Bu8, 0x61];
let pi = build_program_info(&[(0x1011, sci)]);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.streams.len(), 1);
assert_eq!(clip.streams[0].pid, 0x1011);
assert_eq!(clip.streams[0].coding_type, 0x1B);
assert_eq!(clip.streams[0].video_format, 6);
assert_eq!(clip.streams[0].video_rate, 1);
assert_eq!(clip.streams[0].language, "");
}
/// ProgramInfo primary-audio (coding 0x80..=0x86): sci[1] = format/rate
/// nibbles, sci[2..5] = ISO 639 language. Verify TrueHD (0x83) at
/// offset, 5.1 / 48kHz, language "eng".
#[test]
fn program_info_audio_stream_lang_offset() {
// sci = 0x83 + 0x61 (fmt 6, rate 1) + "eng"
let sci = vec![0x83u8, 0x61, b'e', b'n', b'g'];
let pi = build_program_info(&[(0x1100, sci)]);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.streams[0].coding_type, 0x83);
assert_eq!(clip.streams[0].audio_format, 6);
assert_eq!(clip.streams[0].audio_rate, 1);
assert_eq!(clip.streams[0].language, "eng");
}
/// ProgramInfo PG (0x90)/IG (0x91): layout is coding_type(1)+lang(3),
/// so language is at sci[1..4] (NOT sci[2..5] like audio). Verify the
/// PG arm reads from the right offset.
#[test]
fn program_info_pg_lang_offset() {
// sci = 0x90 + "fra" (lang directly after coding_type)
let sci = vec![0x90u8, b'f', b'r', b'a'];
let pi = build_program_info(&[(0x1200, sci)]);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.streams[0].coding_type, 0x90);
assert_eq!(clip.streams[0].language, "fra");
// Audio nibbles must NOT be populated for a PG stream.
assert_eq!(clip.streams[0].audio_format, 0);
}
/// ProgramInfo with multiple streams: PID and coding for each must be
/// read from the correct per-stream offset (pid(2)+sci_len(1)+sci).
/// Three mixed streams must all parse with distinct PIDs in order.
#[test]
fn program_info_multiple_streams_advance_correctly() {
let v = (0x1011u16, vec![0x24u8, 0x81]); // HEVC video
let a = (0x1100u16, vec![0x86u8, 0x61, b'e', b'n', b'g']); // DTS-HD MA
let s = (0x1200u16, vec![0x90u8, b'j', b'p', b'n']); // PG
let pi = build_program_info(&[v, a, s]);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.streams.len(), 3);
assert_eq!(clip.streams[0].pid, 0x1011);
assert_eq!(clip.streams[0].coding_type, 0x24);
assert_eq!(clip.streams[1].pid, 0x1100);
assert_eq!(clip.streams[1].coding_type, 0x86);
assert_eq!(clip.streams[1].language, "eng");
assert_eq!(clip.streams[2].pid, 0x1200);
assert_eq!(clip.streams[2].language, "jpn");
}
/// parse_program_info is best-effort: a stream whose declared sci_len
/// runs past the section (`sci_end > data.len()`) makes it return the
/// streams collected so far (here: none), never panic. Source returns
/// `out` early on the overflow.
#[test]
fn program_info_truncated_sci_no_panic() {
// One stream claiming sci_len = 200 but with no body.
let mut body = Vec::new();
body.push(0); // reserved
body.push(1); // num_programs
body.extend_from_slice(&0u32.to_be_bytes());
body.extend_from_slice(&0u16.to_be_bytes());
body.push(1); // num_streams
body.push(0); // num_groups
body.extend_from_slice(&0x1011u16.to_be_bytes()); // pid
body.push(200); // sci_len = 200, no body follows
let mut pi = Vec::new();
pi.extend_from_slice(&(body.len() as u32).to_be_bytes());
pi.extend_from_slice(&body);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should not panic");
assert!(clip.streams.is_empty());
}
/// parse_program_info rejects sci_len == 0 (`sci_len < 1` → return).
/// A zero-length stream_coding_info is unusable.
#[test]
fn program_info_zero_sci_len_yields_no_stream() {
let mut body = Vec::new();
body.push(0);
body.push(1);
body.extend_from_slice(&0u32.to_be_bytes());
body.extend_from_slice(&0u16.to_be_bytes());
body.push(1);
body.push(0);
body.extend_from_slice(&0x1011u16.to_be_bytes());
body.push(0); // sci_len = 0
let mut pi = Vec::new();
pi.extend_from_slice(&(body.len() as u32).to_be_bytes());
pi.extend_from_slice(&body);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert!(clip.streams.is_empty());
}
/// pts_coarse field is 14 bits: dword0 = ref_to_fine_id<<14 | pts_coarse.
/// A pts_coarse of 0x3FFF (max) with ref_to_fine_id 5 must decode both
/// without bleed. Verify the >>14 and &0x3FFF split.
#[test]
fn coarse_pts_14bit_split() {
let cpi = build_cpi(0x1011, &[(5, 0x3FFF, 0x12340000)], &[(0, 0)]);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
assert_eq!(clip.ep_coarse[0].ref_to_fine_id, 5);
assert_eq!(clip.ep_coarse[0].pts_coarse, 0x3FFF);
assert_eq!(clip.ep_coarse[0].spn_coarse, 0x12340000);
}
/// Fine entry: dword = is_angle(1)+i_end_offset(3)+pts_fine(11)+
/// spn_fine(17). pts_fine occupies bits 17..28 (>>17 & 0x7FF), spn_fine
/// the low 17 bits (& 0x1FFFF). Set high bits (is_angle/i_end_offset)
/// and verify they do NOT bleed into pts_fine.
#[test]
fn fine_entry_bit_layout_isolates_pts_and_spn() {
// Construct a raw fine dword with is_angle=1, i_end_offset=0b111,
// pts_fine=0x5AA, spn_fine=0x1AAAA, then verify decode.
let is_angle: u32 = 1;
let i_end: u32 = 0b111;
let pts_f: u32 = 0x5AA; // 11-bit
let spn_f: u32 = 0x1AAAA; // 17-bit
let dword: u32 = (is_angle << 31) | (i_end << 28) | (pts_f << 17) | spn_f;
// Build the CPI by hand with this raw fine dword.
let mut stream_ep = Vec::new();
let fine_start: u32 = 4; // no coarse entries → fine right after header
stream_ep.extend_from_slice(&fine_start.to_be_bytes());
stream_ep.extend_from_slice(&dword.to_be_bytes());
let num_coarse: u32 = 0;
let num_fine: u32 = 1;
let ep_map_start: u32 = 14;
let ep_stream_type: u32 = 1;
let packed: u128 = ((ep_stream_type as u128) << 66)
| ((num_coarse as u128) << 50)
| ((num_fine as u128) << 32)
| (ep_map_start as u128);
let packed_bytes = packed.to_be_bytes();
let stream_header_bits = &packed_bytes[6..16];
let mut ep_map = Vec::new();
ep_map.push(0);
ep_map.push(1);
ep_map.extend_from_slice(&0x1011u16.to_be_bytes());
ep_map.extend_from_slice(stream_header_bits);
ep_map.extend_from_slice(&stream_ep);
let mut cpi = Vec::new();
cpi.extend_from_slice(&((2 + ep_map.len()) as u32).to_be_bytes());
cpi.extend_from_slice(&[0u8; 2]);
cpi.extend_from_slice(&ep_map);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
assert_eq!(clip.ep_fine.len(), 1);
assert_eq!(clip.ep_fine[0].pts_fine, 0x5AA); // high bits stripped
assert_eq!(clip.ep_fine[0].spn_fine, 0x1AAAA);
}
/// resolved_ep_map assigns fine entries to coarse groups via
/// [ref_to_fine_id .. next coarse's ref_to_fine_id). full_pts combines
/// coarse<<19 + fine<<8 and full_spn ORs masked coarse with fine.
/// Verify the first resolved entry's (pts, spn) for a known fixture.
#[test]
fn resolved_ep_map_combines_coarse_and_fine() {
// coarse 0: ref_to_fine_id=0, pts_coarse=10, spn_coarse=0x00020000
// fine 0: pts_fine=3, spn_fine=0x100
let cpi = build_cpi(0x1011, &[(0, 10, 0x00020000)], &[(3, 0x100)]);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let resolved = clip.resolved_ep_map();
assert_eq!(resolved.len(), 1);
let expected_pts = (10u64 << 19) + (3u64 << 8);
let expected_spn = (0x00020000u32 & 0xFFFE_0000) | 0x100;
assert_eq!(resolved[0].0, expected_pts);
assert_eq!(resolved[0].1, expected_spn);
}
/// get_extents converts an in/out PTS range to a single sector Extent.
/// SPN→byte = spn×192, byte→sector = /2048 (start floored, end ceiled),
/// relative to m2ts file start. Verify the math for a known fixture.
#[test]
fn get_extents_spn_to_sector_math() {
// Two EP points: PTS p0 → SPN 0, PTS p1 → SPN big_spn.
// full_spn ORs (spn_coarse & 0xFFFE0000) with spn_fine, so the SPN
// must be coarse-aligned (low 17 bits clear) to survive intact.
// 0x20000 (131072) is the smallest non-zero coarse-aligned SPN.
let big_spn: u32 = 0x20000;
let cpi = build_cpi(0x1011, &[(0, 0, 0), (1, 100, big_spn)], &[(0, 0), (0, 0)]);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let p0 = 0u64; // PTS of first EP
let p1 = 100u64 << 19; // PTS of second EP
let extents = clip.get_extents(p0, p1);
assert_eq!(extents.len(), 1);
// Mirror production: SPN→byte ×packet, byte→sector with start FLOORed
// and end CEILed (same constants as get_extents).
let start_spn: u64 = 0;
let end_spn = big_spn as u64;
let start_byte = start_spn * BD_SOURCE_PACKET_BYTES as u64;
let end_byte = end_spn * BD_SOURCE_PACKET_BYTES as u64;
let start_sector = (start_byte / SECTOR_BYTES_U64) as u32;
let end_sector = end_byte.div_ceil(SECTOR_BYTES_U64) as u32;
assert_eq!(extents[0].start_lba, start_sector);
assert_eq!(extents[0].sector_count, end_sector - start_sector);
// Concretely: 0x20000 × 192 / 2048 = 12288 sectors.
assert_eq!(extents[0].sector_count, 12288);
}
/// get_extents returns an empty Vec when the EP map is empty (no CPI),
/// since there is no SPN to resolve. Documented early return.
#[test]
fn get_extents_empty_when_no_ep_map() {
let data = build_clpi(1000, None);
let clip = parse(&data).expect("should parse");
assert!(clip.get_extents(0, 1_000_000).is_empty());
}
/// get_extents returns empty when end_spn <= start_spn (degenerate or
/// inverted range). Source has an explicit `if end_spn <= start_spn`
/// guard. Use in_time == out_time on a single-point map.
#[test]
fn get_extents_empty_on_degenerate_range() {
let cpi = build_cpi(0x1011, &[(0, 50, 0x1000)], &[(0, 0)]);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let p = 50u64 << 19;
// in == out → start_spn == end_spn → empty.
assert!(clip.get_extents(p, p).is_empty());
}
/// full_spn masks the LOW 17 bits of spn_coarse (& 0xFFFE0000) before
/// OR-ing fine. A spn_coarse with low bits set must have them cleared,
/// then replaced by spn_fine. Independent of parse, exercises the
/// reconstruction directly with a hostile low-bit pattern.
#[test]
fn full_spn_clears_coarse_low_17_bits() {
let coarse = EpCoarse {
ref_to_fine_id: 0,
pts_coarse: 0,
spn_coarse: 0x0006_FFFF, // low 17 bits all set
};
let fine = EpFine {
pts_fine: 0,
spn_fine: 0x5,
};
// 0x0006_FFFF & 0xFFFE_0000 = 0x0006_0000; | 0x5 = 0x0006_0005.
assert_eq!(ClipInfo::full_spn(&coarse, &fine), 0x0006_0005);
}
/// CPI guard: cpi_length < 4 short-circuits to empty maps (the length
/// field counts bytes after itself, and the EP map needs ≥4). A
/// cpi_length of 0/1/2/3 must yield empty EP maps, not panic.
#[test]
fn cpi_length_below_4_yields_empty() {
for bad_len in 0u32..4 {
let mut cpi = Vec::new();
cpi.extend_from_slice(&bad_len.to_be_bytes());
cpi.extend_from_slice(&[0u8; 20]); // padding so the slice exists
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
assert!(clip.ep_coarse.is_empty(), "len={bad_len}");
assert!(clip.ep_fine.is_empty(), "len={bad_len}");
}
}
/// ep_map_offset that points past the EP map (`ep_map_offset + 4 >
/// ep_map.len()`) → empty maps (bounds guard), not panic. Patch the
/// EP_map_start field to a huge value.
#[test]
fn ep_map_offset_out_of_bounds_yields_empty() {
let cpi = build_cpi(0x1011, &[(0, 10, 0x20000)], &[(5, 100)]);
let mut data = build_clpi(1000, Some(&cpi));
// EP_map_start is the low 32 bits of the 80-bit stream header at
// ep_map[4..14]. In the file: header(60) + cpi_length(4) +
// reserved(2) + ep_map reserved(1) + num_streams(1) + pid(2) = 70,
// then 10 header bytes [70..80]; EP_map_start is the last 4 [76..80].
let off = 60 + 4 + 2 + 1 + 1 + 2 + 6; // = 76
data[off..off + 4].copy_from_slice(&0xFFFF_FFFFu32.to_be_bytes());
let clip = parse(&data).expect("should not panic");
assert!(clip.ep_coarse.is_empty());
assert!(clip.ep_fine.is_empty());
}
/// num_coarse declares more entries than the CPI section holds. The
/// loop must stop at `off + 8 > coarse_data.len()` (break), not read
/// out of bounds. Patch num_coarse to a large value while supplying 1
/// coarse entry's worth of bytes.
#[test]
fn coarse_count_overshoot_truncates_safely() {
let cpi = build_cpi(0x1011, &[(0, 10, 0x20000)], &[(5, 100)]);
let mut data = build_clpi(1000, Some(&cpi));
// num_coarse is bits 14..30 of the 80-bit header. Rather than
// bit-surgery, rebuild with a hand-set num_coarse=255 but only 1
// coarse entry of bytes — done below directly.
let _ = &mut data;
let num_coarse_decl: u32 = 255;
let num_fine: u32 = 1;
let ep_map_start: u32 = 14;
let ep_stream_type: u32 = 1;
let packed: u128 = ((ep_stream_type as u128) << 66)
| ((num_coarse_decl as u128) << 50)
| ((num_fine as u128) << 32)
| (ep_map_start as u128);
let packed_bytes = packed.to_be_bytes();
let stream_header_bits = &packed_bytes[6..16];
// stream EP data: fine_start points past the 1 coarse entry.
let fine_start: u32 = 4 + 8; // 4-byte header + 1 coarse entry x 8 bytes
let mut stream_ep = Vec::new();
stream_ep.extend_from_slice(&fine_start.to_be_bytes());
// exactly ONE coarse entry (8 bytes), though header claims 255.
stream_ep.extend_from_slice(&10u32.to_be_bytes());
stream_ep.extend_from_slice(&0x20000u32.to_be_bytes());
// one fine entry (4 bytes)
stream_ep.extend_from_slice(&(((5u32 & 0x7FF) << 17) | 100).to_be_bytes());
let mut ep_map = Vec::new();
ep_map.push(0);
ep_map.push(1);
ep_map.extend_from_slice(&0x1011u16.to_be_bytes());
ep_map.extend_from_slice(stream_header_bits);
ep_map.extend_from_slice(&stream_ep);
let mut cpi2 = Vec::new();
cpi2.extend_from_slice(&((2 + ep_map.len()) as u32).to_be_bytes());
cpi2.extend_from_slice(&[0u8; 2]);
cpi2.extend_from_slice(&ep_map);
let data2 = build_clpi(1000, Some(&cpi2));
let clip = parse(&data2).expect("should not panic on coarse overshoot");
// Only the 1 real coarse entry was readable.
assert_eq!(clip.ep_coarse.len(), 1);
assert_eq!(clip.ep_coarse[0].pts_coarse, 10);
}
/// resolved_ep_map: the LAST coarse group's fine range extends to
/// ep_fine.len() (no "next coarse" bound). Verify all trailing fine
/// entries are assigned to the final coarse group.
#[test]
fn resolved_ep_map_last_group_to_end() {
// coarse 0 ref_to_fine_id=0, coarse 1 ref_to_fine_id=1.
// 3 fine entries: fine 0 → coarse 0; fine 1,2 → coarse 1.
let cpi = build_cpi(
0x1011,
&[(0, 0, 0), (1, 100, 0)],
&[(0, 10), (0, 20), (0, 30)],
);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let resolved = clip.resolved_ep_map();
// All 3 fine entries resolved (last group picks up fine 1 and 2).
assert_eq!(resolved.len(), 3);
}
}
+125
View File
@@ -0,0 +1,125 @@
//! Physical media constants — the single source of truth.
//!
//! Naming convention: a constant is prefixed by the **narrowest scope where it
//! is valid**. A value common to all optical media carries no prefix; a value
//! specific to a container/format/disc-type is prefixed by it
//! (`TS_`, `BD_`, …). Define each physical quantity here exactly once and import
//! it — never re-declare a bare literal or a local copy.
/// Bytes per logical sector on every optical medium freemkv reads
/// (Blu-ray, DVD-Video, CD-ROM Mode 1). Universal — hence unprefixed.
///
/// `usize` because its dominant use is buffer sizing and slice indexing, where
/// Rust *requires* `usize` (`vec![0u8; SECTOR_BYTES]`, `buf.len() < SECTOR_BYTES`).
/// For byte-offset / capacity arithmetic — which is `u64` because a disc can
/// exceed 4 GiB — use [`SECTOR_BYTES_U64`] instead of casting at each site.
pub const SECTOR_BYTES: usize = 2048;
/// [`SECTOR_BYTES`] as `u64`, for byte-offset and capacity arithmetic. The
/// single `usize → u64` boundary cast lives here, once, so offset math across
/// the workspace reads as `sectors * SECTOR_BYTES_U64` with no per-site cast.
pub const SECTOR_BYTES_U64: u64 = SECTOR_BYTES as u64;
/// Milliseconds per second. For turning a byte count ÷ bytes-per-second into a
/// movie-time figure (`bytes / bps * MILLIS_PER_SEC`) without a bare `1000.0`.
pub const MILLIS_PER_SEC: f64 = 1_000.0;
/// Bytes per MPEG-2 transport-stream packet. Common to all MPEG-TS, not just
/// Blu-ray — prefixed by the format, not a disc type.
pub const TS_PACKET_BYTES: usize = 188;
/// Bytes in an MPEG-2 transport-stream packet header: sync byte, the
/// flags/PID word, and the adaptation/continuity byte.
pub const TS_HEADER_BYTES: usize = 4;
/// Bytes in the arrival-timestamp prefix a Blu-ray M2TS prepends to each TS
/// packet to form a source packet. Same width as a TS header but a distinct
/// quantity ([`TS_HEADER_BYTES`]) — do not conflate.
pub const BD_TIMESTAMP_PREFIX_BYTES: usize = 4;
/// Bytes of payload in an MPEG-2 transport-stream packet:
/// [`TS_PACKET_BYTES`] minus the [`TS_HEADER_BYTES`] header.
pub const TS_PAYLOAD_BYTES: usize = TS_PACKET_BYTES - TS_HEADER_BYTES;
/// Bytes per Blu-ray M2TS *source packet*: a TS packet ([`TS_PACKET_BYTES`])
/// prefixed with the [`BD_TIMESTAMP_PREFIX_BYTES`] arrival-timestamp header.
/// A BDAV/M2TS construct only — DVD VOBs have no source packets — hence `BD_`.
pub const BD_SOURCE_PACKET_BYTES: usize = TS_PACKET_BYTES + BD_TIMESTAMP_PREFIX_BYTES;
/// Elementary-stream coding-type codes — the single source of truth for the
/// byte that identifies a stream's codec.
///
/// This is one registry used in two places that share the same value space:
/// the MPEG-TS PMT `stream_type` (ISO/IEC 13818-1 Table 2-34) and the Blu-ray
/// STN/CLPI `stream_coding_type` (BD-ROM Part 3). The standardized video codes
/// (`0x02`, `0x1B`, `0x24`) are ISO assignments (ISO/IEC 13818-1 Table 2-34);
/// `0xEA` (VC-1) is a BD-ROM convention in the ISO user-private range. The
/// `0x80..=0xA2` audio/graphics codes also sit in the user-private range and follow the
/// Blu-ray Disc Association / ATSC A/52 convention. Because every consumer
/// reads or writes this single byte, the family is unprefixed — the scope is
/// "any elementary stream freemkv parses or muxes".
///
/// Each constant is `u8`: the spec defines an 8-bit field and the code compares
/// it directly against a byte read from the buffer, so no casts are needed.
pub mod coding_type {
/// MPEG-2 video (ISO/IEC 13818-1 Table 2-34).
pub const MPEG2_VIDEO: u8 = 0x02;
/// H.264 / AVC video (ISO/IEC 13818-1 Table 2-34).
pub const H264: u8 = 0x1B;
/// HEVC / H.265 video (ISO/IEC 13818-1 Table 2-34, 2015 amendment).
pub const HEVC: u8 = 0x24;
/// SMPTE VC-1 video (BD-ROM convention, ISO user-private range).
pub const VC1: u8 = 0xEA;
/// LPCM audio (BD-ROM convention).
pub const LPCM: u8 = 0x80;
/// Dolby Digital (AC-3) audio (BD-ROM / ATSC A/52 convention).
pub const AC3: u8 = 0x81;
/// DTS audio (BD-ROM convention).
pub const DTS: u8 = 0x82;
/// Dolby TrueHD audio (BD-ROM convention).
pub const TRUEHD: u8 = 0x83;
/// Dolby Digital Plus (E-AC-3 / AC-3+) audio (BD-ROM convention).
pub const AC3_PLUS: u8 = 0x84;
/// DTS-HD High Resolution audio (BD-ROM Part 3-1).
pub const DTS_HD_HR: u8 = 0x85;
/// DTS-HD Master Audio (lossless) (BD-ROM Part 3-1).
pub const DTS_HD_MA: u8 = 0x86;
/// Presentation Graphics — PG subtitle stream (BD-ROM HDMV).
pub const PG: u8 = 0x90;
/// Interactive Graphics — IG / BD-J menu overlay, NOT a subtitle (BD-ROM HDMV).
pub const IG: u8 = 0x91;
/// Text subtitle stream (BD-ROM HDMV).
pub const TEXT_SUBTITLE: u8 = 0x92;
/// Secondary Dolby Digital Plus audio (BD-ROM convention).
pub const AC3_PLUS_SECONDARY: u8 = 0xA1;
/// Secondary DTS-HD audio (lossless MA, not lossy HR) (BD-ROM convention).
pub const DTS_HD_SECONDARY: u8 = 0xA2;
}
/// MPEG PES `stream_id` codes — the byte after the `00 00 01` start-code prefix
/// that identifies an elementary stream's role in a PES packet (ISO/IEC
/// 13818-1 Table 2-22). Shared by the program-stream demuxer and the TS/M2TS
/// muxers, so defined here once. Each is `u8` (matches the byte on the wire).
pub mod pes_stream_id {
/// Video stream (`110x xxxx`; freemkv emits the base id `0xE0`).
pub const VIDEO: u8 = 0xE0;
/// private_stream_1 — AC-3 / DTS / LPCM / PGS subtitle payloads.
pub const PRIVATE_STREAM_1: u8 = 0xBD;
/// padding_stream — stuffing bytes only, no payload to demux.
pub const PADDING_STREAM: u8 = 0xBE;
/// private_stream_2 — DVD navigation (PCI/DSI); carries no muxable ES.
pub const PRIVATE_STREAM_2: u8 = 0xBF;
/// Highest video stream_id — the `110x xxxx` video range tops out at 0xEF.
pub const VIDEO_MAX: u8 = 0xEF;
/// Inclusive range of every PES `stream_id` that carries demuxable payload:
/// [`PRIVATE_STREAM_1`] (0xBD) through [`VIDEO_MAX`] (0xEF) — i.e. private
/// stream 1/2, padding, MPEG audio (0xC0-0xDF) and video (0xE0-0xEF). The
/// pack (0xBA), system-header (0xBB) and program-end (0xB9) codes sit below
/// this range and are deliberately excluded: they're structural, not ES.
pub const PAYLOAD_RANGE: core::ops::RangeInclusive<u8> = PRIVATE_STREAM_1..=VIDEO_MAX;
}
-639
View File
@@ -1,639 +0,0 @@
//! CSS drive authentication — full key hierarchy.
//!
//! Protocol:
//! 1. Bus authentication (challenge-response) → bus key
//! 2. Read disc key block (READ DVD STRUCTURE) → XOR with bus key → decrypt with player keys → disc key
//! 3. Read title key (REPORT KEY format 0x04) → XOR with bus key → decrypt with disc key → title key
use crate::drive::Drive;
use crate::error::{Error, Result};
// ── Built-in public DVD CSS player keys ────────────────────────────────────
//
// These 31 5-byte player keys are long-public CSS inputs. With them
// compiled in, DVD ripping works with no external key file required.
const PLAYER_KEYS: [[u8; 5]; 31] = [
[0x01, 0xaf, 0xe3, 0x12, 0x80],
[0x12, 0x11, 0xca, 0x04, 0x3b],
[0x14, 0x0c, 0x9e, 0xd0, 0x09],
[0x14, 0x71, 0x35, 0xba, 0xe2],
[0x1a, 0xa4, 0x33, 0x21, 0xa6],
[0x26, 0xec, 0xc4, 0xa7, 0x4e],
[0x2c, 0xb2, 0xc1, 0x09, 0xee],
[0x2f, 0x25, 0x9e, 0x96, 0xdd],
[0x33, 0x2f, 0x49, 0x6c, 0xe0],
[0x35, 0x5b, 0xc1, 0x31, 0x0f],
[0x36, 0x67, 0xb2, 0xe3, 0x85],
[0x39, 0x3d, 0xf1, 0xf1, 0xbd],
[0x3b, 0x31, 0x34, 0x0d, 0x91],
[0x45, 0xed, 0x28, 0xeb, 0xd3],
[0x48, 0xb7, 0x6c, 0xce, 0x69],
[0x4b, 0x65, 0x0d, 0xc1, 0xee],
[0x4c, 0xbb, 0xf5, 0x5b, 0x23],
[0x51, 0x67, 0x67, 0xc5, 0xe0],
[0x53, 0x94, 0xe1, 0x75, 0xbf],
[0x57, 0x2c, 0x8b, 0x31, 0xae],
[0x63, 0xdb, 0x4c, 0x5b, 0x4a],
[0x7b, 0x1e, 0x5e, 0x2b, 0x57],
[0x85, 0xf3, 0x85, 0xa0, 0xe0],
[0xab, 0x1e, 0xe7, 0x7b, 0x72],
[0xab, 0x36, 0xe3, 0xeb, 0x76],
[0xb1, 0xb8, 0xf9, 0x38, 0x03],
[0xb8, 0x5d, 0xd8, 0x53, 0xbd],
[0xbf, 0x92, 0xc3, 0xb0, 0xe2],
[0xcf, 0x1a, 0xb2, 0xf8, 0x0a],
[0xec, 0xa0, 0xcf, 0xb3, 0xff],
[0xfc, 0x95, 0xa9, 0x87, 0x35],
];
// ── CryptKey tables ───────────────────────────────────────────────────────
const CRYPT_TAB0: [u8; 256] = [
0xB7, 0xF4, 0x82, 0x57, 0xDA, 0x4D, 0xDB, 0xE2, 0x2F, 0x52, 0x1A, 0xA8, 0x68, 0x5A, 0x8A, 0xFF,
0xFB, 0x0E, 0x6D, 0x35, 0xF7, 0x5C, 0x76, 0x12, 0xCE, 0x25, 0x79, 0x29, 0x39, 0x62, 0x08, 0x24,
0xA5, 0x85, 0x7B, 0x56, 0x01, 0x23, 0x68, 0xCF, 0x0A, 0xE2, 0x5A, 0xED, 0x3D, 0x59, 0xB0, 0xA9,
0xB0, 0x2C, 0xF2, 0xB8, 0xEF, 0x32, 0xA9, 0x40, 0x80, 0x71, 0xAF, 0x1E, 0xDE, 0x8F, 0x58, 0x88,
0xB8, 0x3A, 0xD0, 0xFC, 0xC4, 0x1E, 0xB5, 0xA0, 0xBB, 0x3B, 0x0F, 0x01, 0x7E, 0x1F, 0x9F, 0xD9,
0xAA, 0xB8, 0x3D, 0x9D, 0x74, 0x1E, 0x25, 0xDB, 0x37, 0x56, 0x8F, 0x16, 0xBA, 0x49, 0x2B, 0xAC,
0xD0, 0xBD, 0x95, 0x20, 0xBE, 0x7A, 0x28, 0xD0, 0x51, 0x64, 0x63, 0x1C, 0x7F, 0x66, 0x10, 0xBB,
0xC4, 0x56, 0x1A, 0x04, 0x6E, 0x0A, 0xEC, 0x9C, 0xD6, 0xE8, 0x9A, 0x7A, 0xCF, 0x8C, 0xDB, 0xB1,
0xEF, 0x71, 0xDE, 0x31, 0xFF, 0x54, 0x3E, 0x5E, 0x07, 0x69, 0x96, 0xB0, 0xCF, 0xDD, 0x9E, 0x47,
0xC7, 0x96, 0x8F, 0xE4, 0x2B, 0x59, 0xC6, 0xEE, 0xB9, 0x86, 0x9A, 0x64, 0x84, 0x72, 0xE2, 0x5B,
0xA2, 0x96, 0x58, 0x99, 0x50, 0x03, 0xF5, 0x38, 0x4D, 0x02, 0x7D, 0xE7, 0x7D, 0x75, 0xA7, 0xB8,
0x67, 0x87, 0x84, 0x3F, 0x1D, 0x11, 0xE5, 0xFC, 0x1E, 0xD3, 0x83, 0x16, 0xA5, 0x29, 0xF6, 0xC7,
0x15, 0x61, 0x29, 0x1A, 0x43, 0x4F, 0x9B, 0xAF, 0xC5, 0x87, 0x34, 0x6C, 0x0F, 0x3B, 0xA8, 0x1D,
0x45, 0x58, 0x25, 0xDC, 0xA8, 0xA3, 0x3B, 0xD1, 0x79, 0x1B, 0x48, 0xF2, 0xE9, 0x93, 0x1F, 0xFC,
0xDB, 0x2A, 0x90, 0xA9, 0x8A, 0x3D, 0x39, 0x18, 0xA3, 0x8E, 0x58, 0x6C, 0xE0, 0x12, 0xBB, 0x25,
0xCD, 0x71, 0x22, 0xA2, 0x64, 0xC6, 0xE7, 0xFB, 0xAD, 0x94, 0x77, 0x04, 0x9A, 0x39, 0xCF, 0x7C,
];
const CRYPT_TAB1: [u8; 256] = [
0x8C, 0x47, 0xB0, 0xE1, 0xEB, 0xFC, 0xEB, 0x56, 0x10, 0xE5, 0x2C, 0x1A, 0x5D, 0xEF, 0xBE, 0x4F,
0x08, 0x75, 0x97, 0x4B, 0x0E, 0x25, 0x8E, 0x6E, 0x39, 0x5A, 0x87, 0x53, 0xC4, 0x1F, 0xF4, 0x5C,
0x4E, 0xE6, 0x99, 0x30, 0xE0, 0x42, 0x88, 0xAB, 0xE5, 0x85, 0xBC, 0x8F, 0xD8, 0x3C, 0x54, 0xC9,
0x53, 0x47, 0x18, 0xD6, 0x06, 0x5B, 0x41, 0x2C, 0x67, 0x1E, 0x41, 0x74, 0x33, 0xE2, 0xB4, 0xE0,
0x23, 0x29, 0x42, 0xEA, 0x55, 0x0F, 0x25, 0xB4, 0x24, 0x2C, 0x99, 0x13, 0xEB, 0x0A, 0x0B, 0xC9,
0xF9, 0x63, 0x67, 0x43, 0x2D, 0xC7, 0x7D, 0x07, 0x60, 0x89, 0xD1, 0xCC, 0xE7, 0x94, 0x77, 0x74,
0x9B, 0x7E, 0xD7, 0xE6, 0xFF, 0xBB, 0x68, 0x14, 0x1E, 0xA3, 0x25, 0xDE, 0x3A, 0xA3, 0x54, 0x7B,
0x87, 0x9D, 0x50, 0xCA, 0x27, 0xC3, 0xA4, 0x50, 0x91, 0x27, 0xD4, 0xB0, 0x82, 0x41, 0x97, 0x79,
0x94, 0x82, 0xAC, 0xC7, 0x8E, 0xA5, 0x4E, 0xAA, 0x78, 0x9E, 0xE0, 0x42, 0xBA, 0x28, 0xEA, 0xB7,
0x74, 0xAD, 0x35, 0xDA, 0x92, 0x60, 0x7E, 0xD2, 0x0E, 0xB9, 0x24, 0x5E, 0x39, 0x4F, 0x5E, 0x63,
0x09, 0xB5, 0xFA, 0xBF, 0xF1, 0x22, 0x55, 0x1C, 0xE2, 0x25, 0xDB, 0xC5, 0xD8, 0x50, 0x03, 0x98,
0xC4, 0xAC, 0x2E, 0x11, 0xB4, 0x38, 0x4D, 0xD0, 0xB9, 0xFC, 0x2D, 0x3C, 0x08, 0x04, 0x5A, 0xEF,
0xCE, 0x32, 0xFB, 0x4C, 0x92, 0x1E, 0x4B, 0xFB, 0x1A, 0xD0, 0xE2, 0x3E, 0xDA, 0x6E, 0x7C, 0x4D,
0x56, 0xC3, 0x3F, 0x42, 0xB1, 0x3A, 0x23, 0x4D, 0x6E, 0x84, 0x56, 0x68, 0xF4, 0x0E, 0x03, 0x64,
0xD0, 0xA9, 0x92, 0x2F, 0x8B, 0xBC, 0x39, 0x9C, 0xAC, 0x09, 0x5E, 0xEE, 0xE5, 0x97, 0xBF, 0xA5,
0xCE, 0xFA, 0x28, 0x2C, 0x6D, 0x4F, 0xEF, 0x77, 0xAA, 0x1B, 0x79, 0x8E, 0x97, 0xB4, 0xC3, 0xF4,
];
const CRYPT_TAB2: [u8; 256] = [
0xB7, 0x75, 0x81, 0xD5, 0xDC, 0xCA, 0xDE, 0x66, 0x23, 0xDF, 0x15, 0x26, 0x62, 0xD1, 0x83, 0x77,
0xE3, 0x97, 0x76, 0xAF, 0xE9, 0xC3, 0x6B, 0x8E, 0xDA, 0xB0, 0x6E, 0xBF, 0x2B, 0xF1, 0x19, 0xB4,
0x95, 0x34, 0x48, 0xE4, 0x37, 0x94, 0x5D, 0x7B, 0x36, 0x5F, 0x65, 0x53, 0x07, 0xE2, 0x89, 0x11,
0x98, 0x85, 0xD9, 0x12, 0xC1, 0x9D, 0x84, 0xEC, 0xA4, 0xD4, 0x88, 0xB8, 0xFC, 0x2C, 0x79, 0x28,
0xD8, 0xDB, 0xB3, 0x1E, 0xA2, 0xF9, 0xD0, 0x44, 0xD7, 0xD6, 0x60, 0xEF, 0x14, 0xF4, 0xF6, 0x31,
0xD2, 0x41, 0x46, 0x67, 0x0A, 0xE1, 0x58, 0x27, 0x43, 0xA3, 0xF8, 0xE0, 0xC8, 0xBA, 0x5A, 0x5C,
0x80, 0x6C, 0xC6, 0xF2, 0xE8, 0xAD, 0x7D, 0x04, 0x0D, 0xB9, 0x3C, 0xC2, 0x25, 0xBD, 0x49, 0x63,
0x8C, 0x9F, 0x51, 0xCE, 0x20, 0xC5, 0xA1, 0x50, 0x92, 0x2D, 0xDD, 0xBC, 0x8D, 0x4F, 0x9A, 0x71,
0x2F, 0x30, 0x1D, 0x73, 0x39, 0x13, 0xFB, 0x1A, 0xCB, 0x24, 0x59, 0xFE, 0x05, 0x96, 0x57, 0x0F,
0x1F, 0xCF, 0x54, 0xBE, 0xF5, 0x06, 0x1B, 0xB2, 0x6D, 0xD3, 0x4D, 0x32, 0x56, 0x21, 0x33, 0x0B,
0x52, 0xE7, 0xAB, 0xEB, 0xA6, 0x74, 0x00, 0x4C, 0xB1, 0x7F, 0x82, 0x99, 0x87, 0x0E, 0x5E, 0xC0,
0x8F, 0xEE, 0x6F, 0x55, 0xF3, 0x7E, 0x08, 0x90, 0xFA, 0xB6, 0x64, 0x70, 0x47, 0x4A, 0x17, 0xA7,
0xB5, 0x40, 0x8A, 0x38, 0xE5, 0x68, 0x3E, 0x8B, 0x69, 0xAA, 0x9B, 0x42, 0xA5, 0x10, 0x01, 0x35,
0xFD, 0x61, 0x9E, 0xE6, 0x16, 0x9C, 0x86, 0xED, 0xCD, 0x2E, 0xFF, 0xC4, 0x5B, 0xA0, 0xAE, 0xCC,
0x4B, 0x3B, 0x03, 0xBB, 0x1C, 0x2A, 0xAC, 0x0C, 0x3F, 0x93, 0xC7, 0x72, 0x7A, 0x09, 0x22, 0x3D,
0x45, 0x78, 0xA9, 0xA8, 0xEA, 0xC9, 0x6A, 0xF7, 0x29, 0x91, 0xF0, 0x02, 0x18, 0x3A, 0x4E, 0x7C,
];
const CRYPT_TAB3: [u8; 288] = [
0x73, 0x51, 0x95, 0xE1, 0x12, 0xE4, 0xC0, 0x58, 0xEE, 0xF2, 0x08, 0x1B, 0xA9, 0xFA, 0x98, 0x4C,
0xA7, 0x33, 0xE2, 0x1B, 0xA7, 0x6D, 0xF5, 0x30, 0x97, 0x1D, 0xF3, 0x02, 0x60, 0x5A, 0x82, 0x0F,
0x91, 0xD0, 0x9C, 0x10, 0x39, 0x7A, 0x83, 0x85, 0x3B, 0xB2, 0xB8, 0xAE, 0x0C, 0x09, 0x52, 0xEA,
0x1C, 0xE1, 0x8D, 0x66, 0x4F, 0xF3, 0xDA, 0x92, 0x29, 0xB9, 0xD5, 0xC5, 0x77, 0x47, 0x22, 0x53,
0x14, 0xF7, 0xAF, 0x22, 0x64, 0xDF, 0xC6, 0x72, 0x12, 0xF3, 0x75, 0xDA, 0xD7, 0xD7, 0xE5, 0x02,
0x9E, 0xED, 0xDA, 0xDB, 0x4C, 0x47, 0xCE, 0x91, 0x06, 0x06, 0x6D, 0x55, 0x8B, 0x19, 0xC9, 0xEF,
0x8C, 0x80, 0x1A, 0x0E, 0xEE, 0x4B, 0xAB, 0xF2, 0x08, 0x5C, 0xE9, 0x37, 0x26, 0x5E, 0x9A, 0x90,
0x00, 0xF3, 0x0D, 0xB2, 0xA6, 0xA3, 0xF7, 0x26, 0x17, 0x48, 0x88, 0xC9, 0x0E, 0x2C, 0xC9, 0x02,
0xE7, 0x18, 0x05, 0x4B, 0xF3, 0x39, 0xE1, 0x20, 0x02, 0x0D, 0x40, 0xC7, 0xCA, 0xB9, 0x48, 0x30,
0x57, 0x67, 0xCC, 0x06, 0xBF, 0xAC, 0x81, 0x08, 0x24, 0x7A, 0xD4, 0x8B, 0x19, 0x8E, 0xAC, 0xB4,
0x5A, 0x0F, 0x73, 0x13, 0xAC, 0x9E, 0xDA, 0xB6, 0xB8, 0x96, 0x5B, 0x60, 0x88, 0xE1, 0x81, 0x3F,
0x07, 0x86, 0x37, 0x2D, 0x79, 0x14, 0x52, 0xEA, 0x73, 0xDF, 0x3D, 0x09, 0xC8, 0x25, 0x48, 0xD8,
0x75, 0x60, 0x9A, 0x08, 0x27, 0x4A, 0x2C, 0xB9, 0xA8, 0x8B, 0x8A, 0x73, 0x62, 0x37, 0x16, 0x02,
0xBD, 0xC1, 0x0E, 0x56, 0x54, 0x3E, 0x14, 0x5F, 0x8C, 0x8F, 0x6E, 0x75, 0x1C, 0x07, 0x39, 0x7B,
0x4B, 0xDB, 0xD3, 0x4B, 0x1E, 0xC8, 0x7E, 0xFE, 0x3E, 0x72, 0x16, 0x83, 0x7D, 0xEE, 0xF5, 0xCA,
0xC5, 0x18, 0xF9, 0xD8, 0x68, 0xAB, 0x38, 0x85, 0xA8, 0xF0, 0xA1, 0x73, 0x9F, 0x5D, 0x19, 0x0B,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x33, 0x72, 0x39, 0x25, 0x67, 0x26, 0x6D, 0x71,
0x36, 0x77, 0x3C, 0x20, 0x62, 0x23, 0x68, 0x74, 0xC3, 0x82, 0xC9, 0x15, 0x57, 0x16, 0x5D, 0x81,
];
const VARIANTS: [u8; 32] = [
0xB7, 0x74, 0x85, 0xD0, 0xCC, 0xDB, 0xCA, 0x73, 0x03, 0xFE, 0x31, 0x03, 0x52, 0xE0, 0xB7, 0x42,
0x63, 0x16, 0xF2, 0x2A, 0x79, 0x52, 0xFF, 0x1B, 0x7A, 0x11, 0xCA, 0x1A, 0x9B, 0x40, 0xAD, 0x01,
];
const SECRET: [u8; 5] = [0x55, 0xD6, 0xC4, 0xC5, 0x28];
const PERM_CHALLENGE: [[usize; 10]; 3] = [
[1, 3, 0, 7, 5, 2, 9, 6, 4, 8],
[6, 1, 9, 3, 8, 5, 7, 4, 0, 2],
[4, 0, 3, 5, 7, 2, 8, 6, 1, 9],
];
const PERM_VARIANT: [[u8; 32]; 2] = [
[
0x0A, 0x08, 0x0E, 0x0C, 0x0B, 0x09, 0x0F, 0x0D, 0x1A, 0x18, 0x1E, 0x1C, 0x1B, 0x19, 0x1F,
0x1D, 0x02, 0x00, 0x06, 0x04, 0x03, 0x01, 0x07, 0x05, 0x12, 0x10, 0x16, 0x14, 0x13, 0x11,
0x17, 0x15,
],
[
0x12, 0x1A, 0x16, 0x1E, 0x02, 0x0A, 0x06, 0x0E, 0x10, 0x18, 0x14, 0x1C, 0x00, 0x08, 0x04,
0x0C, 0x13, 0x1B, 0x17, 0x1F, 0x03, 0x0B, 0x07, 0x0F, 0x11, 0x19, 0x15, 0x1D, 0x01, 0x09,
0x05, 0x0D,
],
];
// ── SCSI constants ────────────────────────────────────────────────────────
const SCSI_READ_DVD_STRUCTURE: u8 = 0xAD;
// ── Public API ────────────────────────────────────────────────────────────
/// Perform CSS bus authentication only.
pub fn authenticate(drive: &mut Drive) -> Result<()> {
let (_, _) = bus_auth(drive)?;
Ok(())
}
/// Full CSS key extraction: bus auth → disc key → title key.
pub fn authenticate_and_read_title_key(drive: &mut Drive, lba: u32) -> Result<[u8; 5]> {
// Session 1: bus auth → disc key (AGID consumed by READ_DVD_STRUCTURE)
let (agid, bus_key) = bus_auth(drive)?;
let disc_key = read_disc_key(drive, agid, &bus_key)?;
// Session 2: fresh bus auth → title key (needs separate AGID)
let (agid2, bus_key2) = bus_auth(drive)?;
let encrypted_title = read_raw_title_key(drive, agid2, lba)?;
// Decrypt title key: XOR with bus key, then decrypt with disc key
let mut title_key = [0u8; 5];
for i in 0..5 {
title_key[i] = encrypted_title[i] ^ bus_key2[i];
}
if title_key == [0u8; 5] {
return Ok(title_key);
}
let title_key = super::lfsr::decrypt_key(0xFF, &disc_key, &title_key);
Ok(title_key)
}
// ── Step 1: Bus Authentication ────────────────────────────────────────────
fn bus_auth(drive: &mut Drive) -> Result<(u8, [u8; 5])> {
let scsi = drive.scsi_mut();
// Invalidate all AGIDs via REPORT KEY format 0x3F
for agid in 0..4u8 {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
// alloc_len = 0 (no data transfer)
cdb[10] = (agid << 6) | 0x3F;
let mut buf = [0u8; 8];
let _ = scsi.execute(
&cdb,
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
);
}
// Allocate AGID
let mut buf = [0u8; 8];
scsi.execute(
&report_key_cdb(0, 0x00, 8),
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
let agid = (buf[7] >> 6) & 0x03;
// Host sends challenge
let host_challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
let mut hc_buf = [0u8; 16];
hc_buf[0] = 0x00;
hc_buf[1] = 0x0E;
for i in 0..10 {
hc_buf[4 + i] = host_challenge[9 - i];
}
scsi.execute(
&send_key_cdb(agid, 0x01, 16),
crate::scsi::DataDirection::ToDevice,
&mut hc_buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
// Get Key1 from drive
let mut dk_buf = [0u8; 12];
scsi.execute(
&report_key_cdb(agid, 0x02, 12),
crate::scsi::DataDirection::FromDevice,
&mut dk_buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
let mut key1 = [0u8; 5];
for i in 0..5 {
key1[i] = dk_buf[4 + (4 - i)];
}
// Brute-force variant (0-31)
let mut variant: Option<u8> = None;
for v in 0..32u8 {
if crypt_key(0, v, &host_challenge) == key1 {
variant = Some(v);
break;
}
}
let variant = variant.ok_or(Error::CssAuthFailed)?;
// Get drive challenge
let mut dc_buf = [0u8; 16];
scsi.execute(
&report_key_cdb(agid, 0x01, 16),
crate::scsi::DataDirection::FromDevice,
&mut dc_buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
let mut drive_challenge = [0u8; 10];
for i in 0..10 {
drive_challenge[i] = dc_buf[4 + (9 - i)];
}
// Compute Key2 and send it
let key2 = crypt_key(1, variant, &drive_challenge);
let mut hk_buf = [0u8; 12];
hk_buf[0] = 0x00;
hk_buf[1] = 0x0A;
for i in 0..5 {
hk_buf[4 + i] = key2[4 - i];
}
scsi.execute(
&send_key_cdb(agid, 0x03, 12),
crate::scsi::DataDirection::ToDevice,
&mut hk_buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
// Bus key = CryptKey(2, variant, key1 || key2)
let mut combined = [0u8; 10];
combined[..5].copy_from_slice(&key1);
combined[5..].copy_from_slice(&key2);
let bus_key = crypt_key(2, variant, &combined);
Ok((agid, bus_key))
}
// ── Step 2: Disc Key ──────────────────────────────────────────────────────
fn read_disc_key(drive: &mut Drive, agid: u8, bus_key: &[u8; 5]) -> Result<[u8; 5]> {
let scsi = drive.scsi_mut();
// READ DVD STRUCTURE, format 0x02 (disc key), 2048+4 bytes
let alloc_len: u16 = 2048 + 4;
let mut cdb = [0u8; 12];
cdb[0] = SCSI_READ_DVD_STRUCTURE;
// bytes 2-5: address = 0
cdb[6] = 0; // layer
cdb[7] = 0x02; // format = disc key
cdb[8] = (alloc_len >> 8) as u8;
cdb[9] = alloc_len as u8;
cdb[10] = agid << 6;
let mut buf = vec![0u8; alloc_len as usize];
let dvd_result = scsi.execute(
&cdb,
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
);
dvd_result.map_err(|_| Error::CssAuthFailed)?;
// Disc key block starts at offset 4 (skip 4-byte header)
let disc_key_block = &mut buf[4..4 + 2048];
// XOR with reversed bus key (per libdvdcss)
for (i, byte) in disc_key_block.iter_mut().enumerate() {
*byte ^= bus_key[4 - (i % 5)];
}
// Try each player key against each of 408 disc key entries.
// Each entry in the block is the disc key encrypted with a specific player key.
// We try all known player keys and verify by checking that two different
// entries produce the same disc key.
let mut candidates: Vec<([u8; 5], usize, usize)> = Vec::new(); // (disc_key, pk_idx, pos)
for (pk_idx, player_key) in PLAYER_KEYS.iter().enumerate() {
for pos in 0..408 {
let offset = pos * 5;
if offset + 5 > disc_key_block.len() {
break;
}
let mut enc = [0u8; 5];
enc.copy_from_slice(&disc_key_block[offset..offset + 5]);
let candidate = super::lfsr::decrypt_key(0x00, player_key, &enc);
// Check if any previous candidate matches (same disc key from different entry/pk)
for (prev, _, _) in &candidates {
if *prev == candidate {
return Ok(candidate);
}
}
candidates.push((candidate, pk_idx, pos));
}
}
Err(Error::CssAuthFailed)
}
// ── Step 3: Title Key ─────────────────────────────────────────────────────
/// Read the raw (bus-encrypted) title key bytes from the drive.
fn read_raw_title_key(drive: &mut Drive, agid: u8, lba: u32) -> Result<[u8; 5]> {
let scsi = drive.scsi_mut();
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
cdb[2] = (lba >> 24) as u8;
cdb[3] = (lba >> 16) as u8;
cdb[4] = (lba >> 8) as u8;
cdb[5] = lba as u8;
cdb[8] = 0x00;
cdb[9] = 0x0C;
cdb[10] = (agid << 6) | 0x04;
let mut buf = [0u8; 12];
let result = scsi.execute(
&cdb,
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
);
result.map_err(|_| Error::CssAuthFailed)?;
let mut key = [0u8; 5];
for i in 0..5 {
key[i] = buf[5 + (4 - i)];
}
Ok(key)
}
#[allow(dead_code)]
fn read_title_key(
drive: &mut Drive,
agid: u8,
lba: u32,
bus_key: &[u8; 5],
disc_key: &[u8; 5],
) -> Result<[u8; 5]> {
let scsi = drive.scsi_mut();
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
cdb[2] = (lba >> 24) as u8;
cdb[3] = (lba >> 16) as u8;
cdb[4] = (lba >> 8) as u8;
cdb[5] = lba as u8;
cdb[8] = 0x00;
cdb[9] = 0x0C;
cdb[10] = (agid << 6) | 0x04;
let mut buf = [0u8; 12];
let tk_result = scsi.execute(
&cdb,
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
);
tk_result.map_err(|_| Error::CssAuthFailed)?;
// Title key at bytes 5..10, byte-reversed
let mut title_key = [0u8; 5];
for i in 0..5 {
title_key[i] = buf[5 + (4 - i)];
}
// XOR with reversed bus key (same pattern as disc key block)
for i in 0..5 {
title_key[i] ^= bus_key[4 - i];
}
// Check for null key (title not encrypted)
if title_key == [0u8; 5] {
return Ok(title_key);
}
// Decrypt with disc key (invert=0xFF for title keys)
let title_key = super::lfsr::decrypt_key(0xFF, disc_key, &title_key);
Ok(title_key)
}
// ── CSSCryptKey ───────────────────────────────────────────────────────────
/// Exposed for testing only.
pub fn test_crypt_key(key_type: usize, variant: u8, challenge: &[u8; 10]) -> [u8; 5] {
crypt_key(key_type, variant, challenge)
}
fn crypt_key(key_type: usize, variant: u8, challenge: &[u8; 10]) -> [u8; 5] {
let perm = &PERM_CHALLENGE[key_type];
let mut scratch = [0u8; 10];
for i in 0..10 {
scratch[i] = challenge[perm[i]];
}
let css_variant = match key_type {
0 => variant as usize,
1 => PERM_VARIANT[0][variant as usize] as usize,
_ => PERM_VARIANT[1][variant as usize] as usize,
};
let cse = VARIANTS[css_variant] ^ CRYPT_TAB2[css_variant];
let mut tmp1 = [0u8; 5];
for i in 0..5 {
tmp1[i] = scratch[5 + i] ^ SECRET[i] ^ CRYPT_TAB2[i];
}
let mut lfsr0: u32 = ((tmp1[0] as u32) << 17)
| ((tmp1[1] as u32) << 9)
| (((tmp1[2] as u32) & !7) << 1)
| 8
| (tmp1[2] as u32 & 7);
let mut lfsr1: u32 = ((tmp1[3] as u32) << 9) | 0x100 | (tmp1[4] as u32);
let mut bits = [0u8; 30];
let mut carry: u32 = 0;
for idx in (0..30).rev() {
let mut val: u8 = 0;
for bit in 0..8u8 {
let lfsr0_out = ((lfsr0 >> 24) ^ (lfsr0 >> 21) ^ (lfsr0 >> 20) ^ (lfsr0 >> 12)) & 1;
lfsr0 = ((lfsr0 << 1) | lfsr0_out) & 0x1FFFFFF;
let lfsr1_out = ((lfsr1 >> 16) ^ (lfsr1 >> 2)) & 1;
lfsr1 = ((lfsr1 << 1) | lfsr1_out) & 0x1FFFF;
let combined = ((!lfsr1_out) & 1) + carry + ((!lfsr0_out) & 1);
carry = (combined >> 1) & 1;
val |= ((combined & 1) as u8) << bit;
}
bits[idx] = val;
}
let mut tmp1 = [scratch[0], scratch[1], scratch[2], scratch[3], scratch[4]];
let mut tmp2 = [0u8; 5];
// Round 1: bits[25..29] ^ scratch -> tmp1 (term from original scratch)
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[25 + i] ^ tmp1[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
tmp1[i] = CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term;
term = scratch[i]; // original challenge, NOT modified tmp1
}
tmp1[4] ^= tmp1[0];
}
// Round 2
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[20 + i] ^ tmp1[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
tmp2[i] = CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term;
term = tmp1[i];
}
tmp2[4] ^= tmp2[0];
}
// Round 3 (uses CRYPT_TAB0)
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[15 + i] ^ tmp2[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
let idx3 = (CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term) as usize;
tmp1[i] = CRYPT_TAB0[idx3] ^ CRYPT_TAB2[idx3];
term = tmp2[i];
}
tmp1[4] ^= tmp1[0];
}
// Round 4 (uses CRYPT_TAB0)
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[10 + i] ^ tmp1[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
let idx3 = (CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term) as usize;
tmp2[i] = CRYPT_TAB0[idx3] ^ CRYPT_TAB2[idx3];
term = tmp1[i];
}
tmp2[4] ^= tmp2[0];
}
// Round 5
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[5 + i] ^ tmp2[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
tmp1[i] = CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term;
term = tmp2[i];
}
tmp1[4] ^= tmp1[0];
}
// Round 6
let mut key = [0u8; 5];
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[i] ^ tmp1[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
key[i] = CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term;
term = tmp1[i];
}
}
key
}
// ── SCSI CDB builders ────────────────────────────────────────────────────
fn report_key_cdb(agid: u8, format: u8, alloc_len: u16) -> [u8; 12] {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
cdb[8] = (alloc_len >> 8) as u8;
cdb[9] = alloc_len as u8;
cdb[10] = (agid << 6) | (format & 0x3F);
cdb
}
fn send_key_cdb(agid: u8, format: u8, param_len: u16) -> [u8; 12] {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_SEND_KEY;
cdb[8] = (param_len >> 8) as u8;
cdb[9] = param_len as u8;
cdb[10] = (agid << 6) | (format & 0x3F);
cdb
}
// ── Tests ─────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn crypt_key_is_deterministic() {
let challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
for v in 0..32u8 {
let r1 = crypt_key(0, v, &challenge);
let r2 = crypt_key(0, v, &challenge);
assert_eq!(r1, r2);
}
}
#[test]
fn crypt_key_varies_by_variant() {
let challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
assert_ne!(crypt_key(0, 0, &challenge), crypt_key(0, 1, &challenge));
}
#[test]
fn crypt_key_varies_by_type() {
let challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
assert_ne!(crypt_key(0, 5, &challenge), crypt_key(1, 5, &challenge));
}
#[test]
fn crypt_key_nonzero() {
let challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
for v in 0..32u8 {
assert_ne!(crypt_key(0, v, &challenge), [0u8; 5]);
}
}
#[test]
fn player_keys_count() {
assert_eq!(PLAYER_KEYS.len(), 31);
}
}
-398
View File
@@ -1,398 +0,0 @@
//! CSS title key recovery — Stevenson's divide-and-conquer attack (1999).
//!
//! Given a scrambled DVD sector with known plaintext (MPEG-2 PES headers),
//! recovers the 5-byte title key by:
//!
//! 1. XORing ciphertext with TAB1[ciphertext] to cancel the mangling
//! 2. Iterating all 2^16 LFSR1 states
//! 3. For each: deducing what LFSR0 must produce, then verifying
//!
//! Total work: ~65536 iterations with 10-byte validation = instant.
//!
//! Algorithm: Frank A. Stevenson, "Divide and conquer attack" (1999).
use super::tables::{TAB1, TAB2, TAB3, TAB4, TAB5};
/// Sector layout constants.
const SECTOR_SIZE: usize = 2048;
const ENCRYPTED_START: usize = 0x80; // byte 128
const SEED_OFFSET: usize = 0x54; // sector seed at bytes 0x54-0x58
const FLAG_BYTE: usize = 0x14;
/// Recover the CSS title key from a scrambled sector using known plaintext.
///
/// The `plain` slice should contain the expected plaintext of the encrypted
/// region (bytes 0x80+). For MPEG-2 sectors, the first bytes are typically
/// a PES header: `00 00 01 [stream_id] ...`
///
/// Returns the recovered 5-byte title key, or None if recovery fails.
pub fn recover_title_key(sector: &[u8], plain: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_SIZE || plain.len() < 10 {
return None;
}
let flags = (sector[FLAG_BYTE] >> 4) & 0x03;
if flags == 0 {
return None;
}
let crypted = &sector[ENCRYPTED_START..];
let seed = &sector[SEED_OFFSET..SEED_OFFSET + 5];
// Phase 1: Cancel the TAB1 mangling layer
// The CSS cipher applies TAB1 as an output permutation.
// XORing ciphertext with TAB1[ciphertext] and plaintext removes it,
// leaving the raw LFSR combination output.
let mut buf = [0u8; 10];
for i in 0..10 {
if i >= crypted.len() || i >= plain.len() {
return None;
}
buf[i] = TAB1[crypted[i] as usize] ^ plain[i];
}
// Phase 2: Stevenson attack — iterate all 2^16 LFSR1 initial states
let mut result_key = [0u8; 5];
let mut found = false;
'outer: for i_try in 0u32..0x10000 {
let mut t1 = (i_try >> 8) | 0x100;
let mut t2 = i_try & 0xFF;
let mut t5: u32 = 0;
// Clock LFSR1 forward 4 steps to reconstruct LFSR0 state
let mut t3: u32 = 0;
for &buf_byte in buf.iter().take(4) {
// Advance LFSR1
let t4 = TAB2[t2 as usize] ^ TAB3[t1 as usize];
t2 = t1 >> 1;
t1 = ((t1 & 1) << 8) ^ t4 as u32;
let t4_perm = TAB5[t4 as usize];
// Deduce LFSR0 output from the buffer and LFSR1 output
let mut t6 = buf_byte as u32;
if t5 > 0 {
t6 = (t6 + 0xFF) & 0xFF;
}
if t6 < t4_perm as u32 {
t6 += 0x100;
}
t6 -= t4_perm as u32;
t5 += t6 + t4_perm as u32;
let t6_inv = TAB4[t6 as usize & 0xFF];
// Build LFSR0 candidate from deduced output bytes
t3 = (t3 << 8) | t6_inv as u32;
t5 >>= 8;
}
let candidate = t3;
// Phase 3: Validate — clock 6 more steps and check against buffer
let mut valid = true;
for &buf_byte in buf.iter().skip(4) {
let t4 = TAB2[t2 as usize] ^ TAB3[t1 as usize];
t2 = t1 >> 1;
t1 = ((t1 & 1) << 8) ^ t4 as u32;
let t4_perm = TAB5[t4 as usize];
// Clock LFSR0 forward
let t6 = ((((((t3 >> 8) ^ t3) >> 1) ^ t3) >> 3) ^ t3) >> 7;
t3 = (t3 << 8) | (t6 & 0xFF);
let t6_perm = TAB4[(t6 & 0xFF) as usize];
t5 += t6_perm as u32 + t4_perm as u32;
if (t5 & 0xFF) as u8 != buf_byte {
valid = false;
break;
}
t5 >>= 8;
}
if !valid {
continue;
}
// Phase 4: Recover the initial LFSR0 state from the candidate
t3 = candidate;
let mut recovery_ok = true;
for _ in 0..4 {
let t1_byte = t3 & 0xFF;
t3 >>= 8;
// Brute-force the byte that was shifted in
let mut found_j = false;
for j in 0u32..256 {
t3 = (t3 & 0x1FFFF) | (j << 17);
let t6 = ((((((t3 >> 8) ^ t3) >> 1) ^ t3) >> 3) ^ t3) >> 7;
if (t6 & 0xFF) == t1_byte {
found_j = true;
break;
}
}
if !found_j {
recovery_ok = false;
break;
}
}
if !recovery_ok {
continue 'outer;
}
// Convert LFSR0 initial state back to key bytes
let t4 = (t3 >> 1).wrapping_sub(4);
for t5_off in 0u32..8 {
let val = t4.wrapping_add(t5_off);
if (val * 2 + 8 - (val & 7)) == t3 {
result_key[0] = (i_try >> 8) as u8;
result_key[1] = (i_try & 0xFF) as u8;
result_key[2] = (val & 0xFF) as u8;
result_key[3] = ((val >> 8) & 0xFF) as u8;
result_key[4] = ((val >> 16) & 0xFF) as u8;
found = true;
break;
}
}
if found {
break;
}
}
if !found {
return None;
}
// XOR with sector seed to get the actual title key
result_key[0] ^= seed[0];
result_key[1] ^= seed[1];
result_key[2] ^= seed[2];
result_key[3] ^= seed[3];
result_key[4] ^= seed[4];
Some(result_key)
}
/// Crack the CSS title key from an encrypted sector using MPEG-2 pattern attack.
///
/// Detects the PES header pattern at byte 0x80 and uses it as known plaintext.
pub fn crack_title_key(sector: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_SIZE {
return None;
}
let flags = (sector[FLAG_BYTE] >> 4) & 0x03;
if flags == 0 {
return None;
}
// The PES header at byte 0x80 typically starts with 00 00 01 [stream_id].
// The next bytes are PES length and flags. We need at least 10 bytes of
// known plaintext for the Stevenson attack.
//
// Strategy: try common PES patterns. The first 3 bytes are always 00 00 01.
// The stream_id varies. Bytes 4-9 depend on PES header structure.
//
// For a standard PES with PTS:
// 00 00 01 [id] [len_hi] [len_lo] [flags] [flags2] [hdr_len] [PTS...]
//
// We try multiple stream IDs and use zeros for unknown bytes (most common).
// Try many PES header patterns at byte 0x80.
// Structure: 00 00 01 [stream_id] [len_hi] [len_lo] [flags1] [flags2] [hdr_len] [data]
let mut patterns: Vec<[u8; 10]> = Vec::with_capacity(128);
// Padding stream (0xBE): payload is 0xFF bytes, various lengths
for len_hi in 0u8..8 {
for len_lo_top in [0x00u8, 0x80, 0xFF] {
patterns.push([
0x00, 0x00, 0x01, 0xBE, len_hi, len_lo_top, 0xFF, 0xFF, 0xFF, 0xFF,
]);
}
}
// Video (0xE0) and audio (0xBD, 0xC0) with typical PES headers
for &sid in &[0xE0u8, 0xBD, 0xC0] {
for &flags1 in &[0x80u8, 0x81, 0x84, 0x85, 0x8C, 0x8D] {
for &flags2 in &[0x00u8, 0x05, 0x80, 0xC0] {
let hdr_len = if flags2 & 0x80 != 0 { 0x05u8 } else { 0x00 };
let pts0 = if flags2 & 0x80 != 0 { 0x21u8 } else { 0x00 };
// Try with several PES lengths
for &len_hi in &[0x00u8, 0x07] {
patterns.push([
0x00, 0x00, 0x01, sid, len_hi, 0x00, flags1, flags2, hdr_len, pts0,
]);
}
}
}
}
// Navigation pack system header (0xBB)
patterns.push([0x00, 0x00, 0x01, 0xBB, 0x00, 0x12, 0x80, 0xC4, 0xE1, 0x04]);
for pattern in &patterns {
if let Some(key) = recover_title_key(sector, pattern) {
let mut test = sector.to_vec();
super::lfsr::descramble_sector(&key, &mut test);
if test[0x80] == 0x00 && test[0x81] == 0x00 && test[0x82] == 0x01 {
return Some(key);
}
}
}
None
}
/// Crack CSS key from multiple sectors.
pub fn crack_from_sectors(sectors: &[Vec<u8>]) -> Option<[u8; 5]> {
for sector in sectors {
if sector.len() < SECTOR_SIZE {
continue;
}
let flags = (sector[FLAG_BYTE] >> 4) & 0x03;
if flags == 0 {
continue;
}
if let Some(key) = crack_title_key(sector) {
return Some(key);
}
}
None
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn crack_unscrambled_returns_none() {
let sector = vec![0u8; 2048];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn crack_too_short_returns_none() {
let sector = vec![0u8; 100];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn recover_needs_10_bytes_plain() {
let sector = vec![0u8; 2048];
let short_plain = [0u8; 5];
assert!(recover_title_key(&sector, &short_plain).is_none());
}
/// Test 3: css_crack_recovers_key_from_scrambled_sector
///
/// Build a plaintext sector with known MPEG-2 PES headers, scramble it
/// with a known title key, then run crack_title_key() on the scrambled
/// sector. If the Stevenson attack succeeds, verify that descrambling
/// with the recovered key produces the original plaintext at bytes 128..132.
#[test]
fn css_crack_recovers_key_from_scrambled_sector() {
use super::super::lfsr::descramble_sector;
let title_key: [u8; 5] = [0x42, 0x13, 0x37, 0xBE, 0xEF];
// Build a plaintext MPEG-2 sector
let mut plaintext = vec![0x00u8; SECTOR_SIZE];
// Pack header at byte 0: 00 00 01 BA
plaintext[0] = 0x00;
plaintext[1] = 0x00;
plaintext[2] = 0x01;
plaintext[3] = 0xBA;
// Scramble flag at byte 0x14
plaintext[FLAG_BYTE] = 0x30;
// Sector seed at bytes 0x54-0x58
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
// PES header at byte 0x80: 00 00 01 E0 (video stream)
// Then typical PES header bytes for a stream with PTS
plaintext[0x80] = 0x00;
plaintext[0x81] = 0x00;
plaintext[0x82] = 0x01;
plaintext[0x83] = 0xE0;
plaintext[0x84] = 0x00; // PES length hi
plaintext[0x85] = 0x00; // PES length lo
plaintext[0x86] = 0x80; // flags: data_alignment, copyright
plaintext[0x87] = 0x80; // PTS flag
plaintext[0x88] = 0x05; // PES header data length
plaintext[0x89] = 0x21; // PTS byte 1
let original_plaintext = plaintext.clone();
// "Scramble" the sector by calling descramble (which XORs the keystream)
// on the plaintext. This produces a scrambled sector.
descramble_sector(&title_key, &mut plaintext);
// The scramble flag was cleared by descramble_sector. Restore it so
// the cracker sees it as encrypted.
plaintext[FLAG_BYTE] = 0x30;
// Now we have a scrambled sector. Try to crack the title key.
let cracked_key = crack_title_key(&plaintext);
match cracked_key {
Some(key) => {
// Verify: descramble with the cracked key should recover plaintext
let mut test = plaintext.clone();
descramble_sector(&key, &mut test);
// Check that the PES header is recovered
assert_eq!(test[0x80], 0x00, "PES byte 0 mismatch");
assert_eq!(test[0x81], 0x00, "PES byte 1 mismatch");
assert_eq!(test[0x82], 0x01, "PES byte 2 mismatch");
assert_eq!(test[0x83], 0xE0, "PES byte 3 mismatch");
// Also verify the rest of the encrypted region matches original
assert_eq!(
&test[0x80..SECTOR_SIZE],
&original_plaintext[0x80..SECTOR_SIZE],
"Decrypted content does not match original plaintext"
);
eprintln!(
"Stevenson attack succeeded: cracked key = {:02X?}, original = {:02X?}",
key, title_key
);
}
None => {
// The Stevenson attack may not always find a key for all title keys
// and sector seeds. This is expected for some combinations where the
// known plaintext pattern doesn't match what crack_title_key tries.
eprintln!(
"Stevenson attack did not find key for title_key={:02X?} seed={:02X?}. \
This can happen when the cipher output doesn't match the tried patterns. \
Testing with recover_title_key directly with exact plaintext.",
title_key,
&[0x11u8, 0x22, 0x33, 0x44, 0x55],
);
// Try with exact known plaintext instead of guessing
let exact_plain: [u8; 10] =
[0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x80, 0x05, 0x21];
let recovered = recover_title_key(&plaintext, &exact_plain);
if let Some(key) = recovered {
let mut test = plaintext.clone();
descramble_sector(&key, &mut test);
assert_eq!(test[0x80], 0x00);
assert_eq!(test[0x81], 0x00);
assert_eq!(test[0x82], 0x01);
eprintln!(
"recover_title_key with exact plaintext succeeded: {:02X?}",
key
);
} else {
eprintln!(
"recover_title_key also returned None. The attack may not converge \
for this particular key/seed combination. This is a known limitation \
of the brute-force LFSR0 recovery phase."
);
}
}
}
}
}
+358 -153
View File
@@ -1,11 +1,15 @@
//! CSS cipher implementation based on the Stevenson 1999 analysis.
//!
//! The CSS cipher uses two table-driven feedback circuits:
//! - LFSR1: 9-bit state (two halves), driven by TAB2/TAB3
//! - LFSR0: 32-bit state, driven by a feedback polynomial through TAB4
//! - LFSR1: 17-bit state (9-bit lo + 8-bit hi register, seeded from
//! key[0..2]), driven by TAB2/TAB3
//! - LFSR0: 24-bit feedback register (seeded from key[2..5] XOR seed[2..5],
//! masked to 0xFFFFFF), driven by a feedback polynomial through TAB4
//!
//! The keystream is the bytewise sum (with carry) of both LFSR outputs.
//! Content descrambling XORs this keystream with the encrypted sector data.
//! Content descrambling computes plain = TAB1[cipher] ^ keystream — a TAB1
//! substitution of each ciphertext byte followed by an XOR with the keystream
//! (NOT a plain XOR; the cipher is not its own inverse).
//!
//! Algorithm: Frank A. Stevenson's divide-and-conquer attack (1999).
//! Tables: CSS specification constants.
@@ -14,122 +18,155 @@ use super::tables::{TAB1, TAB2, TAB3, TAB4, TAB5};
/// Descramble a CSS-encrypted DVD sector in place.
///
/// The sector seed (bytes 0x54-0x58) is XORed with the title key to produce
/// the per-sector key. Bytes 0x80..0x800 (128..2048) are then decrypted
/// using the two-LFSR keystream.
/// Exact port of libdvdcss `dvdcss_unscramble` (css.c). The two content
/// LFSRs are seeded **directly** from `title_key XOR sector_seed` — there is
/// no `decrypt_key` mangling on this path (that is the disc/title-key
/// hierarchy, not the content cipher). Bytes 0x80..0x800 are recovered with
/// `*p = TAB1[*p] ^ (i_t5 & 0xff)`.
///
/// The scramble flag at byte 0x14 (bits 4-5) indicates encryption.
/// After descrambling, the flag is cleared.
/// The scramble flag at byte 0x14 (bits 4-5) indicates encryption. Like
/// libdvdcss, the flag byte is NOT modified here — the caller treats a
/// nonzero `sector[0x14] & 0x30` as "needs unscrambling" and the descramble
/// is its own inverse, so re-running it on plaintext would re-scramble.
/// (freemkv historically cleared the flag; we keep clearing it so callers
/// and the existing tests can distinguish a descrambled sector. This does
/// not affect the recovered body.)
///
/// No-op (returns without modifying `sector`) in two cases:
/// - `sector.len() < 2048`: the encrypted region (0x80..0x800) is not
/// fully present. Callers chunk by 2048, so a trailing partial chunk is
/// left untouched. The `debug_assert!` flags this misuse in debug/test
/// builds; a DVD sector is always exactly 2048 bytes.
/// - scramble flags are zero: the sector is not CSS-encrypted.
///
/// Design reference: libdvdcss `dvdcss_unscramble`. The combiner mirrors
/// `css.c` line-for-line:
/// ```text
/// i_t1 = (key[0] ^ sec[0x54]) | 0x100;
/// i_t2 = key[1] ^ sec[0x55];
/// i_t3 = (key[2]|key[3]<<8|key[4]<<16) ^ (sec[0x56]|sec[0x57]<<8|sec[0x58]<<16);
/// i_t4 = i_t3 & 7; i_t3 = i_t3*2 + 8 - i_t4;
/// // per byte over 0x80..0x800:
/// i_t4 = TAB2[i_t2] ^ TAB3[i_t1];
/// i_t2 = i_t1 >> 1; i_t1 = ((i_t1 & 1) << 8) ^ i_t4; i_t4 = TAB5[i_t4];
/// i_t6 = (((((((i_t3>>3)^i_t3)>>1)^i_t3)>>8)^i_t3)>>5) & 0xff;
/// i_t3 = (i_t3 << 8) | i_t6; i_t6 = TAB4[i_t6];
/// i_t5 += i_t6 + i_t4; *p = TAB1[*p] ^ (i_t5 & 0xff); i_t5 >>= 8;
/// ```
pub fn descramble_sector(title_key: &[u8; 5], sector: &mut [u8]) {
debug_assert!(
sector.len() >= 2048,
"descramble_sector: buffer shorter than one 2048-byte sector"
);
if sector.len() < 2048 {
return;
}
let flags = (sector[0x14] >> 4) & 0x03;
if flags == 0 {
// libdvdcss: `if( !(p_sec[0x14] & 0x30) ) return;`
if sector[0x14] & 0x30 == 0 {
return;
}
// Per-sector key = title_key XOR sector_seed (bytes 0x54-0x58)
let key = [
title_key[0] ^ sector[0x54],
title_key[1] ^ sector[0x55],
title_key[2] ^ sector[0x56],
title_key[3] ^ sector[0x57],
title_key[4] ^ sector[0x58],
];
// LFSR1: seeded directly from (key ^ seed) — NO decrypt_key.
let mut i_t1: u32 = ((title_key[0] ^ sector[0x54]) as u32) | 0x100;
let mut i_t2: u32 = (title_key[1] ^ sector[0x55]) as u32;
// Decrypt the key through the CSS mangling function to get the working key
let working_key = decrypt_key(0xFF, &key, &sector[0x54..0x59]);
// LFSR0 (i_t3): 24-bit feedback register seeded from the remaining three
// key/seed bytes, then transformed `i_t3 = i_t3*2 + 8 - (i_t3 & 7)`.
let mut i_t3: u32 = (((title_key[2] as u32)
| ((title_key[3] as u32) << 8)
| ((title_key[4] as u32) << 16))
^ ((sector[0x56] as u32) | ((sector[0x57] as u32) << 8) | ((sector[0x58] as u32) << 16)))
& 0xFF_FFFF;
let i_t4_seed = i_t3 & 7;
i_t3 = i_t3 * 2 + 8 - i_t4_seed;
// Generate keystream and XOR with encrypted region
let mut lfsr1_lo: u32 = working_key[0] as u32 | 0x100;
let mut lfsr1_hi: u32 = working_key[1] as u32;
let mut i_t5: u32 = 0;
let mut lfsr0: u32 = ((working_key[4] as u32) << 17)
| ((working_key[3] as u32) << 9)
| (((working_key[2] as u32) << 1) + 8 - (working_key[2] as u32 & 7));
lfsr0 = (TAB4[(lfsr0 & 0xFF) as usize] as u32) << 24
| (TAB4[((lfsr0 >> 8) & 0xFF) as usize] as u32) << 16
| (TAB4[((lfsr0 >> 16) & 0xFF) as usize] as u32) << 8
| TAB4[((lfsr0 >> 24) & 0xFF) as usize] as u32;
let mut combined: u32 = 0;
// Generate 1920 keystream bytes (for sector bytes 128..2048)
// Per libdvdcss css_unscramble: TAB1 permutation on ciphertext, no invert on LFSR0
for byte in sector.iter_mut().take(2048).skip(128) {
let o_lfsr1 = TAB2[lfsr1_hi as usize] ^ TAB3[lfsr1_lo as usize];
lfsr1_hi = lfsr1_lo >> 1;
lfsr1_lo = ((lfsr1_lo & 1) << 8) ^ o_lfsr1 as u32;
// Advance LFSR1.
let mut i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
i_t4 = TAB5[i_t4 as usize] as u32;
let o_lfsr0 = (((((((lfsr0 >> 8) ^ lfsr0) >> 1) ^ lfsr0) >> 3) ^ lfsr0) >> 7) as u8;
lfsr0 = (lfsr0 >> 8) | ((o_lfsr0 as u32) << 24);
// Advance LFSR0 (i_t3) and fold both outputs into i_t5.
let mut i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xFF;
i_t3 = (i_t3 << 8) | i_t6;
i_t6 = TAB4[i_t6 as usize] as u32;
i_t5 += i_t6 + i_t4;
combined += TAB5[o_lfsr1 as usize] as u32 + TAB4[o_lfsr0 as usize] as u32;
*byte ^= (combined & 0xFF) as u8;
combined >>= 8;
*byte = TAB1[*byte as usize] ^ (i_t5 & 0xFF) as u8;
i_t5 >>= 8;
}
// Clear scramble flags
// libdvdcss leaves byte 0x14 untouched; freemkv clears the scramble bits
// so downstream code and tests can tell a sector was descrambled.
sector[0x14] &= 0xCF;
}
/// CSS key decryption / mangling function.
/// Exact inverse of [`descramble_sector`]: turn a plaintext sector body into
/// CSS ciphertext under `title_key`.
///
/// Decrypts `p_crypted` using `p_key` with the CSS two-LFSR cipher.
/// The `invert` parameter controls the XOR applied to LFSR0 output
/// (0x00 for disc key decryption, 0xFF for title key / sector key).
pub(crate) fn decrypt_key(invert: u8, p_key: &[u8; 5], p_crypted: &[u8]) -> [u8; 5] {
if p_crypted.len() < 5 {
return *p_key;
/// Descramble computes `plain = TAB1[cipher] ^ (i_t5 & 0xff)`, so the
/// inverse is `cipher = TAB1_INV[plain ^ (i_t5 & 0xff)]` with the identical
/// LFSR keystream. The keystream derivation is byte-for-byte the same as
/// `descramble_sector` (libdvdcss `dvdcss_unscramble`); only the final
/// substitution differs. Bytes 0x80..0x800 are rewritten in place; the
/// scramble flag is set to 0x10 so a subsequent descramble runs.
///
/// Not on any production read path — it exists so the key-recovery tests
/// (and any caller that needs to produce a known CSS-encrypted sector) can
/// build genuine ciphertext rather than approximating it.
#[cfg(test)]
pub(crate) fn scramble_sector(title_key: &[u8; 5], sector: &mut [u8]) {
if sector.len() < 2048 {
return;
}
let mut lfsr1_lo: u32 = p_key[0] as u32 | 0x100;
let mut lfsr1_hi: u32 = p_key[1] as u32;
let mut i_t1: u32 = ((title_key[0] ^ sector[0x54]) as u32) | 0x100;
let mut i_t2: u32 = (title_key[1] ^ sector[0x55]) as u32;
let mut i_t3: u32 = (((title_key[2] as u32)
| ((title_key[3] as u32) << 8)
| ((title_key[4] as u32) << 16))
^ ((sector[0x56] as u32) | ((sector[0x57] as u32) << 8) | ((sector[0x58] as u32) << 16)))
& 0xFF_FFFF;
let i_t4_seed = i_t3 & 7;
i_t3 = i_t3 * 2 + 8 - i_t4_seed;
let mut lfsr0: u32 = ((p_key[4] as u32) << 17)
| ((p_key[3] as u32) << 9)
| (((p_key[2] as u32) << 1) + 8 - (p_key[2] as u32 & 7));
lfsr0 = (TAB4[(lfsr0 & 0xFF) as usize] as u32) << 24
| (TAB4[((lfsr0 >> 8) & 0xFF) as usize] as u32) << 16
| (TAB4[((lfsr0 >> 16) & 0xFF) as usize] as u32) << 8
| TAB4[((lfsr0 >> 24) & 0xFF) as usize] as u32;
let mut i_t5: u32 = 0;
let mut combined: u32 = 0;
let mut k = [0u8; 5];
for byte in sector.iter_mut().take(2048).skip(128) {
let mut i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
i_t4 = TAB5[i_t4 as usize] as u32;
for byte in &mut k {
let o_lfsr1 = TAB2[lfsr1_hi as usize] ^ TAB3[lfsr1_lo as usize];
lfsr1_hi = lfsr1_lo >> 1;
lfsr1_lo = ((lfsr1_lo & 1) << 8) ^ o_lfsr1 as u32;
let mut i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xFF;
i_t3 = (i_t3 << 8) | i_t6;
i_t6 = TAB4[i_t6 as usize] as u32;
i_t5 += i_t6 + i_t4;
let o_lfsr0 = (((((((lfsr0 >> 8) ^ lfsr0) >> 1) ^ lfsr0) >> 3) ^ lfsr0) >> 7) as u8;
lfsr0 = (lfsr0 >> 8) | ((o_lfsr0 as u32) << 24);
// TAB5 for LFSR1 output, TAB4 for LFSR0^invert (per libdvdcss css_DecryptKey)
combined += TAB5[o_lfsr1 as usize] as u32 + TAB4[(o_lfsr0 ^ invert) as usize] as u32;
*byte = (combined & 0xFF) as u8;
combined >>= 8;
// Inverse of `*p = TAB1[*p] ^ ks`: apply ks then TAB1's inverse.
*byte = (*TAB1_INV)[(*byte ^ (i_t5 & 0xFF) as u8) as usize];
i_t5 >>= 8;
}
// Two rounds of chained XOR through TAB1
let mut result = [0u8; 5];
result[4] = k[4] ^ TAB1[p_crypted[4] as usize] ^ p_crypted[3];
result[3] = k[3] ^ TAB1[p_crypted[3] as usize] ^ p_crypted[2];
result[2] = k[2] ^ TAB1[p_crypted[2] as usize] ^ p_crypted[1];
result[1] = k[1] ^ TAB1[p_crypted[1] as usize] ^ p_crypted[0];
result[0] = k[0] ^ TAB1[p_crypted[0] as usize] ^ result[4];
result[4] = k[4] ^ TAB1[result[4] as usize] ^ result[3];
result[3] = k[3] ^ TAB1[result[3] as usize] ^ result[2];
result[2] = k[2] ^ TAB1[result[2] as usize] ^ result[1];
result[1] = k[1] ^ TAB1[result[1] as usize] ^ result[0];
result[0] = k[0] ^ TAB1[result[0] as usize];
result
// Mark the sector scrambled so the descrambler will process it.
sector[0x14] = (sector[0x14] & 0xCF) | 0x10;
}
/// Inverse permutation of [`TAB1`], built at first use. `TAB1` is a
/// bijection on 0..256, so `TAB1_INV[TAB1[x]] == x`.
#[cfg(test)]
static TAB1_INV: std::sync::LazyLock<[u8; 256]> = std::sync::LazyLock::new(|| {
let mut inv = [0u8; 256];
for (i, &v) in TAB1.iter().enumerate() {
inv[v as usize] = i as u8;
}
inv
});
#[cfg(test)]
mod tests {
use super::*;
@@ -144,6 +181,34 @@ mod tests {
assert_eq!(sector, original);
}
/// Cross-check `descramble_sector` against the EXACT output of libdvdcss
/// `dvdcss_unscramble` (css.c) for a fixed sector, computed from the
/// reference C semantics with the reference tables. Pins the content
/// cipher to libdvdcss byte-for-byte.
///
/// key = 42 13 37 BE EF, seed (0x54..0x59) = DE AD BE EF 42, body = 0xAA.
#[test]
fn descramble_matches_libdvdcss_unscramble_vector() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0xAAu8; 2048];
sector[0x14] = 0x30;
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
descramble_sector(&key, &mut sector);
assert_eq!(
&sector[0x80..0x90],
&[
0x81, 0x92, 0x24, 0xA2, 0x46, 0x70, 0x3C, 0x64, 0xA6, 0x91, 0x84, 0xF5, 0x1F, 0x98,
0xA0, 0x31
],
"descramble body head must match libdvdcss dvdcss_unscramble"
);
assert_eq!(
&sector[0x7F8..0x800],
&[0x46, 0x94, 0x80, 0x0E, 0x67, 0x36, 0x65, 0xBC],
"descramble body tail must match libdvdcss dvdcss_unscramble"
);
}
#[test]
fn descramble_modifies_scrambled() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
@@ -174,67 +239,14 @@ mod tests {
assert_eq!(sector[0x14] & 0x30, 0x00);
}
#[test]
fn decrypt_key_produces_output() {
let key = [0x12, 0x34, 0x56, 0x78, 0x9A];
let crypted = [0xAB, 0xCD, 0xEF, 0x01, 0x23];
let result = decrypt_key(0xFF, &key, &crypted);
// Should produce a 5-byte result different from input
assert_ne!(result, key);
assert_ne!(result, [0u8; 5]);
}
/// Test 1: css_decrypt_key_roundtrip
/// Test 2: descramble inverts scramble over the body.
///
/// decrypt_key is not a simple encrypt/decrypt pair — it is a one-way mangling
/// function. However, we can verify consistency: calling it twice with the same
/// parameters produces the same output, and varying the invert byte changes
/// the LFSR0 contribution predictably.
/// The content cipher is NOT a plain XOR involution (it applies TAB1 to
/// the ciphertext: `plain = TAB1[cipher] ^ ks`). The true inverse is
/// [`scramble_sector`]. Scrambling a plaintext body and then descrambling
/// with the same key must reproduce the original body exactly.
#[test]
fn css_decrypt_key_roundtrip() {
let keys: &[[u8; 5]] = &[
[0x12, 0x34, 0x56, 0x78, 0x9A],
[0x00, 0x00, 0x00, 0x00, 0x00],
[0xFF, 0xFF, 0xFF, 0xFF, 0xFF],
[0xAB, 0xCD, 0xEF, 0x01, 0x23],
];
let crypted_inputs: &[[u8; 5]] = &[
[0x11, 0x22, 0x33, 0x44, 0x55],
[0xAA, 0xBB, 0xCC, 0xDD, 0xEE],
[0x00, 0x00, 0x00, 0x00, 0x00],
];
for key in keys {
for crypted in crypted_inputs {
// decrypt_key with invert=0x00 and invert=0xFF should give different results
let r0 = decrypt_key(0x00, key, crypted);
let rff = decrypt_key(0xFF, key, crypted);
// The two results differ because the invert byte XORs the LFSR0 output
// They should not be equal (except by extreme coincidence)
// More importantly, both should be deterministic
let r0_again = decrypt_key(0x00, key, crypted);
let rff_again = decrypt_key(0xFF, key, crypted);
assert_eq!(r0, r0_again, "decrypt_key(0x00) not deterministic");
assert_eq!(rff, rff_again, "decrypt_key(0xFF) not deterministic");
// With different invert values, the keystream differs
assert_ne!(
r0, rff,
"invert=0x00 and 0xFF gave same result for key {:?}",
key
);
}
}
}
/// Test 2: css_descramble_produces_valid_mpeg2
///
/// descramble_sector XORs a keystream into bytes 128..2048. Calling it
/// twice with the same key and restored scramble flag should roundtrip,
/// since XOR is its own inverse.
#[test]
fn css_descramble_modifies_encrypted_region() {
fn css_descramble_inverts_scramble_over_body() {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0xAAu8; 2048];
@@ -242,11 +254,10 @@ mod tests {
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
let original = sector.clone();
descramble_sector(&title_key, &mut sector);
// Flag cleared
assert_eq!(sector[0x14] & 0x30, 0x00);
// Header (0..128) unchanged except flag byte
// Scramble the plaintext body into ciphertext.
scramble_sector(&title_key, &mut sector);
// Header (0..128) unchanged except the flag byte (set by scramble).
for i in 0..128 {
if i == 0x14 {
continue;
@@ -255,9 +266,18 @@ mod tests {
}
// Encrypted region modified
assert_ne!(&sector[128..256], &original[128..256]);
// Descramble restores the plaintext body byte-for-byte.
descramble_sector(&title_key, &mut sector);
assert_eq!(sector[0x14] & 0x30, 0x00, "flag cleared after descramble");
assert_eq!(
&sector[128..2048],
&original[128..2048],
"descramble(scramble(body)) did not restore the body"
);
}
/// Test 4: css_tab1_relationship
/// css_tab1_relationship
///
/// Verify the structure of TAB1: it is a substitution table used in
/// key mangling. Check that no two inputs map to the same output
@@ -281,7 +301,7 @@ mod tests {
}
}
/// Test 5: css_tab4_is_bit_reversal
/// css_tab4_is_bit_reversal
///
/// TAB4 reverses the bits of each byte: TAB4[0x01] = 0x80, TAB4[0x80] = 0x01, etc.
#[test]
@@ -303,4 +323,189 @@ mod tests {
);
}
}
// ── scramble-flag detection (byte 0x14, bits 4-5) ──────────────────────
/// Only bits 4-5 of byte 0x14 are the CSS scramble flag: the code reads
/// `sector[0x14] & 0x30 == 0` (bits 6-7, i.e. 0x40/0x80, are masked out by
/// 0x30). A sector with 0x14 == 0x40 or 0x80 must therefore be treated as
/// UNSCRAMBLED and left byte-for-byte unchanged. This guards against a
/// too-wide mask silently "descrambling" (and thus corrupting) clear data.
///
/// Grounding: CSS sector header byte 0x14 — copyright/scramble bits live
/// in bits 4-5; the masked value 0 means not scrambled.
/// Mutation: widen the mask `0x30` to `0x70`/`0xF0` -> 0x40/0x80 would be
/// seen as scrambled and the body would change.
#[test]
fn descramble_treats_high_bits_of_0x14_as_clear() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
for &flag in &[0x40u8, 0x80, 0xC0, 0x0F, 0x4F, 0x8F] {
let mut sector = vec![0xAA; 2048];
sector[0x14] = flag;
sector[0x54..0x59].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
let original = sector.clone();
descramble_sector(&key, &mut sector);
assert_eq!(
sector, original,
"byte 0x14 = {flag:#04x} has flag bits 4-5 clear; sector must be untouched"
);
}
}
/// Each individual scramble bit (4 and 5) independently marks the sector
/// as encrypted: 0x10 and 0x20 must both trigger descrambling.
///
/// Grounding: `(0x10 >> 4) & 3 == 1`, `(0x20 >> 4) & 3 == 2` — both
/// nonzero.
/// Mutation: change `!= 0` early-return condition to `== 3` -> a sector
/// flagged only 0x10 or 0x20 would be skipped and left scrambled.
#[test]
fn descramble_triggers_on_either_flag_bit() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
for &flag in &[0x10u8, 0x20, 0x30] {
let mut sector = vec![0xAA; 2048];
sector[0x14] = flag;
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
let original = sector.clone();
descramble_sector(&key, &mut sector);
assert_ne!(
&sector[128..256],
&original[128..256],
"flag {flag:#04x} (bits 4-5 nonzero) must descramble the body"
);
}
}
/// After descrambling, ONLY the two scramble bits are cleared (`& 0xCF`);
/// bits 6 and 7 of byte 0x14 must be preserved. A sector with 0x14 == 0xF0
/// becomes 0xC0 (bits 6,7 kept, bits 4,5 cleared), NOT 0x00.
///
/// Grounding: code does `sector[0x14] &= 0xCF`; 0xF0 & 0xCF == 0xC0.
/// Mutation: change `&= 0xCF` to `= 0` or `&= 0x0F` -> the preserved
/// high bits assert fails.
#[test]
fn descramble_clear_preserves_high_bits_of_0x14() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let mut sector = vec![0x00; 2048];
sector[0x14] = 0xF0; // bits 4-7 set; bits 4-5 are the flag
sector[0x54..0x59].copy_from_slice(&[0x00; 5]);
descramble_sector(&key, &mut sector);
assert_eq!(
sector[0x14], 0xC0,
"scramble bits cleared, bits 6-7 preserved (0xF0 & 0xCF)"
);
}
// ── header / body boundary (encrypted region is 0x80..0x800) ───────────
/// The encrypted region is exactly bytes 0x80..0x800. Bytes 0x00..0x80
/// (the header) must NOT be modified by the keystream — except byte 0x14
/// whose flag is cleared. In particular the sector-seed bytes 0x54..0x59
/// (which live inside the header) must survive untouched, since the
/// descrambler reads them but never writes them.
///
/// Grounding: loop is `sector.iter_mut().take(2048).skip(128)` -> indices
/// 128..2048 only.
/// Mutation: change `.skip(128)` to `.skip(0)` -> header bytes (incl. the
/// seed) get XORed and this fails.
#[test]
fn descramble_leaves_header_and_seed_intact() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0x5Au8; 2048];
sector[0x14] = 0x30;
let seed = [0xDE, 0xAD, 0xBE, 0xEF, 0x42];
sector[0x54..0x59].copy_from_slice(&seed);
let original = sector.clone();
descramble_sector(&key, &mut sector);
for i in 0..0x80usize {
if i == 0x14 {
continue;
}
assert_eq!(
sector[i], original[i],
"header byte {i:#04x} must be untouched"
);
}
assert_eq!(&sector[0x54..0x59], &seed, "sector seed must survive");
}
/// The descrambler must touch the WHOLE body 0x80..0x800, not just a
/// prefix. With a constant body and constant key, the keystream is
/// non-degenerate enough that the very last sector byte (index 2047) is
/// altered. This guards the loop bound `.take(2048)` against an
/// off-by-one that would leave the final byte(s) scrambled.
///
/// Grounding: encrypted region end is 0x800 == 2048 (exclusive).
/// Mutation: change `.take(2048)` to `.take(2047)` -> last byte unchanged,
/// assert fires (keystream byte for the last position is verified nonzero
/// below by the round-trip, and this body is all-zero so any XOR shows).
#[test]
fn descramble_covers_final_body_byte() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0x00u8; 2048];
sector[0x14] = 0x30;
sector[0x54..0x59].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
descramble_sector(&key, &mut sector);
// Body was all zero; any nonzero in [0x80,0x800) is keystream. Confirm
// the keystream reaches the final byte. (If the last keystream byte
// happened to be 0 this could be a flaky test, so assert the run-end
// region as a whole differs from zero.)
assert_ne!(
&sector[2040..2048],
&[0u8; 8][..],
"the tail of the body must be descrambled (loop must reach index 2047)"
);
}
/// Descramble is keyed by `title_key XOR seed`: two different title keys
/// produce two different bodies for the same scrambled input. A cipher
/// that ignored the title key (or mixed it in wrongly) would yield
/// identical output — silent wrong-key decryption.
///
/// Grounding: per-sector key = title_key[i] ^ sector[0x54+i].
/// Mutation: in the `key` array drop the `title_key[i] ^` term -> both
/// keys give the same body, assert fires.
#[test]
fn descramble_output_depends_on_title_key() {
let seed = [0xDE, 0xAD, 0xBE, 0xEF, 0x42];
let make = |k: &[u8; 5]| {
let mut s = vec![0x00u8; 2048];
s[0x14] = 0x30;
s[0x54..0x59].copy_from_slice(&seed);
descramble_sector(k, &mut s);
s
};
let a = make(&[0x01, 0x02, 0x03, 0x04, 0x05]);
let b = make(&[0x01, 0x02, 0x03, 0x04, 0x06]); // differs in last byte
assert_ne!(
&a[128..2048],
&b[128..2048],
"different title keys must descramble differently"
);
}
/// Descramble is keyed by the sector seed too: same title key, different
/// seed -> different body. Pins that bytes 0x54..0x59 actually feed the
/// keystream (not just the per-sector XOR key).
///
/// Mutation: replace `seed` array reads with a constant -> both seeds give
/// the same body, assert fires.
#[test]
fn descramble_output_depends_on_seed() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let make = |seed: [u8; 5]| {
let mut s = vec![0x00u8; 2048];
s[0x14] = 0x30;
s[0x54..0x59].copy_from_slice(&seed);
descramble_sector(&key, &mut s);
s
};
let a = make([0x11, 0x22, 0x33, 0x44, 0x55]);
let b = make([0x11, 0x22, 0x33, 0x44, 0x56]);
assert_ne!(
&a[128..2048],
&b[128..2048],
"different seeds must descramble differently"
);
}
}
+838 -69
View File
@@ -1,110 +1,269 @@
//! CSS (Content Scramble System) — DVD disc encryption.
//!
//! CSS uses a weak 40-bit LFSR stream cipher (broken since 1999).
//! No keys needed — the title key is cracked from encrypted content
//! using a known-plaintext attack on MPEG-2 PES headers.
//!
//! The title key is recovered keylessly: [`crack_key`] runs the Stevenson
//! known-plaintext attack (see the [`stevenson`] module) on the scrambled
//! data, needing no player keys, disc-key crack, or external key file.
//! Sectors are then decrypted with [`descramble_sector`].
//!
//! Usage:
//! ```rust,ignore
//! let key = css::crack_key(reader, &extents)?;
//! css::descramble_sector(&key, &mut sector);
//! if let Some(state) = css::crack_key(reader, extents, batch) {
//! css::descramble_sector(&state, &mut sector);
//! }
//! ```
pub mod auth;
pub mod crack;
pub mod lfsr;
pub mod stevenson;
pub(crate) mod tables;
use crate::disc::Extent;
use crate::drive::Drive;
use crate::sector::SectorSource;
/// Consecutive CSS-locked (`05/6F/03`) reads before the crack scan early-bails.
/// The bus-auth read gate is global (all-or-nothing), so a run this long means
/// it is shut and nothing here is crackable — bail instead of grinding the full
/// 50_000-sector budget (which is what made rc5 appear to hang on a wedged USB
/// bridge). The counter resets to 0 on any readable batch.
const CSS_LOCKED_BAIL: u32 = 64;
/// CSS decryption state for a DVD title.
#[derive(Debug, Clone)]
pub struct CssState {
/// Cracked 5-byte title key
/// 5-byte CSS title key (from SCSI auth or the crack fallback).
pub title_key: [u8; 5],
/// LBA half-open span `[start, end)` of the extent set this key was
/// cracked from. CSS title keys are per-VTS: a key cracked from one
/// VTS does NOT descramble a title living in a different VTS. The mux
/// path checks whether the title being opened overlaps this span; if
/// not, it re-cracks from that title's own extents. `None` for keys
/// of unknown provenance (e.g. test fixtures) — treated as "applies
/// everywhere" for backward compatibility.
pub crack_span: Option<(u32, u32)>,
}
/// Inputs for CSS key acquisition.
/// Recover the CSS title key with no keys, by scanning scrambled sectors and
/// running the Stevenson known-plaintext attack (see the [`stevenson`] module).
///
/// The acquisition path depends on which inputs the caller supplies:
///
/// - With `drive` + `auth_lba` set, [`resolve`] runs the full SCSI bus
/// auth + title-key path (live BU40N / DVD drive).
/// - With `reader` + `extents` set, [`resolve`] falls back to the
/// crack path (Stevenson known-plaintext attack on encrypted PES
/// headers; works on disc images and on drives whose CSS auth path
/// is unavailable).
///
/// `live_drive` always wins when both modes are populated.
pub struct CssContext<'a> {
/// Live SCSI drive — when present, [`resolve`] tries the auth path.
pub drive: Option<&'a mut Drive>,
/// LBA of a known-scrambled sector for the auth path's title-key
/// query. Required when `drive` is set.
pub auth_lba: Option<u32>,
/// Sector source for the crack path.
pub reader: Option<&'a mut dyn SectorSource>,
/// Extents to scan for the crack path. Required when `reader` is
/// set.
pub extents: Option<&'a [Extent]>,
/// The crib comes from `AttackPattern`: a scrambled sector's cleartext region
/// (bytes 0x00..0x80) often ends in a short-period repeating run (stuffing /
/// constant fill); the attack assumes that run continues across the 0x80
/// boundary into the encrypted region, giving the known plaintext the 2^16
/// LFSR recovery needs. We scan up to 50000 sectors across the
/// extents and return the first sector that yields a key — no player keys, no
/// disc-key crack. Works on a live drive (after bus-auth unlocks reads) and on
/// disc images alike.
pub fn crack_key(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
) -> Option<CssState> {
crack_key_halt(reader, extents, batch_sectors, None)
}
/// Acquire a CSS title key using whichever inputs the context provides.
/// Outcome of a CSS crack scan that distinguishes the THREE cases the bare
/// `Option<CssState>` conflated (and which caused a silent-failure bug:
/// scrambled-but-uncracked content was treated as "unencrypted" and muxed as
/// plaintext garbage at exit 0):
///
/// Order of attempts:
/// 1. SCSI auth path (when `drive` and `auth_lba` are set).
/// 2. Crack path (when `reader` and `extents` are set).
///
/// Returns `None` if neither path is configured or both fail.
pub fn resolve(ctx: &mut CssContext<'_>) -> Option<CssState> {
if let (Some(drive), Some(lba)) = (ctx.drive.as_deref_mut(), ctx.auth_lba) {
if let Ok(title_key) = auth::authenticate_and_read_title_key(drive, lba) {
return Some(CssState { title_key });
}
}
if let (Some(reader), Some(extents)) = (ctx.reader.as_deref_mut(), ctx.extents) {
return crack_key(reader, extents);
}
None
/// - [`CrackOutcome::Cracked`] — a scrambled sector yielded a title key.
/// - [`CrackOutcome::Unencrypted`] — NO scrambled sector was seen across the
/// scanned extents (`is_scrambled` never true): the content is genuinely
/// plaintext, so proceeding without a key is correct.
/// - [`CrackOutcome::ScrambledUncracked`] — scrambled sectors WERE seen but no
/// key could be recovered (the Stevenson attack found no crackable crib, or
/// the scrambled region was unreadable). The content is encrypted; muxing it
/// as plaintext would emit garbage, so callers MUST surface a hard error
/// ([`crate::error::Error::CssKeyMissing`]) instead of falling through to
/// "unencrypted".
#[derive(Debug, Clone)]
pub enum CrackOutcome {
Cracked(CssState),
Unencrypted,
ScrambledUncracked,
}
/// Crack the CSS title key by reading encrypted sectors and applying
/// a known-plaintext attack on MPEG-2 headers.
impl CrackOutcome {
/// The cracked `CssState`, if any. `None` for `Unencrypted` /
/// `ScrambledUncracked`. Lets the `Option`-returning wrappers stay thin.
pub fn into_state(self) -> Option<CssState> {
match self {
CrackOutcome::Cracked(s) => Some(s),
_ => None,
}
}
/// True when scrambled sectors were seen but no key was recovered — the
/// case callers must surface as a hard error instead of "unencrypted".
pub fn is_scrambled_uncracked(&self) -> bool {
matches!(self, CrackOutcome::ScrambledUncracked)
}
}
/// [`crack_key`] returning the full [`CrackOutcome`] (Cracked / Unencrypted /
/// ScrambledUncracked) so callers can distinguish "genuinely unencrypted" from
/// "encrypted but uncrackable" — the latter must become a hard error, never a
/// silent fall-through to plaintext.
pub fn crack_key_outcome(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
) -> CrackOutcome {
crack_key_scan(reader, extents, batch_sectors, halt, true)
}
/// [`crack_key`] with an optional cooperative-cancellation token.
///
/// Crack the CSS title key by scanning scrambled sectors across extents.
///
/// The Stevenson attack needs a sector where a PES header starts at byte
/// 0x80 (start of the encrypted region). This only happens when a new PES
/// packet begins at exactly sector offset 128. We scan up to 50000
/// scrambled sectors sequentially across all extents.
pub fn crack_key(reader: &mut dyn SectorSource, extents: &[Extent]) -> Option<CssState> {
/// "No silent hangs": the crack scans up to 50_000 sectors, which on a live
/// drive hitting bad sectors can take a long time. This variant polls `halt`
/// once per batch (the same cadence sweep/patch use) so an operator Stop or a
/// scan-level watchdog can interrupt the scan, and emits a
/// `freemkv::heartbeat` beat ("css_crack") each batch so a stuck scan is
/// visible in the log.
pub fn crack_key_halt(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
) -> Option<CssState> {
crack_key_scan(reader, extents, batch_sectors, halt, false).into_state()
}
/// The crack scan, returning the full [`CrackOutcome`]. Tracks a
/// `saw_scrambled` flag so a scrambled-but-uncracked disc is distinguished
/// from a genuinely-unencrypted one (the [`crack_key`] / [`crack_key_halt`]
/// `Option` wrappers collapse both to `None`).
fn crack_key_scan(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
// True only on the INITIAL scan: a fully CSS-locked (`05/6F/03`) result is a
// hard `ScrambledUncracked`. False on the per-VTS re-crack so a lapsed-AGID
// locked read returns None instead of killing a genuinely crackable title.
fail_on_locked: bool,
) -> CrackOutcome {
// Batch the reads: a live optical drive at 1 sector/read is glacial, and the
// crack only needs to FIND one scrambled sector whose 0x80 plaintext matches
// a known PES header. `batch_sectors` MUST be sized to the source — a drive
// rejects a READ(10) larger than its per-command max (DVD = 16) and
// `Drive::read` does not chunk, so an over-large batch fails every read and
// scans nothing. Callers pass `detect_max_batch_sectors(device_path)` for a
// live drive, a file-safe value for an image, or 1 to force per-sector.
let batch = (batch_sectors.max(1)) as u32;
// Record the LBA span the key is being cracked from so the per-title mux
// path can tell whether a later title lives in the same VTS (overlaps the
// span → key applies) or a different one (→ re-crack). Half-open [min,max).
let crack_span = extents
.iter()
.filter(|e| e.sector_count > 0)
.map(|e| (e.start_lba, e.start_lba.saturating_add(e.sector_count)))
.reduce(|(amin, amax), (bmin, bmax)| (amin.min(bmin), amax.max(bmax)));
let mut tried = 0u32;
let max_tries = 50_000;
let max_tries = 50_000u32;
let mut buf = vec![0u8; batch as usize * 2048];
let mut hb = crate::progress::Heartbeat::new("css_crack");
// Track whether ANY scrambled sector was observed. If we exhaust the scan
// budget having seen scrambled data but never recovered a key, the content
// is encrypted-but-uncrackable — a HARD failure the caller must surface,
// NOT silently treat as unencrypted (which would mux scrambled MPEG as
// plaintext → garbage at exit 0). See `CrackOutcome::ScrambledUncracked`.
let mut saw_scrambled = false;
// A read rejected with sense `05/6F/03` ("scrambled sector without
// authentication") is positive proof of CSS encryption — never collapse it
// to "unencrypted". A run of consecutive locked reads means the bus-auth
// gate is shut (it is global, so reads are all-or-nothing), so the scan
// early-bails. `consecutive_locked` resets on any readable batch, so a
// crackable title (gate open) never trips it.
let mut saw_locked = false;
let mut consecutive_locked = 0u32;
for ext in extents {
let mut i = 0;
'outer: for (extent_idx, ext) in extents.iter().enumerate() {
let mut i = 0u32;
while i < ext.sector_count && tried < max_tries {
let mut buf = vec![0u8; 2048];
if reader
.read_sectors(ext.start_lba + i, 1, &mut buf, true)
.is_ok()
&& is_scrambled(&buf)
{
if let Some(key) = crack::crack_title_key(&buf) {
return Some(CssState { title_key: key });
// Cooperative cancellation — poll once per batch, the same cadence
// sweep/patch use, so a Stop / watchdog can interrupt the scan.
if let Some(h) = halt {
if h.is_cancelled() {
break 'outer;
}
}
// Liveness beacon: a long scan over a damaged disc stays visible.
// The heartbeat is time-throttled; only when it actually beats do
// we emit the crack-specific context (tried/lba/extent_idx).
if hb.tick(tried as u64, max_tries as u64) {
tracing::debug!(
target: "freemkv::heartbeat",
phase = "css_crack",
tried,
lba = ext.start_lba + i,
extent_idx,
"scanning"
);
}
let n = (ext.sector_count - i).min(batch);
let want = n as usize * 2048;
match reader.read_sectors(ext.start_lba + i, n as u16, &mut buf[..want], true) {
Ok(_) => {
// A readable batch: the gate is open — reset the locked run.
consecutive_locked = 0;
for s in 0..n as usize {
tried += 1;
let sect = &buf[s * 2048..(s + 1) * 2048];
// Use the HARDENED pack-gated check (Fix 3): a clear stub
// sector with stray bits at 0x14 must NOT count as
// scramble evidence, or a genuinely-unencrypted title
// would falsely report ScrambledUncracked (a false E7023).
if is_scrambled_pack(sect) {
saw_scrambled = true;
if let Some(key) = stevenson::crack_title_key(sect) {
return CrackOutcome::Cracked(CssState {
title_key: key,
crack_span,
});
}
i += 1;
}
if tried >= max_tries {
break;
break 'outer;
}
}
}
// A failed batch still counts toward the budget so a damaged
// region can't loop forever. A CSS-locked failure (`05/6F/03`)
// proves encryption and, in a long enough run, means the read
// gate is shut — track it and early-bail rather than grind.
Err(e) => {
tried += n;
if e.scsi_sense().is_some_and(|s| s.is_css_locked()) {
saw_locked = true;
consecutive_locked += 1;
if consecutive_locked >= CSS_LOCKED_BAIL {
break 'outer;
}
} else {
consecutive_locked = 0;
}
}
}
i += n;
}
}
None
// Budget exhausted / extents walked / early-bailed with no key recovered.
// The disc is ENCRYPTED-but-uncracked (a hard failure on the initial scan)
// when EITHER a scrambled sector was actually seen, OR — on the initial scan
// only (`fail_on_locked`) — every read was CSS-locked (`05/6F/03`), itself
// proof of scrambling. A re-crack (`fail_on_locked` false) stays soft: a
// lapsed-AGID locked read yields None, not a hard fail, so a crackable title
// in another VTS isn't killed. Only a scan that saw neither a scrambled
// sector nor a CSS-lock is genuinely unencrypted.
if saw_scrambled || (saw_locked && fail_on_locked) {
CrackOutcome::ScrambledUncracked
} else {
CrackOutcome::Unencrypted
}
}
/// Descramble a single CSS-encrypted sector in place.
@@ -113,6 +272,616 @@ pub fn descramble_sector(state: &CssState, sector: &mut [u8]) {
}
/// Check if a sector has the CSS scramble flag set.
///
/// This is the RAW flag test — bits 4-5 of the sub-header byte 0x14 — used by
/// the descramble loop (`decrypt::decrypt_sectors`), which has already committed
/// to descrambling a known title's VOB data and only needs to skip the clear
/// NAV packs interleaved in it. For the CRACK SCAN's "did this disc actually
/// contain scrambled content?" decision (which must not false-positive on a
/// clear stub), use [`is_scrambled_pack`] instead.
pub fn is_scrambled(sector: &[u8]) -> bool {
sector.len() >= 2048 && (sector[0x14] >> 4) & 0x03 != 0
}
/// The 4-byte MPEG-2 Program Stream pack-start code (`00 00 01 BA`) every DVD
/// video sector opens with. CSS leaves the clear header (`0x00..0x80`)
/// untouched, so this signature survives scrambling.
pub(crate) const PACK_START: [u8; 4] = [0x00, 0x00, 0x01, 0xBA];
/// Check if a sector is a CSS-scrambled DVD **video pack** — the HARDENED test
/// the crack scan uses to set its `saw_scrambled` evidence flag (Fix 3).
///
/// [`is_scrambled`] keys solely on bits 4-5 of byte 0x14. That single byte is
/// only meaningful inside a real DVD sector — an MPEG-2 Program Stream pack,
/// which ALWAYS begins with the 32-bit pack-start code `00 00 01 BA` at offset
/// 0x00. A tiny clear / nav-only stub (a 0.5 s menu loop, an FBI-warning title)
/// can carry arbitrary bytes that happen to set bits 4-5 of byte 0x14; trusting
/// byte 0x14 alone there would flip the scan's `saw_scrambled` gate and make a
/// genuinely-UNENCRYPTED title report `ScrambledUncracked` — a false E7023.
///
/// Requiring the pack-start signature FIRST means only a sector that is
/// structurally a DVD video pack can be counted as scramble evidence. This does
/// NOT weaken the genuine "encrypted but uncrackable" hard-fail: a real
/// scrambled feature is made of valid PS packs, so its scrambled sectors still
/// pass this check and still drive `ScrambledUncracked` when no key cracks. (The
/// descramble loop keeps the looser [`is_scrambled`]: by the time it runs we
/// already know the title is CSS, and it only needs to skip interleaved clear
/// NAV packs — a wrongly-skipped or wrongly-included sector there is recoverable
/// per-sector, whereas a false scramble verdict in the scan poisons the whole
/// title's outcome.)
pub fn is_scrambled_pack(sector: &[u8]) -> bool {
sector.len() >= 2048 && sector[0x00..0x04] == PACK_START && (sector[0x14] >> 4) & 0x03 != 0
}
#[cfg(test)]
mod tests {
use super::*;
use crate::error::{Error, Result};
// ── is_scrambled ───────────────────────────────────────────────────────
/// is_scrambled returns false for any buffer shorter than one sector,
/// WITHOUT indexing byte 0x14 (which would panic on a tiny buffer). The
/// length guard is short-circuited before the flag read.
///
/// Grounding: `sector.len() >= 2048 && (sector[0x14] >> 4) & 0x03 != 0` —
/// `&&` short-circuits so a 20-byte buffer never reads index 0x14.
/// Mutation: swap the operands so the flag is read first
/// (`(sector[0x14]...) && sector.len() >= 2048`) -> panics indexing a
/// 20-byte slice; this test catches it.
#[test]
fn is_scrambled_short_buffer_is_false_no_panic() {
assert!(!is_scrambled(&[]));
assert!(!is_scrambled(&[0u8; 20])); // shorter than 0x14+1 even
assert!(!is_scrambled(&[0xFFu8; 2047])); // one byte short of a sector
}
/// is_scrambled keys on bits 4-5 of byte 0x14 (the CSS scramble field).
/// A full sector flagged 0x10/0x20/0x30 is scrambled; 0x00 and the
/// high-bit-only values 0x40/0x80 are clear.
///
/// Grounding: `(sector[0x14] >> 4) & 0x03`.
/// Mutation: widen mask to `& 0x0F` -> 0x40 reports scrambled, the 0x40
/// assert fails.
#[test]
fn is_scrambled_uses_bits_4_5_only() {
let mut s = vec![0u8; 2048];
for (flag, expected) in [
(0x00u8, false),
(0x10, true),
(0x20, true),
(0x30, true),
(0x40, false),
(0x80, false),
(0xC0, false),
(0xFF, true), // bits 4-5 set within 0xFF
] {
s[0x14] = flag;
assert_eq!(
is_scrambled(&s),
expected,
"flag byte {flag:#04x} scramble detection"
);
}
}
/// is_scrambled accepts exactly 2048 bytes as the minimum (boundary at the
/// inclusive value 2048).
///
/// Grounding: `sector.len() >= 2048`.
/// Mutation: change `>= 2048` to `> 2048` -> an exact 2048-byte scrambled
/// sector reports false; this fails.
#[test]
fn is_scrambled_exact_sector_length_accepted() {
let mut s = vec![0u8; 2048];
s[0x14] = 0x30;
assert!(is_scrambled(&s), "exactly 2048 bytes must be eligible");
}
/// Fix 3 hardening: `is_scrambled_pack` (the crack-scan evidence gate)
/// requires BOTH the MPEG-PS pack-start code at 0x00 AND the 0x14 scramble
/// bits. A clear / nav-only stub whose bytes happen to set bits 4-5 of 0x14
/// but lacks the pack-start is NOT counted as scramble evidence — without
/// this the scan flips `saw_scrambled` and a genuinely unencrypted title
/// reports `ScrambledUncracked` (the false E7023). The looser `is_scrambled`
/// (descramble gate) still reads the same sector as flagged.
///
/// Grounding: `sector[0x00..0x04] == 00 00 01 BA && (sector[0x14] >> 4)...`.
/// Mutation: drop the pack-start clause -> the 0x14-only sector counts as a
/// scrambled pack; the first assert fails.
#[test]
fn is_scrambled_pack_requires_pack_start_signature() {
let mut s = vec![0u8; 2048];
s[0x14] = 0x30; // scramble bits set, but no pack-start at 0x00
assert!(
!is_scrambled_pack(&s),
"0x14 bits without the MPEG-PS pack-start must NOT count as a scrambled pack"
);
// The looser descramble-gate check still sees the raw flag.
assert!(is_scrambled(&s), "is_scrambled keys on the 0x14 flag alone");
// A near-miss pack-start (wrong final byte) is still rejected.
s[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBB]);
assert!(
!is_scrambled_pack(&s),
"a wrong pack-start byte must not qualify"
);
// The real signature flips it to a scrambled pack.
s[0x00..0x04].copy_from_slice(&PACK_START);
assert!(
is_scrambled_pack(&s),
"valid pack-start + 0x14 bits → scrambled pack"
);
}
// ── crack_key scanning over a mock SectorSource ────────────────────────
/// Records every (lba, count) read; returns a caller-supplied flag byte at
/// 0x14 so we can drive scrambled/clear sectors, or an injected error.
struct MockSource {
reads: std::cell::RefCell<Vec<u32>>,
flag_byte: u8,
fail_all: bool,
/// Every read fails with CSS-locked sense `05/6F/03` (drive refusing
/// scrambled reads because the bus-auth gate isn't open).
lock_all: bool,
/// When set, the sector at `crackable.0` is served as a full
/// Stevenson-crackable scrambled sector (`crackable.1`, 2048 bytes)
/// instead of the uniform `flag_byte` fill. Lets the scan actually
/// reach `CrackOutcome::Cracked` from a synthetic ISO.
crackable: Option<(u32, Vec<u8>)>,
}
impl MockSource {
fn new(flag_byte: u8) -> Self {
Self {
reads: std::cell::RefCell::new(Vec::new()),
flag_byte,
fail_all: false,
lock_all: false,
crackable: None,
}
}
}
/// Build a Stevenson-crackable scrambled sector for `(title_key, seed)`:
/// the cleartext header (0x59..0x80) carries a periodic run that continues
/// across the 0x80 boundary into the encrypted region — the crib
/// `stevenson::crack_title_key` recovers a key from. Mirrors the
/// `synth_periodic_sector` fixture in the stevenson tests but built here
/// from the crate-internal `scramble_sector`.
fn crackable_sector(title_key: &[u8; 5], seed: &[u8; 5], period: usize) -> Vec<u8> {
const RUN_START: usize = 0x59;
const SEED_OFFSET: usize = 0x54;
let mut plaintext = vec![0u8; 2048];
plaintext[0x00..0x04].copy_from_slice(&PACK_START); // valid DVD pack header
plaintext[0x14] = 0x10; // scramble flag
let pat: Vec<u8> = (0..period)
.map(|k| (0xA0u8.wrapping_add(k as u8)) ^ 0x5A)
.collect();
for (i, b) in plaintext.iter_mut().enumerate().skip(RUN_START) {
*b = pat[i % period];
}
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
lfsr::scramble_sector(title_key, &mut plaintext);
plaintext
}
impl SectorSource for MockSource {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
self.reads.borrow_mut().push(lba);
if self.lock_all {
return Err(Error::DiscRead {
sector: lba as u64,
status: Some(2),
sense: Some(crate::scsi::ScsiSense {
sense_key: 0x05,
asc: 0x6F,
ascq: 0x03,
}),
});
}
if self.fail_all {
return Err(Error::DecryptFailed);
}
let n = count as usize * 2048;
let end = n.min(buf.len());
for b in buf[..end].iter_mut() {
*b = 0;
}
// Fill each sector in the batch with the uniform flag byte, EXCEPT a
// designated crackable LBA which gets the full synthetic sector.
for s in 0..count as u32 {
let sect_lba = lba + s;
let base = s as usize * 2048;
if base + 2048 > end {
break;
}
match &self.crackable {
Some((clba, sector)) if *clba == sect_lba => {
buf[base..base + 2048].copy_from_slice(sector);
}
_ => {
// Real DVD video sectors always open with the MPEG-PS
// pack-start code; `is_scrambled` (Fix 3) requires it
// before trusting the 0x14 scramble bits, so the fixture
// must include it for a `flag_byte` of 0x30 to register
// as scrambled.
buf[base..base + 4].copy_from_slice(&PACK_START);
buf[base + 0x14] = self.flag_byte;
}
}
}
Ok(n)
}
}
/// crack_key caps total scanned sectors at 50_000 even when extents are
/// far larger, and counts EVERY scanned sector (clear ones included)
/// toward the budget. With one 200_000-sector extent of clear sectors, it
/// must read exactly 50_000 sectors and return None — never run away.
///
/// Grounding: `let max_tries = 50_000; ... tried += 1` before the read,
/// loop guard `tried < max_tries`.
/// Mutation: change `50_000` to `500_000` -> read count exceeds 50_000;
/// the exact-count assert fails. Removing the `tried += 1` increment ->
/// would read all 200_000; also fails.
#[test]
fn crack_key_caps_total_tries_at_50000() {
let mut src = MockSource::new(0x00); // clear sectors, never a hit
let extents = [Extent {
start_lba: 0,
sector_count: 200_000,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none(), "clear sectors yield no key");
assert_eq!(
src.reads.borrow().len(),
50_000,
"scan must stop at the 50_000-sector budget"
);
}
// ── CrackOutcome: scrambled-but-uncracked vs genuinely unencrypted (Fix 6) ─
/// A scan over CLEAR sectors (scramble flag never set) returns
/// `Unencrypted` — the content is genuinely plaintext, so proceeding
/// without a key is correct.
#[test]
fn crack_outcome_clear_sectors_is_unencrypted() {
let mut src = MockSource::new(0x00); // never scrambled
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
matches!(outcome, CrackOutcome::Unencrypted),
"no scrambled sector seen → Unencrypted, got {outcome:?}"
);
// The Option wrapper collapses Unencrypted → None.
assert!(crack_key(&mut MockSource::new(0x00), &extents, 1).is_none());
}
/// THE Fix 6 regression: a scan that SEES scrambled sectors (flag set) but
/// recovers no key (the mock's zeroed data has no Stevenson crib) must
/// return `ScrambledUncracked` — a HARD failure — NOT `Unencrypted`. The
/// old code conflated this with "unencrypted" and muxed scrambled MPEG as
/// plaintext (garbage at exit 0).
#[test]
fn crack_outcome_scrambled_uncracked_is_hard_failure() {
let mut src = MockSource::new(0x30); // scrambled flag set, no crackable crib
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
outcome.is_scrambled_uncracked(),
"scrambled sectors seen but no key → ScrambledUncracked, got {outcome:?}"
);
// The legacy Option wrapper still collapses this to None (the callers
// that need the distinction now use crack_key_outcome instead).
assert!(crack_key(&mut MockSource::new(0x30), &extents, 1).is_none());
}
/// Even when every read FAILS, a scan that never managed to observe a
/// scrambled sector reports `Unencrypted` (we cannot prove encryption from
/// unreadable data alone — the AACS/keydb paths and the disc-level
/// `css_error` plumbing cover genuinely unreadable encrypted discs).
#[test]
fn crack_outcome_all_reads_fail_is_unencrypted() {
let mut src = MockSource::new(0x30);
src.fail_all = true; // no sector is ever inspected
let extents = [Extent {
start_lba: 0,
sector_count: 10,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
matches!(outcome, CrackOutcome::Unencrypted),
"no readable scrambled sector → Unencrypted, got {outcome:?}"
);
}
/// Fix C (rc.5.1): on the INITIAL scan, a drive that refuses every read with
/// CSS-locked sense (`05/6F/03`) is encrypted-but-locked →
/// `ScrambledUncracked` (a hard failure), NOT `Unencrypted`. This is the
/// rc4.3 bug: every VOB read came back `6F/03`, so the scan saw no scrambled
/// sector and wrongly declared the disc unencrypted → 19 KB garbage.
#[test]
fn crack_outcome_css_locked_initial_is_scrambled_uncracked() {
let mut src = MockSource::new(0x30);
src.lock_all = true; // every read → 05/6F/03
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
outcome.is_scrambled_uncracked(),
"every read 6F/03 on the initial scan → ScrambledUncracked, got {outcome:?}"
);
}
/// MISSING #1 guard: the re-crack path (the `Option`-returning `crack_key`,
/// `fail_on_locked == false`) must NOT hard-fail on a CSS-locked read — it
/// returns `None`. A lapsed-AGID re-crack of another VTS stays soft so a
/// genuinely crackable title isn't killed by a transient locked read.
#[test]
fn crack_key_recrack_locked_is_none_not_hard_fail() {
let mut src = MockSource::new(0x30);
src.lock_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
assert!(crack_key(&mut src, &extents, 1).is_none());
}
/// Fix F: a fully CSS-locked scan early-bails near `CSS_LOCKED_BAIL`
/// consecutive locked reads instead of grinding the whole 50_000-sector
/// budget (the rc5 "stuck Scanning…" hang on a wedged bridge).
#[test]
fn crack_css_locked_scan_early_bails() {
let mut src = MockSource::new(0x30);
src.lock_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 10_000,
}];
let _ = crack_key_outcome(&mut src, &extents, 1, None);
let n = src.reads.borrow().len();
assert!(
n <= (CSS_LOCKED_BAIL as usize) + 1,
"locked scan early-bails near {CSS_LOCKED_BAIL}, not 10000; read {n}"
);
}
/// The budget spans ALL extents, not per-extent: two extents summing past
/// the cap must still stop at 50_000 total reads.
///
/// Grounding: `tried` is declared outside the `for ext in extents` loop;
/// `if tried >= max_tries { break }` after each extent.
/// Mutation: move `let mut tried = 0` inside the extent loop -> each extent
/// gets its own 50_000 budget; total reads would be 80_000, this fails.
#[test]
fn crack_key_budget_is_shared_across_extents() {
let mut src = MockSource::new(0x00);
let extents = [
Extent {
start_lba: 0,
sector_count: 40_000,
},
Extent {
start_lba: 100_000,
sector_count: 40_000,
},
];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
50_000,
"the 50_000 budget is shared across all extents"
);
}
/// crack_key scans sequentially from each extent's start_lba. The first
/// reads must be at the extent's start_lba, start_lba+1, ... pinning the
/// LBA arithmetic `ext.start_lba + i`.
///
/// Grounding: `reader.read_sectors(ext.start_lba + i, 1, ...)`.
/// Mutation: change `ext.start_lba + i` to just `i` -> the recorded LBAs
/// would start at 0, not 5000; this fails.
#[test]
fn crack_key_scans_from_extent_start_lba() {
let mut src = MockSource::new(0x00);
let extents = [Extent {
start_lba: 5_000,
sector_count: 4,
}];
let _ = crack_key(&mut src, &extents, 1);
let reads = src.reads.borrow();
assert_eq!(
&reads[..],
&[5_000, 5_001, 5_002, 5_003],
"sequential scan from start_lba"
);
}
/// A read error on a sector does NOT abort the scan: crack_key keeps
/// scanning subsequent sectors (the error sector still counts toward the
/// budget). With a small failing extent, every sector is attempted and the
/// function returns None.
///
/// Grounding: `if reader.read_sectors(...).is_ok() && is_scrambled(...)` —
/// an Err simply falls through to `i += 1`.
/// Mutation: change the read-error handling to `reader.read_sectors(...)?`
/// (propagate) -> crack_key would stop after the first error and read only
/// 1 sector; this asserts all 10 were attempted.
#[test]
fn crack_key_continues_past_read_errors() {
let mut src = MockSource::new(0x30);
src.fail_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 10,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
10,
"read errors must not abort the scan"
);
}
/// Empty extents (no sectors) -> crack_key reads nothing and returns None.
/// A zero-sector extent must not read its start_lba.
///
/// Grounding: `while i < ext.sector_count` with sector_count == 0 never
/// enters.
/// Mutation: change `i < ext.sector_count` to `i <= ext.sector_count` ->
/// one spurious read at start_lba; this asserts zero reads.
#[test]
fn crack_key_empty_extent_reads_nothing() {
let mut src = MockSource::new(0x30);
let extents = [Extent {
start_lba: 42,
sector_count: 0,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
0,
"zero-sector extent reads nothing"
);
}
/// No extents at all -> immediate None, zero reads.
///
/// Grounding: `for ext in extents` over an empty slice is a no-op.
/// Mutation: any change that reads before the loop would break this.
#[test]
fn crack_key_no_extents_is_none() {
let mut src = MockSource::new(0x30);
let res = crack_key(&mut src, &[], 1);
assert!(res.is_none());
assert_eq!(src.reads.borrow().len(), 0);
}
// ── Scan-level Cracked branch + per-VTS re-crack success (audit §2 / §5 #8) ─
/// SCAN-LEVEL CRACKED (audit gap "MockSource never yields a crackable
/// sector"): drive the full `crack_key_scan` over a synthetic ISO whose
/// scan hits a Stevenson-crackable scrambled sector. The outcome must be
/// `CrackOutcome::Cracked` with a key that round-trips the sector, AND the
/// `crack_span` must be recorded as the half-open extent span (the per-VTS
/// routing key the mux path needs). Previously only the leaf crack and the
/// Uncracked/Unencrypted branches were tested — the Cracked branch and
/// `crack_span` recording were never exercised end-to-end.
#[test]
fn crack_outcome_reaches_cracked_with_span() {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let seed = [0x11, 0x22, 0x33, 0x44, 0x55];
let crackable = crackable_sector(&title_key, &seed, 8);
// The crackable sector sits a few sectors into the extent.
let mut src = MockSource::new(0x00); // surrounding sectors: clear
src.crackable = Some((1003, crackable.clone()));
let extents = [Extent {
start_lba: 1000,
sector_count: 50,
}];
let outcome = crack_key_outcome(&mut src, &extents, 4, None);
let state = match outcome {
CrackOutcome::Cracked(s) => s,
other => panic!("expected Cracked, got {other:?}"),
};
// The recovered key descrambles the crackable sector body.
let mut test = crackable.clone();
descramble_sector(&state, &mut test);
let mut plain = crackable;
lfsr::descramble_sector(&title_key, &mut plain);
assert_eq!(
&test[0x80..],
&plain[0x80..],
"recovered key must round-trip the scrambled sector body"
);
// crack_span = half-open [start, start+count) of the scanned extent.
assert_eq!(
state.crack_span,
Some((1000, 1050)),
"crack_span must record the extent LBA span for per-VTS routing"
);
}
/// CSS_ERROR WIRING (audit §2 / §5 #7): an all-locked synthetic ISO (every
/// VOB read returns CSS-locked sense `05/6F/03` across MULTIPLE extents, as a
/// real encrypted-but-unauthenticated disc image does) must produce the exact
/// outcome the scan converts into `disc.css_error = Some(Error::CssKeyMissing)`
/// — i.e. `CrackOutcome::ScrambledUncracked` / `is_scrambled_uncracked()`,
/// NOT `Unencrypted`. disc/mod.rs's `crack_key_outcome → ScrambledUncracked`
/// arm (where it stamps css_error) is driven by exactly this signal, so this
/// pins the css-layer contract that arm depends on without touching the
/// scan plumbing.
#[test]
fn all_locked_synthetic_iso_yields_css_key_missing_signal() {
let mut src = MockSource::new(0x30);
src.lock_all = true; // every read → 05/6F/03 across the whole "ISO"
let extents = [
Extent {
start_lba: 0,
sector_count: 30,
},
Extent {
start_lba: 5_000,
sector_count: 30,
},
];
let outcome = crack_key_outcome(&mut src, &extents, 16, None);
assert!(
outcome.is_scrambled_uncracked(),
"all-locked ISO → ScrambledUncracked (the css_error=CssKeyMissing \
signal), got {outcome:?}"
);
// The legacy Option wrapper still collapses it to None — callers that
// surface the hard error must use crack_key_outcome, which this proves.
let mut src2 = MockSource::new(0x30);
src2.lock_all = true;
assert!(crack_key(&mut src2, &extents, 16).is_none());
}
/// PER-VTS RE-CRACK SUCCESS (audit gap "success path missing"): the prior
/// re-crack test only covered the locked→None path. Here a re-crack
/// (`crack_key`, `fail_on_locked == false`) over a DIFFERENT VTS's extents
/// finds that VTS's own crackable sector and returns a `CssState` whose
/// `crack_span` matches the new extents — proving a key cracked for one VTS
/// is genuinely re-derived (not reused) for another.
#[test]
fn recrack_succeeds_on_other_vts_extents() {
let title_key = [0xFE, 0xDC, 0xBA, 0x98, 0x76];
let seed = [0x00, 0xFF, 0x80, 0x7F, 0x01];
let crackable = crackable_sector(&title_key, &seed, 5);
let mut src = MockSource::new(0x00);
// The second VTS lives at a disjoint LBA range; its crackable sector is
// the first one in the extent.
src.crackable = Some((9000, crackable));
let other_vts = [Extent {
start_lba: 9000,
sector_count: 20,
}];
let state = crack_key(&mut src, &other_vts, 4).expect("re-crack must recover a key");
assert_eq!(
state.crack_span,
Some((9000, 9020)),
"re-crack span must reflect the OTHER VTS extents, not a reused span"
);
}
}
+593
View File
@@ -0,0 +1,593 @@
//! CSS title-key recovery — Frank A. Stevenson's divide-and-conquer attack
//! (1999), ported exactly from libdvdcss `RecoverTitleKey` + `AttackPattern`
//! (css.c).
//!
//! Recovers the 5-byte CSS title key from a single scrambled DVD sector with
//! no player keys and no disc-key crack, using only known plaintext.
//!
//! # The cipher this attacks
//!
//! The content descrambler ([`super::lfsr::descramble_sector`], = libdvdcss
//! `dvdcss_unscramble`) seeds its two LFSRs **directly** from
//! `key = title_key XOR sector_seed` (seed = `sector[0x54..0x59]`):
//!
//! ```text
//! i_t1 = (key[0] ^ sec[0x54]) | 0x100; // LFSR1 low (9-bit)
//! i_t2 = key[1] ^ sec[0x55]; // LFSR1 high
//! i_t3 = (key[2]|key[3]<<8|key[4]<<16) ^ seed3; // LFSR0 (24-bit feedback)
//! i_t3 = i_t3*2 + 8 - (i_t3 & 7);
//! // per byte: *p = TAB1[*p] ^ (i_t5 & 0xff)
//! ```
//!
//! There is NO `decrypt_key` mangling on the content path. So the recovery
//! is a single inversion of `dvdcss_unscramble`, not the multi-stage
//! working-key inversion the previous (non-CSS) implementation used.
//!
//! # The attack
//!
//! 1. **Known plaintext → keystream.** Because the descramble applies TAB1
//! to the ciphertext, the per-byte keystream is
//! `buf[i] = TAB1[cipher[i]] ^ plain[i]` (matching libdvdcss
//! `RecoverTitleKey`'s `p_buffer`).
//! 2. **Brute the 16-bit LFSR1 seed.** For each of 2^16 seeds, run LFSR1
//! forward; for the first four steps deduce the LFSR0 output bytes from
//! the keystream (carry-tracked), reconstructing `i_t3`. For the next six
//! steps clock LFSR0 normally and check it reproduces the keystream — a
//! wrong LFSR1 seed fails fast.
//! 3. **Back-clock LFSR0.** Run four backward `i_t3` steps (each a 256-way
//! search for the byte shifted in) to reach the initial state, then undo
//! `i_t3 = i_t3*2 + 8 - (i_t3 & 7)` to recover key[2..5].
//! 4. **XOR back the seed.** `key[0..5] ^= sector_seed[0..5]` (plain XOR —
//! the descramble seeds directly, so there is no inversion).
//!
//! `AttackPattern` finds known plaintext for step 1: the longest periodic
//! run in the cleartext `sec[0x00..0x80]`, assumed to continue into the
//! encrypted region at 0x80.
use super::lfsr::descramble_sector;
use super::tables::{TAB1, TAB2, TAB3, TAB4, TAB5};
use crate::consts::SECTOR_BYTES;
const ENCRYPTED_START: usize = 0x80; // byte 128
const SEED_OFFSET: usize = 0x54; // sector seed at bytes 0x54-0x58
const FLAG_BYTE: usize = 0x14;
/// RecoverTitleKey: recover the title key from cipher + known plaintext.
///
/// Exact port of libdvdcss `RecoverTitleKey` (css.c). `crypted` is the
/// ciphertext starting at sector byte 0x80; `decrypted` is the matching
/// known plaintext; `seed` is `sector[0x54..0x59]`. On success returns the
/// recovered 5-byte title key; `None` if no LFSR seed reproduces the
/// keystream.
///
/// At least 10 bytes of `crypted`/`decrypted` are required (the cipher is
/// iterated 10 times: 4 to reconstruct LFSR0, 6 to validate).
fn recover_title_key_from_plain(
crypted: &[u8],
decrypted: &[u8],
seed: &[u8; 5],
) -> Option<[u8; 5]> {
if crypted.len() < 10 || decrypted.len() < 10 {
return None;
}
// buf[i] = TAB1[cipher[i]] ^ plain[i] — the per-byte content keystream.
let mut buffer = [0u8; 10];
for (i, b) in buffer.iter_mut().enumerate() {
*b = TAB1[crypted[i] as usize] ^ decrypted[i];
}
let mut key = [0u8; 5];
let mut found = false;
for i_try in 0u32..0x1_0000 {
let mut i_t1 = (i_try >> 8) | 0x100;
let mut i_t2 = i_try & 0xff;
let mut i_t3: u32 = 0; // not needed yet
let mut i_t5: u32 = 0;
// Iterate the cipher 4 times to reconstruct LFSR0 (i_t3).
for &b in buffer.iter().take(4) {
let i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
let i_t4 = TAB5[i_t4 as usize] as u32;
// Deduce i_t6 (LFSR0 output, pre-TAB4) and the carry.
let mut i_t6 = b as u32;
if i_t5 != 0 {
i_t6 = (i_t6 + 0xff) & 0xff;
}
if i_t6 < i_t4 {
i_t6 += 0x100;
}
i_t6 -= i_t4;
i_t5 += i_t6 + i_t4;
let i_t6 = TAB4[i_t6 as usize] as u32;
i_t3 = (i_t3 << 8) | i_t6;
i_t5 >>= 8;
}
let i_candidate = i_t3;
// Iterate 6 more times to validate the candidate.
let mut i = 4usize;
while i < 10 {
let i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
let i_t4 = TAB5[i_t4 as usize] as u32;
let mut i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xff;
i_t3 = (i_t3 << 8) | i_t6;
i_t6 = TAB4[i_t6 as usize] as u32;
i_t5 += i_t6 + i_t4;
if (i_t5 & 0xff) as u8 != buffer[i] {
break;
}
i_t5 >>= 8;
i += 1;
}
if i != 10 {
continue;
}
// Four backward steps of iterating i_t3 to deduce the initial state.
i_t3 = i_candidate;
for _ in 0..4 {
let i_t1_byte = i_t3 & 0xff;
i_t3 >>= 8;
// Brute-force the byte shifted in (top byte of the 24-bit reg).
for j in 0u32..256 {
i_t3 = (i_t3 & 0x1_ffff) | (j << 17);
let i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xff;
if i_t6 == i_t1_byte {
break;
}
}
}
// Undo `i_t3 = i_t3*2 + 8 - (i_t3 & 7)` to recover key[2..5].
let i_t4 = (i_t3 >> 1).wrapping_sub(4);
for i_t5 in 0u32..8 {
let val = i_t4.wrapping_add(i_t5);
if val.wrapping_mul(2).wrapping_add(8).wrapping_sub(val & 7) == i_t3 {
key[0] = (i_try >> 8) as u8;
key[1] = (i_try & 0xff) as u8;
key[2] = (val & 0xff) as u8;
key[3] = ((val >> 8) & 0xff) as u8;
key[4] = ((val >> 16) & 0xff) as u8;
found = true;
break;
}
}
// First fully-validated candidate wins. The 48-bit keystream constraint
// makes a second match cryptographically negligible on real sectors, but
// continuing would let a later spurious match overwrite a correct key.
if found {
break;
}
}
if found {
for (k, &s) in key.iter_mut().zip(seed.iter()) {
*k ^= s;
}
Some(key)
} else {
None
}
}
/// Recover the CSS title key from a scrambled sector using a known plaintext
/// for the encrypted region.
///
/// `plain` is the expected plaintext at byte 0x80 (at least 10 bytes).
/// Returns the recovered key only if it actually descrambles the sector back
/// to `plain` — guarding against the rare spurious LFSR-seed match.
pub fn recover_title_key(sector: &[u8], plain: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_BYTES || plain.len() < 10 {
return None;
}
if sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
let seed: [u8; 5] = [
sector[SEED_OFFSET],
sector[SEED_OFFSET + 1],
sector[SEED_OFFSET + 2],
sector[SEED_OFFSET + 3],
sector[SEED_OFFSET + 4],
];
let crypted = &sector[ENCRYPTED_START..ENCRYPTED_START + 10];
let key = recover_title_key_from_plain(crypted, plain, &seed)?;
if descramble_matches(sector, &key, plain) {
Some(key)
} else {
None
}
}
/// Verify a title key by descrambling a copy of `sector` and checking the
/// known plaintext reappears at byte 0x80.
fn descramble_matches(sector: &[u8], title: &[u8; 5], plain: &[u8]) -> bool {
let mut test = sector.to_vec();
test[FLAG_BYTE] |= 0x10; // ensure scramble flag set for the descrambler
descramble_sector(title, &mut test);
let n = plain.len().min(SECTOR_BYTES - ENCRYPTED_START);
test[ENCRYPTED_START..ENCRYPTED_START + n] == plain[..n]
}
/// AttackPattern: find a repeating pattern just before the encrypted region
/// and assume the plaintext at 0x80 continues it.
///
/// Functionally-equivalent port of libdvdcss `AttackPattern` (css.c) — finds the
/// same periodic cribs on real DVD data, though its byte-comparison anchor
/// differs from the C on phase-misaligned runs. Scans cleartext
/// `sec[0x00..0x80]` for the longest run that repeats with a cycle length in
/// 2..0x2F. If the run is long enough (`plen > 3` and at least two full
/// cycles), the known plaintext at 0x80 is taken to be the periodic run
/// continuing forward, and [`recover_title_key_from_plain`] is applied.
pub fn crack_title_key(sector: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_BYTES {
return None;
}
if sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
// Runaway guard: a single sector's crack is a bounded 2^16 LFSR search and
// should finish in well under a second on any modern CPU. If it ever
// exceeds ~2s wall-clock, something pathological is happening — log it so a
// hang is never silent.
let crack_t0 = std::time::Instant::now();
let result = crack_title_key_inner(sector);
let elapsed = crack_t0.elapsed();
if elapsed.as_secs_f64() > 2.0 {
tracing::warn!(
target: "freemkv::css",
elapsed_ms = elapsed.as_millis() as u64,
found = result.is_some(),
"css crack: single-sector recovery exceeded 2s (runaway guard)"
);
}
result
}
/// Inner body of [`crack_title_key`] — the actual AttackPattern search. Split
/// out so the public entry point can wall-clock the whole attempt for the
/// runaway guard without threading a timer through every return path.
/// AttackPattern crib: the predicted 10-byte plaintext at byte 0x80.
///
/// Scans the clear header `sec[0x00..0x80]` (never scrambled) for the longest
/// run that repeats with a cycle length in 2..0x2F. If the run is long enough
/// (`plen > 3` and at least two full cycles), the plaintext at 0x80 is taken to
/// be that periodic run continuing forward. Returns `None` for an unscrambled
/// sector or one with no usable run — such a sector can be neither cracked nor
/// key-validated, only descrambled with an externally-cached key.
///
/// The header is untouched by `descramble_sector`, so the crib is identical
/// before and after descramble: the decrypt path uses it as a per-sector
/// "did the cached key descramble correctly?" oracle (the predicted plaintext
/// must reappear at 0x80), and the cracker uses it as its known plaintext.
pub(crate) fn attack_crib(sector: &[u8]) -> Option<[u8; 10]> {
if sector.len() < SECTOR_BYTES || sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
let mut best_plen: usize = 0;
let mut best_p: usize = 0;
// For all cycle lengths from 2 to 0x2F.
for i in 2usize..0x30 {
// Count bytes that repeat with cycle length i, scanning backward from
// 0x7F. `sec[0x7F - (j % i)] == sec[0x7F - j]`.
let mut j = i + 1;
while j < 0x80 && sector[0x7f - (j % i)] == sector[0x7f - j] {
if j > best_plen {
best_plen = j;
best_p = i;
}
j += 1;
}
}
// Need at least a few repeated bytes and at least one full cycle.
if best_plen > 3 && best_p > 0 && best_plen / best_p >= 2 {
// The known plaintext is the periodic run continuing past 0x80. The
// crib starts at `0x80 - (best_plen/best_p)*best_p` and continues
// through the encrypted region; the bytes at and after 0x80 are the
// predicted plaintext (the pattern repeats with period best_p).
let cycles = best_plen / best_p;
let plain_start = 0x80 - cycles * best_p;
// Each predicted byte is the run sample one or more periods back:
// `sec[plain_start + (i % best_p)]`. For in-run offsets
// (`plain_start + i < 0x80`) the run is exactly periodic, so this
// equals `sec[plain_start + i]`; for offsets at/after 0x80 the raw
// byte is ciphertext, so we MUST wrap within the period rather than
// read it. (Reading `&sec[plain_start..+10]` directly — as before —
// pulled ciphertext into the crib whenever the run covered fewer than
// 10 bytes before 0x80, producing false-negative key recovery.)
let mut plain = [0u8; 10];
for (i, p) in plain.iter_mut().enumerate() {
*p = sector[plain_start + (i % best_p)];
}
Some(plain)
} else {
None
}
}
fn crack_title_key_inner(sector: &[u8]) -> Option<[u8; 5]> {
let plain = attack_crib(sector)?;
let seed: [u8; 5] = [
sector[SEED_OFFSET],
sector[SEED_OFFSET + 1],
sector[SEED_OFFSET + 2],
sector[SEED_OFFSET + 3],
sector[SEED_OFFSET + 4],
];
let crypted = &sector[0x80..0x80 + 10];
if let Some(key) = recover_title_key_from_plain(crypted, &plain, &seed) {
// Verify against the same predicted plaintext.
if descramble_matches(sector, &key, &plain) {
return Some(key);
}
}
None
}
#[cfg(test)]
mod tests {
use super::super::lfsr::scramble_sector;
use super::*;
/// Build a synthetic scrambled sector for a given title key and seed,
/// with `plain` placed as the plaintext at byte 0x80, scrambled with
/// EXACTLY the cipher `descramble_sector` inverts. Returns
/// (scrambled_sector, full_plaintext_body).
fn synth_sector(title_key: &[u8; 5], seed: &[u8; 5], plain: &[u8]) -> (Vec<u8>, Vec<u8>) {
let mut plaintext = vec![0u8; SECTOR_BYTES];
plaintext[0..4].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
plaintext[FLAG_BYTE] = 0x10;
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
plaintext[ENCRYPTED_START..ENCRYPTED_START + plain.len()].copy_from_slice(plain);
let body = plaintext.clone();
// scramble_sector turns the plaintext body into ciphertext and sets
// the scramble flag.
scramble_sector(title_key, &mut plaintext);
(plaintext, body)
}
/// Build a synthetic scrambled sector whose CLEARTEXT (0x00..0x80) ends
/// in a periodic run that continues into the encrypted region — the case
/// `AttackPattern` (crack_title_key) is designed to crack.
fn synth_periodic_sector(
title_key: &[u8; 5],
seed: &[u8; 5],
period: usize,
) -> (Vec<u8>, Vec<u8>) {
let mut plaintext = vec![0u8; SECTOR_BYTES];
plaintext[FLAG_BYTE] = 0x10;
// A clean periodic run occupying the tail of the cleartext header
// (RUN_START..0x80) and continuing into the encrypted region. This
// mirrors a real VOB: a periodic data run just before the scrambled
// part. The run must NOT overlap the seed bytes (0x54..0x59), or the
// AttackPattern detector would break mid-run. The phase is anchored to
// offset 0 so the run is consistent across the 0x80 boundary.
// Just above the seed (0x54..0x59); gives a 39-byte run (0x59..0x80)
// — enough for >=2 cycles of every tested period (<=19).
const RUN_START: usize = 0x59;
let pat: Vec<u8> = (0..period)
.map(|k| (0xA0u8.wrapping_add(k as u8)) ^ 0x5A)
.collect();
for (i, b) in plaintext.iter_mut().enumerate().skip(RUN_START) {
*b = pat[i % period];
}
// Seed sits below the run, undisturbed.
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
let body = plaintext.clone();
scramble_sector(title_key, &mut plaintext);
(plaintext, body)
}
#[test]
fn crack_unscrambled_returns_none() {
let sector = vec![0u8; SECTOR_BYTES];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn crack_too_short_returns_none() {
let sector = vec![0u8; 100];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn recover_needs_min_plain() {
let sector = vec![0u8; SECTOR_BYTES];
let short_plain = [0u8; 4];
assert!(recover_title_key(&sector, &short_plain).is_none());
}
/// The known plaintext used at byte 0x80 for the direct-recovery tests.
/// A realistic MPEG-2 PES header start.
const PES: [u8; 10] = [0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x80, 0x05, 0x21];
/// MANDATORY round-trip (Task C.1): synthesize a scrambled sector for a
/// known (title_key, seed), then assert recover_title_key returns a key
/// that descrambles the body back to plaintext. CSS title-key recovery is
/// well-defined up to keys that scramble identically; we assert the full
/// body round-trips (the true correctness property), and additionally
/// that the EXACT key is returned for the common case.
#[test]
fn recover_round_trips_known_keys() {
let cases: &[([u8; 5], [u8; 5])] = &[
(
[0x42, 0x13, 0x37, 0xBE, 0xEF],
[0x11, 0x22, 0x33, 0x44, 0x55],
),
(
[0x01, 0x02, 0x03, 0x04, 0x05],
[0xDE, 0xAD, 0xBE, 0xEF, 0x42],
),
(
[0xFE, 0xDC, 0xBA, 0x98, 0x76],
[0x00, 0xFF, 0x80, 0x7F, 0x01],
),
(
[0x9A, 0x78, 0x56, 0x34, 0x12],
[0xA5, 0x5A, 0x0F, 0xF0, 0xCC],
),
(
[0xFF, 0xFF, 0xFF, 0xFF, 0xFF],
[0x01, 0x01, 0x01, 0x01, 0x01],
),
];
for (title_key, seed) in cases {
let (mut sector, body) = synth_sector(title_key, seed, &PES);
let recovered =
recover_title_key(&sector, &PES).expect("recover_title_key returned None");
descramble_sector(&recovered, &mut sector);
assert_eq!(
&sector[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"recovered key did not descramble the full body for \
title={title_key:02x?} seed={seed:02x?}"
);
}
}
/// MANDATORY (Task C.1): the AttackPattern entry point crack_title_key —
/// no plaintext supplied — recovers a round-tripping key when the
/// cleartext ends in a periodic run that continues into 0x80.
#[test]
fn crack_title_key_recovers_via_attack_pattern() {
for &period in &[2usize, 3, 5, 8, 16] {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let seed = [0x11, 0x22, 0x33, 0x44, 0x55];
let (sector, body) = synth_periodic_sector(&title_key, &seed, period);
let cracked = crack_title_key(&sector)
.unwrap_or_else(|| panic!("crack_title_key returned None for period {period}"));
let mut test = sector.clone();
descramble_sector(&cracked, &mut test);
assert_eq!(
&test[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"crack_title_key key did not round-trip the body (period {period})"
);
}
}
/// recover_title_key_from_plain inverts dvdcss_unscramble exactly: scramble
/// a known body, hand back the keystream-derived key, and the recovered
/// key (XOR-back included) reproduces the plaintext.
#[test]
fn recovered_key_descrambles_back_to_plaintext() {
let cases: &[([u8; 5], [u8; 5])] = &[
(
[0x42, 0x13, 0x37, 0xBE, 0xEF],
[0x11, 0x22, 0x33, 0x44, 0x55],
),
(
[0x9A, 0x78, 0x56, 0x34, 0x12],
[0xA5, 0x5A, 0x0F, 0xF0, 0xCC],
),
(
[0xFF, 0xFF, 0xFF, 0xFF, 0xFF],
[0x01, 0x01, 0x01, 0x01, 0x01],
),
];
for (title_key, seed) in cases {
let (mut sector, body) = synth_sector(title_key, seed, &PES);
let recovered =
recover_title_key(&sector, &PES).expect("recover_title_key returned None");
descramble_sector(&recovered, &mut sector);
assert_eq!(
&sector[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"descramble with recovered key did not reproduce the body \
for title={title_key:02x?} seed={seed:02x?}"
);
}
}
// ── early-return guards ────────────────────────────────────────────────
#[test]
fn recover_rejects_sector_one_byte_short() {
let mut sector = vec![0u8; SECTOR_BYTES - 1];
sector[FLAG_BYTE] = 0x30;
assert!(recover_title_key(&sector, &PES).is_none());
}
#[test]
fn recover_rejects_unscrambled_sector() {
let sector = vec![0x00u8; SECTOR_BYTES];
assert!(recover_title_key(&sector, &PES).is_none());
}
#[test]
fn recover_high_flag_bits_are_not_scramble() {
for &flag in &[0x40u8, 0x80, 0xC0] {
let mut sector = vec![0x11u8; SECTOR_BYTES];
sector[FLAG_BYTE] = flag;
assert!(
recover_title_key(&sector, &PES).is_none(),
"flag {flag:#04x} has scramble bits clear; recover must return None"
);
}
}
#[test]
fn crack_high_flag_bits_are_not_scramble() {
for &flag in &[0x40u8, 0x80, 0xC0] {
let mut sector = vec![0x11u8; SECTOR_BYTES];
sector[FLAG_BYTE] = flag;
assert!(
crack_title_key(&sector).is_none(),
"flag {flag:#04x} clear scramble bits -> crack must return None"
);
}
}
#[test]
fn crack_rejects_sector_one_byte_short() {
let mut sector = vec![0u8; SECTOR_BYTES - 1];
if sector.len() > FLAG_BYTE {
sector[FLAG_BYTE] = 0x30;
}
assert!(crack_title_key(&sector).is_none());
}
/// crack_title_key must never panic on a fully scrambled sector with
/// arbitrary (non-periodic) content — it just returns None.
#[test]
fn crack_full_path_never_panics() {
for seed in 0u32..3 {
let mut sector = vec![0u8; SECTOR_BYTES];
sector[FLAG_BYTE] = 0x30;
let mut x = seed.wrapping_mul(2_654_435_761).wrapping_add(7);
for b in sector.iter_mut().skip(0x80) {
x = x.wrapping_mul(1_103_515_245).wrapping_add(12_345);
*b = (x >> 16) as u8;
}
for (i, b) in sector[SEED_OFFSET..SEED_OFFSET + 5].iter_mut().enumerate() {
*b = (seed.wrapping_add(i as u32) ^ 0xA5) as u8;
}
let _ = crack_title_key(&sector);
}
}
}
+191 -28
View File
@@ -25,6 +25,8 @@ pub const TAB1: [u8; 256] = [
];
/// Table 2: LFSR1 high-byte feedback permutation.
///
/// Byte-identical to libdvdcss `p_css_tab2` (csstables.h).
pub const TAB2: [u8; 256] = [
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x09, 0x08, 0x0b, 0x0a, 0x0d, 0x0c, 0x0f, 0x0e,
0x12, 0x13, 0x10, 0x11, 0x16, 0x17, 0x14, 0x15, 0x1b, 0x1a, 0x19, 0x18, 0x1f, 0x1e, 0x1d, 0x1c,
@@ -40,11 +42,18 @@ pub const TAB2: [u8; 256] = [
0xa4, 0xa5, 0xa6, 0xa7, 0xa0, 0xa1, 0xa2, 0xa3, 0xad, 0xac, 0xaf, 0xae, 0xa9, 0xa8, 0xab, 0xaa,
0xdb, 0xda, 0xd9, 0xd8, 0xdf, 0xde, 0xdd, 0xdc, 0xd2, 0xd3, 0xd0, 0xd1, 0xd6, 0xd7, 0xd4, 0xd5,
0xc9, 0xc8, 0xcb, 0xca, 0xcd, 0xcc, 0xcf, 0xce, 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7,
0xed, 0xec, 0xef, 0xee, 0xe9, 0xe8, 0xeb, 0xea, 0xe4, 0xe5, 0xe6, 0xe7, 0xe0, 0xe1, 0xe2, 0xe3,
0xff, 0xfe, 0xfd, 0xfc, 0xfb, 0xfa, 0xf9, 0xf8, 0xf6, 0xf7, 0xf4, 0xf5, 0xf2, 0xf3, 0xf0, 0xf1,
0xed, 0xec, 0xef, 0xee, 0xe9, 0xe8, 0xeb, 0xea, 0xe4, 0xe5, 0xe6, 0xe7, 0xe0, 0xe1, 0xe2, 0xe3,
];
/// Table 3: LFSR1 low-byte feedback permutation.
/// Table 3: LFSR1 9-bit low-word feedback table (512 entries).
///
/// Byte-identical to libdvdcss `p_css_tab3` (csstables.h): the 8-value
/// block `BASE[i & 7]` repeated 64 times. The CSS LFSR1 step indexes this
/// table with the 9-bit low register (0x100..=0x1FF), but only the low 3
/// bits select the output — the high bits are ignored, hence the constant
/// blocks. The 512-entry width simply lets the 9-bit index be used without
/// masking.
pub const TAB3: [u8; 512] = [
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
@@ -54,30 +63,30 @@ pub const TAB3: [u8; 512] = [
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
];
/// Table 4: LFSR0 byte permutation (used in initialization and output).
@@ -100,8 +109,10 @@ pub const TAB4: [u8; 256] = [
0x0f, 0x8f, 0x4f, 0xcf, 0x2f, 0xaf, 0x6f, 0xef, 0x1f, 0x9f, 0x5f, 0xdf, 0x3f, 0xbf, 0x7f, 0xff,
];
/// Table 5: LFSR1 output permutation for the Stevenson attack.
/// This is the inverse byte-reversal of TAB4.
/// Table 5: LFSR1 output permutation used in the keystream combiner.
/// `TAB5[i] == TAB4[i] ^ 0xFF` (bitwise complement of the TAB4 bit-reversal
/// table). Applied on the normal descramble/recrypt path (lfsr.rs) as well as
/// in the key-recovery fallback (crack.rs).
pub const TAB5: [u8; 256] = [
0xff, 0x7f, 0xbf, 0x3f, 0xdf, 0x5f, 0x9f, 0x1f, 0xef, 0x6f, 0xaf, 0x2f, 0xcf, 0x4f, 0x8f, 0x0f,
0xf7, 0x77, 0xb7, 0x37, 0xd7, 0x57, 0x97, 0x17, 0xe7, 0x67, 0xa7, 0x27, 0xc7, 0x47, 0x87, 0x07,
@@ -120,3 +131,155 @@ pub const TAB5: [u8; 256] = [
0xf8, 0x78, 0xb8, 0x38, 0xd8, 0x58, 0x98, 0x18, 0xe8, 0x68, 0xa8, 0x28, 0xc8, 0x48, 0x88, 0x08,
0xf0, 0x70, 0xb0, 0x30, 0xd0, 0x50, 0x90, 0x10, 0xe0, 0x60, 0xa0, 0x20, 0xc0, 0x40, 0x80, 0x00,
];
#[cfg(test)]
mod tests {
use super::*;
/// Pins the documented relationship `TAB5[i] == TAB4[i] ^ 0xFF` so the
/// table doc cannot drift from the data.
#[test]
fn tab5_is_complement_of_tab4() {
for i in 0..256 {
assert_eq!(
TAB5[i],
TAB4[i] ^ 0xFF,
"TAB5[{i:#04x}] != TAB4[{i:#04x}] ^ 0xFF"
);
}
}
/// TAB1 is a bijection on 0..256. CSS uses it as an invertible output
/// permutation in css_DecryptKey's chained-XOR rounds; if two inputs
/// collided, the key mangling would not be invertible.
///
/// Mutation: duplicate any value (e.g. set TAB1[1] = TAB1[0]) -> the
/// "maps two inputs" assert fires.
#[test]
fn tab1_is_a_permutation() {
let mut seen = [false; 256];
for (i, &v) in TAB1.iter().enumerate() {
assert!(
!seen[v as usize],
"TAB1 maps two inputs to {v:#04x} (collision at index {i:#04x})"
);
seen[v as usize] = true;
}
}
/// TAB1's fixed structural anchors from the CSS spec table:
/// TAB1[0x00] == 0x33 and the inverse TAB1[0x33] == 0x00. These two
/// entries are the canonical first-row / inverse-lookup landmarks of the
/// published CSS TAB1 and pin the table's orientation.
///
/// Grounding: CSS specification TAB1, row 0 col 0 = 0x33; index 0x33
/// (row 3 col 3) = 0x00.
/// Mutation: change the first literal `0x33` in TAB1 -> first assert fails.
#[test]
fn tab1_known_spec_anchors() {
assert_eq!(TAB1[0x00], 0x33, "TAB1[0] is the published 0x33");
assert_eq!(TAB1[0x33], 0x00, "TAB1[0x33] is the published 0x00");
}
/// TAB2 is a permutation of 0..256 (it is the LFSR1 high-byte feedback
/// substitution). A non-bijective TAB2 would bias the LFSR1 keystream.
///
/// Mutation: set TAB2[8] = 0x00 (collides with TAB2[0]) -> assert fires.
#[test]
fn tab2_is_a_permutation() {
let mut seen = [false; 256];
for (i, &v) in TAB2.iter().enumerate() {
assert!(
!seen[v as usize],
"TAB2 maps two inputs to {v:#04x} (collision at index {i:#04x})"
);
seen[v as usize] = true;
}
}
/// TAB3 is the libdvdcss `p_css_tab3`: the 8-value feedback block
/// BASE = [0x00,0x24,0x49,0x6d,0x92,0xb6,0xdb,0xff]
/// repeated 64 times — `TAB3[i] == BASE[i & 7]`. The high bits of the
/// 9-bit index do not affect the output (libdvdcss's LFSR1 step indexes
/// with the full 9-bit low register but only `& 7` matters). This pins
/// all 512 entries to the published table.
///
/// Mutation: flip any single byte in the TAB3 literal -> the formula
/// check fails at that index.
#[test]
fn tab3_matches_lfsr1_generating_formula() {
const BASE: [u8; 8] = [0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff];
for i in 0..512usize {
let expected = BASE[i & 7];
assert_eq!(
TAB3[i], expected,
"TAB3[{i:#05x}] = {:#04x}, formula BASE[i&7] = {expected:#04x}",
TAB3[i]
);
}
}
/// TAB4 is the exact bit-reversal of each byte (CSS uses it to permute
/// LFSR0 bytes on seed and output). TAB4[b] reverses b's 8 bits MSB<->LSB.
/// Therefore it is also an involution: TAB4[TAB4[b]] == b.
///
/// Grounding: TAB4[0x01]=0x80, TAB4[0x80]=0x01, TAB4[0x00]=0x00,
/// TAB4[0xFF]=0xFF.
/// Mutation: set TAB4[1] = 0x40 (not the reversal 0x80) -> bit-reversal
/// check fails at index 1.
#[test]
fn tab4_is_exact_bit_reversal_and_involution() {
for b in 0u16..256 {
let rev = (0..8).fold(0u8, |acc, k| acc | (((b as u8 >> k) & 1) << (7 - k)));
assert_eq!(
TAB4[b as usize], rev,
"TAB4[{b:#04x}] is not the bit-reversal {rev:#04x}"
);
}
for b in 0..256usize {
assert_eq!(
TAB4[TAB4[b] as usize], b as u8,
"TAB4 not an involution at {b:#04x}"
);
}
// Spec landmark entries.
assert_eq!(TAB4[0x01], 0x80);
assert_eq!(TAB4[0x80], 0x01);
assert_eq!(TAB4[0x00], 0x00);
assert_eq!(TAB4[0xFF], 0xFF);
}
/// TAB4 is a permutation (bit-reversal is bijective). Distinct from the
/// reversal test: a table that is "reversal except two swapped entries"
/// would still be a permutation, and a table that is "reversal except one
/// duplicated entry" would fail this but might pass a sampled reversal
/// check — the two tests pin different failure modes.
///
/// Mutation: set TAB4[2] = TAB4[1] -> permutation assert fires.
#[test]
fn tab4_is_a_permutation() {
let mut seen = [false; 256];
for &v in TAB4.iter() {
assert!(!seen[v as usize], "TAB4 maps two inputs to {v:#04x}");
seen[v as usize] = true;
}
}
/// TAB5 is also a permutation (complement of a bijection is a bijection)
/// and its own self-consistency landmark: TAB5[0x00] == 0xFF (TAB4[0]^0xFF)
/// and TAB5[0xFF] == 0x00 (TAB4[0xFF]^0xFF). Pins orientation independent
/// of the complement-loop test.
///
/// Mutation: change the first TAB5 literal 0xff -> 0xfe -> the landmark
/// and permutation checks both catch it.
#[test]
fn tab5_is_permutation_with_anchors() {
let mut seen = [false; 256];
for &v in TAB5.iter() {
assert!(!seen[v as usize], "TAB5 maps two inputs to {v:#04x}");
seen[v as usize] = true;
}
assert_eq!(TAB5[0x00], 0xFF, "TAB5[0] = TAB4[0]^0xFF = 0xFF");
assert_eq!(TAB5[0xFF], 0x00, "TAB5[0xFF] = TAB4[0xFF]^0xFF = 0x00");
}
}
+1164 -52
View File
File diff suppressed because it is too large Load Diff
+753
View File
@@ -0,0 +1,753 @@
//! Structured scan diagnostics — the `--log-level 3` self-diagnosing dump.
//!
//! A bug report log must be self-diagnosing: everything needed to explain
//! *why* freemkv made the choices it did at scan must be in the log, in a
//! compact, machine-parseable form. This module emits one terse line per row
//! (title, cell, stream, decision) under the `tracing` target
//! `freemkv::diag`, which the CLI routes to `log.txt` when `--log-level 3`
//! (debug) is set.
//!
//! Format conventions (stable, greppable):
//! - Every line is prefixed by a `tag=` so a log scraper can filter
//! (`disc`, `title`, `dvd.cell`, `dvd.vattr`, `dvd.aattr`, `bd.clip`,
//! `bd.mark`, `aacs`, `stream`, `decision`).
//! - Raw bytes are shown as `0xNN` next to their decode so a wrong decode
//! is obvious against the raw value.
//! - This module only READS already-parsed scan state — it never re-reads
//! the disc and never mutates anything.
//!
//! The DVD per-cell table (with the raw cell-category byte) is emitted from
//! the IFO scan itself ([`dump_dvd_cells`]), because the per-cell
//! `ifo::DvdCell` detail is lowered away before the `Disc` is built. The
//! `Disc`-level dump ([`dump_disc`]) covers everything that survives
//! lowering: titles, streams, the picked main feature, and AACS state.
use crate::disc::{
AudioChannels, ColorSpace, Disc, DiscTitle, FrameRate, HdrFormat, Resolution, SampleRate,
Stream,
};
use crate::ifo::{CellCategory, DvdTitle};
const DIAG: &str = "freemkv::diag";
// ── small format helpers (pure, unit-testable) ──────────────────────────────
/// Compact name for a [`Resolution`] with the interlace marker preserved.
pub fn res_str(r: Resolution) -> &'static str {
match r {
Resolution::R480i => "480i",
Resolution::R480p => "480p",
Resolution::R576i => "576i",
Resolution::R576p => "576p",
Resolution::R720p => "720p",
Resolution::R1080i => "1080i",
Resolution::R1080p => "1080p",
Resolution::R2160p => "2160p",
Resolution::R4320p => "4320p",
Resolution::Unknown => "res?",
}
}
/// Frames-per-second string for a [`FrameRate`].
pub fn fps_str(f: FrameRate) -> &'static str {
match f {
FrameRate::F23_976 => "23.976",
FrameRate::F24 => "24",
FrameRate::F25 => "25",
FrameRate::F29_97 => "29.97",
FrameRate::F30 => "30",
FrameRate::F50 => "50",
FrameRate::F59_94 => "59.94",
FrameRate::F60 => "60",
FrameRate::Unknown => "fps?",
}
}
/// PAL/NTSC field-rate family inferred from the frame rate (DVD has no
/// explicit field, so this is the colour/standard the muxer stamps).
pub fn tv_system_str(f: FrameRate) -> &'static str {
match f {
FrameRate::F25 | FrameRate::F50 => "PAL",
FrameRate::F23_976 | FrameRate::F29_97 | FrameRate::F59_94 => "NTSC",
_ => "",
}
}
/// CICP-ish short name for a [`ColorSpace`].
pub fn color_str(c: ColorSpace) -> &'static str {
match c {
ColorSpace::Bt709 => "BT.709",
ColorSpace::Bt2020 => "BT.2020",
ColorSpace::Bt470bg => "BT.470BG",
ColorSpace::Smpte170m => "SMPTE-170M",
ColorSpace::Unknown => "color?",
}
}
/// HDR format short name.
pub fn hdr_str(h: HdrFormat) -> &'static str {
match h {
HdrFormat::Sdr => "SDR",
HdrFormat::Hdr10 => "HDR10",
HdrFormat::Hdr10Plus => "HDR10+",
HdrFormat::DolbyVision => "DoVi",
HdrFormat::Hlg => "HLG",
}
}
/// Channel count from an [`AudioChannels`] layout (what lands in the MKV
/// `Channels` element).
pub fn channel_count(ch: AudioChannels) -> u8 {
match ch {
AudioChannels::Mono => 1,
AudioChannels::Stereo => 2,
AudioChannels::Stereo21 => 3,
AudioChannels::Quad => 4,
AudioChannels::Surround50 => 5,
AudioChannels::Surround51 => 6,
AudioChannels::Surround61 => 7,
AudioChannels::Surround71 => 8,
AudioChannels::Unknown => 0,
}
}
/// Sample-rate in Hz for a [`SampleRate`].
pub fn sample_rate_hz(s: SampleRate) -> u32 {
match s {
SampleRate::S44_1 => 44100,
SampleRate::S48 => 48000,
SampleRate::S88_2 => 88200,
SampleRate::S96 => 96000,
SampleRate::S176_4 => 176400,
SampleRate::S192 => 192000,
SampleRate::S48_96 => 96000,
SampleRate::S48_192 => 192000,
SampleRate::Unknown => 0,
}
}
// ── DVD cell-category dump (from the IFO scan, pre-lowering) ─────────────────
/// One formatted cell row for the DVD per-PGC cell table. Returned as a
/// string so it can be unit-tested without a logger.
///
/// Columns: `idx`, raw category (`cat=0xNN`) + decoded fields, first/last
/// sector, duration, and the keep/drop verdict from the bug-4 leading-cell
/// filter.
pub fn dvd_cell_row(idx: usize, cell: &crate::ifo::DvdCell, dropped: bool) -> String {
let c = CellCategory::decode(cell.category);
// Per-cell keep/skip REASON (self-sufficient bug log): a dropped cell is a
// leading secondary angle/interleave block piece; a kept cell is either the
// first feature cell or genuine feature content. This makes the
// leading-cell-filter decision auditable from the log without the disc.
let verdict = if dropped {
"DROP(leading-secondary-block-piece)"
} else if c.is_secondary_block_piece() {
// Kept despite being a secondary piece — only happens past the leading
// run (the filter stops at the first plain feature cell).
"keep(feature-body)"
} else {
"keep(plain-feature)"
};
format!(
"tag=dvd.cell idx={idx} cat=0x{:02X} block_mode={} block_type={} \
seamless={} ilv={} stc={} angle={} plain={} first={} last={} dur={:.1}s {}",
cell.category,
c.block_mode,
c.block_type,
c.seamless_play as u8,
c.interleaved as u8,
c.stc_discontinuity as u8,
c.seamless_angle as u8,
c.is_plain_feature() as u8,
cell.first_sector,
cell.last_sector,
cell.duration_secs,
verdict,
)
}
/// Emit the per-PGC cell table for one DVD title during the IFO scan.
///
/// `vts`/`title` identify the row group; `title` is the `DvdTitle` whose
/// cells (and bug-4 leading-cell verdict) are dumped. Called from
/// `scan_dvd_titles` while the `DvdTitle` is still in scope (the per-cell
/// category byte is lowered away before the `Disc` exists).
pub fn dump_dvd_cells(vts: u8, title_num: u16, title: &DvdTitle) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
let feature_start = title.feature_start_cell();
tracing::debug!(
target: DIAG,
"tag=dvd.pgc vts={vts} title={title_num} cells={} chapters={} \
dur={:.1}s feature_start_cell={feature_start}",
title.cells.len(),
title.chapters,
title.duration_secs,
);
for (i, cell) in title.cells.iter().enumerate() {
tracing::debug!(target: DIAG, "{}", dvd_cell_row(i, cell, i < feature_start));
}
// Chapter/PTT map (program → cumulative start time).
for (i, &t) in title.chapter_times.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.chap vts={vts} title={title_num} ch={} time={:.1}s",
i + 1,
t,
);
}
}
/// Emit the IFO `video_attr` / `audio_attr` decode for one DVD title set,
/// showing the raw bytes next to their decoded meaning. Called from the IFO
/// scan with the still-parsed `ifo::DvdTitleSet` view.
pub fn dump_dvd_attrs(ts: &crate::ifo::DvdTitleSet) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
tracing::debug!(
target: DIAG,
"tag=dvd.vobs vts={} vob_start_sector={}",
ts.vts_number,
ts.vob_start_sector,
);
let v = &ts.video;
tracing::debug!(
target: DIAG,
"tag=dvd.vattr vts={} codec={:?} res={} aspect={:?} std={:?}",
ts.vts_number,
v.codec,
res_str(v.resolution),
v.aspect,
v.standard,
);
for (i, a) in ts.audio_streams.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.aattr vts={} idx={i} codec={:?} ch={} sr={}Hz lang={:?} sub_id={:?}",
ts.vts_number,
a.codec,
a.channels,
a.sample_rate,
a.language,
a.sub_stream_id.map(|x| format!("0x{x:02X}")),
);
}
for (i, s) in ts.subtitle_streams.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.sattr vts={} idx={i} lang={:?}",
ts.vts_number,
s.language,
);
}
}
/// Emit the ACTUAL per-physical-sub-stream AC-3 channel counts read off the VOB
/// during the mux-time sub-stream probe (the Silence-of-the-Lambs wrong-stream
/// fix). This is the ground truth the IFO nibble is compared against: each row
/// is `sub_id=0x8x channels=N` for a physical `private_stream_1` AC-3 sub-stream
/// whose first frame was decoded. An empty probe (scrambled / unreadable / short
/// VOB) logs a single `probed=0` line so the absence is explicit in a bug log.
///
/// Self-sufficiency: with `tag=dvd.aattr` (the IFO's declared sub_id + claimed
/// channels) and these `tag=dvd.substream` rows (the physical reality), a bug
/// log alone shows whether the ordinal `0x80` actually carries the declared
/// channel layout — no disc needed to diagnose a wrong-substream rip.
pub fn dump_dvd_substream_probe(title_id: u16, probed: &std::collections::BTreeMap<u8, u8>) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
if probed.is_empty() {
tracing::debug!(
target: DIAG,
"tag=dvd.substream title={title_id} probed=0 (no AC-3 sync in feature head — scrambled/unreadable/none)",
);
return;
}
for (sub, ch) in probed {
tracing::debug!(
target: DIAG,
"tag=dvd.substream title={title_id} sub_id=0x{sub:02X} channels={ch} (physical acmod read from VOB)",
);
}
}
// ── MKV TrackEntry dump (the ACTUAL container elements written) ──────────────
/// `true` when the `--log-level 3` diagnostic target is enabled. Hot-path
/// callers (the opening-frame capture) check this once and skip all work when
/// off, so a normal run pays nothing.
pub fn diag_enabled() -> bool {
tracing::enabled!(target: DIAG, tracing::Level::DEBUG)
}
/// Cap on the number of codecPrivate bytes rendered to hex in a `tag=mkv.track`
/// line. The sequence header / avcC / hvcC prefix that matters for diagnosis
/// (resolution, frame rate, profile) is at the front; a multi-KB blob past this
/// is summarised as `..(+NB)` rather than flooding the log.
const CODEC_PRIVATE_HEX_CAP: usize = 64;
/// Render a track's codecPrivate as an uppercase-hex string for the diagnostic
/// line, capped at [`CODEC_PRIVATE_HEX_CAP`] bytes (`..(+NB)` suffix beyond).
/// `None` / empty → `"none"`. Pure (no logging) so it is directly unit-testable.
fn codec_private_hex(cp: Option<&[u8]>) -> String {
match cp {
Some(b) if !b.is_empty() => {
use std::fmt::Write;
let shown = b.len().min(CODEC_PRIVATE_HEX_CAP);
let mut s = String::with_capacity(shown * 2 + 8);
for byte in &b[..shown] {
let _ = write!(s, "{byte:02X}");
}
if b.len() > CODEC_PRIVATE_HEX_CAP {
let _ = write!(s, "..(+{}B)", b.len() - CODEC_PRIVATE_HEX_CAP);
}
s
}
_ => "none".to_string(),
}
}
/// Frame the raw bytes of one captured opening frame for the `.opening.bin` side
/// file: `[track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw bytes]`. Pure
/// (no I/O) so the record layout is directly unit-testable; `record` appends the
/// returned bytes to the side file.
fn frame_record(track_idx: usize, pts_ns: i64, keyframe: bool, data: &[u8]) -> Vec<u8> {
let mut rec = Vec::with_capacity(14 + data.len());
rec.push(track_idx as u8);
rec.push(keyframe as u8);
rec.extend_from_slice(&pts_ns.to_le_bytes());
rec.extend_from_slice(&(data.len() as u32).to_le_bytes());
rec.extend_from_slice(data);
rec
}
/// Emit the MKV `TrackEntry` elements the muxer is about to WRITE for one
/// track — the Windows-fps-class metadata (FlagInterlaced, FieldOrder,
/// DefaultDuration, DefaultDecodedFieldDuration, Display dims) plus the
/// codecPrivate as hex. With this row a bug log alone is enough to verify why
/// Windows Explorer reports a given frame rate for an interlaced SD track: the
/// container values that drive its fps derivation are all present, no disc and
/// no MediaInfo needed.
///
/// `track_number` is the 1-based MKV track number; `track` is the built
/// [`crate::mux::mkv::MkvTrack`] whose fields map one-to-one onto the emitted
/// elements (see `MkvMuxer::new`). No-op unless the diag target is on.
pub fn dump_mkv_track(track_number: u64, track: &crate::mux::mkv::MkvTrack) {
if !diag_enabled() {
return;
}
// codecPrivate as hex (capped so a multi-KB hvcC doesn't flood the log; the
// sequence header / avcC prefix that matters for diagnosis is at the front).
let cp = codec_private_hex(track.codec_private.as_deref());
let field_order = match track.field_order {
crate::mux::ebml::FIELD_ORDER_TFF => "TFF",
crate::mux::ebml::FIELD_ORDER_BFF => "BFF",
_ => "",
};
// FlagInterlaced is only written for video tracks (1=interlaced/2=progressive);
// report what the muxer will emit, or "—" for non-video tracks where the
// element is omitted entirely.
let interlaced = if track.track_type == crate::mux::ebml::TRACK_TYPE_VIDEO {
if track.interlaced {
"1(interlaced)"
} else {
"2(progressive)"
}
} else {
""
};
tracing::debug!(
target: DIAG,
"tag=mkv.track num={track_number} type={} codec={} flag_interlaced={interlaced} \
field_order={field_order} default_duration_ns={} field_duration_ns={} \
pixel={}x{} display={}x{} cp_len={} cp_hex={cp}",
track.track_type,
track.codec_id,
track.default_duration_ns,
track.field_duration_ns,
track.pixel_width,
track.pixel_height,
track.display_width,
track.display_height,
track.codec_private.as_ref().map_or(0, |b| b.len()),
);
}
// ── Opening-frame capture (first ~N coded frames per track → side file) ──────
/// Number of coded frames captured PER TRACK before the capture goes dormant.
/// ~100 frames covers a DVD's first few seconds of every track (the
/// opening-GOP / still-frame / menu window where mid-GOP open or PTS-floor bugs
/// show up) while bounding the side file to a few MB even for HD I-frames.
const OPENING_FRAMES_PER_TRACK: usize = 100;
/// Captures the first [`OPENING_FRAMES_PER_TRACK`] coded frames of EACH track to
/// a side file (`<output>.opening.bin`) and logs a per-frame summary line, so an
/// opening-GOP / menu / mid-GOP-open issue is diagnosable from a future log +
/// side file WITHOUT the disc. Gated to `--log-level 3`: constructed only when
/// the diag target is on, so a normal run never opens the file or records a byte.
///
/// Side-file record framing (so a reader can split it back into frames):
/// `[track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw frame bytes]`.
pub struct OpeningCapture {
file: std::fs::File,
/// Frames captured so far, per track index. Capture for a track stops once
/// its counter reaches [`OPENING_FRAMES_PER_TRACK`].
counts: Vec<usize>,
}
impl OpeningCapture {
/// Open `<output>.opening.bin` next to the MKV output. Returns `None` (no
/// capture) when the diag target is off OR the side file can't be created —
/// a diagnostic must never fail the rip. `track_count` sizes the per-track
/// counters.
pub fn new(output_path: &std::path::Path, track_count: usize) -> Option<Self> {
if !diag_enabled() {
return None;
}
let mut name = output_path.as_os_str().to_os_string();
name.push(".opening.bin");
match std::fs::File::create(&name) {
Ok(file) => {
tracing::debug!(
target: DIAG,
"tag=mkv.opening.open path={:?} per_track_cap={OPENING_FRAMES_PER_TRACK}",
std::path::Path::new(&name),
);
Some(Self {
file,
counts: vec![0; track_count],
})
}
Err(e) => {
tracing::debug!(
target: DIAG,
"tag=mkv.opening.open path={:?} failed={e} (capture disabled, rip unaffected)",
std::path::Path::new(&name),
);
None
}
}
}
/// Record one coded frame for `track_idx` if that track is still under its
/// per-track cap. Writes the framed raw bytes to the side file and logs a
/// one-line summary. A write error disables further capture for the track
/// (counter pinned to the cap) but never propagates — the rip is unaffected.
pub fn record(&mut self, track_idx: usize, pts_ns: i64, keyframe: bool, data: &[u8]) {
let Some(count) = self.counts.get_mut(track_idx) else {
return;
};
if *count >= OPENING_FRAMES_PER_TRACK {
return;
}
use std::io::Write;
let rec = frame_record(track_idx, pts_ns, keyframe, data);
if let Err(e) = self.file.write_all(&rec) {
// Stop trying on this track; a broken side file must not stall mux.
*count = OPENING_FRAMES_PER_TRACK;
tracing::debug!(
target: DIAG,
"tag=mkv.opening.frame track={track_idx} write_failed={e} (capture stopped for track)",
);
return;
}
*count += 1;
tracing::debug!(
target: DIAG,
"tag=mkv.opening.frame track={track_idx} n={count} type={} size={} pts_ns={pts_ns}",
if keyframe { "key" } else { "delta" },
data.len(),
);
}
}
// ── Disc-level dump (post-lowering: titles, streams, decisions, AACS) ────────
/// Emit the full scan diagnostic block for a built [`Disc`]. Terse, one line
/// per row, under target `freemkv::diag` at DEBUG. No-op unless that target
/// is enabled, so it costs nothing when `--log-level 3` is off.
pub fn dump_disc(disc: &Disc) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
tracing::debug!(
target: DIAG,
"tag=disc vol={:?} format={:?} content={:?} cap_sectors={} layers={} titles={} encrypted={}",
disc.volume_id,
disc.format,
disc.content_format,
disc.capacity_sectors,
disc.layers,
disc.titles.len(),
disc.encrypted,
);
dump_aacs(disc);
for (ti, title) in disc.titles.iter().enumerate() {
dump_title(ti, title);
}
// freemkv's top-level DECISION: which title is the main feature.
if let Some(main) = disc.titles.first() {
tracing::debug!(
target: DIAG,
"tag=decision pick=main_feature title_idx=0 playlist={:?} dur={:.1}s \
size={}B clips={} reason=canonical_title_order(fits-disc, fewest-clips, longest, richest-audio)",
main.playlist,
main.duration_secs,
main.size_bytes,
main.clips.len(),
);
}
}
fn dump_aacs(disc: &Disc) {
let Some(a) = disc.aacs.as_ref() else {
if disc.css.is_some() {
tracing::debug!(target: DIAG, "tag=aacs none crypto=CSS(DVD)");
} else if disc.encrypted {
tracing::debug!(target: DIAG, "tag=aacs none crypto=encrypted-no-keys");
} else {
tracing::debug!(target: DIAG, "tag=aacs none crypto=clear");
}
return;
};
// CPS-unit / unit-key counts: at scan `unit_keys` is empty (keys are
// resolved later); the unit-key count is the BE16 in the raw
// Unit_Key_RO.inf if captured. Report both: resolved count and raw len.
tracing::debug!(
target: DIAG,
"tag=aacs version={} bus_enc={} mkb_version={:?} disc_hash={} key_source={:?} \
vuk={} unit_keys_resolved={} uk_ro_bytes={} mkb_bytes={}",
a.version,
a.bus_encryption,
a.mkb_version,
a.disc_hash,
a.key_source.name(),
a.vuk.is_some(),
a.unit_keys.len(),
a.uk_ro.len(),
a.mkb.len(),
);
}
fn dump_title(ti: usize, title: &DiscTitle) {
let (mut nv, mut na, mut ns) = (0u32, 0u32, 0u32);
for s in &title.streams {
match s {
Stream::Video(_) => nv += 1,
Stream::Audio(_) => na += 1,
Stream::Subtitle(_) => ns += 1,
}
}
tracing::debug!(
target: DIAG,
"tag=title idx={ti} playlist={:?} id={} dur={:.1}s size={}B clips={} \
extents={} chapters={} v={nv} a={na} s={ns} fmt={:?}",
title.playlist,
title.playlist_id,
title.duration_secs,
title.size_bytes,
title.clips.len(),
title.extents.len(),
title.chapters.len(),
title.content_format,
);
// Per-clip rows (BD: PlayItem/CLPI; DVD has none).
for (ci, c) in title.clips.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=clip title={ti} idx={ci} id={:?} in={} out={} dur={:.1}s src_packets={}",
c.clip_id,
c.in_time,
c.out_time,
c.duration_secs,
c.source_packets,
);
}
// Per-extent rows (the sectors freemkv will actually rip — the bug-4
// decision is visible here: leading non-feature cells are already gone).
for (ei, e) in title.extents.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=extent title={ti} idx={ei} start_lba={} sectors={}",
e.start_lba,
e.sector_count,
);
}
// freemkv's per-stream DECISIONS (what the muxer will write).
for (si, s) in title.streams.iter().enumerate() {
match s {
Stream::Video(v) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=video pid=0x{:04X} codec={:?} \
res={} interlaced={} fps={} std={} color={} hdr={} aspect={:?} secondary={}",
v.pid,
v.codec,
res_str(v.resolution),
v.resolution.is_interlaced(),
fps_str(v.frame_rate),
tv_system_str(v.frame_rate),
color_str(v.color_space),
hdr_str(v.hdr),
v.display_aspect,
v.secondary,
),
Stream::Audio(a) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=audio pid=0x{:04X} codec={:?} \
channels={}({}) sr={}Hz lang={:?} secondary={}",
a.pid,
a.codec,
a.channels,
channel_count(a.channels),
sample_rate_hz(a.sample_rate),
a.language,
a.secondary,
),
Stream::Subtitle(sub) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=subtitle pid=0x{:04X} codec={:?} \
lang={:?} forced={}",
sub.pid,
sub.codec,
sub.language,
sub.forced,
),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn res_str_keeps_interlace_marker() {
assert_eq!(res_str(Resolution::R576i), "576i");
assert_eq!(res_str(Resolution::R480i), "480i");
assert_eq!(res_str(Resolution::R2160p), "2160p");
}
#[test]
fn fps_and_tv_system() {
assert_eq!(fps_str(FrameRate::F25), "25");
assert_eq!(tv_system_str(FrameRate::F25), "PAL");
assert_eq!(fps_str(FrameRate::F29_97), "29.97");
assert_eq!(tv_system_str(FrameRate::F29_97), "NTSC");
}
#[test]
fn color_and_hdr() {
assert_eq!(color_str(ColorSpace::Bt470bg), "BT.470BG");
assert_eq!(color_str(ColorSpace::Bt2020), "BT.2020");
assert_eq!(hdr_str(HdrFormat::Hdr10), "HDR10");
assert_eq!(hdr_str(HdrFormat::DolbyVision), "DoVi");
assert_eq!(hdr_str(HdrFormat::Sdr), "SDR");
}
#[test]
fn channel_count_matches_layout() {
assert_eq!(channel_count(AudioChannels::Mono), 1);
assert_eq!(channel_count(AudioChannels::Stereo), 2);
assert_eq!(channel_count(AudioChannels::Surround51), 6);
assert_eq!(channel_count(AudioChannels::Surround71), 8);
}
#[test]
fn sample_rate_hz_values() {
assert_eq!(sample_rate_hz(SampleRate::S48), 48000);
assert_eq!(sample_rate_hz(SampleRate::S96), 96000);
}
#[test]
fn codec_private_hex_renders_caps_and_handles_empty() {
// None / empty → "none" (no hex). The Windows-fps diagnosis only needs
// the seq-header prefix, so render it but cap long blobs.
assert_eq!(codec_private_hex(None), "none");
assert_eq!(codec_private_hex(Some(&[])), "none");
// Short blob: full uppercase hex, no suffix. An MPEG-2 seq header starts
// 00 00 01 B3 — exactly what a reader greps for in a bug log.
assert_eq!(
codec_private_hex(Some(&[0x00, 0x00, 0x01, 0xB3])),
"000001B3"
);
// Over the cap: first CODEC_PRIVATE_HEX_CAP bytes + a "..(+NB)" summary.
let big = vec![0xABu8; CODEC_PRIVATE_HEX_CAP + 5];
let s = codec_private_hex(Some(&big));
assert!(s.starts_with(&"AB".repeat(CODEC_PRIVATE_HEX_CAP)), "{s}");
assert!(s.ends_with("..(+5B)"), "{s}");
}
#[test]
fn frame_record_layout_is_parseable() {
// The .opening.bin record framing must round-trip so a future tool can
// split the side file back into frames without the disc:
// [track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw bytes].
let data = [0xDEu8, 0xAD, 0xBE, 0xEF];
let rec = frame_record(2, -40_000_000, true, &data);
assert_eq!(rec.len(), 14 + data.len());
assert_eq!(rec[0], 2, "track index");
assert_eq!(rec[1], 1, "keyframe flag");
assert_eq!(
i64::from_le_bytes(rec[2..10].try_into().unwrap()),
-40_000_000,
"pts_ns survives (signed — opening back-anchor can be negative)"
);
assert_eq!(
u32::from_le_bytes(rec[10..14].try_into().unwrap()),
4,
"len"
);
assert_eq!(&rec[14..], &data, "raw frame bytes follow");
// A non-keyframe records the flag as 0.
let delta = frame_record(0, 0, false, &[]);
assert_eq!(delta[1], 0);
assert_eq!(u32::from_le_bytes(delta[10..14].try_into().unwrap()), 0);
}
/// The cell row shows the raw category byte (0xNN) beside the decode, and
/// the keep/drop verdict. A plain feature cell (0x00) is "keep"; a leading
/// secondary-block cell flagged dropped reads "DROP".
#[test]
fn cell_row_shows_raw_byte_and_verdict() {
let plain = crate::ifo::DvdCell {
first_sector: 100,
last_sector: 199,
category: 0x00,
duration_secs: 12.5,
};
let row = dvd_cell_row(0, &plain, false);
assert!(row.contains("cat=0x00"), "{row}");
assert!(row.contains("block_mode=0"), "{row}");
assert!(row.contains("first=100"), "{row}");
assert!(row.contains("last=199"), "{row}");
assert!(row.contains("dur=12.5s"), "{row}");
assert!(row.contains("keep(plain-feature)"), "{row}");
assert!(!row.contains("DROP"), "{row}");
// 0x90 = in-block cell of an angle block (block_mode=2, block_type=1),
// shown dropped as a leading secondary piece.
let sec = crate::ifo::DvdCell {
first_sector: 0,
last_sector: 9,
category: 0x90,
duration_secs: 1.0,
};
let row = dvd_cell_row(0, &sec, true);
assert!(row.contains("cat=0x90"), "{row}");
assert!(row.contains("block_mode=2"), "{row}");
assert!(row.contains("block_type=1"), "{row}");
assert!(row.contains("DROP(leading-secondary-block-piece)"), "{row}");
}
}
+1371 -16
View File
File diff suppressed because it is too large Load Diff
+1205 -15
View File
File diff suppressed because it is too large Load Diff
+466
View File
@@ -0,0 +1,466 @@
//! Physical AC-3 sub-stream probing for DVD audio routing.
//!
//! ## Why this exists (Silence-of-the-Lambs wrong-substream bug)
//!
//! A DVD VTS IFO declares its audio streams in a fixed table, and freemkv's
//! scan assigns each declared stream a `private_stream_1` sub-stream id purely
//! by per-codec ordinal — the first AC-3 stream becomes `0x80`, the second
//! `0x81`, and so on (`ifo::assign_audio_sub_stream_ids`). That assumes the
//! physical sub-stream order on the wire matches the IFO declaration order.
//!
//! On some discs it does NOT. The R2 PAL "The Silence of the Lambs" feature
//! declares ONE AC-3 audio stream the IFO nibble marks as 5.1 (6 channels), but
//! the physical VOB carries the 5.1 main mix and a 2.0 down-mix on DIFFERENT
//! `0x8x` sub-stream ids, and the 2.0 is the one that happens to land at the
//! ordinal `0x80` slot. Routing the declared 5.1 stream to `0x80` by ordinal
//! therefore muxes the 2.0 down-mix while labelling it 5.1 — the wrong physical
//! track.
//!
//! The robust fix is data-driven and codec/disc agnostic: read each physical
//! AC-3 sub-stream's REAL channel count from the VOB (the `acmod`/`lfeon` of its
//! first frame after the `0x0B77` sync) and route each IFO-declared AC-3 stream
//! to the physical sub-stream whose actual channel count matches the IFO's
//! declared count — instead of trusting the ordinal. This never re-reads the
//! disc beyond a bounded head-of-feature probe and degrades to the original
//! ordinal mapping when the probe yields nothing (unreadable/short VOB).
use crate::disc::Stream;
use crate::mux::codec::ac3;
use crate::mux::ps::PsDemuxer;
use crate::sector::SectorSource;
use std::collections::BTreeMap;
/// How many 2048-byte sectors of the first feature extent to probe. The head of
/// a DVD feature opens with logos/warnings whose audio is frequently a thin 2.0
/// bed on the FIRST sub-stream only — the other physical `0x8x` sub-streams and
/// the main 5.1 mix do not appear until a sector or two further in. 512 sectors
/// (1 MiB) was too short: on Greenland it saw ONLY `0x80`, and only its opening
/// 2.0 frames. 1024 sectors (2 MiB) reliably contains at least one frame of
/// every physical AC-3 sub-stream AND enough of `0x80` to reach its 5.1 frames.
/// Still bounded so a live drive is never hammered (see the project "don't
/// hammer the live drive" rule).
const PROBE_SECTORS: u16 = 1024;
/// Decode the real per-sub-stream AC-3 channel count from a buffer of decrypted
/// MPEG-PS (DVD VOB) bytes.
///
/// Demuxes `private_stream_1` (0xBD), and for each AC-3 sub-stream id
/// (`0x80..=0x87`) records the MAXIMUM channel count seen across EVERY decodable
/// frame in the probe window (`acmod` + `lfeon` at each `0x0B77` sync). Pure and
/// unit-testable — takes the already-read bytes, never touches the disc.
///
/// ## Why the maximum, not the first frame
///
/// The first frame of a sub-stream at the head of a feature is NOT
/// representative. A DVD opens with logos/warnings, and the main `0x80`
/// sub-stream there frequently carries a thin 2.0 bed before transitioning to
/// its real 5.1 main mix a fraction of a second later (observed on Greenland:
/// `0x80`'s first frames are acmod=2 → 2 channels, then it becomes acmod=7+lfe →
/// 6 channels within the same 2 MiB window). Recording only the FIRST frame read
/// `0x80=2` and missed the 5.1 entirely, defeating the channel-match routing.
/// The 5.1 capability of a sub-stream is the *maximum* channel count any of its
/// frames carries, so we scan them all and keep the max.
///
/// Returns a map `sub_id -> max channels`. Sub-streams whose frames are all too
/// short to carry the BSI bits, or that never appear in the buffer, are absent
/// from the map.
pub fn probe_ac3_substream_channels(ps_bytes: &[u8]) -> BTreeMap<u8, u8> {
let mut found: BTreeMap<u8, u8> = BTreeMap::new();
let mut demux = PsDemuxer::new();
let mut packets = demux.feed(ps_bytes);
packets.extend(demux.flush());
for p in packets {
// Only private_stream_1 AC-3 sub-streams (0x80..=0x87).
let Some(sub) = p.sub_stream_id else { continue };
if !(0x80..=0x87).contains(&sub) {
continue;
}
// The PS demux strips the 4-byte AC-3 sub-header but does not align to a
// frame. Walk EVERY 0x0B77 sync in this sub-stream's payload, decode
// each frame's channel count, and keep the largest — the sub-stream's
// real (main-mix) channel capability. See the doc comment above for why
// the first frame alone is unreliable.
if let Some(ch) = max_substream_channels(&p.data) {
let slot = found.entry(sub).or_insert(0);
*slot = (*slot).max(ch);
}
}
found
}
/// Largest AC-3 channel count over every decodable frame in a single
/// sub-stream's payload. Returns `None` when no frame carries enough BSI bits.
///
/// Each frame is advanced by its real `ac3_frame_size` so a frame's compressed
/// body (which can contain stray `0x0B77` byte pairs) cannot be mistaken for a
/// new frame; only when a size is unmappable do we fall back to a +2 byte
/// rescan to re-lock the next genuine sync.
fn max_substream_channels(data: &[u8]) -> Option<u8> {
let mut best: Option<u8> = None;
let mut pos = 0;
while pos < data.len() {
let Some(rel) = ac3::find_ac3_sync(&data[pos..]) else {
break;
};
let start = pos + rel;
let frame = &data[start..];
if let Some(ch) = ac3::acmod_channels(frame) {
if ch > 0 {
best = Some(best.map_or(ch, |b| b.max(ch)));
}
}
// Advance past this frame by its declared size when that is mappable;
// otherwise step 2 bytes past the sync and re-scan for the next one.
let size = ac3::ac3_frame_size(frame);
pos = if (6..=8192).contains(&size) {
start + size
} else {
start + 2
};
}
best
}
/// Re-route the title's declared AC-3 audio streams onto the physical
/// sub-stream ids whose REAL channel counts match, using a probed
/// `sub_id -> channels` map.
///
/// For each declared AC-3 audio stream (in IFO order), it picks the physical
/// `0x8x` sub-stream whose probed channel count equals the stream's declared
/// channel count, never re-using a sub-stream already claimed by an earlier
/// stream. The chosen sub-stream's PID (`0xBD00 | sub_id`) is written back onto
/// the `Stream::Audio` so BOTH mux demux paths (`DiscStream` and the file-backed
/// highway) route by it.
///
/// Conservative — it only ever REASSIGNS among the physical sub-streams the
/// probe actually saw, and only when a better (exact-channel) match exists than
/// the stream's current assignment. A stream whose current sub-stream already
/// matches is left alone; a stream with no matching physical sub-stream keeps
/// its ordinal assignment. So a normal disc (physical order == IFO order) is a
/// no-op.
///
/// Returns the number of streams whose PID was changed (for diagnostics).
pub fn remap_audio_pids(streams: &mut [Stream], probed: &BTreeMap<u8, u8>) -> usize {
if probed.is_empty() {
return 0;
}
// Sub-streams already claimed by a remapped (or matching) earlier stream,
// so two declared streams never collide on one physical sub-stream.
let mut claimed: Vec<u8> = Vec::new();
let mut changed = 0usize;
for s in streams.iter_mut() {
let Stream::Audio(a) = s else { continue };
if a.codec != crate::disc::Codec::Ac3 {
continue;
}
let declared = a.channels.count();
// The sub-id this stream currently routes by (low byte of its PID).
let current_sub = (a.pid & 0x00FF) as u8;
// If the stream's current physical sub-stream already matches its
// declared channel count, keep it and claim it.
if probed.get(&current_sub) == Some(&declared) {
claimed.push(current_sub);
continue;
}
// Otherwise find an unclaimed physical sub-stream whose REAL channel
// count equals the declared count.
let pick = probed
.iter()
.find(|(sub, ch)| **ch == declared && !claimed.contains(*sub))
.map(|(sub, _)| *sub);
if let Some(sub) = pick {
let new_pid = 0xBD00 | sub as u16;
if new_pid != a.pid {
tracing::debug!(
target: "freemkv::scan",
old_pid = a.pid,
new_pid,
declared_channels = declared,
"dvd: re-routed AC-3 audio to physical sub-stream matching channel count"
);
a.pid = new_pid;
changed += 1;
}
claimed.push(sub);
} else {
// No physical match — leave the ordinal assignment, but claim its
// current sub so later streams don't steal a slot it may still use.
claimed.push(current_sub);
}
}
changed
}
/// Probe the first feature extent of a DVD title through a (decrypted) sector
/// source and re-route its AC-3 audio PIDs to the physically-correct
/// sub-streams. A bounded, best-effort scan: any read error or empty probe
/// leaves the ordinal assignment untouched.
///
/// `reader` MUST yield PLAINTEXT VOB bytes (i.e. a `DecryptingSectorSource` on a
/// CSS disc) — probing scrambled sectors yields no AC-3 syncs and is a safe
/// no-op. Returns the number of audio streams whose PID changed.
pub fn probe_and_remap<S: SectorSource + ?Sized>(
reader: &mut S,
title: &mut crate::disc::DiscTitle,
) {
// Only DVD (MPEG-PS) titles carry private_stream_1 AC-3 sub-streams.
if title.content_format != crate::disc::ContentFormat::MpegPs {
return;
}
// Nothing to disambiguate unless there is at least one AC-3 audio stream.
let has_ac3 = title
.streams
.iter()
.any(|s| matches!(s, Stream::Audio(a) if a.codec == crate::disc::Codec::Ac3));
if !has_ac3 {
return;
}
let Some(ext) = title.extents.first() else {
return;
};
let count: u16 = ext.sector_count.min(PROBE_SECTORS as u32) as u16;
if count == 0 {
return;
}
let mut buf = vec![0u8; count as usize * 2048];
// `recovery=false`: a single best-effort attempt — the probe must never
// stall the mux or hammer a marginal drive. On any error, bail to ordinal.
let n = match reader.read_sectors(ext.start_lba, count, &mut buf, false) {
Ok(n) => n,
Err(_) => return,
};
buf.truncate(n);
let probed = probe_ac3_substream_channels(&buf);
crate::diag::dump_dvd_substream_probe(title.playlist_id, &probed);
remap_audio_pids(&mut title.streams, &probed);
}
#[cfg(test)]
mod tests {
use super::*;
use crate::disc::{AudioChannels, AudioStream, Codec, LabelPurpose, SampleRate};
/// Build a single, correctly-SIZED AC-3 frame whose `acmod`/`lfeon` encode a
/// known channel count. `byte4` is `fscod=0 | frmsizecod=0`, so
/// `ac3_frame_size` reports 128 bytes and the frame is zero-padded to exactly
/// that — this lets `max_substream_channels` advance frame-by-frame over a
/// multi-frame payload exactly as it does on real VOB data. The BSI bits are
/// laid down with a writer so the test never hand-miscomputes the lfeon
/// offset, matching `acmod_channels`' reader.
fn ac3_frame(acmod: u8, lfeon: bool) -> Vec<u8> {
let mut bits: Vec<u8> = Vec::new();
let push = |val: u32, n: usize, bits: &mut Vec<u8>| {
for i in (0..n).rev() {
bits.push(((val >> i) & 1) as u8);
}
};
push(acmod as u32, 3, &mut bits);
if (acmod & 0x1) != 0 && acmod != 0x1 {
push(0, 2, &mut bits); // cmixlev
}
if (acmod & 0x4) != 0 {
push(0, 2, &mut bits); // surmixlev
}
if acmod == 0x2 {
push(0, 2, &mut bits); // dsurmod
}
push(lfeon as u32, 1, &mut bits);
// Pack the bit vector MSB-first into bytes (byte6 onward).
let mut tail = Vec::new();
let mut cur = 0u8;
for (i, b) in bits.iter().enumerate() {
cur = (cur << 1) | b;
if i % 8 == 7 {
tail.push(cur);
cur = 0;
}
}
let rem = bits.len() % 8;
if rem != 0 {
cur <<= 8 - rem;
tail.push(cur);
}
// AC-3 frame: 0x0B 0x77 crc(2) byte4(fscod=0,frmsizecod=0) bsid<<3 then BSI.
let mut frame = vec![0x0B, 0x77, 0x00, 0x00, 0x00, 8u8 << 3];
frame.extend_from_slice(&tail);
// frmsizecod=0 @ 48kHz → 64 words = 128 bytes. Pad to the real size so
// the frame-stepping in max_substream_channels lands on the next sync.
frame.resize(128, 0);
frame
}
/// Build a minimal `private_stream_1` PES carrying `frames` for `sub_id`,
/// each preceded only by the 4-byte AC-3 sub-header at the PES head. Mirrors
/// the on-disc layout the PS demux expects: PES start `0x000001BD`, length,
/// PES header (no PTS), sub-header `[sub_id, frame_count, ptr_hi, ptr_lo]`,
/// then the concatenated AC-3 frames.
fn ps_ac3_frames(sub_id: u8, frames: &[Vec<u8>]) -> Vec<u8> {
// PES sub-header for AC-3: sub_id + frame_count + 2-byte access ptr.
let mut payload = vec![sub_id, frames.len() as u8, 0x00, 0x04];
for f in frames {
payload.extend_from_slice(f);
}
// PES packet: start code 00 00 01 BD, length(2), flags(2), hdr_len(0).
let pes_payload_len = 3 + payload.len(); // flags(2)+hdrlen(1)+payload
let mut pkt = vec![0x00, 0x00, 0x01, 0xBD];
pkt.extend_from_slice(&(pes_payload_len as u16).to_be_bytes());
pkt.extend_from_slice(&[0x80, 0x00, 0x00]); // no PTS, header_data_len=0
pkt.extend_from_slice(&payload);
pkt
}
/// Single-frame `private_stream_1` PES — the common case in existing tests.
fn ps_ac3(sub_id: u8, acmod: u8, lfeon: bool) -> Vec<u8> {
ps_ac3_frames(sub_id, &[ac3_frame(acmod, lfeon)])
}
fn ac3_stream(pid: u16, channels: AudioChannels) -> Stream {
Stream::Audio(AudioStream {
pid,
codec: Codec::Ac3,
channels,
language: "en".into(),
sample_rate: SampleRate::S48,
secondary: false,
purpose: LabelPurpose::Normal,
label: String::new(),
})
}
/// The probe decodes the real channel count of each physical sub-stream.
/// 0x80 carries a 2.0 frame (acmod=2,no lfe → 2ch); 0x81 carries 5.1
/// (acmod=7 + lfe → 6ch).
#[test]
fn probe_decodes_per_substream_channels() {
let mut bytes = ps_ac3(0x80, 2, false);
bytes.extend(ps_ac3(0x81, 7, true));
let probed = probe_ac3_substream_channels(&bytes);
assert_eq!(probed.get(&0x80), Some(&2), "0x80 is the 2.0 down-mix");
assert_eq!(probed.get(&0x81), Some(&6), "0x81 is the 5.1 main mix");
}
/// GREENLAND regression — the probe must read each sub-stream's TRUE
/// (max-mix) channel count, not be poisoned by an unrepresentative head
/// frame, and must NOT cross-contaminate between sub-streams.
///
/// Mirrors the real on-disc layout that caused the mis-read: the feature
/// head carries `0x80` opening with a 2.0 frame and THEN a 5.1 frame (its
/// real main mix), interleaved with `0x81` carrying only 2.0. The old
/// first-frame probe read `0x80=2` (the logo bed) and missed the 5.1; the
/// max-over-frames probe must report `0x80=6` and `0x81=2`.
#[test]
fn probe_reads_max_channels_no_cross_contamination() {
let mut bytes = Vec::new();
// 0x80 opens with a 2.0 frame (the logo bed)...
bytes.extend(ps_ac3_frames(0x80, &[ac3_frame(2, false)]));
// ...0x81 interleaves a pure-2.0 PES (must NOT bleed 6 into 0x80)...
bytes.extend(ps_ac3_frames(
0x81,
&[ac3_frame(2, false), ac3_frame(2, false)],
));
// ...then 0x80 reaches its real 5.1 main mix (acmod=7 + lfe → 6 ch),
// with a trailing 2.0 frame in the SAME PES to prove we take the max,
// not the last frame.
bytes.extend(ps_ac3_frames(
0x80,
&[ac3_frame(7, true), ac3_frame(2, false)],
));
let probed = probe_ac3_substream_channels(&bytes);
assert_eq!(
probed.get(&0x80),
Some(&6),
"0x80's real 5.1 mix must win over its 2.0 head/tail frames"
);
assert_eq!(
probed.get(&0x81),
Some(&2),
"0x81 is a pure 2.0 stream — must not absorb 0x80's 6-channel frame"
);
}
/// SILENCE-OF-THE-LAMBS regression: the IFO declares ONE 5.1 AC-3 stream and
/// the ordinal mapping put it at 0x80, but physically 0x80 is the 2.0
/// down-mix and the 5.1 lives at 0x81. After probe+remap the declared 5.1
/// stream must route to 0x81 (PID 0xBD81), NOT the ordinal 0x80.
#[test]
fn remap_routes_declared_51_to_physical_51_substream() {
// Physical layout: 0x80 = 2.0, 0x81 = 5.1 (reversed vs ordinal).
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 2u8);
probed.insert(0x81u8, 6u8);
// Declared: one 5.1 stream, ordinally assigned 0x80 (PID 0xBD80).
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 1, "the one 5.1 stream must be re-routed");
let Stream::Audio(a) = &streams[0] else {
panic!("audio")
};
assert_eq!(
a.pid, 0xBD81,
"declared 5.1 must route to physical 0x81 (the real 5.1), not ordinal 0x80"
);
}
/// Conservative no-op: when the physical order already matches the IFO
/// order (0x80 = 5.1 as declared), remap changes nothing.
#[test]
fn remap_noop_when_physical_matches_ordinal() {
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 6u8); // 0x80 really is the 5.1
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 0, "matching physical order is a no-op");
let Stream::Audio(a) = &streams[0] else {
panic!()
};
assert_eq!(a.pid, 0xBD80);
}
/// Two declared streams (5.1 + 2.0) where the physical order is reversed:
/// 0x80=2.0, 0x81=5.1. The 5.1 declaration must claim 0x81 and the 2.0
/// declaration must claim 0x80 — no collision, both correct.
#[test]
fn remap_two_streams_no_collision() {
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 2u8);
probed.insert(0x81u8, 6u8);
// Declared order: 5.1 first (ordinal 0x80), 2.0 second (ordinal 0x81).
let mut streams = vec![
ac3_stream(0xBD80, AudioChannels::Surround51),
ac3_stream(0xBD81, AudioChannels::Stereo),
];
remap_audio_pids(&mut streams, &probed);
let pids: Vec<u16> = streams
.iter()
.filter_map(|s| match s {
Stream::Audio(a) => Some(a.pid),
_ => None,
})
.collect();
assert_eq!(
pids,
vec![0xBD81, 0xBD80],
"5.1→0x81, 2.0→0x80, no collision"
);
}
/// Empty probe (unreadable / scrambled VOB) is a no-op — the ordinal
/// assignment survives so behaviour never regresses below today's.
#[test]
fn remap_empty_probe_is_noop() {
let probed = BTreeMap::new();
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 0);
let Stream::Audio(a) = &streams[0] else {
panic!()
};
assert_eq!(a.pid, 0xBD80, "no probe data → keep ordinal");
}
}
+994 -660
View File
File diff suppressed because it is too large Load Diff
+1661
View File
File diff suppressed because it is too large Load Diff
+1080 -20
View File
File diff suppressed because it is too large Load Diff
+4060 -294
View File
File diff suppressed because it is too large Load Diff
+1059 -1141
View File
File diff suppressed because it is too large Load Diff
+398 -39
View File
@@ -1,11 +1,10 @@
//! Single source of truth for what to do when a sector read fails.
//!
//! Both Pass 1 (`Disc::sweep`) and Pass 2-N (`Disc::patch`) call into
//! `handle_read_error` after every failed `read_sectors`. The handler
//! classifies the error, updates the in-flight context (counters,
//! damage window, retry budgets), and returns a `ReadAction` the caller
//! dispatches on. Every read goes through the same gate — no path can
//! silently skip pause/skip/jump/abort logic.
//! Pass 1 (`Disc::sweep`) calls into `handle_read_error` after every failed
//! `read_sectors`. The handler classifies the error, updates the in-flight
//! context (counters, damage window, retry budgets), and returns a
//! `ReadAction` the caller dispatches on. Pass N patch has its own
//! `handle_read_failure` in `disc/patch.rs` that does not route here.
//!
//! Adding a new error class = add one arm in `handle_read_error`.
//! Adding new logging on errors = one place.
@@ -34,14 +33,20 @@ pub struct ReadCtx {
/// Sliding window of recent read outcomes (true=ok, false=fail).
/// Capped at `damage_window_max`. Drives damage-jump decisions.
pub damage_window: Vec<bool>,
/// Maximum number of outcome entries kept in `damage_window`; the
/// oldest is evicted once this is exceeded. A whole count (e.g. 16).
pub damage_window_max: usize,
/// Fraction of `damage_window` entries that must be failures before
/// the window-based damage-jump fires, as a whole-number percentage
/// (e.g. `12` = 12%).
pub damage_threshold_pct: usize,
/// Trigger a damage-jump after this many consecutive outer-batch
/// failures, even when the damage_window isn't full yet. Pass 1
/// uses a small value (4) so we don't spend ~40 minutes grinding
/// to fill a 16-block window before the first jump on a damage
/// zone we entered cleanly. Pass N uses a larger value (or
/// disables this — see `bisect_on_marginal`) because Pass N's
/// uses a small value (1 — jump on the first outer failure; see
/// the 2026-05-11 rewrite in `for_sweep`) so we don't spend ~40
/// minutes grinding to fill a 16-block window before the first jump
/// on a damage zone we entered cleanly. Pass N uses a larger value
/// (or disables this — see `bisect_on_marginal`) because Pass N's
/// whole job IS to grind on the bad ranges.
pub fast_jump_threshold: u64,
/// Multiplier applied to damage-jump distance. Doubles each jump,
@@ -151,8 +156,12 @@ impl ReadCtx {
/// outer-batch failure — the user's wedge-prevention principle
/// (2026-05-11): once the drive returns ANY recoverable error,
/// retrying the same LBA quickly is what triggers the firmware
/// fast-fail transition. Jump immediately, never retry in Pass 1.
/// Pass N owns retries — it gets per-sector timeouts that don't
/// fast-fail transition. On the damage-jump and marginal paths Pass 1
/// jumps immediately rather than grinding the same LBA. Transient errors
/// (NOT_READY, bridge degradation) are still retried a small bounded
/// number of times (`NOT_READY_MAX_RETRIES` / `BRIDGE_DEGRADATION_MAX_RETRIES`)
/// in both passes before falling through to the skip path.
/// Pass N owns the heavy retries — it gets per-sector timeouts that don't
/// hammer the firmware the same way.
pub fn for_sweep(batch: u16) -> Self {
Self {
@@ -193,9 +202,10 @@ impl ReadCtx {
/// patch loop's whole job is to chip away at bad ranges — being
/// more eager to skip clustered bad sectors converges faster on
/// the recoverable good sectors inside a range. The patch-side
/// `compute_damage_skip` reads its threshold from
/// `PASSN_DAMAGE_THRESHOLD_PCT`; keep the two in sync until the
/// patch loop's damage-skip is unified with `handle_read_error`'s
/// `compute_damage_skip` reads its threshold directly from
/// `PASSN_DAMAGE_THRESHOLD_PCT`, which is an alias for this crate's
/// `PATCH_DAMAGE_THRESHOLD_PCT`, so the two are always in sync.
/// The patch loop's damage-skip is not yet unified with `handle_read_error`'s
/// jump path. (v0.20.8 unification attempt found the unification
/// itself blocked on the size-aware `range_remaining/4` cap that
/// lives in `compute_damage_skip` but not in
@@ -240,6 +250,13 @@ impl ReadCtx {
// drive recovered, so further wedges should reset the skip
// budget instead of accumulating toward a real abort.
self.wedge_count = 0;
// A successful read also means the bridge recovered, so the
// 15s-cooldown retry budget should be available again for the
// next bridge-degradation event. Without this reset the budget
// saturates permanently after 5 cumulative events across the
// whole pass and later degradations skip the cooldown retry,
// needlessly losing data.
self.bridge_degradation_count = 0;
// Outer-success only: a good single-sector read inside a
// bisect doesn't mean we've left the damaged batch. Only an
// outer-batch success resets the outer-failure counter.
@@ -259,6 +276,13 @@ impl ReadCtx {
if self.in_damage_zone && self.consecutive_good >= self.damage_window_max as u64 {
self.in_damage_zone = false;
self.last_error_family = None;
// Reset the damage-jump multiplier so the NEXT zone starts
// from the base jump distance. Without this the multiplier
// stays at whatever the prior zone inflated it to (up to
// MAX_JUMP_MULTIPLIER=64), so the next zone's first jump is
// 64x oversized and skips recoverable data. The field doc
// promises this reset.
self.jump_multiplier = 1;
}
}
@@ -316,7 +340,8 @@ pub enum ReadAction {
// bridge wedges 524 ms after a 5.4-second internal ECC retry. The
// post-failure pauses give the drive — and the bridge — time to settle.
/// Pause between a failed read and the next read attempt — applied
/// uniformly to Pass 1 sweep and Pass N patch.
/// by Pass 1 sweep via `handle_read_error`. Pass N patch uses its own
/// `POST_FAILURE_PAUSE_SECS` (see `disc/patch.rs`).
///
/// 2026-05-11 reframe: a failed read is a failed read, regardless of
/// which pass is running. The prior split (1s for Pass N, 5s for Pass
@@ -334,7 +359,7 @@ const FAIL_PAUSE_SECS: u64 = 5;
/// FIRST read failure after a clean run, before the drive has had a
/// chance to cycle in retries that push it toward fast-fail).
///
/// Empirical: 2026-05-11 Dune Pt 2 wedge incident showed 7 medium
/// Empirical: a 2026-05-11 wedge incident showed 7 medium
/// errors in 6.5 seconds (~1s per attempt + ~1s pause) push the
/// BU40N's firmware into IllegalRequest fast-fail mode permanently.
/// Once there, only physical eject + reload clears it. Giving the
@@ -345,7 +370,7 @@ const FAIL_PAUSE_SECS: u64 = 5;
/// Cost on clean discs: zero (first-error path doesn't trigger).
/// Cost on damaged discs: ~30s × N damage zones; on a 5-zone disc
/// that's 2.5 min extra. Trade for never wedging the drive.
const ZONE_ENTRY_COOLDOWN_SECS: u64 = 30;
pub(crate) const ZONE_ENTRY_COOLDOWN_SECS: u64 = 30;
/// Cooldown when a long streak of failures suggests the drive is
/// stuck in a damage zone and needs MORE breathing room than the
/// standard inter-error pause. Same value as `FAIL_PAUSE_SECS`
@@ -375,9 +400,9 @@ const JUMP_BASE_SECTORS: u64 = 1024;
// When the BU40N (or similar drives) hits a physical-damage cluster,
// its firmware can transition into a "wedge" state where it returns
// HARDWARE_ERROR or ILLEGAL_REQUEST for every subsequent read —
// often for many LBAs after the actual bad sector. Per CLAUDE.md
// "Bad-sector handling" rule #2: "Recovery requires eject+reload OR
// significant cool-down."
// often for many LBAs after the actual bad sector. Once wedged,
// recovery requires either a physical eject + reload or a significant
// cool-down period; hammering the same LBA only deepens the state.
//
// Pass 1's pre-fix behavior was to immediately AbortPass on the
// first HARDWARE_ERROR / ILLEGAL_REQUEST, killing the rip at
@@ -396,10 +421,10 @@ const JUMP_BASE_SECTORS: u64 = 1024;
/// One-gigabyte jump (1024 MiB) on each wedge. Big enough to clear
/// almost any single-cluster damage zone we've seen.
const WEDGE_JUMP_SECTORS: u64 = 524_288;
/// Cooldown pause after each wedge. Per CLAUDE.md the drive needs
/// "significant cool-down"; 30 s strikes a balance between giving
/// the drive a chance to recover and not stalling the rip if the
/// drive is permanently stuck.
/// Cooldown pause after each wedge. A wedged drive needs a
/// significant cool-down to leave fast-fail; 30 s strikes a balance
/// between giving the drive a chance to recover and not stalling the
/// rip if the drive is permanently stuck.
const WEDGE_PAUSE_SECS: u64 = 30;
/// Bail after this many consecutive wedges with no good read in
/// between. At 1 GB jumps this lets us scan ~16 GB worth of fully
@@ -463,8 +488,13 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
.unwrap_or(SenseFamily::Other);
// Zone-entry tracking: this is the first error after a clean run
// (or the first error of the sweep).
if !ctx.in_damage_zone && !ctx.bisecting {
// (or the first error of the sweep). Capture the genuine
// clean->damaged transition here, BEFORE mutating in_damage_zone,
// so the 30s zone-entry cooldown below keys off the real
// transition rather than re-deriving it from a counter that the
// fast-jump path resets after every jump.
let is_zone_entry_transition = !ctx.in_damage_zone && !ctx.bisecting;
if is_zone_entry_transition {
ctx.in_damage_zone = true;
ctx.zones_entered += 1;
}
@@ -501,13 +531,20 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
);
if is_wedge_transition {
// NOTE: this is the FIRST escalation into the hardware/illegal-request
// sense family — NOT a confirmed wedge. Drives frequently recover and keep
// reading after one such error (a single bad spot), so calling it a "wedge"
// here over-claims (it sent past investigations chasing a drive ghost). A
// genuine wedge is PERSISTENT — see the `wedge_skip` / WEDGE_ABORT_THRESHOLD
// path below, which only fires after repeated fast-fails with no recovery.
tracing::warn!(
target: "freemkv::disc",
phase = "wedge_transition",
phase = "fastfail_escalation",
errors_in_zone = ctx.total_errors,
ms_since_last_success,
new_family = ?current_family,
"drive entered wedge / fast-fail family (was returning recoverable medium errors before this)"
"drive escalated into the fast-fail sense family (was returning recoverable medium \
errors before this) often transient; only a PERSISTENT run is a real wedge"
);
}
@@ -519,11 +556,17 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
return ReadAction::AbortPass;
}
// 2. Bridge degradation: NOT_READY with the well-known signature
// (sense_key=2, ASC=0x04, ASCQ=0x3E). Drive's bridge is in a
// semi-stuck state but typically recovers after a long cooldown.
// If we've exhausted our retry budget, fall through to the
// marginal/skip path below.
// 2. Bridge degradation: the SCSI status byte is non-standard —
// neither GOOD (0x00), CHECK CONDITION (0x02), nor TRANSPORT
// FAILURE (0xFF). The USB bridge firmware returns these bogus
// status bytes (e.g. 0x04, 0x05) with empty sense data when it
// enters a semi-stuck state preceding a crash. This is keyed on
// the status byte alone, NOT on sense_key/ASC/ASCQ — a real
// NOT_READY 04/3E bad-sector error arrives as CHECK CONDITION
// (0x02) and is handled by the generic NOT_READY branch below.
// The bridge typically recovers after a long cooldown; if we've
// exhausted our retry budget, fall through to the marginal/skip
// path below.
if err.is_bridge_degradation() && ctx.bridge_degradation_count < BRIDGE_DEGRADATION_MAX_RETRIES
{
ctx.bridge_degradation_count += 1;
@@ -569,9 +612,13 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
// AbortPass after N consecutive wedges with no successful
// read in between.
if sense_key == scsi::SENSE_KEY_HARDWARE_ERROR || sense_key == scsi::SENSE_KEY_ILLEGAL_REQUEST {
if !ctx.bisecting {
// Count every wedge, including bisect-inner ones. A wedge is a
// firmware fast-fail state regardless of whether we're inside a
// bisect; if we did NOT count bisect-inner wedges, a drive that
// wedges mid-bisect would burn a 30s WEDGE_PAUSE cooldown per
// inner sector and never reach WEDGE_ABORT_THRESHOLD from inside
// the bisect — ~16 min of cooldown sleeping on a batch=32 bisect.
ctx.wedge_count += 1;
}
if ctx.wedge_count >= WEDGE_ABORT_THRESHOLD {
tracing::warn!(
target: "freemkv::disc",
@@ -665,8 +712,7 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
// branch for future tuning. Pass N (bisect_on_marginal=true)
// uses the standard pauses — it's running single-sector retries
// on already-known-bad LBAs by design.
let is_zone_entry =
ctx.consecutive_outer_failures == 1 && !ctx.bisecting && !ctx.bisect_on_marginal;
let is_zone_entry = is_zone_entry_transition && !ctx.bisecting && !ctx.bisect_on_marginal;
let pause_secs = if is_zone_entry {
ZONE_ENTRY_COOLDOWN_SECS
} else if ctx.consecutive_failures >= CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD {
@@ -692,7 +738,7 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
// Two triggers, evaluated in order:
//
// a. **Fast-entry** — `consecutive_outer_failures >= fast_jump_threshold`.
// Fires on Pass 1 (threshold=4) so we don't spend ~40 min
// Fires on Pass 1 (threshold=1) so we don't spend ~40 min
// grinding to fill a 16-block damage window before the
// first jump on a damage zone we entered cleanly. Doesn't
// fire on Pass N (threshold=u64::MAX).
@@ -1029,6 +1075,43 @@ mod tests {
}
}
#[test]
fn pass_1_subsequent_in_zone_errors_skip_long_cooldown() {
// Regression: the fast-jump path resets consecutive_outer_failures
// to 0 after each jump, so the next in-zone error re-increments it
// to 1. Zone-entry must key off the genuine clean->damaged
// transition (in_damage_zone), not the counter, otherwise every
// error in a damaged region pays the 30 s cooldown.
let mut ctx = ReadCtx::for_sweep(32);
// First error: genuine zone entry, gets the long cooldown.
let first = handle_read_error(&medium_err(), &mut ctx);
match first {
ReadAction::JumpAhead { pause_secs, .. } => assert_eq!(
pause_secs,
ZONE_ENTRY_COOLDOWN_SECS + POST_JUMP_EXTRA_PAUSE_SECS
),
other => panic!("expected JumpAhead on first error, got {other:?}"),
}
// We are now still in the damage zone; the jump reset the outer
// counter. A second error must NOT re-arm the 30 s cooldown.
assert!(ctx.in_damage_zone);
let second = handle_read_error(&medium_err(), &mut ctx);
let pause = match second {
ReadAction::JumpAhead { pause_secs, .. } => pause_secs,
ReadAction::SkipBlock { pause_secs } => pause_secs,
other => panic!("expected pausing action, got {other:?}"),
};
assert_ne!(
pause,
ZONE_ENTRY_COOLDOWN_SECS + POST_JUMP_EXTRA_PAUSE_SECS,
"subsequent in-zone error must not pay the 30 s zone-entry cooldown"
);
assert!(
pause <= FAIL_PAUSE_SECS + POST_JUMP_EXTRA_PAUSE_SECS,
"subsequent in-zone pause should be the standard fail pause, got {pause}"
);
}
#[test]
fn pass_n_pauses_uniformly_on_failed_read() {
// Pass N (bisect_on_marginal=true) is exempt from the
@@ -1067,6 +1150,66 @@ mod tests {
);
}
#[test]
fn jump_multiplier_resets_after_damage_zone_exit() {
// A zone that doubles the multiplier must not carry the inflated
// value into the next zone — otherwise the next zone's first
// jump is up to 64x oversized and skips recoverable data.
let mut ctx = ReadCtx::for_sweep(32);
// First zone: a few errors push jumps and double the multiplier.
for _ in 0..4 {
handle_read_error(&medium_err(), &mut ctx);
}
assert!(
ctx.jump_multiplier > 1,
"expected the multiplier to inflate inside a damage zone"
);
// Exit the zone: damage_window_max consecutive good reads.
ctx.bisecting = false;
for _ in 0..ctx.damage_window_max {
ctx.on_success();
}
assert!(!ctx.in_damage_zone, "zone should have exited");
assert_eq!(
ctx.jump_multiplier, 1,
"jump_multiplier must reset to 1 on zone exit"
);
}
#[test]
fn bridge_degradation_count_resets_on_success() {
// After a good read the bridge recovered; the 15s-cooldown retry
// budget must be available again instead of staying saturated
// for the whole pass.
let mut ctx = ReadCtx::for_patch(1);
ctx.bridge_degradation_count = BRIDGE_DEGRADATION_MAX_RETRIES;
ctx.on_success();
assert_eq!(ctx.bridge_degradation_count, 0);
}
#[test]
fn wedge_abort_reachable_during_bisect() {
// A drive that wedges mid-bisect must still reach the abort
// threshold rather than burning a WEDGE_PAUSE cooldown per inner
// sector forever.
let mut ctx = ReadCtx::for_patch(32);
ctx.bisecting = true;
let mut aborted = false;
for _ in 0..WEDGE_ABORT_THRESHOLD {
if matches!(
handle_read_error(&hardware_err(), &mut ctx),
ReadAction::AbortPass
) {
aborted = true;
break;
}
}
assert!(
aborted,
"wedge abort threshold must be reachable from inside a bisect"
);
}
#[test]
fn on_success_resets_failure_counters_and_pushes_window() {
let mut ctx = ReadCtx::for_sweep(32);
@@ -1080,4 +1223,220 @@ mod tests {
assert_eq!(ctx.consecutive_failures, 0);
assert!(*ctx.damage_window.last().unwrap());
}
// ----------------------------------------------------------------
// Additional hardening: retry-budget boundaries, transport-abort
// precedence, and the bounded-jump invariant. These guard against
// off-by-one in the retry caps (which would either hammer a wedging
// drive or give up a recovery one attempt early) and against an
// unbounded jump multiplier skipping the rest of the disc.
// ----------------------------------------------------------------
/// NOT_READY check-condition (status 0x02 so it is NOT classified as
/// bridge degradation, which keys off non-standard status bytes).
/// sense_key=2 with a generic ASC routes to the NOT_READY retry path.
fn not_ready_err() -> Error {
Error::DiscRead {
sector: 100,
status: Some(crate::scsi::SCSI_STATUS_CHECK_CONDITION),
sense: Some(ScsiSense {
sense_key: scsi::SENSE_KEY_NOT_READY,
asc: 0x04,
ascq: 0x00,
}),
}
}
/// Transport failure: SCSI status 0xFF (bridge crash). CLAUDE.md
/// "Bad-sector handling": this aborts the copy.
fn transport_failure_err() -> Error {
Error::DiscRead {
sector: 100,
status: Some(crate::scsi::SCSI_STATUS_TRANSPORT_FAILURE),
sense: None,
}
}
/// Bridge degradation: a non-standard status byte (0x04 - neither
/// GOOD/CHECK/TRANSPORT) with empty sense, per `Error::is_bridge_degradation`.
fn bridge_degradation_err() -> Error {
Error::DiscRead {
sector: 100,
status: Some(0x04),
sense: None,
}
}
#[test]
fn not_ready_retries_capped_at_three_then_falls_through() {
// CLAUDE.md "Bad-sector handling" mode 1: NOT READY -> "Pause 3s,
// retry up to 3x, then mark NonTrimmed." NOT_READY_MAX_RETRIES=3.
// The 1st-3rd NOT_READY must Retry; the 4th must NOT Retry (it
// falls through to skip). Pass N (batch=1) so the marginal-bisect
// branch is irrelevant.
// Mutation that makes this RED: change `ctx.not_ready_retries <
// NOT_READY_MAX_RETRIES` to `<=` (retries 4 times) or to `>`
// (never retries).
let mut ctx = ReadCtx::for_patch(1);
for i in 0..NOT_READY_MAX_RETRIES {
let a = handle_read_error(&not_ready_err(), &mut ctx);
assert!(
matches!(a, ReadAction::Retry { .. }),
"NOT_READY attempt {i} should Retry, got {a:?}"
);
}
// Budget exhausted: the next NOT_READY must not Retry.
let a = handle_read_error(&not_ready_err(), &mut ctx);
assert!(
!matches!(a, ReadAction::Retry { .. }),
"NOT_READY past the retry cap must fall through, got {a:?}"
);
}
#[test]
fn transport_failure_aborts_even_mid_bisect() {
// CLAUDE.md "Bad-sector handling" mode 2: a transport failure
// (bridge crash, status 0xFF) aborts the pass so the outer loop
// can re-enumerate the bridge. This must hold even while
// bisecting and even on Pass N - the wedge-skip/jump paths must
// NOT swallow a real transport crash into a JumpAhead.
// Mutation that makes this RED: move the transport-failure check
// below the HARDWARE/ILLEGAL wedge arm, so a transport failure
// that also carried a wedge-family sense would JumpAhead instead.
let mut ctx = ReadCtx::for_patch(32);
ctx.bisecting = true;
assert_eq!(
handle_read_error(&transport_failure_err(), &mut ctx),
ReadAction::AbortPass
);
// And on a fresh Pass 1 context, still AbortPass.
let mut ctx1 = ReadCtx::for_sweep(32);
assert_eq!(
handle_read_error(&transport_failure_err(), &mut ctx1),
ReadAction::AbortPass
);
}
#[test]
fn bridge_degradation_retries_to_budget_then_falls_through() {
// The bridge-degradation cooldown retry is bounded by
// BRIDGE_DEGRADATION_MAX_RETRIES (=5). The first 5 degradation
// errors must Retry with the long bridge cooldown; the 6th must
// fall through to skip/jump rather than retrying forever and
// stalling the pass.
// Mutation that makes this RED: change the budget comparison
// `ctx.bridge_degradation_count < BRIDGE_DEGRADATION_MAX_RETRIES`
// to `<=` (retries 6 times).
let mut ctx = ReadCtx::for_patch(1);
for i in 0..BRIDGE_DEGRADATION_MAX_RETRIES {
let a = handle_read_error(&bridge_degradation_err(), &mut ctx);
match a {
ReadAction::Retry { pause_secs } => {
assert_eq!(
pause_secs, BRIDGE_DEGRADATION_PAUSE_SECS,
"bridge retry {i} should use the bridge cooldown"
);
}
other => panic!("bridge degradation attempt {i} should Retry, got {other:?}"),
}
}
let a = handle_read_error(&bridge_degradation_err(), &mut ctx);
assert!(
!matches!(a, ReadAction::Retry { .. }),
"bridge degradation past the retry budget must fall through, got {a:?}"
);
}
/// The documented BU40N bad-sector signature: NOT_READY
/// (sense_key=2, ASC=0x04, ASCQ=0x3E) delivered as a CHECK CONDITION
/// (status 0x02). This is the case the old comment on the bridge
/// branch wrongly claimed `is_bridge_degradation` matched.
fn not_ready_04_3e_err() -> Error {
Error::DiscRead {
sector: 100,
status: Some(crate::scsi::SCSI_STATUS_CHECK_CONDITION),
sense: Some(ScsiSense {
sense_key: scsi::SENSE_KEY_NOT_READY,
asc: 0x04,
ascq: 0x3E,
}),
}
}
#[test]
fn not_ready_04_3e_does_not_take_bridge_branch() {
// Regression guard for the misleading-comment fix: the bridge
// branch keys on the *status byte* (non-standard, i.e. not
// GOOD/CHECK/TRANSPORT), NOT on the NOT_READY 04/3E sense. A real
// 04/3E bad-sector error arrives as CHECK CONDITION (0x02), so
// `is_bridge_degradation()` must be false for it, and it must
// route to the generic NOT_READY retry (3 s pause) rather than
// the bridge cooldown (15 s pause).
let err = not_ready_04_3e_err();
assert!(
!err.is_bridge_degradation(),
"04/3E arrives as CHECK CONDITION (0x02); it is not bridge degradation"
);
let mut ctx = ReadCtx::for_patch(1);
match handle_read_error(&err, &mut ctx) {
ReadAction::Retry { pause_secs } => {
assert_eq!(
pause_secs, NOT_READY_PAUSE_SECS,
"04/3E must use the generic NOT_READY pause, not the bridge cooldown"
);
assert_ne!(
pause_secs, BRIDGE_DEGRADATION_PAUSE_SECS,
"04/3E must not take the bridge-degradation branch"
);
// Confirm it really went through the NOT_READY path.
assert_eq!(ctx.not_ready_retries, 1);
assert_eq!(ctx.bridge_degradation_count, 0);
}
other => panic!("04/3E should Retry via the NOT_READY path, got {other:?}"),
}
}
#[test]
fn jump_multiplier_caps_and_jump_distance_stays_bounded() {
// CLAUDE.md damage-jump: multiplier doubles per jump but is
// capped at MAX_JUMP_MULTIPLIER=64 (the "4 GiB cap"); a single
// jump must never be allowed to grow without bound and skip the
// rest of the disc. Drive a long single-sector failure streak on
// a sweep ctx with a tiny window so window-trigger jumps fire
// repeatedly, and verify the multiplier saturates at 64 and the
// emitted jump distance equals JUMP_BASE_SECTORS * batch * 64.
// Mutation that makes this RED: remove the
// `.min(MAX_JUMP_MULTIPLIER)` on the multiplier doubling, or use
// wrapping/non-saturating mul -> distance overshoots or panics.
const MAX_JUMP_MULTIPLIER: u64 = 64;
let batch: u16 = 32;
let mut ctx = ReadCtx::for_sweep(batch);
// Small window + 0% threshold so every failure can window-trigger
// a jump and keep doubling the multiplier toward the cap.
ctx.damage_window_max = 2;
ctx.damage_threshold_pct = 0;
let mut last_jump_sectors = 0u64;
for _ in 0..40 {
// Reset bisecting flag defensively; these are outer failures.
ctx.bisecting = false;
if let ReadAction::JumpAhead { sectors, .. } =
handle_read_error(&medium_err(), &mut ctx)
{
last_jump_sectors = sectors;
}
assert!(
ctx.jump_multiplier <= MAX_JUMP_MULTIPLIER,
"jump_multiplier {} exceeded the cap {}",
ctx.jump_multiplier,
MAX_JUMP_MULTIPLIER
);
}
// After saturation, the jump distance is exactly base*batch*cap.
let expected = JUMP_BASE_SECTORS * batch as u64 * MAX_JUMP_MULTIPLIER;
assert_eq!(
last_jump_sectors, expected,
"saturated jump distance must equal base*batch*64"
);
}
}
File diff suppressed because it is too large Load Diff
+43 -50
View File
@@ -8,15 +8,13 @@
//! during the post-read work; throughput tops out at the *sum* of
//! both costs.
//!
//! 0.17.11 introduced a bespoke producer/consumer split (the now-
//! removed `disc/sweep_pipeline.rs`) to overlap the two stages. 0.18
//! collapses that split — together with the analogous splits patch
//! and mux need — onto the generic [`crate::io::Pipeline`] +
//! [`crate::io::Sink`] primitive. This module is the sweep-specific
//! `Sink` impl; the producer-side state machine (read_error context,
//! decrypt, set_speed, halt) stays in `Disc::sweep` in `disc/mod.rs`.
//! A producer/consumer split overlaps the two stages on the generic
//! [`crate::io::Pipeline`] + [`crate::io::Sink`] primitive. This module
//! is the sweep-specific `Sink` impl; the producer-side state machine
//! (read_error context, decrypt, set_speed, halt) stays in
//! `Disc::sweep` in `disc/mod.rs`.
//!
//! Correctness invariants preserved (same as 0.17.11):
//! Correctness invariants preserved:
//! - Mapfile is single-writer (consumer-only). No locking.
//! - All `read_error::ReadCtx` state stays on the producer thread.
//! - `set_speed` calls happen on the producer thread (same thread that
@@ -25,9 +23,8 @@
//! intact in the consumer (write before record), so the on-disk
//! invariant "mapfile only marks Finished what the file has
//! received" survives a crash mid-pass.
//! - The BU40N+Initio bridge wedge concern is unchanged: only one
//! SCSI command in flight at a time, error-path timing identical,
//! no new retry logic.
//! - Only one SCSI command is in flight at a time; error-path timing
//! is identical and no new retry logic is introduced.
use std::io::{Seek, SeekFrom, Write};
use std::sync::mpsc::{Receiver, SyncSender, sync_channel};
@@ -40,7 +37,7 @@ use super::mapfile::{MapStats, Mapfile, SectorStatus};
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. 64 KB
/// matches the existing zero_gap chunk size used by the pre-split
/// sweep loop.
const ZERO_CHUNK: usize = 65 * 1024;
const ZERO_CHUNK: usize = 64 * 1024;
/// Producer → Consumer messages. The consumer applies these in FIFO
/// order; ordering of file writes and mapfile records across items is
@@ -69,6 +66,14 @@ pub(super) enum WorkItem {
/// tell them apart without parsing a flag.
GapFill { pos: u64, len: u64 },
/// Post-read verify downgrade. The producer's `UnitVerifier` found that the
/// just-`Finished` clip unit at `[pos, pos+len)` is confidently undecryptable
/// (a silent bad read). The consumer re-records the range as `NonTrimmed` so
/// the patch pass re-reads it — the ISO bytes (ciphertext) already written by
/// the preceding `Good` are left in place for the patch to overwrite. FIFO
/// pipe ordering guarantees this arrives AFTER the `Good` that wrote them.
MarkBad { pos: u64, len: u64 },
/// Producer wants the latest mapfile stats for the progress
/// callback. Consumer responds on `prog_tx` with a fresh
/// [`ProgressSnapshot`]. Best-effort: if the producer hasn't
@@ -151,63 +156,51 @@ impl Sink<WorkItem> for SweepSink {
WorkItem::Good { pos, buf } => {
// Decrypt is on the producer; consumer assumes plaintext.
let len = buf.len() as u64;
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf)
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, len, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&buf)?;
self.map.record(pos, len, SectorStatus::Finished)?;
}
WorkItem::BisectGood { pos, buf } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf[..])
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, 2048, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&buf[..])?;
self.map.record(pos, 2048, SectorStatus::Finished)?;
}
WorkItem::BisectBad { pos } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&self.zero[..2048])
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, 2048, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&self.zero[..2048])?;
self.map.record(pos, 2048, SectorStatus::NonTrimmed)?;
}
WorkItem::SkipFill { pos, len } | WorkItem::GapFill { pos, len } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file.seek(SeekFrom::Start(pos))?;
// Subsequent writes are sequential; `WritebackFile`'s
// seek-elision keeps them on the writeback pipeline path.
let mut filled = 0u64;
while filled < len {
let chunk = (len - filled).min(self.zero.len() as u64) as usize;
self.file
.write_all(&self.zero[..chunk])
.map_err(|e| Error::IoError { source: e })?;
self.file.write_all(&self.zero[..chunk])?;
filled += chunk as u64;
}
self.map
.record(pos, len, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
self.map.record(pos, len, SectorStatus::NonTrimmed)?;
}
WorkItem::MarkBad { pos, len } => {
// Verify downgrade: the ISO bytes are already written by the
// preceding Good; only the mapfile status changes so patch
// re-reads this range. No file write.
self.map.record(pos, len, SectorStatus::NonTrimmed)?;
}
WorkItem::StatsRequest => {
let stats = self.map.stats();
// DAMAGE only — NOT NonTried. NonTried is the unread remainder
// ahead of the sweep head, not damage; including it made the live
// located drilldown (at-risk movie time + range count) treat the
// whole unread disc as confirmed damage, so at sweep start it
// showed ~full-movie at-risk and melted to 0 as the sweep
// progressed. Matches the one-shot progress path, which already
// excludes NonTried.
let bad_ranges = self.map.ranges_with(&[
SectorStatus::NonTrimmed,
SectorStatus::Unreadable,
SectorStatus::NonScraped,
SectorStatus::NonTried,
]);
// Best-effort: drop on backpressure; producer's cache
// stays current enough.
@@ -230,7 +223,7 @@ impl Sink<WorkItem> for SweepSink {
// Non-regular outputs (/dev/null, pipes) always fail
// sync_all; that's not a real error.
}
self.map.flush().map_err(|e| Error::IoError { source: e })?;
self.map.flush()?;
Ok(ConsumerSummary {
stats: self.map.stats(),
+1021
View File
File diff suppressed because it is too large Load Diff
+80
View File
@@ -25,8 +25,14 @@ pub struct DriveCapture {
/// A single GET CONFIGURATION feature response from the drive.
#[derive(Debug, Clone)]
pub struct CapturedFeature {
/// MMC-6 GET CONFIGURATION feature code (e.g. `0x010D` = AACS).
pub code: u16,
/// Static human-readable label from the internal `FEATURES` table —
/// not a device-reported string.
pub name: &'static str,
/// Raw feature-descriptor payload bytes, with the 8-byte GET
/// CONFIGURATION header stripped (i.e. `buf[8..]`). Unlike
/// [`DriveCapture::gc_010c`], which retains the full header.
pub data: Vec<u8>,
}
@@ -114,3 +120,77 @@ pub fn mask_bytes(data: &[u8]) -> Vec<u8> {
})
.collect()
}
#[cfg(test)]
mod tests {
//! Privacy-masking + capture-orchestration tests.
//!
//! `mask_string` / `mask_bytes` redact identifying characters before
//! a drive capture leaves the machine: every ASCII letter → 'A',
//! every ASCII digit → '0', everything else (punctuation, spaces,
//! control bytes, non-ASCII) is preserved verbatim so structural
//! framing (offsets, separators) survives for diffing.
use super::*;
#[test]
fn mask_string_letters_become_a_digits_become_zero() {
// Mixed case letters all collapse to 'A'; digits to '0'.
assert_eq!(mask_string("HL-DT-ST"), "AA-AA-AA");
assert_eq!(mask_string("BU40N"), "AA00A");
}
#[test]
fn mask_string_preserves_non_alnum_punctuation_and_space() {
// Separators and spaces must be preserved so the masked output
// keeps the same shape as the original (the whole point of a
// structure-preserving redaction).
assert_eq!(mask_string("1.04"), "0.00");
assert_eq!(mask_string("a b-c.d_e"), "A A-A.A_A");
}
#[test]
fn mask_string_preserves_non_ascii_chars() {
// is_ascii_alphabetic/is_ascii_digit are false for non-ASCII, so
// multibyte chars pass through unchanged (no mojibake, no panic).
// 'c','a','f' are ASCII letters → 'A'; 'é' is non-ASCII →
// preserved; '9' → '0'.
assert_eq!(mask_string("café9"), "AAAé0");
}
#[test]
fn mask_bytes_matches_string_masking_for_ascii() {
// mask_bytes is the byte-wise analogue: letters→b'A', digits→b'0'.
assert_eq!(mask_bytes(b"HL-DT-ST"), b"AA-AA-AA".to_vec());
assert_eq!(mask_bytes(b"1.04"), b"0.00".to_vec());
}
#[test]
fn mask_bytes_preserves_non_alnum_and_high_bytes() {
// Control bytes (0x00), high bytes (0xFF), and punctuation are
// not ASCII alnum and must survive verbatim — INQUIRY payloads
// are space-padded binary and the framing must be diffable.
let input = [0x00u8, b'A', 0x20, b'7', 0xFF, b'-'];
assert_eq!(mask_bytes(&input), vec![0x00, b'A', 0x20, b'0', 0xFF, b'-']);
}
#[test]
fn feature_table_has_no_duplicate_codes() {
// capture_drive_data iterates FEATURES once per code; a duplicate
// code would silently capture the same feature twice (and bloat
// the report). Each MMC-6 feature code must be unique.
let mut seen = std::collections::HashSet::new();
for &(code, _name) in FEATURES {
assert!(seen.insert(code), "duplicate feature code {code:#06x}");
}
}
#[test]
fn feature_table_includes_aacs_010d() {
// AACS (0x010D) is the feature that gates UHD decryption capture;
// it must be in the table or AACS drives capture incompletely.
assert!(
FEATURES.iter().any(|&(c, _)| c == 0x010D),
"AACS feature 0x010D must be captured"
);
}
}
+68 -16
View File
@@ -1,18 +1,33 @@
//! Linux drive discovery and device resolution.
use crate::drive::DeviceResolution;
use crate::error::{Error, Result};
use crate::identity::DriveId;
/// SCSI peripheral device type 5 = MMC / optical (CD/DVD/BD), held in the
/// low 5 bits of INQUIRY byte 0 (the high 3 bits are the peripheral
/// qualifier, masked off here).
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives by enumerating `/dev/sg*` SCSI-generic nodes,
/// opening each, running INQUIRY, and keeping only devices whose
/// peripheral device type is optical (MMC, type 0x05).
///
/// Devices where `scsi::open` or `DriveId::from_drive` fail are silently
/// skipped — that is intentional for enumeration (a busy or wedged node
/// shouldn't abort discovery of the others).
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
for i in 0..16 {
let path = format!("/dev/sg{i}");
for name in enumerate_sg_names() {
let path = format!("/dev/{name}");
if !std::path::Path::new(&path).exists() {
continue;
}
if let Ok(mut transport) = crate::scsi::open(std::path::Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty() && (id.raw_inquiry[0] & 0x1F) == 0x05 {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((path, id));
}
}
@@ -21,41 +36,78 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
drives
}
/// Enumerate `sg*` device names. Linux assigns `/dev/sgN` sequentially
/// across *all* SCSI-generic devices (disks, tape, HBAs, optical), so a
/// fixed `sg0..15` range can miss an optical drive on a host with many
/// targets. Prefer the exact present-device list from
/// `/sys/class/scsi_generic/`; fall back to a bounded `sg0..15` probe
/// only when sysfs is unreadable (minimal containers).
fn enumerate_sg_names() -> Vec<String> {
let mut names = Vec::new();
if let Ok(entries) = std::fs::read_dir("/sys/class/scsi_generic") {
for entry in entries.flatten() {
let name = entry.file_name().to_string_lossy().to_string();
if name.starts_with("sg") {
names.push(name);
}
}
} else {
for i in 0..16 {
let name = format!("sg{i}");
if std::path::Path::new(&format!("/dev/{name}")).exists() {
names.push(name);
}
}
}
names.sort();
names
}
/// Resolve a device path to its raw `/dev/sg*` SCSI-generic node.
///
/// - `/dev/sg*` paths pass through unchanged ([`DeviceResolution::Direct`]).
/// - `/dev/sr*` block paths are matched (by vendor/product/serial) to the
/// corresponding `/dev/sg*` node ([`DeviceResolution::SrToSg`]); if no
/// match is found the original path is returned with
/// [`DeviceResolution::SrNoSgMatch`].
/// - Any other existing path passes through as [`DeviceResolution::Direct`].
#[allow(dead_code)]
pub fn resolve_device(path: &str) -> Result<(String, Option<String>)> {
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
if path.contains("/sg") {
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
return Ok((path.to_string(), None));
return Ok((path.to_string(), DeviceResolution::Direct));
}
if path.contains("/sr") {
let mut sr_transport = crate::scsi::open(std::path::Path::new(path))?;
let sr_id = DriveId::from_drive(sr_transport.as_mut())?;
drop(sr_transport);
for (sg_path, sg_id) in find_drives() {
if sg_id.vendor_id == sr_id.vendor_id
// Require a non-empty serial before treating vendor/product/
// serial as a unique match. serial_number falls back to an
// empty string when GET CONFIGURATION 0108h is unavailable
// (common on OEM drives); two same-model drives would then
// both compare equal and the first in enumeration order would
// win silently, resolving sr1 to sr0's sg node. An empty
// serial can't disambiguate, so fall through to the no-match
// path instead.
if !sr_id.serial_number.is_empty()
&& sg_id.vendor_id == sr_id.vendor_id
&& sg_id.product_id == sr_id.product_id
&& sg_id.serial_number == sr_id.serial_number
{
let warning =
format!("{path} is a block device (sr) — using {sg_path} (sg) for raw access");
return Ok((sg_path, Some(warning)));
return Ok((sg_path, DeviceResolution::SrToSg));
}
}
return Ok((
path.to_string(),
Some(format!(
"{path} is a block device (sr) — no matching sg device found"
)),
));
return Ok((path.to_string(), DeviceResolution::SrNoSgMatch));
}
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
Ok((path.to_string(), None))
Ok((path.to_string(), DeviceResolution::Direct))
}
+21 -11
View File
@@ -4,9 +4,20 @@
//! to discover optical drives without exclusive access or unmounts. Only
//! the returned paths are then opened for INQUIRY to build full `DriveId`.
use crate::drive::DeviceResolution;
use crate::error::{Error, Result};
use crate::identity::DriveId;
/// SCSI peripheral device type 5 = MMC / optical, in the low 5 bits of
/// INQUIRY byte 0.
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives via the IOKit registry (`scsi::list_drives`),
/// then open each candidate for INQUIRY to build a full `DriveId`.
///
/// Any drive where `scsi::open` or `DriveId::from_drive` fails, or whose
/// peripheral device type is not optical (MMC, type 0x05), is silently
/// skipped — the same MMC filter the Linux and Windows backends apply.
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
let discovered = crate::scsi::list_drives();
@@ -15,9 +26,13 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
match crate::scsi::open(path) {
Ok(mut transport) => {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((info.path.clone(), id));
}
}
}
Err(_) => {
continue;
}
@@ -26,20 +41,15 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
drives
}
pub fn resolve_device(path: &str) -> Result<(String, Option<String>)> {
// Accept /dev/diskN or /dev/rdiskN paths as-is
if path.contains("/disk") || path.contains("/rdisk") {
/// Resolve a device path on macOS. There is no `sr`→`sg` style
/// substitution here (that is a Linux concern), so any existing path is
/// returned unchanged as [`DeviceResolution::Direct`]; the
/// [`DeviceResolution`] return exists for cross-platform signature parity.
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
return Ok((path.to_string(), None));
}
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
Ok((path.to_string(), None))
Ok((path.to_string(), DeviceResolution::Direct))
}
+1253 -141
View File
File diff suppressed because it is too large Load Diff
+23 -8
View File
@@ -1,9 +1,18 @@
//! Windows drive discovery and device resolution.
use crate::drive::DeviceResolution;
use crate::error::Result;
use crate::identity::DriveId;
use std::path::Path;
/// SCSI peripheral device type 5 = MMC / optical, in the low 5 bits of
/// INQUIRY byte 0.
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives. Probes `\\.\CdRom0..15` first; only if none
/// are found does it fall back to scanning drive letters `D..Z`. Each
/// candidate is opened, INQUIRY'd, and kept only if its peripheral device
/// type is optical (MMC, type 0x05). Returns normalized `\\.\` paths.
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
@@ -12,7 +21,9 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
let path = format!("\\\\.\\CdRom{}", i);
if let Ok(mut transport) = crate::scsi::open(Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty() && (id.raw_inquiry[0] & 0x1F) == 0x05 {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((path, id));
}
}
@@ -25,8 +36,12 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
let path = format!("{}:", letter as char);
if let Ok(mut transport) = crate::scsi::open(Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty() && (id.raw_inquiry[0] & 0x1F) == 0x05 {
drives.push((path, id));
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
// Normalize so returned paths are consistently in
// \\.\ form regardless of which loop matched.
drives.push((normalize_path(&path), id));
}
}
}
@@ -36,8 +51,11 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
drives
}
pub fn resolve_device(path: &str) -> Result<(String, Option<String>)> {
Ok((normalize_path(path), None))
/// Resolve a device path to its normalized Windows `\\.\` form. Windows
/// has no `sr`→`sg` symlink-target indirection, so resolution is purely a
/// path normalization and always reports [`DeviceResolution::Direct`].
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
Ok((normalize_path(path), DeviceResolution::Direct))
}
/// Normalize a device path to Windows \\.\X: format.
@@ -55,9 +73,6 @@ fn normalize_path(path: &str) -> String {
if trimmed.len() == 2 && trimmed.as_bytes()[1] == b':' {
return format!("\\\\.\\{}", trimmed);
}
if path.to_lowercase().starts_with("cdrom") {
return format!("\\\\.\\{}", path);
}
format!("\\\\.\\{}", path)
}
-287
View File
@@ -1,287 +0,0 @@
//! Top-level DRM scheme dispatch.
//!
//! Four content-protection schemes ride through a single
//! detect-then-load pipeline:
//!
//! | Scheme | Discriminator |
//! |---------------------|------------------------------------------------|
//! | [`DrmScheme::Css`] | DVD probe sector flagged scrambled |
//! | [`DrmScheme::Aacs10`] | Content cert type byte `0x00` |
//! | [`DrmScheme::Aacs20`] | Content cert type byte `!= 0x00`, no Variant |
//! | [`DrmScheme::Aacs21`] | Content cert + MKB records `0x82` / `0x83` |
//!
//! Detection happens from a [`DrmProbe`] (raw inputs the caller has
//! already extracted from the disc); resolution runs through a
//! [`DrmContext`] (the full set of inputs the loaders need).
//!
//! The AACS 2.1 arm is wired but disabled. The dispatcher leaves
//! [`crate::aacs::resolve_keys_v21`] reachable as a library entry point
//! for fixture-driven validation, but production consumers go through
//! [`DrmScheme::load`], which short-circuits V21 to `None` until the
//! Variant chain has a real Variant-scheme disc to validate against.
use crate::aacs;
use crate::css;
/// Which content-protection scheme governs a disc.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DrmScheme {
/// DVD Content Scramble System.
Css,
/// AACS 1.0 — original BD-ROM.
Aacs10,
/// AACS 2.0 — UHD-BD, classical Media Key chain.
Aacs20,
/// AACS 2.1 — UHD-BD with Media Key Variant chain.
Aacs21,
}
/// Inputs to [`DrmScheme::detect`]. All borrows — caller retains
/// ownership.
pub struct DrmProbe<'a> {
/// 2048-byte sample sector from inside a DVD title's extents. Used
/// only for CSS scramble-flag detection. `None` for non-DVD discs.
pub dvd_sample_sector: Option<&'a [u8]>,
/// Content Certificate file bytes (typically `/AACS/Content000.cer`).
/// `None` when the disc has no AACS directory.
pub content_cert: Option<&'a [u8]>,
/// MKB file bytes (typically `/AACS/MKB_RW.inf`). Required to
/// distinguish AACS 2.0 from AACS 2.1.
pub mkb: Option<&'a [u8]>,
}
/// Inputs to [`DrmScheme::load`]. Carries everything needed by either
/// the AACS or CSS loader.
pub struct DrmContext<'a> {
/// AACS resolver inputs — required when the scheme is any AACS
/// variant.
pub aacs: Option<aacs::ResolveContext<'a>>,
/// CSS resolver inputs — required when the scheme is [`DrmScheme::Css`].
pub css: Option<css::CssContext<'a>>,
}
/// Resolved key material, tagged by scheme.
#[derive(Debug)]
pub enum ResolvedScheme {
Css(css::CssState),
Aacs(aacs::ResolvedKeys),
}
impl DrmScheme {
/// Detect which DRM scheme protects the disc described by `probe`.
///
/// Returns `None` for unencrypted media. The order is intentional:
/// CSS is checked first (DVD-format probe), then AACS (Blu-ray
/// format).
pub fn detect(probe: &DrmProbe<'_>) -> Option<DrmScheme> {
// CSS — DVD probe sector carries the scramble flag.
if let Some(sector) = probe.dvd_sample_sector {
if css::is_scrambled(sector) {
return Some(DrmScheme::Css);
}
}
// AACS — content cert type byte distinguishes V10 from V20+.
// V21 promotion requires MKB Variant records.
let cc = probe.content_cert.and_then(aacs::parse_content_cert)?;
match cc.version {
aacs::AacsVersion::V10 => Some(DrmScheme::Aacs10),
aacs::AacsVersion::V20 | aacs::AacsVersion::V21 => {
if let Some(mkb) = probe.mkb {
let recs = aacs::variants::walk_mkb(mkb);
if aacs::variants::is_variant_mkb(&recs) {
return Some(DrmScheme::Aacs21);
}
}
Some(DrmScheme::Aacs20)
}
}
}
/// Run key resolution for this scheme against `ctx`.
///
/// Returns `None` when the scheme's resolver could not produce keys
/// (missing context, KEYDB miss, failed crypto walk, etc.) or when
/// the scheme itself is gated off (see the inline comment on the
/// `Aacs21` arm).
pub fn load(self, ctx: &mut DrmContext<'_>) -> Option<ResolvedScheme> {
match self {
DrmScheme::Css => ctx
.css
.as_mut()
.and_then(css::resolve)
.map(ResolvedScheme::Css),
DrmScheme::Aacs10 => ctx
.aacs
.as_ref()
.and_then(aacs::resolve_keys_v1)
.map(ResolvedScheme::Aacs),
DrmScheme::Aacs20 => ctx
.aacs
.as_ref()
.and_then(aacs::resolve_keys_v2)
.map(ResolvedScheme::Aacs),
// AACS 2.1 derivation is wired but disabled. KCD validation
// against a Variant-scheme disc is pending. To enable,
// uncomment the line below.
// DrmScheme::Aacs21 => ctx
// .aacs
// .as_ref()
// .and_then(aacs::resolve_keys_v21)
// .map(ResolvedScheme::Aacs),
DrmScheme::Aacs21 => None,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
// Build a minimal cert: type byte + bus-encryption byte + 6 zero
// cc_id bytes.
fn cert(type_byte: u8) -> Vec<u8> {
let mut v = vec![0u8; 8];
v[0] = type_byte;
v
}
// Synthetic AACS 2.x MKB with no Variant records.
fn mkb_classical() -> Vec<u8> {
vec![
0x10, 0x00, 0x00, 0x0C, 0x48, 0x14, 0x10, 0x03, 0x00, 0x00, 0x00, 0x4D,
]
}
// Synthetic AACS 2.x MKB with a 0x82 + 0x83 record pair.
fn mkb_with_variant() -> Vec<u8> {
let mut m = mkb_classical();
m.extend_from_slice(&[0x82, 0x00, 0x00, 0x14]);
m.extend_from_slice(&[0xEE; 16]);
m.extend_from_slice(&[0x83, 0x00, 0x00, 0x14]);
m.extend_from_slice(&[0x55; 16]);
m
}
// Synthetic scrambled DVD sector — byte 0x14 carries the CSS
// scramble flag in bits 4-5.
fn scrambled_dvd_sector() -> Vec<u8> {
let mut s = vec![0u8; 2048];
s[0x14] = 0x30;
s
}
#[test]
fn detect_returns_none_for_unencrypted() {
let probe = DrmProbe {
dvd_sample_sector: None,
content_cert: None,
mkb: None,
};
assert_eq!(DrmScheme::detect(&probe), None);
}
#[test]
fn detect_returns_css_for_scrambled_dvd() {
let sector = scrambled_dvd_sector();
let probe = DrmProbe {
dvd_sample_sector: Some(&sector),
content_cert: None,
mkb: None,
};
assert_eq!(DrmScheme::detect(&probe), Some(DrmScheme::Css));
}
#[test]
fn detect_returns_aacs10_for_type0_cert() {
let c = cert(0x00);
let probe = DrmProbe {
dvd_sample_sector: None,
content_cert: Some(&c),
mkb: None,
};
assert_eq!(DrmScheme::detect(&probe), Some(DrmScheme::Aacs10));
}
#[test]
fn detect_returns_aacs20_for_type1_cert_no_variant() {
let c = cert(0x01);
let mkb = mkb_classical();
let probe = DrmProbe {
dvd_sample_sector: None,
content_cert: Some(&c),
mkb: Some(&mkb),
};
assert_eq!(DrmScheme::detect(&probe), Some(DrmScheme::Aacs20));
}
#[test]
fn detect_returns_aacs21_for_type1_cert_with_variant() {
let c = cert(0x01);
let mkb = mkb_with_variant();
let probe = DrmProbe {
dvd_sample_sector: None,
content_cert: Some(&c),
mkb: Some(&mkb),
};
assert_eq!(DrmScheme::detect(&probe), Some(DrmScheme::Aacs21));
}
#[test]
fn detect_returns_aacs20_when_mkb_absent() {
// Type-1 cert but no MKB to upgrade with -> Aacs20.
let c = cert(0x01);
let probe = DrmProbe {
dvd_sample_sector: None,
content_cert: Some(&c),
mkb: None,
};
assert_eq!(DrmScheme::detect(&probe), Some(DrmScheme::Aacs20));
}
#[test]
fn load_aacs21_returns_none() {
// The Aacs21 dispatch arm is commented out; load() must
// return None until KCD validation lands.
let uk_ro = vec![0u8; 256];
let vid = [0u8; 16];
let keydb = aacs::KeyDb::empty();
let ctx_aacs = aacs::ResolveContext {
unit_key_ro: &uk_ro,
content_cert: None,
volume_id: &vid,
keydb: &keydb,
mkb: None,
};
let mut ctx = DrmContext {
aacs: Some(ctx_aacs),
css: None,
};
assert!(DrmScheme::Aacs21.load(&mut ctx).is_none());
}
/// Exercises the V21 helper directly. Gated `#[ignore]` because
/// the chain reaches `MediaKeyVariantError::VariantsTableUnavailable`
/// without a real Variant-scheme disc to fix the per-uv table
/// layout against — running it here would assert only the
/// not-yet-wired error code. Kept as a wiring smoke-test for
/// future enablement.
#[test]
#[ignore]
fn resolve_keys_v21_helper_exists() {
let uk_ro = vec![0u8; 256];
let vid = [0xAAu8; 16];
let keydb = aacs::KeyDb::empty();
let mkb = mkb_with_variant();
let ctx = aacs::ResolveContext {
unit_key_ro: &uk_ro,
content_cert: None,
volume_id: &vid,
keydb: &keydb,
mkb: Some(&mkb),
};
// Just confirm the symbol is callable; we don't assert on the
// result.
let _ = aacs::resolve_keys_v21(&ctx);
}
}
+49
View File
@@ -0,0 +1,49 @@
//! DVD-Video navigation — read-only resolver for the **main-feature start
//! point** (issue #40). Mirrors what a DVD player's nav VM resolves: First-Play
//! → menu "Play" → title dispatch → the first cell of the feature, so the rip
//! starts at the movie rather than at raw cell 0 (e.g. skipping a leading
//! logo/warning segment when the disc's own navigation does).
//!
//! Byte layout follows the DVD-Video specification (VMGI/VTSI headers,
//! PGC/cell tables, PCI/HLI button packets); the VM command decoder is
//! verified against libdvdnav's decoder.
//!
//! Current contents: [`vmcmd`] — the VM command decoder (proven against the
//! SOTL/Greenland test discs). The IFO/PCI parsing and the navigation executor
//! that resolves the start cell build on top of this.
pub mod vmcmd;
use crate::sector::SectorSource;
/// Resolve the feature title's **true start cell** (0-based index into the
/// title PGC's cell list) by following the disc's own navigation — First-Play →
/// menu "Play" → title dispatch — the way a player reaches the movie. This is
/// what lets the rip begin at the feature instead of at raw cell 0 when the
/// disc's nav enters the title past a leading logo/warning segment (e.g. a
/// disc whose "Play" resolves to a later cell than cell 0).
///
/// Returns `None` when navigation cannot be resolved, so the caller falls back
/// to the structural leading-cell filter (today's behaviour, ≈ cell 0 / 0:00).
///
/// TODO(#40): the IFO/PCI parsing + nav executor (built on [`vmcmd`]) land
/// incrementally. Until the executor is complete this returns `None`, so wiring
/// it in is behaviour-neutral; improvements to the resolver take effect here
/// without touching the call site.
pub fn resolve_feature_start(
reader: &mut dyn SectorSource,
udf: &crate::udf::UdfFs,
vtsn: u16,
vts_ttn: u16,
) -> Option<usize> {
// `reader`/`udf` are the seam inputs the nav executor will consume to read
// VIDEO_TS.IFO + the VTS IFOs/menu VOBs. Reserved until that lands.
let _ = (reader, udf);
tracing::trace!(
target: "freemkv::dvdnav",
vtsn,
vts_ttn,
"nav start-cell resolver: unresolved — caller falls back to leading-cell filter"
);
None
}
+408
View File
@@ -0,0 +1,408 @@
//! DVD-Video VM command decoder.
//!
//! An 8-byte navigation command as found in PGC command tables (pre/post/cell)
//! and PCI button info. Decoded per the DVD-Video VM instruction set and
//! verified against libdvdnav's command decoder.
//!
//! Bit model: the 8 bytes are a big-endian 64-bit word. `byte0` bits 7-5 are the
//! command **type**; for type 1, `byte0` bit 4 selects Link (0) vs Jump (1), and
//! `byte1` bits 3-0 are the sub-command. Compare predicates live in `byte1`
//! bits 6-4 with the operands in bytes 2-5.
//!
//! This module is pure decode + a register model — no I/O, no English (numeric
//! semantics only), matching libfreemkv conventions. The navigation *executor*
//! and IFO/PCI parsing build on top of this.
/// A decoded navigation instruction. Only the variants freemkv's start-point
/// resolver needs are modelled explicitly; everything else is [`Instr::Other`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Instr {
Nop,
/// Stop executing the current command list (resume cell playback).
Break,
/// Goto command line within the same list (1-based).
Goto {
line: u8,
},
/// Leave the current domain.
Exit,
/// Jump to a VMG title (1-based TT_SRPT index).
JumpTt {
ttn: u8,
},
/// Jump to a title within the current VTS (1-based VTS title index).
JumpVtsTt {
ttn: u8,
},
/// Jump to a part-of-title (chapter) within a VTS title.
JumpVtsPtt {
ttn: u8,
pttn: u16,
},
/// Jump to the First-Play PGC.
JumpSsFp,
/// Jump to a Video-Manager menu (`menu` = menu id).
JumpSsVmgm {
menu: u8,
},
/// Jump to a Video-Title-Set menu.
JumpSsVtsm {
vts: u8,
ttn: u8,
menu: u8,
},
/// Jump to a specific VMGM menu PGC.
JumpSsVmgmPgc {
pgcn: u16,
},
/// Call a sub-domain (raw retained; resume handled by the executor).
CallSs {
sub: u8,
},
/// Link to a PGC number within the current domain.
LinkPgcn {
pgcn: u16,
},
/// Link to a part-of-title within the current PGC's title.
LinkPttn {
pttn: u16,
},
/// Link to a program number within the current PGC (1-based).
LinkPgn {
pgn: u8,
},
/// Link to a cell number within the current PGC (1-based).
LinkCn {
cn: u8,
},
/// A link "subset" op (LinkTopCell/NextPG/RSM/…); `sub` is the raw code.
LinkSub {
sub: u8,
},
/// Set a GPRM. `op` is the set-op code (1=mov, 3=add, …); value is immediate
/// (`imm`) when `immediate`, else the contents of register `src`.
SetGprm {
reg: u8,
op: u8,
immediate: bool,
imm: u16,
src: u8,
},
/// Set a system parameter / unmodelled set — executor may ignore.
SetSystem,
/// Anything not individually modelled (kept as raw bytes).
Other([u8; 8]),
}
/// A compare predicate carried by a command (`byte1` bits 6-4). `None` = always.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Compare {
/// Compare op: 1=&,2===,3=!=,4=>=,5=>,6=<=,7=<.
pub op: u8,
/// Left register index (GPRM 0-15, SPRM 128+).
pub lhs_reg: u8,
/// Right side: immediate when `immediate`, else register `rhs_reg`.
pub immediate: bool,
pub imm: u16,
pub rhs_reg: u8,
}
/// A fully decoded command: its predicate (if any) and the instruction.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Command {
pub compare: Option<Compare>,
pub instr: Instr,
}
// Command types — `byte0` bits 7-5.
const TYPE_SPECIAL: u8 = 0;
const TYPE_LINK_JUMP: u8 = 1;
const TYPE_SET_SYSTEM: u8 = 2;
const TYPE_SET_GPRM: u8 = 3;
// Special (type 0) sub-commands — `byte1` bits 3-0.
const SP_GOTO: u8 = 1;
const SP_BREAK: u8 = 2;
// Jump/Call (type 1, direct=1) sub-commands.
const JP_EXIT: u8 = 1;
const JP_JUMP_TT: u8 = 2;
const JP_JUMP_VTS_TT: u8 = 3;
const JP_JUMP_VTS_PTT: u8 = 5;
const JP_JUMP_SS: u8 = 6;
const JP_CALL_SS: u8 = 8;
// Link (type 1, direct=0) sub-commands. NOTE: sub-op 0 is NOP/no-link and 1 is
// the LinkSub form (libdvdnav `decoder.c` `eval_link_instruction`).
const LK_SUB: u8 = 1;
const LK_PGCN: u8 = 4;
const LK_PTTN: u8 = 5;
const LK_PGN: u8 = 6;
const LK_CN: u8 = 7;
// JumpSS sub-domain selector — `byte5` bits 7-6.
const SS_FP: u8 = 0;
const SS_VMGM_MENU: u8 = 1;
const SS_VTSM: u8 = 2;
// Operand field widths (spec-defined bit counts).
const MASK_TTN: u8 = 0x7F; // 7-bit title number
const MASK_PGN: u8 = 0x7F; // 7-bit program number
const MASK_LINKOP: u8 = 0x1F; // 5-bit link sub-op
const MASK_REG: u8 = 0x0F; // 4-bit GPRM index
const MASK_MENU: u8 = 0x0F; // 4-bit menu id
const MASK_PTTN: u16 = 0x03FF; // 10-bit part-of-title
const MASK_PGCN: u16 = 0x7FFF; // 15-bit PGC number
#[inline]
fn be16(b: &[u8; 8], o: usize) -> u16 {
((b[o] as u16) << 8) | b[o + 1] as u16
}
// Compare-operand layouts ("if_version"s) per libdvdnav `decoder.c`. The op
// nibble is always `byte1` bits 6-4; the immediate flag is `byte1` bit 7. The
// operand *offsets* differ by command family.
//
// v1 (special + link): lhs reg = b[3]; rhs imm = bytes4-5 / rhs reg = b[4].
// v2 (jump + system-set): lhs reg = b[6]; rhs reg = b[7] (registers only).
// v3 (set-GPRM): lhs reg = b[2]; rhs imm = bytes6-7 / rhs reg = b[6].
fn if_v1(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[3],
immediate: b[1] >> 7 != 0,
imm: be16(b, 4),
rhs_reg: b[4],
})
}
fn if_v2(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[6],
immediate: false,
imm: 0,
rhs_reg: b[7],
})
}
fn if_v3(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[2],
immediate: b[1] >> 7 != 0,
imm: be16(b, 6),
rhs_reg: b[6],
})
}
/// Decode an 8-byte VM command.
pub fn decode(b: &[u8; 8]) -> Command {
let typ = b[0] >> 5;
let direct = (b[0] >> 4) & 1;
let setop = b[0] & 0x0F;
let cmd = b[1] & 0x0F;
// Compare predicate, with the operand layout for this command family
// (libdvdnav `decoder.c` `vm_eval_command` type dispatch).
let compare = match (typ, direct) {
(TYPE_SPECIAL, _) => if_v1(b),
(TYPE_LINK_JUMP, 1) => if_v2(b), // jump
(TYPE_LINK_JUMP, 0) => if_v1(b), // link
(TYPE_SET_SYSTEM, _) => if_v2(b),
(TYPE_SET_GPRM, _) => if_v3(b),
_ => None, // 4/5/6 compound — not needed by the resolver
};
// JumpSS sub-domain selector lives in byte5 bits 7-6.
let ss_sel = b[5] >> 6;
let instr = match typ {
TYPE_LINK_JUMP if direct == 1 => match cmd {
JP_EXIT => Instr::Exit,
JP_JUMP_TT => Instr::JumpTt {
ttn: b[5] & MASK_TTN,
},
JP_JUMP_VTS_TT => Instr::JumpVtsTt {
ttn: b[5] & MASK_TTN,
},
JP_JUMP_VTS_PTT => Instr::JumpVtsPtt {
ttn: b[5] & MASK_TTN,
pttn: be16(b, 2) & MASK_PTTN,
},
JP_JUMP_SS => match ss_sel {
SS_FP => Instr::JumpSsFp,
SS_VMGM_MENU => Instr::JumpSsVmgm {
menu: b[5] & MASK_MENU,
},
SS_VTSM => Instr::JumpSsVtsm {
vts: b[4],
ttn: b[3],
menu: b[5] & MASK_MENU,
},
_ => Instr::JumpSsVmgmPgc {
pgcn: be16(b, 2) & MASK_PGCN,
},
},
JP_CALL_SS => Instr::CallSs { sub: ss_sel },
_ => Instr::Nop,
},
TYPE_LINK_JUMP => match cmd {
// direct == 0 (link). sub-op 0 = NOP/no-link.
LK_SUB => Instr::LinkSub {
sub: b[7] & MASK_LINKOP,
},
LK_PGCN => Instr::LinkPgcn {
pgcn: be16(b, 6) & MASK_PGCN,
},
LK_PTTN => Instr::LinkPttn {
pttn: be16(b, 6) & MASK_PTTN,
},
LK_PGN => Instr::LinkPgn {
pgn: b[7] & MASK_PGN,
},
LK_CN => Instr::LinkCn { cn: b[7] },
_ => Instr::Nop,
},
TYPE_SPECIAL => match cmd {
SP_GOTO => Instr::Goto { line: b[7] },
SP_BREAK => Instr::Break,
_ => Instr::Nop,
},
TYPE_SET_GPRM => Instr::SetGprm {
reg: b[3] & MASK_REG,
op: setop,
immediate: direct != 0,
imm: be16(b, 4),
src: b[5],
},
TYPE_SET_SYSTEM => Instr::SetSystem,
_ => Instr::Other(*b),
};
Command { compare, instr }
}
#[cfg(test)]
mod tests {
use super::*;
fn h(s: &str) -> [u8; 8] {
let v: Vec<u8> = (0..8)
.map(|i| u8::from_str_radix(&s[i * 2..i * 2 + 2], 16).unwrap())
.collect();
v.try_into().unwrap()
}
// KATs taken from the real SOTL / Greenland discs (decoded in the PoC).
#[test]
fn greenland_first_play_is_jumptt_1() {
let c = decode(&h("3002000000010000"));
assert_eq!(c.instr, Instr::JumpTt { ttn: 1 });
assert!(c.compare.is_none());
}
#[test]
fn sotl_first_play_is_jumpss_vtsm_root() {
// 30 06 ... byte5=0x83 -> sub 2 (VTSM), vts=byte4=1, menu=byte5&0xF=3 (root)
let c = decode(&h("3006000101830000"));
assert_eq!(
c.instr,
Instr::JumpSsVtsm {
vts: 1,
ttn: 1,
menu: 3
}
);
}
#[test]
fn sotl_title_dispatch_is_conditional_linkpgn_2() {
// 20 a6 ... CmpLink: if GPRM0 == 2 -> LinkPGN 2 (cell 2 = the 5:02 start)
let c = decode(&h("20a6000000020002"));
assert_eq!(c.instr, Instr::LinkPgn { pgn: 2 });
let cmp = c.compare.expect("conditional");
assert_eq!(cmp.op, 2); // ==
assert_eq!(cmp.lhs_reg, 0); // GPRM0
assert!(cmp.immediate);
assert_eq!(cmp.imm, 2);
}
#[test]
fn sotl_root_button_is_linkpgcn_37() {
assert_eq!(
decode(&h("2004000000000025")).instr,
Instr::LinkPgcn { pgcn: 37 }
);
}
#[test]
fn greenland_scene_button_is_linkpgn() {
assert_eq!(
decode(&h("2006000000001401")).instr,
Instr::LinkPgn { pgn: 1 }
);
}
#[test]
fn jumpvts_ptt_decodes_ttn_and_pttn() {
// synthetic: 30 05 | ptt(bytes2-3)=0x0002 | ttn(byte5)=1
let c = decode(&h("3005000200010000"));
assert_eq!(c.instr, Instr::JumpVtsPtt { ttn: 1, pttn: 2 });
}
#[test]
fn setgprm_immediate_mov() {
// SOTL First-Play pre[0]: 71 00 | reg=byte3=6 | imm(bytes4-5)=0x03e8 -> g6 = 1000
match decode(&h("7100000603e80000")).instr {
Instr::SetGprm {
reg,
op,
immediate,
imm,
..
} => {
assert_eq!(reg, 6);
assert_eq!(op, 1); // mov
assert!(immediate);
assert_eq!(imm, 1000);
}
other => panic!("expected SetGprm, got {other:?}"),
}
}
// Regression for the libdvdnav cross-check: link sub-op 0 = NOP, 1 = LinkSub.
#[test]
fn link_subop_zero_is_nop_one_is_linksub() {
assert_eq!(decode(&h("2000000000000000")).instr, Instr::Nop);
assert_eq!(
decode(&h("2001000000000010")).instr,
Instr::LinkSub { sub: 0x10 }
);
}
// if_version_1 register compare: rhs register is byte4 (not byte5).
#[test]
fn link_register_compare_rhs_is_byte4() {
// 20 26: link, cmp=EQ(2), dircmp=0(register) ; cmd=6 LinkPGN
let c = decode(&h("2026000304000002"));
assert_eq!(c.instr, Instr::LinkPgn { pgn: 2 });
let cmp = c.compare.expect("conditional");
assert!(!cmp.immediate);
assert_eq!(cmp.lhs_reg, 3);
assert_eq!(cmp.rhs_reg, 4);
}
// if_version_2 jump compare: both operands are registers in byte6 / byte7.
#[test]
fn jump_compare_uses_bytes6_and_7() {
// 30 22: jump, cmp=EQ(2) ; cmd=2 JumpTT ttn=byte5=5
let c = decode(&h("3022000000050607"));
assert_eq!(c.instr, Instr::JumpTt { ttn: 5 });
let cmp = c.compare.expect("conditional");
assert!(!cmp.immediate);
assert_eq!(cmp.lhs_reg, 6);
assert_eq!(cmp.rhs_reg, 7);
}
}
+936 -35
View File
File diff suppressed because it is too large Load Diff
+33 -1
View File
@@ -8,11 +8,17 @@
//! disc.rip(&mut session, 0, output, |event| {
//! match event.kind {
//! EventKind::BytesRead { bytes, total } => update_progress(bytes, total),
//! EventKind::ReadError { sector, .. } => log_error(sector),
//! EventKind::SectorSkipped { sector } => log_skip(sector),
//! EventKind::BatchSizeChanged { new_size, .. } => note_recovery(new_size),
//! _ => {}
//! }
//! });
//! ```
//!
//! Note: the library currently emits only `BytesRead`, `SectorSkipped`,
//! and `BatchSizeChanged`. The other [`EventKind`] variants are part of
//! the stable event vocabulary for consumers (and future emit sites) but
//! are not produced by the library today.
use crate::error::Error;
@@ -116,3 +122,29 @@ pub enum BatchSizeReason {
/// A no-op event handler. Ignores all events.
pub fn ignore(_event: Event) {}
#[cfg(test)]
mod tests {
use super::*;
/// BatchSizeReason::Shrunk != BatchSizeReason::Probed.
/// These two variants carry distinct meanings (error vs. recovery); they
/// must not compare as equal.
/// Mutation: deriving PartialEq without proper variant discrimination
/// could make two distinct variants equal.
#[test]
fn batch_size_reason_variants_are_not_equal() {
assert_ne!(BatchSizeReason::Shrunk, BatchSizeReason::Probed);
}
/// BatchSizeReason is Clone + Copy: cloning does not move the original.
/// This is required because EventKind::BatchSizeChanged embeds it by value.
/// Mutation: removing Copy would require the caller to clone explicitly;
/// code that passes reason by value would fail to compile.
#[test]
fn batch_size_reason_is_copy() {
let r = BatchSizeReason::Shrunk;
let _r2 = r; // copy, not move
let _r3 = r; // r still usable after copy
}
}
+20 -9
View File
@@ -34,11 +34,9 @@ impl Halt {
Self(Arc::new(AtomicBool::new(false)))
}
/// Wrap an existing `Arc<AtomicBool>` as a `Halt`. Useful as a
/// bridge during the 0.18 deprecation window: callers that already
/// hold an `Arc<AtomicBool>` (e.g. `Drive::halt_flag()`, the
/// deprecated `DiscStream::set_halt`) can adopt the new token API
/// without changing the underlying flag.
/// Wrap an existing `Arc<AtomicBool>` as a `Halt`. A bridge for
/// callers that already hold an `Arc<AtomicBool>` cancellation flag
/// and want to adopt the token API without allocating a new flag.
///
/// Cancelling either side flips the same bit — the wrapping `Halt`
/// and the original `Arc` are two views over one shared flag.
@@ -46,10 +44,9 @@ impl Halt {
Self(flag)
}
/// Borrow the underlying `Arc<AtomicBool>`. Used at boundaries with
/// pre-`Halt` APIs that still take an `Arc<AtomicBool>` directly
/// (`CopyOptions::halt`, the deprecated `DiscStream::set_halt`).
/// Round 3 deletes those boundaries and this accessor with them.
/// Borrow the underlying `Arc<AtomicBool>`. The inverse of
/// [`from_arc`](Self::from_arc): hand the shared flag to an API that
/// still takes a raw `Arc<AtomicBool>` rather than a `Halt`.
pub fn as_arc(&self) -> &Arc<AtomicBool> {
&self.0
}
@@ -177,4 +174,18 @@ mod tests {
arc.store(true, Ordering::Relaxed);
assert!(halt.is_cancelled());
}
// ── New comprehensive tests ────────────────────────────────────────────────
/// POLL_INTERVAL is 250ms — a specific value that the multi-thread halt
/// loops depend on for responsiveness guarantees.
/// Mutation: setting POLL_INTERVAL to 5s makes stop requests take 5s to notice.
#[test]
fn poll_interval_is_250ms() {
assert_eq!(
POLL_INTERVAL,
std::time::Duration::from_millis(250),
"POLL_INTERVAL must be 250ms for the guaranteed ~quarter-second cancel latency"
);
}
}
+106
View File
@@ -0,0 +1,106 @@
//! The single hex → bytes parser for the whole workspace.
//!
//! Key material arrives as hex from three third-party sources — the keydb, an
//! online key service, and the mapfile's `# freemkv-vid:` comment — and each
//! used to parse it slightly differently (one stripped `0x`/`0X`, one stripped
//! nothing, one stripped `0x` only). A key written with a prefix one parser
//! didn't expect was silently dropped → "can't decrypt" with no error. This is
//! the one parser they all call, so the prefix/case/validation rules live in
//! exactly one place.
//!
//! Operates on BYTES, not `&str` char indices: the inputs are untrusted, so a
//! multi-byte UTF-8 scalar must reject as malformed, never panic on a
//! mid-codepoint slice.
/// Parse a hex string into bytes. Accepts an optional `0x`/`0X` prefix
/// (case-insensitive), then requires an even run of ASCII hex digits. Any
/// non-hex byte, or an odd length, yields `None`.
pub fn parse_hex_bytes(s: &str) -> Option<Vec<u8>> {
let body = strip_prefix(s.trim());
let bytes = body.as_bytes();
// Empty → empty Vec (a legitimately-empty variable-length field); odd length
// is malformed. (`parse_hex_fixed` enforces a concrete length separately.)
if bytes.len() % 2 != 0 {
return None;
}
let mut out = Vec::with_capacity(bytes.len() / 2);
for pair in bytes.chunks_exact(2) {
out.push(byte(pair[0], pair[1])?);
}
Some(out)
}
/// Parse a hex string into a fixed `[u8; N]`. Accepts an optional `0x`/`0X`
/// prefix; requires EXACTLY `2*N` ASCII hex digits after it. `None` on any
/// non-hex byte or a length mismatch.
pub fn parse_hex_fixed<const N: usize>(s: &str) -> Option<[u8; N]> {
let body = strip_prefix(s.trim());
let bytes = body.as_bytes();
if bytes.len() != 2 * N {
return None;
}
let mut out = [0u8; N];
for (i, slot) in out.iter_mut().enumerate() {
*slot = byte(bytes[2 * i], bytes[2 * i + 1])?;
}
Some(out)
}
/// Strip a single leading `0x` / `0X` if present (case-insensitive).
fn strip_prefix(s: &str) -> &str {
s.strip_prefix("0x")
.or_else(|| s.strip_prefix("0X"))
.unwrap_or(s)
}
/// Combine two ASCII hex-digit bytes into one byte. `as char` is intentional:
/// for a non-ASCII byte it produces a Latin-1 scalar that `to_digit(16)` then
/// rejects — so non-hex (incl. `+`/`-` sign chars) and multi-byte input fail
/// cleanly rather than slipping through `from_str_radix`'s sign handling.
fn byte(hi: u8, lo: u8) -> Option<u8> {
let hi = (hi as char).to_digit(16)?;
let lo = (lo as char).to_digit(16)?;
Some((hi * 16 + lo) as u8)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn fixed_accepts_0x_0x_and_bare_same_result() {
let want = [0x00, 0x11, 0xab, 0xCD, 0xef, 0x42, 0x99, 0x00];
let bare = "0011abcdef429900";
assert_eq!(parse_hex_fixed::<8>(bare), Some(want));
assert_eq!(parse_hex_fixed::<8>(&format!("0x{bare}")), Some(want));
// The case that used to be dropped by one parser but not another.
assert_eq!(parse_hex_fixed::<8>(&format!("0X{bare}")), Some(want));
assert_eq!(parse_hex_fixed::<8>(&format!(" 0X{bare} ")), Some(want));
}
#[test]
fn fixed_rejects_wrong_length_and_non_hex_and_signs() {
assert_eq!(parse_hex_fixed::<16>("00"), None); // too short
assert_eq!(parse_hex_fixed::<2>("00112233"), None); // too long
assert_eq!(parse_hex_fixed::<2>("zz11"), None); // non-hex
assert_eq!(parse_hex_fixed::<2>("+5-A"), None); // sign chars
}
#[test]
fn does_not_panic_on_multibyte_of_exact_byte_length() {
// "中" is 3 bytes; + 29 'a' = 32 bytes → would mis-slice a &str-indexed
// parser. Must reject, not panic.
let s = "".to_string() + &"a".repeat(29);
assert_eq!(s.len(), 32);
assert_eq!(parse_hex_fixed::<16>(&s), None);
}
#[test]
fn bytes_variable_length_and_odd_rejected() {
assert_eq!(parse_hex_bytes("0xAABBCC"), Some(vec![0xAA, 0xBB, 0xCC]));
assert_eq!(parse_hex_bytes("AABBC"), None); // odd
// Empty (or prefix-only) → empty Vec: a legitimately-empty field.
assert_eq!(parse_hex_bytes(""), Some(vec![]));
assert_eq!(parse_hex_bytes("0x"), Some(vec![]));
}
}
+204 -9
View File
@@ -57,27 +57,48 @@ impl DriveId {
let cdb_inq = [0x12, 0x00, 0x00, 0x00, 0x60, 0x00];
transport.execute(&cdb_inq, DataDirection::FromDevice, &mut inquiry, 5000)?;
// GET CONFIGURATION Feature 010Ch — MMC-6 §6.6
// GET CONFIGURATION Feature 010Ch — MMC-6 §6.6.
// Best-effort: 010Ch (Firmware Information) is an optional feature.
// A drive that lacks it may CHECK CONDITION rather than return an
// empty descriptor, so a failure here is treated as feature-absent
// (empty firmware date + empty raw bytes) instead of aborting the
// whole identity probe.
let mut gc = vec![0u8; 256];
let cdb_gc = [0x46, 0x02, 0x01, 0x0C, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00];
let result = transport.execute(&cdb_gc, DataDirection::FromDevice, &mut gc, 5000)?;
let firmware_date = if result.bytes_transferred > 12 {
String::from_utf8_lossy(&gc[12..24.min(result.bytes_transferred)])
// `bytes_transferred` is device-reported and untrusted; clamp every
// slice end to the actual buffer length before indexing.
let (firmware_date, raw_gc_010c) =
match transport.execute(&cdb_gc, DataDirection::FromDevice, &mut gc, 5000) {
Ok(result) => {
let end = result.bytes_transferred.min(gc.len());
let date = if end > 12 {
String::from_utf8_lossy(&gc[12..24.min(end)])
.trim()
.to_string()
} else {
String::new()
};
(date, gc[..end].to_vec())
}
Err(_) => (String::new(), Vec::new()),
};
// GET CONFIGURATION Feature 0108h — Serial Number
// GET CONFIGURATION Feature 0108h — Serial Number.
// Best-effort, like 010Ch above: the serial-number feature is
// optional, so a drive that lacks it (CHECK CONDITION) or reports
// too few bytes deliberately yields an empty serial rather than
// failing the identity probe.
let mut gc_serial = vec![0u8; 256];
let cdb_serial = [0x46, 0x02, 0x01, 0x08, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00];
let serial_number = if let Ok(r) =
transport.execute(&cdb_serial, DataDirection::FromDevice, &mut gc_serial, 5000)
{
if r.bytes_transferred > 12 {
String::from_utf8_lossy(&gc_serial[12..r.bytes_transferred])
// `bytes_transferred` is device-reported and untrusted; clamp
// the slice end to the buffer length to avoid an out-of-range
// panic on an oversized reported count.
let end = r.bytes_transferred.min(gc_serial.len());
String::from_utf8_lossy(&gc_serial[12..end])
.trim()
.to_string()
} else {
@@ -94,8 +115,8 @@ impl DriveId {
vendor_specific: ascii_field(&inquiry, 36, 43),
firmware_date,
serial_number,
raw_inquiry: inquiry.to_vec(),
raw_gc_010c: gc[..result.bytes_transferred].to_vec(),
raw_inquiry: inquiry,
raw_gc_010c,
})
}
@@ -155,6 +176,49 @@ fn ascii_field(data: &[u8], start: usize, end: usize) -> String {
#[cfg(test)]
mod tests {
use super::*;
use crate::scsi::{ScsiResult, ScsiTransport};
/// Transport that returns the requested data length but reports a
/// bytes_transferred larger than the caller's buffer — models a drive
/// that lies about its transfer count. The old slicing code panicked
/// on this; the clamps must keep it from indexing out of range.
struct OversizedCountTransport;
impl ScsiTransport for OversizedCountTransport {
fn execute(
&mut self,
cdb: &[u8],
_dir: DataDirection,
buf: &mut [u8],
_timeout_ms: u32,
) -> Result<ScsiResult> {
// Fill plausible ASCII so the from_utf8_lossy paths run.
for b in buf.iter_mut() {
*b = b'A';
}
// INQUIRY (0x12): honest count. GET CONFIGURATION (0x46): lie.
let bytes_transferred = if cdb.first() == Some(&0x12) {
buf.len()
} else {
buf.len() + 4096
};
Ok(ScsiResult {
status: 0,
bytes_transferred,
sense: [0u8; 32],
})
}
}
#[test]
fn from_drive_clamps_oversized_bytes_transferred() {
// Must not panic despite the transport reporting a transfer count
// far beyond the 256-byte GET CONFIGURATION buffers.
let mut t = OversizedCountTransport;
let id = DriveId::from_drive(&mut t).expect("from_drive must not error");
// raw_gc_010c is clamped to the 256-byte buffer, never the lie.
assert_eq!(id.raw_gc_010c.len(), 256);
}
#[test]
fn test_bu40n_identity() {
@@ -190,4 +254,135 @@ mod tests {
assert_eq!(id.vendor_specific.trim(), "16/04/");
assert_eq!(id.firmware_date, "201604250000");
}
// ── New comprehensive tests ────────────────────────────────────────────────
/// ascii_field with a buffer shorter than `start` returns empty string
/// rather than panicking.
/// Spec: SPC-4 §6.4.2 — bytes[8:16] are vendor ID; a truncated buffer
/// (e.g. a device that reports fewer than 8 bytes) must not panic.
/// Mutation: removing the `data.len() > start` guard makes it panic on short inputs.
#[test]
fn ascii_field_short_buffer_returns_empty() {
// Buffer of length 5: start=8 is beyond the end → empty string.
let buf = vec![0u8; 5];
let result = ascii_field(&buf, 8, 16); // SPC-4 vendor ID range
assert!(result.is_empty(), "short buffer must yield empty string");
}
/// ascii_field with a buffer that covers start but not end is clamped.
/// Spec: `ascii_field` documents "clamps to data.len()".
/// Mutation: using `end` directly without `min(data.len())` panics here.
#[test]
fn ascii_field_partial_buffer_is_clamped_not_panicked() {
// Buffer of length 12: vendor_id range is [8..16], but only [8..12] present.
let mut buf = vec![0u8; 12];
buf[8..12].copy_from_slice(b"SONY");
let result = ascii_field(&buf, 8, 16);
// Must not panic; the returned string holds what we wrote.
assert_eq!(result, "SONY");
}
/// from_inquiry extracts the product_id field from INQUIRY bytes [16:32].
/// Spec: SPC-4 §6.4.2 — PRODUCT IDENTIFICATION at offset 16, length 16.
/// Mutation: shifting the product_id slice to [8:24] makes this fail.
#[test]
fn from_inquiry_extracts_product_id_at_offset_16() {
let mut inquiry = vec![0u8; 96];
// Leave vendor_id (8..16) as zeros, write product_id at 16..32.
inquiry[16..32].copy_from_slice(b"BD-RW BDR-209M");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.product_id, "BD-RW BDR-209M",
"product_id must come from INQUIRY bytes 16..32 (SPC-4 §6.4.2)"
);
}
/// from_inquiry extracts product_revision from INQUIRY bytes [32:36].
/// Spec: SPC-4 §6.4.2 — PRODUCT REVISION LEVEL at offset 32, length 4.
/// Mutation: reading revision from [36:40] produces the wrong value.
#[test]
fn from_inquiry_extracts_revision_at_offset_32() {
let mut inquiry = vec![0u8; 96];
inquiry[32..36].copy_from_slice(b"1.53");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.product_revision, "1.53",
"product_revision must come from INQUIRY bytes 32..36 (SPC-4 §6.4.2)"
);
}
/// from_inquiry extracts vendor_specific from INQUIRY bytes [36:43].
/// Spec: SPC-4 §6.4.2 — VENDOR SPECIFIC at offset 36, length 8.
/// Mutation: reading vendor_specific from [32:39] returns the revision instead.
#[test]
fn from_inquiry_extracts_vendor_specific_at_offset_36() {
let mut inquiry = vec![0u8; 96];
inquiry[36..43].copy_from_slice(b"MM01234");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.vendor_specific, "MM01234",
"vendor_specific must come from INQUIRY bytes 36..43 (SPC-4 §6.4.2)"
);
}
/// from_inquiry stores the raw inquiry bytes in raw_inquiry unchanged.
/// Mutation: copying only a slice of inquiry into raw_inquiry truncates it.
#[test]
fn from_inquiry_stores_raw_inquiry() {
let mut inquiry = vec![0u8; 96];
inquiry[8..16].copy_from_slice(b"TESTDRVR");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.raw_inquiry, inquiry,
"raw_inquiry must preserve the full 96-byte buffer"
);
}
/// GET CONFIGURATION failure (transport error) must not abort the
/// identity probe — firmware_date is empty, raw_gc_010c is empty.
/// Mutation: propagating the GET_CONFIGURATION error with `?` aborts from_drive.
#[test]
fn from_drive_gc_failure_yields_empty_firmware_date() {
struct GcFailTransport;
impl ScsiTransport for GcFailTransport {
fn execute(
&mut self,
cdb: &[u8],
_dir: DataDirection,
buf: &mut [u8],
_timeout_ms: u32,
) -> Result<ScsiResult> {
if cdb.first() == Some(&0x12) {
// INQUIRY succeeds with a plausible response.
buf[8..16].copy_from_slice(b"TESTDRV ");
buf[16..32].copy_from_slice(b"FAKE DRIVE MODEL");
buf[32..36].copy_from_slice(b"0001");
buf[36..43].copy_from_slice(b"X000001");
Ok(ScsiResult {
status: 0,
bytes_transferred: buf.len(),
sense: [0u8; 32],
})
} else {
// GET CONFIGURATION fails.
Err(crate::error::Error::ScsiError {
opcode: cdb[0],
status: crate::scsi::SCSI_STATUS_CHECK_CONDITION,
sense: None,
})
}
}
}
let mut t = GcFailTransport;
let id = DriveId::from_drive(&mut t).expect("from_drive must succeed despite GC failure");
assert!(
id.firmware_date.is_empty(),
"firmware_date must be empty when GC fails"
);
assert!(
id.raw_gc_010c.is_empty(),
"raw_gc_010c must be empty when GC fails"
);
}
}
+988 -52
View File
File diff suppressed because it is too large Load Diff
+61 -4
View File
@@ -67,10 +67,12 @@ pub(crate) enum BoundedError {
/// The deadline elapsed before the syscall returned. Same leak
/// semantics as `Halted`.
Timeout,
/// The worker thread panicked, or its sender disconnected before
/// sending a result. Treat as a benign no-op (callers usually
/// log and continue) rather than a hard error — by definition no
/// syscall observably ran to completion in this case.
/// The worker thread panicked, the OS rejected the thread spawn,
/// or its sender disconnected before sending a result. Treat as a
/// benign no-op (callers usually log and continue) rather than a
/// hard error — by definition no syscall observably ran to
/// completion in this case. In the spawn-failure case no thread is
/// leaked.
WorkerLost,
}
@@ -101,6 +103,12 @@ where
F: FnOnce() -> R + Send + 'static,
R: Send + 'static,
{
// If the caller already requested halt, don't spawn (and leak) a
// worker that would run `op` to completion in the background.
if halt.is_some_and(|h| h.is_cancelled()) {
return Err(BoundedError::Halted);
}
// Rendezvous channel: the worker sends exactly one value (the
// op's return) and then exits. Capacity-0 means the send blocks
// until we receive — fine on the happy path; on the timeout /
@@ -235,4 +243,53 @@ mod tests {
assert!(matches!(r, Ok("ok")));
assert!(flag.load(Ordering::Relaxed));
}
// ── Added hardening tests ───────────────────────────────────────
/// Doc contract (lines 106-110): "If the caller already requested
/// halt, don't spawn (and leak) a worker that would run `op`."
/// When halt is pre-cancelled the op closure must NEVER run — the
/// short-circuit returns Halted before spawning the worker. We
/// prove the op did not execute by checking a side-effect flag.
#[test]
fn pre_cancelled_halt_never_runs_op() {
let halt = Halt::new();
halt.cancel();
let ran = Arc::new(AtomicBool::new(false));
let r2 = ran.clone();
let r = bounded_syscall(Some(&halt), Duration::from_secs(2), move || {
r2.store(true, Ordering::SeqCst);
7u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
// The op closure must not have been scheduled at all.
assert!(
!ran.load(Ordering::SeqCst),
"op ran despite pre-cancelled halt — short-circuit at line 108 broken"
);
}
/// Timeout boundary: with a tiny deadline and an op that sleeps
/// much longer, the helper must return Timeout and must do so
/// roughly at the deadline — NOT wait for the op to finish (that
/// is the whole point of the bounded wrapper; the worker is
/// leaked). Grounds the `Instant::now() >= deadline` arm (line 141)
/// and the leak contract (doc lines 84-88).
#[test]
fn timeout_returns_near_deadline_not_after_op() {
let started = Instant::now();
let r = bounded_syscall(None, Duration::from_millis(100), || {
thread::sleep(Duration::from_secs(3));
0u32
});
let elapsed = started.elapsed();
assert!(matches!(r, Err(BoundedError::Timeout)));
// Must bail near the 100ms deadline (one POLL_INTERVAL slack at
// most), not after the 3s op. Allow generous CI slack but stay
// well under the op's 3s sleep.
assert!(
elapsed < Duration::from_millis(1500),
"timeout did not return near deadline: {elapsed:?} (op should be leaked, not awaited)"
);
}
}
-330
View File
@@ -1,330 +0,0 @@
//! Byte-sized bounded producer/consumer channel.
//!
//! Wraps `std::sync::mpsc::sync_channel` with a byte-accounting
//! `Mutex<usize> + Condvar` cap. Sender blocks (cooperatively) when
//! `used_bytes + item.byte_size() > capacity_bytes`. Receiver
//! decrements `used_bytes` when it takes the item.
//!
//! Why: the existing producer→consumer channel between `DiscStream`
//! (PES producer) and `MuxSink` (PES consumer) is bounded by frame
//! count. Frame sizes vary 100× between metadata and keyframes, so a
//! count-based cap either starves on small frames or buffers far too
//! much memory on big ones. Byte-sized accounting sizes the buffer for
//! the worst-case input stall (NFS read p99 ≈ 12 s × ~15 MB/s peak
//! compressed bitrate ≈ ~30 MB) directly.
//!
//! The underlying mpsc channel is created with a very large slot count
//! so the byte cap (not the slot count) is the real backpressure. Slot
//! count is only there to give the kernel a small chunk to wake on.
//!
//! See `freemkv-private/memory/project_buffering_architecture.md` §
//! Pipeline channel — sizing.
use std::sync::mpsc::{Receiver as MpscReceiver, RecvError, SendError, SyncSender, sync_channel};
use std::sync::{Arc, Condvar, Mutex};
/// Default byte cap for the muxer's input channel. Sized to hide a
/// worst-case ~2 s NFS read refill at UHD peak compressed bitrate
/// (~15 MB/s); 64 MiB gives headroom. Tweakable; not magic.
pub const BYTE_CHANNEL_DEFAULT_CAPACITY: usize = 64 * 1024 * 1024;
/// Slot capacity of the inner `sync_channel`. Large so the byte cap is
/// the real backpressure mechanism — the mpsc slot count only exists
/// to give the kernel a chunk to wake on. PES frames are typically
/// ~700 B each, so 64 MiB ≈ 90 k frames; 200 k is comfortable headroom.
const INNER_SLOT_CAPACITY: usize = 200_000;
/// Anything whose in-memory cost can be accounted by a single
/// `usize`. Implement on the item type sent through [`Sender`].
pub trait HasByteSize {
/// Bytes this item contributes to the channel's used budget.
/// Must be > 0 to make progress (a 0-byte item would never
/// block the sender no matter the cap; see send_blocks_at_capacity
/// test).
fn byte_size(&self) -> usize;
}
impl HasByteSize for crate::pes::PesFrame {
fn byte_size(&self) -> usize {
// Frame data + the fixed header overhead the serializer
// writes (track + pts + keyframe + len). The `Vec<u8>` heap
// allocation also has alloc-header overhead but that's
// <0.1 % at typical frame sizes — folding it in would just
// add noise to the budget.
self.data.len() + 14
}
}
/// Shared book-keeping between [`Sender`] and [`Receiver`]. Wrapped in
/// an `Arc` because both halves hold it independently.
struct Accounting {
used: Mutex<usize>,
cv: Condvar,
capacity: usize,
}
/// Send half of the byte-bounded channel.
///
/// `send` blocks (on a `Condvar`) when adding the item would push
/// `used_bytes` past `capacity_bytes`. Unblocks when the receiver
/// `recv`s items out and notifies. Returns `Err(item)` if the
/// receiver has been dropped — mirrors `mpsc::SyncSender::send`.
pub struct Sender<T: HasByteSize> {
tx: SyncSender<T>,
acct: Arc<Accounting>,
}
impl<T: HasByteSize> Clone for Sender<T> {
fn clone(&self) -> Self {
Sender {
tx: self.tx.clone(),
acct: self.acct.clone(),
}
}
}
impl<T: HasByteSize> Sender<T> {
/// Push one item. Blocks until adding it would not exceed the
/// capacity, then sends through the inner mpsc channel.
pub fn send(&self, item: T) -> Result<(), SendError<T>> {
let sz = item.byte_size();
// Reserve capacity first. The reservation is observable to
// other senders via `used`; only after we win the slot do we
// hand the item to the inner mpsc channel. That ordering means
// `used` is always a conservative upper bound on what's in the
// mpsc queue + about-to-be-sent.
{
let mut used = self.acct.used.lock().expect("byte_channel poisoned");
// An item bigger than the whole capacity will never fit; let
// it through anyway as a one-shot reservation, otherwise the
// sender deadlocks forever waiting for `used == 0` AND
// nothing in flight. The receiver will drain it on the
// other side. Same behaviour as `std::sync::mpsc` for
// arbitrarily large messages.
while *used + sz > self.acct.capacity && *used > 0 {
used = self.acct.cv.wait(used).expect("byte_channel cv poisoned");
}
*used += sz;
}
match self.tx.send(item) {
Ok(()) => Ok(()),
Err(SendError(returned)) => {
// Receiver dropped — refund the reservation so a later
// sender on a clone doesn't observe phantom used bytes
// (the receiver is gone so nobody will decrement).
let mut used = self.acct.used.lock().expect("byte_channel poisoned");
*used = used.saturating_sub(sz);
self.acct.cv.notify_all();
Err(SendError(returned))
}
}
}
}
/// Receive half of the byte-bounded channel.
///
/// `recv` blocks on the inner mpsc until an item is available, then
/// decrements the byte-accounting and wakes any sender waiting on
/// capacity.
pub struct Receiver<T: HasByteSize> {
rx: MpscReceiver<T>,
acct: Arc<Accounting>,
}
impl<T: HasByteSize> Receiver<T> {
/// Take the next item. Returns `Err(RecvError)` when all senders
/// have been dropped and the channel is empty.
pub fn recv(&self) -> Result<T, RecvError> {
let item = self.rx.recv()?;
let sz = item.byte_size();
let mut used = self.acct.used.lock().expect("byte_channel poisoned");
*used = used.saturating_sub(sz);
// Notify all so multi-sender setups wake every blocked sender,
// not just one. Wasted wakeups are cheap; missed wakeups would
// be a deadlock.
self.acct.cv.notify_all();
Ok(item)
}
}
/// Create a byte-bounded channel with the given capacity in bytes.
/// Returns a `(Sender, Receiver)` pair; clone the `Sender` for
/// multi-producer setups.
pub fn channel<T: HasByteSize>(capacity_bytes: usize) -> (Sender<T>, Receiver<T>) {
let (tx, rx) = sync_channel::<T>(INNER_SLOT_CAPACITY);
let acct = Arc::new(Accounting {
used: Mutex::new(0),
cv: Condvar::new(),
capacity: capacity_bytes,
});
(
Sender {
tx,
acct: acct.clone(),
},
Receiver { rx, acct },
)
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::thread;
use std::time::{Duration, Instant};
/// Test payload — its `byte_size` returns whatever we passed at
/// construction so capacity math is exact and predictable.
#[derive(Clone, Debug, PartialEq, Eq)]
struct Item {
sz: usize,
tag: u32,
}
impl HasByteSize for Item {
fn byte_size(&self) -> usize {
self.sz
}
}
#[test]
fn send_recv_round_trip() {
let (tx, rx) = channel::<Item>(1024);
for i in 0..5 {
tx.send(Item { sz: 100, tag: i }).unwrap();
}
for i in 0..5 {
let got = rx.recv().unwrap();
assert_eq!(got, Item { sz: 100, tag: i });
}
}
#[test]
fn byte_accounting_decrements_on_recv() {
// Internal book-keeping check via observable side-effect: after
// sending K items totalling N bytes and receiving them all, a
// subsequent send of an N-byte item must NOT block (no items
// in flight, all capacity refunded).
let (tx, rx) = channel::<Item>(1024);
for _ in 0..4 {
tx.send(Item { sz: 256, tag: 0 }).unwrap();
}
for _ in 0..4 {
rx.recv().unwrap();
}
// Cap is now fully available again. Send a 1024-byte item; the
// `used > 0` guard means it goes through alone (no wait).
let start = Instant::now();
tx.send(Item { sz: 1024, tag: 99 }).unwrap();
assert!(start.elapsed() < Duration::from_millis(100));
let got = rx.recv().unwrap();
assert_eq!(got.tag, 99);
}
#[test]
fn send_blocks_at_capacity_unblocks_on_recv() {
// Cap = 200 bytes, item = 100 bytes. First two sends fit
// exactly; the third must block until a recv frees capacity.
let (tx, rx) = channel::<Item>(200);
tx.send(Item { sz: 100, tag: 0 }).unwrap();
tx.send(Item { sz: 100, tag: 1 }).unwrap();
let tx2 = tx.clone();
let sent_at = Arc::new(Mutex::new(None::<Instant>));
let sent_at2 = sent_at.clone();
let h = thread::spawn(move || {
tx2.send(Item { sz: 100, tag: 2 }).unwrap();
*sent_at2.lock().unwrap() = Some(Instant::now());
});
// Give the sender thread a head start; it should be parked in
// `cv.wait` because used (200) + 100 > capacity (200).
thread::sleep(Duration::from_millis(100));
assert!(
sent_at.lock().unwrap().is_none(),
"third send should be blocked at capacity"
);
// Drain one. Sender wakes and completes.
let recv_at = Instant::now();
let got = rx.recv().unwrap();
assert_eq!(got.tag, 0);
h.join().unwrap();
let sent_when = sent_at.lock().unwrap().unwrap();
assert!(
sent_when >= recv_at,
"sender must complete AFTER receiver freed capacity"
);
// Drain the remaining two.
assert_eq!(rx.recv().unwrap().tag, 1);
assert_eq!(rx.recv().unwrap().tag, 2);
}
#[test]
fn item_larger_than_capacity_still_goes_through() {
// Pathological case: a single item bigger than the capacity.
// The guard `*used > 0` lets it through when the channel is
// empty (otherwise the sender deadlocks forever). Matches
// `mpsc::SyncSender` semantics for oversize messages.
let (tx, rx) = channel::<Item>(100);
tx.send(Item { sz: 1000, tag: 7 }).unwrap();
let got = rx.recv().unwrap();
assert_eq!(got, Item { sz: 1000, tag: 7 });
}
#[test]
fn concurrent_send_recv_stress() {
// 4 sender threads × 1k items each, 1 receiver. Verify byte
// accounting stays sane (channel never deadlocks, every item
// arrives exactly once) under contention.
const SENDERS: u32 = 4;
const PER_SENDER: u32 = 1000;
const TOTAL: u32 = SENDERS * PER_SENDER;
let (tx, rx) = channel::<Item>(8 * 1024);
let sent = Arc::new(AtomicUsize::new(0));
let mut handles = Vec::new();
for s in 0..SENDERS {
let tx = tx.clone();
let sent = sent.clone();
handles.push(thread::spawn(move || {
for i in 0..PER_SENDER {
// Vary item size so accounting actually has to
// multiplex differently-sized blockers. 1B → 256B.
let sz = 1 + ((i as usize) % 256);
tx.send(Item {
sz,
tag: s * PER_SENDER + i,
})
.unwrap();
sent.fetch_add(1, Ordering::SeqCst);
}
}));
}
// Drop our local sender so the receiver can eventually see
// RecvError once all sender clones are done. Cloning the
// sender into each producer means each clone Drop'd separately.
drop(tx);
let mut received = 0u32;
while let Ok(_item) = rx.recv() {
received += 1;
}
for h in handles {
h.join().unwrap();
}
assert_eq!(received, TOTAL);
assert_eq!(sent.load(Ordering::SeqCst) as u32, TOTAL);
}
#[test]
fn send_after_recv_dropped_returns_err() {
let (tx, rx) = channel::<Item>(1024);
drop(rx);
let r = tx.send(Item { sz: 10, tag: 0 });
assert!(r.is_err());
}
}
+439 -26
View File
@@ -10,11 +10,12 @@
//!
//! This is the byte-stream half of the freemkv mux highway —
//! `BytePrefetcher` feeds [`crate::mux::demux_thread::DemuxThread`]
//! for `m2ts://`, `network://`, `stdio://`, and any other stream
//! whose source is an `io::Read` rather than a `SectorSource`.
//! for `m2ts://` (the only in-tree caller today, via
//! [`crate::mux::resolve`]), and works for any stream whose source is
//! an `io::Read` rather than a `SectorSource`.
use crate::halt::Halt;
use crossbeam_channel::{Receiver, Sender, bounded};
use crate::halt::{Halt, POLL_INTERVAL};
use crossbeam_channel::{Receiver, RecvTimeoutError, SendTimeoutError, Sender, bounded};
use std::io::Read;
use std::thread::JoinHandle;
@@ -38,6 +39,13 @@ pub const DEFAULT_CHUNK_BYTES: usize = 16 * 1024 * 1024;
/// Returned from [`BytePrefetcher::into_channels`]. Owns the
/// producer-thread join handle so dropping the shell joins the
/// producer.
///
/// Drop blocks the calling thread until the producer exits. To
/// guarantee a prompt exit, drop the forward receiver and the recycle
/// sender first so the producer observes channel disconnection (or
/// cancel the [`Halt`] passed to [`BytePrefetcher::new`], which the
/// producer polls at [`POLL_INTERVAL`] granularity even while parked
/// on a channel op).
pub struct PrefetchShell {
producer: Option<JoinHandle<()>>,
}
@@ -52,8 +60,8 @@ impl Drop for PrefetchShell {
/// Spawned byte prefetcher. Drop joins the producer thread.
pub struct BytePrefetcher {
rx: Receiver<Batch>,
recycle_tx: Sender<Vec<u8>>,
rx: Option<Receiver<Batch>>,
recycle_tx: Option<Sender<Vec<u8>>>,
producer: Option<JoinHandle<()>>,
}
@@ -66,7 +74,13 @@ impl BytePrefetcher {
mut reader: R,
chunk_bytes: usize,
halt: Option<Halt>,
) -> Self {
) -> std::io::Result<Self> {
// A zero-length chunk makes every recycled buffer an empty
// slice; `reader.read(&mut [])` returns Ok(0), which the loop
// below treats as EOF — the consumer would see a clean,
// silent zero-byte stream. Callers pass the downstream
// demuxer's batch size, which is always > 0.
debug_assert!(chunk_bytes > 0, "BytePrefetcher chunk_bytes must be > 0");
let (tx, rx) = bounded::<Batch>(FORWARD_DEPTH);
let (recycle_tx, recycle_rx) = bounded::<Vec<u8>>(RECYCLE_DEPTH);
@@ -80,16 +94,49 @@ impl BytePrefetcher {
let producer = std::thread::Builder::new()
.name("freemkv-byte-prefetch".into())
.spawn(move || {
// Wrap the feed loop in catch_unwind so a panic in the inner
// `reader.read` (e.g. a decrypt-on-read slice/arith bug) is NOT
// indistinguishable from a clean finish at the demux boundary. A
// clean exit (EOF, halt, consumer disconnect) returns and drops
// `tx` → the demux loop reads RecvError as EOF (correct). A PANIC
// sends an explicit error sentinel first so the demux loop's
// `Ok(Err(_))` arm fires and propagates a typed error instead of
// converting the dropped channel into a clean `DemuxBatch::Eof`
// that would finalize a TRUNCATED mux while reporting success.
let body = std::panic::AssertUnwindSafe(|| {
let cancelled = || halt.as_ref().map(|h| h.is_cancelled()).unwrap_or(false);
// Liveness heartbeat: the producer blocks on the recycle and
// forward channels; a stalled consumer or a wedged reader shows
// up as the beat going silent. Total is unknown, so `pos` is
// cumulative bytes read.
let mut hb = crate::progress::Heartbeat::new("byte_prefetch");
let mut produced_bytes: u64 = 0;
loop {
if halt.as_ref().map(|h| h.is_cancelled()).unwrap_or(false) {
hb.tick(produced_bytes, 0);
if cancelled() {
return;
}
let mut buf = match recycle_rx.recv() {
Ok(b) => b,
Err(_) => return, // consumer dropped both channels
// Park on the recycle channel, but re-poll halt
// every POLL_INTERVAL: a pure-AtomicBool Halt does
// not disconnect the channel, so a blocking recv()
// would never re-reach the cancel check.
let mut buf = loop {
match recycle_rx.recv_timeout(POLL_INTERVAL) {
Ok(b) => break b,
Err(RecvTimeoutError::Timeout) => {
if cancelled() {
return;
}
}
// Consumer dropped both channels.
Err(RecvTimeoutError::Disconnected) => return,
}
};
// Re-expose the full extent (previous iteration
// may have truncated after a short read).
// Re-expose the full extent. After a short read the
// prior iteration truncated to n < chunk_bytes, so
// this regrows the length back to chunk_bytes
// without reallocating (capacity was fixed at
// construction and never shrinks).
if buf.len() < chunk_bytes {
buf.resize(chunk_bytes, 0);
} else {
@@ -108,19 +155,40 @@ impl BytePrefetcher {
return;
}
};
produced_bytes += n as u64;
buf.truncate(n);
if tx.send(Ok(buf)).is_err() {
return; // consumer dropped
// Hand off the filled buffer, re-polling halt on
// each timeout slice so a cancel can interrupt a
// producer parked on a saturated forward channel.
let mut pending = Ok(buf);
loop {
match tx.send_timeout(pending, POLL_INTERVAL) {
Ok(()) => break,
Err(SendTimeoutError::Timeout(returned)) => {
if cancelled() {
return;
}
pending = returned;
}
// Consumer dropped.
Err(SendTimeoutError::Disconnected(_)) => return,
}
}
})
.expect("freemkv-byte-prefetch thread spawn failed");
}
});
if std::panic::catch_unwind(body).is_err() {
// Producer panicked mid-stream — surface a typed terminal
// error so the demux thread does NOT read the dropped channel
// as a clean EOF and truncate output.
let _ = tx.send(Err(crate::error::Error::DemuxThreadPanicked.into()));
}
})?;
Self {
rx,
recycle_tx,
Ok(Self {
rx: Some(rx),
recycle_tx: Some(recycle_tx),
producer: Some(producer),
}
})
}
/// Peel off the channels for zero-copy pipeline consumption. The
@@ -128,19 +196,364 @@ impl BytePrefetcher {
/// drains `rx`, runs the demuxer in place on each filled buffer,
/// and recycles back through `recycle_tx`.
pub fn into_channels(self) -> (Receiver<Batch>, Sender<Vec<u8>>, PrefetchShell) {
let mut me = self;
let producer = me.producer.take();
let rx = me.rx.clone();
let recycle = me.recycle_tx.clone();
std::mem::forget(me);
// MOVE the three fields out cleanly — never clone. Each of
// `rx` and `recycle_tx` ends up with exactly ONE live copy:
// the one in the returned tuple. The pre-1.0.0 implementation
// cloned both and then `mem::forget`-ed `self`, leaking the
// originals so an extra live receiver + sender survived
// forever. That defeated the channel-disconnection shutdown:
// when the demux consumer exited early (halt, or a `tx.send`
// error in `demux_thread`), the producer's `recycle_rx.recv()`
// and `tx.send()` never saw all-peers-dropped, so the producer
// never returned and `PrefetchShell::drop`'s `join()` hung.
//
// `ManuallyDrop` + `ptr::read` reads each field out by value
// and suppresses `self`'s own `Drop` (which would otherwise
// double-`join`), leaving NO extra live endpoint behind. This
// is the panic-free equivalent of the `Option::take` approach
// and mirrors `sector::prefetched::into_channels`.
let me = std::mem::ManuallyDrop::new(self);
// SAFETY: `me` is `ManuallyDrop`, so none of these fields will
// be dropped by `me`. Each `ptr::read` performs exactly one
// bitwise move out; every field is read exactly once and never
// touched again, so there are no double-frees and no aliasing.
let producer = unsafe { std::ptr::read(&me.producer) };
// SAFETY: `rx` and `recycle_tx` are always `Some` here —
// `into_channels` is the only way to consume a live
// `BytePrefetcher`; `Drop::drop` is suppressed by `ManuallyDrop`.
let rx = unsafe { std::ptr::read(&me.rx) }.expect("rx always Some before drop");
let recycle =
unsafe { std::ptr::read(&me.recycle_tx) }.expect("recycle_tx always Some before drop");
(rx, recycle, PrefetchShell { producer })
}
}
impl Drop for BytePrefetcher {
fn drop(&mut self) {
// Drop channel endpoints BEFORE joining the producer so the
// producer observes SendTimeoutError::Disconnected (forward tx)
// or RecvTimeoutError::Disconnected (recycle rx) and exits
// promptly. Without this, a non-EOF source fills the depth-2
// forward channel and then spins in send_timeout(POLL_INTERVAL)
// forever because rx is never drained, causing join() to
// deadlock.
drop(self.rx.take());
drop(self.recycle_tx.take());
if let Some(h) = self.producer.take() {
let _ = h.join();
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Endless reader: every `read` fills the whole buffer and never
/// hits EOF, so the producer keeps trying to push batches forward
/// until the forward channel disconnects. Exactly the shape that
/// wedged the pre-1.0.0 `clone + mem::forget` `into_channels`.
struct EndlessReader;
impl Read for EndlessReader {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
buf.fill(0);
Ok(buf.len())
}
}
/// Run `f` on a helper thread and fail if it does not finish within
/// `secs`. Turns a join-deadlock into a test failure instead of a
/// hung CI run.
fn within<F: FnOnce() + Send + 'static>(secs: u64, f: F) {
let (done_tx, done_rx) = bounded::<()>(1);
std::thread::spawn(move || {
f();
let _ = done_tx.send(());
});
assert!(
done_rx
.recv_timeout(std::time::Duration::from_secs(secs))
.is_ok(),
"operation did not complete within {secs}s (deadlock)"
);
}
/// The CRITICAL regression: after `into_channels`, dropping the
/// returned forward receiver + recycle sender must let the producer
/// observe disconnection and exit, so dropping the `PrefetchShell`
/// (which joins the producer) returns promptly. With the old
/// clone+forget the leaked endpoints kept the producer blocked and
/// this join hung forever.
#[test]
fn into_channels_drop_releases_producer() {
within(10, || {
// Small chunk so the producer cycles quickly and fills the
// forward channel without allocating much.
let pf = BytePrefetcher::new(EndlessReader, 4096, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
// Consumer goes away early (halt / abort analogue): drop
// both channel endpoints without draining to EOF.
drop(rx);
drop(recycle_tx);
// Joining the producer must not hang.
drop(shell);
});
}
/// Same property via the halt path: cancel the token, then the
/// producer must exit and the shell join must complete.
#[test]
fn halt_releases_producer() {
within(10, || {
let halt = Halt::new();
let pf = BytePrefetcher::new(EndlessReader, 4096, Some(halt.clone())).expect("spawn");
let (_rx, _recycle_tx, shell) = pf.into_channels();
halt.cancel();
drop(shell);
});
}
// ── Added hardening tests ───────────────────────────────────────
use std::io::Cursor;
/// Drain the forward channel, recycling every buffer, and
/// reassemble the bytes. Returns the concatenation of every
/// delivered chunk. Stops on RecvError (producer dropped tx == EOF)
/// or on the first Err batch (which it returns separately).
fn drain_to_vec(pf: BytePrefetcher) -> (Vec<u8>, Option<std::io::Error>) {
let (rx, recycle_tx, shell) = pf.into_channels();
let mut out = Vec::new();
let mut err = None;
while let Ok(batch) = rx.recv() {
match batch {
Ok(buf) => {
out.extend_from_slice(&buf);
// Recycle so the producer can refill. Ignore send
// error (producer may have already exited at EOF).
let _ = recycle_tx.send(buf);
}
Err(e) => {
err = Some(e);
break;
}
}
}
drop(rx);
drop(recycle_tx);
drop(shell);
(out, err)
}
/// CORE CONTRACT: the prefetcher must deliver every source byte,
/// in order, exactly once — never silently truncate or duplicate.
/// Source is 5000 bytes; chunk size 1024 forces multiple chunks
/// (4 full + 1 short of 904). The reassembled stream must equal the
/// source. Mutation: replacing `buf.truncate(n)` (line 141) with a
/// no-op would over-report bytes on the final short read and this
/// fails.
#[test]
fn delivers_all_bytes_in_order_across_chunks() {
within(10, || {
let src: Vec<u8> = (0..5000u32).map(|i| (i & 0xff) as u8).collect();
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 1024, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none(), "unexpected error batch: {err:?}");
assert_eq!(got, src, "prefetcher truncated or reordered bytes");
});
}
/// Short-read truncation: a reader that returns fewer bytes than
/// requested per call must NOT leave stale tail bytes in the
/// delivered chunk. Cursor over 10 bytes with a 4096 chunk yields a
/// single 10-byte chunk; the consumer must see exactly 10 bytes,
/// not 4096. Grounds `buf.truncate(n)` at line 141. Mutation:
/// delete the truncate and the chunk would carry 4086 zero bytes of
/// padding, failing the length assert.
#[test]
fn short_read_truncates_to_actual_length() {
within(10, || {
let src = vec![0xAB; 10];
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 4096, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none());
assert_eq!(got.len(), 10, "delivered chunk padded past actual read");
assert_eq!(got, src);
});
}
/// EOF semantics: an empty source (Cursor over `[]`) yields
/// `read() == Ok(0)` on the first call, which the producer treats
/// as EOF and returns, dropping tx. The consumer sees RecvError
/// (zero batches), NOT an Err batch and NOT a zero-length Ok batch.
/// Grounds the `Ok(0) => return` arm at line 134. Mutation:
/// changing `Ok(0) => return` to `Ok(0) => continue` would spin
/// forever (within() would time out).
#[test]
fn empty_source_yields_clean_eof_no_batches() {
within(10, || {
let pf = BytePrefetcher::new(Cursor::new(Vec::<u8>::new()), 4096, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
// No Ok batch should ever arrive; first recv must be Err
// (producer dropped tx at EOF).
let first = rx.recv();
assert!(
first.is_err(),
"empty source produced a batch instead of clean EOF: {first:?}"
);
drop(rx);
drop(recycle_tx);
drop(shell);
});
}
/// Error propagation: a reader that fails mid-stream must surface
/// the io::Error as an `Err` batch on the forward channel (line
/// 137), not swallow it. We deliver one good chunk then an error.
/// The consumer must see the good bytes followed by the error.
/// Mutation: changing `let _ = tx.send(Err(e)); return;` to a plain
/// `return` would drop the error silently and this fails.
#[test]
fn read_error_is_propagated_as_err_batch() {
within(10, || {
struct OneThenError {
served: bool,
}
impl Read for OneThenError {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if !self.served {
self.served = true;
let n = buf.len().min(8);
buf[..n].fill(0x11);
Ok(n)
} else {
Err(std::io::Error::other("synthetic mid-stream read failure"))
}
}
}
let pf = BytePrefetcher::new(OneThenError { served: false }, 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert_eq!(got, vec![0x11; 8], "good chunk lost");
let err = err.expect("read error must surface as an Err batch");
assert_eq!(err.kind(), std::io::ErrorKind::Other);
});
}
/// PANIC propagation: a reader that PANICS mid-stream must NOT be read as a
/// clean EOF at the demux boundary. The producer's catch_unwind sends an
/// explicit `Err` sentinel before the thread unwinds, so the consumer sees
/// the good bytes followed by an error batch — never a silent truncation.
/// Without the catch_unwind the panic would just drop `tx`, the consumer
/// would see RecvError (== clean EOF) and the partial output would be
/// finalized as if complete.
#[test]
fn read_panic_surfaces_as_err_batch_not_clean_eof() {
within(10, || {
struct OneThenPanic {
served: bool,
}
impl Read for OneThenPanic {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if !self.served {
self.served = true;
let n = buf.len().min(8);
buf[..n].fill(0x22);
Ok(n)
} else {
panic!("synthetic mid-stream reader panic");
}
}
}
let pf = BytePrefetcher::new(OneThenPanic { served: false }, 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert_eq!(got, vec![0x22; 8], "good chunk lost before the panic");
assert!(
err.is_some(),
"a mid-stream producer PANIC must surface as an Err batch, \
not a clean EOF (which would silently truncate the mux)"
);
});
}
/// Recycle-buffer reuse must NOT leak stale bytes between chunks of
/// different lengths. After a full chunk, a short read reuses the
/// same recycled buffer; lines 123-129 regrow it to chunk_bytes
/// before reading, then line 141 truncates to the short count. We
/// verify the short chunk carries only fresh bytes by reassembling
/// the full stream. Source: 8 bytes of 0xAA + 3 bytes of 0xBB, with
/// chunk_bytes=8 → chunk0 = 8×0xAA, chunk1 = 3×0xBB.
#[test]
fn recycled_buffer_carries_no_stale_tail() {
within(10, || {
let mut src = vec![0xAA; 8];
src.extend_from_slice(&[0xBB; 3]);
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none());
assert_eq!(
got, src,
"stale bytes from recycled buffer leaked into short chunk"
);
});
}
/// Exact-multiple boundary: when the source length is an exact
/// multiple of chunk_bytes, the final non-empty chunk is followed
/// by an `Ok(0)` EOF read, NOT a spurious empty Ok batch. 12 bytes
/// with chunk_bytes=4 → three 4-byte chunks then clean EOF. Total
/// bytes must equal 12 and no zero-length batch may appear.
#[test]
fn exact_multiple_length_no_trailing_empty_batch() {
within(10, || {
let src = vec![0x42u8; 12];
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 4, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
let mut total = 0usize;
let mut batch_count = 0usize;
while let Ok(Ok(buf)) = rx.recv() {
assert!(!buf.is_empty(), "producer emitted a zero-length batch");
total += buf.len();
batch_count += 1;
let _ = recycle_tx.send(buf);
}
assert_eq!(total, 12);
assert_eq!(batch_count, 3, "expected exactly 3 full chunks");
drop(rx);
drop(recycle_tx);
drop(shell);
});
}
/// Dropping the BytePrefetcher directly (without into_channels)
/// must join the producer cleanly when the source is finite. The
/// producer reaches EOF, drops tx, and exits; Drop's join returns.
/// Grounds the BytePrefetcher Drop impl (lines 202-208). Mutation:
/// removing the `Ok(0) => return` EOF exit would hang this join.
#[test]
fn drop_finite_prefetcher_joins_cleanly() {
within(10, || {
let pf = BytePrefetcher::new(Cursor::new(vec![1u8; 100]), 4096, None).expect("spawn");
// Drop without consuming — producer fills the forward
// channel (capacity 2), reaches EOF on the third read since
// 100 < 4096 (single chunk + EOF), drops tx, exits.
drop(pf);
});
}
/// Regression: dropping a BytePrefetcher directly (without
/// into_channels) with an ENDLESS source must not deadlock. Before
/// the fix, Drop joined the producer while rx/recycle_tx were still
/// alive (sibling field drop order), so the producer filled the
/// depth-2 forward channel and then spun in send_timeout forever
/// (rx never drained, halt=None). The fix drops rx+recycle_tx
/// BEFORE the join so the producer sees SendTimeoutError::Disconnected
/// and exits.
#[test]
fn drop_endless_prefetcher_joins_cleanly() {
within(10, || {
let pf = BytePrefetcher::new(EndlessReader, 4096, None).expect("spawn");
// Drop without consuming — the old Drop deadlocked here.
drop(pf);
});
}
}
+7 -16
View File
@@ -8,32 +8,22 @@
use std::fs::File;
use std::os::unix::io::AsRawFd;
/// `F_RDADVISE` opcode — not in libc's named constants on all SDKs.
const F_RDADVISE: libc::c_int = 44;
/// Cap on the byte length we pass to `F_RDADVISE`. Asking for a
/// multi-GB readahead window is counterproductive — the OS doesn't
/// have that much cache to throw at one fd. 64 MiB is generous for
/// our use case (sweep, mux) and matches the byte-channel cap so the
/// kernel's prefetch ≥ our app-level pipeline depth.
/// our use case (sweep, mux) so the kernel's prefetch ≥ our app-level
/// pipeline depth.
const RDADVISE_MAX_BYTES: i64 = 64 * 1024 * 1024;
/// `radvisory` per `<sys/fcntl.h>`. repr(C) layout is stable.
#[repr(C)]
struct RadAdvisory {
ra_offset: libc::off_t,
ra_count: libc::c_int,
}
pub(super) fn hint_sequential(file: &File, len_bytes: u64) {
let bytes = (len_bytes as i64).min(RDADVISE_MAX_BYTES);
let mut ra = RadAdvisory {
let mut ra = libc::radvisory {
ra_offset: 0,
ra_count: bytes as libc::c_int,
};
// Best-effort.
unsafe {
libc::fcntl(file.as_raw_fd(), F_RDADVISE, &mut ra);
libc::fcntl(file.as_raw_fd(), libc::F_RDADVISE, &mut ra);
}
}
@@ -52,11 +42,12 @@ pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
/// returns immediately.
pub(super) fn prefetch(file: &File, offset: u64, len: u64) {
let bytes = (len as i64).min(RDADVISE_MAX_BYTES);
let mut ra = RadAdvisory {
let mut ra = libc::radvisory {
ra_offset: offset as libc::off_t,
ra_count: bytes as libc::c_int,
};
// Best-effort — kernel hint only.
unsafe {
libc::fcntl(file.as_raw_fd(), F_RDADVISE, &mut ra);
libc::fcntl(file.as_raw_fd(), libc::F_RDADVISE, &mut ra);
}
}
+184 -24
View File
@@ -17,9 +17,17 @@
//! Without page-cache eviction an 85 GB streaming ISO read pins the
//! entire file in memory, starves the concurrent writer, and collapses
//! mux throughput (observed: 2.7 MB/s mux on 0.21.5 vs. 70 MB/s
//! isolated NFS reads). Every [`READ_DROP_CHUNK_BYTES`] of consumed
//! bytes we call `posix_fadvise(DONTNEED)` over that window, mirroring
//! the write-side [`crate::io::writeback::WritebackPipeline`] policy.
//! isolated NFS reads). Every [`READ_DROP_CHUNK_BYTES_DEFAULT`] of
//! consumed bytes we call `posix_fadvise(DONTNEED)` over that window,
//! mirroring the write-side [`crate::io::writeback::WritebackPipeline`]
//! policy.
//!
//! The drop window is accounted by a monotonic forward byte counter,
//! which matches the sequential streaming pattern the mux highway
//! drives. Under random or backward access the dropped range no longer
//! lines up with the bytes actually read — but `DONTNEED` is purely an
//! advisory cache hint with no correctness impact, so this degrades to
//! a slightly imprecise hint rather than a bug.
//!
//! ## Platform open hint
//!
@@ -64,7 +72,7 @@ use std::path::Path;
use crate::error::{Error, Result};
use crate::sector::SectorSource;
const SECTOR_SIZE: usize = 2048;
use crate::consts::{SECTOR_BYTES, SECTOR_BYTES_U64};
/// Bytes-read threshold per `posix_fadvise(DONTNEED)` drop on the
/// read side. Mirrors `WRITEBACK_CHUNK_BYTES` so the read-side page
@@ -102,7 +110,10 @@ pub struct FileSectorSource {
bytes_read_since_drop: u64,
/// File offset at which the current drop window starts. The next
/// DONTNEED drops from `drop_window_start` for
/// `bytes_read_since_drop` bytes.
/// `bytes_read_since_drop` bytes. This advances monotonically with
/// the byte count, so it tracks the actual reads only under the
/// forward-sequential access the mux highway uses; under random
/// access it degrades to a harmless, imprecise advisory hint.
drop_window_start: u64,
/// Cached drop chunk size (resolved from env once at open).
drop_chunk_bytes: u64,
@@ -116,16 +127,18 @@ impl FileSectorSource {
///
/// Issues the platform's "sequential access expected" hint on the
/// fd (Linux `posix_fadvise(SEQUENTIAL)`, macOS `fcntl(F_RDADVISE)`,
/// Windows TODO stub) so the kernel's readahead widens.
pub fn open(path: &Path) -> std::io::Result<Self> {
let file = File::open(path)?;
let len = file.metadata()?.len();
let sectors = len / SECTOR_SIZE as u64;
/// Windows no-op) so the kernel's readahead widens.
pub fn open(path: &Path) -> Result<Self> {
let file = File::open(path).map_err(|e| Error::IoError { source: e })?;
let len = file
.metadata()
.map_err(|e| Error::IoError { source: e })?
.len();
let sectors = len / SECTOR_BYTES_U64;
if sectors > u32::MAX as u64 {
return Err(Error::IsoTooLarge {
path: path.to_string_lossy().into_owned(),
}
.into());
});
}
let capacity = sectors as u32;
@@ -157,7 +170,7 @@ impl SectorSource for FileSectorSource {
_recovery: bool,
) -> Result<usize> {
let count = count as u32;
let bytes = count as usize * SECTOR_SIZE;
let bytes = count as usize * SECTOR_BYTES;
debug_assert!(
out.len() >= bytes,
"FileSectorSource::read_sectors: out len {} < requested {}",
@@ -167,7 +180,7 @@ impl SectorSource for FileSectorSource {
if count == 0 {
return Ok(0);
}
let offset = lba as u64 * SECTOR_SIZE as u64;
let offset = lba as u64 * SECTOR_BYTES_U64;
self.file
.seek(SeekFrom::Start(offset))
.map_err(|e| Error::IoError { source: e })?;
@@ -211,7 +224,7 @@ mod tests {
/// verify any sector by content alone.
fn make_iso(path: &std::path::Path, sectors: u32) {
let mut f = std::fs::File::create(path).unwrap();
let mut chunk = vec![0u8; SECTOR_SIZE];
let mut chunk = vec![0u8; SECTOR_BYTES];
for n in 0..sectors {
let b = (n & 0xff) as u8;
chunk.iter_mut().for_each(|c| *c = b);
@@ -236,7 +249,7 @@ mod tests {
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), total);
let mut got = vec![0u8; SECTOR_SIZE];
let mut got = vec![0u8; SECTOR_BYTES];
for lba in 0..total {
src.read_sectors(lba, 1, &mut got, false).unwrap();
let expected = (lba & 0xff) as u8;
@@ -257,12 +270,12 @@ mod tests {
let mut src = FileSectorSource::open(&path).unwrap();
let span_lba = TEST_SPAN_SECTORS - 2;
let mut buf4 = vec![0u8; SECTOR_SIZE * 4];
let mut buf4 = vec![0u8; SECTOR_BYTES * 4];
src.read_sectors(span_lba, 4, &mut buf4, false).unwrap();
for i in 0..4 {
let lba = span_lba + i as u32;
let expected = (lba & 0xff) as u8;
for b in &buf4[i * SECTOR_SIZE..(i + 1) * SECTOR_SIZE] {
for b in &buf4[i * SECTOR_BYTES..(i + 1) * SECTOR_BYTES] {
assert_eq!(*b, expected, "byte mismatch at sub-sector {i}");
}
}
@@ -278,7 +291,7 @@ mod tests {
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
let mut got = vec![0u8; SECTOR_SIZE];
let mut got = vec![0u8; SECTOR_BYTES];
src.read_sectors(TEST_SPAN_SECTORS + 1, 1, &mut got, false)
.unwrap();
@@ -298,7 +311,7 @@ mod tests {
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), total);
let mut got = vec![0u8; SECTOR_SIZE];
let mut got = vec![0u8; SECTOR_BYTES];
src.read_sectors(0, 1, &mut got, false).unwrap();
src.read_sectors(total - 1, 1, &mut got, false).unwrap();
let expected = ((total - 1) & 0xff) as u8;
@@ -317,15 +330,15 @@ mod tests {
let mut src = FileSectorSource::open(&path).unwrap();
let req = (TEST_SPAN_SECTORS + 1) as u16;
let req_bytes = req as usize * SECTOR_SIZE;
let req_bytes = req as usize * SECTOR_BYTES;
let mut big = vec![0u8; req_bytes];
src.read_sectors(0, req, &mut big, false).unwrap();
assert!(big[..SECTOR_SIZE].iter().all(|b| *b == 0));
assert!(big[..SECTOR_BYTES].iter().all(|b| *b == 0));
let last_lba = req as u32 - 1;
let exp = (last_lba & 0xff) as u8;
let last_off = (req as usize - 1) * SECTOR_SIZE;
let last_off = (req as usize - 1) * SECTOR_BYTES;
assert!(
big[last_off..last_off + SECTOR_SIZE]
big[last_off..last_off + SECTOR_BYTES]
.iter()
.all(|b| *b == exp)
);
@@ -358,4 +371,151 @@ mod tests {
std::env::remove_var("FREEMKV_READ_DROP_CHUNK_MIB");
}
}
// ---------------------------------------------------------------
// Additional coverage.
// ---------------------------------------------------------------
/// `count == 0` must short-circuit to Ok(0) WITHOUT seeking or
/// reading, even at an out-of-range LBA — the early-return guard
/// runs before any I/O. Grounding: `if count == 0 { return Ok(0) }`.
#[test]
fn zero_count_returns_zero_no_io() {
let dir = tempdir().unwrap();
let path = dir.path().join("zc.iso");
make_iso(&path, 4);
let mut src = FileSectorSource::open(&path).unwrap();
// LBA far past EOF — must not matter because count==0 returns early.
let mut buf = [0u8; 1];
let n = src.read_sectors(1_000_000, 0, &mut buf, false).unwrap();
assert_eq!(n, 0);
}
/// Reading past EOF must ERROR (read_exact's UnexpectedEof), never
/// return a partial/short count. This is the core "never silently
/// truncate / never return fewer bytes than declared" property of
/// the SectorSource contract. Grounding: `self.file.read_exact(...)`
/// — read_exact fails if the file can't supply the full span.
#[test]
fn read_past_eof_errors_not_truncates() {
let dir = tempdir().unwrap();
let path = dir.path().join("eof.iso");
make_iso(&path, 4); // 4 sectors only
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 4);
// Request 2 sectors starting at LBA 3 → sector 4 doesn't exist.
let mut buf = vec![0u8; 2 * SECTOR_BYTES];
let r = src.read_sectors(3, 2, &mut buf, false);
let err = r.expect_err("reading past EOF must error, not short-read");
let io: std::io::Error = err.into();
assert_eq!(
io.kind(),
std::io::ErrorKind::UnexpectedEof,
"partial read at EOF must surface read_exact's UnexpectedEof"
);
}
/// On a successful full read the returned count MUST equal
/// `count * 2048` exactly — the declared byte count. Grounding:
/// `Ok(bytes)` where `bytes = count * SECTOR_BYTES`.
#[test]
fn full_read_returns_exact_declared_bytes() {
let dir = tempdir().unwrap();
let path = dir.path().join("exact.iso");
make_iso(&path, 16);
let mut src = FileSectorSource::open(&path).unwrap();
let mut buf = vec![0u8; 5 * SECTOR_BYTES];
let n = src.read_sectors(2, 5, &mut buf, false).unwrap();
assert_eq!(n, 5 * SECTOR_BYTES, "must return exactly count*2048 bytes");
}
/// Capacity is `file_len / 2048` (floor); trailing bytes that don't
/// complete a sector are NOT counted. A file of 4 sectors + 100
/// extra bytes reports capacity 4. Grounding: `len / SECTOR_BYTES`
/// integer division in `open`.
#[test]
fn capacity_floors_partial_trailing_sector() {
let dir = tempdir().unwrap();
let path = dir.path().join("partial.iso");
make_iso(&path, 4);
// Append 100 stray bytes (a torn final sector).
{
let mut f = std::fs::OpenOptions::new()
.append(true)
.open(&path)
.unwrap();
f.write_all(&[0xee; 100]).unwrap();
f.flush().unwrap();
}
let src = FileSectorSource::open(&path).unwrap();
assert_eq!(
src.capacity_sectors(),
4,
"partial trailing bytes must not inflate the sector capacity"
);
}
/// An empty file opens cleanly with capacity 0. Grounding:
/// `0 / 2048 == 0`, and the IsoTooLarge guard only fires for
/// oversize files.
#[test]
fn empty_file_capacity_zero() {
let dir = tempdir().unwrap();
let path = dir.path().join("empty.iso");
std::fs::File::create(&path).unwrap();
let src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 0);
}
/// Opening a nonexistent path returns an IoError (NotFound), not a
/// panic. Grounding: `File::open(path).map_err(...)`.
#[test]
fn open_missing_file_errors() {
let dir = tempdir().unwrap();
let path = dir.path().join("does-not-exist.iso");
let err = match FileSectorSource::open(&path) {
Ok(_) => panic!("missing file must error"),
Err(e) => e,
};
let io: std::io::Error = err.into();
assert_eq!(io.kind(), std::io::ErrorKind::NotFound);
}
/// A DONTNEED drop crossing the chunk threshold must not corrupt or
/// short subsequent reads — the eviction is a pure page-cache hint.
/// We read past the DEFAULT 32 MiB drop chunk (16384 sectors) so the
/// eviction block fires at least once, asserting every sector still
/// reads correctly. (Avoids mutating FREEMKV_READ_DROP_CHUNK_MIB to
/// sidestep a parallel-test env race with `drop_chunk_size_env_override`.)
/// Grounding: the `bytes_read_since_drop >= drop_chunk_bytes`
/// eviction block calls only `platform::drop_window` (advisory) and
/// resets counters — no data effect.
#[test]
fn dontneed_eviction_does_not_affect_data() {
// 32 MiB default chunk = 16384 sectors; read a bit past it.
let total = (READ_DROP_CHUNK_BYTES_DEFAULT / SECTOR_BYTES_U64) as u32 + 64;
let dir = tempdir().unwrap();
let path = dir.path().join("drop.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
// Read in 16-sector batches to keep the loop fast while still
// crossing the drop boundary by byte count.
let batch = 16u16;
let mut got = vec![0u8; batch as usize * SECTOR_BYTES];
let mut lba = 0u32;
while lba + batch as u32 <= total {
src.read_sectors(lba, batch, &mut got, false).unwrap();
for i in 0..batch as u32 {
let expected = ((lba + i) & 0xff) as u8;
let off = i as usize * SECTOR_BYTES;
assert!(
got[off..off + SECTOR_BYTES].iter().all(|x| *x == expected),
"DONTNEED eviction corrupted sector {}",
lba + i
);
}
lba += batch as u32;
}
}
}
+8 -10
View File
@@ -1,20 +1,18 @@
//! Windows: the canonical sequential-access hint is
//! `FILE_FLAG_SEQUENTIAL_SCAN` passed to `CreateFile` at open time —
//! it cannot be set after the fact via `SetFileInformationByHandle`.
//! Routing the open call through this module would mean a custom
//! `File::from_raw_handle` plumb for every `FileSectorSource::open`
//! caller, which is more invasive than the Phase 1 scope.
//!
//! TODO: replumb `FileSectorSource::open` to take an
//! `OpenOptions`-style builder so the Windows path can flip the flag
//! at open time. For now this is a no-op stub.
//! `FILE_FLAG_SEQUENTIAL_SCAN`, which must be passed to `CreateFile`
//! at open time and cannot be set afterward via
//! `SetFileInformationByHandle`. Since `FileSectorSource::open` uses a
//! plain `File::open`, the hints in this module are no-op stubs.
use std::fs::File;
/// No-op stub. `FILE_FLAG_SEQUENTIAL_SCAN` can only be set at
/// `CreateFile` open time, which the plain `File::open` path does not
/// do, so there is no post-open hint to issue here.
pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {
tracing::debug!(
target: "mux",
"FileSectorSource hint_sequential: windows stub (TODO: FILE_FLAG_SEQUENTIAL_SCAN at open)"
"FileSectorSource hint_sequential: windows no-op stub"
);
}
+92
View File
@@ -0,0 +1,92 @@
//! Platform-aware crash-durability primitives.
//!
//! Two flush operations need OS-specific handling to make a write survive a
//! crash / power loss:
//!
//! - [`dir`] — fsync a directory so a prior `rename(2)` into it is durable.
//! After a crash a renamed file's dirent can otherwise be lost even though
//! the rename returned, because it is still page-cache-only. This is a POSIX
//! concept: on Windows std cannot even open a directory as a `File` (it does
//! not set `FILE_FLAG_BACKUP_SEMANTICS`), and NTFS/ReFS commit the rename's
//! dirent without an explicit directory flush — so it is a no-op there
//! rather than a failed open that logs on every marker write.
//!
//! - [`file_durable`] — fsync a file's contents + metadata. Opens the file
//! **read+write**: on Windows `File::sync_all` maps to `FlushFileBuffers`,
//! which requires a handle with write access and returns
//! `ERROR_ACCESS_DENIED` (os error 5) on a read-only handle. (A read-only
//! `File::open` + `sync_all` is legal on POSIX, which is why that bug only
//! bit Windows.) The open mode is platform-uniform, so this lives here with
//! no dispatch.
//!
//! Per the crate convention (see [`crate::io::writeback_file`]), platform
//! dispatch happens once here via cfg-gated `mod` decls — callers carry no
//! inline `#[cfg(...)]`.
use std::io;
use std::path::Path;
#[cfg(not(windows))]
mod posix;
#[cfg(windows)]
mod windows;
#[cfg(not(windows))]
use posix as platform;
#[cfg(windows)]
use windows as platform;
/// fsync a directory so a prior `rename(2)` into it is durable. Best-effort:
/// failures are logged and swallowed, never propagated — the renamed file's
/// bytes are already synced and the caller's write itself succeeded. No-op on
/// Windows (see module docs).
pub fn dir(path: &Path) {
platform::fsync_dir(path)
}
/// Durably flush an existing file's contents + metadata to stable storage.
///
/// Opens the file read+write (not read-only) so the flush succeeds on every
/// platform — see the module docs for the Windows `FlushFileBuffers` rationale.
/// The file must already exist; its bytes are left intact (no create/truncate).
pub fn file_durable(path: &Path) -> io::Result<()> {
let f = std::fs::OpenOptions::new()
.read(true)
.write(true)
.open(path)?;
f.sync_all()
}
#[cfg(test)]
mod tests {
use super::*;
/// `file_durable` opens read+write (so the flush works on Windows) and
/// syncs an existing file; a missing path surfaces as `Err` so the caller
/// treats it as "not durably synced". Platform-uniform — same on
/// unix/windows.
#[test]
fn file_durable_ok_for_existing_err_for_missing() {
let td = tempfile::tempdir().unwrap();
let f = td.path().join("data.bin");
std::fs::write(&f, b"durable").unwrap();
assert!(
file_durable(&f).is_ok(),
"an existing file must open read+write and fsync cleanly"
);
assert!(
file_durable(&td.path().join("absent.bin")).is_err(),
"a missing file must surface the open failure as Err"
);
}
/// `dir` is best-effort: it must return normally for a real directory
/// (POSIX fsyncs it, Windows no-ops) and must swallow — never panic on —
/// a missing directory.
#[test]
fn dir_is_best_effort_never_panics() {
let td = tempfile::tempdir().unwrap();
dir(td.path());
dir(&td.path().join("does-not-exist"));
}
}
+18
View File
@@ -0,0 +1,18 @@
//! POSIX directory-fsync. Active on unix and any non-Windows fallback target
//! (BSD, illumos, …) — all share the same `File::open(dir).sync_all()`
//! semantics. The Windows no-op lives in the sibling `windows` module.
use std::path::Path;
pub(super) fn fsync_dir(dir: &Path) {
match std::fs::File::open(dir) {
Ok(f) => {
if let Err(e) = f.sync_all() {
tracing::warn!(path = %dir.display(), error = %e, "failed to fsync directory");
}
}
Err(e) => {
tracing::warn!(path = %dir.display(), error = %e, "could not open directory to fsync");
}
}
}
+13
View File
@@ -0,0 +1,13 @@
//! Windows directory-fsync: a no-op.
//!
//! Directory fsync is a POSIX concept. std cannot open a directory as a `File`
//! on Windows (it does not set `FILE_FLAG_BACKUP_SEMANTICS`), so the POSIX impl
//! could only ever fail the open and log a spurious warning on every marker /
//! mapfile write. NTFS/ReFS commit a rename's directory entry without an
//! explicit directory flush, so skipping it here is correct — not a durability
//! regression. (File-content durability is handled platform-uniformly by
//! [`super::file_durable`].)
use std::path::Path;
pub(super) fn fsync_dir(_dir: &Path) {}
+6 -7
View File
@@ -18,18 +18,19 @@
//! consumed window so an 85 GB streaming ISO read doesn't fill the
//! page cache and starve the concurrent MKV write.
//!
//! `Pipeline` + `Sink` (0.18) is the generic producer/consumer primitive
//! `Pipeline` + `Sink` is the generic producer/consumer primitive
//! used by sweep, patch, and mux to overlap reads with writes via a
//! bounded channel + dedicated consumer thread.
//!
//! `byte_channel` is a byte-sized producer/consumer channel for the
//! mux pipeline, sized to absorb worst-case input read stalls (see
//! `freemkv-private/memory/project_buffering_architecture.md`).
//! `byte_prefetcher` is the read-ahead producer feeding the mux
//! pipeline for `io::Read`-backed sources: a worker thread fills a
//! recycled pool of buffers and ships them through a channel, exposing
//! `BytePrefetcher` / `PrefetchShell`.
pub(crate) mod bounded;
pub mod byte_channel;
pub mod byte_prefetcher;
pub mod file_sector_source;
pub mod fsync;
pub mod sink;
mod writeback;
mod writeback_file;
@@ -41,8 +42,6 @@ pub mod pipeline;
pub(crate) use writeback_file::WritebackFile;
// Re-exports for the 0.18 redesign. Sweep, patch, and mux are all
// wired up (disc/sweep.rs, disc/patch.rs, autorip's ripper/mux.rs).
pub use pipeline::{
DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, READ_PIPELINE_DEPTH, Sink, WRITE_PIPELINE_DEPTH,
WRITE_THROUGH_DEPTH,
+795 -116
View File
File diff suppressed because it is too large Load Diff
+74 -3
View File
@@ -12,14 +12,18 @@
//! size patch, Cues index, segment header backpatch) to land on the
//! right offset.
//!
//! `RandomAccessSink` is satisfied via the blanket impl in
//! [`super::mod`]; no explicit impl needed here.
//! [`SequentialSink`](super::SequentialSink) is implemented explicitly
//! (not via a blanket impl) so its `finish()` flushes the `BufWriter`
//! and `fsync`s the file even when called through a `dyn` trait object;
//! [`RandomAccessSink`](super::RandomAccessSink) is implemented over the
//! `Seek` impl below.
use std::fs::{File, OpenOptions};
use std::io::{self, BufWriter, Seek, SeekFrom, Write};
use std::path::Path;
use super::preallocate;
use super::{RandomAccessSink, SequentialSink};
const BUFFER_BYTES: usize = 4 * 1024 * 1024;
@@ -80,13 +84,26 @@ impl LocalFileSink {
/// Drain the internal buffer and `fsync` the underlying file.
/// Idempotent with `Drop` (the `BufWriter` also flushes on drop;
/// this call additionally surfaces fsync errors to the caller).
#[allow(dead_code)] // exposed for parity with WritebackFile::sync_all
/// [`SequentialSink::finish`](super::SequentialSink::finish)
/// delegates here so the durable flush happens through a trait
/// object too.
pub fn sync_all(&mut self) -> io::Result<()> {
self.inner.flush()?;
self.inner.get_ref().sync_all()
}
}
impl SequentialSink for LocalFileSink {
/// Flush the 4 MiB `BufWriter` and `fsync` the file. Overriding the
/// trait default is what makes a `dyn SequentialSink` / `dyn
/// RandomAccessSink` `finish()` actually durable instead of a no-op.
fn finish(&mut self) -> io::Result<()> {
self.sync_all()
}
}
impl RandomAccessSink for LocalFileSink {}
impl Write for LocalFileSink {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.inner.write(buf)
@@ -159,4 +176,58 @@ mod tests {
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..], b"hint-ok");
}
// ── Added hardening tests ───────────────────────────────────────
/// `create` must TRUNCATE an existing file (OpenOptions
/// `.truncate(true)`, lines 52-58). Pre-seed a long file, recreate
/// it via the sink, write a shorter payload — the old tail must be
/// gone. Mutation: dropping `.truncate(true)` would leave the stale
/// tail and the length assert fails.
#[test]
fn create_truncates_existing_file() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("trunc.bin");
std::fs::write(&p, vec![0xFFu8; 4096]).unwrap();
let mut s = LocalFileSink::create(&p).unwrap();
s.write_all(b"short").unwrap();
s.sync_all().unwrap();
drop(s);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(
bytes.len(),
5,
"create must truncate the pre-existing 4096 bytes"
);
assert_eq!(&bytes, b"short");
}
/// Seek must flush the BufWriter FIRST so buffered bytes land at
/// their intended offset, not the post-seek one (lines 121-128, and
/// the module doc's silent-corruption warning). We write into the
/// buffer (no explicit flush), seek backward, write again, and
/// confirm the first write stayed at offset 0. Mutation: removing
/// the `self.inner.flush()?` in `seek` would flush the first 4
/// bytes at the seeked offset, corrupting the file.
#[test]
fn seek_flushes_buffer_before_moving() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("seek-flush.bin");
let mut s = LocalFileSink::create(&p).unwrap();
// These bytes sit in the 4 MiB BufWriter, unflushed.
s.write_all(b"HEAD").unwrap();
// Seek forward to offset 10; the buffered HEAD must be flushed
// to offset 0 BEFORE the position moves.
s.seek(SeekFrom::Start(10)).unwrap();
s.write_all(b"TAIL").unwrap();
s.sync_all().unwrap();
drop(s);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(
&bytes[0..4],
b"HEAD",
"buffered head landed at the wrong offset"
);
assert_eq!(&bytes[10..14], b"TAIL");
}
}
+126 -48
View File
@@ -20,9 +20,6 @@
//! 4 MiB buffer for the common local-disk case; `WritebackFile`
//! (separate module) wraps a `File` with the adaptive-chunk
//! `sync_file_range` machinery for the Linux+NFS case.
//!
//! See `freemkv-private/memory/project_buffering_architecture.md` for
//! the full design and the source/sink matrix.
use std::io::{Seek, Write};
@@ -38,14 +35,22 @@ pub use socket::{SocketSink, UdpSocketSink};
/// trait does not impose or hide any buffering of its own.
///
/// `finish` drains any internal buffering and signals end-of-stream to
/// the underlying transport (close-write on a socket, flush on a
/// buffered writer, etc.). The default impl is a no-op; concrete
/// implementations that need explicit shutdown can override it but the
/// blanket impl below keeps it optional for adapter types like
/// `&mut File`.
/// the underlying transport (close-write on a socket, flush + fsync on
/// a buffered file, etc.). The default impl flushes via [`Write::flush`]
/// — correct for an unbuffered destination — but every concrete sink in
/// this module overrides it to drain its own buffer and run its
/// transport-specific finalisation (socket `shutdown(Write)`, file
/// `fsync`). There is deliberately NO blanket `impl SequentialSink for
/// T`: a blanket impl would force the no-op-style default on every
/// concrete sink (a blanket impl cannot be overridden per-type without a
/// coherence conflict), so a `Box<dyn SequentialSink>` / `&mut dyn
/// SequentialSink` `finish()` call would silently skip the flush and
/// transport shutdown. With explicit per-type impls the vtable dispatches
/// `finish` to the real implementation, so flush + durable-finish
/// actually happen through a trait object.
pub trait SequentialSink: Write + Send {
fn finish(&mut self) -> std::io::Result<()> {
Ok(())
self.flush()
}
}
@@ -54,15 +59,6 @@ pub trait SequentialSink: Write + Send {
/// random-access sink is always usable as a sequential sink.
pub trait RandomAccessSink: SequentialSink + Seek {}
// Blanket impls so any `Write + Send` type acts as a `SequentialSink`
// (with default `finish`), and any sink that also impls `Seek` is
// automatically a `RandomAccessSink`. Keeps call-site ergonomics simple
// — `&mut File`, `LocalFileSink`, `WritebackFile`, `BufWriter<File>`,
// and `Cursor<Vec<u8>>` all satisfy the right trait without per-type
// boilerplate.
impl<T: Write + Send + ?Sized> SequentialSink for T {}
impl<T: SequentialSink + Seek + ?Sized> RandomAccessSink for T {}
/// Pick the right `RandomAccessSink` impl for `dest` based on its
/// filesystem type.
///
@@ -80,18 +76,16 @@ impl<T: SequentialSink + Seek + ?Sized> RandomAccessSink for T {}
///
/// Returns a boxed trait object so the call site (mux construction)
/// stays agnostic of which concrete sink got picked.
#[allow(dead_code)] // wiring to mux::resolve is a follow-up commit
pub fn open_for_mkv(
// Not yet wired into mux::resolve (follow-up commit). Kept `pub(crate)` until
// then so an unfinished signature isn't frozen into the public 1.0 API.
#[allow(dead_code)]
pub(crate) fn open_for_mkv(
dest: &std::path::Path,
size_hint: Option<u64>,
) -> std::io::Result<Box<dyn RandomAccessSink>> {
#[cfg(not(target_os = "linux"))]
use crate::platform::fs_type::detect;
#[cfg(target_os = "linux")]
use crate::platform::fs_type::{FsType, detect};
#[cfg(target_os = "linux")]
{
use crate::platform::fs_type::{FsType, detect};
if detect(dest) == FsType::Nfs {
let wf = match size_hint {
Some(n) => crate::io::WritebackFile::create_with_size_hint(dest, n)?,
@@ -100,12 +94,13 @@ pub fn open_for_mkv(
return Ok(Box::new(wf));
}
}
// Silence the unused-binding warning on non-Linux where the only
// branch above is cfg-gated out.
// Only Linux differentiates the sink by filesystem type (NFS gets
// the WritebackFile machinery); every other OS always uses
// `LocalFileSink`. Reference `detect` as a value (no call, no
// `statfs` syscall) so it isn't flagged dead on non-Linux while
// still avoiding the wasted probe whose result we'd discard.
#[cfg(not(target_os = "linux"))]
{
let _ = detect(dest);
}
let _ = crate::platform::fs_type::detect;
let sink = match size_hint {
Some(n) => LocalFileSink::with_size_hint(dest, n)?,
@@ -117,32 +112,25 @@ pub fn open_for_mkv(
#[cfg(test)]
mod tests {
use super::*;
use std::fs::File;
// Type-level assertion: the blanket impls cover the shapes we care
// about. These functions never run; they just have to type-check.
fn _assert_file_is_sequential(_: &mut dyn SequentialSink) {}
fn _assert_file_is_random_access(_: &mut dyn RandomAccessSink) {}
// Type-level assertion: the concrete sinks satisfy the trait
// objects. These functions never run; they just have to type-check.
fn _assert_is_sequential(_: &mut dyn SequentialSink) {}
fn _assert_is_random_access(_: &mut dyn RandomAccessSink) {}
#[test]
fn blanket_impls_cover_file_and_localfilesink() {
// `File` directly via blanket impls.
fn concrete_sinks_satisfy_traits() {
let dir = tempfile::tempdir().unwrap();
let mut f = File::create(dir.path().join("a.bin")).unwrap();
_assert_file_is_sequential(&mut f);
_assert_file_is_random_access(&mut f);
// `LocalFileSink` ditto.
// `LocalFileSink` is a random-access (and thus sequential) sink.
let mut s = LocalFileSink::create(&dir.path().join("b.bin")).unwrap();
_assert_file_is_sequential(&mut s);
_assert_file_is_random_access(&mut s);
_assert_is_sequential(&mut s);
_assert_is_random_access(&mut s);
// `WritebackFile` — confirms the Phase 1 type still satisfies
// the trait via the blanket impl without needing an explicit
// `impl RandomAccessSink for WritebackFile {}`.
// `WritebackFile` ditto, via its explicit per-type impls.
let mut wf = crate::io::WritebackFile::create(&dir.path().join("c.bin")).unwrap();
_assert_file_is_sequential(&mut wf);
_assert_file_is_random_access(&mut wf);
_assert_is_sequential(&mut wf);
_assert_is_random_access(&mut wf);
}
#[test]
@@ -158,4 +146,94 @@ mod tests {
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..5], b"hello");
}
/// finish() through a `dyn SequentialSink` trait object must
/// dispatch to the concrete sink's override (flush + fsync), not a
/// no-op default. This is the regression test for the silent-no-op
/// finish() bug.
#[test]
fn finish_through_trait_object_flushes_local_file() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("trait-finish.bin");
let sink = LocalFileSink::create(&p).unwrap();
// Box as the trait object the production path uses.
let mut boxed: Box<dyn SequentialSink> = Box::new(sink);
boxed.write_all(b"buffered-tail").unwrap();
// finish() through the vtable must drain the 4 MiB BufWriter and
// fsync; the bytes must be visible to a separate reader BEFORE
// we drop the sink (drop-flush must not be what saves us).
boxed.finish().unwrap();
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..], b"buffered-tail");
}
// ── Added hardening tests ───────────────────────────────────────
use std::io::{self, Write};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
/// A minimal `SequentialSink` that does NOT override `finish`, so it
/// exercises the trait's DEFAULT impl (lines 51-55), which must call
/// `Write::flush`. We record whether flush ran. This pins the
/// documented contract that the default `finish` is "correct for an
/// unbuffered destination" by flushing. Mutation: changing the
/// default `finish` body from `self.flush()` to `Ok(())` would set
/// `flushed=false` and fail.
struct FlushTracker {
flushed: Arc<AtomicBool>,
bytes: Arc<AtomicUsize>,
}
impl Write for FlushTracker {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.bytes.fetch_add(buf.len(), Ordering::SeqCst);
Ok(buf.len())
}
fn flush(&mut self) -> io::Result<()> {
self.flushed.store(true, Ordering::SeqCst);
Ok(())
}
}
// Uses the DEFAULT finish() — deliberately no override.
impl SequentialSink for FlushTracker {}
#[test]
fn default_finish_flushes() {
let flushed = Arc::new(AtomicBool::new(false));
let bytes = Arc::new(AtomicUsize::new(0));
let mut sink = FlushTracker {
flushed: flushed.clone(),
bytes: bytes.clone(),
};
sink.write_all(b"abc").unwrap();
assert!(
!flushed.load(Ordering::SeqCst),
"flush should not run before finish"
);
sink.finish().unwrap();
assert!(
flushed.load(Ordering::SeqCst),
"default SequentialSink::finish must call Write::flush"
);
assert_eq!(bytes.load(Ordering::SeqCst), 3);
}
/// `open_for_mkv` with `None` size hint must still produce a working
/// random-access sink (the `match size_hint { None => ... }` arm,
/// lines 103-106). Round-trip a seek-back patch through it to prove
/// both Write and Seek dispatch. Mutation: if the None arm returned
/// a sequential-only sink the seek would not compile / would fail.
#[test]
fn open_for_mkv_without_size_hint_is_random_access() {
use std::io::{Seek, SeekFrom};
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("nohint.bin");
let mut sink = open_for_mkv(&p, None).unwrap();
sink.write_all(b"AAAABBBB").unwrap();
sink.seek(SeekFrom::Start(4)).unwrap();
sink.write_all(b"CCCC").unwrap();
sink.finish().unwrap();
drop(sink);
assert_eq!(std::fs::read(&p).unwrap(), b"AAAACCCC");
}
}
+5 -2
View File
@@ -5,13 +5,16 @@
//! the file naturally.
use std::fs::File;
#[cfg(unix)]
use std::os::unix::io::AsRawFd;
pub(super) fn preallocate_impl(file: &File, size_bytes: u64) {
let fd = file.as_raw_fd();
// Clamp to the signed `off_t` range fallocate expects; an unchecked
// `as i64` cast would wrap a >= 2^63 size to a negative length that
// fallocate rejects with EINVAL (silent no-op).
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX);
// FALLOC_FL_KEEP_SIZE = 0x01.
let rc = unsafe { libc::fallocate(fd, libc::FALLOC_FL_KEEP_SIZE, 0, size_bytes as i64) };
let rc = unsafe { libc::fallocate(fd, libc::FALLOC_FL_KEEP_SIZE, 0, len) };
tracing::debug!(
target: "mux",
"LocalFileSink fallocate size_hint={size_bytes} rc={rc} ok={}",
+14 -5
View File
@@ -1,7 +1,8 @@
//! macOS `F_PREALLOCATE` extent reservation.
//!
//! `fcntl(F_PREALLOCATE)` with `F_ALLOCATECONTIG` first (try for a
//! contiguous run) and fall back to `F_ALLOCATEALL` (non-contig OK).
//! `fcntl(F_PREALLOCATE)` with `F_ALLOCATECONTIG | F_ALLOCATEALL` first
//! (prefer a contiguous run but accept scattered extents to satisfy the
//! full length) and fall back to `F_ALLOCATEALL` alone on failure.
//! Reported file size is unchanged — the muxer's writes still grow it.
use std::fs::File;
@@ -13,16 +14,24 @@ use crate::io::platform_macos::{
pub(super) fn preallocate_impl(file: &File, size_bytes: u64) {
let fd = file.as_raw_fd();
// Clamp to the signed `off_t` range; an unchecked `as off_t` cast
// would wrap a >= 2^63 size to a negative length.
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX) as libc::off_t;
let mut store = Fstore {
fst_flags: F_ALLOCATECONTIG,
// Prefer a contiguous run but accept scattered extents to
// satisfy the full length. Without F_ALLOCATEALL the first
// attempt is best-effort and can return rc=0 with a partial
// allocation, so the fallback below would never fire. Matches
// writeback_file/macos.rs.
fst_flags: F_ALLOCATECONTIG | F_ALLOCATEALL,
fst_posmode: F_PEOFPOSMODE,
fst_offset: 0,
fst_length: size_bytes as libc::off_t,
fst_length: len,
fst_bytesalloc: 0,
};
let mut rc = unsafe { libc::fcntl(fd, F_PREALLOCATE, &mut store as *mut Fstore) };
if rc == -1 {
// Fall back to non-contiguous.
// Fall back to non-contiguous only.
store.fst_flags = F_ALLOCATEALL;
rc = unsafe { libc::fcntl(fd, F_PREALLOCATE, &mut store as *mut Fstore) };
}
+124 -26
View File
@@ -11,16 +11,21 @@
//! conventional choice). `finish()` is a no-op; UDP has no end-of-stream
//! marker.
//!
//! Both types satisfy [`SequentialSink`] via the blanket impl in
//! `super::mod`. Neither implements `Seek`, so neither satisfies
//! [`RandomAccessSink`] — using one with `MkvMux` is a compile error,
//! which is the design intent.
//! Both types implement [`SequentialSink`] explicitly so their
//! `finish()` dispatches correctly through a `dyn SequentialSink` trait
//! object (the `SocketSink` override drains the buffer and
//! `shutdown(Write)`s; the `UdpSocketSink` override flushes only).
//! Neither implements `Seek`, so neither satisfies [`RandomAccessSink`]
//! — using one with `MkvMux` is a compile error, which is the design
//! intent.
//!
//! [`SequentialSink`]: super::SequentialSink
//! [`RandomAccessSink`]: super::RandomAccessSink
use std::io::{self, BufWriter, Write};
use std::net::{Shutdown, TcpStream, ToSocketAddrs, UdpSocket};
use std::net::{Shutdown, SocketAddr, TcpStream, ToSocketAddrs, UdpSocket};
use super::SequentialSink;
/// `BufWriter` capacity for [`SocketSink`]. 1 MiB matches the typical
/// kernel send-buffer ceiling and keeps small-write amplification from
@@ -47,15 +52,17 @@ impl SocketSink {
/// writing. `sndbuf_bytes`, when present, is forwarded to
/// `setsockopt(SO_SNDBUF)` as a kernel hint — the OS may clamp it.
///
/// `addr` accepts anything `ToSocketAddrs` does: `"10.0.0.1:1234"`,
/// `addr` accepts anything `ToSocketAddrs` does: `"192.0.2.1:1234"`,
/// `("host", 1234)`, a `SocketAddr`, etc.
pub fn connect<A: ToSocketAddrs>(addr: A, sndbuf_bytes: Option<usize>) -> io::Result<Self> {
let stream = TcpStream::connect(addr)?;
// `set_nodelay(true)` keeps small writes (TS packet trains, fMP4
// moof headers) from sitting in Nagle's algorithm until the buffer
// fills. The BufWriter already absorbs syscall overhead; Nagle
// would just add latency without coalescing more.
stream.set_nodelay(true)?;
// would just add latency without coalescing more. It is a latency
// hint, not a correctness requirement, so a platform that rejects
// TCP_NODELAY must not fail the connect — demote the error.
let _ = stream.set_nodelay(true);
if let Some(n) = sndbuf_bytes {
set_send_buffer(&stream, n)?;
}
@@ -76,17 +83,12 @@ impl Write for SocketSink {
}
}
impl SocketSink {
impl SequentialSink for SocketSink {
/// Drain the BufWriter and `shutdown(Write)` the underlying socket
/// so the peer sees a clean EOF.
///
/// Note: [`SequentialSink::finish`](super::SequentialSink::finish)'s
/// blanket-impl default is a no-op. Trait-object call sites that
/// need socket shutdown should call this inherent method directly
/// before dropping the sink, or hold the concrete `SocketSink` type
/// (typical pattern: each muxer's `finish()` calls the appropriate
/// inherent close method on its captured concrete sink).
pub fn finish(&mut self) -> io::Result<()> {
/// so the peer sees a clean EOF. Overriding the trait default is
/// what makes a `dyn SequentialSink` `finish()` send the buffered
/// tail and the EOF instead of silently dropping them.
fn finish(&mut self) -> io::Result<()> {
self.buf.flush()?;
// `shutdown(Write)` signals clean EOF to the peer. Errors here
// are non-fatal — the connection may have already been torn down
@@ -117,10 +119,22 @@ impl UdpSocketSink {
///
/// `sndbuf_bytes`, when present, is a hint to `SO_SNDBUF`.
pub fn connect<A: ToSocketAddrs>(peer: A, sndbuf_bytes: Option<usize>) -> io::Result<Self> {
// Bind to all-zeros / any port. The kernel picks an ephemeral
// source port and the source IP at first send.
let socket = UdpSocket::bind("0.0.0.0:0")?;
socket.connect(peer)?;
// Resolve the peer first so the local bind matches its address
// family. Binding `0.0.0.0:0` (IPv4) and then connecting to an
// IPv6 peer fails with EAFNOSUPPORT, so pick the wildcard that
// matches the resolved family.
let peer_addr = peer
.to_socket_addrs()?
.next()
.ok_or_else(|| io::Error::from(io::ErrorKind::AddrNotAvailable))?;
let bind_addr = match peer_addr {
SocketAddr::V4(_) => "0.0.0.0:0",
SocketAddr::V6(_) => "[::]:0",
};
// Bind to the matching wildcard / any port. The kernel picks an
// ephemeral source port and the source IP at first send.
let socket = UdpSocket::bind(bind_addr)?;
socket.connect(peer_addr)?;
if let Some(n) = sndbuf_bytes {
set_udp_send_buffer(&socket, n)?;
}
@@ -140,10 +154,12 @@ impl Write for UdpSocketSink {
}
}
impl UdpSocketSink {
/// No-op — UDP has no end-of-stream marker. Provided for parity
/// with [`SocketSink::finish`] so call sites can treat them uniformly.
pub fn finish(&mut self) -> io::Result<()> {
impl SequentialSink for UdpSocketSink {
/// UDP has no end-of-stream marker, so there is nothing to shut
/// down; `write` already sent each datagram unbuffered. Flush is a
/// no-op but kept explicit so the trait-object `finish()` matches
/// the concrete behaviour.
fn finish(&mut self) -> io::Result<()> {
Ok(())
}
}
@@ -275,4 +291,86 @@ mod tests {
let n2 = receiver.recv(&mut buf).unwrap();
assert_eq!(&buf[..n2], &[9, 9, 9]);
}
// ── Added hardening tests ───────────────────────────────────────
/// `SocketSink::finish` must signal a clean EOF to the peer via
/// `shutdown(Write)` (lines 91-97). The receiving side's
/// `read_to_end` only returns when it observes that EOF — if
/// `finish` merely flushed without the shutdown, `read_to_end`
/// would block forever (the socket stays half-open). We assert the
/// receiver completes promptly AND sees the buffered tail.
/// Mutation: replacing the `shutdown(Write)` line with `Ok(())`
/// makes the accept thread hang and the join times out.
#[test]
fn finish_signals_eof_to_peer() {
use std::sync::mpsc;
use std::time::Duration;
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let (tx, rx) = mpsc::channel();
thread::spawn(move || {
let (mut sock, _) = listener.accept().unwrap();
let mut buf = Vec::new();
// Returns only when the peer half-closes (shutdown Write).
sock.read_to_end(&mut buf).unwrap();
let _ = tx.send(buf);
});
let mut sink = SocketSink::connect(addr, None).unwrap();
sink.write_all(b"unflushed-tail").unwrap();
sink.finish().unwrap();
// read_to_end must complete because finish() shut down writes.
let received = rx
.recv_timeout(Duration::from_secs(3))
.expect("peer never saw EOF — finish() did not shutdown(Write)");
assert_eq!(received, b"unflushed-tail");
}
/// UDP `write` must emit ONE datagram per call carrying exactly the
/// bytes passed — no buffering, no coalescing (doc lines 100-108).
/// Two writes of different lengths must arrive as two separate
/// datagrams of those exact lengths, in order. Mutation: adding a
/// BufWriter to UdpSocketSink (the doc explicitly forbids it) would
/// merge these into one datagram and the second `recv` would time
/// out.
#[test]
fn udp_write_is_one_datagram_per_call() {
let receiver = UdpSocket::bind("127.0.0.1:0").unwrap();
receiver
.set_read_timeout(Some(std::time::Duration::from_secs(2)))
.unwrap();
let addr = receiver.local_addr().unwrap();
let mut sink = UdpSocketSink::connect(addr, None).unwrap();
// Distinct lengths so a merge would be detectable.
let n_a = sink.write(&[0xAA; 10]).unwrap();
let n_b = sink.write(&[0xBB; 20]).unwrap();
assert_eq!(n_a, 10);
assert_eq!(n_b, 20);
let mut buf = [0u8; 256];
let first = receiver.recv(&mut buf).unwrap();
assert_eq!(first, 10, "first datagram must be exactly 10 bytes");
assert!(buf[..first].iter().all(|&b| b == 0xAA));
let second = receiver.recv(&mut buf).unwrap();
assert_eq!(second, 20, "second datagram must be exactly 20 bytes");
assert!(buf[..second].iter().all(|&b| b == 0xBB));
}
/// UDP `finish` is a documented no-op (lines 157-165): there is no
/// EOF marker for UDP. Calling it must not error and must not
/// affect prior datagrams. Mutation: if `finish` tried to
/// `shutdown` the UDP socket it could error or close it
/// prematurely; here it must just return Ok.
#[test]
fn udp_finish_is_noop_ok() {
let receiver = UdpSocket::bind("127.0.0.1:0").unwrap();
let addr = receiver.local_addr().unwrap();
let mut sink = UdpSocketSink::connect(addr, None).unwrap();
assert!(sink.finish().is_ok());
// A second finish is equally harmless.
assert!(sink.finish().is_ok());
}
}
+195 -28
View File
@@ -54,7 +54,6 @@
use std::collections::VecDeque;
use std::fs::File;
use std::os::unix::io::{AsRawFd, RawFd};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
@@ -79,6 +78,15 @@ pub(crate) struct WritebackPipeline {
/// `WritebackFile` and never exposed outside that wrapper, which
/// is what keeps the alias sound.
fd: RawFd,
/// An owned clone of the file descriptor, held so that any
/// leaked WAIT_AFTER worker thread retains a valid reference to
/// the underlying file description for the duration of its
/// syscall — even if the original `WritebackFile` is closed first
/// and the OS reuses its fd number. `None` only when `try_clone`
/// failed at construction (rare); the pipeline falls back to the
/// pre-clone `fd` integer in that case, which carries the original
/// fd-reuse risk but is no worse than the previous behaviour.
wait_file: Option<File>,
chunk_bytes: u64,
last_flush_pos: u64,
pending: Option<(u64, u64)>,
@@ -94,13 +102,11 @@ pub(crate) struct WritebackPipeline {
is_nfs: bool,
/// Set the first time WAIT_AFTER exceeds [`WAIT_AFTER_TIMEOUT`].
/// Once set, behaviour matches the NFS path for the rest of the
/// pipeline's life. Wrapped in `Arc` only because both this
/// struct and the spawned worker thread (which itself doesn't
/// touch the flag) share-via-fd patterns might one day need it;
/// today it's effectively a single-owner cell — the `Arc` shape
/// keeps the door open for moving the read side into a worker
/// without re-plumbing types.
degraded: Arc<AtomicBool>,
/// pipeline's life. A plain `AtomicBool`: the flag is only ever
/// touched on the owning thread (the spawned WAIT_AFTER worker never
/// reads or writes it). `AtomicBool` over `bool` only because the
/// load/store sites read cleanly; no sharing is needed today.
degraded: AtomicBool,
}
impl WritebackPipeline {
@@ -111,6 +117,19 @@ impl WritebackPipeline {
pub(crate) fn new(file: &File, start_pos: u64, chunk_bytes: u64) -> Self {
let fd = file.as_raw_fd();
let is_nfs = detect_nfs(fd);
// Clone the fd so any leaked WAIT_AFTER worker thread keeps the
// file description alive. Log but continue on clone failure.
let wait_file = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
tracing::warn!(
target: "mux",
"WritebackPipeline fd={fd}: try_clone failed ({e}), WAIT_AFTER workers \
will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
tracing::info!(
target: "mux",
"WritebackPipeline fd={fd} is_nfs={is_nfs} chunk_bytes={chunk_bytes} strategy={}",
@@ -118,13 +137,14 @@ impl WritebackPipeline {
);
Self {
fd,
wait_file,
chunk_bytes,
last_flush_pos: start_pos,
pending: None,
wait_after_window: VecDeque::with_capacity(ADAPTIVE_WINDOW),
chunk_count: 0,
is_nfs,
degraded: Arc::new(AtomicBool::new(false)),
degraded: AtomicBool::new(false),
}
}
@@ -136,6 +156,22 @@ impl WritebackPipeline {
self.is_nfs || self.degraded.load(Ordering::Relaxed)
}
/// Produce a fresh per-call `File` clone for the WAIT_AFTER worker.
///
/// Each call to `wait_after_with_timeout` needs its own owned clone
/// so the worker thread keeps the file description alive for the
/// duration of the syscall. We clone from `self.wait_file` (itself a
/// clone taken at construction) rather than from the original file.
///
/// Returns `None` only if `wait_file` is `None` (construction
/// try_clone failed) or if the second-level try_clone fails — both
/// rare; the fallback raw-fd path in `wait_after_with_timeout`
/// handles that case.
#[inline]
fn clone_for_worker(&self) -> Option<File> {
self.wait_file.as_ref().and_then(|f| f.try_clone().ok())
}
/// Caller advanced the file position to `pos`. If a chunk boundary
/// was crossed, kick async writeback for the just-completed chunk
/// and finalise the previous one.
@@ -143,16 +179,35 @@ impl WritebackPipeline {
if pos < self.last_flush_pos.saturating_add(self.chunk_bytes) {
return;
}
let chunk_off = self.last_flush_pos as i64;
let chunk_len = (pos - self.last_flush_pos) as i64;
// Byte offsets are unsigned throughout; the signed cast happens
// only at the libc call boundary where the kernel ABI requires
// `i64`. `saturating_sub` documents and hardens the line-above
// guard that `pos >= last_flush_pos`.
let chunk_off: u64 = self.last_flush_pos;
let chunk_len: u64 = pos.saturating_sub(self.last_flush_pos);
let mut wait_ms: u64 = 0;
let mut fadvise_ms: u64 = 0;
// Async kickoff for the just-completed chunk runs on every
// path (NFS, degraded, normal) — it's nominally non-blocking
// by spec and gives the kernel an early hint that this range
// is ready to flush.
unsafe {
libc::sync_file_range(self.fd, chunk_off, chunk_len, libc::SYNC_FILE_RANGE_WRITE);
let kickoff_rc = unsafe {
libc::sync_file_range(
self.fd,
chunk_off as i64,
chunk_len as i64,
libc::SYNC_FILE_RANGE_WRITE,
)
};
if kickoff_rc != 0 {
// Non-fatal: the async write-out hint failed, but the data is
// still in the page cache and will be flushed by later fsync /
// kernel writeback. Surface it for diagnosability.
tracing::warn!(
target: "freemkv::io",
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
"sync_file_range(WRITE) kickoff failed"
);
}
if let Some((prev_off, prev_len)) = self.pending.take() {
if self.skip_wait() {
@@ -166,7 +221,8 @@ impl WritebackPipeline {
// we mark the pipeline degraded, log a loud error,
// and fall through to the skip path on subsequent
// calls.
match wait_after_with_timeout(self.fd, prev_off, prev_len) {
match wait_after_with_timeout(self.clone_for_worker(), self.fd, prev_off, prev_len)
{
Some(ms) => {
wait_ms = ms;
let t_fadv = Instant::now();
@@ -187,9 +243,16 @@ impl WritebackPipeline {
// NOT call DONTNEED — if WAIT_AFTER hasn't
// returned, the pages aren't safely flushed.
self.degraded.store(true, Ordering::Relaxed);
// Once degraded we skip DONTNEED, so every subsequent
// chunk's pages stay resident until close — the same
// page-cache exposure profile as NFS. Shrink to the
// floor so that exposure window is as small as the NFS
// path keeps it, instead of whatever the adaptive sizing
// had grown chunk_bytes to (up to 256 MiB).
self.chunk_bytes = CHUNK_BYTES_MIN;
tracing::error!(
target: "mux",
"WritebackPipeline WAIT_AFTER timed out after {}s on chunk off={} len={}, marking writeback degraded (subsequent chunks will skip WAIT_AFTER + DONTNEED)",
"WritebackPipeline WAIT_AFTER timed out after {}s on chunk off={} len={}, marking writeback degraded (subsequent chunks will skip WAIT_AFTER + DONTNEED, chunk_bytes lowered to floor)",
WAIT_AFTER_TIMEOUT.as_secs(),
prev_off,
prev_len
@@ -198,12 +261,12 @@ impl WritebackPipeline {
}
}
}
self.pending = Some((chunk_off as u64, chunk_len as u64));
self.pending = Some((chunk_off, chunk_len));
self.last_flush_pos = pos;
self.chunk_count += 1;
tracing::trace!(
target: "mux",
"WritebackPipeline chunk off={} len={} sync_file_range_ms={wait_ms} fadvise_ms={fadvise_ms} chunk_bytes={} skip_wait={}",
"WritebackPipeline chunk off={} len={} wait_after_ms={wait_ms} fadvise_ms={fadvise_ms} chunk_bytes={} skip_wait={}",
chunk_off,
chunk_len,
self.chunk_bytes,
@@ -231,10 +294,14 @@ impl WritebackPipeline {
if self.wait_after_window.len() < ADAPTIVE_WINDOW {
return;
}
// p95 of 16 samples ≈ sorted[14] (5 % of 16 = 0.8 ≈ 1 above).
// p95 index, derived from the window size so it stays valid if
// ADAPTIVE_WINDOW changes (a hard-coded `[14]` would panic OOB
// for a window <= 14). For the default 16 this is index 15
// (ceil(16 * 95 / 100) - 1 = 15), i.e. the top sample.
let mut sorted: Vec<u64> = self.wait_after_window.iter().copied().collect();
sorted.sort_unstable();
let p95 = sorted[14];
let p95_idx = (ADAPTIVE_WINDOW * 95).div_ceil(100).min(ADAPTIVE_WINDOW) - 1;
let p95 = sorted[p95_idx];
let old = self.chunk_bytes;
let new = if p95 > ADAPTIVE_GROW_MS && self.chunk_bytes < CHUNK_BYTES_MAX {
(self.chunk_bytes * 2).min(CHUNK_BYTES_MAX)
@@ -278,7 +345,7 @@ impl WritebackPipeline {
// paths.
return;
}
match wait_after_with_timeout(self.fd, prev_off, prev_len) {
match wait_after_with_timeout(self.clone_for_worker(), self.fd, prev_off, prev_len) {
Some(_ms) => unsafe {
libc::posix_fadvise(
self.fd,
@@ -321,16 +388,56 @@ fn detect_nfs(fd: RawFd) -> bool {
/// Run `sync_file_range(WAIT_AFTER)` on a worker thread and wait up
/// to [`WAIT_AFTER_TIMEOUT`] for it to return. `Some(elapsed_ms)` on
/// success; `None` on timeout. On timeout the worker thread is
/// 0.20.6 generalizes the worker-thread + recv_timeout pattern into
/// [`crate::io::bounded::bounded_syscall`]; this helper now just adapts
/// the generic primitive to the WAIT_AFTER call shape (returns elapsed_ms
/// instead of the syscall's `()` return, treats `WorkerLost` as a benign
/// no-op to match the original semantics).
fn wait_after_with_timeout(fd: RawFd, off: u64, len: u64) -> Option<u64> {
/// intentionally leaked — it unwinds whenever the syscall eventually
/// returns or the process exits.
///
/// This delegates to [`crate::io::bounded::bounded_syscall`], the
/// generic worker-thread + `recv_timeout` primitive, and just adapts it
/// to the WAIT_AFTER call shape: it returns `elapsed_ms` instead of the
/// syscall's `()`, and treats `WorkerLost` as a benign no-op to match
/// the original semantics.
///
/// ## fd lifetime / fd-reuse safety
///
/// `worker_file` is an *owned* `File` (produced by `File::try_clone` at
/// pipeline construction). It is moved into the worker closure so the
/// file description stays alive for exactly as long as the worker thread
/// lives — even if the original `WritebackFile` is closed and the OS
/// reuses its fd number before the worker's syscall returns.
///
/// `fallback_fd` is used only when `worker_file` is `None` (i.e. the
/// `try_clone` at construction failed). In that case the worker captures
/// the raw fd integer, which carries the original fd-reuse risk but is
/// no worse than the pre-fix behaviour.
fn wait_after_with_timeout(
worker_file: Option<File>,
fallback_fd: RawFd,
off: u64,
len: u64,
) -> Option<u64> {
let started = Instant::now();
match crate::io::bounded::bounded_syscall(None, WAIT_AFTER_TIMEOUT, move || unsafe {
let result = if let Some(owned) = worker_file {
// Happy path: the closure owns a cloned File that keeps the
// file description alive until the worker drops it.
crate::io::bounded::bounded_syscall(None, WAIT_AFTER_TIMEOUT, move || unsafe {
let fd = owned.as_raw_fd();
libc::sync_file_range(fd, off as i64, len as i64, libc::SYNC_FILE_RANGE_WAIT_AFTER);
}) {
// `owned` drops here, closing the cloned fd.
})
} else {
// Fallback: try_clone failed at construction; use the raw fd.
// This carries the pre-fix fd-reuse risk on timeout, but is no
// regression from the original behaviour.
crate::io::bounded::bounded_syscall(None, WAIT_AFTER_TIMEOUT, move || unsafe {
libc::sync_file_range(
fallback_fd,
off as i64,
len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
})
};
match result {
Ok(()) => Some(started.elapsed().as_millis() as u64),
Err(crate::io::bounded::BoundedError::Timeout)
| Err(crate::io::bounded::BoundedError::Halted) => None,
@@ -456,4 +563,64 @@ mod tests {
assert_eq!(p.chunk_count, before);
assert!(p.pending.is_none());
}
// ── Bug-fix regression tests ────────────────────────────────────────
/// Regression for the fd-reuse / use-after-close fix. Verifies that
/// `WritebackPipeline::new` successfully clones the fd into
/// `wait_file` (i.e. `try_clone` doesn't fail for a normal
/// tempfile) and that `clone_for_worker` returns `Some` — meaning
/// the WAIT_AFTER worker will capture an owned `File` rather than a
/// raw fd integer.
///
/// A deterministic test for the actual fd-reuse race is not clean to
/// write (it would require simultaneously closing the original File
/// and re-opening a new one to steal the fd number while the worker
/// is mid-syscall, which is inherently racy). This test instead pins
/// the structural invariant: on a normal local file, the pipeline
/// holds a valid clone and will give the worker an owned File.
#[test]
fn wait_file_clone_is_present_for_local_tempfile() {
let (_f, p) = local_pipeline(32 * 1024 * 1024);
assert!(
p.wait_file.is_some(),
"wait_file must be Some for a normal local tempfile (try_clone should not fail)"
);
// clone_for_worker must return Some — the worker will get an
// owned File, not fall through to the raw-fd fallback.
let worker_clone = p.clone_for_worker();
assert!(
worker_clone.is_some(),
"clone_for_worker must return Some when wait_file is Some"
);
}
/// Structural: the worker `File` clone returned by `clone_for_worker`
/// is a distinct file descriptor (different fd number) that refers to
/// the same underlying file. Closing the original tempfile must not
/// affect the clone's validity — the OS keeps the file description
/// alive until all file descriptors referring to it are closed.
///
/// We verify "distinct fd number" and "still usable as a raw fd"
/// without actually racing a syscall.
#[test]
fn worker_clone_has_distinct_fd_from_original() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
let pipeline = WritebackPipeline::new(f.as_file(), 0, 32 * 1024 * 1024);
let clone = pipeline
.clone_for_worker()
.expect("clone_for_worker returned None");
let clone_fd = clone.as_raw_fd();
// The clone must have a different fd number — it is a separate
// open file description (dup'd by try_clone).
assert_ne!(
clone_fd, original_fd,
"worker clone must have a distinct fd number from the original"
);
// The clone fd must be valid (non-negative on Unix).
assert!(clone_fd >= 0, "clone fd must be non-negative");
}
}
+93 -14
View File
@@ -19,14 +19,10 @@ use std::time::Duration;
pub(super) fn preallocate(file: &File, size_bytes: u64) {
// FALLOC_FL_KEEP_SIZE = 0x01 — keep the reported file size at 0
// (writes grow it normally) while still pre-reserving the extents.
let rc = unsafe {
libc::fallocate(
file.as_raw_fd(),
libc::FALLOC_FL_KEEP_SIZE,
0,
size_bytes as i64,
)
};
// Clamp to the signed `off_t` range; an unchecked `as i64` cast
// would wrap a >= 2^63 size to a negative length (EINVAL no-op).
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX);
let rc = unsafe { libc::fallocate(file.as_raw_fd(), libc::FALLOC_FL_KEEP_SIZE, 0, len) };
tracing::debug!(
target: "mux",
"WritebackFile fallocate size_hint={size_bytes} rc={rc} ok={}",
@@ -34,17 +30,46 @@ pub(super) fn preallocate(file: &File, size_bytes: u64) {
);
}
/// Run `fsync` on `file` with a 60 s deadline. On timeout we log loudly
/// and return `Ok(())` — the kernel will still flush on close, so the
/// data is best-effort durable; the alternative (trap the thread for
/// the rest of the rip) defeats `/api/stop`.
/// Run `fsync` on `file` with a 60 s deadline. On timeout — and
/// likewise on halt or a lost worker — we log and return `Ok(())`: the
/// kernel will still flush on close, so the data is best-effort durable.
/// The alternative (trap the thread for the rest of the rip, or return
/// an error that aborts an otherwise-complete mux) is worse, so all
/// three fallbacks return `Ok(())`. `Ok(())` from these paths is NOT a
/// durability barrier — the durable flush did not complete; only the
/// hang is bounded.
///
/// ## fd-reuse safety
///
/// The `fsync` runs on a bounded worker thread that may be leaked on
/// timeout. To avoid the leaked worker's syscall hitting a recycled fd
/// number after the original `File` is closed, we `try_clone` an owned
/// `File` and move it into the closure. The clone keeps the underlying
/// file description alive for as long as the worker thread lives.
/// On `try_clone` failure (rare) we fall back to the raw fd integer —
/// no worse than the previous behaviour.
pub(super) fn durable_sync(file: &File) -> io::Result<()> {
// Clone so a leaked worker thread retains a valid fd even after the
// original File is closed and its fd number is reused.
let owned = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
let fd = file.as_raw_fd();
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fd={fd}: try_clone failed ({e}), fsync worker will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
let fallback_fd = file.as_raw_fd();
match crate::io::bounded::bounded_syscall(
None,
Duration::from_secs(60),
move || -> io::Result<()> {
let fd = owned.as_ref().map(|f| f.as_raw_fd()).unwrap_or(fallback_fd);
let rc = unsafe { libc::fsync(fd) };
// `owned` (if Some) drops here, releasing the cloned fd.
if rc == 0 {
Ok(())
} else {
@@ -60,7 +85,61 @@ pub(super) fn durable_sync(file: &File) -> io::Result<()> {
);
Ok(())
}
Err(crate::io::bounded::BoundedError::Halted) => Ok(()),
Err(crate::io::bounded::BoundedError::WorkerLost) => Ok(()),
Err(crate::io::bounded::BoundedError::Halted) => {
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fsync skipped (halt requested); data not durably flushed, kernel will flush on close"
);
Ok(())
}
Err(crate::io::bounded::BoundedError::WorkerLost) => {
tracing::error!(
target: "mux",
"WritebackFile::sync_all fsync worker lost before completion; data not durably flushed, kernel will flush on close"
);
Ok(())
}
}
}
#[cfg(test)]
#[cfg(target_os = "linux")]
mod tests {
use super::*;
use tempfile::NamedTempFile;
/// Regression for the fd-reuse / use-after-close fix in `durable_sync`.
///
/// Verifies the structural invariant: `try_clone` succeeds for a normal
/// local tempfile, and the cloned `File` has a distinct fd number from
/// the original. This pins the property that a leaked fsync worker thread
/// captures an owned `File` (and thus keeps the file description alive)
/// rather than a bare fd integer that can be reused after the original
/// `File` closes.
///
/// The actual fd-reuse race is non-deterministic and not cleanly
/// testable without coordinating a simultaneous close + re-open on
/// another thread. A structural test is the accepted substitute.
#[test]
fn durable_sync_worker_uses_owned_clone_with_distinct_fd() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
// try_clone must succeed for a normal local file.
let owned = f
.as_file()
.try_clone()
.expect("try_clone must succeed for a local tempfile");
let clone_fd = owned.as_raw_fd();
// The clone must be a distinct fd (dup'd, not aliased).
assert_ne!(
clone_fd, original_fd,
"owned clone must have a distinct fd number — not an alias of the original"
);
assert!(clone_fd >= 0, "clone fd must be a valid non-negative fd");
// durable_sync must complete without error on the local tempfile.
durable_sync(f.as_file()).expect("durable_sync must return Ok on a local tempfile");
}
}
+77 -5
View File
@@ -1,10 +1,12 @@
//! macOS platform impl for [`super::WritebackFile`].
//!
//! - `preallocate`: `fcntl(F_PREALLOCATE)` — macOS's fallocate-equiv.
//! Reserves a contiguous extent when possible, falling back to a
//! non-contiguous reservation if the FS can't satisfy it. Reported
//! file size is unchanged (`F_ALLOCATEALL` is not set, so allocation
//! is "best effort up to length"; growth happens via writes).
//! First attempt requests `F_ALLOCATECONTIG | F_ALLOCATEALL` (prefer a
//! contiguous run but accept scattered extents to satisfy the full
//! length), falling back to `F_ALLOCATEALL` alone on failure.
//! `F_PREALLOCATE` never advances EOF regardless of the flags — only
//! `ftruncate`/writes grow the file — so the reported file size is
//! unchanged; `F_ALLOCATEALL` governs the contiguity fallback, not size.
//! - `durable_sync`: `fcntl(F_FULLFSYNC)` wrapped in
//! [`crate::io::bounded::bounded_syscall`] with a 60 s deadline.
//! F_FULLFSYNC is HFS+/APFS's true-fsync (flushes the disk's own
@@ -25,11 +27,14 @@ use crate::io::platform_macos::{
const F_FULLFSYNC: libc::c_int = 51;
pub(super) fn preallocate(file: &File, size_bytes: u64) {
// Clamp to the signed `off_t` range; an unchecked `as off_t` cast
// would wrap a >= 2^63 size to a negative length.
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX) as libc::off_t;
let mut fst = Fstore {
fst_flags: F_ALLOCATECONTIG | F_ALLOCATEALL,
fst_posmode: F_PEOFPOSMODE,
fst_offset: 0,
fst_length: size_bytes as libc::off_t,
fst_length: len,
fst_bytesalloc: 0,
};
// First attempt: contiguous.
@@ -47,28 +52,54 @@ pub(super) fn preallocate(file: &File, size_bytes: u64) {
);
}
/// ## fd-reuse safety
///
/// The F_FULLFSYNC / fsync runs on a bounded worker thread that may be
/// leaked on timeout. To avoid the leaked worker's syscall hitting a
/// recycled fd number after the original `File` is closed, we
/// `try_clone` an owned `File` and move it into the closure. The clone
/// keeps the underlying file description alive for as long as the worker
/// thread lives. On `try_clone` failure (rare) we fall back to the raw
/// fd integer — no worse than the previous behaviour.
pub(super) fn durable_sync(file: &File) -> io::Result<()> {
// Clone so a leaked worker thread retains a valid fd even after the
// original File is closed and its fd number is reused.
let owned = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
let fd = file.as_raw_fd();
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fd={fd}: try_clone failed ({e}), F_FULLFSYNC worker will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
let fallback_fd = file.as_raw_fd();
match crate::io::bounded::bounded_syscall(
None,
Duration::from_secs(60),
move || -> io::Result<()> {
let fd = owned.as_ref().map(|f| f.as_raw_fd()).unwrap_or(fallback_fd);
// Try F_FULLFSYNC first. If it isn't supported on this
// filesystem (older HFS, some network mounts) fall back to
// plain fsync — better than nothing.
let rc = unsafe { libc::fcntl(fd, F_FULLFSYNC, 0) };
if rc == 0 {
// `owned` (if Some) drops here, releasing the cloned fd.
return Ok(());
}
let err = io::Error::last_os_error();
if err.raw_os_error() == Some(libc::ENOTSUP) {
let rc = unsafe { libc::fsync(fd) };
// `owned` drops here.
if rc == 0 {
Ok(())
} else {
Err(io::Error::last_os_error())
}
} else {
// `owned` drops here.
Err(err)
}
},
@@ -85,3 +116,44 @@ pub(super) fn durable_sync(file: &File) -> io::Result<()> {
Err(crate::io::bounded::BoundedError::WorkerLost) => Ok(()),
}
}
#[cfg(test)]
#[cfg(target_os = "macos")]
mod tests {
use super::*;
use tempfile::NamedTempFile;
/// Regression for the fd-reuse / use-after-close fix in `durable_sync`.
///
/// Verifies the structural invariant: `try_clone` succeeds for a normal
/// local tempfile, and the cloned `File` has a distinct fd number from
/// the original. This pins the property that a leaked F_FULLFSYNC/fsync
/// worker thread captures an owned `File` (keeping the file description
/// alive) rather than a bare fd integer that can be reused after the
/// original `File` closes.
///
/// The actual fd-reuse race is non-deterministic; a structural test is
/// the accepted substitute.
#[test]
fn durable_sync_worker_uses_owned_clone_with_distinct_fd() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
// try_clone must succeed for a normal local file.
let owned = f
.as_file()
.try_clone()
.expect("try_clone must succeed for a local tempfile");
let clone_fd = owned.as_raw_fd();
// The clone must be a distinct fd (dup'd, not aliased).
assert_ne!(
clone_fd, original_fd,
"owned clone must have a distinct fd number — not an alias of the original"
);
assert!(clone_fd >= 0, "clone fd must be a valid non-negative fd");
// durable_sync must complete without error on the local tempfile.
durable_sync(f.as_file()).expect("durable_sync must return Ok on a local tempfile");
}
}

Some files were not shown because too many files have changed in this diff Show More