Author SHA1 Message Date
matthew 125a8e5bf0 aacs: full-ladder resolve_candidate + correct mk_from_dk
leak-guard / leak-guard (push) Successful in 24s
resolve_candidate(candidate, mkb, unit_key_ro, vid) -> ResolvedChain: one
composed, PURE-DERIVATION boil-down for a candidate key at any ladder rung
(KeyCandidate::{Uk,Vuk,Mk,Pk,Dk}, each carrying the module's newtype). Walks
DK/PK/MK/VUK -> terminal unit keys, parsing Unit_Key_RO.inf at the version
the disc's MKB declares, and returns every CPS unit key as (cps_unit, key)
(matching ResolvedKeys) plus the intermediate chain. No sampling/validation
(that's the caller's unit_key_validates) and no position recovery (Dk is a
positioned DeviceKey; recover_dk_position first). Adds ProcessingKey newtype.
Consumers stop re-composing the ladder; every client hardens one impl.

Correct mk_from_dk: real Subset-Difference walk (derive_media_key_and_pk_from_dk)
instead of the Media-Key-Variant path, which needed integrator KCD absent
in-tree and Err'd for every real disc -- dead for both consumers. Drops the
now-unused vid arg.
2026-07-01 22:34:52 -07:00
Matthew Jackson 741c1ea11d restore freemkv-unlock path dep for local dev (post-v1.2.1) 2026-07-01 19:08:56 -07:00
Matthew Jackson e713b26b87 v1.2.1: bump version (freemkv-unlock git-pinned for the tag) 2026-07-01 19:08:53 -07:00
Matthew Jackson fcbd667add changelog: describe the shipped re-base DTS fix + faithful-passthrough note 2026-07-01 19:02:04 -07:00
Matthew Jackson 88c03152e2 mux/dts: re-base PTS per PES instead of a running clock (fix long-title drift)
The first cut used a global running clock (max(next, own-PES PTS) + advance),
which fixed the same-PES collision but DRIFTED: once accumulated frame
durations exceeded the PES-timestamp spacing, it never re-based, so a
feature-long DVD DTS track ran minutes past its real length (2h44 for a 2h03
film) while AC-3 from the same source stayed exact.

Match the AC-3 path: re-base to each PES's own container timestamp, and advance
by one frame duration ONLY within a run of AUs sharing one PES. Fixes the DVD
multi-frame-per-PES collision without drift; the UHD DTS-HD MA per-PES
attribution (da85f56) is preserved (each AU still takes its own core PES's PTS).
Adds new_pes_rebases_to_its_own_pts_no_drift; full mux suite green (905).
2026-07-01 18:21:25 -07:00
Matthew Jackson c8e7ad5e56 v1.2.1: bump version + changelog (DVD DTS timestamp fix) 2026-07-01 17:30:25 -07:00
Matthew Jackson f122f08628 mux/dts: monotonic per-frame PTS + real frame duration (fixes DVD DTS)
DVD packs several DTS core frames into one PES; the parser stamped every
access unit with that single PES PTS and duration_ns=None, so consecutive
frames collided on one timestamp — ffmpeg rejected the output as 'non
monotonically increasing dts to muxer: X >= X' (deep-decode = corrupt,
e.g. The Punisher). The UHD DTS-HD MA path (one AU per PES, distinct PTS)
was unaffected, which is why this only surfaced on DVD.

Parse the DTS core header for samples ((NBLKS+1)*32) and sample rate
(SFREQ, 48kHz fallback) to derive each AU's duration, and stamp a running
monotonic PTS: max(next_clock, own-core-PES PTS), then advance by the
frame duration. A later PES whose PTS is ahead of the clock still wins
(preserves the UHD per-PES attribution from da85f56/c49a180); frames
sharing one PES advance frame-by-frame instead of colliding.

Tests: the 3 that encoded 'same PES -> same PTS' now assert monotonic
advance; new dvd_many_cores_one_pes_are_strictly_monotonic reproduces the
Punisher bug; duration/SFREQ-fallback unit tests added.
2026-07-01 17:17:35 -07:00
Matthew Jackson b2c3540989 restore freemkv-unlock path dep for local dev (post-v1.2.0) 2026-07-01 16:53:06 -07:00
Matthew Jackson 3a4307def6 v1.2.0: bump version (freemkv-unlock git-pinned for the tag) 2026-07-01 16:53:03 -07:00
Matthew Jackson 917026d566 changelog: DVD DTS/LPCM silent-audio fix + Pass-N recovery specialists + flat scheduler 2026-07-01 16:50:08 -07:00
Matthew Jackson 6b0bcbb43f recovery: flat-pool bandit scheduler (FREEMKV_PATCH_FLAT, opt-in)
The tier ladder (bucket->mug->thimble) forces tier 0 to attempt ALL
ranges before tier 1, and tier 1 all before tier 2 — so on a hardened
residual (late resume, flood already gone) the marginal specialists are
unreachable for hours: you bail near-empty buckets over hundreds of dead
ranges before a thimble ever touches water.

FREEMKV_PATCH_FLAT collapses the 3 tiers into ONE flat pool of all 16
handler configs. run_handlers already sorts best-first by the live
decayed-yield scorecard, so this is a data-driven bandit: the first
ranges try them all (explore), the ranking floats whatever is actually
landing sectors to the front (exploit), re-measured per range; a handler
that doesn't fit stays last but is never dropped (floor -> can revive).
The new handlers self-limit (yield after 4 dead reads), so the flat
scheduler does NOT reintroduce the old depth-first per-range starvation.

Opt-in; unset keeps the proven tier ladder as the default for 1.2.0.
2026-07-01 16:44:29 -07:00
Matthew Jackson 516ff4e581 test: widen profile_03 efficiency guard for tier-2 roster
The tier-2 marginal specialists now additionally probe permanently-bad
residual sectors before leaving them NonTrimmed, so the alternating-bad
profile's finite read count rose past the old tier-0/1 bound. Guard
still catches runaway; behavior asserts unchanged.
2026-07-01 13:55:20 -07:00
Matthew Jackson c27009d443 recovery: wire tier 2 marginal specialists into the Pass-N chain
Add a third breadth-first tier (PATCH_TIERS 2->3) that runs the marginal
specialists on the hardened residual tiers 0-1 leave: SlowSpin (Linear
fwd+rev @ min), FuaRetry (Linear fwd+rev+Bisect @ FUA), SlowFua (Linear @
min+FUA), CachePrime, Oscillate (@ max and @ min), SpeedSweep. Every read is
a wedge-safe read_span, so they inherit wedge-abort / unproductive-yield /
deadline for free. All are new configs, so the EWMA scorecard calibrates each
once then ranks by decayed rate — a specialist that doesn't fit self-
deprioritises. Tiers 0-1 (fast scouts, slow-deep) are unchanged; this is
purely additive. Also switch the scorecard log sort to sort_by_key.

cargo test -p libfreemkv green (2200 passed).
2026-07-01 13:50:35 -07:00
Matthew Jackson 789d988314 recovery: CachePrime + Oscillate handlers + fixtures
CachePrime reads the good run immediately preceding a residual island to lock
the servo/PLL, then reads the island warm (boundary sectors the drive can't
cold-seek). Oscillate reads each residual sector by alternating approach —
forward-into then reverse-into — for direction-dependent tracking. Both go
through the wedge-safe read_span (primes included). FakeDisc models a
direction-dependent sector and a servo-primed boundary sector; fixtures prove
a forward/cold Linear misses each while Oscillate/CachePrime recover them.
2026-07-01 13:50:35 -07:00
Matthew Jackson 1e0dc4c514 recovery: FuaRetry + SlowFua fixtures (FUA / min+FUA levers)
FuaRetry (Linear fwd+rev+Bisect at {max,fua,deep}) and SlowFua (Linear at
{min,fua,deep}) are Linear/Bisect at FUA params, no new struct. FakeDisc now
models a stochastic sector that lands only on the Nth physical (FUA) read (a
cached re-read keeps missing) and a hardest sector needing BOTH min speed and
FUA. Fixtures prove cached reads keep missing while the FUA group lands the
stochastic sector, and that neither lever alone recovers the hardest sector —
only the min+FUA combination does.
2026-07-01 13:50:35 -07:00
Matthew Jackson 629ed32e9e recovery: SpeedSweep handler + slow-spindle fixtures (SlowSpin/SpeedSweep)
SlowSpin is Linear pinned to min speed (no new struct). SpeedSweep is a new
per-sector handler that tries Max->Min until one reads (speed resonance).
FakeDisc now models a sector that reads ONLY at min speed; fixtures prove a
max-speed Linear misses it while SlowSpin and SpeedSweep recover it (and that
SpeedSweep tries fast-then-slow).
2026-07-01 13:50:35 -07:00
Matthew Jackson ac3b3fcfa4 recovery: decayed (EWMA) handler scorecard so the ranking flips
Grade handlers by an exponentially-decayed recovery rate (bytes/second,
alpha=0.5) instead of a cumulative rate. The residual hardens mid-pass, so
the best technique changes: cumulative froze the early winner in the lead
forever; the EWMA re-prices continuously — a handler that stops earning
decays down, a late-starting specialist climbs. Keeps rank()'s untried->top
one-shot calibration and attempted-but-zero-time->bottom.

New flip fixture proves a handler that recovers a lot early then nothing
loses its lead to one that starts recovering later.
2026-07-01 13:50:35 -07:00
Matthew Jackson 0c8153304e recovery: parameterize the read primitive with ReadParams (speed/FUA/timeout)
Add ReadParams { speed: SpeedPref, fua: bool, timeout: TimeoutPref } and
thread it through read_span so every wedge-safe handler read can request a
spindle speed (SET CD SPEED issued only on change, restored to max when the
handler exits), set the READ(10) FUA bit, and pick the 10s vs 60s timeout.

- SectorSource gains read_sectors_fua (default ignores fua); Drive sets the
  CDB bit, DecryptingSectorSource threads fua to its inner read.
- recovery_read gains a fua param.
- Linear becomes { direction, params }; Bisect/Jump take params. Existing
  tier-0/1 instances keep identical behavior (max speed, no FUA, fast/deep).
- Scoreboard keys on the full-config String name (linear:fwd:max:fast, ...).
- FakeDisc observes speed + FUA + approach so specialist techniques are
  provably exercised in later commits.

cargo test -p libfreemkv green (2193 passed).
2026-07-01 13:50:35 -07:00
Matthew Jackson e94319c099 dvd: route private_stream_1 audio sub-ids by positional stream number
The private_stream_1 sub-stream id low nibble is the DVD audio-stream
NUMBER (0-7), shared across codecs, not a per-codec ordinal. A DTS (or
LPCM) track that isn't the disc's first audio stream got a sub-id one
too low, so the demux routing key (0xBD00 | sub_id) never matched and
every packet was dropped -> present-but-silent track. AC-3 at position
0 coincidentally worked (ordinal==position). Route by positional index.

Fixes silent DTS 5.0 track on The Punisher (AC-3 5.1 @ pos0 plays,
DTS 5.0 @ pos1 was 0xBD88, real wire id 0xBD89).
2026-07-01 13:49:55 -07:00
Matthew Jackson a93da78621 lib: remove standalone sector-health verify feature
Delete src/verify.rs (standalone 'read every sector, classify health'
module) + its lib.rs pub mod/pub use. It was old, unused code. The
decrypt-verify gate (disc::verify / UnitVerifier) is a DIFFERENT module and
stays. Relocated the pure chapter_at_offset helper (used by locate_ranges +
autorip's done-card) into disc/mod.rs as a pub fn.
2026-07-01 13:09:21 -07:00
Matthew Jackson cf13838f12 recovery: audit fixes — live at-risk excludes NonTried, latency-gated wedge, drop dead block counters
- Live located drilldown (sweep + patch progress snapshots) now excludes
  NonTried, matching the one-shot path. Including the unread remainder made
  main_at_risk_ms show ~full-movie at sweep start and melt to 0 as it
  progressed — unread is unknown, not damage.
- Wedge abort now requires the failure to be FAST (< WEDGE_FASTFAIL_MS,
  500ms) as well as wedge-family sense: a real uncorrectable sector on
  Hardware-error media spends ECC-recovery time before failing, so it no
  longer false-trips the fast-fail wedge abort. New regression test.
- Removed the always-zero blocks_attempted/read_ok/read_failed/
  unreadable_count from PatchOutcome + PatchLoopState (dead residue from
  the old grind loop; the HandlerScoreboard supersedes them) so the
  patch.done log no longer emits misleading zeros.
2026-07-01 12:07:08 -07:00
Matthew Jackson 9519628954 docs: 1.2.0 changelog — Pass-N handler chain, DVD drive-prep unlock fix, unlocker matrix 2026-07-01 11:05:31 -07:00
Matthew Jackson 21f8cec419 disc: unlocker_matrix reports 'did work this rip', not 'matched'
yes now means the unlocker actually ran and did its job: LibreDrive from
the runtime firmware-unlock success, AACS host-cert only when LibreDrive
didn't do the bus (stock-drive fallback), CSS from the crack succeeding.
On a LibreDrive UHD that correctly reads LibreDrive: yes, AACS: no (LD
removed the bus, the cert route never ran) instead of the misleading
AACS: yes. Names stay registry-driven; runtime logic lives here.
2026-07-01 10:44:39 -07:00
Matthew Jackson d5d72c4e22 drive: disable FUA on READ(10) — the ~10x sequential-read regression
Forcing FUA (Force Unit Access) on EVERY READ(10) bypassed the drive's
readahead/streaming cache on the bulk Pass-1 sweep, collapsing sequential
throughput ~10x (UHD 15-25 → ~2 MB/s, DVD → ~0.5 MB/s), disc-type-
agnostic — the cache IS the streaming throughput. This was the real speed
regression (not unlock/riplock, which the earlier chase suspected). Clear
byte-1 bit 0x08. FUA will return as a dedicated Pass-N recovery handler
that sets/clears it per marginal-sector re-read, where cache-masking of a
stochastic sector actually matters — never blanket on the bulk path (#55).
2026-07-01 10:26:56 -07:00
Matthew Jackson c781fb7193 disc: Disc::unlocker_matrix() — registry-driven unlocker report
Returns each registered unlocker's name + whether it applies to this
drive+disc (via the unlock bridge over all_unlockers()). Disc crypto kind
derived in the library so the CLI and autorip render an identical, always-
current report with no hardcoded names. Also adds a Drive::unlocker_name()
runtime getter.
2026-07-01 10:00:59 -07:00
Matthew Jackson 541ca2139f drive: run drive-prep unlock for EVERY disc, incl. DVD (fix riplock regression)
The v1.0.0-rc.1 unlocker refactor added an `if disc_is_dvd() { return }`
early-return in Drive::init() that skipped the whole drive-prep unlock for
DVDs. That firmware unlock is what lifts riplock and readies max read
speed — a DRIVE-level, disc-independent feature — so skipping it left
every DVD stuck at stock/riplock speed (~0.4x, 3h ETA). UHD was unaffected
because it flows through the unlock.

Remove the skip: init() now runs the identity-keyed drive unlocker for all
discs (disc kind is Unknown at init, so only the drive unlocker matches;
the AACS host-cert handshake and CSS bus-auth still run later, gated on the
real disc kind, on top of the unlocked drive). Speed stays where it
belongs — SET CD SPEED(0xFFFF) at pass-1 start (disc/mod.rs) — not in the
unlocker. Also drop the matching probe_disc DVD skip.

Reverts the SET STREAMING stopgap (separate freemkv-unlock revert): that
treated the symptom; the real bug was the skipped unlock. Drive features
come from the firmware unlock, not a stock speed CDB.
2026-07-01 09:38:51 -07:00
Matthew Jackson 2dd98c3e32 recovery: fresh-eyes audit fixes (handlers + Pass-N engine + sweep)
Handlers (section_recover.rs):
- Bisect expand loops now honor ctx.halted() (were deadline-only, so a
  Stop could hang up to 60s vacuuming a readable island).
- read_span: explicit Transport arm so a bus-abort read isn't counted as
  unproductive grinding; debug_assert the sector-aligned span invariant.
- Scoreboard rank: an attempted-but-zero-time handler (e.g. returned Halted
  on its first check) now ranks BOTTOM, not top — it no longer crowds out
  proven performers.
- Document the wedge tier-size coupling + new regression test that a
  2-handler (tier-1) chain still catches a wedge via cross-section streak.

Pass-N engine (patch.rs):
- Rebuild PatchOutcome stats AFTER the post-read re-verify downgrade flush
  (was snapshotting before it, over-reporting bytes_good / recovered and
  risking a 'perfect rip' verdict on an imperfect one).
- Progress 'recovered' composes the still-bad set to MATCH work_total
  (subtract NonTried, add Unreadable) so the bar can't pin at 0 on a
  partially-swept disc or run backward on the Unreadable→NonTrimmed relabel.
- Remove dead work_done field; rewrite the stale 'adaptive batching' comment
  to describe the handler chain and mark block_sectors/full_recovery as
  informational-only.

Sweep (disc/mod.rs):
- Saturating arithmetic at the damage-jump position math (honor the
  read_error side's documented defence-in-depth guarantee).

Deferred (noted, need focused passes): fast_capture re-introduction,
Pass-1 halt-misclassified-as-jump, bytes_good display inflation, the
always-zero blocks_* telemetry, Pass-1 jump-on-first-error policy.
2026-07-01 09:14:32 -07:00
Matthew Jackson ceaa1da369 drive: apply drive-features for every disc, incl. DVD (fix riplock)
DVD ran the drive at ~0.4x stock speed: init() early-returned for DVD
before any speed was set, and its only speed command was the sweep's lone
legacy SET CD SPEED, which this BU40N ignores for DVD. BD/UHD got up to
speed only via the firmware unlocker's calibration — a path a DVD can't
take (it breaks stock CSS).

Call the new unlock-crate drive-features capability at init() for ALL disc
kinds, before the DVD stock-mode early-return, via a bridge shim. Stock
MMC only (SET STREAMING + SET CD SPEED), no bus unlock, so CSS is
undisturbed.
2026-07-01 08:56:19 -07:00
Matthew Jackson fe14a2d5e5 Pass-N: detect drive fast-fail wedge and abort instead of grinding
The 2026-07-01 overnight rip ground a wedged BU40N for 28 minutes at
0 B/s. Root cause: the handler chain's read_span split errors only into
Transport (bus gone) vs Bad (everything else), so the drive's fast-fail
wedge sense (ILLEGAL REQUEST 0x05/0x24 — it rejects every CDB in <100ms
without attempting recovery) was treated as an ordinary bad sector. The
chain hopped to the next section forever, and the existing wedge detector
in read_error.rs never saw it because the chain reads through
recovery_read directly.

Add a pass-level wedge streak: read_span counts consecutive wedge-family
(Hardware/IllegalRequest) senses; at WEDGE_ABORT_STREAK (16) it escalates
the read to Transport, which every handler already propagates as
TransportFault — aborting the whole pass and setting wedged_exit so the
caller spin-cycles. The streak is carried across sections via PatchCtx
(seeded into and read back from each per-section HandlerCtx) so a wedge is
caught even when every bad sub-range is smaller than the streak. Any Good
or non-wedge (medium-error) read resets it, so scattered bad sectors on
real media never trip it. New fixture test: a wholly-wedged 1000-sector
section aborts in <100 reads, not 1000.
2026-07-01 08:02:52 -07:00
Matthew Jackson 8f6a92ccd4 Bisect: halve at dead boundary + exempt boundary probes from stall
The expand loops broke out of the batch sweep on the first failing read,
overshooting the good/dead boundary by up to a batch and leaving the
readable sectors adjacent to a dead edge for the re-bisect to re-pin. On
top of that, early-yield counted those boundary-probe failures as a
stall, so the forward expand quit early AND the backward expand inherited
a poisoned unproductive counter and yielded instantly.

Now on a failed expand batch we halve the step (down to a single sector)
to recover right up to the dead edge in ~log2(batch) reads, drive the
expand loops off the deadline only (they self-terminate via halving), and
clear the unproductive streak once an island is located. patch_recovers_
good_middle_of_a_bad_range now recovers 50/50 good-middle sectors.
2026-06-30 22:06:04 -07:00
Matthew Jackson 29f76aad68 section_recover: per-rip handler scorecard + Bisect-leads-scouts + proportional Jump
Scorecard: grade each handler by recovery rate (MB/s) per rip, order
best-first on later sections, log the ranking at pass end. Untried
handlers rank top so each is calibrated once before ranking narrows to
the winners. Ephemeral (reset per pass), no persistence.

Tier 0 scouts are now [Bisect, Jump, Linear-fast x2], scorecard-ordered.
Bisect leads: probing the MIDDLE of a range lands on a readable island in
one read where a linear scan grinds the dead front to reach it. Jump now
jumps to the middle of the REMAINING span (proportional) instead of a
fixed 8 MiB that leapt clean over small ranges and missed their readable
middles. Tier 1 is slow deep-recovery on the residue.
2026-06-30 21:26:35 -07:00
Matthew Jackson 840ba8390c section_recover: per-rip handler scorecard + Bisect leads the scouts
Scorecard (ephemeral, reset each pass, no persistence): grades every
handler by recovery rate (bytes/sec). run_handlers orders handlers
best-first by that rate; an untried handler ranks top so each is
calibrated once, then the ranking narrows to the winners. Logged at pass
end (phase=scorecard) so the operator sees which handler is pulling the
weight on this drive/disc and which is a dud.

Tier 0 scouts are now [Bisect, Jump, Linear-fast x2], scorecard-ordered.
Bisect leads because probing a range's MIDDLE lands on a readable island
in one read, where Jump (linear from the front, big skip) can grind the
dead front or overshoot a small range entirely. The scorecard confirms
or overturns that order with real per-disc data.
2026-06-30 21:25:39 -07:00
Matthew Jackson e803905265 section_recover: Jump starts at 8 MiB base to clear big dead spots faster
Each dead probe read costs the drive's full ~10s timeout, so a large dead
region took a dozen escalating probes. Starting the jump at 8 MiB (vs
1 MiB) clears it in a handful; a skipped span stays bad for Bisect to
reclaim readable islands, so an over-jump loses nothing.
2026-06-30 21:12:12 -07:00
Matthew Jackson 11d4c33477 patch: single Jump-scout tier 0, expand-Bisect, live in-handler progress
Tier 0 is now a single fast Jump scout: it streams the big readable
ranges back and skips dead runs in seconds, so the pass reaches every
section fast and converges to the small genuine-dead residue instead of
grinding three handlers x 60s on each dead fragment. Tier 1 (fast
mop-up + slow deep reads + Bisect) works only that residue.

Bisect now expands: on a good probe it reads outward forward and
backward in full batches until a read fails, recovering the whole
readable island in large reads; the two failing ends become smaller bad
sub-ranges it bisects again. One huge bad range becomes many precisely
located small dead clusters.

Progress heartbeat: HandlerCtx gains a throttled tick (250ms) called
from every read, pushing a fresh snapshot to the reporter DURING a
handler. The bar and speed now move continuously as recovery happens
instead of jumping once per section (the reason speed read 0 B/s and the
% looked frozen between range boundaries).
2026-06-30 20:57:36 -07:00
Matthew Jackson 8d775cd341 patch: add Jump handler (lead fast tier) + recovery-based progress %
Jump: on sustained batch failures skip ahead an escalating distance
(1 MiB doubling to 256 MiB) to find where readable data resumes, leaving
the skipped span for Bisect to pin — mirrors the Pass-1 damage-jump. It
leads the fast tier so a large dead run is skipped in seconds instead of
the linear sweeps grinding every dead batch (10 s each) first; on a
readable range it just streams it back. Recovers readable data buried
behind a big dead front (the 192 MB Dune range).

Progress %: report bytes RECOVERED (initial-bad minus still-pending)
instead of a per-range counter that only advanced on the final tier — so
the bar reflects the readable bulk recovered during tier 0 the instant it
lands, matching the 'MB remaining' number.
2026-06-30 20:40:22 -07:00
Matthew Jackson bc07011bcb patch: breadth-first two-tier recovery, largest ranges first
The per-range walk was depth-first: each bad range ran the full handler
chain (fast + slow deep-recovery + bisect) before the next range was
touched. So a handful of tiny dead fragments at one end of the disc
burned the whole pass and the big NonTrimmed ranges elsewhere — usually
sweep-jump over-marks that read straight back — were never attempted.

Now recovery runs in two breadth-first tiers over ALL sections:
- Tier 0 gives every section one fast full-batch attempt (fast reads
  only), largest ranges first, so the recoverable bulk of the disc comes
  back in the first minutes.
- Tier 1 deep-recovers only the residue tier 0 could not pull.
Per-section still-bad sets persist across tiers. Largest-first ordering
means a big readable region is reached before time is spent on tiny dead
fragments.

Linear no longer collapses a failed batch to count=1 single-sector reads
(live probing: a marginal sector recovers in a large read, not a lone
one) — a failed 32-batch stays 32 and is re-attempted at full size by the
next handler/pass; Bisect salvages readable islands.

Adds a handler-start trace line so the debug log shows which handler is
running and the hand-off to the next.
2026-06-30 20:25:48 -07:00
Matthew Jackson d65b776a8e patch: replace grind-until-wedge loop with bounded handler chain
Pass-N recovery is now a chain of time-bounded recovery handlers instead
of one monolithic per-range loop that could grind the front of a bad
range for up to 30 min, wedge the drive, and abort the whole pass.

A bad range is a SubRanges set; recovery is an ordered list of
SectionHandlers (Linear{reverse,fast} covering back/forward x fast/slow,
and Bisect). A coordinator runs each handler with a hard per-handler
deadline: a handler recovers what it can (removing it from the still-bad
set) and hands the rest to the next handler; whatever is still bad after
the chain becomes NonTrimmed residue and we move on to the next range.

Guarantees, now structural rather than bolted-on:
- never hangs: every handler is deadline-bounded; the loop always drains
  to recovered-or-residue.
- always moves on: a range that cannot be finished leaves residue and
  advances; only a genuine transport fault or user halt ends the pass.
- extensible: a new recovery idea is one SectionHandler impl added to the
  chain; a proven-ineffective one is removed. The engine never changes.

Removes ~1.9k lines of the old inner loop (watchdogs, skip escalation,
NOT_READY grind, wedge counters) and their tests. fast_capture is now
inert (the chain supersedes it); breadth-first ordering becomes a future
scheduler concern. New module: disc/section_recover.rs (8 fixture tests,
injectable clock — bounded/never-hang proven without touching a drive).
Two A/B tests updated to the chain's strictly-better recovery counts.
2026-06-30 19:55:37 -07:00
Matthew Jackson ff7f3028a5 drive: always set FUA on READ(10) to bypass the drive cache
Every Drive::read now forces the read from physical media instead of
letting the drive satisfy it from cache. A recovery tool must not trust
the cache: on a re-read of a marginal sector the BU40N can return a
cached miss (~4 ms) instead of giving the surface a fresh physical
attempt (~40-250 ms), masking a sector that would otherwise read on a
real media hit. FUA (READ(10) byte 1 bit 3) is validated on the
BU40N/Initio bridge.
2026-06-30 18:49:14 -07:00
Matthew Jackson 25a7b131b4 drive: Drive::spin_cycle() — soft un-wedge via START STOP UNIT (no eject)
Spin the disc down (START=0, LOEJ=0) then back up (START=1) to clear the
BU40N/Initio fast-fail wedge state a run of HARDWARE_ERROR reads leaves the drive
in — the non-eject power-cycle our notes say the wedge needs. The disc stays
loaded; we never eject (slot-loading — a human eject is a product failure for an
unattended service). Validated live 2026-07-01: took the drive from
failing-every-read back to reading at MB/s.
2026-06-30 17:54:51 -07:00
Matthew Jackson 0cab32a08a disc/patch: fast-capture reads fail-fast (no deep recovery)
A fast_capture pass defers every failed block to a granular pass anyway, so
spending the drive's 60s deep-recovery timeout on it here only freezes the
breadth-first sweep on a pure-bad cluster (~25s per 32-block, incl. NOT_READY
grind). Read with recovery=false in fast_capture so a bad block fails on the
short timeout (~10s) and the sweep steps on; the granular passes
(fast_capture=false) do the deep recovery on what's left. No recovery lost —
the block stays NonTrimmed for the granular retry.
2026-06-30 17:08:26 -07:00
Matthew Jackson 22f0f5eb6f docs: 1.2.0 changelog — fast_capture breadth-first patch recovery 2026-06-30 16:38:56 -07:00
Matthew Jackson 2013ef8c44 disc/patch: fast_capture mode — breadth-first recovery (#50)
A PatchOptions.fast_capture pass reads each bad range ONCE at the full batch and
leaves every FAILED block NonTrimmed for a later pass — no bisect, no per-sector
grind, no retry. This lets a first retry pass grab the readable blocks (the
sweep's good skip-ahead overshoot) of EVERY section quickly, before any single
section's slow per-sector recovery — instead of grinding section 1 to exhaustion
before even touching section 2. A later pass (fast_capture = false) does the
granular bisect/retry on what's left.

Load-bearing invariant (fixture test): NO data is dropped. A failed block becomes
NonTrimmed (pending, retried by a granular pass), NEVER Unreadable. The test pins
that the readable half of a range recovers, the bad half stays NonTrimmed (not
Unreadable), and the bad block is marked in ONE batch read with zero bisection.
Disc::copy's internal patch keeps fast_capture=false (single-call full recovery).
2026-06-30 16:37:59 -07:00
Matthew Jackson acaae3d0a4 docs: 1.2.0 changelog — PassProgress mapfile-free contract + located drilldown 2026-06-30 16:13:05 -07:00
Matthew Jackson 65c6835363 disc: add progress_snapshot_from_mapfile one-shot + locate_ranges test
The library now offers a one-shot PassProgress snapshot built from a mapfile on
disk + title, so a client gets the fully-rendered drilldown without parsing the
mapfile itself — used for autorip's pass-boundary and mux-entry paints (no live
callback yet at those points). Adds a locate_ranges test covering in-feature vs
out-of-feature at-risk time (ported from autorip's removed from_map tests).
2026-06-30 16:09:28 -07:00
Matthew Jackson 0f0c496a4d progress: PassProgress carries the located drilldown (emit side)
Add LocatedRange + LocatedProgress to the progress contract and a 'located'
field on PassProgress, populated by the sweep + patch emitters from the
in-memory bad-range set + title. Move the range->chapter/time annotation
(locate_ranges, range_chapter, byte_offset_in_title) into the library so a
client renders the disc map + at-risk movie time straight from PassProgress
and never reads the mapfile itself — if the mapfile becomes a mapdb, this type
and its producer change, clients don't.

PassProgress is no longer Copy (located carries a Vec); it's built once per
throttled emission and passed by reference. Non-locating phases (verify,
extract) emit LocatedProgress::default(). Adds consts::MILLIS_PER_SEC.
Consumer-side wiring (autorip drops Mapfile::load) follows.
2026-06-30 15:26:49 -07:00
Matthew Jackson 923b9edbf4 disc/patch: remove scatter-recovery (ruled out by live probing); add SubRanges
Live drive probing (cold-single vs in-stream batch vs centered window,
one clean uncached read per sector) showed recovery RATE is identical
across approach: 3/5 every mode. The drive's per-sector ECC is
media-bound and partly stochastic, NOT approach-bound. So scatter
(seek-away recalibration + far-anchor re-read) does not improve recovery
and is dead weight — removed (function, read_good_sectors, SCATTER_*
consts, and its tests).

Add SubRanges: the still-bad (pos,len) sub-ranges of a section with
remove()/split, the work-list foundation for the upcoming per-section
recovery decomposition (recover_section -> stream/bisect/retry phase
helpers). Unit-tested; unused until the decomposition lands.

25 patch + 8 AB profile tests green; precommit clean on Rust 1.86.
2026-06-30 13:56:49 -07:00
Matthew Jackson caf1b03fd4 disc/patch: keep MAX speed on recovery reads + geometric batch re-grow
Live probing on the BU40N/UHD testbed (rprobe/dprobe) settled the speed
question empirically: a marginal sector reads ~12x FASTER at MAX speed
than at slow, and slow NEVER recovered one MAX didn't. Cleanly-tested
(one uncached read per sector) the recovery RATE is identical across
speed/direction/window/batch — the drive's per-sector ECC is media-bound,
not approach-bound. The only host lever that matters is read SPEED.

So the old 'drop to 0x0000 on first failure and hold it for the whole
range' only slowed the GOOD sectors of a bad range — measured ~3x slower
overall. Now the range stays at MAX; the first failure just re-attempts
once (stochastic media) and falls through. Live A/B: 11 KB/s -> 33 KB/s,
bytes_lost unchanged.

Also: batch adaptation is now halve-on-failure / double-on-success
(geometric), so a bisected batch climbs back through clean runs from any
size (the old count==1-only upscale left it stuck at mid sizes). Removed
the dead consecutive_singles_ok counter and the inert batch-retry probe.

25 patch tests green; precommit clean on Rust 1.86.
2026-06-30 13:25:07 -07:00
Matthew Jackson 9cc422bd00 disc/patch: drop cache-priming, bisect batch failures instead of count=1
Two patch-recovery speed fixes, grounded in live rip1 telemetry on the
Dune UHD bad regions.

1. Pull prime_cache. In the reverse walk it issued throwaway reads of
   lba-3..lba-1 — i.e. straight into the not-yet-visited dead L-EC core —
   grinding each ~7-9s at the fast timeout, then discarding the result.
   The live trace showed a good target read (read_duration_ms=1) taking
   9s wall-clock purely from the prime grind ahead of it. The dead-zone
   skip (Tier 1) + scatter's recalibration read (read_good_sectors)
   cover the cache-warm intent without grinding the dead zone, so prime
   is redundant harm — removed (function + CACHE_PRIME_SECTORS + call).

2. Bisect on batch failure. handle_read_failure used to collapse a
   failed count>1 batch straight to count=1 and single-walk the entire
   batch. Now it halves (count/2) and retries the same start: a good
   half recovers in BULK, the bad sector is isolated in O(log n) reads
   instead of O(n). Invariant preserved — only a count==1 failure marks
   NonTrimmed, so no good sector is lumped into a bad mark.

These are Tier 2 (bisect) of the progressive recovery model
(fast-mass-skip -> bisect -> scatter); composable so order can be tuned.
25 lib + 8 AB fixture tests green; precommit clean on Rust 1.86.
2026-06-30 11:20:44 -07:00
Matthew Jackson 5b39a0af2b disc/patch: add scatter-recovery for stuck bad sectors
The 'reset, read good data, come back for one sector' technique. A
genuinely-damaged sector makes the drive grind its full recovery timeout
and still fail; re-reading consecutive bad LBAs at identical conditions
re-fails AND is the rapid-failure cadence that drops the BU40N into a
firmware fast-fail wedge (CLAUDE.md hard-rule #2).

scatter_recover: on a stuck single sector, make up to SCATTER_MAX_ATTEMPTS
fresh tries, each preceded by read_good_sectors() — a seek to a far
known-good anchor + a sustained read that re-seats the head/servo and
breaks the failure cadence. The fresh re-read uses the FAST timeout
(recovery=false) so a recalibrated marginal sector reads quick and a
truly-dead one fails fast instead of burning 60s per attempt. The seek +
good read IS the settle (matches ddrescue/MakeMKV) — no idle sleep.

Gated to genuine single-sector MEDIUM_ERROR (0x03): transport faults
still abort the pass, NOT_READY keeps its own retry path, wedge-family
senses are left for cooldown/eject. A scatter recovery is recorded
exactly like a normal read success (write-then-mark-Finished); nothing
marks a range good without a successful read + write.

Logs anchor_ms/reread_ms per attempt for live tuning. 5 fixture tests
(recover-marginal, give-up-dead, skip-non-medium, skip-batch, AACS
widen); 2 AB read-count guards raised for scatter's bounded extra reads.
Precommit green on Rust 1.86.
2026-06-30 10:29:55 -07:00
Matthew Jackson bd0a21bdb4 disc/patch: extract PatchCtx/patch_region orchestration + unify log taxonomy
Pure structural refactor of the Pass-N recovery path, no behavior change.

- PatchCtx<R> holds the one-pass context (reader, pipe, shared, opts,
  reused read buffer, cross-region state) and owns orchestration.
- PatchCtx::run() walks the ordered bad ranges, applies the inter-range
  cooldown (armed on a grind, consumed at the next range's entry), and
  stops the pass the moment a range reports halt/wedge/transport-fault.
- PatchCtx::patch_region() is the focused single-range loop: read ->
  success/failure -> damage-skip -> per-range watchdog. Every former
  break / break 'outer is now a typed RegionOutcome (Completed,
  SkipLimit, BudgetExceeded, Wedged, Halted, TransportFault) that run()
  maps back to next-range vs stop-pass exactly as before.
- recovery_read() extracts the AACS mid-unit-window widening + plain
  read into one reusable helper (the building block the upcoming
  scatter re-read will share).
- Logging unified onto a single dotted patch.* taxonomy (patch.region.*,
  patch.read.*, patch.probe.*, patch.wedge.*, patch.batch.*,
  patch.mapfile.*, patch.skip.*, patch.stall, patch.speed); the
  freemkv::scan pass-boundary markers are left intact.

20/20 disc::patch tests green; precommit (fmt + clippy -D warnings +
tests) clean on Rust 1.86.
2026-06-30 10:04:51 -07:00
Matthew Jackson c8822ddea3 error: treat IoError/DeviceNotFound as SCSI transport failures (dead bus)
is_scsi_transport_failure recognized only the 0xFF SCSI sentinel, so the two
NON-SCSI dead-bus faults the Linux SG_IO transport returns — Error::IoError
(ioctl(SG_IO) == -1: ENODEV/EIO on an unplugged bridge) and Error::DeviceNotFound
(fd gone) — were classified as recoverable bad sectors. Every caller
(sweep read_error, patch, single-pass fill_extents) would then zero-fill /
skip-ahead across a wedged bus instead of aborting the pass and re-enumerating
the bridge — potentially zero-filling a huge span. Add both to the classifier so
a dead bus aborts the pass everywhere. (Companion to the unlock_bridge fix, which
maps the same two faults to 0xFF across the freemkv-unlock seam.)
2026-06-30 07:19:03 -07:00
Matthew Jackson 7a7ab2c9d8 unlock_bridge: map non-SCSI transport faults (IoError/DeviceNotFound) to 0xFF
Round-2 audit gap in the prior bridge fix: extract_scsi_context only recognizes
Error::ScsiError / DiscRead and collapses every other variant to (0, None). But
the Linux SG_IO transport returns genuine dead-bus faults as non-SCSI variants —
Error::IoError (ioctl(SG_IO) == -1: ENODEV/EIO on an unplugged bridge) and
Error::DeviceNotFound (fd gone) — which were crossing the seam as status 0x00, so
the unlock crate classified neither as a transport fault (needs 0xFF) nor as a
sense-bearing rejection, and kept hammering a dead bus.

Match the error variant in the adapter: a SCSI status (CHECK CONDITION or a
drive-tagged 0xFF) keeps its real status + sense; any other variant is a
non-SCSI transport/IO fault → SCSI_STATUS_TRANSPORT_FAILURE / None. Keying off
`sense.is_none()` would be wrong (a senseless CHECK CONDITION is a rejection).
Add bridge tests covering CHECK-CONDITION sense preservation, 0xFF passthrough,
and IoError/DeviceNotFound → 0xFF.
2026-06-29 22:57:09 -07:00
Matthew Jackson 7a0ef5412c mux: add the B1 drop-to-keyframe resync gate to DiscStream
DiscStream (the live-drive single-pass path) enables AACS decrypt-loss
concealment (NULL-TS fill on an undecryptable unit) but, unlike the file-backed
PipelinedPesStream, had no B1 resync gate — so after a concealed gap it forwarded
inter-coded video frames referencing the now-missing data, producing
decode-broken MKV output (dangling-reference frames).

Mirror PipelinedPesStream: add per-stream ResyncGate + is_video, and route every
codec-frame emit on the TS path (in-stream parse, demuxer EOF flush, parser EOF
flush) through gate.admit(is_video, frame.discontinuity, frame.keyframe) so a
video track drops forward to the next keyframe after a concealment event. Warn
once if a gate is still armed at EOF (a concealed gap in the final GOP with no
trailing keyframe). Audio/subtitle always admit.
2026-06-29 22:27:52 -07:00
Matthew Jackson 8c38cb0918 unlock/patch: preserve SCSI sense across the bridge; init aborts on dead bus; slow-retry first patch failure
Audit fixes (v1.1.0..HEAD regressions in the unlock migration + adaptive patch
speed):

- unlock_bridge ScsiAdapter: libfreemkv's transport returns Err on ANY non-zero
  SCSI status (a normal CHECK CONDITION), not only transport faults. The adapter
  was collapsing every such Err to { status: 0xFF, sense: None }, which discarded
  the parsed sense and defeated the AACS handshake's ILLEGAL_REQUEST wedge guard
  (so it kept hammering the drive — hard-rule #2) and inverted its
  transport-vs-rejection diagnosis. Now reconstruct status + the 32-byte sense
  buffer (sense_key@2, asc@12, ascq@13) and only emit 0xFF/None for a genuine
  transport fault.

- Drive::init: a genuine transport fault during the drive-prep unlock means the
  bus is dead — propagate it (the v1.1.0 invariant) instead of silently
  swallowing it via `if let Ok`. Other errors (no matching unlocker) still fall
  through to stock mode. SET CD SPEED max now runs only when the bus is alive.

- disc::patch: on the first read failure in a range, drop to slow recovery speed
  and RE-ATTEMPT the same position at slow speed before marking it. A
  single-sector range's first failing sector was being marked from a MAX-speed
  read it never got to recover.

- docs: lib.rs architecture diagram (handshake → host_certs) and README (stale
  pluggable-unlock-seam / register-unlocker / crates.io / docs.rs references).
2026-06-29 22:16:28 -07:00
Matthew Jackson b36896564f disc: log adaptive patch speed transitions; issue SET CD SPEED max unconditionally
Two recovery-path fixes:
- patch: log the per-range speed transitions (INFO, phase="patch_speed") — each
  range enters at 0xFFFF (max) and drops to 0x0000 (slow recovery) on its first
  read failure. Previously the adaptive-speed behavior was invisible in the logs.
- drive init: issue the generic SET CD SPEED max UNCONDITIONALLY at drive-open,
  not only when a firmware unlocker matched. A stock-mode BD/UHD drive (no
  firmware unlock) was left riplocked because the call sat inside the
  unlocker-matched branch.
2026-06-29 21:47:50 -07:00
Matthew Jackson e648bde94b README: drop crates.io/docs.rs badges + stale plugin-model line (off crates.io now) 2026-06-29 21:17:06 -07:00
Matthew Jackson c86fa9bfc6 Cargo: publish = false — off crates.io (git-deps the firmware crate)
libfreemkv now git-deps freemkv-unlock (which carries drive firmware and can
never be on crates.io), so libfreemkv itself can only be consumed by git tag.
The freemkv-unlock dep stays an interim path dep for local cross-repo dev; the
release script swaps it to a git tag in the tagged commit.
2026-06-29 21:04:05 -07:00
Matthew Jackson 2ba6274eae unlock: dispatch via freemkv-unlock; delete in-tree handshake/css-auth/registry
Rewire the three unlock dispatch points through the freemkv-unlock crate via a
private `unlock_bridge`: drive-prep (kind=Unknown) at `Drive::init`, AACS cert
(kind=Aacs) at `do_handshake_cert`, CSS bus-auth (kind=Css) at scan. The bridge
news up `all_unlockers()` and runs the first matching one, mapping its
`Unlocked` result to the bus-key gate. After a successful drive unlock,
libfreemkv issues a generic SET CD SPEED (max) itself — the old per-unlocker
trait method is gone.

Delete the in-tree unlock code now owned by freemkv-unlock: the AACS cert
handshake (`aacs/handshake.rs`), the CSS bus-auth (`css/auth.rs`), and the
unlock registry (`unlock.rs`). Host-cert collection (a keysource concern) stays
in a small `aacs/host_certs.rs`. No public unlock surface remains — clients
touch libfreemkv only, oblivious to unlockers (as they are to SCSI). 2277 tests
pass.
2026-06-29 20:45:00 -07:00
Matthew Jackson 3bdb6f8b1a stage 4a: bridge libfreemkv to the freemkv-unlock crate
Add the freemkv-unlock dependency (path dep for local dev; re-pinned to a git
tag at release) and the consumption bridge: one generic ScsiAdapter mapping
libfreemkv's transport to the unlock contract, host-cert + DriveId mapping, and
run_unlockers() which news up all_unlockers() and runs the first matching one.
Additive — validates the dep resolves and the adapter compiles; the in-tree
registry/handshake/css path is still active and gets replaced in 4b.
2026-06-29 19:48:50 -07:00
Matthew Jackson 80e8523db2 aacs: extract run_cert_handshake + collect_host_certs as shared primitives
Pull the wedge-guarded cert loop and host-cert collection out of the in-tree
AacsCertUnlocker into public aacs::handshake primitives (run_cert_handshake +
CertHandshake, collect_host_certs). The in-tree path now delegates to them, so
the external freemkv-unlock-aacs plugin runs the IDENTICAL cert handshake — one
implementation, two callers. Pure refactor of the live AACS path; the existing
handshake + collect_host_certs unit tests validate it unchanged.
2026-06-29 17:33:26 -07:00
Matthew Jackson ab8f09645f unlock: route_unlock returns a structured UnlockRoute; ctx carries opts
Replace route_unlock's Option<(name, Vid)> with a structured UnlockRoute
{ Unlocked(name, Unlocked) | Failed(UnlockError) | NoMatch } so a single
dispatch serves every caller: drive-prep wants "did anything unlock", and the
AACS cert route (next) needs the FAILURE REASON to render "missing keys" vs
"host cert rejected" instead of collapsing it to a bare None. Only a genuine
SCSI transport fault still returns Err (abort). UnlockCtx gains an optional
ScanOptions (the cert route's host-cert source), and read_mkb_from_drive now
takes &mut dyn ScsiTransport — both prerequisites for the cert handshake to
become an external freemkv-unlock-aacs unlocker. Drive-prep + CSS callers fold
the new outcome; no behavior change.
2026-06-29 17:18:11 -07:00
Matthew Jackson 326d17c2f4 unlock: Unlocked carries the bus-key gate nuance; route_unlock returns it
Extend Unlocked with drive_unlocked + read_data_key_err so the registry result
carries everything the downstream bus-key gate needs — a firmware unlocker sets
drive_unlocked=true (bus encryption removed at the drive, no bus key), the cert
route will set read_data_key (+ read_data_key_err on a failed bus-key read).
route_unlock now returns the full Unlocked rather than just a Vid, so a single
dispatch can fold the firmware, cert, and CSS results uniformly. Prep for the
AACS cert handshake becoming a registry unlocker.
2026-06-29 16:59:04 -07:00
Matthew Jackson bac105a022 css: CSS bus-auth becomes a uniform registry Unlocker
Convert the CSS read-unlock into a first-class registry Unlocker (CssUnlocker)
dispatched through route_unlock like every other barrier removal, instead of
a direct call in scan. libfreemkv appends the built-in CSS unlocker (and, next,
the AACS cert handshake) exactly once via ensure_builtins(), AFTER any
client-registered firmware unlocker — so the registry order is firmware → cert
→ css, owned by the lib, not the client.

Defense in depth: the unlocker does NOT trust the caller-declared DiscKind.
matches() filters on the declared kind (Css), but unlock() self-verifies
against the drive's GET CONFIGURATION profile and refuses (UnlockError::
NotApplicable, a new shared "this unlocker doesn't apply" variant) WITHOUT
issuing a single CSS CDB if the drive reports a non-DVD profile — so a
mis-routed Blu-ray is never sent CSS bus-auth. Guard the firmware unlocker the
same structural way (it matches only the drive-prep phase, kind == Unknown).

Tests: CssUnlocker matches only DiscKind::Css; a BD-profile drive yields
NotApplicable with zero CSS CDBs issued.
2026-06-29 16:49:52 -07:00
Matthew Jackson cfcc524367 unlock: introduce UnlockCtx + DiscKind; trait keys off context
Reshape the Unlocker seam so every unlocker is dispatched at ONE place from
ONE ordered registry — the firmware, cert, and CSS routes are all "remove the
bus-encryption barrier", differing only in what they key off. matches() and
unlock() now take an UnlockCtx { drive_id, kind: DiscKind } instead of a bare
DriveId: a firmware unlocker keys off drive_id (kind irrelevant), the cert
unlocker will match DiscKind::Aacs, the CSS unlocker DiscKind::Css. UnlockCtx
is #[non_exhaustive] so a host-cert source can be added without breaking
external unlockers. Drive-prep dispatch passes DiscKind::Unknown (no disc
probed yet); the cert/CSS registry impls + the single post-probe dispatch
point follow in subsequent commits.
2026-06-29 16:27:30 -07:00
Matthew Jackson 03820c68f8 css: auth primitives operate on &mut dyn ScsiTransport; clean up bus_auth
Thread &mut dyn ScsiTransport through unlock_css_reads and its bus-auth /
disc-key primitives instead of &mut Drive — they only ever issued SCSI via
drive.scsi_mut(), so this is mechanical and lossless, and it makes the CSS
unlock speak the same transport interface as the registry Unlocker trait
(prerequisite for CSS becoming a uniform unlocker).

Also clean up the badly-named, half-dead bus_auth:
- rename bus_auth -> establish_authenticated_session: it is run for its
  side effect (sets the drive's ASF=1, unlocking scrambled-sector reads),
  which the name now states.
- drop the derived CSS bus key from the return + computation: it had no
  consumer (descrambling is keyless via the Stevenson attack), so it was
  dead crypto computed on every DVD unlock. Return just the negotiated AGID,
  which the caller genuinely needs for the best-effort disc-key REPORT KEY.
2026-06-29 16:01:07 -07:00
Matthew Jackson f682405973 aacs: handshake primitives operate on &mut dyn ScsiTransport
The AACS cert-auth primitives (aacs_authenticate, the AACS 2.0 P-256
variants, read_volume_id, read_data_keys) and their scsi_read/scsi_write
helpers touched the drive ONLY through Drive::scsi_execute — a pure
pass-through to the transport. Thread &mut dyn ScsiTransport instead of
&mut Drive so these primitives are transport-level, matching the firmware
Unlocker seam (which hands out &mut dyn ScsiTransport for testability).
Pure mechanical signature change, no logic change; the cert orchestrator
(do_handshake_cert) keeps &mut Drive for the OEM-VID shortcut and passes
session.scsi_mut() into the primitives. Step toward making the cert
handshake a uniform registry unlocker.
2026-06-29 15:47:30 -07:00
Matthew Jackson 93fbfac6f0 unlock: Unlocker::unlock returns Unlocked { vid, read_data_key }
Generalize the unlock seam toward uniform unlockers (firmware / cert / CSS).
`unlock` now reports what it LEARNED — a Volume ID and/or an AACS bus key —
instead of just a Vid; libfreemkv files those onto the disc/drive in one
place, so a plugin never touches Disc. The hardware side-effect (extended
mode) still happens inside unlock. route_unlock walks the registry in
registration order (the single ordered place that decides run order); no
priority field. Firmware route reports {vid: Some, read_data_key: None}.
2026-06-29 15:09:17 -07:00
Matthew Jackson a7c8ee09b0 disc: credit firmware unlock in the bus-encryption gate
The bus-key gate only credited the cert handshake's read_data_key as proof
bus encryption was removed. A firmware unlocker removes it AT THE DRIVE
(serves clear content) and yields no read_data_key — so a SUCCESSFUL
firmware unlock (VID present, read_data_key None) tripped the gate and
blocked ALL key resolution, including the online source. That was the
root cause of live UHD discs reporting "missing keys" after an unlock.

Now a single predicate answers "is bus encryption gone?": never-had-it ||
file/ISO || firmware-unlocked || cert-bus-key. The gate is just
`if !bus_encryption_removed { error }` — no enumerated cases. HandshakeResult
gains `drive_unlocked`, and the read_data_key failure reason is captured so
the warn says WHY the bus key is missing.

Also: reword the first hardware-sense escalation as "fast-fail escalation"
(it is often transient — the drive recovers), reserving "wedge" for a
persistent run; and scrub the product name from core comments (it belongs
only in the unlocker crate).
2026-06-29 15:09:16 -07:00
Matthew Jackson 263950622f disc: make bus-key unavailability self-explaining in the log
When a bus-encrypted disc's handshake yields no read_data_key, the gate
logged a bare "bus_key_unavailable" with no indication of WHY — turning
every occurrence into archaeology (is the bus key not-attempted, or did
the read fail?).

Now:
- read_data_keys failure is captured (error code) instead of swallowed by
  `.ok()`, and logged at the handshake with its consequence.
- HandshakeResult carries `read_data_key_err: Option<u16>` so the gate
  distinguishes "never attempted" (None — VID-only/OEM path) from "read
  FAILED" (a code), and the bus_key_unavailable warn now reports the code
  plus whether a Volume ID was present.

No behavior change — purely diagnostic. The handshake_ok debug also now
reports has_volume_id.
2026-06-29 13:33:09 -07:00
Matthew Jackson 9e6af4a729 mux: harden audio discontinuity handling (audit follow-up)
Two defensive hardenings from the post-fix audit (vs FFmpeg/GStreamer):

1. Move the `pes.discontinuity` partial-drop ABOVE the empty-data guard in
   all three audio parsers (ac3/dts/truehd), so a discontinuity signal can
   never be stranded by an empty post-gap PES. The demuxer only emits
   non-empty PES today; this is defense-in-depth for any future caller.

2. A PES with no PTS must not reset the timeline to 0. ac3 now carries
   `flush_pts_ns`, dts continues from the most recent known base; truehd
   already kept its running cadence on a None PTS. Matches OSS behavior
   (PTS rebases off the next PES that actually carries a PTS).

Adds an ac3 regression test (empty-payload discontinuity PES still drops
the stranded partial). Loss accounting was reviewed: TS-demux CC-gaps are
NOT counted toward lost_video_secs / abort (that is sector-based via
DiscStream::errors / mapfile bytes_unreadable), so a source splice never
inflates loss — no gating needed there.
2026-06-29 12:50:52 -07:00
Matthew Jackson be08e3938b mux: drop truncated partial audio frame on concealed gap
The AC-3, DTS and TrueHD parsers buffer access units across PES
boundaries. At a concealed-loss gap the buffered unit is truncated:
splicing post-gap bytes onto it manufactures a corrupt frame on top of
the real loss (FFmpeg "Failed to decode block code(s)" / "Invalid data
found" at the gap) and, for TrueHD, strands the PTS cadence into the
non-monotonic audio-DTS band seen on multi-clip titles.

The video parsers already handle this via the ResyncGate, but the
discontinuity signal was only wired into video — audio parsers ignored
pes.discontinuity and spliced across the gap.

Now, when pes.discontinuity is set, each audio parser drops the partial
(clears buf, and for DTS its PTS marks / pending base) so the post-gap
PES re-bases a fresh unit. A lost gap degrades to a clean single-frame
drop instead of a corrupt spliced frame. No effect on perfect rips: the
branch only runs when concealment inserted a discontinuity marker.

Adds a per-parser test feeding a partial frame then a discontinuity PES,
asserting the truncated partial is dropped (not spliced) and the post-gap
PTS is adopted.
2026-06-29 11:23:08 -07:00
Matthew Jackson cb7d78ac6a disc: adaptive patch speed — enter each range fast, slow on first failure
Pass N (patch) pinned the slow recovery speed (0x0000) for the whole pass.
But Pass 1's damage-jump overshoots, so most of a jumped range is clean data
the reverse-walk reads first — grinding it at slow speed wastes minutes per
gap. Now each range mirrors Pass 1's model:

- Enter at max speed (0xFFFF) + reset to the initial batch (current_batch
  carries across ranges, so a prior single-sector grind would otherwise start
  the next range slow). Reads the clean overshoot fast.
- First read failure in the range → drop to 0x0000 (once; idempotent SET CD
  SPEED) for the rest of that range — grind only the genuine damage.
- 10 s halt-responsive cooldown between ranges, gated on "this range actually
  grinded" so a many-small-range pass (100+ ranges) doesn't stall ~20 min on
  unconditional pauses.

No change to good/bad classification, the mapfile, or the abort math — purely
read speed + an inter-range cooldown. Synthetic speed-transition test to follow.
2026-06-29 10:34:18 -07:00
Matthew Jackson 039a8f9f19 docs: 1.2.0 changelog date → 2026-06-29 (release prep); fix neither/nor grammar in 1.1.0 entry 2026-06-29 09:57:29 -07:00
Matthew Jackson c5515d310f mux/decrypt: audit-nit cleanup + defensive conceal fallback
Non-blocking follow-ups from the 1.2.0 audit:
- conceal loop: if decrypt reported loss but the padding-aware predicate
  matched nothing to conceal (a ~256^-31 contradiction), fall back to the
  strict predicate and conceal whatever it flags, loudly — belt-and-
  suspenders so ciphertext can never reach the mux.
- decrypt_dropped doc: reflect 1.2.0 (mux-path loss is concealed + tallied,
  not silently dropped).
- direct unit test for aacs_unit_still_ciphertext (the padding-aware
  conceal predicate): clear/all-zero/full-decrypted/full-ciphertext/
  decrypted-short-tail.
- fix three stale "v1.1.1" comment refs (the fragment-tail fix ships in
  1.2.0; there is no v1.1.1 release).
2026-06-29 09:51:28 -07:00
Matthew Jackson 789b699f95 mux: make B1 concealment decode-clean on every gap shape
Closes the three residual holes where a concealed/lost gap could still let
a dangling-reference frame reach the muxer (degraded/undecryptable-disc
path only; clean rips are byte-identical and untouched). Root cause: the
discontinuity signal was reconstructed from the 4-bit continuity counter
and applied per-PES, both of which are lossy.

Three coordinated changes:

1. CC-INDEPENDENT marker. fill_null_ts_unit now tags its NULL packets with
   an adaptation-field discontinuity_indicator; the demuxer recognises a
   0x1FFF packet carrying it as a concealed gap and forces a discontinuity
   on every tracked PID (the lost unit's PID is unknowable). This survives
   a loss that is an exact multiple of 16 packets (CC aliases to in-sequence
   — hole 3) and a loss at a PID's very start (no prior CC — hole 4); it
   also drops any open, potentially-truncated partial PES.

2. PUSI ATTRIBUTION. A gap landing on a PES boundary now flags the PES
   STARTING after it, not the one flushed at the boundary (hole 1) —
   stamping the pre-gap frame could arm-then-disarm the gate on a keyframe
   and admit the real post-gap inter frame.

3. PER-FRAME signal. codec::Frame gains `discontinuity`; each parser
   propagates it onto the first post-gap frame. MPEG-2 buffers whole GOPs
   asynchronously, so it associates the gap by ES OFFSET (like PTS/source),
   landing it on the exact post-gap picture mid-GOP (hole 2) — a per-PES
   flag stamped the previous picture. consume_ts (and the EOF flush drain)
   gate on frame.discontinuity.

Tests: CC-independent marker with in-sequence CC + leading-loss; PUSI
attribution flags the post-gap PES; MPEG-2 offset-mark stamps the post-gap
picture through GOP reorder, not the previous one. Existing B1 gate + EOF
tests still green (2270 lib tests).
2026-06-29 09:39:03 -07:00
Matthew Jackson 71b4b09c93 docs: 1.2.0 CHANGELOG — document the breaking public-API changes
Per the release decision: ship the AACS-version-stride API changes under
the 1.2.0 minor bump, with an explicit Breaking section (DiscInputs new
field, DiscInputsCtx::new arity, read_aacs_inputs* 3-tuple return).
2026-06-29 08:41:26 -07:00
Matthew Jackson 067fd207d5 mux: conceal only genuinely-undecryptable units (padding-aware)
The P3 concealment loop in DecryptingSectorSource::read_sectors keyed on
aacs_unit_needs_decrypt, whose sync check is the majority-vote
ts_sync_destroyed (<=16 of 32 syncs). A successfully padding-aware-
decrypted content-fragment TAIL unit (e.g. 11 content packets + 21 zero
padding) has only 11 syncs, so the majority vote called it "still
encrypted" — and when such a good unit shared a read buffer with a
genuinely-undecryptable one (dropped>0), the loop overwrote the GOOD
decrypted tail with NULL-TS, silently discarding correct video and
over-counting concealed units vs the tallied dropped bytes.

Add aacs_unit_still_ciphertext (padding-aware): encrypted AND at least
one non-zero (non-padding) 192-byte packet missing its 0x47 sync — the
same discriminator decrypt_unit uses to accept a fragment tail. The
conceal loop now uses it, so only genuinely-unrestored ciphertext is
concealed. Full and fully-ciphertext units are unchanged. Regression
test: a decrypted short-padding-tail co-resident with a failed unit is
left byte-for-byte intact while the failed unit is concealed.
2026-06-29 00:29:20 -07:00
Matthew Jackson e6180a429b v1.2.0: bump version + reconcile CHANGELOG to [1.2.0] 2026-06-28 23:34:19 -07:00
Matthew Jackson d715a0943a mux: B1 drop-to-keyframe resync after a concealed gap
Pairs with A2 (read-path NULL-TS concealment). When the demux assembler
sees a TS continuity gap it now stamps `discontinuity` on the next
completed PES; the codec-parse stage carries that onto a per-track
ResyncGate. After a gap on an inter-coded video track the gate drops
forward to the next IRAP/IDR keyframe so no frame with a dangling
reference reaches the muxer (an ffmpeg deep scan would otherwise report
a missing-reference / non-existing-PPS error). Audio and subtitle tracks
have no cross-frame references, so the gate is a no-op there.

- ts.rs: PesPacket gains `discontinuity`; PesAssembler tracks a sticky
  pending_discontinuity flag set on CC gap / discontinuity_indicator and
  carried to the next completed/flushed PES.
- resync.rs (new): ResyncGate — per-track arm-on-gap, drop non-keyframes
  until the next keyframe disarms and resumes. Logs the resync + drop
  count once at the keyframe.
- pipelined_stream.rs: precompute per-track is_video, apply the gate in
  consume_ts. Out-of-range track index emits as-is (defensive).

Tests: ResyncGate unit tests; ts.rs gap-stamps-discontinuity; end-to-end
B1 video-drops-to-keyframe and audio-never-drops through PipelinedPesStream.
2026-06-28 23:01:47 -07:00
Matthew Jackson 9a7be7a1a5 1.2.0: mux loss-concealment read path (P3/Edit-2, A2 NULL-TS fill)
Decrypt-verify is a RIP gate, not a MUX gate. On the mux read path an
undecryptable content unit must never abort the mux:

- DecryptingSectorSource gains tolerate_decrypt_loss(): when set, an
  undecryptable in-content unit is tallied, overwritten with valid NULL
  TS packets (PID 0x1FFF) via aacs::fill_null_ts_unit, logged loud with
  its LBA, and the read returns Ok — the stream keeps flowing. The rip
  paths keep the fail-loud DECRYPT_VERIFY_READ decorator (re-read off the
  disc); only the mux opts in.
- Wire it into both mux read paths: the file-backed highway
  (build_iso_pipeline) and the inline DiscStream.
- NULL-TS fill keeps the demuxer byte-synced on the 192-byte stride; the
  lost video/audio PID packets surface as a CC gap the TS assembler
  already drops a partial PES on (the B1 foundation). Ciphertext is never
  passed downstream either way.
- Fix stale resolve_vid_only no-cert test: default is UHD (audit #4).

Tests: conceal-as-NULL-TS, fill well-formedness, fail-loud still holds.
2026-06-28 22:44:19 -07:00
Matthew Jackson a731e7b26b 1.2.0: single MKB framing walker + AACS resolve hardening
- mkb_records() as the one record-framing iterator; rebuild walk_mkb,
  find_record_body, mkb_find_subdiff_records, mkb_content_len, mkb_version,
  mkb_type_raw, mkb_find_mk_dv on it (D4).
- resolve_vid_only / read_aacs_version default to UHD (+warn) on a missing
  content cert instead of BD; route MKB through bounded read_mkb_content.
- AacsVersion major()/from_major() + AACS_MAJOR_BD/UHD as the stride source;
  table + stride-discriminating regression tests.
- read_encrypted_units probes 8 evenly-spaced points per extent (off-midpoint
  scrambled content now sampled); decrypt source-zero mask uses PKT.
2026-06-28 22:28:09 -07:00
Matthew Jackson 4d6f5c0a98 1.2.0: unify hex parsing (one parser) + robust encrypted-unit sampling
- One workspace hex parser (libfreemkv::hex): the keydb / online / mapfile
  parsers had divergent prefix handling (0x vs 0X vs none) — a key written with
  a prefix one parser didn't expect was silently dropped. All three now call it.
- read_encrypted_units probes several points spread across each extent instead
  of only the midpoint-forward window, so a late-starting / sparse encrypted
  body still yields samples (empty samples make decrypt_with skip wrong-key
  validation). A read error at one probe no longer abandons the extent.
2026-06-28 22:12:06 -07:00
Matthew Jackson cad5929afe 1.1.1: unify AACS key-input path on Disc::inputs() + named constants
- read_aacs_inputs* now returns the AACS major version; DiscInputs carries it,
  and DiscInputsCtx parses Unit_Key_RO.inf at the disc's own stride (fixes the
  hardcoded-V20 read-time fetch for V10 discs). One source of truth, no version
  argument to drift.
- Disc::inputs() is the single complete AACS-input source (inf/MKB/VID/hash/
  version); the out-of-band duplicate readers go away.
- Named constants for AACS file paths (aacs::PATH_*) and the AACS majors
  (aacs::AACS_MAJOR_*, AacsVersion::major/from_major) replace magic strings/ints.
- push_ranges saturating (corrupt-disc panic guard).
2026-06-28 21:45:58 -07:00
Matthew Jackson 59681dfd4b 1.1.1: AACS decrypt + key-resolution hardening
- decrypt_unit: padding-aware acceptance — recover real video at content-
  fragment tails (the phantom mux-loss class) without weakening wrong-key
  rejection (a full content unit still needs all 32 TS syncs).
- scan: read the MKB via the bounded read_mkb_content so Disc::inputs()
  carries it. Online key resolution was shipping mkb=0 (a full read of the
  ~128 MiB MKB_RO allocation fails) → the decode service 404'd.
- resolve_vid_only: surface an MKB read error instead of silently emptying.
- fetch: a per-sample dry-set replaces the global fetch_spent latch, so a
  second CPS unit's key can still be fetched after the first came back empty.
- verify::push_ranges: saturating arithmetic (corrupt-disc panic guard).
- Tests for all of the above.
2026-06-28 21:12:57 -07:00
Matthew Jackson eba34f4c20 v1.1.0: bump version 2026-06-28 18:14:18 -07:00
Matthew Jackson 49e5627a69 changelog: 1.1.0 (drop -beta.1 pre-release tag) 2026-06-28 17:58:46 -07:00
Matthew Jackson fb8405385e aacs: rustfmt verify-gate files + correct cert/unit-key test fixtures to libaacs-strict layout; 1.1.0-beta.1 changelog 2026-06-28 16:49:16 -07:00
Matthew Jackson f23338b5dd verify: container-kind seam (HD-DVD-ready) + skip units with unread sectors
ContainerKind {Ts,Ps} + ClipLayout.container thread the post-decrypt structural check per clip; decryptability() dispatches it (TS: unit_is_clean_ts, PS: unit_is_clean_ps). New decrypt_unit_checked(unit,key,accept) decouples the container-agnostic AACS crypto from the format-specific acceptance (decrypt_unit delegates with the TS check). unit_is_clean_ps is the MPEG-2 PS pack-start check, documented UNVALIDATED for HD-DVD (.evo unit/seed/pack alignment must be confirmed on real media). clip_layouts assigns Ts today; .evo->Ps is the one-line HD-DVD hook.

reverify_iso now takes an is_finished predicate and SKIPS any unit with a non-Finished backing sector: we can't verify what wasn't read (a non-Finished sector is zero-filled because the drive read failed there), and must never waste a key lookup on a block the read already knows is bad. observe() (sweep) was already safe (only fed Good bytes).
2026-06-28 16:05:25 -07:00
Matthew Jackson a7bd574c34 verify: post-read decrypt-verify gate + libaacs-strict verify + audit fixes
Post-read verify gate (new src/disc/verify.rs): UnitVerifier buffers/aligns the disc-absolute read stream into clip-file 6144-byte units, then makes one decryptability() decision per unit (CPI gate -> held keys -> key_fetch -> strict TS). POST_READ_VERIFY const kill-switch; fail-safe contract (only ever downgrades units it is confident are undecryptable; every doubt skips). Hooked into Disc::sweep (producer observes ciphertext -> WorkItem::MarkBad after the Good, FIFO-ordered) and Disc::patch (post-loop reverify_iso reads recovered units whole from the patched ISO). extract::clip_layouts enumerates AACS clips for the gate.

Standards-correct AACS verify: aacs::unit_is_clean_ts is a strict port of libaacs _verify_ts (all 32 TS syncs, not a majority vote); decrypt_unit accepts a key only on it; the majority verify_ts is removed. Deleted the Disc::verify_clips post-pass bolt-on (its primitive is absorbed by the read-path gate).

libaacs/DVD audit fixes: content-cert bus_encryption flag now read from bit 7 (was bit 0 - defeated the bus-key fail-loud gate); cc_id read from offset 14; title_cps_unit range-validated + 1->0 index-converted per libaacs. Corrected attack_crib ("functionally-equivalent" not "exact" port) and read_disc_key (READ DVD STRUCTURE 0xAD, not REPORT KEY) doc comments.

Also includes accumulated uncommitted work: key-fetch seam and TrueHD/DTS audio fix.
2026-06-28 15:03:52 -07:00
Matthew Jackson f49ef023cf changelog: enrich 1.1.0-beta.1 and rc.5.2 entries with concrete mechanisms (version label SoT, per-extent AACS anchor, E7017/E7022 no-key split, 3-sector unit grid)
leak-guard / leak-guard (push) Successful in 28s
2026-06-26 22:26:15 -07:00
Matthew Jackson ba5e4fdafa changelog: add 1.1.0-beta.1 version-stamp consistency, DTS-HD MA core-sync, and TrueHD DTS-monotonicity fixes 2026-06-26 22:16:45 -07:00
Matthew Jackson ced89133cc docs(fvi): standardize table headers and multi-line dense cells
Use 'Semantics / reference' as the last-column header across all spec
tables, and break enum/option/note cells onto separate lines with <br>
for readability. Formatting only; no semantic changes.
2026-06-26 21:42:31 -07:00
Matthew Jackson aefd6b6342 mux: single source of truth for the version label (VERSION_LABEL/MUX_APP)
The CLI --version, the MKV muxing/writing-application field, and the FVI generator
all derive from one libfreemkv const, so a binary reports the exact same label it
stamps into the files it produces — no split-brain where an MKV claims one version
and the binary another.
2026-06-26 21:33:19 -07:00
Matthew Jackson 78f78d285e docs(fvi): pretty-print JSON blocks in FVI format spec
Reformat all three JSON code blocks (section 8 ext example, Appendix A
Header schema, Appendix A Record schema) with consistent 2-space
indentation and valid syntax. No field or semantic changes.
2026-06-26 21:21:26 -07:00
Matthew Jackson 80ecb671fd mux: allow build-time version label override (FREEMKV_BUILD_LABEL)
build.rs emits FREEMKV_VERSION = FREEMKV_BUILD_LABEL when that env is set
(non-empty), else the Cargo package version. The muxing/writing-application
field and the FVI generator tag use it, so a pre-release/test build can be
stamped without bumping Cargo.toml and disturbing the tag-pinned [patch]
version matching. The git short hash is still appended either way.
2026-06-26 19:38:13 -07:00
Matthew Jackson c49a180ce7 mux: fix non-monotonic audio DTS (TrueHD + DTS-HD MA) and stamp builds with git hash
TrueHD: when the PES PTS lags the access-unit cadence, resync to the PTS
but never snap the running timestamp backward, so the emitted DTS stays
monotonic across the resync (next_pts_ns = max(next_pts_ns, pts)).

DTS-HD MA: size each EXSS extension substream exactly from its header
(exss_frame_size) and skip it as a unit, so a false 0x7FFE8001 core sync
inside the lossless extension payload can no longer split the access unit
and truncate the extension. Falls back to a bounded scan when the header
is unparseable.

Provenance: build.rs bakes the git short hash into GIT_SUFFIX; the muxing/
writing-application field and the FVI generator tag now record the exact
build (e.g. "freemkv 1.1.0-beta.1 (g835cc99)"), so any output file is
traceable to the revision that produced it.
2026-06-26 19:26:24 -07:00
Matthew Jackson afa218fc8f Remove keydb download/save from the library
keydb I/O moves out of libfreemkv into freemkv-keysources
(KeydbSource::save / ::update). Delete src/keydb.rs entirely (save,
http_get, default_path, write_atomic, UpdateResult) and drop `pub mod
keydb;` — http_get had no real callers. The shared Keydb* Error variants
stay in error.rs (keysources raises them; the every-error-has-a-code
contract depends on them). flate2 is no longer used here, so drop it
from Cargo.toml (zip stays for labels/jar.rs).

CHANGELOG: note the keydb-I/O move; reword the rc.5.2
DefaultDecodedFieldDuration entry to state only the action taken (the
revert) rather than an unverified Windows-fps outcome.
2026-06-26 17:33:57 -07:00
Matthew Jackson 835cc990ad DVD vob_start absolute rebase + rc.5.3 audit fixes
- ifo.rs: rebase VTS title VOBS to absolute disc LBA (file_start_lba +
  vtstt_vobs); fixes DVD rips opening on the menu region instead of the
  movie (e.g. SOTL). Adds absolute-placement regression test.
- aacs/boil.rs: add mk_from_pk primitive (PK -> MK via MKB walk).
- dvdnav/: nav-VM command decoder + start-cell resolver seam, parked
  behind USE_NAV_RESOLVER (kept compiled, never executed).
- mux: FVI src.byte within-sector per spec; Unknown colour -> CICP
  unspecified (2,2,2,1); demux clear PCS -> NORMAL; ts.rs feed() base
  reset + boundary provenance fix.
- Assorted audit fixes (doc/comment/test accuracy) across the crate.
2026-06-26 17:03:58 -07:00
Matthew Jackson d8c323bf9f Magic-number/taxonomy pass: central wire-format + sector + unit consts
- libfreemkv::consts: coding_type::* (ES coding-type bytes), pes_stream_id::*
  + PAYLOAD_RANGE, SECTOR_BYTES (usize) + SECTOR_BYTES_U64 (offset math)
- replace bare wire-code/sector literals across disc, mpls, clpi, labels,
  m2ts_mux, ps, tsmux, file_sector_source, extract
- remove two unreachable secondary-stream match arms in mpls parse_stream_entry
2026-06-26 13:20:21 -07:00
Matthew Jackson decb87a250 AACS pipeline reshape + TrueHD metadata + central consts + clippy/fmt clean
- AACS: delete in-lib keydb parser (Step 3); boil-down primitives
  (mk_from_dk/vuk_from_mk/uk_from_vuk) + newtypes; KeySource->get_uk(ctx)+
  ResolveCtx; Unlocker->unlock()->Result<Vid,UnlockError> + AacsCertUnlocker;
  OEM bus-key gate (AacsBusKeyUnavailable); structured ResolutionTrace (Step 4).
- TrueHD: sample-rate from major-sync, Atmos label, 44.1k AU duration.
- consts: central media/format constants module; 17 duplicate const-defs
  centralized (sector/TS-packet/source-packet); mpls stream-entry + category
  codes named.
- clippy --all-targets -D warnings clean (1.86); fmt clean; 2199 lib tests.
2026-06-26 12:19:24 -07:00
Matthew Jackson 05729f5dfe fix(libfreemkv): rc6 hardening pass — mux timeline/colour/PCR, demux panic sentinel, parser robustness + doc accuracy
Surgical fixes (each with a regression test that fails without the change):

mux/mkv.rs, mux/demux_sink.rs: drive the clip-boundary timeline epoch
off the resolved PRIMARY VIDEO track, not the literal stream index 0.
An M2TS/PMT title can list an audio ES before video, so streams[0] may
be audio; a non-video epoch driver ratchets the frontier and inflates
the timeline. mkv cluster-opening falls back to track 0 for audio-only
titles so they still open clusters.

mux/codec/ac3.rs: correct ACMOD_CHANNELS — acmod=5 (3/1) is 4 channels,
not 3 (was undercounting a 3/1 stream); fix the A/52 Table 5.8 doc.

disc/mod.rs: HDMV coding_type 0x91 (Interactive Graphics / menus) no
longer maps to PGS subtitle — it falls through to Unknown so the PMT/STN
walker drops it instead of surfacing a bogus subtitle track.

mux/videomap.rs + mux/mkv.rs: FVI colour now mirrors the MKV muxer's CICP
precedence (measured CICP authoritative; HDR-driven PQ/HLG transfer
override) via a shared cicp_for_video helper, so the two sinks can't
disagree (HDR10 BT.2020 no longer emits SDR transfer 14).

mux/mkvstream.rs: saturating_add on cluster_ts + rel_ts so an adversarial
CLUSTER_TIMESTAMP near i64::MAX can't overflow/panic before the existing
saturating_mul.

mux/timeline.rs: tighten the tail-straggler clamp so a normal new-epoch
non-video frame leading the sparse video frontier by >3s is not demoted
into the previous clip's epoch.

mux/m2ts_mux/mod.rs: re-stamp PCR per video TS packet (mid-PES), not only
at PES boundaries, so a large UHD I-frame can't open a multi-second PCR
gap; modular 33-bit PTS rebasing so a real 90 kHz clock wrap is not
collapsed to PTS 0 (pre-base frames still floor to 0).

io/byte_prefetcher.rs, sector/prefetched.rs: wrap the producer feed loop
in catch_unwind and emit a typed error sentinel on panic, so a mid-stream
producer panic is not read as a clean EOF at the demux boundary (which
would silently truncate the mux).

mux/codec/h264.rs: extend HIGH_PROFILES to the full ISO/IEC 14496-15 set
that mandates the avcC chroma/bit-depth extension (adds 244 et al.).

Doc/comment accuracy: css/mod.rs (50000 sectors, not scrambled-sectors),
aacs/decrypt.rs (decrypt_unit already-clear path), ifo.rs (TT_SRPT at
0xC4), css/lfsr.rs (LFSR0 24-bit; TAB1-then-XOR cipher; real scramble-flag
predicate), disc/read_error.rs (for_sweep does bounded transient retries).

Skipped: keydb.rs SSRF guard (low/latent, no live caller) — a hard
loopback block breaks an existing behavioral test that exercises the
header-EOF path over a loopback server; a clean fix needs a resolver test
seam beyond this surgical pass. The sibling keydb_fetch.rs comment fix is
out of scope (freemkv crate).
2026-06-25 23:39:03 -07:00
Matthew Jackson dc1d05985b feat(mux): emit HDR10 static metadata from HEVC SEI
Parse the two HDR10 HEVC SEI messages and emit the corresponding
Matroska Colour metadata, only when actually present in the bitstream
(SDR / no-SEI tracks omit it; nothing is fabricated).

Parse (Rec. ITU-T H.265 Annex D):
- Mastering Display Colour Volume SEI, payloadType 137 (D.2.28):
  display_primaries_x/y[3] (SEI order G,B,R), white_point_x/y
  (0.00002 units), max/min_display_mastering_luminance (0.0001 cd/m²).
- Content Light Level Info SEI, payloadType 144 (D.2.35):
  MaxCLL / MaxFALL (cd/m² integers).
HevcParser::scan_sei walks the sei_rbsp ff-extension payloadType/
payloadSize coding and de-emulates (00 00 03) before reading, reusing
the existing strip_emulation_prevention helper. Both SEI are required
before any metadata is surfaced; SEI NALs still pass through unchanged.

Carry: the measured Hdr10Metadata rides PictureInfo (the same per-coded-
picture seam FieldOrder uses), flowing through from_codec_frame onto
PesFrame.coding to the deferred-muxer activate path, where
apply_coding_to_track stamps it on the video track before the header is
written. Set only when both SEI were seen.

Emit (RFC 9559 / Matroska): new Colour children in ebml.rs
(MasteringMetadata 0x55D0, Primary R/G/B + WhitePoint chromaticity
0x55D1..0x55D8, Luminance max/min 0x55D9/0x55DA, MaxCLL 0x55BC,
MaxFALL 0x55BD). write_hdr10 converts chromaticity SEI int × 0.00002 →
Matroska float, luminance SEI int × 0.0001 → cd/m² float; MaxCLL/MaxFALL
are uints verbatim. SEI primary index 0/1/2 (G/B/R) mapped to the
Matroska R/G/B element layout. Emitted only when hdr10 is present.

Tests: SEI parse with exact raw values, requires-both-SEI, SDR omission,
and emulation-prevention stripping (hevc.rs); muxer emit with exact unit
scaling + SDR omission of MasteringMetadata/MaxCLL/MaxFALL (mkv.rs);
apply_coding_to_track HDR10 plumbing (mkvstream.rs).
2026-06-25 21:59:43 -07:00
Matthew Jackson 539b170f7e test(disc): pin the -t 1 = main-feature contract (DVD); fix 2 test clippy nits
Owner-flagged invariant: freemkv -t 1 ALWAYS selects the main feature because the
CLI's title 1 maps to titles[0] and the list is ordered by canonical_title_order
(main feature first). Adds a DVD-shaped contract pin asserting titles[0] is the
movie after sorting (a regression there is a title-ordering bug, not a remux issue),
complementing the existing branching-UHD / normal-disc order tests.

Also clears two pre-existing test-only clippy nits surfaced under --all-targets:
unused `lba` in ClearStubReader::read_sectors, and an unneeded `mut` on the
h264 population test's closure.
2026-06-25 21:42:52 -07:00
Matthew Jackson 998e21c544 test: real-executing coverage for mux codecPrivate/DefaultDuration, sweep damage-jump, patch watchdog clock seam, and AACS CBC KAT
mux/mkv: assert emitted CODEC_PRIVATE bytes verbatim for H.264/HEVC/VC-1/
MPEG-2 (direct TrackEntry child, not nested in Video) and DefaultDuration ns
for all eight frame rates, read back out of a real MkvMuxer.

disc/sweep: end-to-end Disc::sweep against a synthetic MockReader with an
injected bad region, asserting the resulting mapfile marks the clean lead
Finished and the failed batch + zero-filled skip-ahead gap NonTrimmed,
proving the Pass-1 damage-jump engaged.

disc/patch: introduce a minimal clock seam (fn() -> Instant on the internal
PatchLoopState, defaulting to Instant::now) so the per-range and whole-pass
watchdogs are deterministically testable; public API and callers unchanged,
production behavior identical. Add tests that advance a fake clock to trip the
range budget and whole-pass stall predicate.

aacs: add an AES-128-CBC known-answer test for aes_cbc_decrypt using the
published NIST SP 800-38A F.2.2 vector (blocks 1..3 exact; block 0 via the
documented fixed-AACS-IV substitution).
2026-06-25 21:37:20 -07:00
Matthew Jackson 7f55271adb mux: reconcile fvi:// video-index sink onto rc6
Port + adapt the freemkv native per-picture video index (FVI) from the
old feat/fvi-sink branch onto rc6's codec-agnostic PictureInfo model.
This is a surgical adaptation, not a merge.

Adaptations (fvi_sink.rs, videomap.rs, tests/fvi_pipeline.rs):
- Retarget from the removed crate::mux::codec::mpeg2::PictureInfo (raw
  public fields) to rc6's authoritative crate::mux::codec::PictureInfo
  in codec/coding.rs, via its accessors.
- type from coding_type() -> CodingType{I,P,B}; emitted for ANY frame
  that carries coding (every video codec now fills it), with the
  keyframe-flag I/P fallback only when coding is absent.
- Replace the mpeg2-only tff/rff/progressive members with codec-agnostic
  members derived through the accessors: field_order (tff/bff/progressive)
  and progressive, emitted ONLY when the codec measured the signal
  (Option::Some) and omitted otherwise; plus nb_fields.
- Test fixtures rebuilt via PictureInfo::mpeg2(CodingType, Mpeg2Coding{..})
  / coding_type_only(..); added measured_cicp: None to VideoStream
  literals for rc6's struct.

Honesty decision (key / random-access):
- The codec-agnostic PictureInfo carries NO GOP-closure (no closed_gop /
  gop_start), so key is set from the frame's intra / decode-restart flag
  (frame.keyframe == coding.keyframe() for video), NOT a fabricated
  clean-RAP claim. The old gop member is honestly omitted. FVI_FORMAT.md
  is updated to document this as a limitation: key is an intra picture /
  parser-flagged decode-restart point; MPEG-2 open-GOP clean-RAP precision
  (closed_gop) is not currently distinguished. §7.1 rewritten for the
  new field_order/progressive/nb_fields members.

Wiring:
- mux/mod.rs: pub(crate) mod fvi_sink; pub(crate) mod videomap
  (#[allow(dead_code)] on videomap — the VideoMap accumulator is staged
  for side-channel reuse, sink builds records directly); pub use
  fvi_sink::FviSink.
- mux/resolve.rs: add the fvi:// output scheme to StreamUrl, parse_url,
  scheme(), path_str(), input() (write-only reject) and output()
  (constructs FviSink), mirroring the mkv:///demux:// patterns.

Provenance fix surfaced by the end-to-end test:
- pipelined_stream::consume_ps was dropping the PS demuxer's byte-exact
  source stamp (source: None) when rebuilding PesPacket, so PS/DVD-path
  frames reached the mux/index with no provenance (FVI src null). Carry
  ps.source through, matching the TS path; the real-pipeline fvi test now
  sees the stamped src sectors.

Gate: cargo +1.86 fmt + clippy --lib -D warnings clean; cargo +1.86 test
--lib (2182 passed) and --test fvi_pipeline (2 passed); precommit.sh
libfreemkv green.
2026-06-25 21:18:15 -07:00
Matthew Jackson e064bc7055 mux/vc1: measure coding type from progressive PTYPE; carry source
Honest PictureInfo population for VC-1 — completes the set (MPEG-2, H.264, HEVC,
VC-1 all populate now).

- Read the advanced-profile sequence header INTERLACE flag (SMPTE 421M §6.1.1,
  bit 41), de-escaping emulation-prevention bytes as parse_vc1_resolution does.
- For a PROGRESSIVE sequence, decode the picture-layer PTYPE VLC (0=P, 10=B,
  110=I, 1110=BI→I, 1111=Skipped→P) from the first bits after the frame start
  code. For INTERLACED (FCM/FPTYPE precede PTYPE) or simple/main/unknown, decline
  → coding stays None: honestly absent, never read at a wrong bit offset.
- Set coding = coding_type_only(...) and source = pes.source; field order is not
  decoded, so field_order() is honestly None.
- Reuses the shared startcode::BitReader (no new primitive).
- Tests: progressive I/P/B from real PTYPE VLCs + source carry + field-order
  absence; interlaced honest-decline.
2026-06-25 20:50:36 -07:00
Matthew Jackson 6acc26a802 mux/hevc: measure coding type from slice_type; reuse shared BitReader
Honest PictureInfo population for HEVC, on the same principle as H.264.

- Consolidate the bit reader: hevc.rs had its own BitReader (used by the SPS
  parser); h264 just gained one in startcode. Promote startcode's to the single
  shared pub(crate) reader (adds read_bits) and delete hevc's copy — one proven
  primitive, reused (SPS parse + both slice-type decoders).
- hevc: decode slice_type from the first coded slice's slice_segment_header
  (H.265 §7.3.6.1) → I/P/B (§7.4.7.1). The offset to slice_type depends on
  num_extra_slice_header_bits, which lives in the PPS — so we parse it from the
  ACTIVE PPS (§7.3.2.3) and only measure slice_type when that PPS is known.
  With no active PPS we decline rather than guess: coding stays None, honestly
  absent. Set coding = coding_type_only(...) and source = pes.source; field
  order (pic_struct SEI) is not decoded, so field_order() is honestly None.
- Tests: I/P/B from real slice headers, source carry, field-order absence, and
  the no-PPS honest-omission case.
2026-06-25 20:44:15 -07:00
Matthew Jackson 534eca502c mux/h264: measure coding type from slice_type; carry source provenance
Honest PictureInfo population for H.264 — the foundation that lets the video
index claim a frame's type and be believed (a faked "P" on a B-frame would make
the index confidently wrong).

- startcode: shared minimal MSB-first BitReader with Exp-Golomb ue(v), reused by
  H.264 (and next HEVC). Documents the emulation-prevention caveat: only the
  leading slice-header fields are read, where 00 00 03 cannot intervene.
- h264: decode first_mb_in_slice + slice_type (H.264 §7.3.3) from the first
  coded slice and map to I/P/B (§7.4.3 Table 7-6; SP→P, SI→I). Set
  coding = PictureInfo::coding_type_only(...) and source = pes.source. Field
  order is NOT decoded here, so field_order() stays None — honestly absent,
  never guessed.
- Tests: I/P/B from real ue-encoded slice headers, source carry, field-order
  absence; BitReader Exp-Golomb table + truncation.
2026-06-25 20:38:09 -07:00
Matthew Jackson e3dbafcebd mux: codec-agnostic PictureInfo + provenance; measure field order, never guess
Carry per-picture truth and byte-exact source provenance THROUGH the stream so
the muxer (and the upcoming video index) read MEASURED facts instead of
assuming them. Honest data in, honest data out.

- codec/coding.rs: codec-agnostic PictureInfo (CodingType / FieldOrder + the
  accessors field_order/coding_type/nb_fields/progressive/keyframe). Each codec
  folds its raw signals in; consumers use only accessors, never branch on codec.
- mpeg2: builds PictureInfo from the picture coding extension and carries it +
  SourcePos (source_marks, parallel to pts_marks) on every emitted frame.
- pes / codec::Frame: additive `coding` + `source`, forwarded through the
  highway; None for audio/subtitle and the network/stdio deserialize hop.
- mkvstream: DEFER muxer construction until the first coded picture, set the
  video track's FieldOrder from the MEASURED value, THEN write the header —
  right the first time, no guess, no seek-back. An interlaced track that arrives
  with no measured order is LOGGED loudly and left UNDETERMINED, never faked.
- mkv: MkvTrack::video no longer guesses TFF (a bitstream property the scan
  cannot know is UNDETERMINED at build). Removed VideoStream::top_field_first
  (the dead scan-time guess) crate-wide.
- Tests: parser population (every PictureInfo facet + per-PES source carry) and
  mux-stream consumption (measured -> correct; missing -> UNDETERMINED, not
  faked). Two obsolete tests updated only after confirming (their own comments)
  they existed to enforce the deleted hardcoded-TFF.
2026-06-25 20:13:55 -07:00
Matthew Jackson 43fb97f71f mux/mkv: spec-conformance fixes (field order, CICP, VobSub idx, SeekHead, AC-3)
- FieldOrder now derives from the bitstream's measured top_field_first
  (Some(true)→TFF, Some(false)→BFF) instead of hardcoding TFF for all
  interlaced content; falls back to TFF when unmeasured. Adds
  VideoStream::top_field_first; DVD/BD scan sets None with a precise
  TODO(spec) for parser→title plumbing.
- CICP (matrix/transfer/primaries/range) now prefers measured CICP from
  the bitstream (VideoStream::measured_cicp) over the coarse ColorSpace
  enum, so the container stops assuming a colour space the stream may
  contradict. Enum remains the fallback.
- VobSub S_VOBSUB CodecPrivate now emits a `size: WxH` line ahead of the
  palette per the .idx format so players place/scale subs correctly.
- SeekHead: when zero cues are written, the CUES Seek entry is Voided
  instead of leaving a dangling pointer to the Cues offset (now Tags/EOF).
- AC-3 Channels back-patch offset is captured from the writer instead of
  the hardcoded chan_elem_pos+2 (decoupled from the VINT width choice).
- Hoisted inline CICP codes and the dvcC fourcc to named constants citing
  ITU-T H.273 / RFC 9559; fixed the stale FieldOrder comment.
- DefaultDuration vs pulldown: precise TODO(spec) left (needs the same
  parser→title channel as top_field_first).

Tests: BFF-from-measured-flag, measured-CICP-overrides-enum, VobSub size:
line present/omitted, zero-cue SeekHead Void. precommit (1.86) green.
2026-06-25 18:40:30 -07:00
Matthew Jackson 8e0797eab0 wip: rc6 VFR/DVD/CSS base (held for bulletproofing + split) 2026-06-25 18:17:08 -07:00
Matthew Jackson 9b6a48e9d9 demux: solidify sink — reuse canonical primitives, fix 3 bugs
Delete re-implementations in the demux:// sink and wire to proven helpers;
keep only genuinely-new functionality.

- AnnexB reframing: delete the sink's local length_prefixed_to_annexb (it
  break'd on a zero-length NAL, dropping the rest of the access unit) and
  call the canonical append_length_prefixed_as_annex_b in mux::hevc, which
  skips just the empty NAL.
- HEVC param sets: delete hvcc_param_sets; reuse hvcc_to_annex_b.
- avcC param sets: hoist as the new canonical avcc_to_annex_b in mux::hevc,
  next to hvcc_to_annex_b (the symmetry point); the sink calls it.
- PGS .sup: emit a synthetic clear display set (empty PCS + END) at
  pts + duration_ns so subtitles time out instead of lingering to EOF.
- TimelineContinuity: move verbatim into the shared mux::timeline module
  (with the prev_offset straggler-remap intact) and use it from both the
  MKV muxer and the demux sink; delete the sink's drifted TimelineRebase
  copy (which lacked the straggler branch).
- VobSub .idx: emit the conventional 'id: <lang2>, index: 0' line mkvmerge
  reads to assign the subtitle language; palette reuse unchanged.
- output(): seed DemuxOptions.base from title.playlist when non-empty.

New constants for the PGS clear-segment framing and avcC header cite the
public HDMV PGS (BD-ROM Part 3) and ISO/IEC 14496-15 specs.

Tests: a zero-length NAL mid-frame no longer truncates the AU; a frame with
duration_ns produces a .sup clear segment; existing demux tests stay green.
2026-06-25 17:58:49 -07:00
Matthew Jackson 8e2e22af5c mux: highway-level regression — DVD video keyframe + duration survive the pipelined path
The DVD seek-index concern is that the mux highway
(PsDemuxer -> PipelinedPesStream codec parse -> frame out -> muxer)
might drop the keyframe flag or per-frame duration that Mpeg2Parser sets
on each Frame, which would stop the muxer's cluster/cue open
(keyframe && track 0) from ever firing and leave a DVD MKV with many
clusters and zero cues.

Add dvd_highway_preserves_video_keyframe_and_duration: drives a real
Mpeg2Parser through PipelinedPesStream via DemuxBatch::Ps batches (one
PTS-stamped MPEG-2 PS video PES per GOP, decode-order I + P/B), reads the
frames back through the highway's read(), and asserts the GOP-opening
I-frames arrive as keyframes, every frame keeps its duration (BlockGroup
path), and video routes to track 0. Guards from_codec_frame's
keyframe/duration propagation across the demux-thread + recycled-buffer
highway, which the prior cue tests (codec parser straight into the muxer)
did not exercise.

Verified against the real current-VFR-build output
(Greenland-feature.mkv, today): 3365 clusters / 3365 CuePoints, all
BlockGroups, cues resolve to clusters and track the video — the highway
preserves the flags and the seek index is complete.
2026-06-25 16:58:04 -07:00
Matthew Jackson b2e1698b9a mkv: regression test — DVD VFR/BlockGroup output gets per-cluster cues
The existing cue tests (cue_count_equals_cluster_count,
cue_positions_resolve_to_clusters, cue_times_match_cluster_timestamps)
all feed frames with duration_ns=None, so they exercise only the
SimpleBlock write path (UHD/HEVC). DVD MPEG-2 video is now VFR: every
coded picture carries a per-frame duration_ns=Some(..), so it is written
as a BlockGroup, not a SimpleBlock. That cue path was untested.

Add cue_count_equals_cluster_count_blockgroup_vfr, which drives the real
Mpeg2Parser end-to-end (decode-order frames, non-monotonic B-frame
display PTS, telecine field durations) into the muxer and asserts the
output is wholly BlockGroup (no SimpleBlock) and that the Cues index has
exactly one cue per cluster, with every cue resolving to a real cluster.
Guards the DVD seek index (scrub/fast-forward) against regressing to the
chapter-seek-only, zero-cue state.
2026-06-25 16:32:32 -07:00
Matthew Jackson 730af6b1d9 demux: add demux:// per-track elementary-stream sink
New write-only pes::Stream sink that taps the per-track PesFrame stream
(the seam right before MKV muxing) and writes each track to its own
elementary-stream file, plus chapters and per-audio-track delay metadata.
Purely additive — the MKV mux path is untouched.

- mux/demux_sink.rs: DemuxSink + EsWriter dispatch. Pass-through for
  codecs whose Frame.data is already standalone ES (MPEG-2, VC-1, AC3/
  E-AC3, DTS/DTS-HD, TrueHD, LPCM). Non-trivial writers:
  - AnnexBWriter: reframes hvcC/avcC 4-byte-length-prefixed NALs to
    Annex-B and prepends VPS/SPS/PPS parsed out of the codec_private
    configuration record (HEVC .hevc / H.264 .h264).
  - PgsSupWriter: rebuilds the HDMV 'PG' segment framing the parser
    strips, with 90kHz PTS/DTS (.sup).
  - VobSubWriter: writes raw SPUs to .sub and synthesizes the .idx
    sidecar (palette + per-SPU timestamp/filepos).
  - Delay-in-filename (mkvmerge-readable 'DELAY <n>ms') + chapter XML/OGM
    export. TimelineRebase ports the MKV muxer's seamless-branch epoch
    logic so per-track ES timestamps stay continuous across clip joins.
- mux/resolve.rs: StreamUrl::Demux variant + scheme/path_str/parse_url/
  input(write-only)/output arms.
- mux/mod.rs: module + public type re-exports.

16 unit tests: Annex-B reframing, hvcC/avcC param extraction, delay
sign/rounding + mkvmerge-regex match, PGS .sup framing, VobSub .idx
synthesis, chapter XML/OGM, timeline rebase, and end-to-end file-keying
by track + track selection.
2026-06-25 16:23:37 -07:00
Matthew Jackson 52d2e85e3c css: skip clear/uncrackable extra titles instead of failing the whole mux
A genuinely-clear or uncrackable extra title (a tiny menu/nav stub) no
longer poisons a multi-title rip with a false CssKeyMissing (E7023).

- decrypt_keys_for_title_checked: re-crack a non-overlapping VTS via
  crack_key_outcome and report title_is_clear when the title's own
  extents show no scrambling. A genuinely-clear stub on an otherwise-CSS
  disc needs no key.
- ensure_title_decryptable: pass a clear stub without a key; a scrambled-
  but-uncrackable title still hard-fails with CssKeyMissing.
- is_scrambled_pack: hardened scramble-evidence gate for the crack scan —
  requires the MPEG-PS pack-start signature before trusting the 0x14
  scramble bits, so a clear stub with stray 0x14 bits can't flip
  saw_scrambled. The descramble loop keeps the looser is_scrambled.
- mux/resolve: ISO per-title gate routes through the clear-aware check.
2026-06-25 13:43:04 -07:00
Matthew Jackson 6b3014f3e8 progress: expose bytes_retryable so 'lost' counts only failed reads
Add PassProgress::bytes_retryable_total (NonTrimmed/NonScraped — failed
and awaiting retry), distinct from bytes_pending_total which also folds
in not-yet-attempted (NonTried) bytes. Set it at every construction site
(Sweep from the snapshot, Patch from stats, 0 for sequential/placeholder
paths). The disc-level 'lost' display in the CLI can now use
unreadable+retryable instead of unreadable+pending, so a healthy
in-progress rip no longer reports its unread remainder as lost.
2026-06-25 13:04:37 -07:00
Matthew Jackson 8f5968a18f Resume plain disc->iso copy after interrupt
Disc::copy gated its entire mapfile-resume dispatch behind
`if opts.multipass`, so a plain (non-multipass) `disc:// iso://`
copy always called sweep_internal(resume=false) — wiping the
mapfile + ISO and re-sweeping from sector 0. The CLI help and
rip_iso examples promise "auto-resumes if interrupted", and the
sweep already flushes a crash-safe mapfile per block, so the only
missing piece was consulting it on re-run.

Lift the mapfile-resume dispatch out of the multipass guard:
clean->no-op, size-mismatch->fresh sweep, NonTried tail->resume
sweep now apply to both modes. The patch (Pass N) dispatch on
retryable bytes stays multipass-only; a plain copy with a
fully-attempted mapfile returns a terminal result instead of
restarting. autorip (always multipass=true) is unchanged.

Add a regression test that an interrupted plain copy resumes the
NonTried tail and does NOT re-read the Finished prefix.
2026-06-25 08:24:00 -07:00
Matthew Jackson 62d2dfe96a libfreemkv: align E7021→E7017 in comments/test-names (keys-but-no-VID is AacsVidUnavailable) 2026-06-25 07:59:54 -07:00
Matthew Jackson 3c42950ecd libfreemkv: changelog — rc.6 entry (+ backfill rc.5.3, mark rc.5.2 released) 2026-06-24 23:56:41 -07:00
Matthew Jackson 0127c274d2 test: strengthen AACS resolve-reason + unlocker seam coverage (rc.6 WS3)
Targeted tests for the rc.6 surfaces, strengthening (not duplicating) the
regression tests the rc.6 commits already shipped. No production code changes.

aacs/keys.rs — resolve_keys_with_reason / classify_resolve_failure:
The E7021/E7022 split is already proven end-to-end through the
ensure_decryptable gate (disc/mod.rs). These pin the classifier directly at
the keys.rs seam for the branches the gate test does not reach:
  - processing-keys-only + zero VID -> VidUnavailable (the gate test only
    exercises the device-keys arm of has_derivation_material).
  - VID PRESENT + material -> NoMaterial: a non-zero VID must never be
    reported as VidUnavailable however much material is on hand (the has_vid
    short-circuit; the gate test only uses the zero-VID sentinel).
  - VID present + no material -> NoMaterial.
  - version dispatch: version 1 routes the V10 resolver (stamps V10), any
    other value routes the V20->V21 chain; a resolved disc returns Ok, never
    Err(ResolveFailure).

unlock.rs — Unlocker seam introspection + ordering:
  - matching_name reports the first matching unlocker without running it, and
    is None for an unsupported drive; registered_count grows after a
    registration (monotonic check — the registry is process-wide and shared
    across the unlock tests, so no exact-delta assertion).
  - route_unlock first-registered-match-wins: two unlockers matching the same
    identity, the earlier-registered one runs and the later is never consulted.
2026-06-24 23:40:36 -07:00
Matthew Jackson f407c4c693 fix(extract): anchor AACS unit base per-extent + crash-safety hardening
The dir:// extractor set the AACS unit-alignment base ONCE to the first
extent's start, then read every extent against that single base. For a
multi-extent (fragmented / Long-AD / continuation-ICB) file the second
and later extents start at arbitrary LBAs whose offset from the first
extent is generally not a multiple of 3 sectors, so the first read of
each later extent failed the decrypt-on-read gate
(is_unit_aligned(lba, unit_base)), returned DecryptFailed, and recorded
the whole extent as a zero-filled hole even though the data was readable.
Re-anchor the unit base PER extent (matching mux/disc.rs and
sector/prefetched.rs), so each clip's encrypted region gates on its own
unit grid. Same bug class as the rc.5.2 clip-anchor fix.

Also harden the extract + keydb write paths:
- finalize_file: fsync the .partial after set_len (the truncation runs
  on a second handle the content fsync never touched) and fsync the
  parent dir after rename so the new dirent is crash-durable.
- keydb write_atomic: fsync the parent dir after rename (POSIX dirent
  durability), matching the finalize_file pattern.
- AACS tail batch: document that decrypt_sectors' trailing-partial
  contract already handles the short final unit; no math change.
- decrypt-loss delta loads use Acquire (defensive happens-before if
  file extraction is ever parallelised).
- is_windows_reserved: add CONIN$/CONOUT$/CLOCK$; reserved names are
  now substituted (prefix _) instead of aborting the whole tree walk, so
  a legal Linux-authored NUL.cfg extracts.
- http_get header cap: >= MAX_HEADER_BYTES (was > , one byte over).

Regression tests: multi-extent AACS file (Δ4-sector extents) extracts
both extents with zero loss; focused per-extent alignment-arithmetic
test; reserved-name substitution assertions.
2026-06-24 23:20:15 -07:00
Matthew Jackson d4a0f5b786 aacs: split no-key resolution reason into E7021 vs E7022
When key resolution had derivation material (device or processing keys)
but no Volume ID was available to derive the unit key, surface
Error::AacsVidUnavailable instead of the generic NoDiscKey. When there
was no usable key material at all, keep NoDiscKey.

resolve_keys_classical / resolve_keys_v21 still return a bare
Option<ResolvedKeys> (all existing callers unchanged); a new
resolve_keys_with_reason wrapper threads the typed ResolveFailure
(VidUnavailable | NoMaterial) out. decrypt_with uses it; the
ensure_decryptable_keys gate maps a captured AacsVidUnavailable reason to
E7021, otherwise E7022. No decryption math, key derivation, or descramble
logic changed -- only the reason reported on a resolution failure.

Adds ensure_decryptable_aacs_vid_unavailable_vs_no_key proving both
branches (device-keys + zero VID -> E7021; no keys -> E7022).
2026-06-24 22:38:54 -07:00
Matthew Jackson 1854869ab3 v1.0.0-rc.5.3: bump version 2026-06-24 21:50:00 -07:00
Matthew Jackson 275d9eebe0 libfreemkv: plain-English, source-agnostic AACS scan logs
Rewrite the AACS scan/VID trace lines so a reader understands them without
opening the source: name the real thing (AACS host certificate, Volume ID,
decryption key), say "key source" not "keydb", and describe what happened.
The VID flow is unchanged (unlocker OEM VID → cert handshake → continue); a
missing VID is logged, never fatal. All strings are in tracing macros (the
sanctioned debug-log channel) — no English added to any Error.
2026-06-24 21:47:37 -07:00
Matthew Jackson 05b9befc64 libfreemkv: keydb default location is local to the executable
default_path() now resolves <dir of current exe>/keydb.cfg with no OS-config
fallback — freemkv is a portable standalone binary, so the keydb lives next to
it. --keydb still overrides.
2026-06-24 20:46:31 -07:00
Matthew Jackson 1767cf67b6 libfreemkv: dir:// — decrypted file-tree extraction (Disc::extract_tree)
Sibling of Disc::copy specialized to write per-file instead of a whole ISO
image, decrypting on the way out: walk the UDF tree, read each file's extents
through the shared DecryptingSectorSource (AACS unit-aligned, CSS per-VTS),
strip AACS/, sanitize host paths per component, .partial+rename, 1-shot with
per-file loss accounting (no mapfile; recovery stays the iso:// multipass
path). Reuses UdfFs + the decrypt seam; only the per-file orchestration is new.
2026-06-24 20:46:31 -07:00
Matthew Jackson 9a3f6b7313 libfreemkv: hard-error when decryption is needed but no key is available
Adds Disc::ensure_decryptable / ensure_decryptable_keys, the single decrypt
gate consulted before any copy or mux. When the source is encrypted and no
key resolved (and not --raw), abort with a typed error and write nothing,
instead of silently emitting ciphertext at exit 0. Unifies the prior ad-hoc
CSS/AACS checks.
2026-06-24 19:33:09 -07:00
Matthew Jackson 1f91eebb9a v1.0.0-rc.5.2: bump version 2026-06-24 17:14:10 -07:00
Matthew Jackson f4a475c7b9 libfreemkv: rc.5.2 changelog — audio routing, AACS clip-anchor, recovery, TopGun, keysources move 2026-06-24 17:12:17 -07:00
Matthew Jackson 5b0976859f libfreemkv: rc.5.2 SOTL video — full Windows fps, opening-GOP proof, self-sufficient log-level 3
Three Silence-of-the-Lambs (R2 PAL SD-DVD) follow-ups for rc.5.2.

SUB-TASK 1 — Windows Explorer showed 12.5 fps (half) for the 576i25 track.
Root cause: the DefaultDecodedFieldDuration (20 ms field) element rc.5.1
added to "fix" Windows fps did the opposite. With FlagInterlaced=1 +
DefaultDuration=40 ms + DefaultDecodedFieldDuration=20 ms, Explorer halved
to 12.5 fps and MediaInfo flipped to VFR. MakeMKV's correct rip omits the
field-duration element, keeps FlagInterlaced=1 + FieldOrder=TFF +
full-frame DefaultDuration (40 ms), and Explorer shows 25 fps / MediaInfo
CFR. Fix: MkvTrack::video now passes field_duration_ns == 0 so the element
is no longer written; the 1/DefaultDuration = 25 fps signal (the only one
tools trust) is the full-frame value. Interlace signalling (FlagInterlaced,
FieldOrder=TFF) is retained — MediaInfo reads scan type from the MPEG-2 ES
picture coding extension, so it still reports Interlaced / Top Field First.
Tests pin the new TrackEntry elements (element present/absent + values).

SUB-TASK 2 — opening "menu"/still-frame video. Traced the MPEG-2
opening-GOP path; the wrong/last seq header and PTS-floor-to-0 hypotheses
are RULED OUT with file:line evidence: codecPrivate is the FIRST sequence
header (read once at headers-ready, mkvstream.rs:115 + pipelined_stream.rs:289),
DVD VOBU structure guarantees each title opens on seq header + I-frame (no
mid-GOP open), the parser back-anchors leading still-frames to the disc's
real timeline (mpeg2.rs:296-303), and the muxer anchors base on the opening
keyframe's real PTS so the t=0 floor (mkv.rs:963) never corrupts it.
Regression tests pin all three (parser + muxer level).

SUB-TASK 3 — make --log-level 3 self-sufficient (diag.rs + minimal hooks).
(a) dump the ACTUAL MKV TrackEntry elements written per track
(tag=mkv.track: FlagInterlaced, FieldOrder, DefaultDuration, field duration,
Display dims, codecPrivate hex) so Windows-fps-class metadata is verifiable
from a log alone. (b) capture the first ~100 coded frames per track (raw)
to <output>.opening.bin with a per-frame summary line (tag=mkv.opening.frame:
track, key/delta, size, PTS) so opening-GOP/menu issues are diagnosable from
a future log without the disc. Both gated to log-level 3; normal runs open
no side file and record nothing.

CI gate (Rust 1.86): fmt --check, clippy -D warnings, and test --tests all
green.
2026-06-24 17:04:56 -07:00
Matthew Jackson f72a956b5b fix(dvd-audio-probe): read each sub-stream's true max channel count
The DVD AC-3 sub-stream probe recorded the FIRST decodable frame of each
physical 0x8x sub-stream as its channel count. A DVD feature opens with
logos/warnings whose audio is often a thin 2.0 bed on 0x80 before the
real 5.1 main mix begins a fraction of a second later. The probe locked
onto that opening 2.0 frame and reported 0x80=2, missing the 5.1
entirely (confirmed on Greenland: 0x80's head frames are acmod=2, then
acmod=7+lfe). With no 6-channel sub-stream found, channel-match routing
fell back to the ordinal map — harmless on Greenland, but on a disc where
the 5.1 lives on a non-ordinal sub-stream the wrong-substream bug stays
unfixed.

Fix: scan EVERY 0x0B77 frame of each sub-stream in the probe window and
keep the MAXIMUM channel count (the sub-stream's real main-mix
capability), advancing frame-by-frame via ac3_frame_size so a frame body
can't be mistaken for a new sync. Also bump PROBE_SECTORS 512->1024: the
1 MiB head window saw ONLY 0x80; 2 MiB reliably contains a frame of every
physical sub-stream.

Greenland tag=dvd.substream: before 0x80=2 (only); after 0x80=6, 0x81=2,
0x82=2 — matching the IFO and the decoded output.

Adds probe_reads_max_channels_no_cross_contamination regression test.
2026-06-24 16:55:34 -07:00
Matthew Jackson 9cd36427be libfreemkv: fix rc.5.2 audit code findings
1. HEVC CRA->BLA false-trigger on 33-bit PTS wraparound
   (src/mux/codec/hevc.rs): the clip-boundary auto-detect compared the
   RAW 33-bit PES PTS against the high-water mark, so a single-clip title
   crossing 2^33->0 (~26.5h) false-armed pending_clip_boundary and rewrote
   a legitimate in-clip CRA(21)->BLA_W_LP(16), dropping valid RASL pictures
   (visible corruption) and breaking the single-clip byte-identical
   guarantee. Now unwrap the PTS onto a monotonic 64-bit timeline first
   (a near-full-period backstep is a wrap: add 2^33, update the watermark,
   do not arm). Regression test cra_after_33bit_pts_wrap_not_rewritten;
   the genuine-clip-join test still passes.

2. Single-pass recovery read bypassed the transport-failure abort
   (src/mux/disc.rs): the line-442 short-circuit only inspected the 10s
   read res. A transport failure (status 0xFF, wedged USB bridge) on the
   60s recovery read fell into the skip_errors branch and zero-filled/
   advanced, marching the disc at one bridge-recovery per probe
   (run-forever, hard rule #2). Re-check the recovery error for
   is_scsi_transport_failure() before the skip block and abort with
   Error::DiscRead. Test transport_failure_on_recovery_read_aborts_even_with_skip_errors.

3. Recovery-read SUCCESS branch had no coverage (src/mux/disc.rs tests):
   added RecoverableReader (errors when recovery=false, succeeds when
   recovery=true) and test recovery_read_success_muxes_recovered_data_no_skip
   driving fill_extents to the size-1 bottom-out and asserting the recovered
   data is muxed (counters advance, no skip).

4. TrueHD channel-correction probe omitted set_unit_base
   (src/disc/mod.rs correct_truehd_channels): the probe read via a
   DecryptingSectorSource without anchoring the AACS unit-alignment gate,
   so it degraded to absolute start_lba % 3 and returned DecryptFailed on a
   non-3-aligned extent, silently understating Atmos/7.1 as 5.1. Now call
   set_unit_base(ext.start_lba) before the probe read (no-op for CSS/None).

5. is_unit_aligned lba<unit_base latent trap (src/aacs/decrypt.rs):
   wrapping_sub mis-gated when lba < unit_base (2^32 == 1 mod 3). Switched
   to saturating_sub (clamps offset to 0, a unit boundary) and pinned the
   contract with is_unit_aligned_lba_below_base_is_well_defined plus
   is_unit_aligned_relative_to_base.

cargo +1.86 fmt --check / clippy -D warnings / test --tests all green.
2026-06-24 16:31:28 -07:00
Matthew Jackson 674a7dd867 dvd: route AC-3 audio to the physically-correct sub-stream by probed channel count
Fixes the "Silence of the Lambs" R2 PAL wrong-substream rip: the feature's
IFO declares one 5.1 AC-3 stream, but the scan assigned it the on-wire
sub-stream id 0x80 purely by per-codec ordinal (ifo::assign_audio_sub_stream_ids).
On this disc the physical 0x80 carries the 2.0 down-mix and the 5.1 main mix
lives at a different 0x8x sub-stream, so the rip muxed 2.0 while labelling it
"Dolby Digital 5.1" (the acmod fixup in mkv.rs then corrected only the Channels
element, surfacing the mismatch as the "IFO claimed 6 but acmod says 2" warning
— too late to re-route).

New src/disc/dvd_audio_probe.rs probes each physical AC-3 sub-stream's real
channel count from the head of the feature (the acmod/lfeon of its first frame
after the 0x0B77 sync) and re-routes each IFO-declared AC-3 stream onto the
physical sub-stream whose actual channel count matches the declared count,
instead of trusting the ordinal. Wired into both mux demux paths
(DiscStream::new and resolve::build_iso_pipeline) over the decrypting reader,
so it works on CSS discs and the autorip ISO-remux path alike. Bounded
512-sector best-effort read; an empty/unreadable probe degrades to the original
ordinal mapping (no regression on normal discs).

The cell selection is left unchanged: the feature's cell 0 (cat=0x02, 302.4s)
is chapter 1 of the movie (matches MakeMKV's chapter map and 1h53 duration
exactly), so it must NOT be dropped — the perceived "wrong video at the start"
was the wrong 2.0 audio over the opening, the same root cause.

Diagnostics (--log-level 3): new tag=dvd.substream rows dump the ACTUAL acmod
channel count of each physical 0x8x sub-stream read from the VOB, and the
per-cell tag=dvd.cell verdict now spells out the keep/skip reason. With the
existing tag=dvd.aattr (IFO declared sub_id + channels) a bug log alone now
shows whether the ordinal 0x80 really carries the declared layout — no disc
needed to diagnose this class.

expose ac3::find_ac3_sync as pub(crate) for the probe.
2026-06-24 16:28:21 -07:00
Matthew Jackson 1cec2aaaf3 mux/hevc: auto-detect non-seamless clip boundary, rewrite splice CRA→BLA
The "TopGun bug" (Top Gun 1986 UHD, DV Profile 7 dual-layer): the auditor
flags the rip `corrupt` with a flood of HEVC "Could not find ref with POC N"
decode errors (POC 114/210/228/234/240/246/252/318/336/354/372/399 — one
cluster per clip join).

Root cause: the title is multiple .m2ts clips joined at non-seamless
boundaries (mpls connection_condition 0x05/0x06), read as one concatenated
stream. Each next clip opens with a CRA whose RASL leading pictures reference
pre-join frames gone after concatenation. hevc.rs already has the spec remedy
(rewrite splice CRA_NUT 21 → BLA_W_LP 16 so a decoder sets NoRaslOutput and
drops the dangling RASL), behind mark_clip_boundary() — but nothing ever
called it: connection_condition is not plumbed through the threaded mux
pipeline, so the whole CRA→BLA mechanism was dead code.

Fix: detect the boundary inside the parser from the bitstream. Each clip
carries its own PES PTS base, so a non-seamless join is a large backward
PTS reset. HevcParser::parse now tracks a PTS high-water mark and, on a
backward step beyond 3 s (270000 ticks @ 90 kHz — mirroring the mux-side
DISCONTINUITY_BACKSTEP_NS), arms the existing CRA→BLA rewrite for the new
clip's first IRAP. Self-contained: works for every mux path, no cross-thread
plumbing. The 3 s threshold sits above any HEVC B-frame reorder dip and far
below any clip duration, so it never false-triggers in-clip; single-clip /
seamless titles never arm it and stay byte-identical.

Adds regression test cra_at_auto_detected_pts_backstep_rewritten_to_bla
(in-clip dip must not trigger; splice CRA after a backward reset must become
BLA; one-shot). hevc suite 55/55, mux suite 780/780.
2026-06-24 15:41:26 -07:00
Matthew Jackson 794d88f6e7 libfreemkv: rc.5.2 DVD test coverage — depth-aware mux, colour codes, CSS scan
Implements the rc.5.2 quick-units list from the DVD coverage audit and
corrects the "passes-but-encodes-the-bug" tests that could not
distinguish correct from wrong behaviour.

New tests (each with the bug it guards):

mux/mkv.rs
- field_duration_is_direct_trackentry_child_not_in_video: depth-aware
  check that DefaultDecodedFieldDuration (and DefaultDuration) are direct
  TrackEntry children, NOT nested in the Video master. Replaces the flat
  find_id byte-scan that passed either way. Adds master_children /
  first_track_entry depth-walking helpers.
- pal_576i_emits_bt470bg_colour_codes / ntsc_480i_emits_smpte170m_colour_codes:
  assert the actual CICP tuples written into the MKV Colour master —
  PAL (5,5,5,1) vs NTSC (6,6,6,1) — not just stream-layer ColorSpace.
- ntsc_480i_field_order_is_tff_and_encoded: pins NTSC 480i hardcoded TFF
  and its ~33.37ms/16.68ms frame/field durations, asserting the encoded
  FlagInterlaced/FieldOrder bytes (480i was never exercised before).

ifo.rs
- video_attr_absolute_bytes_pin_real_layout: drives parse_video_attr with
  HARDCODED real DVD-Video bytes (PAL/NTSC x 4:3/16:9, plus mpeg_version
  in bits 7-6) instead of v_atr_byte, so a co-edit of the shift constants
  can't re-seed the PAL-as-NTSC bug. Anchors that permitted_df bits (1-0)
  are not read as the TV system.

disc/dvd.rs
- scan_dvd_titles_lpcm_routes_to_a0_pid_range: LPCM (coding 4) → sub-id
  0xA0 → PID 0xBDA0, disjoint from the AC-3 0xBD8x space, channels kept.
- scan_dvd_titles_multiple_vobsub_tracks_distinct_pids: three VobSub
  tracks → distinct 0x20+ordinal PIDs, per-language, shared palette.

css/mod.rs
- crack_outcome_reaches_cracked_with_span: drives the full crack scan to
  CrackOutcome::Cracked via a Stevenson-crackable synthetic sector and
  asserts crack_span recording (the Cracked branch was never exercised).
- recrack_succeeds_on_other_vts_extents: per-VTS re-crack SUCCESS path.
- all_locked_synthetic_iso_yields_css_key_missing_signal: all-locked
  multi-extent ISO → ScrambledUncracked, the signal the scan converts to
  css_error = Some(CssKeyMissing).

Corrected fixtures (passes-but-encodes-the-bug):
- scan_dvd_titles_mixed_audio_codecs_distinct_pids: real channel nibbles
  (AC-3 5.1 = 6ch, DTS 2.0 = 2ch) replacing the 1ch placeholders; asserts
  channel counts and exact canonical PIDs (0xBD80 / 0xBD88).
- ebml.rs FieldOrder comment: drop the stale "PAL DVD (576i) is
  bottom-field-first" line that contradicted the TFF-for-all code.
2026-06-24 15:41:26 -07:00
Matthew Jackson 63ed05bd63 libfreemkv: clip-anchored AACS unit gate + consolidate key mechanism
The AACS unit-alignment gate measured `lba % 3` against absolute disc LBA 0,
but aligned units are anchored at each clip's encrypted-region start. A clip
whose start_lba is not 3-aligned had its readable units wrongly rejected with
"Decryption failed" (the big-title-only failure on some Blu-rays). One
canonical clip-anchored helper (`aacs::is_unit_aligned`) is now the single
source of truth for the decrypt-on-read gate; both mux read paths set the
per-extent `unit_base = start_lba` via a new `SectorSource::set_unit_base`.

Also moves key *mechanism* into the library: the encrypted sample reader
(`read_encrypted_units`) and the candidate-key resolution loop
(`resolve_and_apply`) now live here, so a key source is purely a lookup.
Regression test covers a clip based at a non-3-aligned LBA.
2026-06-24 15:40:50 -07:00
Matthew Jackson 987e26e44d libfreemkv: single-pass disc->MKV recovers marginal sectors before giving up
Single-pass disc->MKV has no Pass N, so its read bottom-out now issues one
bounded recovery read (recovery=true, ~60s ECC) before skipping or aborting,
matching the multipass patch. Fixes a transient/marginal sector surfaced as a
read failure direct-to-MKV while multipass recovered it. One read, not a loop
(hard rule #2); recovered data is used so no bogus-status hole reopens.
2026-06-24 14:58:32 -07:00
Matthew Jackson b76e9d38c5 v1.0.0-rc.5.1: bump version 2026-06-24 14:36:29 -07:00
Matthew Jackson 6592f2a590 libfreemkv: rc.5.1 DVD correctness fixes
- CSS: unlock scrambled-sector reads on enforcing drives via bus-auth
  only; classify sense 6F/03 as CSS-locked; early-bail on a fully locked
  scan; gate the AACS handshake off DVD discs.
- DVD first-play menu no longer prepended to the feature: read the title
  VOBS base from vtstt_vobs (0xC4), not the menu VOBS vtsm_vobs (0xC0).
- Interlaced field-duration (DefaultDecodedFieldDuration) written as a
  direct TrackEntry child rather than inside Video, so Windows reports
  the correct frame rate.
- Audio channel count read from the AC-3 bitstream; FieldOrder set to
  TFF; per-track BPS tags.
- Structured disc diagnostics at --log-level 3; reduced per-operation
  log spam.
2026-06-24 14:34:55 -07:00
Matthew Jackson 315276dd13 Add Contributor Covenant v2.1 Code of Conduct 2026-06-24 10:44:58 -07:00
Matthew Jackson 9d40da3982 v1.0.0-rc.5: bump version 2026-06-24 10:27:30 -07:00
Matthew Jackson e0ce035765 Preserve interlaced scan type in label and MKV output
Interlacing is detected upstream (PAL DVD -> R576i) but was dropped in
two places: the video label hardcoded a 'p' suffix, and the muxer never
wrote any scan-type flag, so MediaInfo inferred progressive and reported
576p for a 576i source.

- Add Resolution::is_interlaced() for the R*i variants.
- generate_video_label now branches i/p for the heights that can be
  interlaced (1080, 576, 480) instead of always emitting 'p'.
- MkvTrack carries interlaced + field_order; the video serializer emits
  FlagInterlaced (0x9A; 1=interlaced, 2=progressive) and, for interlaced
  content, FieldOrder (0x9D) - bottom-field-first for PAL 576i,
  top-field-first otherwise. Adds the EBML constants.
2026-06-24 09:47:16 -07:00
Matthew Jackson 2613a81f09 Fix AC-3 channel count read from wrong nibble in VTS audio attr
The VTS_AST_ATR byte 1 stores (channels - 1) in the low 3 bits, but we
were reading the high nibble. A normal 2.0 AC-3 track has 0x1 in the low
nibble and 0x0 in the high nibble, so we computed 0+1=1 and mislabeled
every stereo track as mono in the muxed Channels element.

Read the low 3 bits instead, and correct the two audio_attr tests that
had encoded (channels - 1) in the high nibble to match the buggy reader.
2026-06-24 09:39:02 -07:00
Matthew Jackson 45a7b74ab9 fix: correct stale FAIL_PAUSE_SECS doc claiming uniform sweep+patch application 2026-06-24 05:57:39 -07:00
Matthew Jackson 8b8ada7802 fix: drop BytePrefetcher channel endpoints before join to prevent deadlock
Drop rx and recycle_tx (now Option fields) before joining the producer
thread in BytePrefetcher::Drop. Without this, a non-EOF source fills the
depth-2 forward channel then spins in send_timeout(POLL_INTERVAL) forever
because rx is never drained; join() deadlocks. Adds a regression test
(drop_endless_prefetcher_joins_cleanly) that directly exercises the path.
2026-06-24 05:50:58 -07:00
Matthew Jackson cdd11ccb25 fix: correct stale doc on PatchItem::Unreadable — promotion shipped via Mapfile::record, not PatchSink 2026-06-24 05:02:24 -07:00
Matthew Jackson c706a94312 fix: correct stale css::crack comment reference to css::crack_key 2026-06-24 04:57:47 -07:00
Matthew Jackson 06ee748689 fix: remove stale sweep_pipeline.rs cross-references in patch.rs comments 2026-06-24 04:52:04 -07:00
Matthew Jackson 95b9762c51 Revert "fix: discard duplicate TS packets in non-PUSI continuation path"
This reverts commit f3c3614a17.
2026-06-24 01:59:49 -07:00
Matthew Jackson fb13f975df fix: enforce WEDGE_FAMILY_COOLDOWN_SECS == ZONE_ENTRY_COOLDOWN_SECS at compile time 2026-06-24 01:52:43 -07:00
Matthew Jackson dbc2225315 fix: correct Profile-07 comment — prime_cache(lba) reads lba-3..lba-1, not lba itself 2026-06-24 01:51:21 -07:00
Matthew Jackson ee0c7cebe3 fix(vc1): assemble keyframe prefix in fixed seq-then-entry order
When a keyframe AU carried an unchanged seq_header (stripped) but a
redefined entry_point (appended), the old append-then-reassert path
produced [entry_point, seq_header] — entry_point before seq_header,
violating SMPTE 421M which requires seq+entry before every RAP.

Replace the single shared prefix Vec + reassert_active() with per-type
temporaries (redefined_seq / redefined_ep) collected during the scan,
then assembled in canonical seq-then-entry order at keyframe time.
Non-keyframes still emit only genuine redefinitions, also seq-before-ep.
Removes the now-unused reassert_active() helper. Adds a regression test
covering the seq-unchanged / entry-redefined trigger case.
2026-06-24 01:47:51 -07:00
Matthew Jackson f3c3614a17 fix: discard duplicate TS packets in non-PUSI continuation path 2026-06-24 01:45:39 -07:00
Matthew Jackson c4f0566fb1 fix: correct comment — patch leaves middle NonTrimmed, not NonTrimmed/Unreadable 2026-06-24 01:11:39 -07:00
Matthew Jackson 77f67dd3ed fix: correct doc comment in patch pipeline test to say NonTrimmed not Unreadable 2026-06-24 01:10:51 -07:00
Matthew Jackson 7f195be894 fix: remove stale TDD-red comment from BytesRead emission test 2026-06-24 00:50:21 -07:00
Matthew Jackson 48e95a7b2c fix: correct hevc.rs doc comments — non-seamless BD join is connection_condition 0x05/0x06 not 0x01
Comments at lines ~63 and ~102 misidentified 0x01 (first-item/seamless) as the
non-seamless trigger and labelled 0x05/0x06 as seamless — inverted vs the BD-ROM
spec and mpls.rs (which documents 1=seamless, 5/6=non-seamless). Corrected all
affected doc blocks; no logic change.
2026-06-24 00:49:22 -07:00
Matthew Jackson 6268f6e5d9 fix: use trim_mkb in resolve_vid_only to avoid zeroing unrecognised MKB 2026-06-24 00:48:02 -07:00
Matthew Jackson 4c50ca2122 fix: correct stale comment in patch.rs work-list ranges_with call
The comment at line 404 claimed "every non-Finished range" but the
immediately-following ranges_with call lists only NonTrimmed,
NonScraped, and Unreadable — deliberately omitting NonTried.
Update the comment to accurately reflect the actual status list and
explain that NonTried is excluded because it is handled by a preceding
sweep pass, not by patch.
2026-06-24 00:09:13 -07:00
Matthew Jackson a324e5c62f fix: correct module doc — only Pass 1 routes through handle_read_error, not Pass N 2026-06-23 23:34:02 -07:00
Matthew Jackson dc7ab01907 fix: update stale doc comment in ReadCtx::for_patch — sync is automatic 2026-06-23 22:51:47 -07:00
Matthew Jackson 55849edd99 fix: correct skip_sectors_for_probe doc comment (8x per index, not 2x per 3) 2026-06-23 22:50:40 -07:00
Matthew Jackson af3666ff3b fix: correct module doc watchdog constant name in patch.rs 2026-06-23 22:49:42 -07:00
Matthew Jackson b82075b41a Fix rc5 audit findings: keydb doc, pipeline ordering, hot-loop Arc, tests
- keydb.rs: separate default_path()/no_home_dir() doc blocks; correct the
  false XDG lock-step claim (Linux write path uses $HOME, ignores
  XDG_CONFIG_HOME; read-side search also checks XDG_CONFIG_HOME).
- io/pipeline.rs: use Release/Acquire on the abandoned flag so a leaked
  consumer reliably skips close() on weak memory models (ARM64/POWER),
  not just x86 TSO.
- mux/disc.rs: cache the decrypt-loss Arc at construction; lost_bytes()
  no longer clones an Arc per frame on the mux hot path.
- disc/dvd.rs: assert display_aspect mapping for both 16:9 (PAL test) and
  4:3 (NTSC test).
- mux/resolve.rs: extract css_error_aborts() helper and unit-test the
  scrambled-but-uncracked CSS guard (Fix 6) incl. the --raw exemption.
- aacs/keys.rs: add unit tests for mkb_type_raw/mkb_type/mkb_is_uhd and
  MkbType (Category C 2.0 UHD, prerecorded 1.0, no-0x10-record None).
- release.yml: publish job needs [verify, test] so a failing test suite
  blocks crates.io publication.
2026-06-23 19:11:09 -07:00
Matthew Jackson 3c3e0b4341 Bump to 1.0.0-rc.4.3 2026-06-23 15:49:25 -07:00
Matthew Jackson e96528ad5b DVD: correct PAL/NTSC, anamorphic aspect, and SD colour
Fix three DVD video-attribute bugs surfaced by a PAL disc detected as
NTSC:

- PAL/NTSC: parse video_format from VTS_V_ATR bits 5-4, not bits 1-0
  (the old mask read permitted_df, so PAL 576i/25fps was mis-detected
  as NTSC 480i/29.97). Named consts replace the magic bit positions.
- Anamorphic aspect: write MKV DisplayWidth/Height from the disc's
  display_aspect (16:9 720x576 -> 1024x576) instead of square pixels,
  so 16:9 DVDs no longer render as 4:3.
- Colour: stamp SD colorimetry (PAL=BT.470BG, NTSC=SMPTE-170M) instead
  of BT.709 (HD).

Adds VideoStream.display_aspect (threaded through every muxer) plus
TvSystem/DvdAspect/ColorSpace plumbing, with regression tests. Removes
the deprecated Disc mux set_halt bridge (use with_halt).
2026-06-23 15:38:49 -07:00
Matthew Jackson 705857f117 docs: changelog entry for rc.4.2 Windows fixes 2026-06-23 12:49:08 -07:00
Matthew Jackson 845e20e508 v1.0.0-rc.4.2: bump version 2026-06-23 12:41:07 -07:00
Matthew Jackson d5afeb6088 io: add platform-aware fsync helpers (dir + durable file sync)
Add an io::fsync module with a per-OS split (posix/windows) mirroring the
writeback_file convention, replacing two duplicated dir-fsync copies:

- dir(): POSIX directory fsync; a no-op on Windows, where std cannot open
  a directory as a File and the failed open logged a spurious warning on
  every mapfile write.
- file_durable(): opens the target read+write before sync_all so the flush
  succeeds on Windows, where FlushFileBuffers rejects a read-only handle
  with ERROR_ACCESS_DENIED.

Point the mapfile writer at the shared dir() helper.
2026-06-23 12:38:02 -07:00
Matthew Jackson 008c1f143e v1.0.0-rc.4.1: bump version 2026-06-23 10:45:42 -07:00
Matthew Jackson e633a7d3af test(scsi/windows): cross-check all FFI structs + constants vs SDK headers
Audited every #[repr(C)] struct and IOCTL/flag constant in scsi/windows.rs
against the authoritative Windows SDK headers (ntddscsi.h, winioctl.h,
devioctl.h, winnt.h, fileapi.h). All correct except the already-reverted
ScsiPassThroughDirect packing. Add the missing regression guards:
- StoragePropertyQuery layout (STORAGE_PROPERTY_QUERY: 0/4/8, size 12).
- IOCTL/flag constants, with IOCTLs asserted against an independent CTL_CODE
  re-derivation (not a tautological literal) so a mistyped code is caught.
Validated compiling via cargo xwin check --target x86_64-pc-windows-msvc.
2026-06-23 10:44:21 -07:00
Matthew Jackson f177d61bbf fix(scsi/windows): revert wrong packed(4) on ScsiPassThroughDirect (rc.4 drive-detection regression)
rc.4 added #[repr(C, packed(4))] to ScsiPassThroughDirect on the false premise
that ntddscsi.h wraps SCSI_PASS_THROUGH_DIRECT in #pragma pack(push, 4). It does
NOT — verified against the Windows SDK ntddscsi.h: the struct has no pragma pack
and uses natural alignment. On 64-bit Windows (LLP64) that puts DataBuffer at
offset 24 and the struct at 56 bytes, which bare #[repr(C)] produces and which
DeviceIoControl expects.

packed(4) instead imposed offset 20 / 48 bytes — the layout of the SDK's
SEPARATE 32-bit thunk struct SCSI_PASS_THROUGH_DIRECT32 (VOID* POINTER_32). Using
that on a 64-bit host malformed every IOCTL_SCSI_PASS_THROUGH_DIRECT, so the
INQUIRY in drive enumeration failed and autorip/CLI reported zero drives
('RC4 no longer detects my drive'). rc.3.1 (bare repr(C)) worked for the same
users; this restores that layout.

Replace the tautological packed-layout test (which asserted the same wrong
offsets the struct produced) with one cross-checked against the SDK header:
DataBuffer@24, SenseInfoOffset@32, Cdb@36, size 56. Verified compiling via
cargo xwin check for x86_64-pc-windows-msvc.
2026-06-23 10:40:32 -07:00
Matthew Jackson 618524ecb8 ci: cache Rust builds with Swatinem/rust-cache to speed up release + CI 2026-06-23 10:01:37 -07:00
Matthew Jackson 4277ee32dd v1.0.0-rc.4: bump version 2026-06-23 09:31:56 -07:00
Matthew Jackson 6ace16293b keysource: add KeySource::label() for source identification 2026-06-23 09:09:06 -07:00
Matthew Jackson 8efe2fcbfd aacs: add MkbType (BD vs UHD generation) accessor API
Expose the MKB Type field (record 0x10) as a typed MkbType enum with mkb_type()
/ mkb_type_raw() / mkb_is_uhd() helpers, so callers can distinguish AACS 1.0
(Blu-ray) from AACS 2.0/2.1 (UHD) discs without poking raw bytes.
2026-06-23 08:19:40 -07:00
Matthew Jackson 31d07fde6e CHANGELOG: add 1.0.0-rc.4 — decrypt-loss accounting, durable writes, Windows SCSI, truthful error causes 2026-06-23 07:22:05 -07:00
Matthew Jackson ecee9f4ec0 Account for decrypt-time loss so partial AACS/CSS failures can't pass as a perfect rip
When a scrambled AACS unit fails to decrypt under every available key
(a missing/wrong CPS sub-key, or a marginal unit that fails the TS-sync
verify), decrypt_sectors restored the original encrypted bytes and
returned Ok with no signal. Those still-encrypted bytes flowed to the TS
assembler, which silently dropped the non-syncing packets with no loss
counter. The only loss accounting was DiscStream's read-error zero-fill
path, so mux reported lost_video_secs=0 for decrypt-dropped content and
the abort gate accepted the rip even under abort_on_lost_secs=0. A rip
missing real video/audio segments was published as a perfect success.

decrypt_sectors now returns the number of bytes in scrambled units that
no key could decrypt. DecryptingSectorSource accumulates that into a
shared counter exposed via decrypt_loss(); both mux pipelines fold it
into lost_bytes() — the inline DiscStream path directly, and the
file-backed highway via PipelinedPesStream sharing the producer's
counter. Restore-to-original is unchanged, so clear nav-files are never
corrupted; metadata-probe callers that don't read the counter are
unaffected. Adds regression tests at the decrypt and decorator layers.
2026-06-23 06:15:24 -07:00
Matthew Jackson 9220f03f3b keydb: correct read_capped_to_string doc for non-UTF-8 case
The doc claimed Error::KeydbInvalid for non-UTF-8 input, but the code
returns Error::KeydbParse (KeydbInvalid is reserved for the size-cap
violation). Correct the doc to match behavior and add a regression test
asserting non-UTF-8 yields KeydbParse.
2026-06-23 05:27:01 -07:00
Matthew Jackson e52689579b keydb: classify server-dropped connection as KeydbConnect, not KeydbParse
In http_get, when the server closes the TCP connection before the HTTP
header block completes (n == 0 on the byte-by-byte header read), or sends
a header block exceeding 64 KiB, the code returned KeydbParse (E8004).
Both are connection/protocol-level faults from the server, not parse
failures of keydb content — the keydb bytes were never received. Return
KeydbConnect (E8000) instead, which already covers TCP-level exchange
failures. A CLI user hitting a transient drop or a redirect target that
immediately closes now sees the correct 'server hung up' diagnostic
rather than 'the downloaded file was malformed'.

Add a regression test that stands up a loopback listener which accepts
then drops the connection before headers, asserting KeydbConnect.
2026-06-23 05:21:46 -07:00
Matthew Jackson 97ae47b3ea fix(scsi/windows): pack ScsiPassThroughDirect to match ntddscsi.h layout
ntddscsi.h wraps SCSI_PASS_THROUGH_DIRECT in #pragma pack(push, 4),
forcing the PVOID DataBuffer field to 4-byte alignment even on 64-bit
hosts. The Rust struct used bare #[repr(C)], so the compiler applied
natural 8-byte pointer alignment and inserted 4 padding bytes after
TimeOutValue. That shifted DataBuffer to offset 24 (SDK: 20),
SenseInfoOffset to 32 (28), and Cdb to 36 (32), and grew the struct to
56 bytes (48). DeviceIoControl reads at the SDK offsets, so every SPTI
ioctl on 64-bit Windows either got rejected or interpreted garbage as
the CDB and DataBuffer pointer.

Add #[repr(C, packed(4))] to ScsiPassThroughDirect and the companion
SptwbDirect (so offset_of!(SptwbDirect, sense) stays correct for
SenseInfoOffset), plus a layout regression test asserting DataBuffer at
offset 20 and a 48-byte struct size.
2026-06-23 05:16:22 -07:00
Matthew Jackson f596dcb40e prefetched: test that event_fn fires BytesRead per batch
The prefetch producer thread fires a BytesRead event after every
batch it reads, carrying a cumulative byte count. Nothing asserted
this callback actually fired, so a consumer that passed None for the
event_fn would silently get no progress events. Add a fixture-based
regression test that drives a finite extent through new_with_events,
captures the events, and asserts the cumulative count is
non-decreasing and reaches the full extent size at EOF.

This locks the contract autorip's mux progress bar and soft-stall
watchdog depend on.
2026-06-23 04:54:40 -07:00
Matthew Jackson 0e18bfa035 aacs: preserve transport-failure errors through the auth handshake
A SEND KEY / REPORT KEY step in the bus-auth handshake mapped every
SCSI error to a cert/key-specific code (AacsCertRejected, AacsCertRead,
AacsKeyRead, AacsKeyRejected, etc.) via map_err(|_| ...). That discarded
the underlying SCSI error, so a transport-layer wedge (bridge crash / USB
disconnect) mid-handshake was reported as 'drive rejected your host cert',
sending operators down a keydb/host-cert dead end for what is really a
replug/power-cycle situation.

Add a handshake_err() helper that keeps the original error when it is a
transport failure (is_scsi_transport_failure) and only substitutes the
handshake-specific code for genuine SCSI rejections. Apply it at every
SEND KEY / REPORT KEY / REPORT DISC STRUCTURE step in both the AACS 1.0
and AACS 2.0 paths. Add a regression test covering both branches.
2026-06-23 04:20:54 -07:00
Matthew Jackson 89fa0a791e disc: warn when READ CAPACITY fails instead of silently using 0 sectors
read_udf treated a READ CAPACITY SCSI failure as a 0-sector disc via
unwrap_or(0) with no diagnostic. capacity=0 then skews the layer
heuristic (always reports 1 layer, even for dual-layer discs) and the
canonical title-ordering sort, with nothing in /api/state or info to
indicate the command actually failed. Emit a tracing::warn carrying the
original error at the fallback site so a transient capacity failure is
visible. Recovery behavior is unchanged: 0 is still used as the
fallback.
2026-06-23 04:11:46 -07:00
Matthew Jackson f68a66c4be keydb: map missing home dir to NotFound, not keydb-parse error
default_path returned Error::KeydbParse (E8004, rendered as 'failed to
parse the keydb file') when HOME/USERPROFILE was unset. That misreports
an environment failure — a process with no home directory, typically a
stripped container or CI config — as a corrupt keydb file the code never
read. Map it to an IoError(NotFound) in the I/O category instead, so no
display path blames the keydb. Add a regression test.
2026-06-23 04:05:41 -07:00
Matthew Jackson 662594ff40 Preserve I/O error from read_aacs_inputs ISO open
Disc::read_aacs_inputs opened the ISO via FileSectorSource::open and
mapped any failure to Error::AacsNoKeys (E7000), discarding the real
Error::IoError (E5000) and its OS errno. A missing or unreadable ISO
(ENOENT/EPERM) is an I/O fault, not a key-resolution failure; callers
that dispatch on the error code would wrongly tell the user to check
their keys when the ISO simply does not exist.

Propagate the open error unchanged and add a regression test asserting
a nonexistent ISO yields E_IO_ERROR, not E_AACS_NO_KEYS.
2026-06-23 03:59:55 -07:00
Matthew Jackson c9bf92cd6f Fix oversized read batch on non-sysfs (Windows) optical drives
detect_max_batch_sectors() is a Linux-sysfs probe with no platform
gate. It derived the device name with rsplit('/'), which never splits a
Windows \.\CdRom0 / \.\D: path, so the whole path became the device
name, no /sys node matched, is_optical fell to false, and the function
returned the 8192-sector block default (16 MiB/request) instead of the
60-sector optical default. That value then took the Some(b) arm in
Disc::copy and bypassed the 510-sector optical clamp that lives only in
the sysfs branch, leaving every Windows rip/verify running ~16x over the
optical cap (coarser bad-sector recovery, 16 MiB UDF reads).

Gate the sysfs probe behind a new sysfs_batch_probe_supported() helper
(Linux-only, requires a '/'-delimited path) and return the optical
default for any path the probe can't handle. Add regression tests for
the \.\ device-path forms.
2026-06-23 03:51:51 -07:00
Matthew Jackson 4a76deadeb scsi(windows): fix StorageAdapterDescriptor.BusType width (u8 -> u32)
STORAGE_ADAPTER_DESCRIPTOR.BusType is STORAGE_BUS_TYPE, an int-sized
(4-byte) enum, but the repr(C) struct declared it as u8. The total
size stayed 32 bytes by coincidence of alignment padding, and the two
fields actually read today (MaximumTransferLength at offset 8 and
AlignmentMask at offset 16) sit ahead of BusType, so there was no
runtime impact. But BusMajorVersion and BusMinorVersion landed at
offsets 26 and 28 instead of the SDK's 28 and 30, so any future reader
of those fields would have gotten wrong values.

Widen BusType to u32 so BusMajorVersion/BusMinorVersion fall at the
correct offsets, and add a layout regression test asserting every
field offset and the 32-byte total against the winioctl.h layout.
2026-06-23 03:45:34 -07:00
Matthew Jackson 8e6d494e54 aacs: fix stale mkb_version offset doc comment
The doc comment claimed the version was a BE u32 at offset 8 of the
record body (offset 12 from pos), but the code correctly reads pos+8
(body offset 4): a 4-byte record header at pos, the Type field at body
offset 0, then the version at body offset 4. Rewrite the comment to
match the actual read so a maintainer does not 'correct' the offset and
break MKB version parsing. Clarify the matching test comment too.
2026-06-23 02:28:34 -07:00
Matthew Jackson 24ed1d1d19 disc: reject partial CPS-unit key coverage in AACS validation gate
aligned_unit_keys_validate accepted a unit-key set as soon as ONE
scrambled sample decrypted. On a multi-CPS-unit disc a set covering
CPS unit 0 but not CPS unit 1 therefore passed: decrypt_with committed
it, the sweep proceeded, and CPS-unit-1 sectors passed through as raw
encrypted bytes into the ISO/MKV with no error surfaced anywhere.

Require every scrambled sample to be descrambled by some unit key.
A sample no key covers now fails the gate, so an incomplete set is
rejected (AacsKeyRejected) and the caller falls through to the next
candidate, ultimately surfacing a key error instead of silently
writing ciphertext. Wholly-wrong-key rejection is unchanged.

Add a regression test for the partial-coverage case.
2026-06-23 01:43:36 -07:00
Matthew Jackson f2c2ff0eb3 disc: fix misleading bridge-degradation comment, add 04/3E regression test
The comment on the bridge-degradation branch in handle_read_error
claimed it matched the NOT_READY 04/3E sense signature, but
is_bridge_degradation() keys solely on a non-standard SCSI status byte
(anything that is not GOOD/CHECK CONDITION/TRANSPORT FAILURE) and
ignores sense_key/ASC/ASCQ. A real 04/3E bad-sector error arrives as
CHECK CONDITION (0x02), so it never took this branch — it falls through
to the generic NOT_READY retry. Rewrite the comment to describe the
status-byte condition the predicate actually detects, and drop the
parallel misleading note in the not_ready_err test helper.

Add a regression test asserting a NOT_READY 04/3E error is not
classified as bridge degradation and routes to the NOT_READY retry
(3 s pause) rather than the bridge cooldown (15 s pause).
2026-06-23 01:20:04 -07:00
Matthew Jackson a3987e67f2 scsi(windows): only sleep on successful device reset
SptiTransport::reset() unconditionally slept 2 seconds after sending
IOCTL_STORAGE_RESET_DEVICE, even when the IOCTL failed (e.g.
ERROR_INVALID_FUNCTION on a driver that does not support the reset).
On failure no reset occurred, so there is nothing to settle and the
2-second penalty was pure waste. Gate the settle sleep on the IOCTL
return so it only fires when the drive was actually reset.
2026-06-23 00:59:26 -07:00
Matthew Jackson 1b008008dd keydb: write keydb.cfg atomically (temp + fsync + rename)
keydb::save() overwrote the live keydb.cfg with a bare in-place
std::fs::write. keydb.cfg is the single source of AACS truth and this
path runs unattended (first-boot download + daily-refresh thread, with
a container restart on every release), so a SIGKILL, OOM-kill, power
loss, or ENOSPC mid-write could leave the file truncated with the prior
good copy already gone. A truncated keydb does not error at write time;
it surfaces later as failed key resolution on every AACS rip.

Factor the write into write_atomic(): create the parent dir, write a
unique sibling temp file, fsync, then rename (atomic within a
filesystem). On any write/fsync/rename failure the temp is removed and
the existing keydb is left untouched. Same pattern already used by the
settings and mover write paths. Add regression tests covering in-place
replacement (no stray temp) and prior-copy preservation on failure.
2026-06-23 00:24:58 -07:00
Matthew Jackson 980eeb3de9 mux: track skipped bytes for accurate loss estimation
DiscStream skips a whole AACS unit (3 sectors = 6144 bytes) per
read-error event, but only the skip-event count was exposed. Loss
estimates built from errors*2048 therefore undercounted AACS loss ~3x.

Add a lost_bytes field that accumulates the actual zero-filled byte
count at each skip, expose it via a new Stream::lost_bytes() accessor
(default 0; DiscStream and CountingStream override), so consumers can
scale lost-video time by real bytes lost rather than the event count.

Regression tests assert the AACS path records 6144 B/event (and
exceeds the errors*2048 undercount) while the align=1 path records
2048 B/event.
2026-06-23 00:12:56 -07:00
Matthew Jackson c3c5259f84 pipeline: leaked consumer must not finalise an abandoned output
When finish_with_halt's grace period expires it detaches from the
consumer thread and returns an error to the caller, but the leaked
consumer kept running to completion: once its wedged write syscall
returned it would fall through to sink.close(). For the mux writer
close() finalises the MKV (Cues block + segment-header patch), so a
leaked consumer could finalise — and keep writing to — an output file
the caller had already reported as failed, racing a fresh rip for the
same device over the same path.

Add a shared abandonment flag the consumer polls in its drain loop and
again before close(). finish_with_grace sets it before dropping the
JoinHandle, so the moment the wedged syscall returns the consumer skips
any further apply and skips close() entirely, then exits. This does not
interrupt the in-flight syscall (only its return or process exit can),
but it bounds the damage to the write already in flight instead of a
full finalise of an abandoned file.

Regression tests cover both sides: a consumer leaked past the grace
period skips close(), while one that finishes inside the grace window
still calls close() and finalises normally.
2026-06-22 23:22:57 -07:00
Matthew Jackson ae411df8f9 scsi/windows: surface IOCTL_STORAGE_RESET_DEVICE failures
SptiTransport::reset() discarded the DeviceIoControl return value, so a
wrong or unsupported reset IOCTL would fail with ERROR_INVALID_FUNCTION
and silently no-op while the unconditional 2s settle sleep made it look
like a reset happened. That is exactly the regression class the doc block
records for the two earlier (incorrect) code values.

Bind the result and warn (with GetLastError) when the reset fails, debug
on success. Lift IOCTL_STORAGE_RESET_DEVICE to module scope and add a
test recomputing it from the CTL_CODE formula so a wrong value can't slip
back in unnoticed.
2026-06-22 23:08:52 -07:00
Matthew Jackson 60daf63c09 mapfile: fsync parent directory after rename for durable resume checkpoint
Mapfile::flush() wrote the new state to a .tmp sibling, sync_all()'d the
temp file, then rename(2)'d it over the final mapfile path — but never
fsynced the parent directory. After the rename the new dirent lives only
in the directory's page cache, so a crash or power loss in the
rename-commit window (the wide window on NFS, the very case the temp
fsync guards) can lose it: resume then reads a stale or absent mapfile
even though the data bytes were durable, silently discarding multi-pass
recovery progress.

Add a best-effort fsync_dir() on the path's parent after the rename,
mirroring the established dirent-durability pattern in autorip's mover.
A directory that can't be opened or synced is logged and ignored rather
than failing the write, since the file bytes are already durable.

Adds a regression test exercising the parent-fsync branch against a real
subdirectory and asserting the helper is a no-op on a missing directory.
2026-06-22 23:00:04 -07:00
Matthew Jackson d3c58791ff release CI: parallelize test/publish/release off the critical path
Tests, crates.io publish, and the GitHub Release no longer serialize
behind each other: each job only needs the version-check (verify) gate.
Binary consumers git-tag-pin libfreemkv, so they start building the
instant the tag exists rather than waiting on the crates.io publish.
crates.io publish runs as an independent job for external consumers and
uses --no-verify (CI already compiled this commit in the test job).
2026-06-22 21:59:31 -07:00
Matthew Jackson b85744d120 Demote per-read Drive::read trace event to TRACE
Drive::read fires hundreds of thousands of times per rip. Logging its
entry at DEBUG floods a diagnostic log and buries the events that
actually matter. Move it to TRACE so a level-3 (debug) bug-report log
stays readable; level-4 (trace) still captures it for deep dives.
2026-06-22 21:39:27 -07:00
Matthew Jackson 633a22c6bd Merge branch 'feat/windows-hardening' 2026-06-22 18:08:41 -07:00
Matthew Jackson ab959dd770 v1.0.0-rc.3.1: silent-failure guards (mux empty/zero-frame, CSS crack-vs-unencrypted), Windows keydb path, AlignmentMask, English errors 2026-06-22 18:07:48 -07:00
Matthew Jackson 9f422e6ebb docs: describe the generic Unlocker seam, drop in-tree firmware specifics
The docs still documented the old in-tree firmware unlocker: the MediaTek
MT1959 variant table, the READ BUFFER unlock CDB bytes, the profiles.json
schema (unlock_mode/unlock_buf_id/unlock_cdb), the platform/mt1959 driver
listings, and the 'why unlock is needed' handshake mechanism. None of that
lives in libfreemkv anymore — the core is firmware-clean and ships only the
pluggable Unlocker trait + registry (src/unlock.rs).

Rewrite drive-access, architecture, api-design, disc-to-rip, and the README
to describe only the generic Unlocker seam: the trait, register_unlocker, the
registry routing, and the host-cert fallback when no unlocker matches. Point
readers to the freemkv-unlock repo for concrete unlockers. No source change.
2026-06-22 17:10:51 -07:00
Matthew Jackson 63ca840b7e v1.0.0-rc.3: bump version 2026-06-22 16:13:07 -07:00
Matthew Jackson e9108e8b6b scsi/windows: correct IOCTL_STORAGE_RESET_DEVICE to 0x002D5004
The dual-model Windows audit (Sonnet) caught that the prior 'fix' (0x002DD000)
was also wrong: that decodes to the OBSOLETE RESET_BUS code (function 0x400,
R|W access) which class drivers reject. Canonical ntddstor.h:
  IOCTL_STORAGE_RESET_DEVICE = CTL_CODE(0x2D, 0x0401, METHOD_BUFFERED, FILE_READ_ACCESS)
  = (0x2D<<16)|(1<<14)|(0x401<<2)|0 = 0x002D5004.

Only affects the best-effort drive-reset recovery path (return value is
discarded); the normal read/unlock/rip path never calls it, so this does not
change normal Windows operation. Verified correct-by-construction; the
AlignmentMask finding is deferred to rc4 (benign on USB optical bridges,
needs a real Windows SCSI-HBA rig to validate).
2026-06-22 15:53:45 -07:00
Matthew Jackson fa8913800c Fix 3 Opus-audit findings: patch transport-abort, patch AACS align, reset IOCTL
- patch (Pass N) now aborts immediately on transport failure (status=0xFF),
  symmetric with the sweep and single-pass mux. Previously a USB-bridge crash
  was treated as an ordinary bad sector and the pass hammered the crashed
  device sector-by-sector until the per-range watchdog expired. (medium)
- patch AACS recovery reads are now unit-aligned: a mid-unit single-sector
  read on an AACS disc was rejected by the decrypting reader (DecryptFailed)
  and the sector abandoned without asking the drive. The read is now widened
  to the enclosing whole 3-sector unit and the requested window copied out,
  leaving all recovery accounting (pos/block_bytes/cursor) untouched so it
  cannot desync. Only affected CLI decrypt-to-ISO --multipass re-runs. (low)
- IOCTL_STORAGE_RESET_DEVICE corrected 0x002D1004 -> 0x002DD000 (the old value
  decoded to function 0x401 with the access bits cleared, so DeviceIoControl
  would fail ERROR_INVALID_FUNCTION instead of resetting). Windows-only. (low)

Adds a transport-failure classification regression test.
2026-06-22 15:50:22 -07:00
Matthew Jackson f863e9a4be mux/disc: abort single-pass rip on transport failure (USB-bridge crash)
A direct disc://→mkv:// single-pass rip drives fill_extents in
skip_errors mode. On a read failure it shrank the batch, retried, and
once bottomed out zero-filled + skipped the unit and continued. A SCSI
transport failure (status=0xFF) is a USB-bridge crash, NOT a skippable
bad sector: the bridge is wedged and every subsequent read fails the
same way. So the loop marched the entire disc at one ~15s bridge-
recovery per probe, producing no MKV — the user-reported 'hundreds of
0x28/0xff warnings, runs forever, Movie.mkv never created'.

Fix: short-circuit to an error on transport failure before any
shrink/skip, even under skip_errors — mirroring the multipass sweep's
transport-failure rule in read_error::handle_read_error. The CLI
surfaces it so the user power-cycles the drive or switches to multipass
recovery. Regression test asserts exactly one read is issued and no skip
is counted (no infinite march).
2026-06-22 15:18:52 -07:00
Matthew Jackson 4d81affb45 Merge branch 'rc3-topgun' into rc3-integration 2026-06-22 15:08:06 -07:00
Matthew Jackson c73a3dbcb6 Fix Windows multi-drive selection, disk:// alias, and READ chunking
Three fixes for a Windows ASUS Blu-ray drive that failed/spammed errors:

- resolve.rs: accept disk:// as an alias for disc:// (identical behavior;
  empty = auto-detect, path = device). Windows users commonly type
  disk://i: after the drive-letter convention.

- drive::find_drive: prefer a drive that reports media present. Enumerate
  all optical drives, query Drive::drive_status() (GET EVENT STATUS, works
  regardless of firmware), and return the first reporting DiscPresent;
  fall back to the first enumerated drive when none report a disc so
  single-drive / quirky setups don't regress. Selection policy split into
  select_drive_with_media() for unit testing.

- READ chunking: add ScsiTransport::max_transfer_bytes() (default 1 MiB).
  Windows SPTI overrides it with the adapter MaximumTransferLength queried
  via IOCTL_STORAGE_QUERY_PROPERTY / StorageAdapterProperty, clamped to a
  64 KiB floor (fallback on query failure). Drive::read now caps each
  READ(10) to that limit: small reads take the unchanged single-CDB path,
  larger reads loop over read_one() chunks, reporting the failing chunk's
  LBA on error. This stops the 16 MiB single read that exceeded the
  adapter limit, made DeviceIoControl fail, and spammed transport-failure
  warnings with slow tiny-read fallbacks.

Tests added for the disk:// alias, media-preference selection, and READ
chunk decomposition / per-chunk error LBA.
2026-06-22 15:02:24 -07:00
Matthew Jackson 4f606ae9a3 aacs: source OEM host certs from keysource layer
Complete the OEM/AACS cert baseline so host certs are a KeySource output,
never compiled in. With an unlocker present the OEM route is unused
(unlocker_read_volume_id short-circuits); without one, the cert handshake
runs when a keysource supplies a host cert and fails gracefully when none
does.

- KeySource trait gains host_certs() (default empty), reusing the existing
  aacs::HostCert type. A source holds certs as its second kind of AACS
  material alongside decryption keys.
- ScanOptions gains key_sources so the handshake can collect certs across
  the app's keysource layer, unioned with DriveCredentials.
- do_handshake_cert collects certs via collect_host_certs (credentials +
  every key source). Zero certs from any source now returns the new
  graceful Error::AacsNoHostCert (code 7024, sentinel <no host cert>)
  instead of silently skipping; resolution still falls back to the
  path-1 disc-hash -> VUK lookup, which drops the error on a hit.
- error.rs: add E_AACS_NO_HOST_CERT / Error::AacsNoHostCert, wired into
  code(), Display, and the round-trip + sentinel tests.

HandshakeResult { volume_id, read_data_key } unchanged: the cert path
still yields both the VID and the bus key.
2026-06-22 11:23:38 -07:00
Matthew Jackson 25acd09504 unlock: finalize Unlocker 3-capability contract
Rename the trait to a generic, drive-neutral capability contract so future
unlockers don't conform to LibreDrive specifics:

  - unlock(...)   -> unlock_drive(...)        (the one required capability)
  - read_vid(...) -> read_volume_id(...)      (no-op default)
  - add set_max_read_speed(...)               (no-op default)

The trait doc now states the contract in one place: unlockers are optional
drive-capability providers; the AACS layer is the always-present baseline and
falls back to the full cert handshake when no unlocker matches. Implement only
the capabilities your drive supports.

Registry: route_unlock now calls unlock_drive; unlocker_read_vid renamed to
unlocker_read_volume_id; add unlocker_set_max_read_speed (mirrors route_unlock
resolution, first matching unlocker, no-op if none match). drive::init calls
it on a matched drive in the post-unlock path; a speed-set failure is logged
and does not fail the rip. encrypt.rs handshake updated to the new VID helper.

Tests updated for the renames; added a set_max_read_speed routing test
(match invokes, no-match is a safe no-op).
2026-06-22 11:05:21 -07:00
Matthew Jackson 159e967760 unlock: add OEM read_vid capability to Unlocker seam
An Unlocker unlocks drive functionality, not just the disc: unlock() is
one capability, OEM VID retrieval is another. Widen the Unlocker trait
with a default-no-op read_vid(), add an unlocker_read_vid registry helper
that mirrors route_unlock resolution, and consult it in do_handshake_cert
before the cert-based VID read. A matching unlocker that serves a VID via
its OEM path short-circuits the cert handshake — VID is obtained without
the host certificate + HRL (restoring the pre-refactor decoupled OEM VID
path, now living inside the unlocker). Non-matching drives, and unlockers
without an OEM VID path, fall through to cert auth unchanged.

is_unlocked() now reports the honest signal (a registered unlocker matched
this drive) instead of const false.
2026-06-22 10:50:47 -07:00
Matthew Jackson 6dc62bcd84 Extract drive unlock behind pluggable Unlocker seam
libfreemkv must stay firmware-clean for crates.io. Move ALL drive-unlock
knowledge — firmware blobs, WRITE_BUFFER/MODE SELECT upload, unlock CDBs,
the MT1959 variant-A/B handshake, the 800 KB profiles.json database, and
the DriveProfile parsing — out into the freemkv-unlock-ld crate.

libfreemkv now keeps only the seam:
  - Unlocker trait (name/matches/unlock) + a process-wide ordered registry
    (register_unlocker / route_unlock) in src/unlock.rs
  - Drive::init() walks the registry; the first unlocker whose matches(id)
    is true runs unlock(scsi, id); if none match the drive is left in
    stock mode and the host-cert AACS handshake (the OEM route) carries
    the disc.

The unlocker issues its own CDBs through the public ScsiTransport::execute,
so libfreemkv knows nothing about how unlocking happens.

Removed:
  - profiles.json
  - src/platform/mt1959/{mod,variant_a,variant_b}.rs
  - src/profile.rs (DriveProfile, ProfilesFile, find_by_drive_id, ...)
  - the PlatformDriver trait

Because the Unlocker seam reports only success/failure (no extended-access
marker), VID acquisition is now always via the cert-based handshake; the
per-drive OEM-VID-CDB shortcut and Drive::is_unlocked() (now const false)
are removed/neutralized. Disc-speed calibration moved into the unlocker's
unlock(); Drive::probe_disc() is a no-op.

git grep over src/ is firmware-blob/profiles/WRITE_BUFFER/mt1959-free.
All tests pass on Rust 1.86 (precommit green).
2026-06-22 10:31:51 -07:00
Matthew Jackson 9250f5bb30 wip: top gun EL/decrypt follow-up (in progress, rc3) 2026-06-22 10:02:53 -07:00
Matthew Jackson e960c2f1be mux/mkv: video-only timeline epochs + finer 0.1ms TimestampScale
Fixes corrupt MKV seek index on single-clip titles with many
interleaved tracks (Top Gun UHD: 2 video, 11 audio, 32 PGS).

TimelineContinuity previously shared one high_ns frontier + offset_ns
across ALL tracks. A sparse, lagging non-video frame (subtitle/audio)
ratcheted the frontier up; the next normal video frame then sat >3s
below it and was misread as a clip-boundary discontinuity, permanently
bumping offset_ns. On a one-clip title this fired thousands of times
and inflated Cue/cluster timestamps into the billions of ms, destroying
the seek index (ffmpeg then seeked to wrong positions and emitted
spurious 'Could not find ref with POC N' errors).

Now only the VIDEO track drives epoch decisions: video alone advances
the frontier and opens a new epoch on a real backward PTS jump.
Non-video tracks are remapped under the current offset and never touch
the frontier or offset. A lagging non-video tail straggler at a genuine
multi-clip boundary (old-epoch raw PTS under the new offset) is
recognised via the previous offset and remapped to the seam, so it
neither flies forward nor forces a back/forward-dated split cluster.
Genuine multi-clip seamless rebasing is preserved.

Also drop TimestampScale from 1ms to 0.1ms (100_000 ns/tick) so
23.976fps frames and 0.833ms TrueHD AUs stop colliding on a single
tick (the source of the non-monotonic-DTS warnings and the audio
cadence flattening). The finer scale shrinks the i16 block-relative
span to ~3.27s, so: cluster duration is set to 2s nominal (keeps
keyframe-driven clusters within the i16 range for typical GOPs), and
the i16-overflow cluster-split path now emits a Cue for the split
cluster so the seek index has no gaps.

Regression tests: single-clip late-subtitle must not inflate offset_ns;
non-video must not advance the frontier; non-video straggler remapped
to seam at a real boundary; every cluster (incl. i16-split) carries a
Cue. Existing tick/duration assertions updated for the new scale.
2026-06-22 09:45:45 -07:00
Matthew Jackson f74979bdb4 docs: scrub AACS internals + key-coverage claims from changelog; condense pre-1.0 history 2026-06-22 09:39:40 -07:00
Matthew Jackson dc2cac1b5f v1.0.0-rc.2: bump version 2026-06-22 09:06:54 -07:00
Matthew Jackson 337e77951c rc2: macOS cross-compile fix + security/recovery hardening
- build.rs: pass target -arch to cc so macos_shim cross-compiles (x86_64-apple-darwin)
- AACS/CSS: unit-aligned decrypting sweep; per-VTS CSS title keys (hard-fail on wrong VTS);
  reject truncated Unit_Key_RO; AACS 2.0 sig-verify skip; CSS bus-auth random nonce
- recovery: gap-filling mapfile load; sweep/copy resume reconciliation; stale-mapfile abort;
  patch wedge/damage-window range reset
- mux: TS continuity + PSI CC desync guards; HEVC numTemporalLayers clamp; MPEG-2 pending
  byte-cap; PS parse_pts marker-bit validation; HdrFormat strict parse; Unknown-variant metadata
- net/keydb: network:// SSRF parity (IPv4-mapped, CGNAT, 0.0.0.0/8, Class-E); bounded keydb
  header read + size cap + error context
- io: durable mapfile fsync; NFS writeback degrade; sync_file_range error capture;
  Windows SCSI u32 transfer guard
2026-06-22 08:58:10 -07:00
Matthew Jackson 5941c059c6 v1.0.0-rc.1
CSS keyless decrypt (Stevenson), AACS 1.0/2.0/2.1, MPEG-2 DVD, multi-OS SCSI, multipass recovery, mux highway, audit hardening
2026-06-21 21:06:07 -07:00
Matthew Jackson e8bb6225ac libfreemkv 0.31.10: parallelise recover_dk_position slot scan
The subset-difference slots are independent, so rayon find_map_any scans them
in parallel and cancels on first match. UHD MKB no-match scan ~26s -> ~4.6s on
8 cores. Bit-identical result; 58 aacs tests pass.
2026-06-17 19:56:07 -07:00
Matthew Jackson 9c80ef8245 libfreemkv 0.31.9: ~3x fewer AES ops in the subset-difference PK walk
calc_pk_from_dk derived all three children (left/pk/right) at every tree
level but used only the one it descended into; the Processing Key only
matters at the final node. Derive just the descended child per level + the
PK once at the end. Bit-for-bit identical; speeds every DK->MK derivation
(disc decryption + unpositioned-DK recovery). 60.8s -> 22.9s on a UHD
worst-case recovery scan.
2026-06-17 15:43:39 -07:00
Matthew Jackson dc87962e50 libfreemkv 0.31.8: recover unpositioned device key position from an MKB
Add recover_dk_position: boil a position-less device key down against a
disc MKB to its invariant subset-difference position (node/uv/u_mask_shift)
— zero-descent probe + ancestor walk-up, hoisted verify. Consolidate the
SD-walk surface (drop the research-only probe::walk_pk_against_tables;
make derive_media_key_from_pk_walked internal).
2026-06-17 15:29:51 -07:00
Matthew Jackson 4221cd6a86 libfreemkv 0.31.7: expose Processing Key from the DK walk 2026-06-17 13:18:34 -07:00
Matthew Jackson dda4e7482b aacs: unify the SD-walk; derive_media_key_from_dk now exposes the PK
Two byte-identical copies of the subset-difference walk lived in keys.rs
and variants.rs. Consolidate the pure helpers (aesg3, calc_v_mask,
calc_pk_from_dk) into keys.rs (pub(super)); variants.rs imports them.

Add derive_media_key_and_pk_from_dk(mkb, dks) -> Option<(mk, pk)>, which
returns the intermediate Processing Key the walk already computes;
derive_media_key_from_dk becomes a thin wrapper. This lets callers bank
the PK on a DK boil instead of re-deriving it via a second, divergent
walk (the classical-vs-variant cvalues order made that miss silently).

216 AACS tests pass (incl. a new (mk,pk) regression); precommit (1.86) green.
2026-06-17 13:11:03 -07:00
Matthew Jackson f80551f278 udf: read AACS MKB length-aware; honor inline allocation descriptors
read_file rejected the padded ~128 MiB MKB_RO.inf via the 0.31.0
MAX_FILE_BYTES cap, so read_aacs_inputs failed and the online
key-resolve path never contacted the keyserver. Read the MKB's real
record length from its header and read exactly that. Also honor
inline/embedded (AD type 3) files so small AACS .inf files read from
the ICB payload instead of being misparsed as allocation descriptors.

Release 0.31.6.
2026-06-08 19:20:41 -07:00
Matthew Jackson 41a6d89cd1 libfreemkv 0.31.5: do not force monotonic block timestamps on video
B-frame video PTS is legitimately non-monotonic in decode/storage order; the
audio-oriented monotonic nudge was clobbering it to prev+1ms, which decoders
flagged as non-monotonic DTS (thousands per title). Apply the nudge to
audio/subtitle only; video keeps its true PES PTS. + regression test.
2026-06-08 09:00:35 -07:00
Matthew Jackson f79c2a0aa9 libfreemkv 0.31.4: prune 144 vacuous tests (keep spec-grounded subset) 2026-06-08 07:28:55 -07:00
Matthew Jackson d181362460 libfreemkv 0.31.3: use RFC 5737 doc address in network URL test fixture 2026-06-07 22:42:09 -07:00
Matthew Jackson 8000bae177 libfreemkv 0.31.2: comprehensive spec-grounded test suite (~950 tests)
Test-hardening release, no runtime changes. Adds spec-grounded unit tests
across the silent-corruption surfaces — UDF/MPLS/CLPI/IFO parsing, BD/DVD
title + extent assembly, AACS/CSS key handling, TS/PS demux + codec parsers,
MKV/EBML container output, the mux pipeline, sector prefetch + decrypt
decorator, drive/SCSI sense decoding, label extraction, and core I/O. Each
test is grounded in the format spec or real on-disc behavior and verified to
fail under a targeted source mutation. No behavior changed.
2026-06-07 22:28:29 -07:00
Matthew Jackson 2a55bab3ed tests: read_aacs_inputs Long-AD coverage; scrub title names from comments
Add a precommit fixture proving Disc::read_aacs_inputs reads a Long-AD,
multi-extent /AACS/Unit_Key_RO.inf in full — the exact input the online
key-request path depends on (no disc/deploy needed). Make
read_aacs_inputs_from_reader pub(crate) for the test.

Reword internal comments/doc examples to generic descriptions instead of
specific media titles.
2026-06-07 21:25:29 -07:00
Matthew Jackson 222a596c55 libfreemkv 0.31.1: UDF Long-AD stride fix + MKB trim guard + regression tests 2026-06-07 20:49:55 -07:00
Matthew Jackson c1b4f3cbb3 tests: harden UDF allocation-descriptor + bad-sector recovery paths
Spec-grounded unit tests for the silent-corruption surfaces, each verified to
fail under a targeted source mutation (no vacuous tests).

udf (10): Extended-AD 20-byte stride + extent LBA at off+12, type-1 sparse
extents skipped not emitted, zero-length type-0 terminator, continuation-loop
bound (anti-hang), UTF-16BE and 8-bit name decoding, FID L_IU offset, parent
(..) FID skip, d-string length-byte cap. Locks the spec branches a future
allocation-descriptor refactor must not silently break.

recovery (9): Pass-N damage-skip range bounds (forward/reverse cursor stays in
range), one-quarter-of-remaining skip cap, below-threshold no-op, work-done
accounting, and bridge-degradation retry-to-budget fall-through.
2026-06-07 20:46:00 -07:00
Matthew Jackson 06c30aa466 udf: honor ICB allocation-descriptor type (Short/Long/Extended AD)
read_icb_extents hardcoded an 8-byte Short-AD stride for every file. Large
BD-ROM .m2ts streams use 16-byte Long ADs; striding them as Short ADs reads
descriptor #0 correctly (length+lba align) but lands #1 in the middle of the
first Long AD (its zero impl_use bytes). The AD-list terminator (data_len==0
=> break) then fired on that zero and stopped after the first extent, so every
multi-extent title truncated at ~1 GiB. The same reader backs read_file, so
disc AACS-input files (/AACS/*.inf) and the m2ts mux extents were both
affected.

Read the ICB Tag flags (AD type) and stride 8/16/20 bytes for Short/Long/
Extended ADs accordingly; Extended ADs carry the lba at off+12.

aacs: extract trim_mkb and restore its guard so an MKB whose content length
the parser cannot determine (mkb_content_len == 0) is returned intact instead
of truncated to empty.

Regression tests: Long-AD read_icb_extents returns all extents; Long-AD
read_file returns full content; trim_mkb never zeroes an unrecognised MKB.
2026-06-07 19:53:40 -07:00
Matthew Jackson 061f68594a 0.31.0: hardening and correctness pass across mux, codec, AACS/CSS, UDF/MPLS/CLPI, recovery, drive/SCSI, labels, and I/O
Library-wide review-and-fix pass: tightened AACS keydb/handshake/variant
handling and trailing-partial-unit policy, corrected MPLS mark offset and
added UDF allocation bounds, hardened the mux/codec framing and M2TS paths,
guarded SCSI READ CAPACITY short transfers and unified error mapping, added
overflow guards on untrusted disc input, and made prefetch shutdown
deterministic. Release profile now builds with thin LTO + single codegen unit.
2026-06-07 17:37:38 -07:00
MattJackson 5b6ea8f5c4 v0.30.7: mux correctness pass
- hevc: real chroma/bit-depth in hvcC (10-bit Main10), emit redefined
  param sets in-band; clamp oversized param sets
- h264: emit redefined SPS/PPS in-band, clamp oversized avcC
- ac3: flush final frame at EOS, per-frame PTS
- dts: attribute each access unit's PTS to its own core PES
- truehd: capture PTS only at access-unit boundary
- meta: preserve color_space round-trip
- dvd: fix subtitle/audio track PID collision
- pgs/vobsub: flush trailing subtitle at EOF; multi-PES SPU reassembly
- ts: unify codec table via from_coding_type (+LPCM), warn on unknown
2026-06-06 21:46:49 -07:00
MattJackson eeba94b21d Merge branch 'worktree-agent-a184f5a97a37e2c1a' 2026-06-06 21:40:42 -07:00
MattJackson cfc12774f2 mux: fix DVD subtitle/audio track collision, PGS/VobSub flush, unify TS codec table
Subtitle/DVD output-corruption + stream-mapping coverage fixes.

1. DVD subtitle/audio track-mapping collision (CRITICAL). The PS path
   routed 0xBD private-stream packets to a track via (sub_id & 0x1F)+1,
   so VobSub subtitle sub-id 0x20+j aliased audio track j+1: subtitle
   PES was fed to the AC-3 parser and the real subtitle track got
   nothing. Route by the canonical DVD PID instead via a new
   PsPacket::dvd_pid() that mirrors scan_dvd_titles' PID assignment
   (video 0xE0, audio 0xBD00+i, subtitle 0x20+j), then look up the
   track in pid_to_track. Fixed identically at all three sites
   (pipelined_stream consume_ps, disc.rs live feed, disc.rs EOF flush).
   Unmappable/unmapped packets now WARN instead of silently dropping.

2. PGS flush() missing. PgsParser inherited the no-op default flush, so
   the last subtitle of every PGS track (emitted only when a following
   PCS arrives) was dropped at EOF. Implemented flush() to drain the
   pending display set (duration_ns: None for the trailing block).

3. DVD VobSub multi-PES SPU not reassembled. A subpicture unit larger
   than one PES spans multiple PES (only the head carries a PTS).
   DvdSubParser is now stateful: it buffers per sub-stream until the
   leading 2-byte SPU_size is satisfied, inherits the head PTS, and
   emits one Frame. flush() drains a truncated trailing SPU at EOF.

4. One-table hygiene. scan_streams had a duplicate stream_type->Codec
   table that had drifted from Codec::from_coding_type (missing 0x80
   LPCM, 0x85 mapped to DTS-HD MA vs HR, etc.). scan_streams now uses
   from_coding_type plus a new Codec::kind()/CodecKind category split,
   so the two mappings can never diverge. Silent drops in
   scan_streams and bluray STN parsing now WARN with PID + type.

Tests: dvd_pid mapping + subtitle/audio collision regression, PGS
final-subtitle flush, VobSub multi-PES reassembly + EOF flush,
scan_streams 0x80 LPCM via from_coding_type.
2026-06-06 21:33:11 -07:00
MattJackson fd543c058b mux: rustfmt (Rust 1.86) on touched codec/meta files 2026-06-06 21:32:04 -07:00
MattJackson cd8ce708ac mux/h264: emit redefined SPS/PPS in-band + clamp oversized avcC
H.264 stored SPS/PPS in single Option slots and always stripped them
from frame data, so a mid-title parameter-set redefinition (same id,
different body) was lost — those frames decoded against the stale avcC
copy the player re-applies at each keyframe. Same defect class as the
HEVC PPS-redefinition bug. Mirror that fix: emit a changed SPS/PPS
in-band at every occurrence; strip only the first-seen / identical ones.

Also guard avcC's 16-bit NAL length fields: a param set > 65535 bytes
truncated the length while appending all bytes. Return None instead.
2026-06-06 21:24:16 -07:00
MattJackson 2eee777b8f mux/hevc: real chroma/bit-depth in hvcC + clamp oversized param sets
codec_private() hardcoded 8-bit 4:2:0 in the hvcC fixed header, wrong for
10-bit Main 10 UHD (essentially all UHD). Parse chroma_format_idc and
bit_depth_luma/chroma_minus8 from the SPS RBSP (with emulation-prevention
removal and sub-layer profile_tier_level handling) and emit the real
values; fall back to 8-bit 4:2:0 only if the SPS can't be parsed.

Also guard the 16-bit NAL length fields: a param set larger than 65535
bytes would truncate the length while appending all bytes, mis-framing
the record. Return None instead of emitting a corrupt hvcC.
2026-06-06 21:22:44 -07:00
MattJackson a5962be86c mux/truehd: capture PTS only at an access-unit boundary
A continuation PES (one that merely extends a TrueHD AU spanning PES
packets) carries its own later PTS, which the parser was adopting
mid-assembly, snapping the AU's timestamp and breaking the monotonic
per-AU cadence (A/V drift). Capture the PTS base only when the
reassembly buffer is empty, i.e. when a PES actually begins a new AU.
2026-06-06 21:19:53 -07:00
MattJackson 7306f661b9 mux/dts: attribute each access unit's PTS to its own core PES
When two access units flushed in one parse() call (a core arriving in an
earlier PES than the PES that closes the unit, the standard DTS-HD MA
core+extension-as-separate-PES layout), the second AU inherited the
latest PES's PTS instead of its own core's.

Track per-PES PTS markers keyed by buffer offset and stamp each emitted
AU with the PTS of the PES covering its first byte, rebasing markers on
every front drain. Each AU now keeps its own core's timestamp.

Extends two_cores_back_to_back to assert PTS; adds cross-PES coverage.
2026-06-06 21:19:07 -07:00
MattJackson 77f7aced83 mux/ac3: flush final frame at EOS and stamp per-frame PTS
Two defects:
- Ac3Parser inherited the no-op default flush(), so a complete final
  frame still buffered at end-of-stream was dropped (~32 ms of audio
  lost). Add a flush() that drains a complete buffered frame, mirroring
  dts.rs.
- Every frame in one parse() call was stamped with the single PES PTS,
  collapsing their timecodes and drifting A/V. Compute a base PTS once
  per call, then advance per frame by the frame's own duration (AC-3 =
  1536 samples; E-AC-3 from numblkscod), converting samples->ns at the
  stream sample rate (fscod). Each Frame now carries duration_ns.
2026-06-06 21:16:17 -07:00
MattJackson 97ae452e40 mux/meta: preserve color_space round-trip in m2ts metadata
MetaStream::Video dropped color_space on from_title/to_title, hardcoding
BT.709 on the way back. HDR titles (BT.2020) lost their color metadata.

Add a color_space field, populate it in from_title, and use it in
to_title. For pre-0.30.7 metadata that has no color_space, derive it from
the preserved hdr field (all HDR formats are BT.2020, SDR is BT.709).

Adds ColorSpace::id() + FromStr for serialization round-trip.
2026-06-06 21:13:29 -07:00
MattJackson eec0594a30 Merge branch 'main' into worktree-agent-a91cd2cf29779b84e 2026-06-06 21:11:05 -07:00
MattJackson dab6ea4359 mux/hevc: emit redefined param sets in-band at every keyframe
Fight Club redefines PPS id 0 mid-title; the parser froze the first PPS into
codecPrivate and stripped the rest, so the redefined segment decoded against
the wrong PPS (CABAC/cu_qp_delta desync, intact framing). Now any VPS/SPS/PPS
whose body differs from the codecPrivate copy is emitted in-band at every
occurrence, overriding the hvcC copy a player re-applies per keyframe. Proven:
Fight Club re-mux decode errors 320+ -> 0 across all corrupt regions.

Also adds aacs::unit_key_validates (1-block early-reject UK validation) and
ts_sync_count/ts_packet_total helpers.
2026-06-06 21:09:04 -07:00
MattJackson 9a5ed57044 mux: enforce strictly-monotonic per-track block timestamps
Fixes the non-monotonic DTS ffmpeg/players reject (observed on Fight Club audio
streams 16/18: "14061 >= 14060"). Some audio PES PTS truncate to the same
millisecond as the prior frame, or tick back 1ms from rounding. The MKV writer
now tracks the last block timecode per track and nudges a non-increasing one to
prev+1ms (sub-frame, inaudible, A/V sync unaffected at ms granularity), recorded
only for frames actually written. New helper monotonic_ts() + unit test.

Independent of the separate HEVC slice-payload corruption investigation.
2026-06-06 15:57:00 -07:00
MattJackson 7d58ba7b08 v0.30.0: DiscInputs carries the disc's volume label (title)
Add `DiscInputs.volume_label: Option<String>` — the disc's human title (UDF/ISO
volume identifier, else BDMV <di:name>), populated by Disc::inputs() from the
scan. Identity only, no secret, not used in any AACS derivation; lets a key
source forward the title so a key service can catalog disc_hash → title.
2026-06-06 09:05:15 -07:00
MattJackson 4f1dbfd042 v0.29.0: bump version 2026-06-05 20:36:19 -07:00
MattJackson 5b702a76a7 mux: write PTS (presentation), not DTS, as the MKV block timecode
The video codec parsers (HEVC, H.264, VC-1, MPEG-2) used
pes.dts.or(pes.pts) as each frame's timestamp. MKV block timecodes
are presentation timestamps; frames are stored in decode order and
the player reorders for display by timecode. Using DTS makes the
timecode monotonic in storage order, presenting B-frames in decode
order — visible motion judder / wrong frames on playback, and
PTS-based seeking lands on the wrong frame.

The compressed video was always byte-correct (verified by NAL-level
diff against a known-good demux); this was purely a timestamp defect
affecting every B-frame title. Fix: prefer PTS (pes.pts.or(pes.dts)).
Verified on a real UHD iso->mkv: emitted PTS now reorders for
B-frames identically to a reference muxer.

Update the two tests that asserted the old DTS-preferred behavior and
add an HEVC regression test pinning PTS as the block timecode.
2026-06-05 20:01:13 -07:00
MattJackson 6be5198886 libfreemkv: v1.0 hardening — codec/EBML/TS robustness + DTS parser fixes
Audit-driven fixes (rounds 1–3):
- hevc: correct hvcC profile/level SPS offsets (HEVC has a 2-byte NAL header)
- mkv: map all DTS variants to the registered A_DTS codec id; force a new
  cluster before the i16 cluster-relative timestamp can overflow
- ebml/mkvstream: bound untrusted EBML sizes (no multi-GB allocs); reject
  uint>8 (was an OOB panic) and non-{0,4,8} float widths (were a desync)
- ts: skip PES-header bytes that span a TS packet boundary; add the PMT
  section_len/prog_info_len bounds the PAT parser already had
- ac3: preserve a 0x0B77 syncword split across a PES boundary; cap buffer
- dts: validate each next-core boundary by decoded core size (a 0x7FFE8001
  pattern inside XLL payload no longer false-splits/drops the lossless
  extension); reject sub-minimum core frames; fix forced-emit PTS base
- lpcm: DVD program-stream PCM no longer double-strips the BD LPCM header
- vc1/mpeg2: do not emit a parameter-set-only PES as a standalone frame
- pgs/truehd: cap the pending reassembly buffer (parity with ac3/dts)
- aacs: ts_syncs_intact uses the exact packet count
- prefetched: capacity-guard the recycled-buffer set_len
- Cargo.toml: exclude project docs from the published crate

Convergence: a third independent audit pass found no remaining material
(CRITICAL/HIGH/MEDIUM) issues. Full precommit (fmt + clippy -D + tests,
Rust 1.86) green.
2026-06-05 16:23:39 -07:00
MattJackson e2aa9abd6d aacs: validate a resolved key against content before applying it
decrypt_with now takes the disc's encrypted content samples and, after
deriving the candidate unit keys, confirms at least one de-scrambles a real
aligned unit before committing them. A wrong key (a keydb VK that doesn't
match the disc, a stale UK) is rejected with AacsKeyRejected instead of
silently applying garbage unit keys. Conservative by design: with no samples
(resume / mapfile cache) it accepts as before, leaving those paths unchanged.

The KeySource trait becomes a stateful provider — next_key hands one candidate
at a time (the source owns the order) and reports exhaustion, replacing the
all-at-once resolve; errored() distinguishes a failed source from a clean
no-key.
2026-06-05 09:23:08 -07:00
MattJackson 3b7bee9ed4 Fix DTS-HD MA muxing as lossy core only (#10)
DTS-HD MA/HRA access units on Blu-ray are a DTS core frame (sync
0x7FFE8001) followed by one or more DTS extension substreams (sync
0x64582025) carrying the lossless audio. Ground-truthing the Dunkirk
ISO showed the m2ts demuxer hands these out as SEPARATE PES packets on
the same PID: one core PES (exactly core-sized, nothing trailing), then
the extension substreams in following PES packets with their own later
PTS.

The old DtsParser emitted one frame per PES the moment a core frame was
complete, and dropped any PES with no core sync. So every core became a
core-only (lossy) frame and the extension PES packets were discarded as
junk -- silently downgrading the track to lossy DTS core (1557 kb/s CBR,
16-bit) instead of DTS-HD MA (VBR, 24-bit lossless).

Rewrite the parser to assemble across PES boundaries: an access unit
runs from its core sync up to (but not including) the NEXT core sync, so
the core plus every following extension substream stays together. Add a
CodecParser::flush() (default empty) called at end-of-stream by both the
pipelined and inline DiscStream mux paths to drain the final buffered
unit. A 64 KiB cap guarantees forward progress and never stalls if a
boundary can't be found.

Validated on the rip1 testbed: Dunkirk eng+ger and Fight Club eng main
audio now ffprobe as profile=DTS-HD MA (Fight Club eng at 24-bit), with
VBR packet sizes (~2716-2788 B) well above the old fixed 2012 B lossy
core. Genuinely-lossy DTS dub tracks are left untouched.
2026-06-05 06:40:27 -07:00
MattJackson b7405e2d27 mux: fail iso:// with no usable AACS key instead of muxing garbage
When an AACS-encrypted ISO is muxed with decryption requested (not --raw)
but key resolution yielded no usable key, input() proceeded to mux the
still-encrypted stream — emitting ~100 MB of garbage (no TS syncs, demuxer
emits nothing) and sometimes spinning for tens of minutes.

Add a cheap result-check in resolve::input()'s Iso branch via the pure
predicate aacs_key_missing(raw, has_aacs, keys): when decryption is
requested AND the disc carries AACS state AND decrypt_keys() is None,
return new Error::NoDiscKey { disc_hash } (E7022) before muxing. The
40-hex disc hash is sourced from AacsState::disc_hash. --raw and
non-AACS (unencrypted / CSS) discs are unaffected. Unit-tested.
2026-06-05 05:06:15 -07:00
MattJackson e04d79c593 v0.28.1: bump version 2026-06-04 21:46:41 -07:00
MattJackson b34af1fa74 mux: wire TrueHD channel probe into iso:// mux setup
correct_truehd_channels() probes the first DECRYPTED access units of the
chosen title (TrueHD PIDs, bounded 8 MiB read of the first extent) and
sets AudioStream.channels from the MLP major sync — fixing the MPLS
audio_format understatement (5.1 declared on a 7.1/Atmos TrueHD track).
Regenerates the basic codec label for the corrected count; richer
editorial labels are left untouched. Wired in resolve.rs input() for
iso:// after decrypt_with (the m2ts is only decryptable post-key), using
a fresh reader so the mux reader is undisturbed. generate_audio_label
made pub(crate).
2026-06-04 20:28:37 -07:00
MattJackson a7317f8885 mux/truehd: decode true channel count from MLP major-sync format_info
Adds truehd_channels(format_info) + truehd_channels_from_stream(): the
real presentation channel count (8ch/7.1 when present, else 6ch/5.1)
from the TrueHD major sync, which the MPLS audio_format base field
understates (declares 5.1 even on a 7.1/Atmos track — the Dune case).
Per the MLP spec presentation channel-assignment masks. 3 unit tests.

This is the decoder; wiring it to correct AudioStream.channels requires
a mux-time probe of the DECRYPTED audio payload (the channel truth isn't
readable at scan — the m2ts is AACS-encrypted until the key is applied
at mux), to be added + validated on a real rip.
2026-06-04 19:51:46 -07:00
MattJackson 474273afc0 mux/mkv: signal Dolby Vision via dvcC BlockAdditionMapping
With the EL now carried (mpls PID fix), the MKV writer tags the Dolby
Vision layer so players/mediainfo recognise it: a BlockAdditionMapping
(0x41E4) with BlockAddIDType 'dvcC' and a DOVIDecoderConfigurationRecord
(profile 7, bl+el+rpu present) on the DV-layer video track. Disc
Profile 7 dual-layer is preserved losslessly as a second video track
(archival-correct, MakeMKV-equivalent); M2TS carries the EL as a native
second video PID with no extra signaling needed.

Adds dolby_vision_config() + a profile-7 dvcC unit test. End-to-end DV
recognition (mediainfo / playback) to be confirmed on a full rip.
2026-06-04 19:42:14 -07:00
MattJackson 0a2bab5789 mpls: resolve PID for sub-path / DV-EL stream entries (types 2/3/4)
parse_stream_entry only read the PID for stream-entry type 1 (a stream
in the PlayItem's own clip), returning PID 0 for every other type. The
Dolby Vision enhancement layer uses type 4 (verified on Wicked + Dune:
se_len=9 type=0x04 [00 10 15] -> PID 0x1015), so the EL fell through to
PID 0x0000 and was silently dropped by the demux/mux — Dolby Vision lost.

Now the PID offset is keyed off the entry type per the BD stream_entry()
layout: type 1 -> +2, type 2 -> +4, type 3/4 -> +3. The DV EL now
resolves to its real PID (0x1015) so it is demuxed and carried as a
stream through the PES layer to every writer (M2TS carries dual-PID DV
natively; the MKV writer's DV signaling is the format-specific piece).
2026-06-04 19:28:34 -07:00
MattJackson fa4d7ef871 mux/dts: keep DTS-HD extension substream across PES boundaries
The DTS parser emitted a core-only frame and discarded the trailing
DTS-HD extension substream whenever the extension straddled a PES
boundary (it advanced past only the core, then re-synced on the core
syncword, skipping the leftover extension bytes). That silently
downgrades DTS-HD MA / HRA to lossy DTS core.

Now: when an extension sync is visible after the core — full, or a
partial prefix at the buffer edge — wait for the full extension instead
of splitting it off; only 'nothing after the core' (final unit / EOF,
no parser flush) is taken as a genuine lossy core-only unit. Adds a
boundary-split regression test asserting core+extension is preserved.
2026-06-04 19:16:26 -07:00
MattJackson 36d1af1b7f labels+disc: codec from stream, audio-richness title tiebreak
Two validated audio-correctness fixes (proven on real discs Wicked/
Paddington/Dune/Fight Club via head-captures):

labels: apply_labels now derives the codec/channel descriptor from the
stream's OWN codec/channels unless the parser's codec_hint is BOTH
consistent with it AND richer (e.g. "Dolby Atmos" on a TrueHD stream).
A mis-bound hint ("AC-3 2.0" on a TrueHD track) is rejected and the
stream's own codec used — killing the cross-labeled shuffle (Wicked) and
the compat-core mislabel (Paddington), while keeping rich hints and
normalizing plain ones to uniform marketing names. (codec_hint_consistent
+ codec_hint_adds_detail, 5 tests.)

disc: canonical_title_order gains an audio-richness tiebreak
(lossless > channels > track-count) for titles that tie on
duration+clips — so a movie authored as a full-audio playlist plus a
stereo-only twin (Fight Club 00800 vs 00004) picks the full-audio one
instead of falling to array order.
2026-06-04 19:10:11 -07:00
MattJackson b96f6206fe 0.28.0: version bump (keydb decoupling) 2026-06-04 16:41:04 -07:00
MattJackson bb32fb993b 0.28.0 review fixes: propagate mux key-apply error; scrub key-shaped fixtures
- mux input(): propagate a failed decrypt_with rather than muxing an
  undecryptable stream silently (review M6).
- Scrub synthetic but key-SHAPED hex (HOST_PRIV_KEY/DEVICE_KEY/disc-entry) from
  the keydb parser tests to obvious zero/repeated-byte placeholders. No real or
  real-looking key material in code.
2026-06-04 16:38:58 -07:00
MattJackson c1eb74dfa5 AACS: cutover — lookup-free lib; DriveCredentials for the handshake
The library no longer loads a keydb anywhere. The scan path always captures the
disc's AACS inputs (MKB, VID, Unit_Key_RO.inf) and resolves NO key; a caller
resolves a Key from a key source and applies it via Disc::decrypt_with.

- ScanOptions loses keydb_path / unit_key / disable_keydb (and the path search);
  it now carries only optional DriveCredentials (host certs) for the live-drive
  AACS handshake. do_handshake_cert uses those instead of loading the keydb.
  An unlocked / LibreDrive drive takes the OEM Volume-ID path and needs none.
- The mux input() path takes caller-resolved unit_keys instead of a keydb_path,
  and applies them via decrypt_with.
- Deleted the now-dead inline resolve_encryption / resolve_encryption_static.

All 700+ lib tests pass.
2026-06-04 16:04:06 -07:00
MattJackson 46838c63ca AACS: trim the MKB captured at keyless scan to its record length
The VID-only scan path stashed MKB_RO/RW raw — those files are allocated to a
fixed ~128 MiB and zero-padded, so the MKB on AacsState (consumed by
Disc::inputs() and the device/processing-key decrypt_with derivation) was the
full pad, not the ~few-MB record stream. Trim to mkb_content_len, matching
read_aacs_inputs.
2026-06-04 15:32:12 -07:00
MattJackson 1e60220ff6 AACS: KeySource::needs_samples() gates the content-sample read
A source that validates server-side against ciphertext (an online key service)
needs encrypted content samples; one that keys on disc identity (keydb, mapfile)
does not. needs_samples() lets the caller skip the extra disc read unless a
configured source actually needs it. Defaults false.
2026-06-04 14:45:40 -07:00
MattJackson 080f03e8ed AACS: DiscInputs gains app-populated samples for ciphertext-validating sources
An online key service validates a candidate against real ciphertext, so it
needs a few encrypted content sample units. Add a samples field to DiscInputs;
Disc::inputs() leaves it empty (reading content needs the disc reader, which
scan does not retain) for the application to fill.
2026-06-04 14:42:11 -07:00
MattJackson 35de6101d4 AACS: add KeySource trait + DiscInputs; Media key carries a candidate pool
The library now describes the key-source abstraction it consumes, while the
concrete sources live in a companion crate. A KeySource looks a disc up from
its DiscInputs (disc hash, volume id, MKB, Unit_Key_RO.inf) and returns a Key;
the library derives down and decrypts. Sources do no derivation.

Disc::inputs() exposes those inputs from a scanned disc.

Key::Media now carries a Vec of candidate media keys rather than one: a media
key is MKB-scoped, so a source hands its whole pool and the library picks the
one that verifies against the disc's MKB. That mirrors Device/Processing and
keeps key selection (a derivation step) inside the library.
2026-06-04 14:29:32 -07:00
MattJackson 9012101573 AACS: decrypt_with accepts higher-level keys and derives down
Disc::decrypt_with now takes Device / Processing / Media / Volume keys in
addition to Unit. A caller hands in whatever level it resolved and the
library derives down the AACS chain to the per-CPS-unit keys, then
decrypts:

  Device   -> MKB walk         -> media key -> VUK -> per-CPS unit keys
  Processing -> MKB            -> media key -> VUK -> per-CPS unit keys
  Media      -> Volume ID      -> VUK              -> per-CPS unit keys
  Volume     -> Unit_Key_RO.inf, one unit key per CPS unit
  Unit       -> used directly (terminal)

Derivation stays centralized in the version-dispatched resolver
(1.0 / 2.0 / 2.1), fed by a single-key provider built from the supplied
key — no new crypto. Volume notably does NOT stop at the volume key: it
decrypts every CPS unit's key.

Scan stashes the AACS inputs (Unit_Key_RO.inf and MKB) on AacsState so an
out-of-band decrypt_with can derive without re-reading the disc.

Non-breaking: Key is #[non_exhaustive] and the existing Unit path is
unchanged. New tests cover the Volume -> per-CPS derive-down, the
missing-inputs error, and the no-units rejection.
2026-06-04 14:20:16 -07:00
MattJackson 8bc1de6c9b 0.27.5 (step 2, Phase 1): expose AACS inputs (uk_ro, mkb) on AacsState
scan now stashes the raw Unit_Key_RO.inf + MKB bytes on AacsState (via
resolve_vid_only, the disable_keydb path), so an external key-resolver can
derive unit keys from a resolved VUK without re-reading the disc — the
foundation for moving lookup/derivation out of libfreemkv. Additive: the keydb
path is untouched, all existing constructors default the new fields empty.
584 lib tests green. Builds on the KeyOrigin rename + the Key/decrypt_with API.
2026-06-04 13:20:27 -07:00
MattJackson d94a4d3444 0.27.5 (step 2, item 2): rename provenance enum KeySource -> KeyOrigin
Disambiguates the key vocabulary: Key (the input handed to decrypt_with),
key sources (the resolver's lookup list), and KeyOrigin (how a key was
resolved). Internal-only rename — no dependents import it.
2026-06-04 11:42:21 -07:00
MattJackson 1bd7e1de2a 0.27.4: lookup-free Key API (Disc::decrypt_with) + E8005 deferred-mux fix
Add libfreemkv::Key (non_exhaustive; Unit wired) + Disc::decrypt_with(Key) — the
caller resolves a key from any source and hands it in; the library decrypts with
no keydb/network of its own. inject_unit_keys now synthesizes a minimal
ExternalUk AACS state when a scan built none (no keydb at scan time), so a disc
keyed at sweep but scanned without a keydb still decrypts from its mapfile UK
instead of reporting E8005 and deferring the mux forever. Tests cover the
synthesize, replace, empty->None, and decrypt_with paths.
2026-06-04 11:08:33 -07:00
MattJackson da62ee7cf2 v0.27.3: bump version 2026-06-03 21:47:43 -07:00
MattJackson 575c76156f mapfile: persist resolved unit keys (keys XOR VID)
A keyed disc now writes its decrypted AACS unit keys to the mapfile header
(# freemkv-uk: <cps>:<hex>); an unresolved disc writes only the VID. The two are
mutually exclusive (set_unit_keys clears the VID) — unit keys are the final
answer, so deferred-mux / resume decrypts directly with no key lookup, while the
VID alone is the 'still unresolved, retry' marker. CopyOptions/SweepOptions carry
the keys (written when present, else the VID); Disc::inject_unit_keys applies
mapfile-recovered keys to a scanned disc. Round-trip test added.
2026-06-03 21:45:37 -07:00
MattJackson b518860d9c v0.27.0: detect AACS-scrambled units by raw TS sync, not flag bits
Rename is_unit_encrypted -> is_aacs_scrambled and decide encryption from the unit's MPEG-TS sync bytes (destroyed by the encrypted body) instead of the TP_extra copy-control (byte 0) or TS scrambling-control (byte 7) flags, which discs do not set reliably. One shared predicate now backs the decrypt gate and out-of-band key validation, so callers agree on what 'encrypted' means. Decryption restores the syncs, so a decrypted unit reads as clear and there is no flag to clear.
2026-06-03 07:35:50 -07:00
MattJackson c8eb42b490 v0.26.11: detect AACS unit encryption via TS scrambling-control bits
is_unit_encrypted read the TP_extra copy-control bits (byte 0), which are a
copy-permission flag, not an encryption flag. On discs whose sampled units are
clear navigation packets (PAT/PMT) those bits can be set while the unit is not
scrambled, so a correct Unit Key was used to 'decrypt' already-plaintext data,
produced garbage, and the key was wrongly treated as failing.

Read the actual flag instead: the TS transport_scrambling_control bits (top two
of TS-header byte 3 = byte 7 of the aligned unit, inside the clear seed). AACS
encrypts whole aligned units, so this one packet's TSC reflects the unit.
decrypt_unit now clears the TSC bits of every packet on the way out so the
result is valid unscrambled TS. Tests updated to the TSC flag.
2026-06-02 20:37:56 -07:00
MattJackson 48570ac065 Local resolve: MK-pool brute via km_verifies (path 2.5)
keydb stores Media Keys per-disc, but an MK is MKB-scoped (shared across a
pressing/MKB-family). A disc whose own hash/VID isn't keyed can still resolve
if any stored MK verifies against its MKB. New path 2.5 (between PK and the
VID lookup) collects the distinct MK pool from the providers, km_verifies each
against the disc MKB, and on a UNIQUE pass derives VUK (with the disc VID) then
the UK — matching the online resolver's behavior so local keydb mode resolves
the same discs (e.g. an MK present in keydb under a sibling pressing).
km_verifies is one AES-D + magic check per candidate (cheap). Adds
KeyProvider::media_keys() + a path-2.5 unit test.
2026-06-02 16:14:09 -07:00
MattJackson 08e46640fd v0.26.8: bump version 2026-06-02 15:41:38 -07:00
MattJackson a9195824ff Trim MKB to real record length; empty unit-keys means no keys
- mkb_content_len walks the MKB record stream and returns where it ends,
  so callers can drop the trailing zero padding. MKB files are allocated
  to a fixed size (~128 MiB) with records at the front; both MKB_RO and
  MKB_RW can be padded. read_aacs_inputs* now trim to the real records
  (~few MB) instead of shipping ~128 MiB of zeros.
- decrypt_keys() returns None when unit_keys is empty: an AACS state that
  carries only a VID (out-of-band resolution, pre-key) is 'encrypted, no
  keys', not a usable empty key set.
2026-06-02 15:41:27 -07:00
MattJackson 401fe23988 v0.26.7: bump version 2026-06-02 15:15:58 -07:00
MattJackson f89bce5851 Read MKB_RO not the padded MKB_RW; keep VID when keydb is disabled
- read_aacs_inputs / read_aacs_inputs_from_drive now read MKB_RO.inf first.
  MKB_RW.inf is a fixed ~128 MiB rewritable region that is mostly zero
  padding; reading it shipped 124 MiB of nothing. MKB_RO is the real,
  correctly-sized MKB (a few MB). Fall back to RW only if RO is absent.
- disable_keydb no longer drops the Volume ID. A caller resolving Unit Keys
  out-of-band needs the VID (on-disc content read during the handshake).
  New resolve_vid_only builds a keys-free AacsState carrying just the VID +
  version metadata, so the disc reports 'encrypted, no keys' (resolved
  out-of-band) instead of discarding the VID.
2026-06-02 15:15:45 -07:00
MattJackson dfccb85e15 v0.26.6: bump version 2026-06-02 14:11:13 -07:00
MattJackson ebedffb762 ScanOptions: add disable_keydb to skip all keydb lookup
A caller that resolves Unit Keys out-of-band can now set disable_keydb
so the scan consults no keydb at all — neither an explicit keydb_path
nor the standard search locations. Without it, a keydb that merely sits
in a default location ('~/.config/...') silently shadows the out-of-band
path. unit_key still takes precedence over everything.
2026-06-02 14:09:59 -07:00
MattJackson fdf63ccb7f v0.26.5: bump version 2026-06-02 13:37:32 -07:00
MattJackson c735d284da v0.26.4: bump version 2026-06-02 13:06:10 -07:00
MattJackson 94ab7bc73c scan: genericize the caller-supplied unit-key docs (no external-source naming)
The ScanOptions.unit_key path is a generic primitive — a caller-supplied Unit
Key that bypasses keydb lookup. Doc comments + a tracing log named a specific
external source; reworded to neutral 'out-of-band / external key service' so
the library makes no assumptions about where the key came from.
2026-06-02 13:00:11 -07:00
MattJackson 9f209fe066 v0.26.3: bump version 2026-06-02 10:56:56 -07:00
MattJackson 32a1a6e095 mapfile: persist AACS Volume ID as ddrescue-safe comment
Add an optional 16-byte AACS Volume ID to the Mapfile, persisted as a
'# freemkv-vid: <32-hex>' header comment so it survives to deferred-mux
and resume without touching the ISO payload. ddrescue treats the line
as a comment, so the pos/size/status data parser is unaffected.

- Mapfile gains set_vid/vid accessors; save() emits the comment in the
  header block, load() parses it back (malformed/absent -> None, never
  errors the load). Load->save round-trip preserves it (patch pass).
- Thread vid through CopyOptions and SweepOptions; sweep persists it
  when creating/opening the mapfile.
- Unit test: round-trips the VID, asserts the saved comment, and
  confirms data ranges parse identically with and without the comment.
2026-06-02 08:50:19 -07:00
MattJackson 1d3b8f5fb6 chore: gitignore scratch/ (prevent accidental commit of session artifacts) 2026-06-01 22:52:58 -07:00
MattJackson 97b0ae7be2 chore: scrub internal refs from changelog/makefile/cargo 2026-06-01 22:46:54 -07:00
MattJackson 7756f1feca aacs: restore read_aacs_inputs_from_drive (live-drive key-file extractor) 2026-06-01 21:52:27 -07:00
MattJackson 1565da610a chore: scrub non-shippable references from tests/comments 2026-06-01 21:36:57 -07:00
MattJackson e134616422 v0.26.2: bump version 2026-06-01 21:06:56 -07:00
MattJackson 8d54a3c64e aacs: KeyProvider abstraction + PK walker + external-UK key source
Two coherent additions to the AACS resolver:

KeyProvider abstraction (provider.rs) — key material comes from pluggable
backends; KeyDb implements it (device/processing keys, host certs,
disc-by-hash / disc-by-vid lookup) plus orphan-DK parsing. ResolveContext
takes a provider array. Adds the SD-tree PK walker
(derive_media_key_from_pk_walked) and a `probe` module (km_verifies MK
oracle, mkb_* record parsers) used for offline key verification. Cvalue
record selection prefers 0x05, falls back to 0x07.

External-UK key source — the second, mutually-exclusive key source for the
keyserver path. ScanOptions/InputOptions gain `unit_key`; when set,
resolve_encryption_static skips keydb entirely and uses the caller-supplied
Unit Key directly (KeySource::ExternalUk). Disc::read_aacs_inputs exposes a
disc's Unit_Key_RO.inf + MKB so a caller can fetch the UK out-of-band; the
library makes no network call itself.

CHANGELOG: redact test-disc title in historical notes.
2026-06-01 20:47:14 -07:00
MattJackson c0478e1273 labels(pixelogic): number audio from the FPL playlist, not menu segments
Pixelogic discs that authored a SEG_MainFeature menu/preview segment
containing a stray commentary token made the parser anchor on that
segment instead of the real FPL_MainFeature playlist. Combined with
'Audio Stream N' placeholder tokens not advancing the audio counter, a
lone eng_ACOM_ commentary at STN slot 4 collapsed onto slot 1 — tagging
the main feature track as Commentary (seen on Wicked: For Good).

Fix:
- Anchor the feature section on the FPL_ playlist when present; only
  fall back to SEG_MainFeature on discs with no FPL_ section.
- Count 'Audio Stream N' placeholders so editorial audio tokens get
  their true STN ordinal. Subtitle (PG Stream N) numbering is left
  unchanged — counting it regressed several corpus discs.

Extract the section/numbering loop into assign_labels with unit tests.
Verified: 11/11 labels-corpus discs still PASS; Wicked commentary now
lands on the AC-3 2.0 stereo track instead of the main TrueHD.
2026-05-31 21:16:11 -07:00
MattJackson 7d29168fec v0.26.1: AACS resolver path 5 — KEYDB unit-keys direct fallback
Adds a 5th key-resolution path that consumes pre-decrypted unit keys
directly from KEYDB when the entry has no VUK field. Covers ~4,572
entries in the public keydb (~2.5%), heavily skewed toward MKBv76+ UHD
discs where DVDFab/FindVUK can no longer extract a VUK but does extract
unit keys. Partial CPS-unit coverage is rejected so a disc is never
half-decrypted.

Resolver path order reordered root-to-leaf: DK (1) → PK (2) →
KEYDB-derived MK+VID (3) → KEYDB VUK (4) → KEYDB unit keys (5).
Previous order was leaf-first.

API:
- AacsState::vuk is now Option<[u8; 16]> (was [u8; 16])
- ResolvedKeys::vuk is now Option<[u8; 16]> (was [u8; 16])
- KeySource variants reordered + new KeyDbUnitKeys variant

3 new resolver tests (path 4 still works without VID; path 5 succeeds
with pre-decrypted unit keys; path 5 rejects partial CPS coverage).
2026-05-22 12:51:37 -07:00
MattJackson 8e14c9b850 v0.26.0: bump version 2026-05-21 15:18:39 -07:00
MattJackson e1c8343f77 aacs: OEM-driven VID retrieval — per-drive CDB from profile, cert fallback
When the drive is in extended-access state (unlocked), retrieve VID via
the per-drive `read_vid_cdb` from the bundled profile instead of the
cert-based AACS REPORT_KEY handshake. Cert handshake remains the
fallback for drives that don't enter extended-access state, or whose
profile lacks the required CDB.

Empirically verified on the BU40N (signature 999ec375) against
Barbie UHD: drive returns 36 bytes from buffer 0x44 at offset
0x10E291, VID at response[4..20]. The 16 bytes match Dune Part Two's
known VID in keydb.cfg byte-for-byte, cross-validating the path
against an independent oracle.

Architectural impact:

- Renames `Drive::is_libredrive_active()` → `Drive::is_unlocked()`.
  Internal `Mt1959::libredrive_active` becomes `Mt1959::unlocked`;
  the prior `unlocked` (init-success flag) becomes `init_complete`
  to avoid the name collision.
- `disc/encrypt.rs::Disc::read_vid` is the single entry point.
  When `is_unlocked()` is true, calls `read_vid_oem` (issues the
  per-drive CDB, validates the response signature high-3-bytes
  `00 22 00`, returns bytes [4..20]). Otherwise delegates to
  `read_vid_cert` (the existing AACS REPORT_KEY format 0x80 path).
- `DriveProfile` gains the per-drive CDB templates and identifier
  blocks extracted from each per-drive firmware payload — including
  `read_vid_cdb`, `read_disc_keys_cdb`, `drive_nominal_speed_cdb`,
  `set_speed_max_cdb`, two cache-prime canary CDBs, the buffer-0x45
  verify CDB, the firmware-upload CDB, and the unlock probe CDB.
  Variants A and B differ in which fields are populated. All optional;
  consumers fall back to the cert/handshake path when fields are
  absent.
- New error variants `Error::DriveProfileMissing` (E7020) and
  `Error::VidCdbUnavailable` (E7021). Both treated as
  "OEM unavailable → try cert path" by `read_vid`, not terminal.

Closes the v0.25.x gap where HRL-burned host certs (the public
libaacs leaked cert is on every recent drive's HRL) blocked all
post-handshake VID retrieval. With OEM-driven VID:

- AACS 1.0 BD on supported drives: rips end-to-end with our existing
  DKs walking the MKB.
- AACS 2.x UHD: fails honestly at the DK wall (E7018 "No usable DK"
  for v77+ MKBs) instead of the misleading E7017 "No Volume ID"
  the prior code surfaced. We have VID; we just don't have v77+ DK
  material — that gap is a key-acquisition problem, not a code
  problem.

Empirically verified on rip1 (BU40N + Barbie UHD, MKB v77,
2026-05-21): error code flipped from E7017 to E7018 as predicted.
The DK wall is now correctly the proximate failure for unrippable
modern UHD discs, instead of the indirect VID-retrieval wall the
v0.25.x cert-only path produced.

Renames and comment scrubs eliminate upstream-RE-vocabulary
references in the public crate per `feedback_no_breadcrumbs.md`.

674 tests pass (565 lib + 109 integration). No tradename leaks in
any modified file.
2026-05-21 15:18:21 -07:00
MattJackson a956c6ad94 v0.25.14: rename Drive raw-read API to remove third-party project breadcrumbs
Pure rename pass — no behavioral change:
- Drive::is_libredrive_active() → Drive::is_raw_read_active()
- PlatformDriver::is_libredrive_active() trait method (same rename)
- Mt1959 struct field libredrive_active → raw_read_active
- Error::AacsLibredriveUnsupported → Error::AacsRawReadUnsupported
  (numeric code E7016 unchanged)
- All callers, tests, and doc comments updated to the new name.

Old identifiers removed entirely; downstream consumers must update.
Mirrored in bdemu, freemkv, autorip, freemkv-tools.
2026-05-21 14:43:20 -07:00
MattJackson 1805d92ca4 v0.25.13: DrmScheme dispatcher + AACS 2.1 framework + libredrive cleanup
- Introduce DrmScheme enum (Css/Aacs10/Aacs20/Aacs21) + drm module with
  uniform detect/load dispatch across all four protection schemes.
- Land AACS 2.1 Media Key Variant framework in aacs::variants: chain
  derivation, MKB record types 0x82/0x83, bit-0x02 SoftKCD and bit-0x04
  online-challenge detection. Aacs21 dispatcher arm wired but commented
  out pending validation against a Variant-scheme disc.
- Replace aacs2: bool with AacsVersion enum across ContentCertificate,
  UnitKeyFile, ResolvedKeys. resolve_keys splits into _v1/_v2/_v21.
- Delete the libredrive raw-read VID shortcut from do_handshake; the
  drive enforces the AGID requirement regardless of firmware-upload
  state, so the shortcut spuriously dispatched E7017 instead of
  surfacing the real downstream walls.
2026-05-21 13:57:45 -07:00
MattJackson 823f0ad430 v0.25.12: bump version 2026-05-21 11:37:53 -07:00
MattJackson 477bdf1835 v0.25.11: bump version 2026-05-21 11:14:22 -07:00
MattJackson 4d83b69c20 aacs: libredrive raw-read VID path + revert v0.25.9 built-ins + walker fix follow-through
Three coherent threads landing for v0.25.11:

1. Libredrive raw-read VID path. When Mt1959::do_unlock sees both the
   MMkv active-mode marker at [12..16] and the LbDr mode-ID marker at
   [16..20], Drive::is_libredrive_active() returns true and
   do_handshake skips the AACS cert dance — VID is retrieved via
   READ_DISC_STRUCTURE format 0x80 with AGID=0 and bus encryption is
   already off. This unblocks UHD ripping on drives whose leaked host
   cert is on the AACS HRL.

   - platform/mt1959/mod.rs: detection + active flag + 4 unit tests.
   - platform/mod.rs: PlatformDriver::is_libredrive_active trait method.
   - drive/mod.rs: Drive::is_libredrive_active accessor.
   - disc/encrypt.rs: do_handshake branches on the flag; new
     read_volume_id_libredrive helper. Return type widened to
     (Option<HandshakeResult>, Option<Error>) so callers see which
     specific failure happened.
   - disc/mod.rs: scan_with plumbs the new tuple through and preserves
     handshake errors as disc.aacs_error.

2. Revert v0.25.9 built-in AACS keys + plugin slot. Single source of
   AACS truth: keydb.cfg. The compiled-in DKs/PKs were a slim
   convenience that didn't move the hard problem (no v77+ DKs) and
   added a maintenance surface. Plugin slot was overlapping
   functionality with the main keydb.

   - Deleted src/aacs/builtin_keys.rs (4 DKs + 3 PKs).
   - Removed KeyDb::with_builtins, load_or_builtins, merge_from,
     merge_local_plugin, local_plugin_path, internal dedup helpers.
     KeyDb::empty kept for unit-test use.
   - KeyDb::load reverts to pre-0.25.9 form: read file or return I/O
     error; no fallback.
   - disc::encrypt::resolve_encryption keydb_path back to required
     (&Path), not Option<&Path>.
   - disc::scan_with surfaces KeydbLoad { path: "<no keydb in search
     paths>" } sentinel when encrypted + no keydb — same sentinel
     autorip's message switch already handles.
   - CSS player keys in src/css/auth.rs stay compiled in; they're
     1999-era public inputs separate from AACS and pre-date the 0.25.9
     additions.

3. Walker fix follow-through (libaacs-parity validate_processing_key,
   cvalues 0x07-then-0x05 preference, path-2/3/4 short-circuit on
   zero VID) + NIST AES-CMAC KAT + VID MAC round-trip / mutation /
   zero-rejection tests.

5 new Error variants for finer-grained AACS failure reporting:
AacsHostCertRejected (E7015), AacsLibredriveUnsupported (E7016),
AacsVidUnavailable (E7017), AacsMkUnavailable (E7018),
AacsVukNotInKeydb (E7019). Lets CLIs/UIs render which piece of the
AACS chain failed instead of always saying "no keys."
2026-05-21 11:10:35 -07:00
MattJackson 7dbbfc6726 v0.25.10: bump version 2026-05-20 15:35:05 -07:00
MattJackson e635c9556f v0.25.9: built-in AACS keys + plugin slot + MKB record-type fix
Two changes that make AACS 1.0 / DVD self-sufficient:

1. MKB record-type identification bug fix. `mkb_find_mk_dv` was
   searching for type 0x10 (which is Type-and-Version, 12 bytes)
   when the Verify Media Key Record is actually type 0x81 for
   AACS 1.0 or type 0x86 for AACS 2.0/2.1. `mkb_version` had the
   inverse bug. PK and DK derivation paths therefore silently
   failed on every disc, masking how often the fallback paths
   could have worked. Fix searches the correct types; tests added
   covering both the 0x81 and 0x86 verify-record forms and the
   0x10 version record at offset 8 of the body.

2. Built-in AACS keys + operator plugin slot. Four device keys
   (covering MKB v01-v82+) and three processing keys (covering
   v63-v68) compiled directly into the library. Combined with the
   31 CSS player keys already in css/auth.rs, DVDs and Blu-rays
   (AACS 1.0) now decrypt with zero external files. New plugin
   path at ~/.config/freemkv/local_keys.cfg (same syntax as
   keydb.cfg) layered additively on top of built-ins and main
   keydb. `Disc::scan` no longer errors when keydb.cfg is absent;
   AACS 2.0 / UHD still surfaces a specific error when the disc
   needs keys none of the layers provide.

Public docstrings in project docs + README updated to describe the
three additive layers (built-ins → keydb.cfg → local_keys.cfg).
2026-05-20 09:00:32 -07:00
MattJackson d7b5c30f5d v0.25.8: bump version (unified release with autorip 0.25.8) 2026-05-19 21:45:45 -07:00
MattJackson 739a276a39 v0.25.7: BU40N firmware wedge fix in do_handshake
Pre-0.25.7 the AACS authenticate loop fired up to 16 host-cert
attempts back-to-back with no pause. Each attempt is 5-10 SCSI
REPORT_KEY/SEND_KEY exchanges, so on a disc whose host cert isn't
in our KEYDB (or one the drive rejects), the drive saw 80-160 SCSI
commands in a few hundred ms and entered a fast-fail firmware
wedge state where every subsequent CDB returns sense 05/24 until
power-cycled.

Three defences:
- MAX_CERT_ATTEMPTS capped at 3 (was 16)
- 1-second sleep between attempts
- Bail immediately on any sense_key == 0x05 (ILLEGAL_REQUEST) so
  the loop can't deepen the wedge if a regression undoes the
  attempt cap.
2026-05-19 21:18:45 -07:00
MattJackson 5f1028a62a v0.25.6: sync to autorip 0.25.6 (image diet) 2026-05-19 18:17:10 -07:00
MattJackson ea15d212de v0.25.5: sync to autorip 0.25.5 2026-05-19 18:01:14 -07:00
MattJackson 00673c8ec3 v0.25.4: sync to autorip 0.25.4 2026-05-19 17:44:23 -07:00
MattJackson bfefb4cb5b v0.25.3: sync to autorip 0.25.3 release (no behavioural changes) 2026-05-19 17:11:11 -07:00
MattJackson 1b95193517 v0.25.2: DTS-HD codec ID + PGS BlockDuration
- MkvTrack::audio emits A_DTS/MA, A_DTS/HR, A_DTS per the DTS family
  instead of mislabelling everything as A_DTS. Plex transcoder and
  strict hardware decoders reject DTS-HD MA payload under a plain
  A_DTS track.
- PgsParser is now stateful: pairs display PCS with the following
  empty PCS to compute a duration. Frame::duration_ns + PesFrame::duration_ns
  carry it through; MkvMuxer::write_frame gains a final Option<u64>
  parameter that emits BlockGroup + BlockDuration when set. Fixes
  subtitle bitmaps lingering past their intended end-time.
2026-05-19 16:11:54 -07:00
MattJackson 7dcac44136 v0.25.1: bump version 2026-05-19 14:20:44 -07:00
MattJackson 4da559e39f mux: PrefetchedSectorSource event_fn + delete DiscStream::new_pipeline
* `PrefetchedSectorSource::new_with_events` adds an optional
  `event_fn` callback that fires `BytesRead` after every successful
  batch from the producer thread. The original `new()` becomes a
  thin no-events wrapper. Lets autorip wire the highway and still
  get UI progress events without polling the consumer side.

* `build_iso_pipeline` grows an `event_fn` arg so the autorip
  multipass mux can pipe BytesRead straight through to its progress
  UI.

* Stream trait gains a default `errors() -> u64` method (= 0) so
  Box<dyn Stream> callers (autorip's mux loop) can read the
  skip-on-error counter without downcasting. `DiscStream` overrides
  to return its `errors` field.

* Delete `DiscStream::new_pipeline` and the pipeline-mode fields
  (`demux_thread`, `demux_rx`) plus the `read_pipeline` helper.
  All pipeline construction now goes through
  `PipelinedPesStream` via `build_iso_pipeline`; `DiscStream`
  becomes the single-thread-only inline path used by the drive
  single-pass read.

* `lib.rs` re-exports `build_iso_pipeline`.
2026-05-19 14:19:37 -07:00
MattJackson eeca250f69 v0.25.0: bump version 2026-05-19 13:37:24 -07:00
MattJackson c51b3181f2 mux: pipelined PES highway — read+decrypt → demux → parse on 3 threads
Introduces the freemkv mux throughput highway: a three-stage thread
pipeline that replaces the inline single-thread read path for any
file-backed source (ISO and m2ts file URLs both route through it).

  Thread A: read + decrypt  (PrefetchedSectorSource / BytePrefetcher)
  Thread B: M2TS demux      (DemuxThread)
  Thread C: codec parse     (PipelinedPesStream, on caller thread)

Each handoff uses a bounded crossbeam channel with a recycled buffer
pool — no allocations or memcpys in the steady-state hot loop.

Component map:

* io/byte_prefetcher.rs (new) — std::io::Read producer thread with
  recycled Vec<u8> pool. Pairs with PrefetchedSectorSource (sector
  side) so demux_thread::spawn_zero_copy can wire either upstream.
* sector/prefetched.rs — recycled buffer pool added; into_channels()
  peels off the rx/recycle_tx/shell triple for zero-copy demux.
* mux/demux_thread.rs (new) — owns the TsDemuxer/PsDemuxer, runs
  feed() on its thread, ships Vec<PesPacket> batches.
* mux/pipelined_stream.rs (new) — the read-side Stream impl. Pulls
  packets from the demux thread and runs codec parse on the caller.
* mux/resolve.rs — build_iso_pipeline (public) / build_m2ts_pipeline
  (private) assemble the three stages; iso:// and m2ts:// both
  return PipelinedPesStream.
* mux/m2ts.rs — collapsed to a write-only sink (Mode::Read deleted;
  the read direction lives on the highway now).
* mux/codec/h264.rs — find_start_code uses memchr SIMD memmem::find.
* mux/codec/hevc.rs — tightened frame_data initial capacity.
* mux/ts.rs — boundary-packet handling avoids the per-batch 16 MiB
  remainder copy; PesAssembler starts at 16 KiB to dodge the 64-page
  first-touch fault tax that the previous 256 KiB pre-alloc paid on
  every PES boundary.
* mux/disc.rs — gains DiscStream::new_pipeline + read_pipeline as
  the legacy autorip ingress (drive + multipass paths still need
  on_event / skip_errors before they migrate to the highway).
* io/file_sector_source/* — per-OS prefetch() syscall hook
  (Linux readahead, macOS F_RDADVISE, Windows/other no-op).
* decrypt.rs — FREEMKV_DECRYPT_THREADS renamed to FREEMKV_THREADS;
  pool sized to all cores by default.

Measured on rip1 testbed (Civil War UHD, 62 GiB ISO → null://):

  60 → 322 MB/s warm cache (old new_pipeline path)
  60 → 660 MB/s warm cache (highway path, this commit)
  60 → 126 MB/s sustained disk-bound

The IsoSectorReader baseline reader was deleted in favour of
FileSectorSource so the freemkv CLI and autorip exercise the same
read path.
2026-05-19 13:35:32 -07:00
MattJackson 2a31a47434 mux: keyframe-align MKV clusters + SeekHead; set TS RAI on keyframe PES
MKV: cluster boundaries now require a video keyframe in addition to the
5s minimum, so every cluster has a CuePoint at its start. Pre-first-
keyframe frames are dropped. Adds a SeekHead at Segment start with
fixed-width back-patched SeekPositions for Info/Tracks/Chapters/Cues.

Before this change a 2h26m UHD rip had 52 CuePoints across ~1750
clusters and a 16.5-minute gap between adjacent seek entries; players
seeking inside that gap had to scan from the prior cue. After, one
CuePoint per cluster.

TS (tsmux production path + m2ts_mux): PesFrame.keyframe is plumbed
end-to-end. Codec-private parameter sets are prepended on the first
keyframe (not the first frame); non-key video before any keyframe is
dropped. The first TS packet of a keyframe video PES carries an
adaptation field with random_access_indicator=1. m2ts_mux previously
hardcoded RAI=1 on every PCR packet; that is now gated on the current
PES being a keyframe video PES, combining correctly with PCR when both
land on the same packet.

Adds 17 tests covering keyframe alignment, cue count/position/timing,
SeekHead correctness, RAI set/clear, codec_private gating, non-key drop,
and PCR+RAI combination.
2026-05-17 15:31:16 -07:00
MattJackson b08662f95c Revert "iter14: re-engage FileSectorSource 4 MiB readahead buffer"
This reverts commit 4594196b6e.
2026-05-17 10:50:00 -07:00
MattJackson 4594196b6e iter14: re-engage FileSectorSource 4 MiB readahead buffer
iter13 strace finding: producer thread (mux ISO reader) spends 80%
of wall-clock in state D (NFS RPC wait), doing 207 preads/sec at
103 us each. Pipe is 78 MB/s isolated read; we use ~25 MB/s.

FileSectorSource was using direct per-sector pread (0.21.3 bypass).
That bypass was justified under Phase 2.5 + 0.21.7 producer-poll cap
("32 MiB refill bursts the TCP connection enough to starve the
writer thread"). Both of those conditions are gone now (iter8
baseline: no Phase 2.5, no producer poll cap).

The 4 MiB buffer infrastructure was preserved with #[allow(dead_code)]
in case re-engagement was ever wanted. iter14 just routes the hot
path through buffer_covers + refill + memcpy. Net diff: ~10 LOC of
business logic, 5 unit tests already cover the contract.

Expected: producer's effective read rate jumps from 25 → 60+ MB/s
(matches dd ceiling for 1+ MiB block reads). If consumer side keeps
up, mean mux climbs from 28.7 → 35-50 MB/s. If consumer is now the
cap, we see a clear ceiling around 30-35 and we know where to look
next.

Critically: this change ONLY affects mux-from-ISO. Disc→ISO sweep,
Pass N bad-sector retry, AACS, drive ops, mapfile, recovery — all
untouched (they use DriveSectorSource which is a separate impl).
2026-05-17 10:36:04 -07:00
MattJackson 9d6974df3e iter13 base: lock chunk back to 32 MiB after iter12 revert
iter12 revert restored chunk to 128 (iter11 value) by way of the
revert chain. Setting it back explicitly to 32 for iter13. We're
back to iter8 baseline config (no Phase 2.5, 32 MiB chunks,
DONTNEED, no app-level FileSectorSource buffer, channel depth 32).

iter13 is instrumentation: strace the mux thread during a known-dip
to see where time is actually going.
2026-05-17 09:49:03 -07:00
MattJackson 3394a5b3fe Revert "iter12: fallocate without KEEP_SIZE + truncate_at_sync; chunk back to 32 MiB"
This reverts commit e2c7e20329.
2026-05-17 09:48:35 -07:00
MattJackson e2c7e20329 iter12: fallocate without KEEP_SIZE + truncate_at_sync; chunk back to 32 MiB
iter11 (128 MiB) crashed to 16.6 mean — chunk-size sweet spot is
firmly 32 MiB. Locking in.

iter12 hypothesis: each NFS write past the file's EOF triggers a
server-side SETATTR (metadata commit) to update the file length.
With ~62 GiB of MKV output and writes at chunk boundaries, that's
thousands of SETATTRs per rip. By using plain fallocate (mode 0)
the file is pre-extended to size_bytes up front; subsequent writes
overwrite the pre-extended region with no metadata ops.

Adds high_water tracking + truncate_at_sync so we ftruncate down
to actual content size if the size hint was an overestimate.
2026-05-17 09:35:07 -07:00
MattJackson ff22385a61 iter11: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB
Prediction: ~26-27 mean (regression from iter8). Confirms chunk-size
sweet spot at 32 MiB. Then move off chunk-size as a lever entirely.
2026-05-17 09:21:00 -07:00
MattJackson d39d3933f5 Revert "Reapply "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point""
This reverts commit f7165a2020.
2026-05-17 09:19:44 -07:00
MattJackson 31dc3d35c3 iter10: chunk back to 32 MiB
iter9 (64 MiB on top of no-P2.5) regressed -1.2 from iter8. 32 MiB
remains best on the no-P2.5 path. Net iter10 = iter8 minus DONTNEED
(via the prior commit re-applying f4a881b).
2026-05-17 09:06:34 -07:00
MattJackson f7165a2020 Reapply "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point"
This reverts commit 8ec767339e.
2026-05-17 09:06:14 -07:00
MattJackson d836f8b1e7 iter9: 64 MiB chunks on top of iter8 (Phase 2.5 disabled)
iter8 (Phase 2.5 disabled, 32 MiB chunks) hit 28.7 MB/s mean
— 1.3 MB/s below R2 floor. Without Phase 2.5, WAIT_AFTER blocks the
mux thread directly once per chunk. Halving the WAIT_AFTER frequency
(32 → 64 MiB chunks) should raise the mean without changing the
underlying architecture.
2026-05-17 08:52:51 -07:00
MattJackson 8919de5b03 iter8: chunk back to 32 MiB on top of Phase-2.5 revert
iter7's 64 MiB chunks didn't help (23.1 vs iter4's 24.0). Reverting
to 32 MiB so iter8 differs from iter4 by exactly one variable:
Phase 2.5 disabled / direct passthrough mux thread.

Test question: is Phase 2.5 helping at all? If iter8 mean > iter4,
Phase 2.5 has been a net negative on this rig the whole time. If
iter8 mean < iter4, Phase 2.5 is doing what it claimed.
2026-05-17 08:39:27 -07:00
MattJackson 0dc1f12108 Revert "Reapply "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)""
This reverts commit 335e8b68cc.
2026-05-17 08:39:06 -07:00
MattJackson 6abc25a7e8 iter7: WRITEBACK_CHUNK_BYTES 32 -> 64 MiB
iter6 (8 MiB chunks) regressed mean -8.2 MB/s vs iter4 (32 MiB).
Trend: bigger = better in this workload, against the page-age
theory. Try 64 MiB to see if fewer/larger syncs raise mean further.
0.21.14 went to 128 MiB and was reverted; 64 is between.
2026-05-17 08:25:44 -07:00
MattJackson 4d22cf632f iter6 cleanup: stop tracking (agent dir)worktrees/ 2026-05-17 08:10:29 -07:00
MattJackson 611ad77586 iter6: revert depth bump + WRITEBACK_CHUNK_BYTES 32->8 MiB
iter5 (256-frame channel) regressed -1.8 MB/s vs iter4. Reverting to
32 frames.

iter4 sample pattern shows clear ~30 s oscillation (peak 45 MB/s →
dip 3 MB/s → recovery). Matches Linux vm.dirty_expire_centisecs
default (30 s). 32 MiB chunks at 25 MB/s issue WAIT_AFTER every
~1.3 s, which can't outrun the kernel's own page-age limit, so
pages buildup then flush in bursts. Smaller 8 MiB chunks
(WAIT_AFTER every ~0.33 s) should keep the dirty-page set young
and eliminate the periodic flush-burst dip.
2026-05-17 08:09:57 -07:00
MattJackson 60590193e3 iter5: READ_PIPELINE_DEPTH 32 -> 256 frames
iter4 measurement (Phase 2.5 + DONTNEED, no FileSectorSource buffer)
showed mean 24.0 MB/s but with dips to 2.8 MB/s. Channel at 32 frames
(~1.6 MiB at 50 KB/frame avg) drains in <100 ms on any producer pause,
starving the consumer.

256 frames = ~12 MiB ≈ 3-4 sec of consumer drain at the rolling-avg
floor we want to hit. Should coast through producer micro-stalls
without idle gaps.
2026-05-17 07:53:41 -07:00
MattJackson 5f24843d3c Revert "iter2: restore FileSectorSource readahead buffer (32 MiB)"
This reverts commit ecf9a4d01e.
2026-05-17 07:38:34 -07:00
MattJackson ecf9a4d01e iter2: restore FileSectorSource readahead buffer (32 MiB)
iter1 baseline (Phase 2.5 + DONTNEED restored) measured at 18.4 MB/s
mean on Civil War remux — well below the rig's 37 MB/s concurrent-r+w
ceiling. Per-sector pread is the producer-side bottleneck: ~50us per
pread on NFS = ~19k preads/sec = effective ceiling near what we see.

The 0.21.3 bypass commit cited an A/B test showing the buffer hurt
throughput on NFS bidirectional workloads. That test was taken under
the 0.21.7 producer polling cap; once the cap is gone, the cap was
the bottleneck, not the buffer. Same invalidation pattern as the
0.21.5 Phase 2.5 revert.

Restored the buffered path. DONTNEED page-cache eviction from 0.21.6
stays intact.
2026-05-16 22:46:22 -07:00
MattJackson 8ec767339e Revert "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point"
This reverts commit f4a881b25d.
2026-05-16 17:02:20 -07:00
MattJackson 335e8b68cc Reapply "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)"
This reverts commit 579c94bd23.
2026-05-16 17:02:15 -07:00
MattJackson 1137b4ea1f v0.23.2: bump version 2026-05-16 12:36:28 -07:00
MattJackson 7788c560d1 v0.23.1: bump version 2026-05-16 11:35:41 -07:00
MattJackson 84e0c2ca69 v0.23.0: bump version 2026-05-16 11:17:34 -07:00
MattJackson f4a881b25d io/writeback: medium-agnostic by construction, no DONTNEED, single detection point
Three coupled changes that together meet the "stable max throughput
on any medium" bar:

1. Remove `is_nfs` from `skip_wait`. WAIT_AFTER runs on every medium
   now. The `bounded_syscall` 30s safety net (already in place) covers
   the wedged-FS case generically — no need to predict NFS-hangs at
   compile time. Re-applies the 0.21.12 fix that landed the flat
   25 MB/s on NFS in the first place.

2. Drop the `posix_fadvise(DONTNEED)` calls after WAIT_AFTER. Bounded
   *dirty* pages (the actual invariant) does not require evicting
   *clean* pages. DONTNEED was forcing read-modify-write on any
   in-window seek-then-write — exactly the pattern matroska
   cluster-size backpatches produce. Empirical 2026-05-15: write side
   of mux fed NFS at 43 MB/s while the file grew at 25 MB/s, an 18
   MB/s overhead almost certainly composed of those RMW cycles. The
   kernel reclaims clean pages under LRU when memory is actually
   needed — we don't have to ask.

3. Replace `detect_nfs(fd) -> bool` with `detect_storage_class(fd)
   -> (StorageClass, chunk_bytes_seed)`. Medium detection happens
   *once* at construction and produces *one* output: the initial
   `chunk_bytes` seed for the autotuner. NFS gets 64 MiB (commit ack
   ~10-30 ms; needs bigger chunks to amortize); other media use the
   caller's hint. No hot-path branches on medium. The autotuner drives
   all subsequent decisions from measured WAIT_AFTER p95 latency,
   identically on every (OS, FS) combination.

Module-level doc rewritten to match: no more "NFS escape hatch", no
"is_nfs" framing. The whole writeback module is now medium-agnostic
except for one labelled detection point.
2026-05-16 11:17:30 -07:00
MattJackson 1edda7fedb v0.22.1: bump version 2026-05-16 10:12:34 -07:00
MattJackson 3dd71a0650 v0.22.0: bump version 2026-05-16 09:39:44 -07:00
MattJackson 579c94bd23 Revert "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)"
This reverts commit 2a33364253.
2026-05-16 09:30:44 -07:00
MattJackson 3b2a68fe7a Revert "io/pipeline: bump WRITE_PIPELINE_DEPTH 16 → 32"
This reverts commit d703ce439b.
2026-05-16 09:30:44 -07:00
MattJackson 5b56f112a0 Revert "io/writeback_file: coalesce consecutive Cmd::Writes in the writer thread"
This reverts commit fc13268dc6.
2026-05-16 09:30:44 -07:00
MattJackson f27e4c3088 Revert "io/writeback/linux: drop is_nfs skip — bounded cache works on every medium"
This reverts commit dffee56102.
2026-05-16 09:30:44 -07:00
MattJackson b3576f2ae6 Revert "io/writeback: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB"
This reverts commit 50867516b8.
2026-05-16 09:30:44 -07:00
MattJackson 39f2d0e992 v0.21.14: bump version 2026-05-15 12:02:43 -07:00
MattJackson 50867516b8 io/writeback: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB
The pipeline's adaptive autotuner grows chunk_bytes only when the p95
WAIT_AFTER latency exceeds 200 ms. On NFS, sync_file_range(WAIT_AFTER)
translates to an NFS COMMIT RPC whose ack lands within ~10 ms — so
the autotuner never triggered and the pipeline stayed at the original
32 MiB initial value forever.

That capped sustained mux throughput by paying NFS COMMIT-RPC overhead
roughly once per second of writes. Bidirectional mountstats on rip1
2026-05-15: write side 29 MB/s (RTT 30 ms but exec_time 257 ms — 220 ms
queue/serial waiting) while concurrent dd on the same disk shows
~91 MB/s write + ~65 MB/s read available.

128 MiB initial drops COMMIT cadence 4x while keeping the bounded-
cache invariant intact (worst-case dirty pages ~2 x chunk = 256 MiB,
well under vm.dirty_ratio = 6.6 GB on the 32 GB rig). The adaptive
autotuner can still grow further (up to 256 MiB) or shrink if
WAIT_AFTER ever measures sub-20 ms p95 on faster media.
2026-05-15 12:02:40 -07:00
MattJackson 193e7c1680 v0.21.13: bump version 2026-05-15 10:20:37 -07:00
MattJackson b6463729e8 v0.21.12: bump version 2026-05-15 09:55:41 -07:00
MattJackson dffee56102 io/writeback/linux: drop is_nfs skip — bounded cache works on every medium
The WritebackPipeline's WAIT_AFTER + posix_fadvise(DONTNEED) dance
keeps dirty pages bounded at ~2 × chunk_bytes by waiting for each
chunk's writeback to commit before issuing the DONTNEED hint to drop
it from cache. The original 0.18-era design unconditionally skipped
this on NFS on the premise that "NFS clients have their own buffering
and commit semantics that handle dirty-page bounds without us forcing
the issue."

Empirically wrong. On unraid-1 NFS the kernel client buffers dirty
pages up to vm.dirty_ratio (default 20% of RAM = ~6.6 GB on the rip1
host) before the kernel forces writeback and throttles app writes.
Result on 0.21.11 mux measured 2026-05-15: mux throughput cycled
between ~45 MB/s (cache absorbing) and ~7 MB/s (cache draining under
throttle) on a ~100 s period — exactly the burst-flush pathology this
pipeline was built to fix, but disabled on the medium it actually
runs on. /proc/meminfo Dirty: column climbed lockstep with mux
write rate during the slow half of every cycle, confirming the cause.

The original safety concern — `sync_file_range(WAIT_AFTER)` hanging
indefinitely on a wedged NFS server — is already handled by
`wait_after_with_timeout`'s `bounded_syscall` wrapper (30 s deadline).
If a real WAIT_AFTER call exceeds the deadline the pipeline flips to
the `degraded` state and skips WAIT_AFTER + DONTNEED for the rest of
its life — same effect as the old NFS branch, but only triggered when
something is genuinely broken rather than as a blanket exception.

This change is medium-agnostic: every medium goes through the same
path now, every medium gets the same safety net, and the
ADAPTIVE_WINDOW chunk-size autotuner (lines 226-255 — measures p95 of
WAIT_AFTER and resizes between 4 MiB and 256 MiB) finally activates
on NFS where previously it was dead code. Slow medium auto-grows
chunks to amortise per-chunk overhead; fast medium auto-shrinks to
keep cache pressure tight; nothing in the code special-cases the
filesystem type.

`is_nfs` is still detected (for logging + observability) but no
longer keys `skip_wait`. Module doc + startup log line updated to
match.
2026-05-15 09:55:38 -07:00
MattJackson d8f27cdee0 v0.21.11: bump version 2026-05-14 22:18:33 -07:00
MattJackson fc13268dc6 io/writeback_file: coalesce consecutive Cmd::Writes in the writer thread
Pre-coalescing the Phase-2.5 writer thread issued one `file.write_all`
syscall per `Cmd::Write` dequeued. The mux side calls
`WritebackFile::write_all(buf)` per PES frame, typically 30-200 KB.
On NFS that translates to one RPC per syscall, capping per-thread
throughput at `(wsize / rtt) × inflight` — well below what the same
disk delivers under a 1 MiB `dd oflag=direct` workload (empirical
2026-05-15: dd 71 MB/s vs mux ~25 MB/s sustained, with instantaneous
samples bursting 7→108 MB/s as the kernel page cache filled and
drained on its own cadence).

Coalesce instead: dequeue drains consecutive `Cmd::Write` items off
the ring up to a 1 MiB byte budget, returns them as
`DequeuedWork::Writes(Vec<Vec<u8>>)`, and the run loop concatenates
into one contiguous buffer and issues a single `file.write_all`.
Non-write commands (Seek, Flush, SyncAll, Finish) break the run and
are returned one at a time as `DequeuedWork::Other`, preserving their
ordering relative to the writes.

Single-buffer fast path avoids the concat allocation when only one
write is in the queue at dequeue time. A single oversize write (e.g.
the rare matroska cluster larger than 1 MiB) is admitted alone so it
still makes progress — the kernel splits internally.

This is generic across mediums: bigger app writes are at-least-as-
good on local SSD, HDD, or NFS. On fast storage the ring rarely fills
so coalescing is mostly a no-op; on slow storage with significant
per-RPC overhead it materially improves throughput.

Tests in `tests/` (write_then_drop_persists_bytes, sync_all_drains,
seek_then_patch_roundtrip, flush_is_observed_in_order) still pass —
ordering and durability semantics are unchanged.
2026-05-14 22:18:23 -07:00
MattJackson d9b55f02d0 v0.21.10: bump version 2026-05-14 20:43:21 -07:00
MattJackson 35d66d2059 v0.21.9: bump version 2026-05-14 20:21:36 -07:00
MattJackson d703ce439b io/pipeline: bump WRITE_PIPELINE_DEPTH 16 → 32
The depth was conservative because pre-0.21.8 a full sync_file_range
stall on NFS could traverse this channel and pin the producer. With
0.21.8's restored Phase 2.5 writer thread + 128 MiB byte-bounded ring
inside WritebackFile, every blocking syscall happens downstream of
this channel — never on it. The original "smaller buffer reduces
backpressure risk" rationale no longer applies.

Empirical (2026-05-15 Civil War UHD remux on 0.21.8): 30.9 MB/s
sustained but instantaneous samples spanning 9-53 MB/s, stdev 9.3.
Distribution clusters 52% of samples in 25-35 MB/s but has a long
9-15 MB/s tail. The tail corresponds to brief stalls in the matroska
builder when the sink momentarily lags — exactly the case a deeper
inter-thread channel covers. 32 frames at PES-frame sizes is still
well under a megabyte of additional memory, so the cost is zero.
2026-05-14 20:21:20 -07:00
MattJackson 6112a26d15 v0.21.8: bump version 2026-05-14 18:54:10 -07:00
MattJackson 2a33364253 io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)
Reverts 523af46. That revert was made on the premise that Phase 2.5
caused a ~60% mux throughput regression on NFS bidirectional workloads.
The premise was wrong: at the time of measurement the producer was
capped at ~8 MB/s by a 50 ms thread::sleep poll in
Pipeline::send_with_halt (fixed in v0.21.7's io/pipeline change), so
the comparison was measuring the polling cap on both sides.

With the polling cap removed, direct passthrough exposes the kernel's
default dirty-page writeback pathology on NFS: writes accumulate, the
kernel periodically bursts a flush, app writes block for the burst.
Observed empirically on Civil War UHD remux 2026-05-14: 5-45 MB/s
spiking around a ~21 MB/s sustained mean, dominated by burst-flush
back-pressure cycles.

Phase 2.5 decouples the mux thread from the file syscall:
  * mux writes complete instantly into a 128 MiB byte-bounded SPSC ring,
  * a dedicated writer thread executes the real File writes, seeks,
    and sync_file_range calls; can sit in a kernel burst without
    blocking the mux pipeline,
  * backpressure via Condvar notify/wait, no polling primitive,
  * the ActiveClusterBuffer fast-path preserves the original MKV
    cluster-backpatch optimisation so in-window seeks don't drain
    the current writeback chunk.

Halt-safety is preserved: every blocking writeback syscall on the
writer thread still routes through bounded_syscall with a 60 s
deadline. A wedged NFS server cannot trap the writer indefinitely;
the muxer keeps queueing into the ring; the kernel page cache and
the ring together absorb the stall.
2026-05-14 16:42:00 -07:00
MattJackson fa3872ddcc v0.21.7: bump version 2026-05-14 11:10:57 -07:00
MattJackson c427389f36 io/pipeline: replace polling-send with kernel-wakeup channel
The halt-aware send loop polled try_send on a 50 ms sleep slice when
the channel was full. That capped producer throughput at 1 / 50 ms =
20 frames/sec ≈ 1 MB/s at typical PES frame sizes — way below NFS,
let alone local SSD/NVMe. Multi-day diagnostic 2026-05-13/14 surfaced
it as the root cause of the 18 → 3 MB/s mux throughput regression on
0.21.x.

Replaced std::sync::mpsc::sync_channel with crossbeam_channel::bounded
and switched send_with_halt to send_timeout. Producer now BLOCKS on
consumer drain (kernel-wakeup) instead of polling — the timeout slice
(250 ms) only fires when the producer is waiting for stop signal
observation, never on the happy path.

Result: no throughput cap from this primitive at any medium speed.
Mux is bounded by actual storage / network bandwidth, not by our
channel implementation.

Same fix needs to land in autorip's mux.rs producer-side
sync_channel (separate sibling commit).

Documented in (internal)/memory/
feedback_send_with_halt_poll_throttle.md.
2026-05-14 11:10:39 -07:00
MattJackson 6980f562df v0.21.6: bump version 2026-05-14 09:16:34 -07:00
MattJackson e3b2c9d850 file_sector_source: restore read-side DONTNEED + SEQUENTIAL (the actual fix)
Empirical: isolated NFS read 70 MB/s + write 93 MB/s on the rip1 setup
right now, but mux throughput pinned at 2.7 MB/s on 0.21.5. NOT
environmental — code regression.

Root cause: Phase 1 silently dropped the read-side
posix_fadvise(POSIX_FADV_DONTNEED) eviction that the pre-Phase-1 (0.20.7)
hot path had. Without it, an 85 GB streaming ISO read pins the entire
file in the kernel page cache, starving concurrent MKV writeback. 0.21.2
then also dropped the POSIX_FADV_SEQUENTIAL hint on the same theory,
compounding the regression.

Restored both, per-OS split:
- linux: posix_fadvise(SEQUENTIAL) at open + posix_fadvise(DONTNEED)
  on consumed 32 MiB windows
- macos: F_RDADVISE hint at open (kept); drop_window no-op (macOS unified
  buffer cache less prone to the pin pathology)
- windows / other: both no-op stubs

Target mux speed restored to 20+ MB/s (per concurrent-NFS math:
70/2 read × 0.73 MKV/ISO ratio ≈ 25 MB/s achievable).
2026-05-14 09:16:23 -07:00
MattJackson a383200ef1 v0.21.5: bump version 2026-05-14 01:46:59 -07:00
MattJackson 51d15f551e file_sector_source: allow(dead_code) on residual buffer fields post-0.21.3 bypass 2026-05-14 01:46:51 -07:00
MattJackson 523af461a5 io/writeback_file: revert write path to direct passthrough (0.20.7 baseline)
The Phase 2.5 writer-thread + bounded ring + ActiveClusterBuffer
architecture introduced a ~60% mux throughput regression on NFS
bidirectional workloads (18 MB/s -> 7-8 MB/s in 0.21.x). Other
candidates (FileSectorSource readahead buffer, per-OS sink split)
ruled out empirically across 0.21.1-0.21.4.

Reverts WritebackFile's hot path to direct File passthrough matching
the 0.20.7 baseline:
  * Write/write_all/flush call straight through to self.file.
  * Seek calls through to self.file and notifies the pipeline.
  * sync_all runs pipeline.finalize() then per-OS durable_sync.

Kept (intentional): the per-OS file split established in earlier work
(writeback_file/{linux,macos,windows,other}.rs). create_with_size_hint
still dispatches to platform::preallocate and sync_all still routes
through platform::durable_sync (bounded_syscall 60s deadline on
Linux/macOS), so halt-safety on a wedged NFS fsync is preserved.

Removed:
  * Cmd / RingState / Shared / WriterState / writer_thread_main
  * ActiveClusterBuffer (active-cluster window for in-window seeks)
  * push_command / dequeue / publish_error / mark_writer_gone
  * RING_CAPACITY_BYTES, ACTIVE_CLUSTER_WINDOW_BYTES,
    MAX_WRITE_CHUNK_BYTES, WRITER_THREAD_NAME constants
  * writer JoinHandle field + Drop join logic
  * Tests that asserted internal writer-thread state
    (backpressure_blocks_when_ring_full, in/out_of_window_seek_*,
    active_cluster_buffer_*, writer_thread_panic_surfaces_on_drop)

Kept tests (correctness-of-output only):
  * write_then_drop_persists_bytes
  * sync_all_drains_and_flushes (renamed from sync_all_blocks_until_ring_drains)
  * seek_then_patch_roundtrip (replaces in_window_seek_then_patch_roundtrip)
  * flush_is_observed_in_order

mod.rs: 1026 -> 303 lines. linux/macos/windows/other.rs untouched.
2026-05-14 01:40:37 -07:00
MattJackson 695b65149e v0.21.4: bump version 2026-05-14 00:09:28 -07:00
MattJackson d3f9560689 test/file_sector_source: drop buf-state assertions after 0.21.3 bypass
The three tests (multi_sector_read_spanning_buffer_boundary,
backward_seek_rebuffers, partial_buffer_at_eof) were asserting
internal buf_start_lba / buf_len_sectors state. With 0.21.3's
read-path bypass, those fields are no longer mutated. The
byte-level contract assertions (read returns correct bytes for
every scenario the tests cover) remain intact.
2026-05-14 00:09:21 -07:00
MattJackson 1fa5a7d27f v0.21.3: bump version 2026-05-13 23:57:53 -07:00
MattJackson 55219070f1 io/file_sector_source: bypass app-level buffer — direct pread per call
The 32 MiB readahead window (0.21.0–0.21.1) regressed mux throughput
on NFS bidirectional workloads vs the pre-Phase-1 0.20.7 baseline
(18 -> 7-8 MB/s). The 0.21.2 4 MiB shrink made it worse (5-6 MB/s).
Both signs point at the application-level buffer itself, not the size.

This commit bypasses the buffer entirely on the read path — every
read_sectors call seeks and pread()s direct to the file. That matches
0.20.7's hot path. Kernel readahead handles the policy; on NFS that
interleaves naturally with concurrent writes on the same TCP
connection.

Buffer state fields and refill/buffer_covers are kept so the
structure is preserved for a future per-source-type policy (e.g. a
local-disk source where batched reads ARE beneficial), and so the
existing tests still exercise that machinery.
2026-05-13 23:57:46 -07:00
MattJackson a34c419521 v0.21.2: bump version 2026-05-13 22:28:09 -07:00
MattJackson b179846f5d io/file_sector_source: throttle readahead for NFS bidirectional workloads
Empirical regression observed during 0.21.1 mux test on rip1/unraid-1:
historical 0.20.7 baseline averaged ~18 MB/s mux throughput; 0.21.1
dropped to ~7-8 MB/s flat. Same NFS source + destination, same disc.

Suspect: 32 MiB FileSectorSource readahead + posix_fadvise(SEQUENTIAL)
together saturate the TCP connection on read bursts, starving the
writer thread's concurrent NFS writes (mux reads the source ISO and
writes the MKV over the same connection).

- READAHEAD_BUF_BYTES: 32 MiB -> 4 MiB. Matches NFS rsize=1 MiB * 4
  round-trips per refill, interleaves cleanly with writes.
- linux/hint_sequential: now no-op. Kernel's default ~128 KiB
  readahead is what we want on NFS-backed ISOs (the dominant case).
  Per-OS file stays so we can re-enable a hint cleanly later if a
  different path benefits.
2026-05-13 22:27:57 -07:00
MattJackson e110e80e6e lint: silence clippy::unnecessary_cast on glibc + fix doc list indent
CI's lint workflow runs clippy on linux target where:
- platform/fs_type/linux.rs and io/writeback/linux.rs: the i64 cast
  on buf.f_type / NFS_SUPER_MAGIC is unnecessary on glibc x86_64 (both
  already i64) but required on musl (c_ulong); silence the lint via
  inline allow with explanatory comment.
- mux/m2ts_mux/packet.rs: doc comment continuation across lines was
  parsed as an unindented list item. Reworded to a single flowing
  sentence.
2026-05-13 20:53:26 -07:00
MattJackson 597fa34099 v0.21.1: bump version 2026-05-13 20:30:09 -07:00
MattJackson 52eeb949e3 mux: Rust 1.86 compat + fmt
- m2ts_mux: replace u64::is_multiple_of (stable in 1.87+) with %.
  CI on the pinned 1.86 toolchain rejected the unstable feature use.
- mux/{fmp4,hevc,m2ts_mux}: rustfmt drift cleanup (test-code wrapping).
2026-05-13 20:30:01 -07:00
MattJackson 6ca62b8411 v0.21.0: bump version 2026-05-13 20:17:47 -07:00
MattJackson 04195c27d8 io+mux: phase 3 — streaming sinks + sequential container muxers
SocketSink + UdpSocketSink (`src/io/sink/socket.rs`) — sequential-only
TCP/UDP write destinations. SocketSink wraps BufWriter<TcpStream> with
1 MiB capacity, tunes SO_SNDBUF on construction, calls shutdown(Write)
on finish(). UdpSocketSink emits one datagram per write — caller
packetizes. Both impl Write+Send and thus satisfy SequentialSink via
the Phase 2 blanket; neither impls Seek, so RandomAccessSink is
correctly inaccessible (compile error to mux MKV onto a socket).

New sequential container muxers in src/mux/:

  - hevc/ — raw HEVC Annex B elementary stream. Length-prefixed NALU
    → 00 00 00 01 NALU. hvcC parsing emits VPS/SPS/PPS once at stream
    head. Fully ships.

  - m2ts_mux/ — standard MPEG-TS (188-byte packets). Single program,
    HEVC video on PID 0x100, optional AC3/TrueHD audio on PID 0x101.
    PAT+PMT re-emitted every 250 packets; PCR stamped on video every
    40 packets. Hand-rolled, no new deps. Distinct from the existing
    BD-TS (192-byte) `mux::m2ts::M2tsStream` — that path stays as-is.

  - fmp4/ — fragmented MP4. STUB: ftyp + minimal moov skeleton with
    one HEVC video trak + mvex/trex. Media fragments (moof+mdat) are
    TODO for v0.22.0 — write_video accumulates frames into a pending
    buffer that finish() clears. Init segment is well-formed enough
    that init_segment_starts_with_ftyp_then_moov asserts the box
    chain.

17 new unit tests added (socket round-trip, HEVC Annex B conversion,
M2TS packet alignment + PAT/PMT cadence + per-PID CC, fMP4 box chain).
All 514 lib tests + 17 new = pass on Rust 1.86 (fmt + clippy + test
via (internal)/scripts/precommit.sh libfreemkv).

No new dependencies. No version bump. Don't-touch list clean.
2026-05-13 20:17:33 -07:00
MattJackson 5a8f8e54e1 io: phase 2.5 — writer thread + bounded ring + active-cluster buffer
WritebackFile now offloads all File I/O to a dedicated writer thread.
Muxer's Write/Seek/sync_all push into a bounded byte ring; the writer
drains the ring and runs the syscalls.

- Commands: Write(Vec<u8>), Seek(SeekFrom), SyncAll(oneshot),
  Finish(oneshot). One ring carries all four; ordering preserved.
- ActiveClusterBuffer mirrors the last ACTIVE_CLUSTER_WINDOW_BYTES
  written. Seek-back within window → in-memory patch + re-emit
  (no forced drain). Seek-back outside window → real drain+seek.
  Wins for MKV: cluster size patches almost always land inside the
  active window; only the end-of-mux Cues + segment header backpatch
  fall outside.
- Ring capacity RING_CAPACITY_BYTES = 128 MiB. Backpressure on full
  blocks the muxer (correct semantics for archival workflows).
- All syscalls in writer thread wrapped in bounded_syscall(60s) so
  a wedged NFS doesn't pin the thread forever.
- Drop drains via Finish + joins the writer thread; sync_all blocks
  until ring drained AND underlying fsync completes.
2026-05-13 20:17:29 -07:00
MattJackson 1a2c830cf6 fmt: rustfmt reorder of cfg-gated use/mod decls in sink + fs_type 2026-05-13 20:10:57 -07:00
MattJackson 637c2bfe36 v0.20.10: bump version 2026-05-13 19:59:07 -07:00
MattJackson 0d28357b4d io+platform: phase 2 — sink trait split + fs_type detection
Introduces the SequentialSink / RandomAccessSink trait pair under
io::sink and an open_for_mkv dispatch helper that picks WritebackFile
on Linux+NFS and LocalFileSink everywhere else. LocalFileSink wraps
BufWriter<File> with a 4 MiB buffer and exposes a per-OS preallocate
path (fallocate on Linux, F_PREALLOCATE on macOS, no-op fallback).

Adds platform::fs_type::detect with a per-OS split (statfs on Linux /
macOS, UNC heuristic on Windows, Unknown elsewhere) so construction-
site dispatch has a single primitive to call.

Blanket impls cover the common shapes: any Write+Send is a
SequentialSink, and any SequentialSink+Seek is a RandomAccessSink.
WritebackFile satisfies the random-access trait via the blanket impl
without needing an explicit per-type impl. No callers wired yet — the
mux::resolve construction sites stay on WritebackFile pending Phase 3.

Tests: 5 new sink/preallocate tests + 3 fs_type tests (1 ignored,
needs a real NFS mount). cargo +1.86 fmt + clippy + tests all green.
2026-05-13 19:58:54 -07:00
MattJackson 5495332f07 v0.20.9: bump version 2026-05-13 19:52:48 -07:00
MattJackson e22fc6fd47 io: phase 1 buffering — read-side flatness
Three changes targeting 0.20.9's "muxer never read-stalls on NFS read
latency" invariant:

A. FileSectorSource gets a 32 MiB internal read-ahead buffer
   (READAHEAD_BUF_BYTES). Splits out from src/sector/file.rs into
   src/io/file_sector_source/ with per-OS open hints (Linux
   posix_fadvise(SEQUENTIAL), macOS fcntl(F_RDADVISE) with 64 MiB
   cap, Windows TODO stub, BSD/illumos no-op). Backward seeks
   rebuffer; partial reads at EOF return only the bytes that exist;
   oversize-request bypass for count > BUF_SECTORS.

B. WritebackFile inline #[cfg(target_os = "linux")] blocks split
   into per-OS files under src/io/writeback_file/. Linux unchanged
   (fallocate KEEP_SIZE, fsync via bounded_syscall). macOS gets a
   real F_PREALLOCATE + F_FULLFSYNC impl (was a "skipped (non-linux)"
   debug log before). Windows is a stub (FlushFileBuffers via
   std sync_all; TODO for SetFileValidData). BSDs/illumos fall back
   to std sync_all.

C. New byte_channel module — byte-bounded producer/consumer wrapping
   std sync_channel with Mutex/Condvar byte accounting. Sender blocks
   when used_bytes + item.byte_size() > capacity. HasByteSize impl
   for PesFrame. Default cap BYTE_CHANNEL_DEFAULT_CAPACITY = 64 MiB,
   sized to absorb worst-case NFS read p99 (~2 s × UHD peak compressed
   ~15 MB/s). The mux call site lives in autorip (out of scope here);
   this lands the primitive in libfreemkv for autorip to adopt.

Test counts: byte_channel +6, file_sector_source +5, sector::file
round-trip suite (3) preserved. passn_handler_ab.rs A/B fixture
(8 profiles) still green.

precommit.sh libfreemkv: fmt + clippy + test all green on Rust 1.86.

No version bump; no Cargo.lock changes; no forbidden-file edits
(disc/patch.rs, disc/read_error.rs, io/pipeline.rs,
tests/passn_handler_ab.rs).
2026-05-13 19:48:23 -07:00
MattJackson 5b98c13e47 v0.20.8: bump version 2026-05-13 19:18:51 -07:00
MattJackson 8f8f1a62a2 io+disc: bundle 0.20.8 dev work
- io/pipeline.rs: add send_with_halt + finish_with_halt for cooperative
  halt during blocking producer-consumer handoffs; 5 new tests
- disc/patch.rs: split Disc::patch body (1168 -> 316 LOC) into named
  helpers (compute_initial_state, prime_cache, check_range_watchdog,
  handle_skip_limit, compute_damage_skip, handle_read_success,
  handle_read_failure, report_patch_progress, build_outcome) with
  PatchLoopState / RangeFrame structs; references shared
  PATCH_DAMAGE_THRESHOLD_PCT constant
- disc/read_error.rs: add pub const PATCH_DAMAGE_THRESHOLD_PCT = 6;
  ReadCtx::for_patch() now references the shared constant (was a
  latent 12 / 6 inconsistency)
- tests/passn_handler_ab.rs: 8-profile A/B fixture locking current
  patch-side recovery behavior (clean / all-medium / alternating /
  edge-bad-good-middle / single-bad / deep-pit / medium-then-good /
  batch-fail). Goldens captured pre-unification; will catch any
  future refactor that breaks the size-aware skip cap.
2026-05-13 19:15:48 -07:00
MattJackson e3cfd27942 v0.20.7: version bump for unified release (no source changes)
The 0.20.7 work lives downstream in autorip — process-level safety net
(hard watchdog 5-min mux escalation + std::process::exit, restart-loop
counter + auto-quarantine after 3 attempts, partial-state preservation,
failure_reason surfaced in /api/state).

Bumping libfreemkv to 0.20.7 keeps the 4 crates on the unified release
train per project docs.
2026-05-13 14:29:40 -07:00
MattJackson aa12bdad62 v0.20.6: io::bounded — halt-safe wrapper for blocking syscalls
Generalizes 0.20.5's hand-written wait_after_with_timeout into a
reusable primitive. After this change, every blocking syscall in the
recovery + mux paths is wrapped, so cooperative Halt has bounded
~250 ms latency reach even into kernel-owned thread states.

New module src/io/bounded.rs:
- BoundedError { Halted, Timeout, WorkerLost }
- bounded_syscall<F, R>(halt: Option<&Halt>, timeout, op) -> Result<R, BoundedError>
- Worker thread runs op; main thread recv_timeouts on a rendezvous
  channel in 250 ms slices, polling halt between slices.
- Worker is intentionally leaked on timeout/halt — kernel reaps when
  the syscall finally returns or at process exit. Calling thread is
  NEVER trapped inside a kernel call.
- 6 unit tests cover the happy path + each error variant.

Refactored callsites:
- src/io/writeback/linux.rs::wait_after_with_timeout now delegates
  to bounded_syscall. ~30 LOC of duplicated channel/thread plumbing
  deleted. Same semantics, cleaner.
- src/io/writeback_file.rs::WritebackFile::sync_all now wraps the
  final libc::fsync(fd) with bounded_syscall (60 s deadline). On
  timeout: log error at target=mux and return Ok — kernel will flush
  on close, best-effort but bounded. Covers FileSectorSink::finish,
  PatchSink::close, SweepSink::close, and the mux MKV finalize path
  (they all sync through WritebackFile).

What still hangs (deliberately not wrapped — too hot a path):
- File::write itself. Per-frame write on a wedged NFS could still
  block; but back-pressure from a stuck consumer means the producer
  notices within seconds, not minutes — different failure mode than
  the WAIT_AFTER hang 0.20.5/0.20.6 fix.
2026-05-13 14:22:24 -07:00
MattJackson ef3895cdc5 v0.20.5: NFS-aware writeback + bounded sync_file_range timeout
Targets the recurring mux hang on NFS dest where the consumer thread
sits indefinitely inside libc::sync_file_range(SYNC_FILE_RANGE_WAIT_AFTER)
because the NFS server never returns a commit ack. The whole rip
wedges; halt is cooperative and can't reach inside a kernel syscall.

A. NFS detection at WritebackPipeline construction (fstatfs f_type ==
NFS_SUPER_MAGIC 0x6969). When NFS:
- Skip SYNC_FILE_RANGE_WAIT_AFTER entirely.
- Skip posix_fadvise(DONTNEED) — NFS client handles its own buffering.
- Still issue async SYNC_FILE_RANGE_WRITE (harmless hint).
Cannot hang on a syscall not made. fstatfs failure fails open (assume
local). Logged at info on construction so operators see which strategy
is active. The whole hang vector is removed for NFS deployments.

B. Hard timeout on WAIT_AFTER for non-NFS (defense in depth, since
even a degraded local disk could in principle hang the syscall).
Each WAIT_AFTER runs on a worker thread; main thread waits on a
sync_channel rendezvous with 30s deadline. On timeout: log error,
set per-pipeline 'degraded' Arc<AtomicBool>, downgrade to NFS-style
skip for the rest of the pipeline's life. Worker thread leaks
intentionally — it'll unwind when the syscall eventually returns or
the process exits. Converts indefinite freeze into 'log loud +
downgrade + keep ripping'.

C. Diagnostic logging for the 73%-of-this-movie reproduction:
- WritebackFile::seek logs every non-trivial seek (from, to, signed
  delta) at target=mux so we can see if MkvMuxer seeks back before
  a stall.
- WritebackPipeline::finalize logs the chunk being finalised before
  any WAIT_AFTER call, so a hung chunk is identifiable by offset.

No new dependencies. macOS / Windows noop stubs unchanged. Net
+198 LOC libfreemkv (mostly writeback/linux.rs).
2026-05-13 14:09:55 -07:00
MattJackson 2dcf969ac8 v0.20.4: mux performance + observability — universal across storage
Four targeted changes to maximize mux throughput regardless of storage
backend (local SSD, local HDD, NFS, network share) and surface enough
log data to diagnose 'mux slow' reports without a re-rip:

1. POSIX_FADV_SEQUENTIAL on FileSectorSource::open (Linux only).
   Widens the kernel readahead window for sequential ISO reads. One
   syscall at open, free on every storage type.

2. POSIX_FADV_DONTNEED on the ISO read side after every 32 MiB chunk.
   Mirrors the writeback DONTNEED that already runs on the write
   side. Keeps the read-side page cache bounded during multi-GB ISO
   reads — eliminates the OOM-pressure / eviction-storm risk on
   long mux runs. Linux only; per-drop trace at target="mux".

3. WritebackFile::create_with_size_hint(path, size_bytes) calls
   fallocate(FALLOC_FL_KEEP_SIZE) on Linux to pre-reserve extents
   for the output. Reported file size stays 0 (writes grow it
   naturally) but the on-disk extent allocation is contiguous —
   reduces extent fragmentation for big sequential muxes. Wired
   into mkv:// and m2ts:// output paths via DiscTitle::size_bytes.
   No-op on macOS/Windows; old create() kept with #[allow(dead_code)]
   for callers without a size hint.

4. Adaptive WRITEBACK_CHUNK_BYTES in the Linux writeback pipeline.
   Tracks sync_file_range(WAIT_AFTER) elapsed_ms in a rolling
   16-sample window. p95 > 200 ms → double chunk size (cap 256 MiB).
   p95 < 20 ms → halve (floor 4 MiB). One algorithm, both
   fast-storage (small chunks, responsive) and slow-storage (big
   chunks, fewer commit round-trips) optimized. Per-chunk trace +
   per-32-chunk debug snapshot + info-on-resize so an operator can
   see where the autoscaler settled.

All four are universal — no storage-type detection, no env vars to
flip, no per-deploy tuning required. Total +201/-6 across four files.
2026-05-13 13:50:44 -07:00
MattJackson 005f887bf9 v0.20.3: add halt check to Disc::patch backtrack inner loop
WO-5 (partial): the patch backtrack inner loop ('while bt_pos <
backtrack_end' in disc/patch.rs) issues per-sector reads to fill the
gap created by a damage-window skip. A long backtrack span can run
minutes; without an inline halt poll, the outer halt only takes
effect when control returns to the per-range loop. Adds a halt poll
at the top of each iteration so cancellation propagates inside the
backtrack span.

Per-sector read failures inside the backtrack already drop through
to the main fail path; this only changes the cancellation latency
between an /api/stop call and the producer actually unwinding. Drops
worst-case unwind from 'whole backtrack span × per-sector recovery
timeout' (minutes) to 'one in-flight SCSI command' (seconds).

The broader Arc<AtomicBool> → Halt migration on CopyOptions /
SweepOptions / PatchOptions / Drive::halt and Pipeline::send halt-
awareness is deferred — separate cycle, larger API impact.
2026-05-13 11:55:52 -07:00
MattJackson d7243a6044 v0.20.2: delete dead retry block + wedge-family cooldown in Disc::patch
WO-3c: Delete dead non-NOT_READY retry block (~100 LOC). The block
declared retry_count = 0 inside the per-iteration Err arm, so the
'MAX_NON_NOT_READY_RETRIES=3' budget actually fired exactly once
(1s pause + 1 retry) before falling through to NonTrimmed. The
'exponential backoff: 2s, 4s, 8s' comment was wrong by construction.
Cross-pass NonTrimmed retry (each pass gives the same sectors another
shot) already covers the recovery case, and gives the drive minutes
between attempts instead of 1-8 seconds — empirically what stochastic
recovery on the BU40N actually needs.

WO-4 (targeted slice): Add wedge-family cooldown on HARDWARE_ERROR /
ILLEGAL_REQUEST senses. These are what the BU40N's firmware fast-fail
state returns; every subsequent read in that state comes back in
<100ms. Pre-fix patch hammered the drive: mark NonTrimmed, sleep 1s,
advance, hit next wedge, mark, sleep 1s — exactly the rapid-retry
cadence the firmware is sensitive to. Now a wedge-family sense triggers
WEDGE_FAMILY_COOLDOWN_SECS=30 cooldown (matches read_error.rs's
ZONE_ENTRY_COOLDOWN_SECS), and WEDGE_ABORT_THRESHOLD=16 consecutive
wedges aborts the pass for autorip eject+reload. Any non-wedge read
clears the counter.

Also drops the duplicate NonTrimmed dispatch (Mapfile::record is
idempotent so it wasn't a correctness bug, but it doubled per-failure
consumer work).
2026-05-13 11:53:57 -07:00
MattJackson f1926c38dc v0.20.1: delete SectorReader, extract Disc::patch, doc/stub cleanup
WO-2 (delete SectorReader trait):
- The 0.18 trait split into SectorSource (read-only) and SectorSink
  (write-only) is final; the legacy SectorReader alias was a bridge.
- Renames every internal &mut dyn SectorReader (~25 sites) to
  &mut dyn SectorSource. The trait method capacity() becomes
  capacity_sectors() with a default of 0 (preserves SectorReader's
  default-0 behavior).
- Deletes the SectorReader trait, its blanket-to-Source bridge, and
  the FileSectorReader type alias. Adds explicit forwarding impls
  for Box<dyn SectorSource> and &mut dyn SectorSource so generic
  decorators like DecryptingSectorSource<S: SectorSource> compose.

WO-3a (extract Disc::patch):
- Moves Disc::patch (1230 lines) and bytes_bad_in_title from
  disc/mod.rs into disc/patch.rs as a split inherent impl. Zero
  behavior change — pure mechanical relocation. disc/mod.rs drops
  from 3,945 to 2,714 LOC.

WO-6 (partial):
- Deletes src/labels/png_filenames.rs — was a 72-LOC stub with
  detect() returning false, never wired into the PARSERS registry.

project docs doc drift fixes (audited 2026-05-13):
- JUMP_BASE_SECTORS: 256→1024 (64 MB base for UHD, not 8 MB)
- PASSN_DAMAGE_THRESHOLD_PCT: 12→6
- PASSN_SKIP_SECTORS_BASE: 64→32
- MAX_RANGE_SECS=180: replaced by proportional range_sectors × 25,
  capped at RANGE_BUDGET_CAP_SECS=1800.
2026-05-13 11:36:55 -07:00
MattJackson 4709a73c80 v0.20.0: delete FrameSource/FrameSink, keep single Stream trait
The 0.18 trait split into FrameSource (read-only) and FrameSink
(write-only) was an over-engineered API. Consumers don't think
"frame source backed by MKV" — they think "open MKV for reading".
The split paid a real API-complexity cost (two trait names, two
re-exports, dual impls per bidirectional type, deprecation bridge)
for one marginal property: compile-time direction-safety at the
trait-object boundary. The runtime error path on a wrong-direction
call (StreamReadOnly / StreamWriteOnly) is unambiguous and rare in
practice.

Deletions:
- pes::Stream is no longer #[deprecated]
- pes::FrameSource trait + its blanket-from-Stream bridge
- pes::FrameSink trait + the trampoline impls on every concrete type
- The compile-time-direction-safety test scaffolding
- Crate-root FrameSource / FrameSink re-exports

Additions:
- Stream is now Send-bounded (Stream: Send supertrait). Every
  concrete impl was already Send-compliant — Box<dyn Read + Send>
  and Box<dyn Write + Send> were already in place on the trait
  objects MkvStream / M2tsStream / etc hold internally. Promoting
  Send into the trait makes Box<dyn Stream> Send too, which lets
  autorip drop its SendStream unsafe newtype.

The public API is now: one Stream trait, one concrete type per
format, two constructors (open/create or input/output). Bidirectional
types route through internal Mode { Read | Write } discriminants.

Net: -347 lines libfreemkv, -38 lines autorip, -5 lines freemkv.
2026-05-13 08:42:14 -07:00
MattJackson 55bd1ee868 v0.19.1: repair Cargo.toml after botched 0.19.0 bump
v0.19.0 was tagged with a search-and-replace gone wrong:
rust-version, serde, and zip all had their version strings
replaced with "0.19.0". Edition 2024 rejected rust-version
0.19.0 (< 1.85), failing every CI build. No artifacts shipped
to crates.io.

Repair:
- rust-version: 0.19.0 → 1.86 (CI pin)
- serde: 0.19.0 → 1
- zip: 0.19.0 → 2

Also drops an unused start_lba binding in mux/disc.rs that
clippy 1.86 catches.
2026-05-12 21:29:07 -07:00
MattJackson 5285e8b6ec v0.19.0: bump version 2026-05-12 19:24:06 -07:00
MattJackson c5af61e46b v0.18.25: bump version 2026-05-12 16:27:28 -07:00
MattJackson ed73c1ab68 v0.18.25: add mux debug logging for reader type and stall detection 2026-05-12 16:27:16 -07:00
MattJackson 5ce2df0888 v0.18.24: bump version 2026-05-12 15:46:47 -07:00
MattJackson ea591a36a5 v0.18.23: bump version 2026-05-12 10:59:45 -07:00
MattJackson 368e10486e v0.18.22: bump version 2026-05-12 09:33:00 -07:00
MattJackson a99c4f8487 v0.18.21: format code 2026-05-12 09:18:52 -07:00
MattJackson 4df470e572 v0.18.21: fix borrow checker, make io public 2026-05-12 09:01:52 -07:00
MattJackson a0b13941fe v0.18.21: fix borrow checker error in debug logging 2026-05-12 08:49:33 -07:00
MattJackson 275c9da009 v0.18.21: verbose debug logging for pipeline stalls (fixed) 2026-05-12 08:37:42 -07:00
MattJackson da183722fa v0.18.20: bump version 2026-05-11 22:33:15 -07:00
MattJackson 4a5424eabe v0.18.20: bump version 2026-05-11 22:28:57 -07:00
MattJackson 293d89c586 v0.18.20: separate read/write pipeline depths 2026-05-11 22:18:56 -07:00
MattJackson 4840f5134d libfreemkv: increase DEFAULT_PIPELINE_DEPTH to 32 for smoother mux speed 2026-05-11 22:01:27 -07:00
MattJackson 11c8211605 v0.18.18: bump version 2026-05-11 20:16:30 -07:00
MattJackson 1038beaf06 v0.18.17: bump version 2026-05-11 19:58:36 -07:00
MattJackson f1df57196c libfreemkv: add Clone derive to MuxAtomics 2026-05-11 19:54:50 -07:00
MattJackson 8ba9dc1c0b v0.18.16: bump version 2026-05-11 15:57:03 -07:00
MattJackson 49b131f39a add debug logging for MKB processing 2026-05-11 15:53:11 -07:00
MattJackson 8238ec4ce7 optical vs block batch sizing 2026-05-11 15:35:45 -07:00
MattJackson 4fd5d27df8 v0.18.15: bump version 2026-05-11 11:30:19 -07:00
MattJackson 1fbe272832 disc/scan: surface AACS resolution error on Disc.aacs_error
scan_with() collapsed every failure path from resolve_encryption() into
None via .ok(), so callers couldn't tell the difference between "no
KEYDB found", "KEYDB failed to parse", "disc hash not in KEYDB and
fallback derivation failed", "AACS files unreadable on disc", and a
handshake that rejected every host cert. autorip's UI was stuck
printing "no decryption keys found (check KEYDB)" for all of them,
which is a particularly bad message when the user has actually loaded
a KEYDB and the real failure is something else.

Changes:
- New pub field Disc.aacs_error: Option<Error>. Populated by scan_with
  whenever encrypted && aacs.is_none(). Sentinel KeydbLoad path
  "<no keydb in search paths>" distinguishes the no-keydb case from
  a real load failure without adding a new Error variant (which would
  be a breaking change for downstream exhaustive matches).
- tracing::warn in scan_with at scan_aacs_resolve_failed and
  scan_aacs_no_keydb, with error_code and keydb path for grepping.
- tracing in do_handshake: keydb load failure, host-cert exhaustion
  (with cert count and last error code), VID read failure post-auth,
  and a debug-level success log. Lets us see whether handshake even
  got off the ground for a given disc.

Test fixtures updated to set aacs_error: None.
2026-05-11 11:29:48 -07:00
MattJackson 6bd635725e v0.18.14: bump version 2026-05-11 08:27:10 -07:00
MattJackson 09b77b4dea disc/patch: relabel "possible wedge" heuristic log to avoid confusion
The 'All probes failed — possible wedge condition' log fired during patch
probing whenever 10+ consecutive failures hit AND a probe sweep at the
local zone returned 0 successes. This was distinct from the read_error.rs
'wedge_transition' log that fires when the SCSI sense family ACTUALLY
flips into Hardware/IllegalRequest fast-fail mode.

Two logs both saying 'wedge' caused operator confusion during the
2026-05-11 Dune Pt 2 wedge investigation — was the drive wedged, or was
it just a zone of fully-bad sectors? They mean different things.

Relabel to 'patch_zone_fully_bad' with explicit pointer to read_error.rs
for the canonical wedge detection. Same triggering condition; just clearer
wording in the log stream.
2026-05-10 22:20:44 -07:00
MattJackson 688058b3e8 labels/bdmt: drop description fields that are just XML child elements
Disc-04 (Top Gun: Maverick) re-test 2026-05-11 surfaced a real-world
bdmt_eng.xml where <di:description> contained no prose, only nested
<di:thumbnail href="…"/> elements. The previous parser surfaced
the raw XML fragment as the description string ("<di:thumbnail
href=\"tgm_meta_sm.jpg\" />\\r\\n      <di:thumbnail
href=\"tgm_meta_lg.jpg\" />"). Worse than no description.

Fix: filter description candidates that begin with `<` after
trimming. Real prose never starts with an angle bracket; XML-only
content always does. Net: title extraction unaffected (it uses its
own element-priority path); description field drops when it would
otherwise carry XML noise.

Two new bdmt tests, 12 of 12 passing.
2026-05-10 22:12:59 -07:00
MattJackson c96bac7977 labels: append CLPI orphan streams after gap-fill
Three layered sources of stream labels now, in precedence order:
1. **Framework parser** (paramount/criterion/pixelogic/ctrm/dbp/deluxe)
   — editorial labels with purpose/qualifier ("English Atmos",
   "Director's Commentary", "English SDH"). High or Medium confidence.
2. **MPLS gap-fill** (`fill_gaps_from_mpls`) — every stream the
   playlist references gets at least a basic lang+codec label, even
   when the framework parser missed it.
3. **CLPI orphan append** (`append_clpi_orphans`) — streams in
   /BDMV/CLIPINF/*.clpi ProgramInfo that NO MPLS playlist references.
   Empirical (2026-05-11): ~5% of streams across the 11-disc corpus,
   most dramatic on disc-02 (HDMV-only) at 40% CLPI-only.

Orphan numbering: each appended orphan gets
`stream_number = max(existing per type) + N` so playlist-reachable
streams keep their original positions and orphans sort cleanly at
the tail.

Orphan dedup: (stream_type, language, codec_hint) tuple — fuzzier
than PID matching (PIDs aren't carried on StreamLabel) but it's the
only signal available downstream of the gap-fill. False positives
(genuine orphan that happens to share lang+codec with an existing
entry) silently drop, which is the conservative failure mode — the
user-facing display would just see a confusing duplicate otherwise.

`mpls_universal::language_display_name` and `::codec_name` promoted
from private fn to pub(crate) so this module can build orphan labels
with consistent naming.

Tests: 2 new in gap_fill_tests — synthetic-input verification of the
dedup tuple logic and the stream_number assignment. 6/6 tests in the
gap-fill module now passing.
2026-05-10 22:08:14 -07:00
MattJackson c32acff3e4 disc/read_error: rustfmt fixup for wedge-prevention commit 2026-05-10 22:00:08 -07:00
MattJackson b58e2d9873 disc/read_error: wedge PREVENTION — jump on first error + 30s cooldown
Rewrites the Pass 1 wedge handling from "slow skip after the drive
has already wedged" to "prevent the wedge transition in the first
place." Driven by 2026-05-11 empirical data: the BU40N transitioned
into IllegalRequest fast-fail mode at exactly 7 medium errors in
6.5 seconds (~1 read/sec retry cadence). Once there, only physical
eject + reload clears it — 30s pauses + 1 GB jumps do not.

The fix is the user's mental model from that session:

  "We can detect bad reads, failed reads, and asking to read again
   fast after causes a wedge. We need to prevent the wedge in the
   first place."

Two changes to the centralized error handler:

1. **`for_sweep().fast_jump_threshold = 1`** (was 4). Pass 1 now
   JumpAheads on the FIRST outer-batch failure, not the 4th. The
   drive never gets back-to-back retries at the same LBA in Pass 1
   — every error → jump 64 MB forward + long cooldown. Pass N keeps
   `fast_jump_threshold = u64::MAX` because retries on already-known-
   bad LBAs are its whole job.

2. **`ZONE_ENTRY_COOLDOWN_SECS = 30`**. The FIRST error after a
   clean run (when `consecutive_outer_failures == 1` and we're not
   bisecting) uses this long pause instead of the standard 5 s
   FAIL_PAUSE_SECS. Gives the BU40N's firmware / bridge internal
   retry counters 30 s of breathing room before the next read,
   preventing the "7 errors in 6.5 s" cascade. Subsequent errors
   in the same zone use the standard 5 s pause (we've already
   jumped past the initial damage; further errors mean we landed
   in another bad cluster).

Pass N exempt from the zone-entry cooldown — `bisect_on_marginal=
true` skips the long-pause arm. Pass N's per-sector retries on
known-bad LBAs would multiply uselessly with 30 s/error.

Test updates: 4 tests' expected behavior changed under the new
policy. Renamed `pass_1_marginal_skips_instead_of_bisecting` →
`pass_1_marginal_jumps_immediately_not_bisecting`. Renamed
`pass_1_jumps_after_4_consecutive_outer_failures` →
`pass_1_jumps_immediately_on_first_outer_failure`. Updated
`both_passes_pause_on_failed_read_for_wedge_avoidance` (now
`pass_1_zone_entry_uses_long_cooldown` + `pass_n_pauses_uniformly_on_failed_read`).

Cost analysis:
- Clean disc (no errors): unchanged. 0% overhead.
- Lightly damaged (1-2 zones): +30 s per zone = ~1 min total. Fine.
- Heavily damaged (10+ zones): +5+ min total. The trade for never
  wedging the drive and getting a usable Pass N afterwards.

Expected behavior on the next damaged-disc rip:
- Pass 1 hits damage at LBA X → jumps 64 MB forward immediately,
  pauses 30 s
- Drive's firmware never accumulates the retry pressure that triggers
  IllegalRequest fast-fail
- bytes_maybe accumulates faster (we skip more), but Pass N picks up
  the slack with proper per-sector recovery — and Pass N can actually
  RUN because the drive isn't wedged
2026-05-10 21:56:27 -07:00
MattJackson a9e802c1c2 clpi+labels: extract program_info stream table + CLPI vs MPLS audit
Two layered changes, in service of the empirical question "is CLPI
truly redundant with MPLS for label data?":

1. **clpi.rs ProgramInfo parser**. The existing CLPI parser only
   walked the EP map (for sector-range lookups). Added a parser for
   the ProgramInfo section's per-stream stream_coding_info table:
   pid, coding_type, audio_format/rate, video_format/rate, ISO 639-2
   language. Spec layout per libbluray clpi_parse.c. Best-effort —
   malformed program_info leaves `streams: vec![]`, EP map keeps
   working. `ClipInfo` gains a `streams: Vec<ClpiStream>` field.

2. **labels/clpi_audit.rs**. Diagnostic that walks both
   `/BDMV/CLIPINF/*.clpi` (via the new program_info parser) and
   `/BDMV/PLAYLIST/*.mpls`, builds a (PID → fields) merged view, and
   classifies each row:
   - `Match`: both sources agree (same coding_type + language)
   - `ClpiOnly`: PID in CLPI but no MPLS playlist references it
     (orphan stream on disc — reachable via low-level access, not via menu)
   - `MplsOnly`: PID in MPLS but no CLPI lists it (would indicate a
     parser bug; verified empirically that this NEVER happens)
   - `Divergent`: same PID, different coding_type or language between
     sources (playlist re-tagged or attribute encoding mismatch)
   Surfaced via `labels-analyze` as `clpi_vs_mpls_audit: {matches,
   clpi_only, mpls_only, divergent, total_pids}`. Doesn't affect the
   label output — pure diagnostic.

Empirical findings on the 11-disc corpus (excl. disc-04 truncated):
- 226 matches / 0 mpls_only / 8 clpi_only / 5 divergent across 239 PIDs
- 6 of 10 non-truncated discs have CLPI-only streams (orphans)
- disc-02 (HDMV-only) is the most dramatic: 40% of its 5 streams are
  CLPI-only — MPLS sees 3, CLPI sees 5
- Conclusion: CLPI is NOT truly redundant. ~5% of streams disc-wide
  are CLPI-exclusive. Future work: layer CLPI as a tertiary source
  below MPLS in the labels pipeline (orphan streams marked with even
  lower confidence than MPLS).
2026-05-10 21:50:39 -07:00
MattJackson a876ce846b labels: surface MPLS chapter summary in LabelAnalysis
LabelAnalysis gains `chapter_summary: Vec<ChapterSummary>` — one row
per .mpls file in /BDMV/PLAYLIST/, with chapter count (PlaylistMark
entries with mark_type ≤ 1) and approximate playlist duration in
seconds. Sorted by playlist filename.

Sourced from the existing crate::mpls parser (no new format work).
Useful for identifying the main feature playlist at a glance — it's
the one with the longest duration. Verified on disc-11 (Dune Pt 2):
00800.mpls correctly identified as 2h 45m 49s with 18 chapters
amid 30+ shorter playlists.

Doesn't touch the per-title `disc::DiscTitle::chapters` field which
disc::bluray.rs already populates from the same marks during disc
init — this is purely the diagnostic surface for labels-analyze.
2026-05-10 21:32:54 -07:00
MattJackson 058fd8396f labels: gap-fill MPLS streams when framework parser under-yields
When a framework parser (paramount, criterion, pixelogic, ctrm, dbp,
deluxe) is chosen but its label list covers only a subset of the
stream slots MPLS knows about, merge MPLS-derived entries for the
uncovered (stream_type, stream_number) slots. Framework labels keep
their richer fields (purpose=Commentary, codec_hint with "Atmos",
qualifier=Sdh); MPLS only fills slots the framework left unnamed.

Implementation:
- `fn fill_gaps_from_mpls` walks the MPLS label list, pushing any
  entry whose (type, number) tuple isn't already in the framework
  output. Stable sort by (type, number) groups audios before
  subtitles in the merged result.
- Called from both `extract()` and `analyze()`. Skipped when the
  chosen parser is itself `mpls_universal` (no gaps possible).
- `LabelAnalysis::gap_fill_added` field reports how many slots got
  filled — useful diagnostic from `labels-analyze`.
- `StreamLabelType` gains `Eq + Hash` so the dedup HashSet works.

Tested via 4 new unit tests (155 of 155 labels tests passing, was
151). End-to-end on partial-yield corpus discs:
- disc-05 (Oppenheimer): pixelogic 4/5 already covered, gap_fill_added=0
- disc-11 (Dune Pt 2):   pixelogic 8/11 already covered, gap_fill_added=0

(Real-world gap-fill activations are rare in the current corpus because
pixelogic already incorporates MPLS-equivalent data when matching;
the merge is defensive for less-thorough frameworks.)
2026-05-10 21:28:01 -07:00
MattJackson 5ee28c08b9 labels/mpls_universal: dense stream numbering across playlists
Per-playlist stream_number counters were resetting between MPLS
files, so a disc with 2 MPLS files each listing the same 8 audio
streams produced labels with stream_number 1..8 then 1..8 again
(dedup kept whichever PID was different, leaving the numbering
visibly broken — multiple "audio1: eng" rows).

Move the counters outside the per-file loop and increment only
when an entry survives dedup. Surviving entries now get dense
1..N numbering across the whole disc per stream_type.

Verified on corpus disc-02 (HDMV-only): was `audio1, audio1, audio1`
for the 3 distinct audio codecs (TrueHD/AC-3/DTS-HD MA), now
`audio1, audio2, audio3`. Same fix applies to disc-01 (12 audio
streams across multiple MPLS) and disc-09 (14 audio streams).
2026-05-10 20:58:33 -07:00
MattJackson 764230b1eb labels: universal MPLS fallback + bdmt disc metadata + png stub
Three new modules in the labels platform, all layered so framework-
specific parsers (paramount, criterion, pixelogic, ctrm, dbp, deluxe)
always take precedence over the universal layer.

**mpls_universal.rs** (~600 LOC, 9 tests): consumes the already-parsed
`crate::mpls::Playlist::streams` and emits StreamLabel entries with
language + codec_hint per stream. Returns `Confidence::Low` (new
variant) so framework parsers' Medium/High always win the registry's
max-by-confidence tiebreaker; MPLS only gets picked when no framework
matched. Closes the "no BD-J disc" case (HDMV-only navigation) that
previously produced zero labels — language and base codec are
spec-mandated in MPLS STN tables on every Blu-ray ever made.

**bdmt.rs** (~350 LOC, 10 tests): reads `/BDMV/META/DL/bdmt_<lang>.xml`
files into a new `DiscMetadata` struct (localized title names per
ISO 639-2 code, descriptions, optional box-set position). Runs
independently of the parser registry — disc-level metadata, not
per-stream, so the registry's confidence selection doesn't apply.
Surfaced on a new `LabelAnalysis::disc_metadata` field.

**png_filenames.rs** (noop stub): pattern documentation + dead-code
detect/parse for future reactivation. Deferred because MPLS already
delivers per-stream lang/codec/type on every disc; PNG filename
language tokens only add studio variant disambiguation (FRC vs FRP,
LAS vs CSP) — niche enough to not justify the implementation cost
right now.

Wiring changes in `mod.rs`:
- New `Confidence::Low` variant (PartialOrd places it below Medium/High)
- New `ParseResult::low()` constructor
- `mpls_universal` appended last to `PARSERS` registry
- `LabelAnalysis::disc_metadata: Option<DiscMetadata>` field
- `analyze()` runs `bdmt::parse` independently and surfaces result
- `pub use bdmt::DiscMetadata` re-export so the labels-analyze tool
  in freemkv-tools can construct the JSON payload

Total: 151 of 151 labels tests passing (was 132 — added 19 new).
2026-05-10 20:53:56 -07:00
MattJackson 7fb2e07aed v0.18.13: bump version 2026-05-10 19:57:44 -07:00
MattJackson 376aadb335 disc/patch: adaptive batching — 32 sectors, drop to 1 on failure
Pass N now reads at 32 sectors per attempt and drops to 1 only on
batch-read failure to probe each sector individually. After 16
consecutive clean single-sector reads it climbs back to 32. Net
effect: NonTrimmed regions walk ~32x faster in clean stretches
without sacrificing per-sector recovery quality — the drop-to-1
retry from the same cursor position guarantees every sector in a
failed batch is individually attempted.

Design contract:
- A batch-read failure (count > 1) is NOT a recorded failure: no
  NonTrimmed mark, no consecutive_failures bump, no damage_window
  push, cursor stays put. We just drop current_batch to 1 and the
  loop re-attempts the same position at single-sector granularity.
- A single-sector failure (count == 1) follows the existing path:
  NonTrimmed mark, consecutive_failures++, damage_window.push(false),
  post-failure pause, wedge probes.
- Backtrack always at count=1: this path fills a gap that the main
  loop's damage-window skip jumped over. Using batched reads there
  would lump good sectors into NonTrimmed marks when the gap
  contains even one bad sector.

State machine adds:
- `initial_batch` (from opts.block_sectors, default 32 in patch_internal)
- `current_batch` (mutable, starts at initial_batch, drops to 1 on
  batch failure)
- `consecutive_singles_ok` (counter, resets on upscale + failure)
- `ADAPTIVE_UPSCALE_THRESHOLD = 16` (matches sweep's pattern for
  "16 consecutive good = back to fast mode")

Tests:
- pass_n_size_aware_skip.rs PatternedSectorReader now fills each
  sector with its OWN LBA byte (not the starting LBA's byte). This
  matches real drive behavior — the pre-0.18.13 fixture's
  "fill whole batch with one byte" was a shortcut that only worked
  when patch read 1 sector at a time. Existing recovery-quality
  assertions all still pass under adaptive batching.

User spec: "try 32, pass, great, fail -> do 1 sector"
2026-05-10 19:57:27 -07:00
MattJackson 586495b2ee v0.18.12: bump version 2026-05-10 19:44:57 -07:00
MattJackson 72f2224efe disc/patch: leave failed reads NonTrimmed mid-multipass, not Unreadable
User design call after watching Pass 2 mark ~20 KB as "Cosmetic"
(permanently Unreadable) after just 10 retries within a single pass:
"i think it's good or maybe until all passes are done. then it's
gone."

That contradicts what the multi-pass design promises a user. The
project goal in project docs is "recover 100% of readable data from any
optical disc, automatically." Marking sectors Unreadable after a
SINGLE pass's per-range retry budget gives up on sectors that
subsequent passes might recover — drive reads are stochastic, the
sector that fails 10 times in Pass 2 may succeed on attempt 1 in
Pass 3 after temperature / bus state / prior-read patterns shift.
The patch.rs doc comment already noted ~36% of patch-marked
Unreadable sectors turned out to be readable in re-rip experiments.

Three sites in `Disc::patch` were emitting `PatchItem::Unreadable`
mid-pass:
  - backtrack hit damage (line ~2659)
  - all-retries-exhausted on a single LBA (line ~2846)
  - redundant second mark after the wedge-suspicion log (line ~2970)

All three now emit `PatchItem::NonTrimmed` instead. Failed bytes
stay "maybe" (NonTrimmed) so the next pass gets another shot. The
per-range skip-limit (10) and per-pass wedge-threshold (50) still
bound time-per-pass; they just no longer turn the bytes terminal.

The `PatchItem::Unreadable` variant stays in the enum (with
#[allow(dead_code)]) because the orchestrator-side end-of-recovery
promotion will use it: autorip, after the final retry pass
completes, scans the mapfile and promotes still-NonTrimmed →
Unreadable. That promotion lands in a follow-up commit on the
autorip side — separable from this libfreemkv change.

Loss accounting unchanged: `bytes_pending + bytes_unreadable` is
the "lost or pending" total that `abort_on_lost_secs` consults
(disc/mod.rs:1327). Moving bytes from one bucket to the other
mid-pass doesn't affect whether the rip would abort; it only
affects display (UI shows "Maybe" vs "Cosmetic") and whether
subsequent passes retry the bytes (the actual fix).

Test update: `test_pass_progress_separates_unreadable_from_pending`
was renamed to `test_pass2_leaves_failed_reads_as_pending_not_unreadable`
and rewritten to assert the new invariant — Pass 2 leaves all
failed bytes as bytes_pending (no mid-pass Unreadable promotion).
Original assertions were checking the pre-design-call behavior.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 18:47:05 -07:00
MattJackson 0bca7a11bd disc/read_error: unify Pass 1 and Pass N error handling
User's design call after watching the avoidance work prevent a wedge
on the live rip (no wedge events across 6 read errors): "Pass N
and 1 should both be very very similar in recovery. almost identical
just smaller sectors imo in pass n. pause times the same imo as a
failed read is a failed read."

The error-handling code path was already centralized (one
handle_read_error fn, called by both Disc::sweep and Disc::patch).
The TUNING was split — Pass 1 used 5 s inter-error pauses + a
wedge-skip-and-continue policy; Pass N used 1 s pauses + immediate
AbortPass on HARDWARE_ERROR / ILLEGAL_REQUEST. That asymmetry made
Pass N vulnerable to the same wedge that Pass 1's avoidance fixed.

Changes:

1. FAIL_PAUSE_SECS = 5 — single constant, applied uniformly to both
   passes. Dropped PASS_1_FAIL_PAUSE_SECS and POST_FAILURE_PAUSE_SECS
   in favor of one value. CONSECUTIVE_FAIL_LONG_PAUSE_SECS kept as a
   distinct (but currently equal) value for future tuning escalation.

2. HARDWARE_ERROR / ILLEGAL_REQUEST path is now symmetric:
   - Pass 1: JumpAhead WEDGE_JUMP_SECTORS (1 GB) + WEDGE_PAUSE_SECS
     cooldown, mark skipped region NonTrimmed.
   - Pass N: JumpAhead WEDGE_PASS_N_SKIP_SECTORS (64 sectors / 128 KB)
     + WEDGE_PAUSE_SECS cooldown. Pass N's batch=1 means a 1 GB skip
     would abandon the entire current NonTrimmed range; small skip
     moves past the bricked LBA + buffer, outer patch loop picks up
     the next sector.
   - Both share WEDGE_ABORT_THRESHOLD — same 16-skip budget before
     real AbortPass on a permanently stuck drive.

3. wedge_skip / wedge_abort tracing logs now include `pass=1|N`
   so post-mortems can see which pass hit the wedge condition.

Cost analysis:

Pre-reframe worry was "5 s × 5500 NonTrimmed sectors per Pass N
pass × 7 passes = 53 hours." Reality: most NonTrimmed sectors
recover on first or second retry, so most reads are successful and
pay 0 pause. The few that DON'T recover hit the 10-skip budget and
get marked Unreadable — bounded at 10 × 5 s = 50 s per truly-bad
sector. Worst-case Pass N pause overhead on a typical damaged disc
is single-digit minutes, not hours. And it's strictly cheaper than
the alternative (wedge kills the entire multi-pass recovery).

Tests:

- `both_passes_pause_on_failed_read_for_wedge_avoidance` — locks the
  unified pause-tuning policy (was pass_1_pauses_briefly).
- `pass_n_hardware_error_also_skips_not_aborts` — was
  `pass_n_hardware_error_still_aborts`. New behavior verified:
  JumpAhead with WEDGE_PASS_N_SKIP_SECTORS + WEDGE_PAUSE_SECS.
- `pass_n_hardware_error_aborts_after_threshold` — new. Confirms
  Pass N respects the same WEDGE_ABORT_THRESHOLD as Pass 1.
- pass_1_does_not_pause_on_skip is gone (it was the old "Pass 1
  pause=0" assertion, irrelevant after the avoidance work).

Empirical validation: avoidance was already proven on a live rip
tonight — 6 read errors on a damaged disc, sense_family=Medium
throughout, wedge_count=0, Pass 1 continued cleanly past 40%
where it previously died at 48%. This commit extends the same
discipline to Pass N's recovery loop.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 18:04:01 -07:00
MattJackson 72ab714c1e labels/deluxe: Phase D rewrite against ground-truth binding pattern
Replaces the speculative arg-position heuristic with type-presence
detection driven by real disc bytecode. Ground truth captured in
(internal)/research/deluxe-poc/data/ via POC v0.3 binding-
bytecode dumps against disc-01 (Disney) and disc-09 (Warner).

What changed:

1. StackVal::CodingType(String) — new variant. getstatic against
   org/bluray/ti/CodingType (the BD-J spec codec enum) now pushes
   this, carrying the field name (e.g. DOLBY_LOSSLESS_AUDIO). The
   pre-fix code was treating codecs as a Deluxe-internal enum
   subclass walk (Phase B), which is the wrong model — codecs are
   standard BD-J API references.

2. coding_type_to_codec_hint(field) — new function. Maps
   org.bluray.ti.CodingType field names to human-readable codec
   strings (DOLBY_LOSSLESS_AUDIO -> "Dolby TrueHD", DOLBY_AC3_AUDIO
   -> "Dolby Digital", etc.). Unknown field names pass through
   verbatim so future codec values still surface something.

3. find_binding_classes — multi-class variant. Some Deluxe discs
   split per-stream tables across two binding classes (audio +
   subtitle). Returns top-K candidates by getstatic count, filtered
   to >=40% of the top count and capped at 4. Replaces the old
   single-class find_binding_class (which was unused after this
   change).

4. interpret_streams — rewritten. Args identified by TYPE not
   position:
   - First EnumRef{kind:"Language"} -> language
   - First EnumRef{kind:"Purpose"}  -> purpose
   - First CodingType(name)         -> codec_hint
   - First Int(n)                   -> stream index hint (traced
     only; per-type sequential stream_number still wins because BD
     spec stream-numbering is anchored on MPLS)
   - Construction has CodingType -> Audio stream; otherwise Subtitle
   - No Language -> skip (not a stream construction)

   This handles BOTH the Disney 5-arg pattern (I, Lbe, Llp, I,
   LCodingType) and the Warner 4-arg pattern (I, Law, Lgp,
   LCodingType) automatically — same code path because args are
   identified by type rather than constructor-signature shape.

5. parse() now walks all binding-class candidates and unions
   their constructions before calling interpret_streams. Logs each
   candidate at INFO with getstatic_count for diagnosis.

Tests:
- 2 new tests verify the CodingType -> codec_hint mapping for
  known + unknown field names.
- Existing interpret_streams tests updated to use the new
  signature (dropped CodecTable arg).
- Audio-emission test rewritten to use CodingType arg instead of
  the old binding_type substring-match approach.

Confidence is still Medium for now (single-corpus verification);
ready to promote to High once tested against a third Deluxe disc.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 17:32:29 -07:00
MattJackson d7fb1b35ed disc: emit Pass 1 summary INFO log at sweep exit
Wires the existing PassSummary infrastructure (in read_error.rs as
of a832bad) into the sweep loop's exit path. One INFO log line per
Pass 1 completion gives operators an at-a-glance damage profile
without grepping per-error WARN lines:

  INFO pass1_summary  total_reads_ok=384521 total_errors=5
                      zones_entered=1 jumps_taken=2
                      bytes_good=38_725_644_288 bytes_pending=46_GB
                      copy_elapsed_ms=1751650

Particularly useful for post-mortem analysis when combined with
the per-error structured WARN logs (ms_since_last_error /
ms_since_last_success / sense_family / wedge_transition) shipped
in 0.18.10. Single line tells you the pass shape; preceding WARN
lines tell you the per-error detail.

Pass N (Disc::patch) intentionally NOT covered in this commit —
Pass N has its own retry-budget summary semantics that warrant a
separate design pass. Pass 1 sweep is where wedge incidents
originate, so it gets the diagnostic surface first.

Staged for 0.18.11. 0.18.10 already shipped the per-error WARN
layer; this is the finishing companion log.
2026-05-10 17:21:37 -07:00
MattJackson 06be4defd2 v0.18.10: bump version 2026-05-10 17:14:52 -07:00
MattJackson a832bad697 disc: structured timing + transition diagnostics for read errors
Adds the observability we need to debug wedge incidents from logs
alone — without needing to enable verbose TRACE-level SCSI tracing.
Goal stated by user: "when error occurs we can debug and code
correctly."

Pre-fix the WARN log on each read error showed only sense codes
and consecutive_failures. Missing: timing context (was the failed
read fast or slow?), gap to previous events (cumulative vs.
immediate failure?), and family transitions (did the drive just
flip into wedge mode, or has it been there?).

New fields on ReadCtx (no caller signature change):

  last_success_at: Option<Instant>
  last_error_at: Option<Instant>
  last_error_family: Option<SenseFamily>
  total_errors: u64
  total_reads_ok: u64
  zones_entered: u64
  jumps_taken: u64
  in_damage_zone: bool

New SenseFamily enum (NotReady / Medium / Hardware / IllegalRequest
/ Other) with is_wedge_family predicate.

handle_read_error WARN log now carries:
  consecutive_failures
  consecutive_outer_failures
  ms_since_last_error    NEW gap between this and previous error
  ms_since_last_success  NEW gap to last good read
  total_errors           NEW aggregate this pass
  total_reads_ok         NEW
  wedge_count
  sense_family           NEW typed category, easier to filter
  sense_key / asc / ascq (existing)

NEW WARN log "wedge_transition" fires once when the sense family
changes from non-wedge to wedge (Medium to Hardware/IllegalRequest).
That's the moment the drive's firmware flipped into fast-fail
mode. Single timestamped event in the log so post-mortems can
pinpoint the transition without scanning thousands of TRACE lines.

Worked example: if the next wedge incident shows

  read_error  ms_since_last_success=18234  ms_since_last_error=null
  read_error  ms_since_last_success=28000  ms_since_last_error=10000
  read_error  ms_since_last_success=43000  ms_since_last_error=68
                                          (drive returned <100ms = wedge symptom)
  wedge_transition  errors_in_zone=5  ms_since_last_success=43000

we can immediately tell cumulative damage, 5 errors over 43 s,
drive went into fast-fail mode at the 5th. If instead we see

  read_error  ms_since_last_success=200  ms_since_last_error=null  sense_family=Hardware
  wedge_transition  errors_in_zone=1

the wedge was triggered by ONE read at a physically-bricked LBA
(immediate fast-fail, no warm-up).

These two patterns demand different tuning responses (longer
pause vs. larger initial jump), and now we can distinguish them
from a single WARN log line each instead of needing TRACE
verbose for the whole rip.

Plus jumps_taken / zones_entered counters that feed an end-of-pass
INFO summary (PassSummary). Caller invokes pass_summary at sweep
end and logs structured stats: "Pass 1 saw N errors / M ok reads
/ K zones / J jumps". Single-line post-mortem for any rip.

No caller signature change (timing is internal to the handler;
end-of-pass summary is a new method callers opt into). Precommit
green; 433+ tests pass. Staged for the 0.18.10 release once we
have user-validation data on 0.18.9's avoidance tuning.
2026-05-10 17:10:52 -07:00
MattJackson 26e2d847d5 v0.18.9: bump version 2026-05-10 17:02:24 -07:00
MattJackson 4442fa2df6 disc: wedge AVOIDANCE on Pass 1 — inter-error pause + larger jumps
Complements the wedge-skip backstop (fbdb50c) with proactive
avoidance so we don't HIT the wedge in the first place. User's
take after seeing the Dune Pt 2 rip wedge at 48%: 'we shouldn't be
wedging.'

Empirical observations from the 23:09:12-23:09:55 wedge timeline:

  5 read errors over 43 s, ~8 s apart (drive's own ECC recovery
  takes 5-10 s per failure). Not 'hammering' in any usual sense,
  but cumulative firmware-state buildup over 5 in-cluster errors
  was enough to tip the BU40N into wedge mode at the 5th error.

  Damage cluster spanned ~140 MB (LBAs 19.898M-19.965M). Current
  damage-jump base of 256 sectors × batch=32 = 16 MB first jump,
  doubling to 32 MB, 64 MB... Each jump landed BACK INSIDE the
  140 MB cluster, exposing the drive to MORE in-cluster errors.

Two avoidance levers:

1. Inter-error pause on Pass 1 (PASS_1_FAIL_PAUSE_SECS = 5 s):
   pre-fix Pass 1 ran pause_secs=0 on all errors to 'zoom past'
   damage zones. Successful reads still zoom at zero pause — the
   pause applies only to FAILED reads, giving the drive's firmware
   cool-down between cluster exposures. Cost: ~5 s per scattered
   failure (~30-60 s total on a damage cluster); trivial vs.
   crashing the rip at 48%.

2. Larger damage-jump base (JUMP_BASE_SECTORS = 1024, up from
   256): first jump at batch=32 now covers 64 MB instead of 16 MB,
   second jump 128 MB instead of 32 MB. Two jumps clear 192 MB —
   well past most single-cluster damage patterns. Smaller jumps
   were landing inside the cluster and adding to the wedge counter.

Plus a halt-aware sleep helper (sleep_secs_or_halt) so the new
inter-error pause doesn't degrade halt response time. Halt poll
granularity 100 ms — halt fires within ~100 ms regardless of
remaining pause time. Updated three sleep call sites in disc/mod.rs
(SkipBlock pause, JumpAhead post-pause, Retry pause).

The wedge-SKIP backstop (fbdb50c) stays — combined with this
avoidance work, the flow becomes:
  damage cluster encountered →
    pause 5 s, mark NonTrimmed →
    second failure →
    pause 5 s, mark NonTrimmed →
    ...
    threshold hit →
    damage-jump 64 MB (clears 95% of clusters) →
    if jump lands in another cluster: 128 MB next jump →
    only if drive STILL wedges after all this:
      wedge-skip kicks in (1 GB jump + 30 s cooldown × 16 budget)

Tests:
  pass_1_pauses_briefly_on_skip_for_wedge_avoidance — locks the
    new 5 s pause behavior in place (replaces the old pause=0 test).
  integration test threshold bumped from 5 s to 60 s with comment
    explaining the new bound is 'not infinite' rather than
    'milliseconds-fast'.
  All 433+ tests green on cargo +1.86 fmt + clippy + test.

Precommit green.
2026-05-10 16:55:02 -07:00
MattJackson 23dc55661c labels: apply_labels integration tests + class_reader robustness fuzz tests
Closes the final two audit items from this session.

labels::apply_labels: factored out of apply() so the matching logic
is unit-testable without needing a SectorReader / UdfFs. 11 new
tests in apply_tests cover:
  - codec_hint + variant flow through to AudioStream.label
  - purpose set on audio with no label English text
  - name fallback only when purpose=Normal (CLI owns purpose i18n)
  - subtitle SDH qualifier set; forced flag flipped on Forced
  - per-type 1-based indexing (audio #2 maps to 2nd audio stream,
    not 2nd stream overall)
  - labels for nonexistent streams are no-ops
  - empty labels list leaves streams untouched
  - fill_defaults generates audio + video labels; preserves existing

class_reader: robustness smoke tests. ClassFile::parse must NEVER
panic on adversarial input — only return Err. 9 new tests:
  - empty input
  - short magic (0..4 bytes)
  - wrong magic
  - truncated after magic
  - bad CP tag
  - truncated UTF-8 in CP
  - 200 random byte buffers (deterministic xorshift)
  - 100 magic + random tail (most adversarial — magic check passes,
    everything else garbage)
  - instructions iter on random code (200 buffers)
  - instruction_size on every opcode 0..255 with varied tail buffers
  - modified_utf8 on random byte buffers (500)

The xorshift PRNG keeps the tests deterministic (no rand dep) and
reproducible — failures will be the same buffer every time. This is
the lightweight alternative to a cargo-fuzz setup; if/when we adopt
cargo-fuzz, these tests stay as regression cases.

All 451 tests passing on cargo +1.86 fmt + clippy + test.
2026-05-10 16:38:19 -07:00
MattJackson fbdb50c79f disc: Pass 1 wedge-skip instead of abort-on-first-wedge
Pre-fix: when the drive returned HARDWARE_ERROR or ILLEGAL_REQUEST
during Pass 1 sweep, libfreemkv immediately returned ReadAction::
AbortPass. Autorip surfaced this as a fatal error and stopped the
rip at whatever progress percentage Pass 1 had reached — typically
40-50%. On a disc with one physical-damage cluster, the user would
see Pass 1 die at ~48% with the cryptic message 'E6000: <lba>
0x02/0x04/0x3e' and have no rip output to work with.

Root cause analysis: BU40N firmware transitions into a fast-fail
state when it hits cumulative read failures in a small LBA range —
returns HARDWARE_ERROR for every subsequent read near that LBA, even
sectors that aren't physically damaged. Per project docs 'Bad-sector
handling' rule #2, 'Recovery requires eject+reload OR significant
cool-down.' Aborting on first wedge throws away the rest of the
disc; the right response is to SKIP the wedged region (mark as
NonTrimmed for Pass N), pause for drive cooldown, and continue.

Fix: in handle_read_error, the HARDWARE_ERROR / ILLEGAL_REQUEST arm
now branches on bisect_on_marginal:

  Pass 1 (bisect_on_marginal=false): JumpAhead with WEDGE_JUMP_SECTORS
    (1 GB at 2048 bytes/sector) and WEDGE_PAUSE_SECS (30 s cooldown).
    Tracks wedge_count in ReadCtx; resets on any successful read.
    Truly aborts only after WEDGE_ABORT_THRESHOLD (16) consecutive
    wedges with no good read in between — generous enough to clear
    most physical-damage clusters, bounded enough to not loop forever
    on a permanently bricked drive.

  Pass N (bisect_on_marginal=true): unchanged AbortPass. Pass N's
    job is single-sector recovery; if the drive won't talk near a
    specific LBA, skipping doesn't help. Pass N exits and lets the
    outer layer decide retry/eject/surface.

5 unit tests cover the new policy:
  pass_1_hardware_error_jumps_ahead_not_aborts — JumpAhead emitted
    with correct sectors+pause, wedge_count incremented.
  pass_1_hardware_error_aborts_after_threshold — AbortPass kicks in
    on the WEDGE_ABORT_THRESHOLD-th consecutive wedge.
  pass_1_good_read_resets_wedge_count — on_success clears
    wedge_count; subsequent wedge gets fresh skip budget.
  pass_n_hardware_error_still_aborts — Pass N's AbortPass behavior
    intact.
  pass_1_illegal_request_also_routes_to_wedge_skip — both wedge
    sense families get the skip treatment.

Impact: on the Dune Pt 2 disc that consistently wedged at 48%
(physical damage at LBA ~19.9M), Pass 1 will now jump ahead 1 GB
on the wedge, give the drive 30 s cooldown, and continue scanning
the rest of the disc. The damaged region becomes Pass N's job to
revisit. Worst case if the drive stays wedged: 16 GB of NonTrimmed
disc area before honest AbortPass.

Precommit (cargo +1.86 fmt + clippy + test) green; 430 passing.
2026-05-10 16:37:59 -07:00
MattJackson 7cc74f0087 labels/xml: shared tolerant XML helper, paramount + criterion onto it
Replaces two near-duplicate hand-rolled XML scrapers in paramount.rs
and criterion.rs with a single labels::xml module that's robust to:

- Case-insensitive tag / attribute names ('<Playlist>' matches the
  same as '<playlist>'; 'Name=...' matches 'name=...').
- XML namespace prefixes (matches '<ns:tag>' for tag='tag').
- Arbitrary whitespace inside open tags and around '=' separators
  ('<tag  name = "X">' works).
- Both quote styles for attribute values (" and ').
- Self-closing tag forms ('<tag/>' and '<tag />').
- '>' chars inside quoted attribute values (no premature end-of-tag).

Three functions:
  xml::attr(element, name) -> Option<String>
      Extract attribute value from an open-tag fragment.
  xml::text(xml, tag) -> Option<String>
      Trimmed text content of first <tag>...</tag>.
  xml::find_element(xml, tag, from) -> Option<(start, end)>
      Locate next <tag>...</tag> for iteration; handles self-closing.

22 unit tests cover the robustness properties: case-insensitivity,
namespace stripping, whitespace tolerance, quote styles, self-close
forms, no-substring-false-positive (looking for 'lang' must NOT
match 'lang_id' or 'language'), '>' inside quoted attrs, iteration
across repeated elements.

paramount.rs: drops local extract_attr; find_feature_playlist now
walks xml::find_element('playlist', ...) so it works regardless of
case and self-closing style. Pre-refactor: required exactly
'<playlist ' (single space, exact case) and '/>' for self-close.

criterion.rs: drops local extract_tag; parse_stream_infos and
parse_playback_config iterate via xml::find_element. Same case-
sensitivity + namespace gains. The 'COMMENTARY' / 'SDH' / 'DS'
content-value match is now case-insensitive too (previously a disc
authored with 'commentary' would have been miscategorized as Normal).

Pre-refactor known failure modes (none observed yet, but trivial
to trip on a future disc): vendor switches whitespace around '=',
uses single quotes, capitalizes a tag, prefixes a namespace. All
now handled.

Out of scope by design: XML entity decoding (&amp;, &lt;), CDATA
sections, comments, processing instructions. None observed in BD-J
authored label data. If a future disc trips them, the entity
decoder is a localized addition.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:21:51 -07:00
MattJackson 3aa1e528c5 labels/deluxe: full Phase B/C/D buildout — codec walk, binding decode
Completes the Deluxe parser pipeline. Phase A (master enums) was
already shipping; this commit lands Phases B/C/D so the parser now
emits per-stream StreamLabel records on Deluxe-authored discs.

Phase B (decode_codec_enum): walks the codec enum's subclass
references (one .class per codec ordinal) and extracts the codec
name string from each subclass's constant pool. Heuristic: pick the
first Utf8 entry that's uppercase + underscored + >=4 chars, or one
of the known codec roots (ATMOS/DOLBY/DTS/TRUEHD/MLP/AC3/EAC3/PCM)
when no underscored candidate is found. CodecTable maps ordinal ->
codec string; empty string for ordinals where extraction failed
(logged via tracing, not fatal).

Phase C (find_binding_class): identifies the class that builds the
per-stream label table by counting getstatic operations targeting
any of the master enum classes from Phase A. Class with the highest
count >= 4 wins. Threshold is empirical (real binding classes have
50+ matches; floor of 4 admits small discs while rejecting incidental
single-reference classes).

Phase D (decode_binding + BindingDecoder): symbolic stack machine
that walks the binding class's <clinit> bytecode. Handles:
  - constant pushes: iconst_<n>/bipush/sipush/ldc(Integer)
  - new <X>: pushes uninit-object marker
  - dup: stack copy
  - getstatic <Y.Z>: pushes EnumRef when Y is in MasterEnumTable,
    else Unknown
  - invokespecial X.<init>(...)V: pops args per descriptor; when the
    receiver is NewObj(X), emits a Construction { binding_type: X,
    args: [...] }
  - invokevirtual/invokestatic/invokeinterface: pop args per
    descriptor, push return placeholder unless void
  - pop/pop2/aastore/putstatic/putfield: standard stack effects
  - branches/returns: clear stack (conservative resync — binding
    <clinit> is straight-line in practice)
  parse_method_arg_count: JVMS field-descriptor parser, handles
  primitives, references (L...;), arrays ([...).

interpret_streams: converts Constructions to StreamLabels using
the master enum table + CodecTable. Each construction with a
Language ref becomes a stream. Audio when codec_hint resolves via
binding_type substring match against CodecTable; subtitle otherwise.
Purpose ordinal -> LabelPurpose via the verified Deluxe Purpose enum
order (Normal/Commentary/PiP/Trivia/Descriptive/Score/NoForced/
NoForcedDescriptive). Stream index = sequential per type. Language
goes through vocab::lang for ISO code + variant.

deluxe::parse now returns Some(ParseResult::medium(labels)) when
all four phases produce labels. Medium confidence — the bytecode
mechanism is rigorously tested but the signal-to-StreamLabel
mapping (which arg is which, audio vs subtitle classification) is
heuristic until corpus binding-class bytecode confirms the exact
pattern.

Test coverage: 13 new unit tests in deluxe.rs
  parse_method_arg_count: 3 tests (basic types, references, malformed)
  BindingDecoder: 4 tests (simple construction, with int pushes,
    skips unmatched invokespecial, resolves master-enum ordinal)
  interpret_streams: 4 tests (subtitle on no codec, audio on codec
    match, purpose routing, skips no-language)
  MasterEnumTable: 3 tests (resolve, value, class_name_set)
  extract_codec_name: 1 test (uppercase+underscore matching)

class_reader.rs gained a #[cfg(test)] ConstantPool::from_entries
test-only constructor so Phase D tests can build synthetic CP
fixtures without writing raw .class bytes.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:15:39 -07:00
MattJackson 7c6b0f82ab labels: per-parser confidence + highest-confidence-wins registry
Replaces 'first-match-wins by array order' with 'highest-confidence-
wins, array order tiebreaker'. Removes the arbitrariness when more
than one parser can claim a disc (e.g. one with both
bluray_project.bin and playlists.xml).

New types in labels::mod:
  pub enum Confidence { Medium, High }
  pub struct ParseResult { labels: Vec<StreamLabel>, confidence }
  ParseResult::high(labels) / ::medium(labels) constructors

Parser signature change: every parse() now returns
Option<ParseResult> instead of Option<Vec<StreamLabel>>. Updated all
six parsers in lockstep:
  paramount: High (fully structured XML)
  criterion: High (fully structured XML)
  pixelogic: High by default, Medium when an unknown token component
             is encountered (the skip-unknown path now propagates the
             coverage gap to the caller instead of silently degrading)
  ctrm:      High (structured key-value)
  dbp:       High (anchor scan with vocab routing)
  deluxe:    still returns None pending Phase D — signature aligned

Registry behavior:
  extract() iterates all detect-positive parsers, picks highest
  Confidence with non-empty labels. Equal confidence falls to array
  order (deterministic). Same selection logic in analyze().

LabelAnalysis grew a confidence: Option<Confidence> field so the
diagnostic surface (freemkv-tools labels-analyze) exposes which
confidence tier the selected parser claimed. labels-analyze JSON
and labels-corpus-check structural diff both gained the field.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 16:01:20 -07:00
MattJackson 4226a53e73 labels: fresh-eyes audit — capture variant, dedupe detect, lock registry
Three targeted fixes from a second-pass audit of the labels module.

1. vocab::lang now returns Option<LangInfo> with both code AND a
   human-readable variant string. Pre-fix: 'Brazilian Portuguese 5.1'
   became language=por, variant='', dropping the dialect info the
   disc had explicitly authored. Post-fix: language=por,
   variant='Brazilian' — matches the convention pixelogic / ctrm /
   criterion already use for their region variants. dbp now
   populates StreamLabel::variant from this. Compound table grew a
   3-tuple (needle, code, variant); bare matches still return
   variant=''.

2. dbp and deluxe had duplicated detect() boilerplate (any top-level
   .jar in /BDMV/JAR/). Both now call jar::has_any_top_level_jar.
   The trait-level detect contract — see super::PARSERS — can't peek
   inside a jar without a SectorReader, so loose-detect-plus-real-
   check-in-parse is the unavoidable pattern for jar-content parsers.
   Consolidating in jar.rs at least makes the duplication visible.

3. mod.rs comment about parser ordering said 'dbp last'; deluxe is
   actually now last. Updated to explain the dbp-before-deluxe order
   is by cost (cp-iteration cheaper than bytecode walking when Phase
   D lands).

Plus a registry-level lock test in mod.rs::registry_tests — asserts
the PARSERS array order is exactly [paramount, criterion, pixelogic,
ctrm, dbp, deluxe]. This was previously implicit; if someone reorders
the array (which changes which parser wins on overlapping signals),
unit tests would have stayed green. Now they fail with an explanatory
message about why the order matters.

Audit findings deferred to follow-ups (each its own commit + design
discussion):
- Stronger detect contract — current loose-detect-real-check pattern
  is forced by SectorReader-not-in-detect-signature; could be fixed
  by changing the trait to take an Option<&mut dyn SectorReader> or
  similar.
- Per-parser confidence scoring — registry currently first-match-wins.
  A high-confidence parser ought to beat a low-confidence one
  regardless of array order.
- class_reader fuzzing — handles malformed input via Result but no
  adversarial corpus yet.

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 15:50:32 -07:00
MattJackson 92f34a289e labels: refactor pixelogic + ctrm onto shared platform + hardening
Closes the platform unification: every label parser now routes
purpose/qualifier/codec classification through one source of truth
(vocab.rs) instead of N hand-rolls, and every binary-blob byte
scanner goes through one helper (text::extract_ascii_strings).

pixelogic.rs:
- Drop local extract_strings (~20 lines) — use text::extract_ascii_strings.
- HARDENING: replace  with skip-unknown-component +
  trace log. Pre-refactor behavior: any single uncatalogued token part
  (e.g. a future codec ID, new framework variant) silently dropped the
  entire stream record. New behavior: skip just the unknown part,
  surface what we know about the stream.
- 8 new unit tests cover basic audio/subtitle paths, commentary,
  descriptive, region variant, the new skip-unknown-component
  regression, and the non-audio/non-subtitle early-out.

ctrm.rs:
- Replace  with
  vocab::purpose(&name). Now word-boundary matched — 'Commenter Pro
  Track' no longer false-matches Commentary.
- Replace  with vocab::qualifier(&name).
  Same word-boundary tightening, plus picks up Forced and
  DescriptiveService for free.
- Preserved structural commentary signal via
  as a fallback when name is silent (e.g. 'audio_commentary_1.name=Track 2').
- 6 new unit tests including the 'Commenter' false-positive regression
  and the SDH-only-on-subtitles boundary.

text.rs:
- Drop module-level dead_code allow now that pixelogic uses
  extract_ascii_strings.

Net: all 5 framework parsers now on the unified platform. Future work
(deluxe Phase D, paramount/criterion XML hardening) builds on the
same scaffolding.

Precommit green.
2026-05-10 15:33:22 -07:00
MattJackson fdbe469d50 labels: add deluxe parser (Phase A — master enum identification)
Closes the 'detected but no parser' gap on Deluxe-authored BD-J discs
(com/bydeluxe/ package signature; ~20% of UHD discs in our corpus per
the 2026-05-10 11-disc capture session).

What ships in this commit:

- detect() — registers the parser in the chain (loose pre-check at the
  /BDMV/JAR/ level; real signal in parse via has_path_prefix on
  'com/bydeluxe/').
- Phase A master enum identification — walks every .class's <clinit>
  ldc sequence and matches against framework-stable fingerprints for:
    Language    (70 ldcs starting English/French/Spanish/Dutch)
    Purpose     (8  ldcs starting Normal/Commentary/PiP/Trivia)
    VideoFormat (7  ldcs starting HD/HDR10 Plus/HD Dolby)
    Region      (22 ldcs starting USA_D1/LIC1/LIC2/LIC3 — Disney-only)
    Studio      (6  ldcs starting Disney/Marvel/Pixar — Disney-only)
  All identifications verified out-of-tree on corpus disc-01 (Disney,
  The Amateur) and disc-09 (Warner, Dune Part 1) via the standalone
  POC.
- Phase B structural skeleton (find_codec_enum) — identifies the
  codec enum class by structural signature (>=20 'new' ops, 0 ldcs),
  returns the ordered subclass list. Codec string extraction from
  subclasses is dead-coded pending the follow-up commit.

What does NOT ship yet:

- Phase D (per-stream binding-class decoder). parse() returns None
  intentionally — the master enums alone don't yield StreamLabels
  without the streamTable.put(...) bytecode walker. analyze() will
  show 'deluxe' in parsers_detected with the enum identification in
  tracing logs, so the analyzer reports honestly: 'detected, can't
  emit labels yet' rather than silent failure.

Why ship A without D: A is proven on real corpus discs; D's design
needs ground-truth binding bytecode from at least 2 corpus discs
side-by-side to verify the stack-machine pattern. The Phase A
infrastructure (master enum identification + ordinal->name table)
is what D will consume — landing it now unblocks D's design without
holding back the parser registration.

5 unit tests cover the fingerprint matcher logic + a roster lock that
forces explicit consideration when adding/removing fingerprints.

Precommit green.
2026-05-10 15:22:11 -07:00
MattJackson 4ec75a03f2 labels: shared platform (vocab/text/jar) + dbp refactor
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.

New modules:

  vocab.rs       expanded from 27 -> ~370 lines
                 + lang(text) -> Option<&'static str>      (English/multi-word
                                                            -> ISO 639-2; ~45
                                                            languages, compound
                                                            phrases like
                                                            'Brazilian Portuguese'
                                                            and 'Castilian Spanish')
                 + purpose(text) -> LabelPurpose            (Commentary,
                                                            Descriptive, Score,
                                                            Ime; word-boundary
                                                            matched)
                 + qualifier(text) -> LabelQualifier        (SDH, Forced,
                                                            DescriptiveService)
                 + has_word internal primitive — enforces word-boundary
                   matching so 'Commenter' no longer matches 'commentary' and
                   'engineering' no longer matches 'english'. Existing parsers
                   used .contains() and got lucky on the corpus; vocab now
                   guarantees the boundary in one place. 20+ unit tests.

  text.rs        NEW (~85 lines)
                 + extract_ascii_strings(data, min_len) — promoted from two
                   near-duplicate copies (pixelogic min=4, dbp min=5);
                   threshold passed in. 7 unit tests including
                   trailing-without-terminator + high-bit-byte handling.

  jar.rs         NEW (~120 lines)
                 + for_each_jar(reader, udf, fn)  — walk every top-level
                                                    .jar under /BDMV/JAR/,
                                                    yield to callback.
                 + has_path_prefix(archive, prefix) — cheap 'is this MY
                                                      framework's jar?' check
                                                      via central-dir filenames.
                 + for_each_class(archive, fn)    — parse every .class entry
                                                    through class_reader,
                                                    yield (name, &ClassFile).
                 + try_each_class(archive, fn)    — same with early-return on
                                                    first Some(R) match.

Refactored:

  dbp.rs         v2 on the new platform:
                 - dropped extract_printable raw byte scan
                 - dropped its own English -> ISO 639-2 map
                 - dropped its own parse_attributes hand-roll
                 + iterates CpInfo::Utf8 via class_reader (structurally clean,
                   no false-positive risk from method bytecode bytes)
                 + routes language/purpose/qualifier through vocab
                 All 7 prior dbp tests still pass; +2 new ones cover
                 vocab routing.

dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.

Precommit green (cargo +1.86 fmt + clippy + test).
2026-05-10 15:16:25 -07:00
MattJackson 99236ffd55 labels: add class_reader, hand-rolled JVM .class file parser
Foundation for label parsers that need structured access to .class
files inside /BDMV/JAR/<x>.jar. Replaces noak (~3KLOC dep) with a
~1000-line std-only reader.

Public API:
- ClassFile::parse(&[u8]) -> Result<ClassFile>
- ConstantPool::{get, utf8, class_name, string, integer, member_ref, iter}
- Member::code(&pool) -> Option<CodeAttribute>
- CodeAttribute::instructions() -> Instructions iterator
- Instruction::{name, operand_u8, operand_u16, cp_index}
- Opcode constants (LDC, AASTORE, NEW, GETSTATIC, INVOKESPECIAL, ...)

Spec coverage:
- Constant pool: all 17 tag types incl. Long/Double 2-slot quirk
- Modified UTF-8 incl. 0xC0 0x80 -> U+0000 special case
- Bytecode iteration with full opcode size table
- Variable-length tableswitch / lookupswitch / wide

12 unit tests cover the opcode table edge cases (padded switch tables,
wide-iinc 6-byte form), modified-UTF-8 decoder, and iterator
stop-on-truncated behaviour.

Module is currently #![allow(dead_code)] — the public API is staged
for labels::deluxe (Phases A-E bytecode walker) and a labels::dbp
refactor onto the constant-pool iterator. Tests exercise the API
in isolation. The allow comes off as those callers land.

Also fixes two pre-existing clippy lints that 1.86's stricter checks
flagged after I touched the labels module:
- src/mux/disc.rs: while-let-loop in test fixture
- tests/pass_n_size_aware_skip.rs: type_complexity in helper signature

Precommit (cargo +1.86 fmt + clippy + test) green.
2026-05-10 15:06:34 -07:00
MattJackson 9e3327a569 v0.18.8: bump version (Pass-1 fast-skip never reached autorip 0.18.7 — Cargo.lock mis-resolved) 2026-05-10 14:01:23 -07:00
MattJackson 0081955686 labels: add dbp parser (Magnolia Pictures BD-J framework)
5th BD-J authoring framework recognized. Discriminator: any top-
level .jar in /BDMV/JAR/ that contains com/dbp/ package paths.
Identified during the 2026-05-10 corpus session via string-mining
disc-07's BD-J jar — perm files reference bd-live.magpictures.com
(Magnolia / Magnet Releasing).

Stream labels live as plain ASCII strings inside compiled .class
files in the form

    LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,...
    HTextField,Subtitle1,English SDH,Fontstrip_Composite,...
    ATextField,Subtitle0,None,Fontstrip_Composite,...

— a quirk of the menu-rendering layer encoding TextField positions
and content as constant strings the Java compiler retained in the
class string pool. The leading single-letter prefix is string-pool
ordering noise; parser anchors on `TextField,`. Subtitle0 is the
disable-subtitles button and is skipped.

The parser:
  - reads each top-level .jar via udf.read_file
  - opens it with the existing zip dependency
  - confirms com/dbp/ presence in the central directory
  - walks .class entries, extracts printable strings, matches the
    `TextField,(Audio|Subtitle)<N>,<label>,...` pattern
  - maps human-readable language names ("English", "Castilian
    Spanish", "Brazilian Portuguese", "Canadian French", ...) to
    ISO 639-2 codes via a parser-local table (per the rules-of-
    engagement memo, each parser knows its own format)
  - preserves the full disc-authored label string in `name` so
    consumers display it raw without the lib guessing further
    structure
  - detects SDH / Forced qualifiers and Commentary / Descriptive
    purposes from substring matches; everything else falls through
    to fill_defaults using BD-spec MPLS data

Verified live on the corpus: disc-07 (Civil War UHD) now matches
parser=dbp with 3 audio + 2 subtitle labels, exactly the count
visible in the disc's authored TextField definitions and what BD
spec MPLS reports.

Limitation: dbp's detect() returns true for ANY top-level .jar in
/BDMV/JAR/ (every BD-J disc has one), since the discriminator
trait function takes only `&UdfFs` and can't read jar contents.
parse() does the real com/dbp/ check — a non-dbp disc gets
parsed-as-dbp, archive_has_dbp returns false, parse() returns
None, and we fall through. Diagnostic noise: parsers_detected
includes "dbp" on non-dbp BD-J discs. Real fix is refactoring the
DetectFn signature to take a SectorReader; deferred.

7 unit tests cover the TextField extraction, language detection
(simple + compound: "Brazilian Portuguese", "Castilian Spanish",
"Canadian French", "Latin American Spanish", "Australian English"
plus the disc-corpus typo "Austrailian English"), SDH/Forced/RNIB
qualifier detection, and Commentary/Descriptive purpose detection.
Don't-guess discipline preserved: unknown languages return ""
(consumer falls back to MPLS spec data via fill_defaults).
2026-05-10 13:15:07 -07:00
MattJackson 3244fdd683 v0.18.7: Pass 1 fast-skip, defer recovery to Pass N
Pass 1 sweep was grinding through damage zones because the marginal-
media handler returned `Bisect` for every failed 32-sector batch —
forcing 32 single-sector reads per bad block at ~5s each on a real
BU40N-vs-Dune-Pt-2 trace. AND the JumpAhead trigger required a 16-
block damage window to fill before firing, so entry into a
contiguous damage zone took ~40 minutes of grinding before the
first jump fired. Architecturally wrong: Pass 1's job is "fast and
accurate, get the most data in the shortest time." Bisection +
recovery is Pass N's purpose-built role.

ReadCtx now carries two new fields:
  - `consecutive_outer_failures: u64` — outer-batch failures since
    last outer success. Bisect inner failures don't count.
  - `bisect_on_marginal: bool` — whether to return Bisect on a
    marginal-media batch failure.
  - `fast_jump_threshold: u64` — outer-failures count that triggers
    JumpAhead before the damage window has filled.

`for_sweep` (Pass 1) sets `bisect_on_marginal=false`,
`fast_jump_threshold=4`, and zeroes the post-failure pause. Failed
batches become SkipBlock → whole block NonTrimmed → advance, no
sleep. After 4 consecutive outer failures: JumpAhead with the
existing escalating multiplier.

`for_patch` (Pass N) sets `bisect_on_marginal=true`,
`fast_jump_threshold=u64::MAX`, keeps the original cooldown pauses.
Pass N's whole reason to exist is to grind on bad ranges with
proper recovery semantics — single-sector reads, 60s recovery
timeout, retry budget, escalating skip — and that's unchanged.

`on_success` resets `consecutive_outer_failures` only when not
bisecting, so a good single-sector read inside Pass N's bisect
doesn't pretend we've escaped the damaged batch.

Tests:
  - `pass_n_marginal_with_batch_gt_1_bisects` — Pass N still bisects.
  - `pass_1_marginal_skips_instead_of_bisecting` — Pass 1 doesn't.
  - `pass_1_jumps_after_4_consecutive_outer_failures` — fast-entry.
  - `pass_n_does_not_fast_jump` — fast-entry is Pass-1-only.
  - `outer_success_resets_consecutive_outer_failures` — counter reset.
  - `bisect_inner_success_does_not_reset_outer_counter` — semantics.
  - `pass_1_does_not_pause_on_skip` — explicit zero-pause contract.
  - `long_failure_streak_extends_pause_on_pass_n` — Pass N still
    extends pauses on long failure streaks (renamed from the old
    sweep-based test).

Integration test `test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed`
updated: it used to assert Pass 1 recovers all sectors via bisect
(bytes_good=total). New contract: Pass 1 marks NonTrimmed; Pass N
recovers. Test now asserts Pass-1-only outcome (bytes_pending=total,
complete=false) consistent with the redesign.

Real-world impact on the user's BU40N + Dune Pt 2 trace from this
session: a damage zone that was on track to take ~40 minutes of
Pass-1 grinding will now jump in ~20 seconds. Pass N still has the
full 7-pass recovery budget to revisit those NonTrimmed ranges.
2026-05-10 12:50:12 -07:00
MattJackson ef5487e5a5 labels: distinguish "parser detected" from "parser succeeded"
Adds `parsers_detected: Vec<&'static str>` to `LabelAnalysis`. Records
every parser whose discriminator matched, regardless of whether its
parse() then returned Some/None.

Why: when `parser=None` we currently can't tell apart:
  (a) no parser recognized this disc — missing parser, candidate for a
      new module
  (b) a parser recognized it but parse() returned None / empty —
      capture truncated, or genuine empty authoring data, or a parser
      bug

Surfaced concretely on the 11-disc capture session 2026-05-10:
disc-04 had `bluray_project.bin` in jar_inventory (pixelogic detect()
returned true) but parse() returned None because file content was
past the 1 GB capture window. Old API: parser=None — looked like
"missing parser." New API: detected=[pixelogic], parser=None — clearly
"capture problem, not a parser gap."

The tracing log line on the no-parser-emitted-labels path now
distinguishes the two cases too.

No behavior change to production rip path. extract() is unchanged;
only the diagnostic analyze() returns the richer result.
2026-05-10 12:36:05 -07:00
MattJackson e7dffa63d2 v0.18.6: bump version (unified release with bdemu/freemkv/autorip) 2026-05-10 10:03:03 -07:00
MattJackson e5989c8270 labels: expose analyze() for corpus regression tooling
Promotes `mod labels` to `pub mod labels` and adds `analyze()` plus
`LabelAnalysis` (both `#[doc(hidden)]`) so an out-of-tree diagnostic
binary (freemkv-tools labels-analyze) can introspect which BD-J parser
matched a given disc, what JAR files the discriminators saw, and what
labels came out — without going through the production `apply()` path
that mutates DiscTitles.

Also adds `tracing::info!(parser = name, "label parser matched")` /
"no label parser matched" inside `extract()`. Dev-only signal: users
get the same seamless behavior; developers can finally tell whether a
disc hit a real parser or fell through to the codec-name fallback in
fill_defaults().

The new `jar_inventory()` helper deduplicates and sorts filenames
under any `/BDMV/JAR/<x>/` subdirectory — same plumbing the existing
`jar_file_exists()` discriminators use, just enumerated rather than
predicate-tested. Used by `analyze()` to surface unrecognized
parser-source files when no parser matches, which is the input to
"do we need a new parser?" triage.

No behavior change to the production label path. `apply()` and
`extract()` remain functionally identical; the new public surface
exists alongside.
2026-05-10 07:42:45 -07:00
MattJackson e1c938b82a ci: drop --locked from libfreemkv workflows
libfreemkv is a library — Cargo.lock is gitignored (standard for
libs). --locked refuses to create a lockfile on a fresh runner,
so it always fails CI. --locked stays in the binary crates
(freemkv, autorip, bdemu) which DO track Cargo.lock and benefit
from the dependency-race hard-fail behaviour.
2026-05-09 20:40:31 -07:00
MattJackson ed801a708b fmt: rustfmt-mandated reflow of canonical_title_order tests
The 0.18.4 commit landed with rustfmt diffs in the new
canonical_order tests because my local validation script piped
'cargo fmt --check' to 'tail -1', masking the diff output and
reporting green when fmt was actually unhappy. CI's lint job
caught it immediately. No code change — pure formatting.
2026-05-09 20:34:40 -07:00
MattJackson d94451954c v0.18.4: cargo --locked everywhere — hard-fail dependency races 2026-05-09 20:30:34 -07:00
MattJackson 385c9f094c v0.18.3: canonical_title_order — main feature first on branching UHDs 2026-05-09 20:08:56 -07:00
MattJackson 9f2a13739d Disc title order: main feature first on branching UHDs
Disc::titles previously sorted purely by duration_secs descending,
which puts a play-all virtual playlist at index 0 on UHDs that ship
one. Such playlists reference the same source clips multiple times
for seamless alternate-angle / alternate-ending playback and report
inflated duration AND inflated size_bytes that exceeds the disc's
physical capacity.

Concrete observed case (The Amateur 2025 4K UHD, 58.5 GB BD-100):
  Title 1 — 00020.mpls — 4h13m — 92.4 GB — 253 clips  ← impossible
  Title 2 — 00800.mpls — 2h02m — 57.2 GB — 1 clip      ← the movie

92.4 GB > 58.5 GB capacity is proof of clip double-counting. With
the duration-only sort, freemkv -t 1 / disc.titles.first() / autorip's
main-feature picker all selected the 4-hour composite instead of the
2-hour movie.

New canonical_title_order:
  1. Real titles (size_bytes <= capacity_bytes) before virtual
     composites — capacity gate is hard physical truth.
  2. Among real titles, fewer clips first (1-clip wins as the
     canonical main feature; multi-clip is either chapter-stitched
     or composite).
  3. Tiebreak on longer duration first.

Behaviour:
- Non-branching discs: unchanged. The longest 1-clip title is
  already the movie.
- Branching UHDs: virtual composite drops to the back, the real
  movie surfaces at index 0.

Comparator exposed as Disc::canonical_title_order for downstream
consumers that need the same logic on custom title sets.

Three regression tests (disc::tests::canonical_order_*):
- pushes_oversize_play_all_behind_real_main (The Amateur)
- preserves_natural_ranking_on_normal_disc
- fewer_clips_wins_tiebreak
2026-05-09 19:57:07 -07:00
MattJackson d9ce69bc9d v0.18.2: fix AACS nav-file scramble + sweep progress non-regression
decrypt::decrypt_sectors now restores chunks when decrypt_unit_full's
TS-sync verification fails, preventing 0.18.1's silent corruption of
MPLS/CLPI navigation files when DecryptingSectorSource decorates the
sweep reader. Fixes E6009 NoStreams on info iso:// for AACS-encrypted
UHDs ripped without --raw.

Disc::sweep progress takes max(snapshot.bytes_good, bytes_done) so
the user-visible counter never regresses below what the producer has
already sent.
2026-05-09 17:19:47 -07:00
MattJackson 7cd2c937ed 0.18.1 docs: refresh README, CHANGELOG, and docs/ for the trait split
The library's public-facing docs were sitting on the 0.17 trait
surface — Disc::copy, pes::Stream, SectorReader, etc. — even though
all in-tree callers migrated in 0.18 rounds 1-3. With 0.18.1 about
to ship, a user copy-pasting the README sample from crates.io would
have hit a compile error.

This commit is purely doc-side:

- README.md: Quick Start rewritten onto Disc::sweep + Disc::patch
  with caller-orchestrated multipass; Streams table footnote and
  Architecture row reference FrameSource / FrameSink.
- CHANGELOG.md: 0.18.1 entry describing the redesign — primitives,
  trait splits, deprecations (kept alive through 0.18.x, deletion
  target 0.18.2), throughput numbers.
- docs/{rip-recovery,api-design,architecture,disc-to-rip,
  drive-access,udf}.md: every Disc::copy / pes::Stream /
  SectorReader reference updated to the 0.18 trait surface.
- FEATURES.md: deleted (8+ versions stale; capabilities live in
  README.md and CHANGELOG.md now, matching the workspace-top
  FEATURES.md removal in 84acd65).
- examples/iso_dump.rs: verified compiles against 0.18.1.

No code changes.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 12:13:51 -07:00
MattJackson 107524da5e 0.18.1: bump version 2026-05-09 11:38:24 -07:00
MattJackson 055a3c5276 0.18 round 3: make Disc::sweep + Disc::patch pub (was pub(crate))
Round 3 step 1: lift the visibility on the two flat rip-phase verbs
so consumers (autorip + freemkv CLI) can call them directly instead
of going through Disc::copy's multipass dispatcher. Also lift their
option/outcome types and re-export at crate root.

- fn sweep -> pub fn sweep (with rustdoc explaining its role)
- fn patch -> pub fn patch (ditto)
- pub(crate) struct SweepOptions -> pub struct SweepOptions
- pub(crate) struct PatchOpts -> pub struct PatchOptions (renamed
  for consistency — both are 'Options')
- pub(crate) struct PatchOutcome -> pub struct PatchOutcome
- libfreemkv::{SweepOptions, PatchOptions, PatchOutcome} re-exports
  at crate root.

Disc::copy still exists and still calls Disc::sweep / Disc::patch
through the now-private sweep_internal / patch_internal wrappers.
Migration of the two autorip callers + the freemkv CLI's
disc_to_iso to direct sweep/patch is a follow-up; once those land
Disc::copy + CopyOptions + CopyResult delete in the same commit.

See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.

Single contributor: MattJackson.
2026-05-09 11:19:46 -07:00
MattJackson 50b04790ce 0.18 round 2: thread Halt through DiscStream construction
# Conflicts:
#	src/mux/disc.rs
2026-05-09 11:04:07 -07:00
MattJackson a00b2c884b 0.18 round 2: adopt DecryptingSectorSource decorator at sweep + patch + DiscStream 2026-05-09 11:03:43 -07:00
MattJackson 44e2c64d7e 0.18 round 2 (Halt threading): DiscStream accepts Halt at construction
Adds a Halt field to DiscStream, populated via the new
`with_halt(halt)` builder. The internal recovery / fill_extents
loops check `halt.is_cancelled()` directly. The existing
`set_halt(Arc<AtomicBool>)` method stays through the deprecation
window for callers (autorip mux) that haven't migrated; marked
#[deprecated] with a pointer to the constructor-time path.

Both signals are unified inside DiscStream: either Halt or the
legacy Arc<AtomicBool> triggers cancellation, so callers can mix
during the deprecation window without breaking stop behaviour.

See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.

Single contributor: MattJackson.
2026-05-09 10:58:54 -07:00
MattJackson d6535b8f57 0.18 round 2 (decrypt dedup): adopt DecryptingSectorSource at the two
existing call sites — sweep producer and DiscStream demux

Round 1 shipped the DecryptingSectorSource decorator
(libfreemkv/src/sector/decrypting.rs) but the existing decrypt
sites kept calling crate::decrypt::decrypt_sectors inline. This
commit migrates both:

- Disc::sweep (disc/mod.rs): producer wraps the input reader
  in DecryptingSectorSource::new(reader, keys) before the read loop.
  The inline decrypt_sectors call goes away — read_sectors yields
  plaintext directly.

- DiscStream (mux/disc.rs): constructor wraps the underlying
  Box<dyn SectorReader> in DecryptingSectorSource so the internal
  fill_extents / read path sees plaintext bytes. The DecryptKeys
  field stays on DiscStream for metadata-side use; it just no
  longer drives decryption.

Disc::patch carried the same inline decrypt step at three call
sites (main read, backtrack read, non-NOT_READY retry read). All
three migrated onto the same wrapping for a single audit surface.

Two small support changes carry the migration without touching
the round-1 decorator shape:
- sector/mod.rs gains specific SectorSource impls for
  &mut dyn SectorReader and Box<dyn SectorReader>, mirroring
  std's Read forwarding pattern. Generic blankets would conflict
  with the existing SectorReader → SectorSource blanket under the
  orphan rule (downstream could impl SectorReader for &mut U), so
  the impls are scoped to the dyn-trait shape we actually consume.
- sector/decrypting.rs gains DecryptingSectorSource::set_keys so
  DiscStream::set_raw() can flip the wrapped reader to a
  DecryptKeys::None pass-through without rebuilding the decorator
  (which would require moving the inner Box out from behind &mut self).

After this commit, grep `decrypt_sectors` in src/ shows the
function definition, its single use inside DecryptingSectorSource,
plus comments only. One audit surface for AACS / CSS / passthrough
correctness.

Behaviour-preserving: same plaintext bytes flow through; the only
difference is which type owns the decrypt step.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:49:53 -07:00
MattJackson 90ab00ed45 0.18 round 2: cargo fmt after sweep+patch+DiscStream merge
Auto-fmt nit on the multi-line map.flush() expression that landed
when the sweep + patch + DiscStream-FrameSource branches were merged
together. No semantic change.
2026-05-09 10:36:40 -07:00
MattJackson 6bfd7dfd13 0.18 round 2: confirm DiscStream as FrameSource via blanket impl 2026-05-09 10:32:56 -07:00
MattJackson b9a7f601d4 0.18 round 2: refactor Disc::patch onto Pipeline + PatchSink
# Conflicts:
#	src/io/mod.rs
#	src/io/pipeline.rs
2026-05-09 10:32:51 -07:00
MattJackson cf0a61f8b4 0.18 round 2: refactor Disc::sweep onto Pipeline + SweepSink (delete sweep_pipeline.rs) 2026-05-09 10:32:05 -07:00
MattJackson f98f07b2d3 0.18 round 2: refactor Disc::sweep onto Pipeline + SweepSink
Sweep was the original producer/consumer split that motivated the
generic Pipeline primitive (round 1, commit 198268b). Now that
Pipeline + Sink exist, sweep stops shipping its own bespoke
threading.

- New SweepSink: Sink<WorkItem> impl in src/disc/sweep.rs. Owns
  WritebackFile + Mapfile + ProgressSnapshot back-channel. apply()
  carries the file-write + mapfile.record per WorkItem; close()
  drains writeback, fsyncs, flushes mapfile.
- Disc::sweep: constructs SweepSink, calls Pipeline::spawn_named
  (so the consumer thread keeps showing up as
  freemkv-sweep-consumer), sends WorkItems, calls pipe.finish().
  The producer-side ReadCtx state machine, decrypt, set_speed,
  halt — all unchanged.
- Pipeline gains spawn_named(name, depth, sink) so callers can
  preserve identifiable thread names without the primitive baking
  one in. Also adds Pipeline::try_send for the throttled
  StatsRequest path that must not block the producer.
- Deleted src/disc/sweep_pipeline.rs entirely. WorkItem,
  ProgressSnapshot, ConsumerSummary moved into disc/sweep.rs as
  module-private types. WorkItem::Finish dropped — dropping the
  channel is the end-of-stream signal Pipeline already uses.

Behaviour-preserving: the sweep algorithm, mapfile invariants,
back-pressure via channel depth (DEFAULT_PIPELINE_DEPTH = 4) all
match the 0.17.13 implementation. New synthetic regression test
(sweep_pipeline_full_good_100_batches) exercises ~100 batches of
clean reads end-to-end through the new Pipeline path and verifies
bytes_good and ISO file size.

See (internal)/memory/0_18_redesign.md.
2026-05-09 10:31:05 -07:00
MattJackson b53454fa09 0.18 round 2: refactor Disc::patch onto Pipeline + PatchSink
Patch was strictly serial (per-sector recovery: read → seek+write
→ mapfile.record → next). Lifting the write+record onto a consumer
thread lets the drive issue the next per-sector retry while the
previous block's recovered bytes are being committed — small but
real win on damaged discs with many bad sectors, and uniform with
sweep's threading model.

- New PatchSink: Sink<PatchItem> impl in src/disc/patch.rs. Owns
  WritebackFile + Mapfile. apply() seeks+writes recovered bytes
  and records mapfile state per item; close() runs sync_all and
  mapfile.flush.
- Channel depth: WRITE_THROUGH_DEPTH (1). Patch wants minimum
  buffering — back-pressure should kick in immediately so the
  drive's per-sector retry budget isn't ahead of the writer.
- Disc::patch: keeps every existing recovery decision on the
  producer (reverse walk, damage-window skip, NOT_READY pauses,
  bridge-degradation handling, wedge exit, range watchdog).
  WritebackFile ownership moves to the sink.

Behaviour-preserving: per-sector single-shot read budget unchanged
(BU40N+Initio bridge wedge concern still respected); recovery
algorithm bit-identical.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:28:14 -07:00
MattJackson f28b6ee6d9 0.18 round 2: re-export Pipeline + Sink + Flow at the crate root
Round 2 #1 landed `Pipeline` / `Sink` / `Flow` / `DEFAULT_PIPELINE_DEPTH`
in `crate::io::pipeline` but only re-exported them through
`crate::io` (which is `pub(crate)`), so no out-of-tree consumer could
reach them. autorip's round 2 #2 (lifting the mux loop onto Pipeline +
MuxSink) is the first such consumer; surface the primitives at the
crate root for ergonomic access.

No behaviour change — the items themselves are unchanged from round
2 #1; this is just `pub use` plumbing.

Single contributor: MattJackson.
2026-05-09 10:22:44 -07:00
MattJackson d9d778fa64 0.18 round 2: confirm DiscStream as FrameSource
Mirror of the FrameSink concrete migrations slice (f52e4c5) on the
read side. DiscStream is the only meaningful source impl in tree;
all the mux/* impls are sinks.

The round-1 blanket impl<T: Stream + Send> FrameSource for T
already covers DiscStream if it's Send. This slice:

- Audits DiscStream's interior types for Send (its Box<dyn
  SectorReader> already requires Send via the trait's super-bound;
  verify nothing else interior breaks Send).
- Adds a synthetic-input test that constructs Box<dyn FrameSource>
  over a DiscStream, reads frames through the trait object, and
  exercises info() / headers_ready() / codec_private().
- (Conditional) Adds a direct FrameSource impl on DiscStream only
  if call-site ergonomics demand it; otherwise relies on the
  blanket.

No caller migrated. mux::resolve::input still returns
Box<dyn Stream>; autorip / CLI consumers still call Stream::read.
Per-caller migration is a later slice.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:13:14 -07:00
MattJackson bbfb887a35 0.18 round 1+2 integration fixes
Two clippy issues surfaced when round 1 polish + round 2 FrameSink
migrations both landed on libfreemkv main:

- src/halt.rs: clippy::new_without_default fires when a public new()
  exists without Default. The polish pass dropped the derive thinking
  it was redundant — clippy disagrees, so add a manual impl that
  forwards to new(). Doc-comment notes why both exist.

- src/disc/read_error.rs:372: pre-existing
  assert_eq!(.., true) trips clippy::bool_assert_comparison. Pre-0.18
  precommits passed because that lint sat outside the gate; the
  round-2 commits brought enough new clippy surface that it now
  shows up. Trivial cleanup: assert!(...) instead of assert_eq!.

Single contributor: MattJackson.
2026-05-09 10:00:06 -07:00
MattJackson 9884346c24 0.18 round 2: FrameSink concrete migrations for mux/* sinks 2026-05-09 09:53:12 -07:00
MattJackson 766b7c6636 0.18 round 1 polish: address libfreemkv code-review findings 2026-05-09 09:53:06 -07:00
MattJackson 925c30686b 0.18 round 1 polish: address libfreemkv code-review findings
Applies must-fix + in-scope should-fix items from the round-1 code
review:

- M1: FileSectorSource::open takes &Path (was &str — non-UTF-8 panic)
- M2: drop FileSectorSource's BufReader (defeated by absolute seeks)
- M3: WritebackFile Drop impl finalises the writeback pipeline
- M4: Pipeline::finish preserves panic payload in error message
- M5: pes::Stream is left without a : Send supertrait — concrete
  in-tree impls (MkvStream, M2tsStream) hold Box<dyn Read> /
  Box<dyn Write> trait objects that aren't Send, so the simple
  trait tightening would cascade into a wider Send audit. Per the
  review's escape clause the FrameSource blanket impl keeps its
  T: Send bound and the constraint is documented loudly there.
- S6: document Pipeline::send post-Flow::Stop semantics
- S9: truncate stale Stream docs (E9001/E9000 was runtime-only)
- S10: document WritebackPipeline.fd lifetime invariant
- S11: pub use pes::Stream as PesStream to disambiguate from
  disc::Stream codec enum at crate root
- S12: rename DEFAULT_DEPTH → DEFAULT_PIPELINE_DEPTH; add
  WRITE_THROUGH_DEPTH constant
- N14: drop Halt's Default derive (redundant with Halt::new)
- N17: Pipeline::spawn propagates thread-spawn error instead of expect
- N19: deprecation since = "0.18.0" (was "0.18.0-dev", non-conventional)
- N21: rename Apply enum to Flow

Deferred to follow-up commits: SectorReader/SectorSource competition
(migration commit), WritebackFile::create/open orphans (migration
commit), AACS round-trip test (design doc defers), various nits.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:52:25 -07:00
MattJackson f52e4c5d22 0.18 round 2: add FrameSink impls to concrete mux sinks
Per-impl migration of MkvStream / M2tsStream / NetworkStream /
NullStream / StdioStream from the deprecated pes::Stream trait
to the typed pes::FrameSink trait. Both impls coexist during
the 0.18 deprecation window — the existing Stream impls are
unchanged.

The FrameSink::finish signature differs (Box<Self> vs &mut self),
which is why this couldn't be a blanket impl. Each migration
re-borrows the box and delegates to the underlying Stream::finish
body.

FrameSink: Send forced two struct fields (M2tsStream's boxed
Write/Read, MkvStream's boxed WriteSeek/Read) to gain `+ Send`
bounds — minimum surface needed to make the Send-bounded trait
impl-able. mux::resolve::output's local Box<dyn WriteSeek>
construction picks up the same `+ Send`. tests/streams.rs's
shared `stream.write/.finish/.info/.read` calls were
disambiguated to `PesStream::*` to resolve the now-multiple
candidates from coexisting trait impls.

Caller migration (mux::resolve::output return type, autorip,
CLI) is a later slice. This commit only adds new impls; nothing
removed.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:49:18 -07:00
MattJackson 8d67790d2e 0.18: thread WritebackFile rename through FileSectorSink
The SectorSource/Sink agent and the WritebackFile-rename agent both
branched from main concurrently; the sector branch wrote against the
0.17 Writer name and only the rename branch knew about WritebackFile.
This integration commit reconciles the two: FileSectorSink::create /
::open / the inner-field type all use WritebackFile directly, and the
module-level + struct-level docs are corrected.
2026-05-09 09:17:39 -07:00
MattJackson 246a439990 0.18: FrameSource/FrameSink trait split (deprecate Stream) 2026-05-09 09:13:39 -07:00
MattJackson bd19041648 0.18: SectorSource/SectorSink trait split + DecryptingSectorSource decorator 2026-05-09 09:13:34 -07:00
MattJackson 198268b725 0.18: add crate::io::Pipeline + Sink trait
# Conflicts:
#	src/io/mod.rs
2026-05-09 09:13:30 -07:00
MattJackson 31424c8203 0.18: add crate::halt::Halt cancellation token 2026-05-09 09:13:07 -07:00
MattJackson c747ecc589 0.18: rename crate::io::Writer → WritebackFile 2026-05-09 09:12:59 -07:00
MattJackson 283a561c12 0.18 primitive: SectorSource/SectorSink trait split + DecryptingSectorSource
Splits the unidirectional read trait from a (planned) write trait at
the sector level, eliminating runtime "wrong direction" potential.
Keeps SectorReader alive as a pre-deprecation alias via blanket impl
so existing callers compile unchanged through the migration window.

Adds DecryptingSectorSource decorator: wrap any SectorSource in this
to get plaintext sectors out. Replaces the duplicate decrypt code
paths in sweep_pipeline and DiscStream (those migrations are
follow-up commits).

The formal #[deprecated] attribute on SectorReader is held back to a
follow-up commit because internal call sites in disc/, udf/, mux/,
and verify/ still go through the legacy trait, and the CI gauntlet
treats deprecation lints as errors. Behavioural intent — "this trait
is going away" — is documented on the trait itself.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:04:16 -07:00
MattJackson 2667d68675 0.18 primitive: FrameSource/FrameSink trait split (deprecate Stream)
Splits the bidirectional pes::Stream into one-direction traits so
calling read() on a write-only sink is a compile error instead of
runtime E9001. Keeps Stream alive as deprecated through 0.18 with a
blanket FrameSource impl so existing concrete types compile unchanged.

FrameSink can't be blanket-impl'd from Stream (different finish
signature), so concrete impls migrate per-type in a follow-up.

Concrete `impl pes::Stream for X` blocks in mux/* and the existing
tests gain a one-line `#[allow(deprecated)]` to keep `-D warnings`
clean during the deprecation window — no behavior changes.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 08:57:48 -07:00
MattJackson 57f2c22e30 0.18 primitive: crate::io::Pipeline + Sink trait
Generic bounded producer/consumer pipeline. The same shape applies to
sweep, patch, and mux today via three near-duplicate implementations
(or, in patch's and mux's case, no implementation at all). 0.18
collapses them onto one primitive. See
(internal)/memory/0_18_redesign.md for full context.

Single contributor: MattJackson.
2026-05-09 08:56:31 -07:00
MattJackson 5f3545d244 0.18 primitive: rename crate::io::Writer → WritebackFile
The type's job is the bounded-cache writeback pipeline (sync_file_range
+ posix_fadvise(DONTNEED)) — not generic writing. The 0.17 name was
ambiguous; reading `Writer::new(file)` gave no hint about what was
special. New name makes the role obvious at every call site.

Adds `WritebackFile::create(path)` and `WritebackFile::open(path)`
constructors so callers don't have to assemble a `File` first.

No alias kept; this is a clean 0.18 rename. See
(internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 08:53:17 -07:00
MattJackson 6e17ef0859 0.18 primitive: crate::halt::Halt cancellation token
One-bit cooperative cancellation flag. Replaces ad-hoc Arc<AtomicBool>
patterns scattered across libfreemkv (DiscStream::set_halt) and the
HALT_FLAGS global registry in autorip. See
(internal)/memory/0_18_redesign.md.

Single contributor: MattJackson (no attribution trailers).
2026-05-09 08:50:13 -07:00
MattJackson 6ec97af104 v0.17.13: thread Writer through patch + mux for big-write consistency
The bounded-cache writeback wrapper (crate::io::Writer) was added in
0.17.10 and wired into Disc::sweep in 0.17.11, but the other two
paths in the crate that write large amounts of data sequentially —
Disc::patch and the MKV/M2TS mux — were still operating on raw
std::fs::File. That meant the dirty-page burst pathology the wrapper
exists to prevent could still bite on slow / network-attached staging
during recovery and mux phases.

This release plugs those gaps:

- Disc::patch (disc/mod.rs:1981) now wraps the reopened ISO in
  Writer before any seek / write. sync_all on Writer cleanly drains
  the in-flight chunk before the existing fsync.
- mux/resolve.rs MKV and M2TS branches wrap the output File in
  Writer underneath BufWriter. UHD MKV mux routinely produces 70+ GB
  of sequential output; the page cache no longer absorbs that as a
  single hot blast on slow targets.

Mapfile, log, settings, history, and stream-pipeline byte buffers
remain unchanged: those are either small one-shot writes (where
the wrapper has zero benefit and adds a stream_position syscall) or
already use bounded persistence (mapfile time-batched in 0.17.12).
The principle: any path that writes substantial sequential data to
a single file uses Writer; trivial writes don't.
2026-05-09 06:32:08 -07:00
MattJackson 3a6c1aa5a3 v0.17.12: mapfile time-batched persistence — unblock NFS staging
Pre-0.17.12 every Mapfile::record() persisted the full mapfile via
tempfile-create + write + atomic-rename. On local LVM that's
microseconds; on NFS each rename is multiple RPCs through the
unraid user-share fuse layer, dragging a Black Mass UHD rip from
~11 MB/s on local to ~1.5 MB/s on NFS — the mapfile path alone burned
multiple seconds of wall time per real-world second of work.

Mapfile now batches the rename to once per second:

- record() always updates in-memory state and stats; only fires
  write_to_disk when last_flushed.elapsed() >= FLUSH_INTERVAL (1 s).
- New flush() API forces a persist; called by sweep_pipeline's
  consumer at end-of-sweep and by Disc::patch at end-of-patch,
  after the file's sync_all.
- Drop impl best-effort flushes so an early-return / unwind doesn't
  silently lose pending state.

Crash-safety changes from "lose at most one block" to "lose at most
1 s of recorded progress" — the ISO file's payload bytes are unaffected;
only the mapfile's authority over which sectors are already-good is at
risk, and a resume re-reads anything Pass 1 had already covered.

Measured on the BU40N test bed against Black Mass UHD inner zone:
- NFS staging: 1.5 MB/s → 16.48 MB/s (10.9× recovery)
- Local LVM staging: 11.09 MB/s → 11.83 MB/s (+6.7 % bonus)

Internal round_trip_load test now flushes before reading back from
disk. External patch / copy tests are unaffected: patch and
sweep_pipeline flush at completion before returning.
2026-05-08 23:00:52 -07:00
MattJackson 1ba3264747 v0.17.11: sweep producer/consumer split — overlap drive read with file write
Pre-0.17.11 sweep ran strictly serialised: SCSI read → decrypt → seek
+ write → mapfile.record → next read. Drive idled for the post-read
work; throughput capped at the sum of both costs. On a healthy disc
that's ~7-12 ms read + ~5-15 ms write/record per 64 KB batch, limiting
sustained throughput to ~10-12 MB/s on the test bed (BU40N + UHD inner
zone), well below the ~14-16 MB/s drive ceiling.

Decouples them: producer thread (caller's) owns SectorReader +
read_error state + decrypt + set_speed + halt; consumer thread (one
spawn) owns Writer + Mapfile, receives WorkItem messages, applies
file write + mapfile record. Bounded mpsc::sync_channel(4) gives
natural back-pressure. While the consumer writes batch N, the
producer is already reading batch N+1 — steady-state throughput is
now bound by the slower of the two pipelines (drive on healthy
discs), not their sum.

Side effects:
- Bisect path now decrypts. Pre-0.17.11 the bisect inner loop wrote
  raw cyphertext for single-sector recoveries on encrypted discs —
  quiet correctness bug exercised only by batch-fail-then-
  bisect-succeed on encrypted media. New producer-side decrypt
  covers main + bisect success paths uniformly.
- All read_ctx state stays single-threaded on producer (damage
  window, jump multiplier, etc.). No locking added.
- Mapfile remains single-writer on consumer. No locking.
- Halt latency: producer breaks loop, sends Finish, consumer drains
  ≤4 in-flight items + sync_all. ~1 batch (~12 ms) typical.
- BU40N + Initio bridge wedge concern unchanged: still single SCSI
  command in flight, error-path timing identical, no new retries.

New module: src/disc/sweep_pipeline.rs (WorkItem, ProgressSnapshot,
ConsumerInputs, spawn_consumer, consumer_loop, helpers). Public API
unchanged — Disc::copy / CopyOptions / CopyResult identical.

Patch (Pass N) is NOT changed; it's bound by drive recovery time, not
the read/write serialisation.
2026-05-08 21:32:11 -07:00
MattJackson ae2909fe8d v0.17.10: bounded-cache writeback pipeline for big sequential writes
Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
settings. Empirical cause: the kernel's vm.dirty_ratio (~20% of RAM)
lets hundreds of MB of dirty pages accumulate, then bursts a flush at
99% disk utilisation that blocks app writes for ~1 s. Confirmed on
the BU40N test bed — dirty pages grew 112 → 563 MB between bursts;
lowering vm.dirty_bytes to 64 MB at the host sysctl level eliminated
the dips. Shipping the equivalent inside libfreemkv so users do not
need to tune the host kernel.

- New crate::io::Writer: drop-in File wrapper (impl Write + Seek).
  Wraps a per-platform WritebackPipeline that on Linux schedules
  sync_file_range(WRITE) + lagging sync_file_range(WAIT_AFTER) +
  posix_fadvise(DONTNEED) in 32 MB chunks, bounding dirty cache at
  ~64 MB. macOS and Windows ship a no-op stub.
- Disc::sweep wraps its output File in Writer. Loop body unchanged.
- Module is purpose-built so any large sequential output (patch,
  mux) can adopt the same wrapper as a one-line change later.
2026-05-08 19:54:25 -07:00
MattJackson 6a3d19a453 v0.17.7: sync release — no functional changes
Version bump to keep the four freemkv crates at unified versioning
after autorip's v0.17.6 + v0.17.7 work today. No libfreemkv code
changes; republished to crates.io so downstream consumers stay
aligned on the latest patch version.
2026-05-08 16:20:28 -07:00
MattJackson a35596d2d1 v0.17.5: Pass N kernel block-device fallback + per-range fixes
Direct-SATA BU40N + Dune Part Two UHD live testing exposed that the
v0.17.3 single-shot SCSI READ path matched 0/22 of the small bad-
sector LBAs that dd if=/dev/sr0 recovers on the same drive. This
release closes that gap and fixes adjacent bugs silently capping
recovery.

- /dev/sr0 pread fallback in Drive::read (Linux only): on SCSI READ
  Err, fall back to posix_fadvise(DONTNEED) + pread() against the
  corresponding block device. Kernel sr_mod runs ~5 internal retries
  with no per-attempt mid-layer escalation overhead — the mechanism
  behind dd's recovery advantage. End-to-end byte verification
  confirms the fallback path returns real disc data.

- Disc::patch per-range watchdog fix: MAX_RANGE_SECS was breaking
  'outer (one slow range killed the entire patch). Now skips to the
  next range. Pre-fix patch died after 4 sectors of range 1 of 47.

- Per-sector range budget: range_budget = sectors × 25 s, capped at
  1800 s. Replaces the flat 180 s/range that was unfair to medium
  ranges and pointlessly generous to single-sector ones.

- consecutive_failures resets per range. The wedge-exit detector is
  for stuck-on-one-range, not many-small-ranges-with-one-fail-each.

- Reverted inline 5× retry experiment (was hurting: each retry paid
  kernel SCSI escalation overhead). Restored READ_RECOVERY_TIMEOUT_MS
  to 60 s. The kernel-auto-retry pattern is now provided by sr0
  fallback.

Empirical: pass 1 recovered 94.6 MB / 11 s of main title (33 sr0
saves). Pass 2 added 0.6 MB. Remaining ~233 MB on the test disc
appears physically unrecoverable on this hardware.
2026-05-08 12:58:48 -07:00
MattJackson 318f654fed v0.17.3: fix scsi_recovery.rs Linux compile (missing Duration import)
tests/scsi_recovery.rs:
- Add `use std::time::Duration` inside both `#[cfg(target_os = "linux")]`
  blocks. Locally on macOS the linux blocks are cfg-out so the missing
  import was invisible to precommit on macOS.
- Bug pre-dated this branch but only surfaced when v0.17.2 release CI
  ran the test compile on Linux.

Cargo.toml: 0.17.2 -> 0.17.3.
2026-05-07 19:16:21 -07:00
MattJackson 5e82ef65de v0.17.2: bump version (skip 0.17.1 — stray tag exists)
Cargo.toml: 0.17.1 -> 0.17.2. Functionally identical to the prior
commit; 0.17.1 was never published to crates.io but a tag exists on
the remote pointing at an unrelated commit. Bumping past it.
2026-05-07 19:10:39 -07:00
MattJackson 8534607329 v0.17.1: cache priming, NonTrimmed marking, decrypt regression test
src/disc/mod.rs:
- Cache priming (3-sector lookback) before patch's single-sector reads.
  Drive read-ahead pulls in adjacent pages so the target may already be
  cached when we ask for it. Throwaway reads — failures here don't
  update mapfile state.
- When patch hits skip-limit on a range, leave remaining sectors
  NonTrimmed instead of marking Unreadable. We never tried to read those
  sectors, so don't give them terminal status — drive state evolves
  between passes (cache, mechanical settle), and a later pass may
  succeed.

tests/pass_n_patch_fix.rs:
- New regression test for the decrypt key inversion bug at
  src/disc/mod.rs:1938-1942. Asserts decrypt_sectors is invoked with
  the correct key when opts.decrypt=true.

tests/pass_n_size_aware_skip.rs:
- rustfmt-only changes.

Cargo.toml: 0.17.0 -> 0.17.1.
2026-05-07 19:09:17 -07:00
MattJackson 97e1a4cad3 unified read-error handler + pass N size-aware skip
New disc/read_error.rs as the single entry point all read failures flow
through. Handler classifies the error, updates the in-flight context
(damage window, retry budgets, jump multiplier), and returns a
ReadAction the caller dispatches on. Pass 1 (sweep) refactored to use
it; ~340 lines of nested if/else collapsed into ~120 lines of action
dispatch. Adding a new error class = one match arm. Logging is in one
place. Bisect inner failures don't poison the damage window. Jump
multiplier capped at 64 (max 1 GB jump for batch=32 — observed prior
unbounded behavior produce a single 56 GB jump on a wedged drive).

Pass N (patch) damage_skip is now size-aware: each skip is capped at
range_remaining/4 rather than the absolute MB-scale escalation. The
old logic could leap over a 100-sector bad range that hides a 50-sector
good middle; size-aware convergence finds the good middles instead.

Tests in tests/pass_n_size_aware_skip.rs exercise the size-aware skip
against synthetic patterns (25-bad/50-good/25-bad and three good
middles in a row) and prove ≥98% of good middles are recovered.
Existing test test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed
updated to reflect that MEDIUM_ERROR now triggers single-sector
bisect (which the BlockSizeFailingReader succeeds at).
2026-05-07 09:08:43 -07:00
MattJackson 3f6bf33de5 Format code 2026-05-04 20:06:10 -07:00
MattJackson 359301e8fc Pass 1 transport failure recovery without user intervention 2026-05-04 20:02:01 -07:00
MattJackson bfa527162a 0.17.0: unwrap safety fix, patch pass algorithm, clippy compliance
- Fix unwrap in disc/mod.rs sweep() hot path using pattern matching
- Patch pass excludes Unreadable sectors from work list
- Expose bytes_bad_in_title for accurate UI reporting
- All 256 tests pass, cargo clippy clean with -D warnings
2026-05-04 09:42:07 -07:00
MattJackson 45defd47f3 fix patch pass: exclude Unreadable from work list; expose bytes_bad_in_title; clippy 1.86 fixes
- patch(): only process NonTrimmed + NonScraped ranges (Unreadable=terminal, NonTried=not-yet-swept)
- bytes_bad_in_title: pub fn for autorip main-movie lost_ms computation
- Clippy 1.86: saturating_sub, unused assignments, unused variable
- fmt: rustfmt formatting
2026-05-03 16:35:06 -07:00
MattJackson fe8933dad2 v0.16.3: i18n damage display
- Add rip.damage_lost, rip.damage_lost_movie, rip.damage_lost_simple, rip.damage_none keys
- Update all 6 locale files (en, es, fr, de, it, pt, nl)
- CLI progress shows 'Xs lost (Y in movie)' instead of 'Xs unreadable'
- Perfect rips show '0s' instead of '0s unreadable'
2026-04-30 20:51:02 -07:00
MattJackson 0a3c1b7b70 fix: scsi_recovery test variable shadowing and missing imports for Linux CI 2026-04-30 19:57:16 -07:00
MattJackson 2bcdc97341 fix: add missing Duration import in scsi_recovery test 2026-04-30 19:52:53 -07:00
MattJackson 4863e9c545 v0.16.2: sticky escalation in patch, title-aware damage display, PASS1/PASSN constant naming 2026-04-30 19:50:08 -07:00
MattJackson 03db038dd9 docs: update project docs, CHANGELOG, drive-access.md, rip-recovery.md for v0.16.x 2026-04-30 15:43:58 -07:00
MattJackson 95001ba2c3 v0.16.1: bump version 2026-04-30 15:23:44 -07:00
MattJackson a1f4dff6f6 v0.16.0: IOKit registry-based drive enumeration, BSD name → IOBDServices matching, reverse patch default 2026-04-30 15:17:20 -07:00
MattJackson 7dd5001d45 v0.15.1: fix damage-jump detection (window=16, threshold=12%), fix dispatch covers_disc check, sweep resume from mapfile
- DAMAGE_WINDOW 50→16, DAMAGE_THRESHOLD_PCT 25→12: triggers on 2nd scattered failure
- Previous 50/25% was too diluted by good reads between sparse failures
- copy() dispatch checks mapfile total_size == disc capacity_bytes (covers_disc)
- NonTried regions → sweep with resume=true (preserves mapfile)
- Only NonTrimmed/NonScraped/Unreadable → patch
- sweep_internal takes resume param: true when dispatching from existing mapfile
- Verified: Pass 1 from 30-100% completes in ~20 min with correct jumps through 3 damage zones
2026-04-30 13:15:03 -07:00
MattJackson 67fe93c0b8 v0.15.0: multipass CopyOptions, auto-detect sweep vs patch, speed control on damage zone entry/exit
- CopyOptions: replace resume/skip_on_error/batch_sectors with single multipass bool
- Disc::copy() auto-detects pass from mapfile state: no mapfile or NonTried → sweep, only NonTrimmed/Unreadable → patch
- Fix bug where mapfile with NonTried regions incorrectly dispatched to patch mode
- SectorReader::set_speed() default method, Drive impl sends SET CD SPEED
- On damage zone entry: set_speed(0x0000) for better error recovery
- On damage zone exit (50 consecutive good): set_speed(0xFFFF) to restore max speed
- Disc::mapfile_for() returns /tmp/<name>.mapfile for null:// output
- patch_internal/sweep_internal as private helpers, CopyResult gains recovered_this_pass
2026-04-30 11:31:31 -07:00
MattJackson 5c4f575c46 v0.14.0: bump version 2026-04-30 08:45:32 -07:00
MattJackson e2cb7a78bb v0.13.46: damage-jump algorithm replaces probe, bridge degradation detection, ecc_sectors() 2026-04-29 22:08:52 -07:00
MattJackson a6f1bd19bc v0.13.45: multipass adaptive probe, NOT_READY retry, progress bytes_bad_total
- Adaptive probe algorithm in Disc::copy skip_on_error mode: after 4
  consecutive errors, probe 1 sector at 256x batch (8 MB) ahead. If
  good, zero-fill gap, mark NonTrimmed, jump. Clears bad zones in
  seconds instead of hours.
- NOT_READY sense key (0x02) now retries up to 3x with 3s pause before
  marking NonTrimmed. BU40N returns NOT READY for bad sectors, not
  MEDIUM ERROR.
- PassProgress struct gains bytes_bad_total field for consumer-side
  bad/retryable byte counts.
- Mapfile header version string fix: no longer duplicates 'libfreemkv v'
  prefix on each write.
- Structured sense_key/asc/ascq logging in copy error path.
2026-04-29 19:41:34 -07:00
MattJackson 9e3d0f1383 v0.13.44: macOS raw CDB transport via IOKit exclusive access
macOS SCSI transport rewritten from hybrid MMC+pread to single-path
raw CDB dispatch through SCSITaskDeviceInterface. All CDBs (INQUIRY,
READ, REPORT KEY, etc.) now go through ExecuteTaskSync — 1:1 with
the Linux SG_IO backend.

Key changes:
- New macos_shim.c: diskutil unmount → find IOBDServices →
  ObtainExclusiveAccess → raw CDB dispatch. Eliminates Rust-side
  IOKit COM vtable complexity.
- build.rs compiles macos_shim.c via cc into static lib
- macos.rs simplified to three FFI calls (open/close/execute)
- disc/mod.rs: graduated batch restore after errors, skip-ahead
  through bad zones, configurable error pause
2026-04-29 15:59:35 -07:00
MattJackson 1e6eb0698d v0.13.43: Pass 1 transport-failure recovery loop 2026-04-29 08:53:56 -07:00
MattJackson bd744171e4 v0.13.42: transport failure skips instead of aborting copy 2026-04-29 07:46:42 -07:00
MattJackson 7de9d4d42c v0.13.41: debug logging for sector-0 regression 2026-04-29 07:09:46 -07:00
MattJackson 9ac0c2fcdf v0.13.40 2026-04-28 21:50:32 -07:00
MattJackson 3aa27d5d1b fix CI: Rust 2024 match ergonomics - remove ref binding modifier 2026-04-28 21:47:19 -07:00
MattJackson 5083f70cff fix CI: import ScsiTransport trait in integration test 2026-04-28 21:45:34 -07:00
MattJackson 477cf7ee0e fix CI: pub scsi::linux module and SgIoTransport fields for integration tests 2026-04-28 21:43:14 -07:00
MattJackson 0b079e27ce fix CI: make scsi::linux pub(crate) for integration test access 2026-04-28 21:39:50 -07:00
MattJackson 5f69fd2a44 update CHANGELOG and README for v0.13.39 2026-04-28 21:36:37 -07:00
MattJackson befe6ef69b remove (internal) references from public code 2026-04-28 21:32:15 -07:00
MattJackson 29491d6eaf v0.13.39 2026-04-28 21:19:51 -07:00
MattJackson 2e602f7e37 fix scsi_recovery test: dereference status in pattern match 2026-04-28 21:17:11 -07:00
MattJackson a126f31b70 v0.13.38 2026-04-28 21:12:05 -07:00
MattJackson 6ff10df83a fix scsi_recovery test compilation (old Drive::read signature) 2026-04-28 21:11:37 -07:00
MattJackson 52b8522a75 v0.13.37: Pass 1 is pure ECC-block sweep — read 32 sectors, fail → skip, no single-sector reads 2026-04-28 21:08:58 -07:00
MattJackson 13eb4331b6 v0.13.36: remove unused reset() 2026-04-28 17:57:25 -07:00
MattJackson 1a3f7ce676 remove unused reset() 2026-04-28 17:56:42 -07:00
MattJackson e1a1f730be v0.13.35: pause 3s after first MEDIUM ERROR before retry to prevent USB bridge crash 2026-04-28 17:54:12 -07:00
MattJackson 646c22ae93 v0.13.34: open() just opens, drive_has_disc() is side-effect-free direct TUR, enumerate_sg_names skips unreadable type files, Disc::copy read_err_count fix 2026-04-28 16:00:00 -07:00
MattJackson 2f6092a686 scsi/linux: fix as_bytes() → as_encoded_bytes() for OsStrExt
- as_bytes() requires OsStrExt import which is platform-specific
- as_encoded_bytes() is the portable API available on all platforms
- Fixes CI failure on Linux (Ubuntu) in GitHub Actions
2026-04-28 14:59:55 -07:00
MattJackson e5a90a6567 disc: fix hysteresis, SgIoTransport recovery, wallclock budget, patch instrumentation
- Fix hysteresis: use_single now correctly forces block_count=1 (was computed before the check)
- Fix SgIoTransport: spawn close+reopen in background thread on transport error
- Fix autorip: rip_disc() spawns wallclock watcher thread, caps entire rip at max(disc_runtime, 1h)
- Fix Disc::patch: add unreadable_count counter + tracing instrumentation
- Simplify Disc::copy() error handling per RIP_DESIGN.md §2.1
2026-04-28 14:48:16 -07:00
MattJackson afce1031d5 disc: add eprintln debug 2026-04-27 23:46:34 -07:00
MattJackson 2c0bff1303 disc: add debug logging for error type 2026-04-27 22:39:03 -07:00
MattJackson 2067ebe485 disc: add warn trace for MEDIUM ERROR skip 2026-04-27 21:38:03 -07:00
MattJackson 2ce4c20221 disc: skip MEDIUM ERROR sectors instead of bailing
0.13.28 - when drive returns MEDIUM ERROR (bad sector), skip the sector
and continue instead of retrying or bailing. Write zero-fill, mark as
Unreadable for pass 2+ recovery.

Closes: (internal)#20260427
2026-04-27 20:44:59 -07:00
MattJackson da3004b120 disc: DiscRead carries SCSI status/sense
Fix extract_scsi_context() and Error::scsi_sense() to handle Error::DiscRead
in addition to Error::ScsiError, so is_marginal_read() works for DiscRead
errors and disc::copy() can properly route MEDIUM ERROR as a marginal
(bad sector) instead of bailing.

Closes: (internal)#20260427
2026-04-27 18:38:57 -07:00
MattJackson b5ed97801b libfreemkv: extend DiscRead with SCSI status/sense for 30% wedge diagnostics 2026-04-27 16:47:27 -07:00
MattJackson ae76aaf0fa v0.13.24 — MapStats: split bytes_pending into nontried / retryable
bytes_pending was an opaque aggregate of NonTried + NonTrimmed +
NonScraped. UIs that wanted a "will retry in Pass 2-N" bucket were
stuck showing the entire unread disc as Maybe at pct=0.

Adds two granular fields to MapStats:

  bytes_nontried   — Pass 1 hasn't read these yet
  bytes_retryable  — NonTrimmed + NonScraped, Pass 2-N will retry

bytes_pending stays for back-compat (= bytes_nontried + bytes_retryable).

Also picks up the cargo fmt --check lint that's been red on main CI
since v0.13.18 (rustfmt fold differences on a few long format-string
layouts; functional no-op).
2026-04-26 19:28:35 -07:00
MattJackson 2cd4fbead7 v0.13.23 — stop discarding the drive's SCSI sense data
Through the entire 0.13.x line, every CHECK CONDITION reply from the
drive (the standard way SCSI tells you why a sector failed) was being
collapsed into a synthetic status=0xFF, sense_key=0 transport-wedge
sentinel and the actual sense data was thrown away. Confirmed live on
the BU40N reading Dune 2 on 2026-04-27: drive returned host_status=0,
driver_status=8, status=2, exec_elapsed_ms=1416 on every bad sector
— a clean CHECK CONDITION carrying full sense data — and Disc::copy
was bailing on it as if the bridge had wedged.

Root cause: scsi/linux.rs's wedge check was
  `host_status != 0 || driver_status != 0`
SG's DRIVER_SENSE bit (0x08) is set on every CHECK CONDITION reply
just to flag "sense buffer is populated" — it's not a transport
failure on its own. Pre-fix we conflated the two and silently lost
every drive-reported error reason. macOS and Windows backends had
the same shape: they extracted sense_key only, dropping ASC/ASCQ.

API restructure (clean separation):

  Error::ScsiError {
      opcode: u8,
      status: u8,                  // 0xFF = synthetic transport-failure
      sense: Option<ScsiSense>,    // None ⇔ no sense delivered
  }

  pub struct ScsiSense { sense_key: u8, asc: u8, ascq: u8 }
  impl ScsiSense {
      pub fn is_marginal(&self) -> bool       // keys 0/1/3/B
      pub fn is_medium_error(&self) -> bool
      pub fn is_hardware_error(&self) -> bool
      pub fn is_unit_attention(&self) -> bool
      pub fn is_data_protect(&self) -> bool
      pub fn is_not_ready(&self) -> bool
      pub fn is_illegal_request(&self) -> bool
      pub fn is_aborted_command(&self) -> bool
  }

  impl Error {
      pub fn scsi_sense(&self) -> Option<&ScsiSense>
      pub fn is_scsi_transport_failure(&self) -> bool
      pub fn is_marginal_read(&self) -> bool
  }

SCSI protocol constants (SCSI_STATUS_*, SENSE_KEY_*) moved from
error.rs to scsi/mod.rs where they belong alongside SCSI_INQUIRY,
SCSI_READ_10, etc. parse_sense replaces parse_sense_key (returns the
full triple, not just the key); inline tests now exercise ASC/ASCQ
extraction at the right offsets for both descriptor (0x72/0x73) and
fixed (0x70/0x71) sense formats.

Disc::copy + Disc::patch sense-aware dispatch:
  - marginal sense (MEDIUM ERROR / ABORTED COMMAND / RECOVERED ERROR
    / NO SENSE) → engage hysteresis (Block→Single, bpt=1)
  - non-marginal sense (HARDWARE / DATA PROTECT / UNIT ATTENTION /
    NOT READY / ILLEGAL REQUEST / transport failure / kernel
    IoError) → bail with full sense info preserved; caller (autorip)
    surfaces "physical replug" / "drive failing" / "media changed"

  Pre-fix: every CHECK CONDITION → 0xFF synthetic → Disc::copy bailed
  → bytes_good froze at the bad zone. The hysteresis from v0.13.22
  was correct but never got to run. This release unblocks it.

  Disc::patch's wedged_threshold (50 consecutive failures) stays as
  defense-in-depth for chains of marginal failures; a single
  non-marginal sense now short-circuits it.

New phase=bail trace event records the bail reason with the sense
triple. phase=transport_err remains for genuine bridge wedges /
kernel timeouts; phase=scsi_err carries the parsed sense_key, asc,
ascq for drive-reported errors.

All 350 tests pass. Clippy clean across all targets.
2026-04-26 19:06:19 -07:00
MattJackson ebffc6eb88 v0.13.22 — replace bisect-on-fail with hysteresis Block↔Single
The v0.13.21 bisect-on-fail recovery was correct (100% of recoverable
sectors picked up) but slow on dense damage clusters. Live test on
Dune 2 v0.13.21 burned ~30 s per damaged 60-block — paying a ~5 s
kernel ABORT/timeout at every level of a log₂(60) ≈ 6 deep DFS, on
the failing branch each time.

Replaced with a two-state hysteresis machine in Disc::copy:

  Block(batch):
    read(batch) ok    → write, advance, stay Block
    read(batch) fail  → switch to Single, retry SAME range at bpt=1

  Single:
    read(1) ok    → write, consecutive_good++
                    if consecutive_good >= BPT1_EXIT_THRESHOLD:
                      switch to Block, reset counter
    read(1) fail  → mark NonTrimmed, consecutive_good = 0

BPT1_EXIT_THRESHOLD = 10_000 sectors (= 20 MB clean run). Calibrated
from the 2026-04-26 BU40N empirical probe data; tunable.

Per-block math on a damaged 60-block with 1 truly bad sector:

  Bisect      (v0.13.21): ~30 s  (5 s × 6 levels)
  Hysteresis  (v0.13.22): ~10 s  (5 s bpt=batch fail
                                  + 59 × 1 ms good
                                  + 1 × 5 s bad)

Inside a damaged cluster spanning many 60-blocks the win compounds:
hysteresis pays the bpt=batch fail cost ONCE on entry, then stays at
bpt=1 across the cluster; bisection re-paid it every 60 sectors. For
Dune 2's ~1248-sector boundary cluster that's ~21 fewer 5-sec
kernel timeouts ≈ 100 s saved per pass.

Telemetry: new phase=mode_change trace event with from, to, lba, and
consecutive_good. Replaces v0.13.21's phase=bisect. Worklist DFS is
gone — single iterative for s in 0..count on the failure path.

Test rename, same fixture and same 100% recovery expectation:
  test_disc_copy_bisect_recovers_via_single_sector_reads
  → test_disc_copy_hysteresis_recovers_via_single_sector_reads

Also adds DamageSeverity (Clean / Cosmetic / Moderate / Serious) +
classify_damage(bad_sectors, lost_ms), re-exported from libfreemkv,
so applications can render structured severity instead of formatting
their own from raw counters.
2026-04-26 17:27:57 -07:00
MattJackson 424d3cd4f2 v0.13.21 — bisect-on-fail in Disc::copy + 10s caller READ timeout
Fixes the BU40N wedge cycle that has been chasing us through
v0.13.18-20. Two changes, both backed by empirical live-hardware
probes recorded in (internal)/docs/TEST_PLAN.md:

1. scsi/mod.rs: READ_TIMEOUT_MS 1500 → 10000 ms.
   Cold-start seek on the BU40N takes ~1.5 s. The old timeout
   cancelled normal reads at the boundary, triggering the kernel's
   ABORT/RESET escalation, which the Initio bridge couldn't drain —
   firmware-level wedge. 10 s catches every legitimate slow read
   (max successful ECC recovery: 2.6 s; cold-start: 1.5 s) with
   margin and short-circuits truly bad sectors at ~10 s.

2. disc/mod.rs: Disc::copy bisect-on-fail (replaces skip-forward).
   Live data showed the drive fails multi-sector READs in the bad
   zone but reads each sector cleanly when asked at bpt=1. Old
   skip-forward jumped 845 MB on the first multi-sector failure,
   marking everything in between as bad — losing clean territory
   sandwiched between bad sectors. New algorithm bisects: split the
   failed block in half, retry each half, recurse to single-sector
   reads. Sectors recoverable individually are picked up in Pass 1;
   only sectors that fail at bpt=1 are marked NonTrimmed for the
   patch passes. Stack-based DFS, log2(batch) = 6 levels for the
   default 60-sector batch.

Multi-pass machinery is untouched. Pass 2..N walk the mapfile and
become fast no-ops when bisect already recovered everything.
Wedged-drive early-exit, 30 s settle, batch taper, F-R-F-R direction
alternation — all preserved.

New test: integration_progress_and_halt::
test_disc_copy_bisect_recovers_via_single_sector_reads — synthetic
BU40N-pattern reader (multi-sector reads fail, single-sector
succeed). Pre-patch: lost everything to skip-forward. Post-patch:
100 % bytes_good. Plus the 10 sense-key parser tests from the
0.13.20 test-coverage pass.

Empirical recovery on Dune 2 UHD on the BU40N (per TEST_PLAN.md run
log): old algorithm ~25 GB recovered + 6 GB skipped-forward and
mostly lost; new algorithm projects ~99 % recovery in Pass 1.

Audits + raw probe data:
- (internal)/docs/TEST_PLAN.md (run log)
- (internal)/docs/audits/2026-04-26-scsi-architecture-research.md
2026-04-26 15:57:44 -07:00
MattJackson d2905ba7bb v0.13.20 — sync blocking SG_IO + cross-platform parity strip
- scsi/linux.rs: full rewrite from async write/poll/read+1.5s timeout+
  close-on-timeout to one synchronous ioctl(fd, SG_IO, &hdr). Kernel
  honors hdr.timeout and runs its own ABORT/RESET escalation. Errors
  check host_status and driver_status (both 0xFF-synthesised) plus
  status. Sense-key parser handles descriptor (0x72/0x73) + fixed
  (0x70/0x71) formats. Deleted fd_recovery, bg close+open thread, fd
  swap dance. -331/+155 lines.

- scsi/macos.rs: try_recover() removed (userspace handle-recovery on
  task failure was the same anti-pattern stripped from Linux). bsd_name
  field deleted. Errors bubble up directly.

- scsi/windows.rs: try_recover() removed, wide_path field deleted,
  INVALID_HANDLE guard removed.

- scsi/mod.rs: parse_sense_key() helper extracted (used by all three
  platforms now — single canonical sense-key parse rather than three
  inlined copies). +10 unit tests covering descriptor format, fixed
  format, truncated buffers, unknown response codes.

- drive/mod.rs: Drive::reset() deleted (escalating eject + STOP/START +
  reinit recovery — per audit, kernel handles its own escalation;
  userspace shouldn't).
  pub fn find_drives() -> Vec<Drive> deleted (opened N drives just to
  throw most away). find_drive() now uses discover_drives() directly.
  wait_ready() simplified — drops the reset path on sense_key=5,
  just keeps polling TUR for 60 iterations.

- lib.rs: find_drives re-export removed.

- benches/sgio_read.rs: switched to find_drive() (no longer iterates a
  drive list).

Net: 9 files changed, 226 insertions(+), 473 deletions(-). 329 tests
pass, clippy -D warnings clean. No consumer breakage (CLI, autorip,
bdemu compile + test green).

Architecture decision documented in
(internal)/docs/audits/2026-04-26-scsi-architecture-research.md
(primary-source survey of MakeMKV, sg_dd, ddrescue, and the kernel
mid-layer's own scsi_eh.rst escalation ladder).
2026-04-26 09:51:46 -07:00
MattJackson b4de5d343d v0.13.18 — sync release (no functional changes; autorip two-bar UI fix) 2026-04-26 07:47:37 -07:00
MattJackson 8beeac7df9 v0.13.17 — sync release (no functional changes; actual fix is autorip hot-plug) 2026-04-26 07:27:07 -07:00
MattJackson b33f41e219 v0.13.16 — single Progress trait + PassProgress (RIP_DESIGN.md §16)
Pre-0.13.16 the rip API leaked internal mapfile concepts (pos,
bytes_good, work_done, bytes_pending, Finished/NonTrimmed) into per-pass
positional callbacks Fn(u64, u64, u64). Consumers reinvented the math
each time, and v0.13.15's UI bug surfaced exactly because of this —
autorip's web JS computed pct from bytes_good while the backend
computed from pos, silent drift, frozen UI bar.

This release replaces both Disc::copy::on_progress and
Disc::patch::on_progress callbacks with a single Progress trait +
PassProgress struct (new progress module).

  pub struct PassProgress {
      pub kind: PassKind,            // Sweep | Trim {reverse} | Scrape {reverse} | Mux
      pub work_done: u64,
      pub work_total: u64,
      pub bytes_good_total: u64,
      pub bytes_total_disc: u64,
  }

  pub trait Progress {
      fn report(&self, p: &PassProgress);
  }

  impl<F: Fn(&PassProgress)> Progress for F { ... }   // closures work directly

CopyOptions::on_progress and PatchOptions::on_progress are renamed to
progress: Option<&dyn Progress>. Closure callers update trivially via
the blanket impl; struct callers gain a clean named-field shape with no
positional-arg confusion.

PassKind carries the semantic (sweep vs trim vs scrape vs mux) so
consumers can label phases without reinventing detection logic.
Disc::patch reports Trim {reverse} for retry passes with block_sectors
>= 2 and Scrape {reverse} when block_sectors == 1. Direction comes
through reverse: bool. Mux variant is reserved for v0.13.17 when the
mux pipeline emits progress.

Tests + clippy clean across all 4 crates.
2026-04-26 07:15:26 -07:00
MattJackson e85e20f436 v0.13.15 — pos in on_progress, PatchOptions::reverse, wedged_threshold
Breaking: CopyOptions::on_progress + PatchOptions::on_progress now take
Fn(bytes_good, pos, total). Consumers display `pos` for "% swept" — the
true Pass 1 progress that advances through skip-forward bad zones, where
bytes_good (Finished sectors only) freezes. v0.13.14 live trace proved
the existing UI was lying for ~14 minutes about Dune 2 being "stuck at
30%" while Pass 1 was actually 83% through the disc via skip-forward.

PatchOptions::reverse: walk bad ranges from highest LBA to lowest. For
drives that wedge after a forward read of a bad sector, approaching the
post-bad-zone NonTrimmed range from end-of-disc reads good sectors before
the drive sees a bad one. Hypothesis informed by the BU40N + Initio
bridge live data — Pass 2 forward saw zero successful reads in 7 min
while Pass 1's pos walked all the way to end-of-disc.

PatchOptions::wedged_threshold: > 0 → exit early after that many
consecutive failures with zero successes in the same pass. Saves the
wallclock budget for productive grinding when the drive has wedged on
the bad zone for THIS pass; a different direction or block size in the
next pass may still recover. New PatchResult::wedged_exit reports it.

Trace: patch_start (block_sectors, recovery, reverse, wedged_threshold,
num_ranges) and patch_done (blocks_attempted, blocks_read_ok,
blocks_read_failed, wedged_exit, halted, bytes_recovered) at the
freemkv::disc target.
2026-04-25 20:06:03 -07:00
MattJackson 7bc54be8d3 v0.13.14 — sync release, no functional changes (autorip subscriber filter fix in autorip 0.13.14) 2026-04-25 18:38:17 -07:00
MattJackson 6d9083743b v0.13.13 — telemetry: tracing instrumentation in SgIoTransport + Disc::copy
v0.13.12 shipped the async fd_recovery design but a live test on Dune 2
showed Pass 1 sat for 14 minutes with bytes_good=0 — the inner loop iterates
(throttled on_progress log fires every 78s) but each iteration evidently
takes ~60s instead of the microseconds the design promises on fast-fail.
Without trace-level telemetry at the SCSI + Disc::copy boundaries we
can't tell where the time goes.

This release is instrumentation only — no behavior change.

- New dep: tracing 0.1. Per project docs, debug/trace logging is allowed in
  libfreemkv (the no-English rule applies to errors). Consumers wire a
  tracing subscriber.
- SgIoTransport::execute (Linux): trace at every state transition (entry,
  recovery_swap_ok, recovery_pending, write_ok / write_err, poll_done,
  timeout_spawn_recovery, scsi_err, read_err, ok). Each event includes
  opcode + elapsed timing. The bg recovery thread also traces close_ms +
  open_ms so we can see if the kernel really takes 60s to close+open on a
  wedged Initio bridge.
- Disc::copy: trace at copy_start, outer_loop, region_enter, every 100
  inner-loop iterations (iter_progress with pos/region_end/skip_size/
  bytes_good/read_ok_count/read_err_count/last_read_ms/copy_elapsed_ms),
  copy_done.
- All trace events use targets `freemkv::scsi` and `freemkv::disc` so
  consumers can filter by subsystem (e.g. autorip /api/debug?q=freemkv::scsi).

Next: run the live test on Dune 2 again, read the autorip JSONL log,
diagnose why each iter is slow, fix the actual bug.
2026-04-25 18:32:05 -07:00
MattJackson c49a68054f style: cargo fmt on integration test 2026-04-25 17:36:06 -07:00
MattJackson 4bd38787c6 test(integration): make halt-on-skip-forward test deterministic
The wallclock-based halt timing failed on fast CI runners where a 2 GB
synthetic-disc skip-forward sweep finishes in <100 ms — well under the
200 ms halt fire delay. Reader now signals halt on first read; the
inner-loop halt check on iteration 2 breaks 'outer. No wallclock race.
2026-04-25 17:33:52 -07:00
MattJackson b4951b1c5b v0.13.12 — Fix 1+2+4 + cross-platform SCSI parity (RIP_DESIGN.md §6, §7, §15.1)
Fix 1: delete stall guard from Disc::copy. Pass 1 must sweep end-to-end
per ddrescue model (RIP_DESIGN.md §2.1, §3, §9). The v0.13.9 guard at
disc/mod.rs broke Pass 1 at 30% on Dune 2 with 56 GB still NonTried.
Removed stall_secs field, narrative comment in scsi/linux.rs, and the
broken regression test. Replaced with test_disc_copy_completes_full_disc_
with_failing_reader and test_disc_copy_halts_promptly_on_failing_reader.

Fix 2: async SCSI transport recovery. Added Arc<AtomicI32> fd_recovery
on SgIoTransport. On poll timeout: spawn close + spawn open in
background, return Err immediately. Top of execute() swaps fd from
recovery atomic. Main thread never blocked beyond ~1.5s poll budget
(was up to ~60s per timeout because kernel serialized main-thread
open() against in-flight close()). Drop drains pending recovery fd.

§15.1 cross-platform parity: Windows + macOS now have the same
observable recovery contract. SptiTransport gets try_recover()
(synchronous CloseHandle + CreateFileW; Windows close is fast, no
in-flight CDB drain like Linux). MacScsiTransport gets try_recover()
(release IOKit interface + reacquire via new acquire_device_iface()
helper); stores bsd_name for re-resolution. Drop guards null'd-out
interfaces. Stripped English error strings ("try as root" / "run as
administrator") on Linux + Windows. Fixed Windows TimeOutValue
ms→s ceiling so 1500ms gets 2s (was 1s; broke Drive::read fast path).

Fix 4: instrument Disc::patch arms. PatchResult exposes
blocks_attempted, blocks_read_ok, blocks_read_failed so the v0.13.11
mystery (Dune 2 Pass 2 recovered 0 bytes in 100 min) is diagnosable
from the live device log without re-instrumenting from outside.

Cleanup: honor PatchOptions::full_recovery (was read into _ and
ignored; now routed to read_sectors recovery arg). Updated
CopyOptions::batch_sectors doc to describe the actual production
path (sysfs detect_max_batch_sectors, typically 60 sectors / ~120 KB
on BU40N) rather than the test-only 32-sector internal default.

All four crates clippy-clean and tests green on the host targets
(macOS native + cargo check on Linux). Cross-platform CI watches
Linux + Windows + macOS builds + tests.
2026-04-25 17:30:25 -07:00
MattJackson 92dc145864 v0.13.11: revert SgIoTransport timeout path — keep transport alive
v0.13.10's 'fd=-1 on first poll timeout' was too aggressive: a single
transient killed the entire transport, Pass 1 finished in 45ms with
0 GB good on Dune 2.

Revert to spawn-close + main-thread-reopen (the v0.13.5/8 pattern).
Per-timeout cost is up to ~60s while the kernel completes the
abandoned command, but the v0.13.9 Disc::copy stall guard caps
catastrophic stalls at 120s of bytes_good non-advance. Pass 1 bails
cleanly with NonTrimmed ranges; Pass 2 has a working Drive for
retries with recovery=true + 30s timeouts.
2026-04-25 08:47:10 -07:00
MattJackson 36b65f526a v0.13.10: version sync 2026-04-25 08:25:08 -07:00
MattJackson ca8ebf418f v0.13.9: Disc::copy stall guard + SgIoTransport no-reopen-on-timeout
Fixes the silent Pass 1 hang observed on Dune 2 with v0.13.8 (drive
grinding through bad sectors at 0 KB/s, errs=0, no error surfaced).

Root cause: SgIoTransport::execute's reopen-after-poll-timeout opened
a fresh /dev/sg* fd on the main thread, which serialized against the
spawned close() of the old fd via the kernel's per-device state lock.
The userspace 1.5s timeout still fired, but the abandon-and-reopen
recovery itself blocked the main thread for as long as close() took.
Net: reads returned slowly, skip-forward fired on every iteration,
bytes_good never advanced.

- SgIoTransport::execute: on poll timeout, spawn close, set fd=-1,
  return Err. No reopen on the main thread. The transport is now
  invalidated until the consumer creates a fresh Drive.
- Disc::copy: add stall guard. CopyOptions.stall_secs (default 120s).
  If bytes_good doesn't advance for the threshold, break 'outer
  cleanly with complete=false, bytes_pending > 0 so Pass 2 retries
  pick up the NonTrimmed ranges with recovery=true 30s timeouts.
- New regression test: test_disc_copy_stall_detection_triggers_
  skip_forward in tests/integration_progress_and_halt.rs.
2026-04-25 08:12:01 -07:00
MattJackson 34182d956a v0.13.8: version sync 2026-04-25 07:11:47 -07:00
MattJackson 0f967a2084 v0.13.7: version sync 2026-04-25 06:58:43 -07:00
MattJackson 0341995c0d fix: allow dead_code on Drive::emit (DiscStream owns BytesRead emission post-0.13.6 strip) 2026-04-24 21:43:30 -07:00
MattJackson fe0ec0c5cf fix: drop unused INQUIRY_* constants (clippy -D warnings) 2026-04-24 21:42:30 -07:00
MattJackson 022657d481 docs: complete 0.13.6 docs sweep (architecture, api-design, disc-to-rip, README)
Follow-up to the rip-recovery + drive-access updates: aligns the
remaining docs with the v0.13.6 single-shot read model and the
three-layer recovery architecture.

- architecture.md: module map says single-shot read; new paragraph on
  layered recovery with postmortem pointer.
- api-design.md: EventKind enum example expanded; emission notes
  document that BytesRead now fires from DiscStream::fill_extents and
  Retry/SectorRecovered are no longer emitted in 0.13.6+.
- disc-to-rip.md: Step 10 of the pipeline diagram + module table
  reflect single-shot read.
- docs/README.md: added rip-recovery.md to the TOC.
2026-04-24 21:36:49 -07:00
MattJackson fc8eca44e1 docs: rewrite rip-recovery + drive-access for 0.13.6 single-shot model
Updates docs/ to reflect the recovery-loop strip:
- rip-recovery.md: drops Phase 1/2/3 description, replaces with three-layer
  model (Disc::patch multi-pass / DiscStream batch halving / Drive::read
  single-shot). Notes that no SCSI resets fire from any retry path.
- drive-access.md: removes SG_SCSI_RESET + STOP/START UNIT escalation
  references; SgIoTransport::reset is now kernel SG_IO flush + ALLOW
  MEDIUM REMOVAL only.
- src/mux/disc.rs + tests/: cargo fmt cleanup.
2026-04-24 21:35:33 -07:00
MattJackson 43836865be v0.13.6: strip Drive::read inline recovery + reset escalation; emit BytesRead
Drive::read is now single-shot. Phase 1/2/3 retries + scsi::reset+reopen
removed (~80 lines). recovery=true bumps timeout to 30s; recovery=false
stays at 1.5s. On any failure returns Err(DiscRead) immediately — caller
(Disc::patch outer loop, DiscStream batch halver) handles retries.

Inline reset+reopen WAS the wedge primitive on the LG BU40N. Per prior
post-mortem, every USB/SCSI reset path tested fails to recover the
wedged Initio bridge — the inline retry was pure cost.

SgIoTransport::reset (Linux) trimmed to kernel SG_IO state flush +
ALLOW MEDIUM REMOVAL. SG_SCSI_RESET ioctl + STOP/START UNIT escalation
removed. macOS reset removed (no-op). scsi::reset() top-level family
removed (no callers).

EventKind::BytesRead { bytes, total } now actually emitted from
DiscStream::fill_extents after each successful sector read. Was
declared in 0.13.0, never fired. Drives autorip per-device progress
in direct mode.

EventKind::Retry / SectorRecovered no longer emitted (variants kept
for forward compat). SpeedChange still emitted via Drive::set_speed
public path.

Tests: new tests/integration_progress_and_halt.rs (5 tests). 233 unit
tests + 5 integration green.
2026-04-24 21:32:45 -07:00
MattJackson 5e22199441 v0.13.5: version sync (autorip-side fixes) 2026-04-24 20:13:07 -07:00
MattJackson 761b77bbd9 v0.13.4: roll back wedge recovery + add sysfs identity fallback
USB/SCSI recovery escalation in drive_has_disc (0.13.1-0.13.3) tested
on LG BU40N USB BD-RE: USBDEVFS_RESET, authorized toggle, driver
unbind/rebind, SCSI host rescan — all succeed at the USB transport
layer but the drive firmware below the bridge stays locked. Only
physical unplug-replug clears it. Rolled back so consumers can
surface the real failure to the user.

New: list_drives falls back to sysfs-cached vendor/model/rev from
/sys/class/scsi_generic/sgN/device/ when live INQUIRY returns empty,
so wedged drives still show their identity in UIs.

Removed: scsi::usb_reset, usb_reset_with_timeout, per-platform
usb_reset methods, recover_then_probe, is_wedge_signature. Breadcrumb
comment in scsi/linux.rs::drive_has_disc points at v0.13.3 tag for
the full implementation if future hardware needs it back.

Linux/macOS/Windows pass-through symmetric; 233 tests passing.
2026-04-24 19:53:44 -07:00
MattJackson cc05ef2a3a v0.13.3: broaden is_wedge_signature — fix dead-code wedge recovery
0.13.2's is_wedge_signature gated on opcode=SCSI_INQUIRY (0x12), but
drive_has_disc issues TEST UNIT READY (0x00). Production wedge errors
(E4000: 0x00/0xff/0x00) never matched → SCSI reset + USB reset
escalation never fired.

Drop the opcode gate. Status byte 0xFF is synthesised by our own
execute() path on poll() timeout — it's the ground-truth wedge
marker for any opcode.

Linux-only; macOS/Windows use sense-key-based wedge detection.
2026-04-24 19:26:11 -07:00
MattJackson e27b82ce5b v0.13.2: list_drives + drive_has_disc; SCSI primitives pub(crate)
Architectural cleanup. autorip + freemkv CLI were reimplementing drive
discovery (sysfs walking, type-5 filtering, sg-path construction) and
calling SCSI reset primitives directly. All of that hardware-aware code
moves into libfreemkv with two cheap public probes:

- DriveInfo + list_drives() — multi-OS enumeration (Linux/macOS/Windows)
  with peripheral-type-5 filtering and INQUIRY identity. Cheap.
- drive_has_disc(path) — single TUR with internal wedge recovery
  escalation (SCSI reset → USB reset → retry) hidden from callers.

USB-layer reset (USBDEVFS_RESET / IOUSBDeviceInterface::ResetDevice /
storport's combined reset) wired across all three platforms.

Visibility tightening — scsi::reset, scsi::usb_reset, and the timeout
constants are now pub(crate). Compile-time guarantee that no consumer
crate can issue SCSI commands directly.

233 lib tests pass; clippy clean.
2026-04-24 17:31:15 -07:00
MattJackson 8af47e4c19 v0.13.1: scsi::reset() bounded by wallclock timeout
Production incident: autorip's poll loop called scsi::reset() on a
wedged BU40N USB drive. The Linux SG_SCSI_RESET ioctl blocked
indefinitely (kernel SCSI subsystem waiting for a bus-wedged device to
ack a reset that will never come). Caller's poll loop hung for 60+
seconds before manual intervention.

scsi::reset() now spawns a detached worker for the platform-specific
reset and bounds the caller's wait via mpsc::recv_timeout. Default
30 s (DEFAULT_RESET_TIMEOUT_SECS); reset_with_timeout(device, dur)
exposes the bound for callers that want a different value. Returns
DeviceResetFailed on timeout. Worker thread keeps running until the
kernel eventually unblocks — leaks one OS thread per hard wedge, but
the daemon stays responsive instead of hanging forever.

Follow-up flagged for 0.13.2: USB-attached drives wedge at the USB
Mass Storage layer below SCSI; SG_SCSI_RESET doesn't help. A
scsi::usb_reset(path) using USBDEVFS_RESET is the proper escalation.
2026-04-24 16:58:13 -07:00
MattJackson d1f09439a5 v0.13.0: zero English in library + API hygiene + dead-code sweep
Audit pass against the project docs "no English text in library code" rule.
Found 9 call sites that violated the contract by stuffing English into
io::Error::new(kind, "…") or by abusing Error::DeviceNotFound { path }
as a free-form description field. Each is now a typed Error variant.

New variants and codes: ScsiInterfaceUnavailable (E1004), DeviceLocked
(E1005), IoKitPluginFailed (E1006), UnsupportedPlatform (E2003),
PlatformNotImplemented (E2004), MapfileInvalid (E6011), DiscUrlNotDirect
(E9009).

labels::apply() previously pushed Commentary/Descriptive/Score/IME and
" (Secondary)" English literals into AudioStream.label, leaking into
MKV titles + autorip UI. AudioStream now exposes structured `purpose:
LabelPurpose`, SubtitleStream `qualifier: LabelQualifier`. Callers
translate to localized text. label keeps codec-formatting only.

API hygiene: 11 mux/* modules dropped from `pub` to `pub(crate)` —
their *types* are still re-exported from lib.rs, but the modules were
leaking low-level EBML/TS/network primitives. Stream trait gets a real
rustdoc explaining read-vs-write split. lib.rs grouped re-exports into
documented sections. ScanOptions::with_keydb() removed (one-method-per-
action rule); use struct literal.

Dead-code sweep: removed lookahead.rs (orphan, never declared as mod),
tsreader.rs (TsDemuxReader unused), ebml::{write_int,read_vint,SEEK_*},
ts::{scan_first/last_pts,scan_duration,SCAN_HEAD/TAIL_SIZE,take/set_
remainder}, MkvMuxer codec_private_slots/filled fields and
fill_codec_private method (deferred-codecPrivate path never used since
the v0.10 PES rewrite). cargo clippy --all-targets -D warnings clean.

Tests: new error::tests for variant codes + Display "no English" guard +
io::ErrorKind mapping. 233 lib tests, all green (was 230).

Breaking: ScanOptions::with_keydb removed; mux/* modules pub(crate);
AudioStream and SubtitleStream gained required fields; UnsupportedDrive
{ product_revision: "Renesas not yet implemented" } no longer produced
(use PlatformNotImplemented).
2026-04-24 16:41:02 -07:00
MattJackson 37e721ee7e v0.12.2: hide halt behind checked_sleep / checked_exec primitives
Drive::read is now halt-check-free in its body. Previously, the halt
flag was checked in 4 places and the sleep logic was scattered across
4 "if halt_aware_sleep { return Halted }" call sites — correct, but
the ugliness invited drift: a new sleep added by someone unfamiliar
with the pattern would silently swallow Stop requests.

Two private primitives now own halt awareness:

  checked_sleep(Duration) -> Result<()>
  checked_exec(cdb, dir, buf, timeout_ms) -> Result<ScsiResult>

Both return Err(Halted) instead of a bool. The ? operator in read()
then propagates halts for free. The recovery path reads top-to-bottom
with no halt vocabulary.

sleep_until_halted lives as a free function so it's unit-testable
without a live Drive. 4 new tests: completes normally, bails on
pre-set flag within one slice, wakes mid-sleep, zero-duration no-op.

Public API unchanged — halt_flag/halt/clear_halt still exposed, the
refactor is entirely internal.
2026-04-24 13:12:24 -07:00
MattJackson 481b47d90d v0.12.1: halt-aware sleeps in Drive::read recovery
Stop was waiting up to 30 s to register when the drive hit L-EC
recovery mid-read — the recovery phase does 30 s sleeps between
retries and the halt flag was only checked at the start of each sleep.
UI feels broken ("stop isnt working") even though the halt was set.

halt_aware_sleep breaks each wait into 100 ms slices and returns
early on halt. Applied to all 4 sleeps in the recovery path (both
30 s retry delays, both 5 s reset-phase delays).
2026-04-24 12:59:36 -07:00
MattJackson dd7cbd70a9 v0.12.1: cut non-recovery read timeout 5s → 1500ms
Disc::copy fast pass (skip_on_error=true, recovery=false) was giving
the drive 5 s per 64 KB block. On structure-protected / marginal UHD
sectors the drive grinds L-EC for nearly the full budget per block,
pinning throughput at ~13 KB/s even though skip_forward would happily
skip past the region.

1500ms bounds the floor at ~43 KB/s/block. Recoverable sectors that
would have succeeded at 3-5 s get picked up on Disc::patch (pass 2+)
where recovery=true and the per-read budget is 30 s.
2026-04-24 12:57:53 -07:00
MattJackson 3dea679dac style: cargo fmt 2026-04-24 12:23:42 -07:00
MattJackson 64ecca0d12 fix(windows): Rust 2024 requires unsafe extern blocks
Missed in the prior 2024 edition sweep because local builds don't
exercise the cfg(target_os = "windows") path. check-windows CI caught it.
2026-04-24 12:21:43 -07:00
MattJackson 84f6baba38 v0.12.0: Rust 2024 edition migration
- edition = "2024" bump.
- FFI block in src/scsi/macos.rs wrapped in `unsafe extern "C" { }`.
- vtable_fn body gets an explicit unsafe block (unsafe_op_in_unsafe_fn).
- Match-ergonomics cleanup in mux/meta.rs, mkvstream.rs, network.rs,
  stdio.rs — removed redundant `ref` / `ref mut` bindings.

MSRV unchanged at 1.86. 226 tests pass. No behavior change.
2026-04-24 12:07:05 -07:00
MattJackson bd5b7795bd docs: multi-pass recovery — README snippet + new rip-recovery.md
- README quick-start gains a multi-pass example using the new
  Disc::copy + Disc::patch primitives.
- New docs/rip-recovery.md documents the two-stage rip model: mapfile
  format (ddrescue-compatible), CopyOptions/PatchOptions surface, the
  pass-1/pass-2 algorithm, and the design decisions (why no MODE
  SELECT, why ISO intermediate, why ddrescue mapfile).

No code change.
2026-04-24 11:53:28 -07:00
MattJackson a57506c1ce v0.11.22: version sync — no functional changes
autorip 0.11.22 ships the full multi-pass UI (live mapfile stats,
bad-range viz, Recovery settings). libfreemkv API is unchanged from
0.11.21. Part of the 0.11.22 ecosystem sync.
2026-04-24 11:43:32 -07:00
MattJackson c33f3e9557 v0.11.21: multi-pass rip — Disc::copy + Disc::patch + mapfile module
New primitives for two-stage rip workflows: fast forward pass with
zero-fill on failures, then targeted retries of bad ranges via a
ddrescue-compatible mapfile.

- Disc::copy now takes &CopyOptions (breaking change from positional
  args). Always writes a sidecar .mapfile. Opt-in skip_on_error +
  skip_forward give ddrescue-style fast sweep: 64 KB blocks,
  exponential skip-forward on failure, zero-fill bad blocks. Defaults
  preserve pre-0.11.21 behavior (recovery reads, abort on bad sector).

- Disc::patch is new and idempotent. Reads the mapfile, re-reads every
  non-finished range with full drive recovery, patches good bytes back
  into the ISO at exact offsets. Call N times for N retry attempts.

- disc::mapfile is a new module. ddrescue text format, crash-safe
  (flushed on every record()), greppable, human-editable, tool-compatible.
  Status chars match ddrescue: ? / * / / / - / +.

- Re-exports FileSectorReader from the crate root.

- freemkv CLI caller (pipe.rs) updated to the new Disc::copy signature
  in lockstep — shipped in the 0.11.21 freemkv CLI release.

Part of the 0.11.21 ecosystem sync (libfreemkv + freemkv + bdemu +
autorip all on 0.11.21).
2026-04-24 09:24:32 -07:00
MattJackson 166fc4bf8c v0.11.18: DiscStream halt flag — Stop works in dense bad-sector regions
DiscStream::fill_extents loops internally while the demuxer waits for
enough clean data to emit a PES frame. In a dense bad zone that loop
can run for minutes without returning to the outer read() call, so
the caller's Stop signal never gets serviced until a frame is finally
emitted — which may be very far away.

Add DiscStream::set_halt(Arc<AtomicBool>) — typically wired to
Drive::halt_flag() for unified Stop across drive recovery phases and
stream sector processing. fill_extents checks the flag at the top of
every retry iteration; raising it returns Err(Error::Halted) within
one SCSI round-trip.

No behavior change for callers that don't call set_halt. Unblocks the
architectural fix for the "Stop doesn't stop" bug observed on a
damaged UHD disc.
2026-04-24 07:41:13 -07:00
MattJackson 9aaddaa9b8 v0.11.17: adaptive batch sizer — no per-sector descent
Replace read_with_binary_search + 3×5s light recovery with an adaptive
sizer that shrinks on failure (halve, 3-aligned ≥6) and probes back up
after 100 MiB (51,200 sectors) of clean reads. Descent cost is paid
once per bad region, not once per bad sector.

Emit BatchSizeChanged { new_size, reason } on shrink and probe-up.
Remove BinarySearch event — no longer produced.

Side fix: scsi/macos.rs one-liner for manual_c_str_literals clippy
lint that surfaced on a newer toolchain.
2026-04-23 20:24:40 -07:00
Matt Jackson 40ec1e1eba v0.11.16: API cleanup — one method per action 2026-04-21 19:17:50 +00:00
Matt Jackson 8843833ea7 v0.11.15: lint cleanup — fmt + clippy clean 2026-04-21 18:52:55 +00:00
Matt Jackson afae37c8c3 fix: iso_dump example missing recovery arg 2026-04-21 18:41:41 +00:00
Matt Jackson d8089ec491 v0.11.14: audit fixes — trailing sectors, verify stop, SCSI sense, O_CLOEXEC
Fix trailing sectors dropped at extent boundaries when sector_count % 3 != 0.
Add verify_title stop support via progress callback returning bool.
Add O_CLOEXEC on all SCSI fd opens to prevent leak to child processes.
Fix SCSI sense descriptor format detection (0x72/0x73 vs 0x70/0x71).
2026-04-21 18:40:04 +00:00
Matt Jackson ce8132d181 Update docs: async sg transport, Drive::read recovery phases 2026-04-21 18:01:48 +00:00
Matt Jackson 71e1f57364 Async SG_IO: enforceable timeouts via write/poll/read
Replace blocking ioctl(SG_IO) with the sg driver's async interface.
Commands are submitted via write(), waited on via poll() with a hard
wall-clock timeout, and completed via read(). If poll() times out,
the fd is abandoned and a fresh one opened — the kernel can no longer
hold us hostage during USB error recovery.

- write() submits command, returns immediately
- poll() enforces exact timeout (EINTR-safe with deadline tracking)
- read() retrieves result + copies data to caller's buffer
- On timeout: old fd closed in background thread, new fd opened
- No SG_FLAG_DIRECT_IO — kernel buffers for safe timeout abandonment
- Store device_path for fd reopen after timeout
- Drop guards fd=-1 (abandoned fd)
2026-04-21 17:57:58 +00:00
Matt Jackson d8092ee093 Clean API: merge read() and read_fast() into read(recovery: bool) 2026-04-21 03:27:17 +00:00
Matt Jackson dd49d55b11 v0.11.13: all rip reads use fast timeout, no full recovery in read path 2026-04-21 02:16:49 +00:00
Matt Jackson 6ba958fe8a Fix: initial batch read uses fast read, not full recovery 2026-04-21 02:02:13 +00:00
Matt Jackson f30b958408 v0.11.12: halt, events, light recovery 2026-04-21 00:25:44 +00:00
Matt Jackson 6971f00be2 Drive halt flag, sector events, binary search light recovery (3x5s) 2026-04-21 00:18:46 +00:00
Matt Jackson ad86f92d85 v0.11.11: binary search error recovery in fill_extents 2026-04-20 21:51:46 +00:00
Matt Jackson cce1be29b2 v0.11.10: skip_errors, read_sectors_recover API 2026-04-20 15:27:56 +00:00
Matt Jackson 081ad4f619 DiscStream skip_errors, read_sectors_recover(recovery) API 2026-04-20 03:38:32 +00:00
Matt Jackson 43ecad2d43 Clean API: read_sectors_recover(recovery: bool) replaces read_sectors_fast 2026-04-20 01:12:59 +00:00
Matt Jackson 8870efa77b v0.11.9: fast verify reads — 5s timeout, no recovery loop 2026-04-20 01:01:44 +00:00
Matt Jackson a693ef965b v0.11.8: disc verify module 2026-04-20 00:03:26 +00:00
Matt Jackson 0596307c19 Add verify module: sector-by-sector disc health check 2026-04-20 00:01:02 +00:00
Matt Jackson 18c6365e08 Improve label generator: proper video labels, all resolutions, no false Atmos claims 2026-04-19 17:50:17 +00:00
Matt Jackson 981f30b1b0 Move label generation to labels system — fill_defaults() for all stream types 2026-04-19 17:23:50 +00:00
Matt Jackson 04086d73a0 Generate audio track labels when disc doesn't provide them 2026-04-19 16:34:35 +00:00
Matt Jackson 20e404e59a README: fix dep version 0.10 → 0.11, update share command 2026-04-19 04:57:55 +00:00
Matt Jackson 1bea8eb650 v0.11.7: TrueHD parser rewrite — 12-bit length, AC-3 skip, cross-PES buffering 2026-04-19 01:36:56 +00:00
Matt Jackson 0165f18fad v0.11.6: fix TrueHD BD-TS header corruption 2026-04-18 19:27:12 +00:00
Matt Jackson ca3914d537 Fix TrueHD: strip BD-TS access unit header before muxing to MKV 2026-04-18 19:26:14 +00:00
Matt Jackson 80fcffd190 v0.11.5: MKV container fixes — timestamps, frame rate, HDR, chapters, disposition 2026-04-18 16:29:21 +00:00
Matt Jackson 7437af39c8 v0.11.3: unified versioning across all repos 2026-04-18 15:01:53 +00:00
Matt Jackson bcd6925170 v0.10.10: fix dual-layer disc rips, propagate read errors
Bump version, update CHANGELOG and FEATURES for 0.10.8-0.10.10.
2026-04-18 02:10:20 +00:00
Matt Jackson fba1eb189c Fix truncated rips on dual-layer discs, propagate read errors
Use UDF file_extents() to read actual allocation descriptors instead
of assuming m2ts files are contiguous from file_start_lba. Dual-layer
UHD discs split large files across 70+ extents (~1 GB each) — the old
code created one extent from packet count which only covered the first
chunk, causing silent truncation at ~37%.

Also changed fill_extents() to return io::Result<bool> so read errors
propagate instead of being silently treated as EOF.
2026-04-18 02:08:34 +00:00
Matt Jackson a328ad3cd8 Add freemkv keydb path to KEYDB search, fail on encrypted disc without keys 2026-04-17 22:10:56 +00:00
Matt Jackson 9791e60c65 v0.10.8: prefetch all metadata file sectors — scan 2min to 18s on USB 2026-04-17 19:51:32 +00:00
Matt Jackson 8b1ded84cd Document buffered sector reads in UDF docs 2026-04-17 18:41:53 +00:00
Matt Jackson 3e4febe7f5 Remove debug timing from scan pipeline 2026-04-17 18:26:04 +00:00
Matt Jackson aa7afc0c98 Buffer UDF sector reads — eliminates scan hang on USB drives 2026-04-17 18:24:29 +00:00
Matt Jackson 29f9e916d6 DiscStream::new() replaces open_drive/open_iso/from_reader
A stream is a stream. DiscStream::new() takes reader + title + keys +
batch + format — same pattern as every other stream constructor.

Deleted: open_drive(), open_iso(), from_reader() — these were helper
functions that chained multiple operations. Library provides primitives,
callers decide the sequence.

Removed disc:// case from input() — callers use Drive::open() +
Disc::scan() + DiscStream::new() directly for disc sources.
2026-04-17 15:32:48 +00:00
Matt Jackson 30d59063d5 Fix drive discovery in Docker — remove sysfs check 2026-04-16 22:05:25 +00:00
MattJackson 6e958b0f71 v0.10.5: Buffer audio parsers across PES boundaries 2026-04-16 19:55:57 +00:00
MattJackson 911d260695 Buffer audio parsers across PES boundaries
- DTS: buffer with core sync detection + frame size from header
- TrueHD: buffer with unit length field parsing
- Same pattern as AC3 fix: incomplete frames held for next PES
- When PES boundaries align (normal case), buffering is a no-op
2026-04-16 19:02:23 +00:00
MattJackson 59d4eb8854 Fix AC3 parser: buffer across PES boundaries, proper frame sizing
- Add state to Ac3Parser (was stateless, split frames at PES boundaries)
- Buffer leftover bytes from incomplete frames for next PES packet
- Calculate exact AC3 frame size from fscod/frmsizecod table
- Calculate EAC3 frame size from frmsiz field
- Skip invalid frame sizes (0 or >8192)
- Eliminates all AC3 decode errors on BD and UHD output
2026-04-16 18:49:30 +00:00
MattJackson 8820f7a460 Fix cargo fmt formatting 2026-04-16 17:51:45 +00:00
MattJackson 8de99bb8e8 Update README and FEATURES.md for v0.10.4 DVD support 2026-04-16 17:36:41 +00:00
MattJackson 3e5e570573 v0.10.4: CSS decryption, MPEG-2 PS demuxer fixes
Full CSS key hierarchy (bus auth, disc key, title key), correct
descramble cipher, MPEG-2 codec parser routing for DVD PS path,
sequence header extraction with quantizer matrices.
2026-04-16 17:33:29 +00:00
MattJackson f95894bba3 Expand CSS Stevenson crack patterns, scan 50K sectors
- Add padding stream (0xBE) with 0xFF payload patterns
- Add video/audio PES with multiple flag/header combinations
- Add navigation pack system header pattern
- Scan up to 50K consecutive scrambled sectors (was 500 sampled)
2026-04-16 15:15:17 +00:00
MattJackson 597f512eca Fix MPEG-2 PS demuxer: codec parsers, sequence header extraction, descramble
- Route DVD PS packets through codec parsers (was bypassing them)
- Extract MPEG-2 sequence header with quantizer matrices for MKV codec_private
- Calculate exact sequence header size from intra/non-intra matrix flags
- Capture sequence extension (B5) from subsequent PES packets
- Revert TAB1 permutation in descramble (XOR-only is correct)
- Fix CSS roundtrip tests for new descramble behavior
2026-04-16 15:06:35 +00:00
MattJackson 8e907393a3 Route DVD PS demuxer through codec parsers, fix CSS test expectations
- DVD PS path now calls parser.parse() like BD-TS path does
- MPEG-2 sequence headers extracted for codec_private
- Keyframe detection from parser instead of always-true
- Fix CSS roundtrip tests: descramble uses TAB1 permutation, not pure XOR
2026-04-16 04:49:46 +00:00
MattJackson c47b9a9c3c Fix CSS decryption: full key hierarchy, correct cipher tables
- Implement complete CSS key chain: bus auth → disc key → title key
- Add 31 player keys for disc key decryption
- Read disc key via READ DVD STRUCTURE format 0x02
- Read title key via REPORT KEY format 0x04
- Fix CryptKey round 1: use original scratch for term, not modified tmp1
- Fix decrypt_key: use TAB5 for LFSR1 output, TAB4 for LFSR0^invert
- Fix descramble_sector: use TAB5 for LFSR1, TAB4 for LFSR0 (no invert),
  and apply TAB1 permutation to ciphertext before XOR
- Fix title key bus XOR: forward order (bus_key[i]), not reversed
- Two-session auth: disc key and title key need separate AGID sessions
- Fix crack_key: scan across extents for scrambled sectors
- Fix TsDemuxer: dynamic PID table size for DVD PIDs
- Set max read speed after scan for DVD riplock removal
2026-04-16 04:42:42 +00:00
MattJackson d7b9d87075 v0.10.3: CSS drive authentication for DVD ripping 2026-04-16 00:01:48 +00:00
MattJackson e0f40583c4 Fix cargo fmt formatting 2026-04-15 22:32:53 +00:00
MattJackson d06a37d3dc Add 0.10.2 changelog entry 2026-04-15 22:29:33 +00:00
MattJackson add9a929a5 Bump to v0.10.2 2026-04-15 22:26:51 +00:00
MattJackson 9bb1d02245 Fix batch overflow in Disc::copy(), DVD PGC parsing, demuxer flush at EOF
- Disc::copy() hardcoded batch=64 sectors, exceeding BU40N's 60-sector
  hw limit. Now accepts batch_sectors param, defaults to 60.
- IFO PGC: playback time at offset 0x04 not 0x02, cell time at cell+4
- DiscStream: set demuxer from content_format (TS for BD, PS for DVD)
- Flush TS/PS demuxers at EOF to avoid losing last PES frame
- M2tsStream: flush demuxer at EOF
- StdioStream: FMKV metadata header for roundtrip compatibility
2026-04-15 22:26:07 +00:00
MattJackson dc706e8153 Fix integration tests for new PES-only API
Update tests/streams.rs: IOStream → PES Stream, MkvStream::new → create,
M2tsStream::new → create, NullStream::new takes title, open_input → input,
open_output → output. All 317 tests pass.
2026-04-15 19:54:09 +00:00
MattJackson 0f18906ede v0.10.1: Streams are PES, Disc::copy() for sector dumps, zero English
Architecture:
- One stream per format, bidirectional PES (read/write on same type)
- IsoStream merged into DiscStream (one type, any SectorReader)
- Disc::copy() for disc→ISO raw sector dump
- IOStream trait deleted, all byte-level Read/Write removed
- ContentReader/OpenDisc/open_title/open_input/open_output deleted
- CountingStream wrapper for progress tracking

Error codes:
- All io::Error English strings replaced with Error enum variants
- From<Error> for io::Error conversion
- Unused variants removed, new stream/mux variants added

Deleted: mkvout.rs, pesout.rs, isowriter.rs, mkv-muxer-plan.md
Updated: all docs, README stream table, CHANGELOG

238 tests, 0 clippy warnings.
2026-04-15 19:46:01 +00:00
MattJackson bd7220c62b v0.10.0: PES pipeline audit, codec_privates on DiscTitle, streams not files 2026-04-15 17:22:52 +00:00
MattJackson 404c005e0b Remove Seek/File dependencies from stream readers
Streams are streams — they take impl Read, not Read+Seek or File.

- MkvStream::open takes impl Read (was Read+Seek)
  - EBML element skipping uses skip_bytes() instead of seek(Current)
  - Byte position tracking uses remaining-bytes counter, not stream_position()
  - Removed file_size from open (progress is CLI concern)
- M2tsStream::open takes impl Read (was Read+Seek)
  - Buffers first 1MB for FMKV header / PMT scan
  - Uses chain reader (buffered head + rest) for sequential reading
  - Duration unknown without seeking (0.0) — CLI can set from metadata
- Removed ReadSeek trait (no longer needed)
- WriteSeek kept (MKV muxer container format requires seeking internally)
2026-04-15 17:05:50 +00:00
MattJackson 87342290c5 Move codec_privates onto DiscTitle, eliminate duplicate methods
Design fix: codec_privates are now a field on DiscTitle, not a separate
parameter passed through the pipeline. This eliminates the root cause of
the network codec_private bug (forgot to pass the separate param).

API changes:
- output() takes (url, &DiscTitle) — no separate codec_privates param
- MkvOutputStream::create, M2tsOutputStream::create, NetworkOutputStream::connect
  all read codec_privates from title.codec_privates
- M2tsMeta::from_title() takes only &DiscTitle — reads privates from title
- Deleted from_title_with_privates (was the wrong-name duplicate)
- Merged read_header + read_header_from_stream into one read_header(impl Read)
- Deleted finish(self) from TsMuxer, keep only finish(&mut self)

Rule: ONE public method per action. No _with_X, _from_Y, _ref variants.
2026-04-15 16:52:06 +00:00
MattJackson 8bbf630d82 Fix all PES pipeline audit findings (20 issues)
Critical:
- C1: PES serialize validates track < 256 and data < 4GB
- C2: PES deserialize caps frame size at 256MB (OOM protection)
- C3: TsMuxer stuffing uses static buffer, no per-packet alloc
- C4: PES length uses unbounded (0x0000) for audio >65535 bytes
- C6: TsDemuxer validates AF length <= 183

Warning:
- W1: parse_timestamp validates marker bits, returns Option
- W2: PES header data_start clamped to data.len()
- W3: TsMuxer PTS conversion uses saturating_mul, rejects negative
- W4: AC3/DTS replace debug_assert with runtime bounds check
- W6: MKV block_vint handles 3-4 byte VINTs
- W7: meta.rs to_title() uses unwrap_or fallbacks instead of panic
- W8: MKV reader skips frames for non-existent tracks
- W9: DVD PTS uses higher-precision conversion (1e9/90000)
- FMKV read_header caps JSON at 10MB
- PAT section_len underflow guard

Suggestion:
- S2: TsMuxer uses static STUFF_FF buffer
- S3: HEVC parser single-pass NAL scan (was duplicated)
- S4: TsDemuxer caps remainder at one packet
- S5: PTS 90kHz→ns uses round-to-nearest
2026-04-15 16:22:28 +00:00
MattJackson 45dddc1810 Fix M2TS/MKV roundtrip: codec_private in FMKV header, Annex B conversion
Bug 1: M2TS roundtrip dropped frames — TsMuxer converts length-prefixed
NALs to Annex B, prepends VPS/SPS/PPS from HEVCDecoderConfigurationRecord.

Bug 2: MKV remux lost codec_private — MkvStream.codec_private() now returns
data from EBML header.

FMKV header carries codec_private (base64) per video stream for lossless
M2TS roundtrip.
2026-04-15 16:09:34 +00:00
MattJackson cfa80cb881 Fix test for network open_input error message change 2026-04-15 04:54:19 +00:00
MattJackson b510ae9b29 Fix M2TS/Network output: write FMKV header for roundtrip compatibility
- M2tsOutputStream writes FMKV metadata header before TS data
- NetworkOutputStream sends FMKV header on connect
- Enables M2TS→MKV and network roundtrip to work correctly
2026-04-15 04:50:19 +00:00
MattJackson 9a1c3cc218 NetworkStream PES-only: remove old IOStream/Read/Write interface
- Remove IOStream, Read, Write impls from NetworkStream
- PES write sends FMKV header before first frame (protocol fix)
- PES finish sends TCP shutdown for clean EOF
- Update tests to use PES roundtrip instead of byte-level
- Remove NetworkStream from open_input/open_output (use input/output)
2026-04-15 04:36:47 +00:00
MattJackson 9ae7d6b38a Fix audit findings: SCSI constants, sg_io_hdr assert, handshake cap, sector overflow check
- Replace magic SCSI opcodes with named constants (S2)
- Add compile-time sg_io_hdr size assertion — 88 bytes on 64-bit (W2)
- Cap handshake cert attempts at 16 (W8)
- Validate IsoSectorReader/FileSectorReader against u32 overflow for >8TB (S8)
- encrypt.rs: limit host cert loop iterations
2026-04-15 04:29:45 +00:00
MattJackson d983985faa Fix all clippy warnings: dead code, match patterns, type complexity, docs
- Remove unused pes_buf field from M2tsStream and unused TS_PACKET/BD_TS_PACKET constants
- Replace match-with-single-pattern with if let (3 instances in drive/mod.rs)
- Replace match-can-be-? with ? operator for scsi::open call
- Add type aliases PesSetup and MkvHeaderResult to reduce type complexity
- Collapse identical if/else branches in tsmux.rs build_pes_header
- Use RangeInclusive::contains instead of manual range checks
- Make WriteSeek trait pub (was pub(crate) but leaked through pub fn)
- Remove empty line after doc comment in disc.rs
- Fix doc list item indentation in scsi/linux.rs (12 instances)
2026-04-15 04:09:56 +00:00
MattJackson 56fe26c9b8 All streams complete — DVD PS demux, network/stdio PES, MKV input
- DiscStream: BD (TsDemuxer) or DVD (PsDemuxer) auto-detected
- NetworkStream: Stream impl with PES serialize/deserialize
- StdioStream: Stream impl with PES serialize/deserialize
- MkvStream: Stream read returns PesFrame from EBML blocks
- M2tsStream: Stream read via TsDemuxReader
- PesFrame: serialize/deserialize for wire format
- TsDemuxReader: shared BD-TS demux helper
- All inputs and outputs support PES
2026-04-15 04:03:56 +00:00
MattJackson 7a83f3244d Complete PES pipeline — all streams, clean API
- Unified Stream trait: read() and write() on one type
- PesFrame serialize/deserialize for wire format
- TsDemuxReader: shared BD-TS demux for any Read source
- MkvStream: PES read from MKV (EBML → PesFrame)
- M2tsStream: PES read via TsDemuxReader
- Network/Stdio output: PES serialization directly (no BD-TS wrap)
- Network/Stdio input: deferred (needs PES deserialization protocol)
- TsMuxer for M2TS output from PES frames
- input() and output() functions return Box<dyn Stream>
2026-04-15 03:56:15 +00:00
MattJackson 15f5b49fc3 M2TS implements Stream (read PES frames via TsDemux)
M2tsStream::read() → TsDemux → CodecParser → PesFrame.
M2TS input now works through PES pipeline.
MKV input deferred (needs EBML → PES extraction).
2026-04-15 03:41:03 +00:00
MattJackson ff6004a567 Unified Stream trait: read() and write() on one type
Stream trait: read() returns PesFrame, write() accepts PesFrame.
A stream is a stream — you read from it or write to it.
No separate Input/Output traits.

API: libfreemkv::input(url) and libfreemkv::output(url, title, codecs)
Returns Box<dyn Stream>.
2026-04-15 03:33:29 +00:00
MattJackson bd644d2f60 100% PES pipeline — all streams produce/consume PES frames
- TsMuxer: PES frames → BD-TS packets (new, reverse of TsDemuxer)
- M2tsOutputStream: PES → TsMuxer → file
- NetworkOutputStream: PES → TsMuxer → TCP
- StdioOutputStream: PES frames → stdout
- NullOutputStream: discard
- MkvOutputStream: PES → MKV mux
- All outputs via open_pes_output()
- All inputs via open_pes_input() (ISO, disc)
- No byte-level fallback — everything is PES
2026-04-15 03:19:03 +00:00
MattJackson ab63950ef1 PES pipeline: InputStream on DiscStream + IsoStream, MkvOutputStream
- DiscStream: next_frame() reads sectors → decrypts → demuxes → returns PesFrame
- IsoStream: same pattern, reads from ISO file
- MkvOutputStream: accepts PesFrame, writes MKV via MkvMuxer
- InputStream trait: next_frame(), info(), codec_private(), headers_ready()
- OutputStream trait: write_frame(), finish()
- FileSectorReader: SectorReader backed by a file
- PesFrame: track + pts + keyframe + data

Old Read/Write IOStream impls preserved for backward compatibility.
Next: replace pipe() in CLI to use PES pipeline.
2026-04-15 03:03:45 +00:00
MattJackson cab87ffdf4 DiscStream implements InputStream — produces PES frames
DiscStream now has TsDemuxer + CodecParsers internally.
next_frame() reads sectors → decrypts → demuxes → parses → returns PesFrame.
Old Read/Write impls preserved alongside for backward compatibility.
2026-04-15 02:54:08 +00:00
MattJackson ccb1fadedf Add PES frame types and FileSectorReader — foundation for stream refactor
- pes.rs: PesFrame, InputStream, OutputStream traits
- sector.rs: SectorReader now public, added FileSectorReader (ISO = file)
- Foundation for unified DiscStream that handles both disc and ISO
2026-04-15 02:37:54 +00:00
MattJackson 610f1b62b6 Fix ISO → MKV producing empty files (TsDemuxer remainder lost)
When the MKV muxer transitions from Scanning to Streaming phase,
it creates a fresh TsDemuxer. The old demuxer's remainder bytes
(partial 192-byte BD-TS packets) were lost, causing the new demuxer
to lose sync. All subsequent feed() calls found 0 packets.

Fix: transfer remainder via take_remainder/set_remainder so the
new demuxer maintains packet alignment.
2026-04-15 02:17:34 +00:00
MattJackson f400e4fc72 Remove eprintln from library — library code should not print to stderr
Drive recovery is silent. Results communicated through return values.
2026-04-15 02:00:45 +00:00
MattJackson 5a9980cb6b Fix unused variable warning from handshake fix 2026-04-15 01:56:57 +00:00
MattJackson 2f48877992 Fix handshake returning fake success on failure
Previously returned HandshakeResult with zeros when all host certs
failed. Now returns None. Also propagates volume_id read failure
instead of silently using zeros.
2026-04-15 01:54:58 +00:00
MattJackson 7c59d063ee decrypt_sectors returns Result — fail instead of silent corruption
Previously used all-zeros AES key when unit_key_idx was out of range,
producing silently corrupted output. Now returns DecryptFailed error.
Also wired --raw flag through InputOptions → set_raw() on streams.
2026-04-15 01:52:36 +00:00
MattJackson e4ffe7cc91 Wire --raw through InputOptions to streams
set_raw() on IsoStream and DiscStream sets keys to None.
open_input passes raw flag to streams via InputOptions.
Streams skip decrypt when raw=true.
2026-04-15 01:38:18 +00:00
MattJackson 2e23e848df Decrypt back in streams — streams handle their own decryption
IsoStream decrypts in its read path using keys from scan.
IOStream trait gets keys() method with default DecryptKeys::None.
Pipeline no longer handles decrypt — just reads decrypted bytes.
2026-04-15 01:34:30 +00:00
MattJackson fbdfccdd83 Add keys() to IOStream trait — streams know their own decrypt keys
- IOStream::keys() default returns DecryptKeys::None
- IsoStream and DiscStream override with real AACS/CSS keys
- Pipeline calls input.keys() instead of separate scan_keys()
- Streams are self-contained: read bytes + provide keys
2026-04-14 23:42:14 +00:00
MattJackson 2f52188f77 Drive recovery, reset on open, simplified DiscStream
- SgIoTransport::reset() — open/close/TUR/escalate on every open
- Drive::read() — single read method with error recovery (min speed,
  sleep 30s, retry, phase 1/2/3 escalation)
- Removed read_timeout, read_sectors, read_range — one read() method
- DiscStream simplified — no on_error/on_success/Recovery, delegates
  all error handling to Drive::read()
- IsoStream no longer decrypts — streams return raw bytes, pipeline
  handles decryption
- reset() on all platforms (Linux real, Windows/macOS stubs)
- Watchdog thread removed — kernel handles USB timeouts
2026-04-14 23:32:22 +00:00
MattJackson 252e58cab0 Bump to 0.9.0 2026-04-13 02:13:15 +00:00
MattJackson 3b96976a7a Add decrypt module, merge to one drive.read(), Disc::decrypt_keys()
- New decrypt.rs: DecryptKeys enum (AACS/CSS/None) + decrypt_sectors()
- Single drive.read() replaces read_disc/read_content (same SCSI READ(10))
- ContentReader and DiscStream use decrypt_sectors() (no duplicated crypto)
- Disc::decrypt_keys() exposes resolved keys for disc-to-ISO
2026-04-13 02:08:58 +00:00
MattJackson 3385de5704 Add Drive::read_capacity() for raw sector dump 2026-04-13 01:45:32 +00:00
MattJackson e9c907a2ca Pin Rust 1.86 MSRV in Cargo.toml and CI workflows 2026-04-13 00:35:35 +00:00
MattJackson db1f6bc446 Fix is_multiple_of nightly API, bump to 0.8.3
Replace s.len().is_multiple_of(2) with s.len() % 2 != 0 for stable Rust.
This was fixed previously but regressed.
2026-04-13 00:31:54 +00:00
MattJackson c43ba06b5b Fix macOS build: mark MacScsiTransport as Send
IOKit COM interface pointers are Mach port references, safe to send
between threads. The Send bound added in 0.8.1 broke macOS builds.
2026-04-13 00:22:19 +00:00
MattJackson 077e4d018f Bump to 0.8.1, make profile module public, fix unused import 2026-04-13 00:17:57 +00:00
MattJackson f8b5a1eaf1 API: Drive object, typed StreamUrl, tray lock/unlock, Send traits
- Rename DriveSession → Drive across entire codebase
- find_drives() returns Vec<Drive>, find_drive() returns Option<Drive>
- resolve_device() now pub(crate) — internal only
- StreamUrl is now a typed enum (Disc, Mkv, M2ts, Iso, Network, Stdio, Null)
  with scheme() and path_str() accessors, replacing struct of Strings
- Add lock_tray() / unlock_tray() for safe disc access during rips
- Improve reset() with eject cycle that clears LibreDrive stuck state
- Add Send bounds to ScsiTransport and PlatformDriver traits
- DiscOptions uses PathBuf instead of String for device/keydb paths
- Update doc example to use new Drive API
2026-04-13 00:13:41 +00:00
MattJackson fb1c35e653 DriveStatus API + reset() + wait_ready with fallback
- DriveStatus enum: TrayOpen, NoDisc, DiscPresent, NotReady, Unknown
- drive_status(): GET EVENT STATUS NOTIFICATION with TUR fallback
- reset(): PREVENT ALLOW → START STOP → init() escalation
- wait_ready(): tries reset on Illegal Request, falls back to drive_status
- Remaining: standard READ(10) still fails in LibreDrive stuck state
2026-04-12 23:36:16 +00:00
MattJackson 9268ea2fdb WIP: LibreDrive stuck state detection in wait_ready + scan
- wait_ready: detects MMkv vendor probe when TUR returns Illegal Request
- scan: capacity fallback to 0 when READ CAPACITY fails
- Still needs: re-init to restore standard SCSI commands, or use raw reads for UDF
2026-04-11 22:09:26 +00:00
MattJackson 9078aa8718 Fix UHD remux: skip DV EL from video_pending, dynamic lookahead
- Secondary video streams (Dolby Vision EL) no longer block codec detection
- Lookahead buffer: 10 MB default, 100 MB for UHD (>15 streams)
- Verified: Dune UHD 84.6 GB remux completes at 101 MB/s
- Verified: V for Vendetta BD 21.3 GB remux completes at 131 MB/s
2026-04-11 21:43:51 +00:00
MattJackson c55e6991b8 Doc comments, format string inlining, long literal separators
- Doc comments on DriveSession, find_drives, all Error variants, Result type
- 24 format! strings inlined (clippy pedantic)
- 25 long hex literals with separators (0xFFFFFFFF → 0xFFFF_FFFF)
- README install example updated to 0.8
2026-04-11 21:04:44 +00:00
MattJackson ca931b6522 Doc comments on public API, README version fix, format string cleanup 2026-04-11 21:01:16 +00:00
MattJackson d4d98ce593 Granular SCSI query methods on DriveSession, capture uses them
- get_config_feature(code) → Option<Vec<u8>>
- report_key_rpc_state() → Option<Vec<u8>>
- mode_sense_page(page) → Option<Vec<u8>>
- read_buffer(mode, buf_id, length) → Option<Vec<u8>>

capture.rs now uses these methods — zero raw CDB construction.
CLI info.rs has zero SCSI references.
autorip ejects via library, not shell command.
2026-04-11 20:54:04 +00:00
MattJackson 4d92e15224 v0.8.0: DVD support, 100% codecs, 327 tests, 4 audit rounds clean 2026-04-11 20:35:33 +00:00
MattJackson 113d6b9e9e CSS + AACS cross-validation test vectors
AACS: encrypt with aes crate independently, decrypt with our code, verify match
- 3 tests: unit decrypt, alternate key, bus decrypt
- Uses independent AACS IV constant (not imported from library)

CSS: roundtrip snapshots + Stevenson attack validation
- 4 tests: snapshot regression, multi-key roundtrip, attack validation
- Documents limitation: synthetic sectors may not converge on attack

320+ tests total.
2026-04-11 20:31:12 +00:00
MattJackson 75dfd06a02 Fix all v4 audit findings (22 items)
HIGH: ISO writer multi-extent for >4GB, end-to-end MKV mux test
MEDIUM: AACS cvalue bounds, UV offset, macOS discovery, VC-1 resolution
  from sequence header, HEVC profile flags from SPS, ISO CRC + reserve AVDP
LOW: PS AC3 sub-header, CSS crack first-match break, TrackUID unique,
  AC3 no-sync empty return, --all for iso://, --min warning, dead code removed

320 tests, all passing.
2026-04-11 20:29:00 +00:00
MattJackson f48b4925c1 Zero clippy warnings: fix all 32 remaining
- Iterator::find() replaces manual loops (6 sites)
- Index-only loops → iterators (4 sites)
- Identical if-blocks merged
- Box large MkvStream WriteState enum variant
- Vec macro initializers, late init fixes
- Unused fields prefixed with underscore (format spec fields)
- Dead code removed or documented

0 clippy warnings. 319 tests passing.
2026-04-11 19:33:13 +00:00
MattJackson 75f15cae62 Audit v3 fixes: all 3 tiers (19 findings)
Tier 1 (compilation + correctness):
- Fix nightly-only is_multiple_of → % 2 != 0 (stable Rust compat)
- Fix parse_sample_rate: check 192 before 96 (was returning wrong rate)
- macOS drive discovery: split unix.rs → linux.rs + macos.rs
- Linux: EACCES returns DevicePermission not DeviceNotFound
- CLI pipe.rs: Ctrl+C signal handler added

Tier 2 (correctness + security):
- MkvStream: reset demuxer after scanning→streaming transition
- Windows SPTI: zero data buffer before ioctl
- AACS cert verification: documented why silently skipped
- KEYDB: HOME + USERPROFILE fallback for Windows
- Library modules: pub(crate) for internal modules
- AACS: explicit re-exports, AES primitives pub(crate)

Tier 3 (performance + polish):
- IsoStream: batch 64-sector reads (was 1 sector at a time)
- DiscStream: buffer swap instead of copy in decrypt_and_buffer
- Vec capacity hints in TS/PS demuxer hot paths
- NetworkStream: TLS warning documented
- Batch rip: per-title progress display
- cargo fmt: 0 violations

319 tests, 0 fmt violations.
2026-04-11 19:24:25 +00:00
MattJackson e25035fe8c Fix audit v2 criticals: DiscStream read loop, CSS crack, ISO writer
Critical fixes:
- DiscStream: persistent read state (was creating new ContentReader per call)
  Full error recovery, AACS/CSS decryption, extent tracking across reads
- CSS crack: labeled 'outer continue (was targeting wrong loop)
- CSS crack: LFSR0 polynomial fixed to match cipher (shifts 8,1,3,7)
- CSS lfsr: operator precedence clarified in LFSR0 init

Warning fixes:
- ISO writer: UDF tag checksums computed (was zeros)
- DVD extents: saturating_add for overflow safety
- Removed dead fields: HandshakeResult.error, ContentReader.content_format
- Added TODO for ISO long_ad >4GB support

319 tests, 20 clippy warnings remaining.
2026-04-11 19:11:25 +00:00
MattJackson ffa0eaba4d cargo fmt + clippy --fix: 104 format violations fixed, 8 clippy auto-fixes 2026-04-11 19:10:20 +00:00
MattJackson 96a65de3ff Chapters, DVD subtitle palette, MKV track flags, progress total_bytes
Chapters:
- MPLS PlayList marks parsed (mark_type 1 = chapter)
- Chapter struct on DiscTitle (time_secs, name)
- MKV Chapters element with EditionEntry/ChapterAtom per mark
- 3 MPLS mark tests + 2 MKV chapter tests

DVD subtitle palette:
- IFO palette extraction (PGC offset 0xA4, 16 × YCbCr colors)
- YCbCr→RGB conversion for VobSub .idx format
- DvdSubParser codec_private returns formatted palette
- codec_data field on SubtitleStream flows through pipeline
- 5 palette tests (YCbCr conversion, formatting, overflow)

MKV track flags:
- FlagDefault: primary video/audio = 1, secondary = 0
- FlagForced: forced subtitles = 1
- Language: set from stream language code
- Already implemented, verified with 4 new tests

Progress total_bytes:
- IOStream trait: total_bytes() -> Option<u64>
- DiscStream, IsoStream: from disc_title.size_bytes
- M2tsStream, MkvStream: from file metadata on open
- NetworkStream, StdioStream, NullStream: None

316 tests total, all passing.
2026-04-11 17:43:47 +00:00
MattJackson cd575b7221 Add MKV muxer, IsoWriter, disc pipeline, and network tests
- MKV muxer: EBML header, segment, cluster, cues, multi-track, keyframe flags — 6 tests
- MkvStream: BD-TS roundtrip, metadata preservation — 2 tests
- IsoWriter: valid UDF, file size update, custom names, empty content — 4 tests
- Disc pipeline: format detection (UHD/BD/DVD), content format, capacity, duration — 5 tests
- Network: listen/connect roundtrip, metadata flow — 2 tests (ignored for CI)
- Encryption: no AACS dir, no keydb — 2 tests
- 297 tests total, all passing
2026-04-11 17:27:36 +00:00
MattJackson 515f2f6bc2 100% codec coverage + disc/ and aacs/ module refactors
Codec coverage (DVD + BD + UHD):
- E-AC-3 (Dolby Digital Plus): bsid detection, frame size calc — 8 tests
- DTS-HD MA/HR: extension substream (0x64582025) detection — 8 tests
- LPCM: BD header skip, raw PCM extraction — 6 tests
- DVD VobSub subtitles: passthrough parser — 5 tests
- Dolby Vision: verified RPU NAL type 62 preserved in HEVC — 1 test

Module refactors:
- disc.rs → disc/mod.rs + bluray.rs + dvd.rs + encrypt.rs
- aacs/mod.rs (1661 lines) → mod.rs (21) + keydb.rs + keys.rs + decrypt.rs
- All public APIs preserved, all tests pass

270 tests total, 0 failures.
2026-04-11 17:22:18 +00:00
MattJackson fe723a7759 Add crypto roundtrip tests: CSS + AACS validation
- CSS: decrypt_key determinism, descramble XOR roundtrip, TAB1 permutation,
  TAB4 bit-reversal involution, Stevenson attack on scrambled sector
- AACS: decrypt_unit roundtrip, disc hash deterministic, VUK derivation,
  unit key parsing, EC point-on-curve and ECDSA already covered
- 239 tests total
2026-04-11 17:17:29 +00:00
MattJackson e4c5c88909 CSS crypto tests + DVD pipeline fully wired
- CSS roundtrip tests: decrypt_key determinism, descramble XOR roundtrip
- CSS table verification: TAB1 is permutation, TAB4 is bit-reversal involution
- DVD scan pipeline confirmed: scan_dvd_titles, CSS crack, ContentReader descramble
- 229 tests, all passing
2026-04-11 17:14:58 +00:00
MattJackson 648ce28ac6 Rewrite CSS from Stevenson 1999 paper — proper table-driven cipher
- tables.rs: 5 CSS specification tables (TAB1-TAB5, mathematical constants)
- lfsr.rs: Table-driven LFSR1 (TAB2/TAB3) + LFSR0 (TAB4), sector seed XOR,
  decrypt_key() mangling function, descramble_sector() with proper feedback
- crack.rs: Stevenson divide-and-conquer attack (2^16 LFSR1 iteration,
  LFSR0 deduction from known plaintext, 10-byte validation)
- No external code copied — original Rust implementation from the 1999 paper
- 225 tests, 0 ignored
2026-04-11 17:07:21 +00:00
MattJackson ff5547363b Audit fixes + DVD support foundation (IFO, PS demux, MPEG-2, CSS crack)
Audit fixes (14 critical, 22 warnings):
- UDF: bounds checks on all ICB/FID parsing from disc data
- SCSI Linux: saturating_sub on residual, CDB length guard, buffer size guard
- SCSI macOS: SCSITaskStatus u32 (was u8 — stack corruption)
- AACS: EC mod_inv returns infinity instead of panic, key reduced mod n
- AACS: do_handshake tries all host certs (was returning on first failure)
- H.264: bounds check on SPS < 4 bytes
- ContentReader: error on missing unit key (was zero-fill)
- KEYDB: flat redirect loop (was recursive), 100MB response limit, Windows HOME fallback
- ISO writer: AVDP extent order, partition length, allocation cap
- Network: removed TCP_NODELAY on bulk stream
- MKV: guard on u64::MAX seek
- disc.rs: saturating_sub on extent offset, simplified dead region code
- cargo fmt (610 violations), cargo clippy --fix (55 auto-fixes)

DVD support (new files):
- src/ifo.rs — IFO parser (VIDEO_TS.IFO, VTS_XX_0.IFO, PGC chains, cells, streams) — 13 tests
- src/mux/ps.rs — MPEG-2 Program Stream demuxer (pack headers, PES, private stream 1) — 12 tests
- src/mux/codec/mpeg2.rs — MPEG-2 video parser (sequence headers, I-frame detection) — 15 tests
- src/css/crack.rs — split-attack algorithm (LFSR cipher needs verification — test ignored)

226 tests total (was 186), 1 ignored (CSS crack needs cipher verification).
2026-04-11 16:52:22 +00:00
MattJackson 6e771a1867 v0.7.2: Windows SPTI, 177 tests, platform file separation 2026-04-11 16:15:14 +00:00
MattJackson 8999db68ea Refactor: drive discovery into platform files, no inline cfg
- drive/unix.rs: find_drives() + resolve_device() for Linux/macOS
- drive/windows.rs: find_drives() + resolve_device() + normalize_path() for Windows
- drive/mod.rs: clean delegation, no cfg branches
- scsi/windows.rs: SPTI transport only, no drive discovery
2026-04-11 16:12:53 +00:00
MattJackson 2d67b11a2c Add Windows SPTI backend, CI check, platform support complete
- scsi/windows.rs: SCSI_PASS_THROUGH_DIRECT via DeviceIoControl
- Device path normalization (D:, \\.\CdRom0, \\.\D:)
- Windows drive discovery (CdRom0-15 + drive letters)
- CI: cargo check on windows-latest
- Platform table: Linux + macOS + Windows all supported
2026-04-11 16:08:30 +00:00
MattJackson 995525d3ff Add 113 tests, update CI to checkout@v5, add FEATURES.md
Test suite: 64 → 177 tests
- MPLS parser: 6 tests (synthetic binary, streams, errors)
- CLPI parser: 6 tests (EP map, PTS/SPN math, errors)
- H.264: 12 tests (NAL parsing, SPS/PPS, keyframes)
- HEVC: 13 tests (VPS/SPS/PPS, IRAP range, codec private)
- AC3: 12 tests (syncword, frame extraction)
- VC1: 15 tests (BITMAPINFOHEADER, start codes)
- DTS: 5, TrueHD: 4, PGS: 4 tests
- EBML: 6 tests (size/ID/string/float roundtrips)
- UDF: 10 tests (MockSectorReader, filesystem parsing, error paths)
- Disc: 8 tests (scan_image, DiscTitle helpers)
- Streams: 5 new (meta roundtrip, MkvStream)
- NullStream: 4, StdioStream: 2, IsoSectorReader: 2

CI: actions/checkout@v4 → v5 (all workflows)
FEATURES.md: created for v0.7.1
2026-04-11 16:02:49 +00:00
MattJackson dc4ebd7d9b v0.7.1: SectorReader trait, IsoStream, StdioStream, resolve_encryption
- SectorReader trait decouples disc scanning from SCSI
- Disc::scan_image() for ISO and any sector source
- resolve_encryption() handles AACS 1.0/2.0/none in one path
- IsoStream: full UDF/MPLS/CLPI/labels pipeline from ISO files
- StdioStream: stdin/stdout pipe
- Strict scheme:// URL format with validation
- Labels module refactored to SectorReader
- 7 stream types total
2026-04-11 15:49:29 +00:00
MattJackson 168005ef34 Fix update-readme: use ORG_DISPATCH_TOKEN to bypass branch protection 2026-04-11 15:04:22 +00:00
MattJackson 6dc308f245 Bump to v0.7.0 2026-04-11 14:55:52 +00:00
MattJackson cccb6b4631 Add StdioStream, IsoStream; enforce scheme:// URL format
- StdioStream: stdin/stdout pipe, format-agnostic
- IsoStream: read BD-TS from Blu-ray ISO images
- URL resolver: bare paths rejected, all URLs require scheme:// prefix
- Validation: empty paths, missing ports, read-only/write-only errors
- Tests: 22 passing (URL parsing, validation, metadata roundtrip)
- Docs: full stream table with 7 stream types
2026-04-11 14:54:28 +00:00
MattJackson 8c2f3898b8 Add IOStream trait and stream-based I/O architecture
Introduce IOStream trait for uniform read/write across disc, file,
network, and null streams. Rename Title→DiscTitle, add stream URL
resolver, split old stream.rs into focused modules (m2ts, mkvstream,
network, disc, null, resolve, meta).
2026-04-10 19:13:53 -07:00
MattJackson 37c98e8826 Support multiple host certs, expose handshake_error on AacsState
- host_cert: Option → host_certs: Vec (try all until one succeeds)
- handshake_error field shows why auth failed (e.g. cert rejected/revoked)
2026-04-10 09:48:29 -07:00
MattJackson 891dee3db9 Bump to v0.6.1 — republish with open() all drives, AACS 2.0, has_profile() 2026-04-10 09:30:15 -07:00
MattJackson e34b41967d v0.6.0: changelog, open() all drives, AACS 2.0, MKV muxer 2026-04-10 09:26:45 -07:00
MattJackson c063dbd76e Update README for v0.6.0: AACS 2.0 done, open() all drives, .cargo gitignore 2026-04-10 09:24:01 -07:00
MattJackson d1f8db93d6 Remove orphaned jar.rs — replaced by labels/ module 2026-04-10 09:17:11 -07:00
MattJackson 926939f24b open() works on all drives, AACS 2.0 handshake wired, raw_gc_010c on DriveId
- DriveSession::open() no longer requires profile match — works on any optical drive
- init()/probe_disc() return error gracefully for unknown drives
- find_drives() returns all optical drives (PDT 0x05), not just profile-matched
- has_profile() check for callers
- AACS 2.0: handshake wired into resolve_aacs() — real VID + read_data_key
- DriveId: added raw_gc_010c field for GET_CONFIG 010C response bytes
2026-04-10 08:48:41 -07:00
MattJackson 48d70c092c Fix test failures: correct EBML size assertion, fix doc tests
- Fix write_size test: 127 encodes as [0x40, 0x7F] not [0xC0, 0x7F]
- Mark doc examples as ignore (use incomplete pseudo-code)
2026-04-10 08:23:32 -07:00
MattJackson c14b319212 Fix compiler warnings: remove unused imports and dead code
- Remove unused NAL_SLICE and NAL_SEI constants from h264 parser
- Remove unused aacs import from Disc::scan()
2026-04-10 08:21:55 -07:00
MattJackson 55b22bfb9d Add design docs for API and MKV muxer architecture 2026-04-10 08:19:53 -07:00
MattJackson 48b212c9d2 Add MKV muxer and event system
- Add event.rs: structured event system for progress reporting
- Add mux/: MKV muxer pipeline (EBML writer, TS demuxer, stream assembly)
  - Codec parsers: H.264, HEVC, AC-3, DTS, TrueHD, PGS, VC-1
  - Lookahead buffer for codec private data extraction
  - Direct m2ts-to-MKV streaming without intermediate files
2026-04-10 08:19:40 -07:00
MattJackson 074f21ba58 Refactor error types: replace generic AacsError/DiscError with typed variants
- Split AacsError { detail } into 13 specific error variants (AacsCertShort,
  AacsAgidAlloc, AacsCertRejected, etc.) with unique error codes E7001-E7012
- Split DiscError { detail } into 7 specific variants (DiscRead, MplsParse,
  ClpiParse, UdfNotFound, DiscNoTitles, DiscTitleRange, DiscNoExtents)
- Add WriteError (E5001), KeydbLoad (E8005), MuxLookahead (E9000), MuxWrite (E9001)
- Add OpenDisc API for single-call open+scan+rip workflow
- Remove all English text from error Display impl (code-only output)
- Normalize doc comments to use -- instead of em dash for ASCII consistency
2026-04-10 08:19:28 -07:00
MattJackson 9241d767ee Code audit: fix unused variables, update docs
- Fix unused variable warnings in profile parsing (|e| -> |_|)
- Fix outdated 'freemkv info --share' reference in README (now drive-info)
2026-04-10 08:19:01 -07:00
MattJackson 65996ade59 Remove SpeedTable, add probe_disc(), named constants, clean architecture
- Removed SpeedTable entirely — drive manages speeds after probe
- Renamed read_speed_table() → probe_disc()
- Named all SCSI constants: SUB_CMD_UNLOCK, SUB_CMD_INIT, SUB_CMD_PROBE,
  INIT_ADDR_BD, INIT_ADDR_UHD, PROBE_COARSE_END, PROBE_FINE_END, etc.
- Auto-detect BD vs UHD from disc capacity for correct probe init address
- Fixed NOMINAL_SPEED_B (was invalid CDB, removed — single max instead)
- Added session.set_speed() for simple speed control
- Error recovery: re-init on first error, BD2x on repeated errors
- Batch size uses full kernel limit (was 80%, now 100%)
- Clean variant_a/variant_b with named constants

API: open() → wait_ready() → init() → probe_disc() → scan() → read
2026-04-09 15:17:25 -07:00
MattJackson 09f5bb7816 Speed table: generic zone-based speed management
- SpeedTable: maps disc positions to optimal speeds
- Default: max speed everywhere (drive manages itself)
- After read_speed_table(): calibrated per-zone speeds
- One u32 comparison per read on hot path
- Error recovery: reduce() / resume() override table temporarily
- Replaces old tier-based speed management in ContentReader
- MT1959 split into mod.rs + variant_a.rs + variant_b.rs
- PlatformDriver: init() + read_speed_table() + is_ready()
2026-04-09 13:33:02 -07:00
MattJackson 12a5985a58 Update README: v0.6.0 API, real sample output 2026-04-09 13:03:00 -07:00
MattJackson d23632931b Profiles v2: chipset+variant top-level keys, minimal per-drive data
profiles.json: { "mt1959_a": [...], "mt1959_b": [...], "renesas": [] }
Each profile: identity + signature + firmware (3 fields)
Platform enum replaces Chipset — section determines variant
2026-04-09 12:50:44 -07:00
MattJackson 92e7692779 Bump to v0.6.0 2026-04-09 12:39:50 -07:00
MattJackson 0c11623666 v0.6.0: Clean API, chipset-keyed profiles, streamlined platform driver
- API: open() is OEM-only, wait_ready() separate, init() optional
- Profiles: chipset-keyed JSON ({ "mt1959": [...], "renesas": [] })
- Profiles: identity group, variant + signature + firmware per drive
- Platform constants: mode, buffer_id, nominal speed, verify commands
  moved from profiles to code (variant-determined, not per-drive)
- Removed unused fields: register CDBs, speed tables, status data
- Platform driver: unlock + firmware upload + calibrate + speed only
- Cross-compile fix: build.rs uses CARGO_CFG_TARGET_OS for framework linking
2026-04-09 12:38:02 -07:00
MattJackson b454d100c6 Update profiles: fix data extraction offset
Firmware payload was extracted 12 bytes too early — included handler
table pointers instead of actual microcode. Fixed by scanning for
function table position (VM address pattern) instead of magic offsets.

All 206 profiles regenerated. Cold boot firmware upload should now work.
2026-04-09 08:37:23 -07:00
MattJackson 097473d371 Update README and CHANGELOG for v0.5.0
- 12-23 MB/s read speeds, kernel transfer limit auto-detection
- Full custom firmware init pipeline (all 10 handlers)
- MPLS parser fixes (PGS language offset, secondary streams)
- 206 profiles with full per-drive data
2026-04-09 07:27:10 -07:00
MattJackson 5e3157b18d Fix subtitle language parsing: PGS in audio slots, secondary PG entries
Two MPLS parsing issues:

1. PGS subtitle (0x90) appearing in audio stream slot: language was read
 from audio offset sa[2..5] instead of PG offset sa[1..4], causing
 truncation ("ng " instead of "eng"). Fixed by detecting PGS coding
 type in audio slots and using correct offset.

2. Secondary PG (PiP subtitle) entries were not consumed, causing
 position tracking to drift. Added missing n_pip_pg loop.

3. Added stream_type 5/6/7 attribute parsing for secondary audio/video.
2026-04-08 22:41:07 -07:00
MattJackson d9c5f7a0d2 Fix read speed: detect kernel max transfer size, fix phantom streams
Root cause: READ_10 requested 510 sectors (1MB) but kernel sg driver
limits to max_hw_sectors_kb=120 (60 sectors). All bulk reads errored,
error handler fell to 3-sector reads → 4.8 MB/s instead of 12+ MB/s.

- detect_max_batch_sectors() now resolves sg→block device via sysfs
 and uses 80% of kernel limit for safety margin
- Default fallback reduced from 510 to 48 sectors
- ContentReader uses per-device detected max, not hardcoded constant
- Filter out coding_type 0x00 (empty/padding) stream entries
- Add MPLS stream_type 5/6/7 attribute parsing (secondary audio/video)

Tested on BU40N: 4.8 → 12.5 MB/s sustained, 23.7 MB/s peak.
2026-04-08 22:30:08 -07:00
MattJackson 6456e24bb5 Lock down Platform trait: pub(crate), only init/set_read_speed/is_ready
Platform trait is no longer publicly exported. External code uses
DriveSession only — cannot call unlock, load_firmware, calibrate directly.

- Platform trait: pub(crate) with 3 methods only
- All handlers are private methods on Mt1959
- DriveStatus moved to mt1959 internal struct
- init() has guard: no re-init if already ready
- set_read_speed() has guard: no-op if not calibrated
- Removed open_unlocked() — open() is the only entry
- Removed Platform and DriveStatus from public exports

Prevents: out-of-sequence SCSI commands, double-init, wrong firmware writes.
2026-04-08 21:41:57 -07:00
MattJackson 0b19154bd3 Clean pipeline: one open, one init, no double-init
- Removed open_unlocked() — open() is the only entry
- Removed redundant init() call from open_title()
- init() called once in open(), handles everything
- Each function does one thing: open→init→scan→read
2026-04-08 21:35:48 -07:00
MattJackson ecc6cd9f6b Fix init() hang: make status() non-fatal, single attempt
status() (sub_cmd 0x13) returns ILLEGAL REQUEST on some drives.
Was retrying 6× with 30s timeouts = 180s hang during init().

Steps 1-10 all pass on hardware:
 unlock: OK, load_firmware: OK, calibrate: OK,
 register_a: OK, register_b: OK
Only status fails — not needed for reads.
2026-04-08 21:27:34 -07:00
MattJackson 71a21b5c6f Fix PGS subtitle misclassified as audio in STN parsing
When MPLS STN table parsing drifts (disc-specific alignment issue),
a PGS subtitle entry (coding_type 0x90/0x91) can appear in the audio
stream section. Previously this showed as garbled "ng PGS 5.1" audio.

Fix: guard in stream builder checks if audio-typed streams have
subtitle codecs and reclassifies them as subtitles.

Also: unknown stream types now filtered out (filter_map) instead of
creating fake Video entries that showed as "?" in output.

Tested on V for Vendetta BD — "ng PGS 5.1" gone, clean output.
2026-04-08 21:14:39 -07:00
MattJackson 79b1b4d5b5 Bump to v0.5.0 2026-04-08 20:53:54 -07:00
MattJackson 85b38de4a7 Update profiles.json: 206 drives with complete data
Generated by: profile generator --profiles sdf0.bin keys.json --drive-db drive_profiles.json
206 profiles (66 A + 140 B), all with identity from brute-force dictionary.
No manual merging. One tool, three inputs, complete output.
2026-04-08 20:51:15 -07:00
MattJackson 686bf91bb5 Update profiles.json: 206 drives
New profile format includes all -verified fields:
- ld_microcode (base64): firmware payload for WRITE_BUFFER/MODE_SELECT
- hardware_register_a/b_cdb: pre-built 10-byte SCSI CDBs
- drive_nominal_speed_cdb: calibration SET_CD_SPEED
- speed_zone_table, speed_calc_table: operation speed constants
- unlock_init_value, unlock_response_size_minus_init
- drive_signature: per-drive unlock check

206 profiles (66 mt1959_a + 140 mt1959_b), 661KB.
Identity fields merged from drive database.
All 32 tests pass.
2026-04-08 20:46:40 -07:00
MattJackson f07f8210c3 Fix variant B firmware upload: MODE SELECT not WRITE_BUFFER
B firmware upload () byte-level verification reveals:
- Step 1: MODE SELECT (0x55), NOT WRITE_BUFFER — sends 2496 bytes (0x9C0)
- Step 2: Check result == 2
- Step 3: READ_BUFFER mode=6 offset=0x3000 (16B firmware metadata)
- Step 4: WRITE_BUFFER mode=6 (16B from fw_write_data)
- Step 5: Vendor verify CDB (0xF1 opcode from blob)
- Step 6: do_unlock × 5 retries + 1 confirmation

Key differences from A:
- A uses WRITE_BUFFER (0x3B), B uses MODE SELECT (0x55)
- A sends 1888 bytes, B sends 2496 bytes
- B has extra READ metadata + WRITE 16B + vendor verify steps
- B retries unlock 5 times (A does 2)

Added profile fields: fw_write_data (16B), verify_cdb (10B) for B-only.
2026-04-08 20:40:37 -07:00
MattJackson 4153d23652 mt1959: separate A/B firmware upload paths
A (): single WRITE_BUFFER → verify 0x45 → unlock×2
B (): WRITE handshake → READ 0x3000 → WRITE 16B → verify → unlock×5

9/10 handlers are identical A/B. Only load_firmware has different logic.
Both paths end with do_unlock() — firmware upload is a prerequisite for
unlock, not a substitute. init() tries unlock first, falls back to
load_firmware only on failure (cold boot).
2026-04-08 20:35:43 -07:00
MattJackson 760bab0893 mt1959.rs: complete platform implementation
Every handler traced instruction-by-instruction from operation: do_unlock with configurable response size
operation: WRITE_BUFFER + verify buf=0x45 + unlock×2
operation: do_unlock → validate → send pre-built CDB → [4:20]
operation: same with CDB B
operation: init → scan 0x0000-0x5800 → build table → triple speed
operation: ↔x86 VM only (host_write 16B), no SCSI
operation: do_unlock → validate → probe 0x13 → check sig → features
operation: 3 paths by param count (1/5/9), dynamic READ_BUFFER
operation: search 64-entry table → position probe →
 set_cd_speed_max → custom SET_CD_SPEED with matched value
operation: ↔x86 VM only (host_read 8B), no SCSI

init() matches x86 dispatch exactly:
 Phase 1: unlock → [load_fw] × 6
 Phase 2: calibrate × 6
 Phase 3: probe (drive info)
 Phase 4: register A + B × 5
 Phase 5: status × 6

Handlers 5/9 are VM communication (no SCSI equivalent in Rust).
All other handlers send real SCSI commands.
2026-04-08 20:30:46 -07:00
MattJackson ce8ddb48bb DriveProfile with all per-drive fields, drive.rs uses init() as single entry
DriveProfile now has every field traced from firmware:
- drive_signature, unlock_init_value, unlock_response_size_minus_init
- ld_microcode (base64, ~1888B firmware payload)
- hardware_register_a_cdb, hardware_register_b_cdb (10B pre-built CDBs)
- drive_nominal_speed_cdb (12B calibration speed)
- speed_zone_table (28B), speed_calc_table (25B)

drive.rs simplified:
- open() calls init() instead of unlock()
- init() is the ONLY entry point — handles full dispatch sequence internally
- Removed read_config, read_register, maintain_speed, read_sectors from public API
- Added set_read_speed() for per-zone speed during content reads
- disc.rs updated to call init() instead of unlock()

Compiles clean, all tests pass.
2026-04-08 20:15:25 -07:00
MattJackson 377cbe0aec Rewrite mt1959.rs: complete platform driver with full profile support
Complete rewrite of MT1959 platform driver:
- All 10 handlers implemented matching firmware logic 1:1
- load_firmware(): WRITE_BUFFER ld_microcode on cold boot
- calibrate(): full zone probe + speed table + triple SET_CD_SPEED
- init(): x86 dispatch sequence (unlock → fw × 6, calibrate × 6)
- read_register_a/b(): use pre-built CDBs from profile
- set_read_speed(): speed table lookup per zone
- status(), probe(), keepalive(), timing()

Platform trait updated:
- Renamed read_config → load_firmware (matches actual function)
- Added init() for full x86 dispatch sequence
- Renamed read_sectors → set_read_speed (operation sets speed, not reads)
- Split read_register into read_register_a/b (separate CDBs)

Profile fields used:
- drive_signature, unlock_init_value, unlock_response_size_minus_init
- ld_microcode (1888B firmware payload)
- hardware_register_a_cdb, hardware_register_b_cdb (pre-built CDBs)
- drive_nominal_speed_cdb (calibration triple-play)
- speed_zone_table, speed_calc_table (operation lookups)
2026-04-08 20:11:26 -07:00
MattJackson a4c10887cf Make unlock non-fatal in open_title — BD discs work without it
BD drives reject the MediaTek unlock command (sense 0x05).
Unlock is only needed for UHD raw access. Standard BD reads
work with standard READ(10) without vendor unlock.
2026-04-08 17:30:09 -07:00
MattJackson 3fcab4d8d9 Strip to bare minimum for speed test: no calibration, no maintain_speed
Back to basics: open, unlock, SET CD SPEED max, read.
Remove all calibration probes, register reads, maintain_speed calls.
This is closest to the build that hit 17 MB/s earlier.

Also: drive discovery moved to libfreemkv (find_drive, resolve_device),
AACS via UDF only, clean pipeline, sg device support.
2026-04-08 15:46:42 -07:00
MattJackson f46d9706eb Fix rip: extent LBA offset, u16 truncation, batch reads, read_content
- Fixed: extents were relative to m2ts file, not absolute disc LBAs
- Fixed: u16 truncation of remaining sector count (13M → 36!)
- Added: UdfFs::file_start_lba() for m2ts LBA lookup
- Added: DriveSession::read_content() with 30s timeout for bulk reads
- Added: SET CD SPEED 0xFFFF on title open
- Added: adaptive batch reading (96→48→3 on error, ramp back up)
- Rip working end-to-end: scan → AACS → decrypt → write
2026-04-08 10:12:07 -07:00
MattJackson 9ecbf0ada5 Clean warning, AACS 2.0 full handshake + P-256 2026-04-08 08:27:13 -07:00
MattJackson 82746ed592 AACS 2.0: full P-256 handshake, HC2 parsing, stubbed for credentials
- Full aacs2_authenticate_p256(): AGID → P-256 cert exchange → ECDSA
 signatures → ECDH bus key. Complete SCSI payload format (132-byte
 certs, 128-byte key+sig).
- Falls back: tries AACS 1.0 first, P-256 only if drive rejects v1.
- HC2 KEYDB parsing: | HC2 | HOST_PRIV_KEY 0x... | HOST_CERT 0x...
 (32-byte private key, 132-byte certificate)
- P-256 key pair generation for ephemeral handshake
- AACS 2.0 cert verification with LA public key
- Waiting on: AACS 2.0 host credentials (HC2 entry in KEYDB)
- SKB: detected (use_skb_mkb flag) but not processed — VUK from KEYDB
 still works for all discs

32 tests passing.
2026-04-08 08:03:40 -07:00
MattJackson da0eeae6a7 AACS 2.0: P-256 curve, SHA-256 ECDSA, bus decryption pipeline
- Added P-256/secp256r1 curve constants
- Added AACS 2.0 LA public key for drive cert verification
- Implemented P-256 ECDSA sign/verify (SHA-256)
- Implemented P-256 ECDH for bus key derivation
- Added aacs2_authenticate() entry point (falls back to AACS 1.0)
- Extended HostCert with optional v2 credentials
- Added sha2 dependency
- Added keydb updater (HTTP GET via raw TCP, zip/gzip extraction)
- 4 new P-256 tests: generator, order, ECDSA, ECDH — all passing
- 32 total tests passing
2026-04-07 21:30:09 -07:00
MattJackson 262f9a7f8d Add keydb updater, update README with labels + multi-lingual + real output 2026-04-07 21:21:18 -07:00
MattJackson ca732d57e7 Bump to v0.4.1 2026-04-07 21:01:31 -07:00
MattJackson f357e0fe26 README: update dep version to 0.4, update architecture 2026-04-07 20:57:32 -07:00
MattJackson 976aeb4848 Labels: catch panics — never break disc scan 2026-04-07 20:36:26 -07:00
MattJackson 2bea69c7d9 Release v0.4.0: labels rewrite, eject, capture fix 2026-04-07 20:35:11 -07:00
MattJackson 0bb815ca22 Restructure labels: detect-then-parse, named parsers, raw disc data
Architecture:
- Each BD-J format in own file: paramount.rs, criterion.rs, pixelogic.rs, ctrm.rs
- Standard interface: detect() → bool, parse() → Option<Vec<StreamLabel>>
- PARSERS array in mod.rs — drop in a new parser with one line
- Shared vocab.rs for BD spec codec names only (MLP→TrueHD, AC3→Dolby Digital)
- All other label data passes through raw from disc — no guessing

Changes:
- New: paramount.rs (playlists.xml — Paramount/onQ format)
- Renamed: bluray_project.rs → pixelogic.rs
- Renamed: stream_properties.rs → criterion.rs
- Merged: language_streams.rs + menu_base.rs → ctrm.rs
- Removed: jar module (superseded by labels), dead apply functions
- Added: DriveSession::eject() with PREVENT ALLOW MEDIUM REMOVAL
- Added: DiscRegion enum (Free/BluRay/Dvd)
- Fixed: capture sector ranges now include all files (only skip STREAM/)
- Renamed: StreamLabel.region → variant (not a BD spec field)
2026-04-07 20:29:44 -07:00
MattJackson aa188751e4 Simplify labels API: one call, labels on streams or nothing
labels::apply(session, udf, titles) does everything internally.
Disc::scan() is one line: crate::labels::apply().
No intermediate variables, no fallback logic in disc.rs.

App reads disc.titles[].streams — labels already applied if
disc had config files, empty otherwise. MPLS data always there.
2026-04-07 18:46:29 -07:00
MattJackson ba44f7d928 Add labels module: 4 disc file parsers for stream labels
src/labels/ with 4 parsers tried in order:
1. language_streams.txt (Warner CTRM CSV)
2. menu_base.prop (Warner CTRM properties)
3. streamproperties.xml + playbackconfig.xml (Criterion XML)
4. bluray_project.bin (Pixelogic binary tokens)

Disc::scan() calls labels::extract() → apply_disc_labels().
If no disc files found, streams keep MPLS data as-is.
No JAR fallback — disc files or nothing.

Covers 4/8 discs with JARs (Dunkirk UHD, V for Vendetta BD,
Being There, Barbie). Remaining 3 (Civil War, Dune, V for
Vendetta UHD) have no disc config files.
2026-04-07 18:42:33 -07:00
MattJackson 8a95787426 JAR bytecode tracer POC — extract display names from BD-J enum classes
Traces <clinit> bytecode to find ldc/putstatic pairs that map
enum field names to display name strings. Pattern: new X, dup,
ldc "English", invokespecial X.<init>, putstatic X.a.

Proven on Dune UHD JAR: aw.a="English", aw.b="French", etc.
Next: trace mapping class (ISO code → enum field) to complete the chain.
2026-04-07 16:54:30 -07:00
MattJackson aebe6a256b Fix JAR label matching: by language+codec for label format, by index for TextField
Labels like eng_MLP_ now match to the correct stream by checking language
and codec hint against the stream's properties. Fixes Barbie where labels
were swapped (TrueHD label was on DD stream). TextField format (Civil War)
still uses index matching.

TrackLabel now carries language + codec_hint for structured matching.
2026-04-07 16:27:47 -07:00
MattJackson 791b832f25 Rewrite JAR parser: multi-format with TextField support
- Extract all class strings once, try format parsers in chain
- Format 1: TextField,Audio{N} (A24/Lionsgate) — fixes Civil War
- Format 2: eng_MLP_ label strings (Warner UHD) — Barbie, Dune
- Format 3: playlist-only (MAIN_FEATURE etc.)
- Simplified TrackLabel to just description + raw
- 5 unit tests for both formats
- Tested on 12 disc captures: Civil War now gets 3 audio + 2 sub labels

Known issues: Barbie labels swapped, Dune returns 0 labels (different format?)
2026-04-07 16:23:14 -07:00
MattJackson 580ce2277d Add serial_number to DriveId from GET_CONFIG 0x0108 2026-04-07 16:14:55 -07:00
MattJackson 8a556c5c63 Merge JAR labels into streams during Disc::scan()
apply_jar_labels() matches JAR audio/subtitle labels to streams by
position and sets AudioStream.label. Apps read labels directly from
streams instead of doing their own JAR matching.
2026-04-07 16:10:43 -07:00
MattJackson 520a3f912c Refactor Stream to enum with typed variants (Video, Audio, Subtitle)
Each stream type has only its relevant fields. No more HDR on audio
or channels on video. Added SubtitleStream.forced field (TODO: parse).
Removed display helpers from lib (belongs in CLI).
2026-04-07 16:08:45 -07:00
MattJackson 6f0eea48fa Add Clip struct to Title — expose clip references for playlist analysis
Title.clips[] contains clip_id, in/out times, duration, source_packets.
Apps can detect fake/scrambled playlists by checking unique clip count
vs total (253 clips referencing 2 unique = fake). Removed clip_count
field (use clips.len()).
2026-04-07 16:03:56 -07:00
MattJackson b01010f2d1 Fix title sizes: use pkt_count*192, fix metadata partition range
- Size from source_packet_count * 192 (matches libbluray approach)
- metadata_sector_ranges() uses actual metadata partition size from ICB
 instead of arbitrary +256 margin (fixes Top Gun missing CLIPINF)
- Store metadata_sectors in UdfFs
- Fix CLPI CPI bit-packed field parsing (partial, EP map still needs work)
2026-04-07 15:49:53 -07:00
MattJackson fd443e3a2f Add disc title, format, streams to Disc::scan() — move logic from CLI to lib
- Disc.volume_id: UDF Volume ID from PVD (always present)
- Disc.meta_title: from META/DL/bdmt_eng.xml (falls back to other languages)
- Disc.format: UHD/BluRay/DVD detected from video codec
- Disc.capacity_bytes, Disc.layers
- Disc.jar_labels: extracted from BDMV/JAR
- Fixed MPLS STN parsing: 16-byte header (was 8), proper stream entry offsets
- Streams now include: HDR, color space, Dolby Vision EL, secondary audio/video
- parse_dstring() for UDF d-string fields
- wait_ready() polls TEST UNIT READY before unlock

Tested on 12 disc captures — all return correct titles, streams, format.
2026-04-07 15:36:05 -07:00
MattJackson 8d85a9d778 Add metadata_sector_ranges() and max_metadata_sector() to UdfFs
Walk UDF tree to discover sector ranges for all metadata files
(MPLS, CLPI, JAR, AACS certs, etc). Skips STREAM/, BACKUP/,
DUPLICATE/, MKB_RO.inf, ContentHash. Exposes partition_start()
and metadata_start() getters. Used by bdemu smart capture.
2026-04-07 14:43:42 -07:00
MattJackson 237528b385 Add macOS SCSI support via IOKit SCSITaskDeviceInterface
IOKit backend for macOS optical drives. Accepts BSD device paths
(/dev/disk2), walks IORegistry to find authoring device, sends
SCSI commands through SCSITaskDeviceInterface COM vtable.
2026-04-07 13:15:06 -07:00
MattJackson 492a2966c0 Add docs/README.md — reading order and index for all documentation
Start with disc-to-rip.md (the big picture), then architecture,
drive access, disc formats (UDF/MPLS/CLPI), and AACS encryption.
2026-04-07 12:36:29 -07:00
MattJackson 5fa6b064da Remove standalone binaries — functionality lives in freemkv CLI and tests
Deleted:
- freemkv_info.rs — duplicate of freemkv drive-info CLI command
- freemkv_test.rs — duplicate of bdemu capture-disc
- aacs_test.rs — covered by inline #[test] functions (31 tests)

libfreemkv is a library. CLI tools belong in the freemkv repo.
Dev/debug tools belong in (internal).
2026-04-07 12:34:10 -07:00
193 changed files with 117967 additions and 8716 deletions
+35 -5
View File
@@ -6,16 +6,46 @@ on:
pull_request: pull_request:
jobs: jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
with:
components: clippy, rustfmt
- uses: Swatinem/rust-cache@v2
- run: cargo fmt --check
# libfreemkv is a library — Cargo.lock is gitignored. --locked
# would always fail on a fresh runner because there's no committed
# lockfile to lock against. The binary crates (freemkv, autorip,
# bdemu) track Cargo.lock and DO use --locked.
- run: cargo clippy -- -D warnings
test: test:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable - uses: dtolnay/rust-toolchain@1.86.0
- run: cargo test - uses: Swatinem/rust-cache@v2
- run: cargo test --tests
check-macos: check-macos:
runs-on: macos-latest runs-on: macos-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable - uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
- run: cargo check - run: cargo check
check-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
# Build the tests (not just `cargo check`): catches errors in test
# code and forces full codegen of the Windows-only SPTI transport
# (src/scsi/windows.rs), which never compiles on the Linux/macOS dev
# hosts. We don't `cargo test` here — the suite needs no drive but the
# extra build is the value; running tests is covered by the Linux job.
- run: cargo build --tests
+35
View File
@@ -0,0 +1,35 @@
name: leak-guard
# Self-contained public-repo leak gate. Public CI cannot reach the private
# tooling, so this encodes only the generic net: internal-infra references,
# tracked CLAUDE.md/.claude paths, and AI-attribution in commit messages.
# No project-specific reverse-engineering vocabulary lives here.
on: [push, pull_request]
jobs:
leak-guard:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Compute commit range
id: range
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
base="${{ github.event.pull_request.base.sha }}"
head="${{ github.event.pull_request.head.sha }}"
echo "range=$base..$head" >> "$GITHUB_OUTPUT"
else
before="${{ github.event.before }}"
after="${{ github.sha }}"
# New branch / first push: github.event.before is all-zeros.
if [ -z "$before" ] || [ "$before" = "0000000000000000000000000000000000000000" ]; then
echo "range=$after" >> "$GITHUB_OUTPUT"
else
echo "range=$before..$after" >> "$GITHUB_OUTPUT"
fi
fi
- name: Run leak-guard
run: bash ci/leak-guard.sh "${{ steps.range.outputs.range }}"
+33 -9
View File
@@ -12,7 +12,7 @@ jobs:
verify: verify:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v5
- name: Verify Cargo.toml version matches tag - name: Verify Cargo.toml version matches tag
run: | run: |
CARGO_VER="v$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')" CARGO_VER="v$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')"
@@ -22,30 +22,54 @@ jobs:
fi fi
echo "Version match: $CARGO_VER" echo "Version match: $CARGO_VER"
# Tests run as a PARALLEL TRIPWIRE: they fail the run if they fail, but the
# publish/release jobs do NOT `needs:` this job. The tag decision was already
# gated by the local precommit (same Rust 1.86, same commit). Binary consumers
# (freemkv/autorip/bdemu) git-tag-pin libfreemkv and therefore start building
# the instant this tag exists — so this test job and the crates.io publish
# below must NOT sit on their critical path.
test: test:
needs: verify needs: verify
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable - uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
# libfreemkv is a library — Cargo.lock isn't tracked, so --locked
# would always fail (no lockfile to lock against on a fresh runner).
- run: cargo test - run: cargo test
# crates.io publish is an INDEPENDENT job: it serves EXTERNAL consumers only.
# The freemkv binaries no longer depend on it (they git-tag-pin libfreemkv via
# a committed [patch.crates-io]), so this publish runs in parallel with their
# release builds rather than gating them. It `needs: [verify, test]` so a
# failing test suite still blocks publication to crates.io — external
# consumers who `cargo add libfreemkv` must never receive a release whose
# tests were failing. (The two upstream jobs run in parallel, so this gate
# does not serialize publish behind test beyond their own completion.)
publish: publish:
needs: test needs: [verify, test]
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable - uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
# --no-verify: CI already compiled this exact commit (in the `test` job
# and on every push via ci.yml). cargo publish's default re-verify does a
# full cold release build of the packaged tarball, which here is pure
# redundant work (~a cold lib build). Skip it.
- name: Publish to crates.io - name: Publish to crates.io
run: cargo publish run: cargo publish --no-verify
env: env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
release: release:
needs: test # Only needs `verify`; the GitHub Release can be cut as soon as the version
# check passes, in parallel with test + publish.
needs: verify
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v5
- name: Create GitHub Release - name: Create GitHub Release
uses: softprops/action-gh-release@v2 uses: softprops/action-gh-release@v2
with: with:
+2 -1
View File
@@ -11,9 +11,10 @@ jobs:
update: update:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v5
with: with:
ref: main ref: main
token: ${{ secrets.ORG_DISPATCH_TOKEN }}
- name: Update version in README - name: Update version in README
run: | run: |
+11
View File
@@ -3,3 +3,14 @@ Cargo.lock
*.swp *.swp
*.swo *.swo
.DS_Store .DS_Store
.cargo/
# session scratch — never track (may contain RE breadcrumbs)
scratch/
# stray local build artifact
/rust_out
# internal agent context — never publish (path AND dir; leak-guard blocks both)
CLAUDE.md
.claude/
+646
View File
@@ -0,0 +1,646 @@
# Changelog
## [1.2.1] — 2026-07-02
### Fixed
- **DVD DTS audio no longer muxes with non-monotonic timestamps.** A DVD
Program Stream packs several DTS core frames into one PES packet; the parser
stamped every access unit with that single PES timestamp and no per-frame
duration, so consecutive frames collided on one PTS and a strict decode/remux
(ffmpeg) rejected the track — `non monotonically increasing dts to muxer`.
The DTS parser now derives each core frame's duration from its header
(`(NBLKS+1)*32` samples ÷ the `SFREQ` sample rate) and re-bases to each PES's
own container timestamp, advancing by a frame duration only *within* a single
PES — so the track stays monotonic and does not drift past its real length on
a feature-long title. The UHD DTS-HD MA path (one access unit per PES) is
unaffected: each unit keeps its own PES timestamp, preserving the 1.2.0 per-PES
attribution. Completes the DVD DTS fix begun in 1.2.0 (which corrected the
silent-track routing, exposing this timing bug). Note: genuinely corrupt
source DTS frames — valid framing, bad audio blocks — are passed through
faithfully; freemkv never fabricates or drops audio it can't prove is bad.
## [1.2.0] — 2026-07-01
### Breaking
The disc's AACS version is now carried through the key-resolution path as the
single source of truth for the `Unit_Key_RO` stride (AACS-1.0 = 48-byte,
AACS-2.x = 64-byte), so keys are always read at the disc's own layout. That
threaded one new value through three public signatures. In-tree consumers
(`freemkv`, `autorip`, `freemkv-keysources`) are updated; external callers must
adjust:
- **`DiscInputs` gains a `version: u8` field** (between `volume_id` and `mkb`).
Code constructing it with a struct literal must add the field. It is normally
obtained from `Disc::inputs()`, not constructed by hand.
- **`keysource::DiscInputsCtx::new` takes one argument, not two** — the version
is now read from `inputs.version` (`new(inputs)` instead of
`new(inputs, version)`).
- **`disc::read_aacs_inputs` / `read_aacs_inputs_from_drive` return a 3-tuple**
`(inf, mkb, version)` instead of `(inf, mkb)`.
- **`PassProgress` is no longer `Copy` and gains a `located: LocatedProgress`
field.** It now carries a `Vec` (the rendered bad-range drilldown), so it's
`Clone` only — still built once per throttled emission and passed by reference
to `Progress::report`. Struct-literal constructors must add the field (empty:
`located: Default::default()`). New public types `LocatedRange` /
`LocatedProgress`.
These are source-breaking for external crates.io consumers. Shipped under a
minor bump (1.2.0): libfreemkv's surface is not yet frozen and the only known
consumers are the in-tree toolchain crates.
### Added
- **Pass-N marginal-sector recovery specialists.** The patch pass gained a
roster of parameterized recovery techniques — read speed (max/min), cache
bypass (FUA), and traversal (linear fwd/rev, bisect, cache-prime, oscillate,
per-sector speed-sweep) — each targeting a distinct physical failure mode of
marginal media. A per-rip **decayed (EWMA) scorecard** grades every technique
by its recent recovery rate and re-orders them best-first, so the engine
hardcodes no conclusion: a technique that fits *this* disc floats to the front
and one that doesn't self-deprioritises (but is never dropped). Every read is
wedge-safe and deadline-bounded; the existing fast/deep recovery behavior is
unchanged (the specialists are additive, tried only on the hardened residue).
- **Opt-in flat-pool recovery scheduler (`FREEMKV_PATCH_FLAT`).** Collapses the
breadth-first recovery tiers into one flat pool so every technique gets a shot
at each bad range immediately, scorecard-ordered — a data-driven bandit for a
hardened residual (e.g. a late resume) where the tiered ladder would spend a
long time on cheap techniques before reaching the specialists. Unset keeps the
proven tier ladder as the default.
- **`PassProgress` is the complete, mapfile-free progress contract.** Every
emission now carries the fully-rendered "where is the damage" drilldown
(`located`): the bad ranges annotated with chapter + movie-time offset, the
main-feature at-risk time, the section count and the largest gap — computed by
the library from its in-memory mapfile + title. A client (autorip, a future
GUI/CLI) renders the disc map + at-risk time straight from it and never parses
the mapfile, so a mapfile→mapdb change is invisible to clients. Adds
`disc::locate_ranges`, the one-shot `disc::progress_snapshot_from_mapfile`
(builds a snapshot from a mapfile on disk so a boundary/verdict paint stays
mapfile-free client-side), and `consts::MILLIS_PER_SEC`.
- **`PatchOptions::fast_capture` — breadth-first patch recovery.** A fast-capture
pass reads each bad range once at the full batch and leaves every failed block
`NonTrimmed` for a later pass — no bisect, no re-read, no per-sector grind — so
a first retry pass grabs the readable blocks (a sweep's good skip-ahead
overshoot) of EVERY range before any single range's slow per-sector recovery.
No data is dropped: a failed block stays `NonTrimmed` (retried by a granular
pass), never `Unreadable`. A transport fault still aborts. `Disc::copy`'s
internal patch leaves it `false` (single-call full recovery).
- **Mux loss concealment — a logged gap still produces a decode-clean file.**
When a unit genuinely cannot be decrypted on the mux read path (a key the disc
never yielded, after the rip's own decrypt-verify already failed loud and
re-read), the mux no longer passes ciphertext downstream or emits a broken
frame. The undecryptable aligned unit is concealed as NULL transport-stream
packets (PID 0x1FFF, invisible to every real stream), and the codec layer
**drops forward to the next keyframe** so no frame with a dangling reference
reaches the muxer. An ffmpeg deep scan of the result is clean — no missing
references, no partial frames. The loss is tallied and logged, never silently
dropped, and the mux always completes. Audio and subtitle tracks have no
cross-frame references, so only the directly-affected frames are dropped there.
Decrypt-verify remains a **rip** gate (fail loud → re-read), never a mux gate.
- **`Disc::unlocker_matrix()` — registry-driven unlocker did-work report.** Returns
each registered unlocker's name alongside a `did_work` flag recording whether it
performed authentication steps during the current rip. Callers (autorip, the CLI)
surface this so an operator can confirm at a glance which unlock paths —
LibreDrive firmware, AACS, CSS — actually ran, with no hardcoded names on the
caller side.
### Changed
- **One hex parser.** All hex parsing (keys, IDs, key-source inputs) routes
through a single `libfreemkv::hex` parser instead of several ad-hoc decoders,
so length/odd-nibble/invalid-digit handling is identical everywhere.
- **Robust encrypted-unit sampling + a single MKB framing walker.** Up-front
AACS sampling tolerates content layouts that previously yielded too few
encrypted units to resolve a key, and the Media Key Block is now walked by one
framing routine shared across the in-band and out-of-band readers (no
divergent record-stride logic). AACS resolution hardened around these paths.
- **One reader, one `DiscInputs`.** `Disc::inputs()` is now the single, complete
source of a disc's AACS inputs (inf, MKB, VID, disc_hash, version), and
`read_aacs_inputs*` returns the version alongside inf+MKB. Both the CLI and
autorip resolve through `Disc::inputs()`; the duplicate out-of-band readers
(autorip's `key_files()`/`volume_id()`) and the stale mapfile-VID read are
removed. AACS file paths and the AACS major versions are now named constants
(`aacs::PATH_*`, `aacs::AACS_MAJOR_*`, `AacsVersion::major`/`from_major`) so a
fallback or stride change lives in exactly one place.
- **Pass-N recovery rebuilt as a bounded, never-hang handler chain.** The 1.1.0
patch loop retried each bad range sector-by-sector until a per-range budget was
exhausted, with no escape from a wedged drive short of the watchdog firing after
tens of minutes. 1.2.0 replaces that with a two-tier handler chain dispatched
breadth-first, largest bad range first:
- **Jump** (lead tier): reads each range in large forward-skipping batches to
quickly locate readable islands — clearing a multi-gigabyte dead spot in
seconds rather than sector-by-sector.
- **Bisect** (trailing tier): binary-searches the boundaries of each remaining
bad block, converging to within a single sector of the last-readable LBA.
Boundary-probe reads are exempt from the early-yield stall so the boundary
walk always completes.
- **Handler scorecard**: handlers that make progress stay at the front of the
rotation per rip; an idle handler is ranked last so proven performers lead.
- **Wedge detection**: a pass-level streak counter tracks consecutive
wedge-family senses (HARDWARE ERROR / ILLEGAL REQUEST) across section
boundaries. At the threshold the pass aborts and a soft un-wedge
(`Drive::spin_cycle()` — START STOP UNIT, no eject) runs before the next retry
pass, instead of grinding at near-zero throughput until the pass watchdog
fires.
No data is dropped: a block that neither handler recovers in a pass stays
`NonTrimmed` for the next pass.
### Fixed
- **DVD DTS/LPCM audio tracks no longer mux silent.** On DVD-Video the
`private_stream_1` sub-stream id's low nibble is the audio-stream *number*
(shared across codecs), not a per-codec ordinal. A DTS or LPCM track that
wasn't the disc's first audio stream got a sub-id one too low, so the demux
routing key (`0xBD00 | sub_id`) never matched and every packet was dropped —
the track appeared in the container but played silent (AC-3 at position 0
worked by coincidence). Audio sub-stream ids are now assigned by positional
stream number, so a DTS 5.0 track after an AC-3 5.1 track routes correctly.
- **ISO mux no longer drops real video at content-fragment tails.** A title's
encrypted content can end mid-AACS-unit, with the disc zero-padding the rest
of the 6144-byte aligned unit to the next fragment. The decrypt-verify
demanded the TS sync byte on *all 32* source packets, so it rejected such a
tail unit over its legitimate padding — discarding the real video packets it
contained. On a flawless rip this surfaced as a small phantom "loss" at mux
(and, once retries were exhausted, a truncated MKV). Unit acceptance is now
**padding-aware**: only packets whose *source* (pre-decrypt) bytes are
non-zero must restore their TS sync; the zero padding is excluded from the
check and emitted as clean zeros. A full content unit still requires all 32
(unchanged — no wrong-key relaxation), and a unit whose *non-zero* tail fails
to decrypt is still rejected as a genuine bad read.
- **ISO online key resolution now sends the Media Key Block.** Capturing a
disc's AACS inputs at scan read the MKB with a full `read_file` of the
~128 MiB `MKB_RO`/`MKB_RW` allocation, which fails on file-backed readers —
leaving the MKB empty, so `Disc::inputs()` shipped `mkb=0` to an online key
service and the request was rejected (no key → no decrypt). Scan now reads the
MKB through the same bounded prefix-grow + trim reader as the out-of-band
path, so `Disc::inputs()` is the single complete source of AACS inputs — one
reader for every caller.
- **Read-time key-fetch parses `Unit_Key_RO.inf` at the disc's own AACS stride.**
The on-demand fetch (for a CPS unit not sampled up front) hardcoded the V20
64-byte stride, so an AACS-1.0 (V10) disc whose key arrived as a VUK derived
the wrong unit keys. `DiscInputs` now carries the disc's `version`, and the
context parses at the matching stride — the disc is the single source of truth
for its own stride (no separate version argument to drift).
- **A dry key-fetch for one unit no longer blocks fetching a different unit.**
A global "fetch spent" latch meant that once the key service returned nothing
for one CPS unit's ciphertext, no further unit was ever asked — so a multi-CPS
disc could strand a unit whose key the service *would* have served. Replaced
with a per-unit "already-asked-dry" set (still bounded by the fetch budget).
- **`verify::push_ranges` uses saturating arithmetic** so a corrupt-disc LBA near
`u32::MAX` can't panic (matches `udf::merge_ranges`).
- **Audio no longer corrupts at a stream discontinuity.** At a transport-stream
discontinuity — a continuity-counter break, an adaptation-field
discontinuity_indicator, or a concealed-loss gap — the AC-3 / DTS / TrueHD
parsers held a *truncated* partial access unit and spliced the post-gap bytes
onto it, manufacturing a corrupt frame (ffmpeg "exponent out of range" /
"Failed to decode block code(s)" / "Invalid data found") and, for TrueHD, a
non-monotonic timestamp band on multi-segment titles. The video path already
resynced via the keyframe gate; the audio parsers now do too — on a
discontinuity they drop the un-completable partial and resync on the next
syncword, rebasing the timestamp from the post-gap PES. A discontinuity becomes
a clean single-frame gap instead of a corrupt splice. Audio has no inter-frame
references, so dropping the truncated partial is the complete fix; the approach
matches FFmpeg's parser layer and GStreamer's `tsdemux`.
- **Drive-prep firmware unlock skipped for DVD discs.** An
`if disc_is_dvd() { return }` guard in `Drive::init()` (present since
1.0.0-rc.1) bypassed the entire drive-prep unlock step for DVDs. That unlock is
what removes riplock and raises the drive to maximum read speed — a drive-level,
disc-independent feature — so every DVD rip ran at riplock speed (~0.4× rated,
multi-hour ETA). The guard is removed; all disc types now go through the full
drive-prep sequence. UHD and Blu-ray were unaffected (they already ran through
the unlock path).
## [1.1.0]
### Added
- **Post-read decrypt-verify gate.** Every AACS unit read off the disc is now
buffered, re-aligned to its clip-file 6144-byte unit grid, and verified
(CPI flag → decrypt → strict all-32 TS-sync, matching libaacs `_verify_ts`)
before it is signed off as good. A unit that no held or freshly-fetched key
decrypts is treated exactly like a bad read — re-read by
the patch pass, terminal loss only if truly unrecoverable — closing the
"silent bad read" class where a sector reads OK but its ciphertext is subtly
wrong. **Fail-safe:** it only ever downgrades a unit it is *confident* is bad;
every uncertainty (no keys, a merely-missing key, an unread/zero-filled sector,
a non-AACS disc) leaves the read byte-for-byte as before. Gated by a
compile-time kill-switch (`POST_READ_VERIFY`), and container-pluggable (BD/UHD
transport stream today, with an HD-DVD program-stream seam in place).
- **Every error is now `Error: E<code> <message>`, with an Error Codes
reference.** User-facing errors show their code so you can look it up, and a
new **Error Codes** page lists every code with its message, cause, and next
steps. A contract test guarantees every error variant has a code, a message in
all seven languages, and a Codes-page entry. Messages are source-agnostic
("key source", never a specific database).
### Changed
- **AACS decrypt acceptance is now standards-strict.** A key is accepted only
when the decrypted unit has the TS sync byte on *all* 32 source packets
(libaacs `_verify_ts`), replacing a majority-vote heuristic where a wrong key
could coincidentally restore enough syncs to pass and silently corrupt a unit.
- keydb download/save moved out of the library into freemkv-keysources;
libfreemkv no longer has any keydb I/O (it already held no keys).
### Fixed
- **AACS content-certificate bus-encryption flag read from the wrong bit.** The
flag is bit 7 of byte 1 (libaacs `p[1] >> 7`) but was read as bit 0, so a
bus-encrypted disc parsed as *not* bus-encrypted — defeating the fail-loud
guard that refuses to decrypt bus-wrapped data to garbage when no bus key was
obtained. Also corrected the cc_id offset (byte 14) and the AACS2 type marker
(`0x10`). Confirmed against real retail content certificates.
- **DVD rips now start on the movie, not the disc menu.** A VTS title VOB's
start sector was read from the IFO as a VTS-relative pointer but used as an
absolute disc address, so a DVD title's read extents began `ifo_lba` sectors
too early — the rip opened on the disc's menu / VMGI region and only drifted
into the feature minutes later (Silence of the Lambs, for example, showed
several minutes of the main menu before the movie). The title VOB is now
rebased to its absolute on-disc location, so the rip begins at the first frame
of the feature. Aspect ratio and chapter timing were already correct; only the
starting sector was wrong. (Covered by a new absolute-placement regression
test.)
- **Container metadata correctness.** Unknown colorimetry now emits the CICP
"unspecified" code point (2) consistently across the MKV track and the FVI
sidecar (previously 0); PGS subtitle wipes use the NORMAL composition state
rather than a full epoch reset; and FVI source-byte offsets are written
within-sector per the format spec.
- **Multi-extent AACS alignment in `dir://` extraction.** AACS encrypts in
aligned units of 3 sectors (6 KiB), and the decrypt-on-read gate accepts a read
only when its LBA is unit-aligned against a base. The `dir://` file-tree
extractor set that base once, to the file's first extent. A fragmented file
(Long-AD / continuation-ICB allocation) has later extents starting at arbitrary
LBAs whose distance from the first extent is generally not a multiple of 3
sectors, so the first read of every later extent failed the gate, returned a
decrypt error, and the whole extent was written as a zero-filled hole — even
though the sectors were readable. The unit base is now re-anchored per extent
(matching the mux read paths), so each extent gates on its own unit grid.
Decryption math is unchanged. Same class as the rc.5.2 clip-anchor fix.
- **Distinct "no key" reasons.** When AACS key resolution has usable material
(device or processing keys) but cannot obtain the disc's Volume ID — needed to
derive the unit key — freemkv now reports a distinct "AACS Volume ID
unavailable" error (E7017) instead of collapsing it into the generic "no key"
error (E7022), which is now reserved for a genuine absence of any key material.
No key derivation or descramble logic changed — only the reason reported on a
resolution failure.
- **autorip keydb writes go to the right path.** Auto-download, daily refresh,
the "Update KEYDB" button, and the startup existence-check now resolve to the
service's config path (matching where reads look); they previously used the
CLI's executable-local default.
- **Crash-safety hardening** in `dir://` extraction and keydb writes (fsync of
files and parent directories around rename).
- **Windows-reserved filenames** (`CON`, `NUL`, `COM1`…) inside a disc's file
tree are safely renamed on extraction instead of aborting the walk.
- **`--version` now matches the build stamped into MKVs.** The CLI's `--version`
string and the `MuxingApp` / `WritingApp` fields written into every MKV now
derive from a single libfreemkv constant — the package version plus the git
short hash (e.g. `freemkv 1.1.0 (g835cc99)`). The muxer previously kept
its own copy of that string, so the two could drift; a binary and the files it
produces can no longer report different versions.
- **DTS-HD Master Audio: a false core-sync inside the lossless extension no
longer splits an audio frame.** A byte pattern in the extension substream that
resembled the `0x7FFE8001` core sync word could truncate the lossless
extension and produce decode errors on the affected frames. The extension
substream is now sized exactly from its header, so that pattern is skipped as
data.
- **TrueHD: decode timestamps no longer step backward.** In a case where the
source PES timing lagged the audio access-unit cadence, the muxed decode
timestamp could regress (non-monotonic-DTS warnings to the muxer); the running
timestamp is now clamped so it never goes backward.
### Tests
- 58 new tests across the toolchain (AACS key resolution, the unlocker seam, the
key sources, DVD/CSS, `dir://` routing, and autorip keydb resolution).
## [1.0.0-rc.5.3]
### Added
- **`dir://` output** — write a decrypted `VIDEO_TS` / `BDMV` file tree straight
from a disc or ISO instead of a single muxed file.
### Changed
- **Source-agnostic key errors** — decryption messages no longer assume a local
key database is *the* key source.
- **The default `keydb.cfg` location is next to the executable** (portable CLI);
the autorip service keeps its container path.
- **Simpler flags** — dropped `-k` (use `--keydb`) and removed `--device` (the
drive is named in the source URL, e.g. `disc:///dev/sgN`).
### Fixed
- **Fail loud on missing keys or bad input** instead of silently writing an
undecrypted file.
## [1.0.0-rc.5.2]
### Fixed
- **Reverted the rc.5.1 `DefaultDecodedFieldDuration` experiment for interlaced
SD-DVD.** rc.5.1 added a 20 ms `DefaultDecodedFieldDuration` field element to
the 576i/480i track header on the theory that Windows derives fps from it.
Captured evidence showed that element made Windows Explorer report 12.5 fps
(half) and MediaInfo flip the track to "Frame rate mode: Variable", while
MakeMKV's rip of the same disc omits it. The element is therefore no longer
written (`MkvTrack::video` now passes `field_duration_ns == 0`); the track
keeps `FlagInterlaced=1` + `FieldOrder=TFF` and the full-frame 40 ms
`DefaultDuration` (`1/DefaultDuration` = 25 fps), matching MakeMKV. How a given
player or shell handler chooses to display interlaced fps is not guaranteed.
- **Correct AC-3 audio track selected on DVDs with non-standard sub-stream
ordering.** freemkv assigned each declared audio stream a physical sub-stream
by ordinal (`0x80+n`), assuming the IFO's first stream lives at `0x80`. On
discs where the 5.1 main mix sits on a different sub-stream and `0x80` carries
a 2.0 down-mix (e.g. Silence of the Lambs), the 2.0 was muxed under a "5.1"
label. freemkv now probes each physical sub-stream's actual channel count from
the disc — scanning every AC-3 frame and taking the maximum, so a brief 2.0
logo bed at the feature head can't mask the real 5.1 — and routes each declared
stream onto the sub-stream that genuinely matches.
- **"Decryption failed" on large AACS Blu-ray titles fixed.** AACS encrypts in
aligned units of 3 sectors (6 KiB); the unit-alignment gate measured `lba % 3`
against absolute disc LBA 0, but the unit grid is actually anchored at each
clip's encrypted-region start. A clip whose start is not 3-sector-aligned had
its readable units wrongly rejected — failing the feature/large titles of some
discs while short clips passed. The gate is now clip-anchored.
- **Single-pass disc→MKV recovers marginal/transient sectors before failing.**
The direct-to-MKV path now gives the drive its full ECC recovery budget on a
bad sector (matching the multipass rip) instead of reporting a read failure a
multipass rip would have recovered.
- **4K decode glitches at non-seamless clip joins fixed (Top Gun class).**
Titles assembled from clips joined at non-seamless boundaries no longer drop
reference frames at the join ("Could not find ref" stutter); the splice
keyframe is rewritten so the decoder discards only the genuinely-dangling
leading pictures.
### Changed
- **`freemkv-keysources` is now a pure key lookup.** The encrypted content-sample
reader and the candidate-key resolution loop moved into libfreemkv (they read
the disc and validate keys — decryption mechanism, not lookup). A key source
now only looks a key up and hands it back. Downstream API: use
`libfreemkv::read_encrypted_units` / `libfreemkv::resolve_and_apply` (was
`freemkv_keysources::read_sample_units` / `…::resolve_and_apply`).
### Added
- **`--log-level 3` is now self-sufficient for MKV/opening-frame diagnosis.**
The diagnostic pass now (a) dumps the ACTUAL MKV `TrackEntry` elements written
per track (`tag=mkv.track`: FlagInterlaced, FieldOrder, DefaultDuration,
DefaultDecodedFieldDuration via field-duration, Display dims, codecPrivate as
hex) so the Windows-fps-class metadata is verifiable from a log alone, and
(b) captures the first ~100 coded frames per track (raw bytes) to a
`<output>.opening.bin` side file with a per-frame summary line
(`tag=mkv.opening.frame`: track, key/delta, size, PTS) so opening-GOP / menu
issues are diagnosable from a future log without the disc. Both are gated to
log-level 3; a normal run opens no side file and records nothing.
### Verified
- **DVD opening-GOP / still-frame open handling is correct (no change needed).**
The hypothesis that the opening pictures get the wrong (last-seen) sequence
header or have their PTS floored to t=0 was traced and ruled out: the
codecPrivate is the FIRST sequence header (read once at headers-ready, before
any later AU), DVD VOBU structure guarantees each title opens on a sequence
header + I-frame (no mid-GOP open), the parser back-anchors leading
still-frames to the disc's real timeline, and the muxer anchors its timestamp
base on the opening keyframe's real PTS so the t=0 floor never corrupts it.
Regression tests pin all three.
## [1.0.0-rc.5.1]
### Fixed
- **CSS reads unlocked on enforcing drives.** CSS-protected DVDs on
drives that enforce CSS authentication previously produced an empty MKV
at exit 0, or hung indefinitely. The read path now issues the bus-auth
handshake (`css::auth::unlock_css_reads`) to unlock scrambled-sector
reads before attempting any data transfer, so the drive gates lift
correctly.
- **Keyless title-key recovery always runs.** The Stevenson known-plaintext
attack (`css::crack_key` / `src/css/stevenson.rs`) now recovers the
title key even when the bus-auth scan detects a CSS drive, removing a
code path that fell through to locked reads on certain disc/drive
combinations. A wrong key still fails cleanly (confirmed by a sector
descramble check) rather than producing silent garbage.
- **Early bail on undecryptable discs.** When CSS authentication succeeds
but no valid title key can be recovered, the mux path now terminates
with a clear error code instead of writing an empty (or zero-byte)
output file.
- **DVD audio channel count from AC-3 bitstream.** The audio channel count
is now parsed from the AC-3 elementary-stream bitfield rather than from
the IFO audio attributes, so the reported channel count always matches the
actual muxed audio even when the IFO attribute disagrees. Passthrough only
— no downmix is performed. (Selecting the correct audio sub-stream on discs
with non-standard ordering is a separate item — see Known issues.)
- **Interlaced MKV frame rate on Windows.** Interlaced content (576i/480i)
now emits a `DefaultDecodedFieldDuration` element in the MKV track
header, which Windows Media Foundation and Explorer use to derive the
display frame rate. Without it, players reported an incorrect or zero
frame rate on interlaced tracks.
- **Per-track `BPS` bitrate tags populated.** The `BPS` tag is written for
each track so players and shell extensions (Windows Explorer, MPC-HC,
etc.) can display the per-stream bitrate without reading the full file.
- **Interlaced field order corrected to TFF.** 576i tracks were written
with a bottom-field-first (BFF) container flag that disagreed with the
top-field-first order carried in the MPEG-2 stream; the MKV `FieldOrder`
element now matches the stream (TFF) so deinterlacers use the correct
field parity.
- **DVD first-play menu no longer prepended to the feature.** The title
VOBS base sector was read from the VTS menu-VOBS pointer (`vtsm_vobs`,
offset 0xC0) instead of the title-VOBS pointer (`vtstt_vobs`, 0xC4), so on
a disc that authors a per-title menu the entire menu VOB — e.g. a studio
first-play "the parental level has been set, press yes" prompt — was
prepended to the movie and every cell extent shifted back. The rip now
opens on the feature's first frame.
### Changed
- **AACS handshake skipped on DVDs.** The AACS authentication sequence is
no longer attempted on DVD discs (it never applied to CSS-encrypted
media); attempting it on a DVD drive was a no-op at best and surfaced
spurious errors at worst.
### Added
- **Structured disc diagnostics at `--log-level 3`.** A new diagnostic
pass emits structured log events at INFO level when the log level is 3
or higher: DVD PGC/cell layout and IFO video/audio attributes; BD/UHD
playlist, clip, and AACS metadata. Provides a single-command snapshot
for diagnosing mux or authentication issues without instrumenting the
source.
- **Reduced per-operation log spam.** Mux-read and seek operations are
demoted to TRACE (were DEBUG); benign navigation-packet drops are
summarized as a single counter at the end of the title rather than
logged per-packet.
### Known issues
- **Wrong audio track on discs with non-standard substream ordering.**
Audio sub-stream ids are assigned by per-codec ordinal rather than read
from the IFO/PGC stream-number table, so a disc whose physical substream
order diverges from the convention may select the wrong audio track
(e.g. a 2.0 stream in place of 5.1). Diagnose with
`freemkv info disc://… --log-level 3`; fix tracked for the next release.
## [1.0.0-rc.4.2]
### Fixed
- **Windows durability.** New platform-aware `io::fsync` module: directory
fsync is a no-op on Windows (std cannot open a directory there, which
logged a spurious warning on every mapfile write — including from the
CLI), and a shared `file_durable` helper opens files read+write before
`sync_all` so the flush succeeds on Windows, where `FlushFileBuffers`
rejects a read-only handle with `ERROR_ACCESS_DENIED`.
## [1.0.0-rc.4] — UNRELEASED
An audit-driven round of correctness, durability, and Windows-transport
fixes. No API changes; behavior is more conservative on damaged media and
on partial decryption.
### Fixed
- **Decrypt-time loss is accounted for.** A partial AACS/CSS decryption
failure can no longer pass as a perfect rip — skipped/undecryptable
bytes are folded into the loss total — and partial CPS-unit (per-title)
key coverage is rejected in the AACS validation gate instead of
producing partly-garbage output.
- **Durable writes.** `keydb.cfg` is written atomically (temp file +
fsync + rename), and the mapfile fsyncs its parent directory after the
rename so a resume checkpoint survives a crash.
- **Truthful error causes.** A server-dropped keydb download is
classified as a connection error, not a parse error; a missing home
directory maps to "not found" rather than a keydb-parse failure; the
I/O error from opening an AACS-inputs ISO is preserved; and a
transport failure is preserved through the AACS auth handshake instead
of being relabeled.
- A failed `READ CAPACITY` now warns instead of silently using a
zero-sector disc.
- A leaked pipeline consumer can no longer finalize an abandoned output.
- **Windows SCSI.** `ScsiPassThroughDirect` is packed to match the
`ntddscsi.h` layout, `StorageAdapterDescriptor.BusType` width is
corrected (`u8``u32`), oversized read batches on non-sysfs
(Windows) drives are bounded, `IOCTL_STORAGE_RESET_DEVICE` failures are
surfaced, and a device reset only sleeps on success.
- Mux now tracks skipped bytes so a partly-read title reports accurate
loss.
### Changed
- The per-read `Drive::read` trace event was demoted to TRACE so a debug
log isn't flooded by per-sector reads.
## [1.0.0-rc.2]
Second release candidate for 1.0. libfreemkv is the core library: disc scan,
multipass sector recovery, content decryption (CSS, AACS 1.0/2.0), and the
threaded mux pipeline that turns a disc or ISO into an MKV. This candidate adds
keyless DVD/CSS support and correct DVD video, on top of security and recovery
hardening.
### Added
- **Keyless DVD/CSS title-key recovery.** A CSS-protected DVD decrypts with no
key database — the title key is recovered directly from the scrambled disc
data via the Stevenson known-plaintext attack (ported from libdvdcss) and
validated by descrambling a sector and confirming the known plaintext
reappears, so a wrong key fails cleanly instead of producing silent garbage
(`src/css/stevenson.rs`). `Disc::scan_image` recovers the same title key from
a raw, still-scrambled CSS ISO, so a raw image can be muxed without
pre-decryption.
- **MPEG-2 Program-Stream access-unit reassembler** (`src/mux/codec/mpeg2.rs`).
Buffers elementary-stream bytes across PES packets and emits exactly one
coded picture per MKV block, with presentation timestamps reconstructed from
the stream — fixing corrupted DVD video. Bounded buffer so a malformed stream
cannot exhaust memory.
### Changed
- Self-contained keyframes: the active param sets (HEVC VPS/SPS/PPS, H.264
SPS/PPS, VC-1 sequence/entry headers) are re-asserted at every keyframe and
any mid-title param-set change is emitted in-band, fixing whole-segment
HEVC/H.264/VC-1 corruption when a source stops repeating or reverts a param
set.
- Block timestamps use presentation order keyed on track type, so B-frame video
(including a Dolby Vision enhancement layer) keeps its true presentation
timestamps instead of decode-order timecodes.
- Mux unit alignment is scheme-aware (AACS vs CSS/none), so DVD extents are no
longer rejected for unit misalignment.
- MKV output records `freemkv <version>` in the Muxing/Writing application
fields, so every output file is traceable to its build.
- Subtitle `BlockDuration` values are scaled by the segment timecode scale, so
display durations are correct when the scale is not 1 ms.
- The NOT_READY retry pause in the patch (Pass N) loop is halt-responsive: a
stop request interrupts the drive-recovery wait immediately instead of
blocking shutdown.
- Bounded the keydb decompressed-plaintext reader (caps a malformed or
zip-bombed download).
### Fixed
- A `READ(10)` that returns GOOD status with a residual underrun is treated as a
failed read (routed to retry) instead of committing stale buffer data —
closing a silent-corruption hole in the sweep and patch paths.
- `raw_command` on Linux masks the `DRIVER_SENSE` bit before treating a result
as an error, preventing false transport errors on commands that return sense
alongside a GOOD response.
- `READ CAPACITY (10)` rejects the "capacity exceeds 32-bit" sentinel instead of
silently wrapping to 0 and misreporting disc size.
### Security
- Content keys (CSS disc/title keys, AACS unit/volume keys) are redacted in log
output (logged as `<redacted>` with a 1-byte fingerprint); a test guards
against any key field being logged with a raw value.
- The macOS SCSI shim uses `posix_spawn` directly instead of `system()` / `sh
-c`, eliminating a command-injection vector on the device-path string.
## [1.0.0-rc.1]
First release candidate for 1.0 — the first tagged 1.0 milestone of the core
library. Established the full feature set: multipass sector recovery, content
decryption (CSS, AACS 1.0/2.0) from `keydb.cfg`, disc parsing, and the threaded
mux pipeline (see "Pre-1.0 development" for the consolidated feature list).
## Pre-1.0 development
Versions 0.x were the iterative development series leading up to 1.0. The
highlights, condensed:
- **Multipass recovery engine.** Pass 1 sweeps the whole disc sequentially,
tolerating bad sectors with an adaptive damage-jump algorithm (mark the bad
range, keep going). Pass N retries the bad ranges with per-sector recovery
timeouts, reverse-direction reads, and range bisection. A mapfile tracks
per-sector state across passes so a rip can resume.
- **Drive and SCSI layer.** Single-shot, synchronous SG_IO transport on Linux
(with IOKit on macOS and SPTI on Windows), full SCSI sense decoding, and
drive enumeration / presence probes. Single-shot reads by design — recovery
lives in the multipass orchestration, not inline in the read path.
- **Content decryption.** CSS for DVDs and AACS 1.0/2.0 for Blu-ray and UHD,
with keys read from `keydb.cfg`. A single decrypting decorator wraps the
sector source so decryption is one audited surface, and a resolved key is
verified against disc content before it is applied.
- **Disc parsing.** UDF, MPLS/CLPI (Blu-ray), and IFO (DVD) parsing for title
and extent assembly, with bounds checks on values derived from untrusted disc
input. Canonical main-title selection picks the real feature over a
play-all virtual playlist on branching discs.
- **Mux pipeline (the "highway").** A three-stage threaded pipeline —
read+decrypt, demux, codec parse — with a recycled buffer pool, taking
file-backed mux from ~60 MB/s to several hundred MB/s warm-cache. Codec
parsers for HEVC, H.264, VC-1, MPEG-2, TrueHD, DTS(-HD), and PGS feed an
EBML/Matroska writer.
- **I/O stack.** Bounded-cache writeback (`sync_file_range` +
`posix_fadvise(DONTNEED)`) keeps the kernel dirty-page cache bounded on long
sequential writes, and time-batched mapfile persistence keeps NFS-staged rips
fast.
- **Library hygiene.** No user-facing English in the library — all errors are
numeric codes handled by the application layer. A large spec-grounded,
mutation-verified test suite guards the silent-corruption surfaces. Rust 2024
edition; release builds use thin LTO.
+83
View File
@@ -0,0 +1,83 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at matthew@pq.io. All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of actions.
**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
+46 -12
View File
@@ -1,38 +1,72 @@
[package] [package]
name = "libfreemkv" name = "libfreemkv"
version = "0.3.0" version = "1.2.1"
edition = "2021" edition = "2024"
rust-version = "1.86"
license = "AGPL-3.0-only" license = "AGPL-3.0-only"
description = "Open source raw disc access library for optical drives" description = "Open source raw disc access library for optical drives"
repository = "https://github.com/freemkv/libfreemkv" repository = "https://github.com/freemkv/libfreemkv"
keywords = ["bluray", "uhd", "optical", "scsi", "disc"] keywords = ["bluray", "uhd", "optical", "scsi", "disc"]
categories = ["hardware-support", "multimedia"] categories = ["hardware-support", "multimedia"]
# Keep internal AI-instruction / private notes out of the published crate.
exclude = ["CLAUDE.md"]
# OFF crates.io: libfreemkv git-deps freemkv-unlock (firmware, never published),
# so libfreemkv itself can only be consumed by git tag. Clients git-tag-pin it.
publish = false
[profile.release]
lto = "thin"
codegen-units = 1
[dependencies] [dependencies]
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1" serde_json = "1"
sha1 = "0.10" sha1 = "0.10"
sha2 = "0.10"
aes = "0.8" aes = "0.8"
cbc = "0.1" cbc = "0.1"
flate2 = "1" # Interim path dep for local cross-repo dev; the release script re-pins this to
# `{ git = ".../freemkv-unlock", tag = "vX.Y.Z" }` before tagging libfreemkv (so
# the released tag resolves freemkv-unlock from git, not a sibling path).
freemkv-unlock = { path = "../freemkv-unlock" }
num-bigint = "0.4" num-bigint = "0.4"
num-traits = "0.2" num-traits = "0.2"
num-integer = "0.1" num-integer = "0.1"
rand = "0.8" rand = "0.8"
cmac = "0.7" cmac = "0.7"
zip = { version = "2", default-features = false, features = ["deflate"] } zip = { version = "2", default-features = false, features = ["deflate"] }
base64 = "0.22.1"
# Trace-level instrumentation for Disc::copy + SgIoTransport::execute. Permitted
# under CLAUDE.md ("Acceptable strings: debug/trace logging"). Consumers (autorip)
# wire a tracing subscriber and pipe events into the JSONL debug log.
tracing = "0.1"
# Bounded MPSC channel with kernel-wakeup send_timeout. Used by `io::pipeline`
# so the halt-aware send/finish loops can BLOCK on consumer drain instead of
# polling — the 50 ms poll cadence of the previous mpsc-based impl capped mux
# throughput at ~1 MB/s (0.21.7).
crossbeam-channel = "0.5"
# Persistent work-stealing thread pool for parallel AACS unit
# decryption. Per-call std::thread::scope spawned fresh OS threads
# and that overhead dominated for typical batch sizes (60 units).
# rayon's global pool initialises once on first use.
rayon = "1"
# SIMD-accelerated bytestring search. Drives the HEVC/H.264 start-code
# scan in `mux::codec::h264::find_start_code` — naive byte-by-byte
# walk is ~500 MB/s single-thread on x86_64; memchr's vectorised
# `memmem::find` for the 3-byte `00 00 01` needle hits ~5 GB/s on
# AVX2-capable hosts.
memchr = "2"
[target.'cfg(target_os = "linux")'.dependencies] [target.'cfg(target_os = "linux")'.dependencies]
libc = "0.2" libc = "0.2"
[[bin]] [target.'cfg(target_os = "macos")'.dependencies]
name = "freemkv-info" libc = "0.2"
path = "src/bin/freemkv_info.rs"
[[bin]] [dev-dependencies]
name = "freemkv-test" tempfile = "3"
path = "src/bin/freemkv_test.rs"
[[bench]]
name = "sgio_read"
harness = false
[[bin]]
name = "aacs-test"
path = "src/bin/aacs_test.rs"
+19
View File
@@ -0,0 +1,19 @@
# libfreemkv — local dev helper.
# Mirrors the workspace-wide CI checks but scoped to this single crate.
.PHONY: test build check ci clean
test:
cargo test --tests
build:
cargo build --release
check:
cargo fmt --check
cargo clippy --all-targets -- -D warnings
ci: check build test
clean:
cargo clean
+123 -28
View File
@@ -1,68 +1,161 @@
[![Crates.io](https://img.shields.io/crates/v/libfreemkv)](https://crates.io/crates/libfreemkv)
[![docs.rs](https://img.shields.io/docsrs/libfreemkv)](https://docs.rs/libfreemkv)
[![License: AGPL-3.0](https://img.shields.io/badge/license-AGPL--3.0-blue)](LICENSE) [![License: AGPL-3.0](https://img.shields.io/badge/license-AGPL--3.0-blue)](LICENSE)
# libfreemkv # libfreemkv
Rust library for 4K UHD / Blu-ray optical drives. Drive access, disc scanning, AACS decryption, and content reading in one crate. Bundled drive profiles — no external files needed. Rust library for 4K UHD / Blu-ray / DVD optical drives. Drive access, disc scanning, stream labels, AACS decryption, CSS decryption, KEYDB updates, and content reading in one crate. Drive-level unlocking is handled internally; consumers work with disc access and decryption only.
**[API Documentation](https://docs.rs/libfreemkv)** · **[Technical Docs](docs/)** DVDs (CSS) decrypt out of the box. Blu-ray and UHD (AACS) require a `keydb.cfg` (default `~/.config/freemkv/keydb.cfg`) supplying disc-specific volume unique keys; no AACS key material is compiled in.
**12+ MB/s** sustained read speeds on BD. Drive prep (`init()`) handles unlocking internally via the `freemkv-unlock` crate — clients never see it; when no drive unlock applies, the library rips via the host-certificate AACS handshake.
Multi-lingual by design — the library outputs structured data and numeric error codes, never English text. Build any UI or localization on top.
**[Source & API](https://github.com/freemkv/libfreemkv)** · **[Technical Docs](docs/)**
Part of the [freemkv](https://github.com/freemkv) project. Part of the [freemkv](https://github.com/freemkv) project.
## Install ## Install
Consumed by git tag (not published to crates.io):
```toml ```toml
[dependencies] [dependencies]
libfreemkv = "0.3" libfreemkv = { git = "https://github.com/freemkv/libfreemkv", tag = "vX.Y.Z" }
``` ```
## Quick Start ## Quick Start
```rust ```rust
use libfreemkv::{DriveSession, Disc, ScanOptions}; use libfreemkv::{Drive, Disc, ScanOptions};
use std::path::Path; use std::path::Path;
// Open drive — profiles are bundled, auto-identified // Open drive — identified via INQUIRY
let mut session = DriveSession::open(Path::new("/dev/sr0"))?; let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?; // wait for disc
drive.init()?; // unlock + prep (handled internally)
drive.probe_disc()?; // probe disc surface for optimal speeds
// Scan disc — UDF, playlists, streams, AACS (all automatic) // Scan disc — UDF, playlists, streams, AACS (all automatic)
let disc = Disc::scan(&mut session, &ScanOptions::default())?; let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
for title in &disc.titles { for title in &disc.titles {
println!("{}{} streams", title.duration_display(), title.streams.len()); println!("{}{} streams", title.duration_display(), title.streams.len());
} }
// Read content (decrypted transparently if AACS keys available) // Stream pipeline — read PES frames from any source, write to any output
let mut reader = disc.open_title(&mut session, 0)?; let opts = libfreemkv::InputOptions::default();
while let Some(unit) = reader.read_unit()? { let mut input = libfreemkv::input("iso://Disc.iso", &opts)?;
// 6144 bytes of content per aligned unit let title = input.info().clone();
let mut output = libfreemkv::output("mkv://Movie.mkv", &title)?;
while let Ok(Some(frame)) = input.read() {
output.write(&frame)?;
} }
output.finish()?;
```
### Multi-pass recovery rip
For damaged discs the library exposes two flat verbs — `Disc::sweep` for the
forward Pass 1 and `Disc::patch` for retrying bad ranges. The library never
loops; the multipass policy is the caller's job. See
[`docs/rip-recovery.md`](docs/rip-recovery.md).
```rust
use libfreemkv::{SweepOptions, PatchOptions};
use libfreemkv::disc::{mapfile, mapfile_path_for};
use std::path::Path;
let iso = Path::new("disc.iso");
// Pass 1: disc → ISO. Skip-on-error, zero-fill, write the sidecar mapfile.
disc.sweep(&mut drive, iso, &SweepOptions {
decrypt: true,
resume: false,
batch_sectors: None,
skip_on_error: true,
progress: None,
halt: None,
})?;
// Pass 2..N: retry every non-finished range. Idempotent.
loop {
let map = mapfile::Mapfile::load(&mapfile_path_for(iso))?;
let stats = map.stats();
if stats.bytes_pending + stats.bytes_unreadable == 0 { break; }
let outcome = disc.patch(&mut drive, iso, &PatchOptions {
decrypt: true,
block_sectors: None,
full_recovery: true,
reverse: true,
wedged_threshold: 50,
progress: None,
halt: None,
})?;
if outcome.bytes_recovered_this_pass == 0 { break; }
}
// Mux from the ISO via the normal stream pipeline (no drive involvement).
``` ```
## What It Does ## What It Does
- **Drive access** — open, identify, unlock for raw reads - **Drive access** — open, identify, internal unlock + prep, speed control, eject
- **Disc scanning** — UDF 2.50 filesystem, MPLS playlists, CLPI clip info, BD-J labels - **12+ MB/s reads** — auto-detects kernel transfer limits, sustained full speed
- **AACS decryption** — transparent key resolution and content decrypt (1.0 + 2.0) - **Disc scanning** — UDF 2.50 filesystem, MPLS playlists, CLPI clip info
- **Content reading** — sector reads with automatic decryption - **Stream labels** — 5 BD-J format parsers (Paramount, Criterion, Pixelogic, CTRM, Deluxe)
- **AACS decryption** — transparent key resolution and content decrypt (1.0 + 2.0 bus decryption)
- **KEYDB updates** — download, verify, save from any HTTP URL (zero deps, raw TCP)
- **Content reading** — adaptive batch reads with automatic decryption
- **Stream I/O** — unified stream pipeline for reading and writing any format
AACS decryption requires a KEYDB.cfg file. If available at `~/.config/aacs/KEYDB.cfg` or passed via `ScanOptions`, the library handles everything — handshake, key derivation, and per-sector decryption — without the application needing to know anything about encryption. ### Streams
| Stream | Input | Output | Transport |
|--------|-------|--------|-----------|
| DiscStream | Yes | -- | Optical drive via SCSI |
| IsoStream | Yes | -- | Blu-ray ISO image file (read via stream pipeline; written via `Disc::sweep()`) |
| MkvStream | Yes | Yes | Matroska container |
| M2tsStream | Yes | Yes | BD transport stream with FMKV metadata header |
| NetworkStream | Yes (listen) | Yes (connect) | TCP with FMKV metadata header |
| StdioStream | Yes (stdin) | Yes (stdout) | Raw byte pipe |
| NullStream | -- | Yes | Discard sink (byte counter for benchmarks) |
Streams implement a single unified `pes::Stream` trait (re-exported as `PesStream`) exposing `read()` and `write()` on one type. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
### Keys
DVDs (CSS) decrypt out of the box, with no external key file needed.
Blu-rays and UHD (AACS) require a `keydb.cfg` at `~/.config/freemkv/keydb.cfg` (or passed via `ScanOptions`). No AACS key material is compiled into the binary.
## Architecture ## Architecture
```text ```text
DriveSession — open, identify, unlock, read sectors Drive — open, identify, init, single-shot read
├── ScsiTransport — SG_IO (Linux), IOKit (macOS planned) ├── ScsiTransport — SG_IO (Linux), IOKit (macOS), SPTI (Windows)
── DriveProfile — per-drive unlock parameters (bundled) ── unlock_bridge — private seam to the freemkv-unlock crate
└── Platform — MediaTek (supported), Renesas (planned) (firmware / AACS cert / CSS bus-auth unlockers)
Disc — scan titles, streams, AACS state Disc — scan titles, streams, AACS/CSS state
├── UDF reader — Blu-ray UDF 2.50 with metadata partitions ├── UDF reader — Blu-ray UDF 2.50 with metadata partitions
├── MPLS parser — playlists → titles + clips + streams ├── MPLS parser — playlists → titles + clips + streams
├── CLPI parser — clip info → EP map → sector extents ├── CLPI parser — clip info → EP map → sector extents
├── JAR parser — BD-J audio track labels ├── IFO parser — DVD title sets, PGC chains, cell addresses
── AACS — key resolution + content decryption ── Labels — 5 BD-J format parsers (detect + parse)
├── AACS — key resolution + content decryption
├── CSS — DVD CSS (bus auth → player-key disc crack → known-plaintext title-key attack)
└── KEYDB — download + verify + save
Streams — unified PES pipeline
├── PesStream — pes::Stream: one trait, read()/write() PES frames
├── DiscStream — sectors → decrypt → TS demux → PES
├── IsoStream — ISO file → decrypt → TS demux → PES
├── MkvStream — MKV mux/demux
├── M2tsStream — BD transport stream
├── NetworkStream — TCP with FMKV metadata header
├── StdioStream — stdin/stdout pipe
└── NullStream — discard sink
``` ```
See [docs/](docs/) for detailed technical documentation on each module. See [docs/](docs/) for detailed technical documentation on each module.
@@ -80,18 +173,20 @@ All errors are structured with numeric codes. No user-facing English text — ap
| E5xxx | I/O errors | | E5xxx | I/O errors |
| E6xxx | Disc format errors | | E6xxx | Disc format errors |
| E7xxx | AACS errors | | E7xxx | AACS errors |
| E8xxx | KEYDB update errors |
| E9xxx | Stream / mux errors (URL, PES, ISO, pipeline, demux) |
## Platform Support ## Platform Support
| Platform | Status | Backend | | Platform | Status | Backend |
|----------|--------|---------| |----------|--------|---------|
| Linux | Supported | SG_IO ioctl | | Linux | Supported | SG_IO ioctl |
| macOS | Planned | IOKit | | macOS | Supported | IOKit SCSITask |
| Windows | Planned | SPTI | | Windows | Supported | SPTI |
## Contributing ## Contributing
Run `freemkv info --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to contribute your drive's profile. Run `freemkv info disc:// --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to capture your drive's identity for contribution. Drive-unlock profiles are maintained in the [freemkv-unlock](https://github.com/freemkv/freemkv-unlock) repository.
## License ## License
+312
View File
@@ -0,0 +1,312 @@
# Troubleshooting Guide
Common problems and solutions for optical drive ripping with freemkv.
---
## 1. USB-SATA Bridge Issues
This is the single most common source of problems when ripping discs over USB.
### Symptoms
- The drive disappears mid-rip. The ripping tool reports the device is gone, and `ls /dev/sg*` no longer shows it.
- The device re-enumerates under a different name: `sg4` becomes `sg5`, then `sg7`, then `sg11` after each USB port reset.
- `dmesg` shows USB port resets: `usb X-Y: reset high-speed USB device`, `xhci_hcd 0000:00:14.0: Cannot enable. Maybe the USB cable is bad?`, or `usb-storage: device reset failed`.
- The SCSI layer reports `host_status=7` (Linux USB transport error) in sense data.
- The drive works fine for reading data discs or burning, but crashes when hitting damaged sectors during a rip.
- After the crash, the drive is completely invisible until physically unplugged and reconnected.
### Root Cause
USB-SATA bridges translate between the USB Mass Storage protocol (BOT or UAS) and the drive's native SATA interface. When the optical drive encounters an unreadable sector, it returns a SCSI CHECK CONDITION with sense key 0x03 (MEDIUM ERROR). Some bridge chipsets -- particularly the Initio INIC-36xx family -- have buggy firmware that mishandles this error response.
Specific failure modes:
- **Incorrect residue reporting.** The bridge claims a different number of bytes transferred than what actually occurred. The Linux USB storage driver sees this discrepancy as a protocol violation and resets the port to recover. The `US_FL_IGNORE_RESIDUE` quirk exists specifically for this class of bug (see `drivers/usb/storage/transport.c` in the Linux kernel).
- **Bridge firmware crash.** On some Initio bridges, a malformed SCSI error response from the drive causes the bridge MCU to hang entirely. The USB controller sees the device stop responding and initiates a port reset. The bridge recovers (it re-enumerates), but the rip is dead -- all state is lost.
- **Sense data corruption.** The bridge forwards garbled or truncated sense data to the host, which the SCSI midlayer cannot parse, leading to a transport reset.
This is a hardware + firmware problem, not a software bug. The same drive connected via direct SATA does not exhibit these symptoms.
### Known Affected Bridges
| Chipset | USB IDs | Notes |
|---------|---------|-------|
| Initio INIC-3609 | `13fd:3609` | Very common in cheap SATA-to-USB enclosures. Highly problematic. |
| Initio INIC-3619 | `13fd:3940` | Same firmware family as INIC-3609. |
| Initio INIC-3069 | `13fd:0840` | Older variant, same residue bug. |
| ASMedia ASM1051 | `174c:5106` | Early ASM SATA bridge. Residue issues on error paths. |
| JMicron JMB36x | `152d:0561` | Some firmware versions. Not all JMicroon chips are affected. |
If your drive came in a pre-built external enclosure (Vantec, Sabrent, OWC, etc.), it almost certainly uses one of these bridge chips internally.
### The Fix: USB Storage Quirk
Apply the `US_FL_IGNORE_RESIDUE` kernel quirk for your bridge. This tells the Linux USB storage driver to ignore the residue field in SCSI response frames, preventing the port reset on mismatched byte counts.
**Step 1: Identify your bridge's vendor:product ID.**
```bash
lsusb
```
Look for your drive's entry. Example output:
```
Bus 002 Device 005: ID 13fd:0840 Initio Corporation INIC-3609
```
Here the vendor ID is `13fd` and the product ID is `0840`.
**Step 2: Apply the quirk at runtime.**
```bash
echo "13fd:0840:i" > /sys/module/usb_storage/parameters/quirks
```
Replace `13fd:0840` with your device's actual IDs. The `:i` flag means `US_FL_IGNORE_RESIDUE`.
You can combine multiple flags. Common additions:
- `:i` -- ignore residue (`US_FL_IGNORE_RESIDUE`)
- `:u` -- force BOT mode instead of UAS, for bridges with UAS bugs
**Step 3: Reconnect the drive.** Unplug and replug the USB cable, or bind/unbind the device. The quirk is applied per-module-load, so existing sessions may need the drive reconnected.
### Making It Persistent
Add the quirk to your kernel boot parameters so it survives reboots.
Edit `/etc/default/grub` (GRUB) and add to `GRUB_CMDLINE_LINUX_DEFAULT`:
```
GRUB_CMDLINE_LINUX_DEFAULT="quiet usb_storage.quirks=13fd:0840:i"
```
Then rebuild the GRUB config:
```bash
sudo update-grub
```
For systemd-boot, add to your loader entry or `/etc/kernel/cmdline`:
```
usb_storage.quirks=13fd:0840:i
```
Multiple devices can be separated by commas:
```
usb_storage.quirks=13fd:0840:i,174c:5106:u
```
### Recommended Bridges
If you are buying a USB-SATA adapter or enclosure for optical drive use:
| Bridge | USB IDs | Notes |
|--------|---------|-------|
| ASMedia ASM1153 | `174c:1153` | Reliable. Widely available in SATA-USB 3.0 cables. |
| JMicron JMS578 | `152d:0578` | Good firmware. Supports UASP. |
| Icy Box IB-AC640-C3 | N/A | Uses a known-good bridge internally. Plug-and-play. |
Avoid any enclosure or adapter listing an Initio chipset.
### Best Solution: Direct SATA
Connect your optical drive directly to a motherboard SATA port. This eliminates the USB-SATA bridge entirely and is the most reliable configuration:
- No USB protocol overhead or translation errors.
- No bridge firmware bugs.
- No port resets or re-enumeration.
- Full SATA error recovery handled natively by the kernel's libata driver.
- Sustained read speeds are limited only by the drive, not the USB bus.
If your machine has a free SATA port, use it.
---
## 2. Damaged Disc Handling
### Symptoms
- SCSI MEDIUM ERROR (sense key 0x03) at specific LBAs. `dmesg` shows `sr X:0:0:0: [srY] Unrecoverable read error` or similar.
- Read speed drops to near zero when approaching a damaged area.
- The drive makes audible retrying noises (laser repositioning, spindle speed changes).
- On USB-connected drives: the bridge crashes (see section 1 above) when the drive returns the error.
### How freemkv Handles This
freemkv uses a three-layer recovery model. See [`docs/rip-recovery.md`](docs/rip-recovery.md) for full details.
- **Pass 1 (Disc::copy):** Fast sweep with 64 KB reads. On failure, zero-fills the block and skips forward. Writes a ddrescue-format mapfile for later retry.
- **Pass 2+ (Disc::patch):** Targeted re-reads of bad ranges with a long 30-second timeout per CDB. The drive firmware performs its own ECC and laser power retries within that window.
- **In-stream (DiscStream):** Adaptive batch halving -- reduces request size on failure to isolate bad sectors within a larger block.
This means a disc with some bad sectors will still produce a usable ISO. The damaged areas are zero-filled in pass 1 and retried in subsequent passes. Structure-protected sectors (deliberate unreadable regions from copy protection) will never yield, which is expected.
### The Drive Taint Issue (LG BU40N)
Some drives, notably the LG BU40N, exhibit a "taint" behavior after encountering MEDIUM ERRORs:
1. The drive hits a damaged sector and returns a MEDIUM ERROR.
2. From that point forward, **all subsequent reads fail** -- even reads to sectors that were previously successful.
3. The only recovery is to physically unplug and reconnect the drive (or power-cycle it).
This is not a freemkv bug. It is a drive firmware behavior triggered by the interaction between the drive's internal error recovery and the USB-SATA bridge's handling of the error response. The drive firmware enters a degraded state that it does not recover from without a power cycle.
Workarounds:
- **Use a direct SATA connection.** This eliminates the bridge interaction that triggers the taint.
- **Use a different bridge.** The ASM1153 and JMS578 are less likely to trigger this behavior.
- **Accept the partial ISO.** freemkv's skip-forward recovery will zero-fill the unreadable blocks and continue. The resulting ISO may be playable with minor glitches in the affected areas.
- **Physical replug between retry passes.** If running multi-pass patch, replug the drive between passes to clear the taint state.
freemkv deliberately does not attempt inline SCSI resets or eject cycles to recover from this state, because those operations were found to make the problem worse on affected hardware (see the design rationale in [`docs/rip-recovery.md`](docs/rip-recovery.md)).
---
## 3. Drive Not Detected
### Check Hardware Visibility
```bash
lsusb
```
Verify the drive appears in the USB device list. If it does not show up, the drive is not visible to the host at all -- check cables, power, and USB port.
```bash
ls /dev/sg*
```
On Linux, optical drives appear as `/dev/sg*` devices (the SCSI Generic interface). freemkv uses `/dev/sg*`, not `/dev/sr*`. If `lsusb` shows the device but no `/dev/sg*` entry exists, the `sg` kernel module may not be loaded:
```bash
sudo modprobe sg
```
### Check Kernel Messages
```bash
dmesg | grep -i usb | tail -30
dmesg | grep -i sg | tail -10
```
Look for:
- USB enumeration errors or failed port resets.
- `sg_add` messages confirming the sg device was registered.
- Permission denied or access errors.
### Permission Issues
On most Linux distributions, `/dev/sg*` devices are owned by `root:disk` or `root:cdrom` with restricted permissions. Running freemkv as an unprivileged user will fail with permission errors.
Options:
- Add your user to the appropriate group:
```bash
sudo usermod -aG disk $USER
```
Then log out and back in for the change to take effect. On some distributions the group is `cdrom` or `optical` instead of `disk`.
- Run with elevated privileges:
```bash
sudo freemkv ...
```
- Install a udev rule for persistent per-device permissions. Create `/etc/udev/rules.d/99-sg-optical.rules`:
```
SUBSYSTEM=="scsi_generic", ATTRS{type}=="5", MODE="0666"
```
Then reload udev rules:
```bash
sudo udevadm control --reload-rules && sudo udevadm trigger
```
### Spin-Up Delay
Optical drives take 30-60 seconds to spin up and become ready after hot-plug or disc insertion. During this window, SCSI commands may return NOT READY or timeout.
freemkv's `Drive::wait_ready()` handles this automatically by polling with TEST UNIT READY until the drive responds. If you are writing your own code using the library, always call `wait_ready()` before `init()`:
```rust
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?; // blocks until disc is ready, up to 30s
drive.init()?;
```
If the drive was just plugged in, wait a full minute before concluding it is not detected.
---
## 4. How to Identify Your USB-SATA Bridge
If you are experiencing the issues described in section 1, you need to know which bridge chipset your adapter or enclosure uses.
### Step 1: Find the Device
```bash
lsusb
```
Look for entries matching your drive or enclosure. Bridges may appear under their own manufacturer name or as a generic SATA device. Common examples:
```
Bus 002 Device 005: ID 13fd:0840 Initio Corporation
Bus 002 Device 006: ID 174c:1153 ASMedia Technology Inc. ASM1153
Bus 002 Device 007: ID 152d:0578 JMicron Technology Corp. JMS578
```
### Step 2: Match the IDs
| Vendor | Product ID | Chipset | Status |
|--------|-----------|---------|--------|
| `13fd` | `3609` | Initio INIC-3609 | Affected. Apply quirk. |
| `13fd` | `3940` | Initio INIC-3619 | Affected. Apply quirk. |
| `13fd` | `0840` | Initio INIC-3069 | Affected. Apply quirk. |
| `174c` | `5106` | ASMedia ASM1051 | Affected (early firmware). Apply quirk. |
| `174c` | `1153` | ASMedia ASM1153 | Good. No quirk needed. |
| `152d` | `0561` | JMicron JMB36x | Affected (some firmware). Apply quirk if issues occur. |
| `152d` | `0578` | JMicron JMS578 | Good. No quirk needed. |
### Step 3: Check dmesg for the Bridge Name
```bash
dmesg | grep -i "usb-storage\|uas\|initio\|asmedia\|jmicron"
```
This often reveals the bridge chipset even when `lsusb` shows a generic name.
### Step 4: If the Enclosure Is Sealed
Many external drive enclosures (Vantec NexStar, Sabrent, OWC, etc.) do not advertise the bridge chipset on the packaging. In this case:
1. Check `lsusb` while the enclosure is connected.
2. Search the vendor:product ID online -- there are community-maintained lists of which chipsets popular enclosures use.
3. If you cannot determine the chipset and are experiencing bridge crashes, assume it is an Initio and apply the quirk with its IDs.
4. The definitive test: connect the bare drive to a motherboard SATA port. If the problems disappear, the bridge was the cause.
---
## 5. General Debugging Checklist
When something goes wrong during a rip, gather this information before filing an issue:
1. **freemkv version:** `freemkv --version` or the crate version in `Cargo.toml`.
2. **Drive model:** from the drive label, or from `freemkv info`.
3. **Connection type:** USB (with bridge chipset if known) or direct SATA.
4. **Operating system and kernel:** `uname -a`.
5. **Kernel messages during the failure:** `dmesg | tail -50` immediately after the crash.
6. **SCSI device:** which `/dev/sg*` the drive was on, and whether it changed after the failure.
7. **The disc:** title, format (BD/DVD/UHD), condition.
Include all of the above in bug reports. SCSI transport errors that resolve with the `US_FL_IGNORE_RESIDUE` quirk or by switching to direct SATA are bridge firmware bugs, not freemkv bugs.
+87
View File
@@ -0,0 +1,87 @@
// Mimics ISO dump exactly — read + write + progress
use libfreemkv::Drive;
use std::io::Write;
use std::path::Path;
use std::time::Instant;
fn main() {
let device = std::env::args()
.skip(1)
.find(|a| !a.starts_with('-'))
.unwrap_or_else(|| match libfreemkv::find_drive() {
Some(d) => d.device_path().to_string(),
None => {
eprintln!("No drives found");
std::process::exit(1);
}
});
let mut drive = Drive::open(Path::new(&device)).unwrap_or_else(|e| {
eprintln!("Cannot open {}: {}", device, e);
std::process::exit(1);
});
eprintln!("wait_ready...");
let _ = drive.wait_ready();
eprintln!("read_capacity...");
let cap = drive.read_capacity().unwrap();
eprintln!("capacity: {} sectors", cap);
let batch = libfreemkv::disc::detect_max_batch_sectors(drive.device_path());
let mut buf = vec![0u8; batch as usize * 2048];
// Open /dev/null writer like ISO dump does
let file = std::fs::File::create("/dev/null").unwrap();
let mut writer = std::io::BufWriter::with_capacity(4 * 1024 * 1024, file);
eprintln!(
"Reading 1000 batches ({:.1} MB) with write + progress...",
1000.0 * batch as f64 * 2048.0 / 1_048_576.0
);
let start = Instant::now();
let mut ok = 0u32;
let mut fail = 0u32;
let mut bytes: u64 = 0;
// Recovery flag: true matches pre-0.11.13 bench behavior — full SCSI
// ECC retry loop on errors (slower, what the rip path used before the
// adaptive batch sizer landed). Flip to `false` for the fast-fail path
// that current rips use; benches are configurable via this constant.
const READ_WITH_RECOVERY: bool = true;
for i in 0..1000u32 {
let lba = i * batch as u32;
match drive.read(lba, batch, &mut buf, READ_WITH_RECOVERY) {
Ok(_) => {
writer.write_all(&buf).unwrap();
ok += 1;
}
Err(e) => {
fail += 1;
if fail <= 5 {
eprintln!(" FAIL LBA {}: {}", lba, e);
}
buf.fill(0);
writer.write_all(&buf).unwrap();
}
}
bytes += buf.len() as u64;
if i % 50 == 0 && i > 0 {
let elapsed = start.elapsed().as_secs_f64();
let mb = bytes as f64 / 1_048_576.0;
eprint!("\r {:.1} MB | {:.1} MB/s ", mb, mb / elapsed);
}
}
let elapsed = start.elapsed().as_secs_f64();
let mb = ok as f64 * batch as f64 * 2048.0 / 1_048_576.0;
eprintln!(
"\n{} ok, {} fail, {:.1} MB in {:.1}s = {:.1} MB/s",
ok,
fail,
mb,
elapsed,
mb / elapsed
);
}
+97
View File
@@ -0,0 +1,97 @@
fn main() {
emit_git_suffix();
let target = std::env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
if target == "macos" {
println!("cargo:rustc-link-lib=framework=IOKit");
println!("cargo:rustc-link-lib=framework=CoreFoundation");
let out_dir = std::env::var("OUT_DIR").unwrap();
let obj = format!("{out_dir}/macos_shim.o");
let lib = format!("{out_dir}/libmacos_scsi.a");
// Build the shim for the TARGET arch, not the host's. A bare `cc` on an
// Apple-Silicon CI runner defaults to arm64, so cross-building to
// x86_64-apple-darwin would link a host-arch object against x86_64 Rust
// code → "Undefined symbols for architecture x86_64". (Still raw `cc`,
// not the `cc` crate, which breaks IOKit exclusive access.)
let target_arch = std::env::var("CARGO_CFG_TARGET_ARCH").unwrap_or_default();
let clang_arch: &str = if target_arch == "aarch64" {
"arm64"
} else {
&target_arch // x86_64 → x86_64
};
std::process::Command::new("cc")
.args([
"-arch",
clang_arch,
"-c",
"src/scsi/macos_shim.c",
"-o",
&obj,
"-framework",
"IOKit",
"-framework",
"CoreFoundation",
"-Wall",
"-O2",
])
.status()
.expect("failed to compile macos_shim.c");
std::process::Command::new("ar")
.args(["rcs", &lib, &obj])
.status()
.expect("failed to create static lib");
println!("cargo:rustc-link-search=native={out_dir}");
println!("cargo:rustc-link-lib=static=macos_scsi");
println!("cargo:rerun-if-changed=src/scsi/macos_shim.c");
}
}
/// Bake the git short hash into the build as `GIT_SUFFIX` so any muxed MKV or
/// FVI index is traceable to the exact source revision (e.g. ` (g835cc99)`).
/// Empty when git or the repo is unavailable (e.g. a crates.io tarball build),
/// leaving just the package version. Always emitted so `env!("GIT_SUFFIX")`
/// resolves on every target.
fn emit_git_suffix() {
// Version label for the muxing-app / FVI generator tag. `FREEMKV_BUILD_LABEL`
// overrides the Cargo package version when set (non-empty) — used to stamp a
// pre-release/test build without bumping Cargo.toml and disturbing the
// tag-pinned [patch] version matching. Unset → the package version.
let version = std::env::var("FREEMKV_BUILD_LABEL")
.ok()
.filter(|s| !s.trim().is_empty())
.or_else(|| std::env::var("CARGO_PKG_VERSION").ok())
.unwrap_or_default();
println!("cargo:rustc-env=FREEMKV_VERSION={version}");
println!("cargo:rerun-if-env-changed=FREEMKV_BUILD_LABEL");
let suffix = git_short_hash()
.map(|h| format!(" (g{h})"))
.unwrap_or_default();
println!("cargo:rustc-env=GIT_SUFFIX={suffix}");
// Re-run when HEAD (or the branch it points at) moves so the stamp stays
// current without a clean rebuild.
println!("cargo:rerun-if-changed=.git/HEAD");
if let Ok(head) = std::fs::read_to_string(".git/HEAD") {
if let Some(ref_path) = head.strip_prefix("ref: ") {
println!("cargo:rerun-if-changed=.git/{}", ref_path.trim());
}
}
}
fn git_short_hash() -> Option<String> {
let out = std::process::Command::new("git")
.args(["rev-parse", "--short=7", "HEAD"])
.output()
.ok()?;
if !out.status.success() {
return None;
}
let h = String::from_utf8(out.stdout).ok()?.trim().to_string();
if h.is_empty() { None } else { Some(h) }
}
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
#
# leak-guard.sh — self-contained public-repo leak gate.
#
# This is the LAST line of defense in CI. It is intentionally self-contained:
# public CI cannot reach the private tooling, so this script encodes ONLY the
# generic net — internal infrastructure references, agent-context files, and
# AI-attribution in commit messages. It deliberately contains NO project-
# specific reverse-engineering vocabulary (those words would themselves be a
# leak). The richer private scanner stays private.
#
# Fails (exit 1) if any of the following appear in the repo:
# 1. a tracked CLAUDE.md or .claude/ path (agent context — never public),
# 2. tracked file content matching the internal-infra net,
# 3. a commit message (in the given range) with AI attribution.
#
# Usage:
# leak-guard.sh [<commit-range>]
# <commit-range> optional git rev-list range to scan commit messages
# (e.g. "abc..def"). If omitted, commit-message scan is
# skipped (path + content checks always run).
set -euo pipefail
# Absolute path to this script, resolved before any cd, so we can exclude it
# from the content scan (it necessarily contains the detection patterns).
SELF_ABS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
REPO="$(git rev-parse --show-toplevel)"
cd "$REPO"
fail=0
note() { printf ' ✗ %s\n' "$1"; fail=1; }
# Internal-infra net — GENERIC ONLY. This script ships in the public repo, so
# the patterns themselves must not name any org-specific identifier (doing so
# would itself leak the infra they guard). We catch the leak *class*:
# - RFC1918 private IPv4 ranges (10/8, 172.16/12, 192.168/16),
# - private/internal/non-routable TLDs (.internal/.local/.lan/.corp/.invalid),
# - docker.internal.
# The full org-specific net (literal hostnames, service names, repo paths,
# vendor tooling, …) lives ONLY in the private scanner and never ships here.
INFRA_RE='\b10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|\b172\.(1[6-9]|2[0-9]|3[01])\.[0-9]{1,3}\.[0-9]{1,3}|\b192\.168\.[0-9]{1,3}\.[0-9]{1,3}|\.internal\b|\.local\b|\.lan\b|\.corp\b|\.invalid\b|docker\.internal'
# Home-path net — GENERIC ONLY. Catches an absolute developer home path
# committed into a tracked file (a macOS /Users/<user>/… or Linux /home/<user>/…
# path). This names NO specific user — it matches the leak *class* (any home
# path), so the pattern itself reveals nothing org- or person-specific. A real
# leak (e.g. /Users/alice/Developer/x slipping into a public RELEASE.md) trips
# this regardless of whose machine it came from. The username segment is a
# literal-username class ([A-Za-z0-9._-]) so dynamic/templated paths that build
# the user at runtime — shell `/home/$USER/`, doc `/home/<rip>/`, Rust
# `/home/{user}/` — do NOT false-positive; only a baked-in literal home leaks.
HOMEPATH_RE='/Users/[A-Za-z0-9._-]+/|/home/[A-Za-z0-9._-]+/'
# AI-attribution net (case-insensitive). "claude" matches only as a standalone
# word — NOT preceded by a dot/slash/alnum and NOT followed by .md — so legit
# mentions of CLAUDE.md / .claude/ in a commit message don't false-positive.
ATTR_RE='co-authored-by|generated with|🤖|(?<![.\/A-Za-z0-9])claude(?!\.md)'
echo "── leak-guard: tracked agent-context paths ──"
while IFS= read -r f; do
case "$f" in
CLAUDE.md|*/CLAUDE.md|.claude|.claude/*|*/.claude|*/.claude/*)
note "tracked agent-context file: $f (CLAUDE.md/.claude must never be tracked in a public repo)" ;;
esac
done < <(git ls-files)
# Match a PCRE against a file, emitting "LINE: MATCH". The pattern is passed as
# an argument (not interpolated into a //) so metacharacters like the "/" in a
# path-style token can't break the regex. Reads raw bytes so non-UTF-8 blobs
# don't abort the scan.
pcre_matches() {
perl -e '
my ($file, $re) = @ARGV;
open(my $fh, "<:raw", $file) or exit 0;
my $rx; eval { $rx = qr/$re/i }; exit 0 if $@;
while (my $l = <$fh>) { if ($l =~ /$rx/) { print "$.: $&\n"; } }
' "$1" "$2" 2>/dev/null
}
# This script's own source necessarily contains the detection patterns (e.g.
# the regex tokens in INFRA_RE), so scanning it would always self-flag. Skip it.
SELF="$(git ls-files --full-name -- "$SELF_ABS" 2>/dev/null | head -1)"
echo "── leak-guard: internal-infra references in tracked files ──"
while IFS= read -r f; do
case "$f" in *.png|*.jpg|*.jpeg|*.ico|*.gif|*.bin|*.crate|*.gz|*.zip|*.pdf) continue ;; esac
[ -n "$SELF" ] && [ "$f" = "$SELF" ] && continue
[ -f "$f" ] || continue
while IFS= read -r hit; do
[ -z "$hit" ] && continue
note "internal-infra reference: $f:$hit"
done < <(pcre_matches "$f" "$INFRA_RE")
while IFS= read -r hit; do
[ -z "$hit" ] && continue
note "[HOME-PATH] absolute home path: $f:$hit (no local home path may be committed to a public repo)"
done < <(pcre_matches "$f" "$HOMEPATH_RE")
done < <(git ls-files)
RANGE="${1:-}"
if [ -n "$RANGE" ]; then
echo "── leak-guard: AI-attribution in commit messages ($RANGE) ──"
while IFS= read -r sha; do
[ -z "$sha" ] && continue
msg="$(git log -1 --format='%B' "$sha" 2>/dev/null || true)"
# Pass the pattern as an argument (not interpolated into a //) so the
# lookbehind char class and "/" don't break the regex.
hit="$(printf '%s' "$msg" | perl -e '
my $re = $ARGV[0]; my $rx = qr/$re/i;
while (my $l = <STDIN>) { if ($l =~ /($rx)/) { print "$1\n"; last; } }
' "$ATTR_RE" | head -1 || true)"
[ -n "$hit" ] && note "commit ${sha:0:12}: message contains \"$hit\" (owner rule: zero AI attribution, ever)"
done < <(git rev-list "$RANGE" 2>/dev/null || true)
fi
echo
if [ "$fail" -ne 0 ]; then
echo "✗ leak-guard: blocking finding(s) above — DO NOT MERGE/PUBLISH"
exit 1
fi
echo "✓ leak-guard: clean"
+302
View File
@@ -0,0 +1,302 @@
# FVI — Freemkv Video Index Format
**Specification version:** 1.0 (DRAFT)\
**File extension:** `.fvi`\
**Media type:** `application/vnd.freemkv.fvi+jsonl`\
**Status:** Draft for review. This document is the normative reference for the FVI
format; implementations and downstream tools cite it by section.
---
## 1. Scope and purpose
FVI is an open, codec-agnostic, byte-exact **index of the coded pictures** in a
video bitstream, together with **provenance** back to the source medium.
An FVI document answers, for every picture in a stream, three questions:
1. **Where is it?** — the byte-exact offset of its first byte in the *source*
(the disc/ISO/file), so a reader can extract or seek to any picture without
re-parsing the whole bitstream.
2. **What is it?** — coding type, random-access capability, GOP boundary, and
(where the codec defines them) field/pulldown attributes.
3. **When is it?** — decode and presentation timestamps on a declared timescale.
FVI is **not** a container, a codec, or a copy of the bitstream. It indexes; it
never stores coded samples. It is the serialized form of an indexer's per-picture
truth — carried from the demuxer, **never reconstructed** (§9).
## 2. Conformance
The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**,
**SHOULD**, **SHOULD NOT**, **MAY**, and **OPTIONAL** are to be interpreted as
described in BCP 14 (RFC 2119, RFC 8174) when, and only when, they appear in all
capitals.
A **conformant writer** MUST emit a document that satisfies §4–§10. A
**conformant reader** MUST accept any such document and MUST ignore unknown
object members (§11) so that forward-compatible extensions do not break it.
## 3. Terminology
- **Picture** — one coded video frame (or pair of fields coded as a frame). The
unit FVI indexes.
- **Access unit (AU)** — the set of bitstream bytes that decode to exactly one
picture (ISO/IEC 14496-10 §3; ISO/IEC 23008-2 §3).
- **Coded order** — the order pictures appear in the bitstream. FVI records are
emitted in coded order.
- **GOP / coded video sequence** — a self-contained run beginning at a
random-access point.
- **Provenance** — the mapping from an AU back to the exact bytes of the physical
source it was read from (§9).
- **Source position (`src`)** — `{ file, sector, byte }`, the provenance anchor of
an AU.
## 4. Encoding
An FVI document is a sequence of **UTF-8** text lines separated by a single LF
(`U+000A`). Each non-empty line is exactly one JSON value (RFC 8259), forming a
**JSON Lines / NDJSON** stream. A writer MUST NOT emit a UTF-8 BOM. A writer MUST
NOT pretty-print: each JSON value occupies exactly one line.
The first line MUST be the **Header** object (§6). Each subsequent line is one
**Picture record** (§7), in coded order.
Rationale: line-delimited JSON is streamable (a writer appends as it indexes; a
reader processes without loading the whole file), line-addressable (picture *n*
is near line *n+1*), append-safe, and parseable by every language without a
custom grammar — while remaining a precisely specified format, not an ad-hoc dump.
A document MAY be concatenated for multiple elementary streams: each stream is its
own header line followed by its records. Readers MUST treat a Header line as the
start of a new stream section.
## 5. Document structure
```
<header> line 1 (exactly one Header object)
<record> line 2 .. N (one Picture record per picture, coded order)
[<header> <record>…] (OPTIONAL further stream sections)
```
## 6. Header object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `format` | string | MUST | Constant `"freemkv/video-index"`. Signature: a document begins with these bytes. |
| `fvi_version` | integer | MUST | Document format version. This spec defines `1`. |
| `generator` | string | SHOULD | Producing tool + version, e.g. `"freemkv/1.0.0-rc.6"`. |
| `stream` | object | MUST | The indexed elementary stream (§6.1). |
| `source` | object | MUST | Provenance root (§6.2). |
| `timescale` | integer | MUST | Ticks per second for all `pts`/`dts` (§10). E.g. `90000`. |
| `picture_count` | integer | MAY | Total pictures, if known at header time; OMITTED when streaming. |
### 6.1 `stream` object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `codec` | string | MUST | Registered codec id (Appendix B), e.g. `"mpeg2video"`, `"hevc"`. |
| `width`,`height` | integer | MUST | Coded luma dimensions in pixels. |
| `dar` | `[int,int]` | SHOULD | Display aspect ratio as `[num,den]`. |
| `frame_rate` | `[int,int]` | SHOULD | Nominal rate as exact rational `[num,den]` (e.g. `[24000,1001]`). |
| `scan` | string | MUST | `"progressive"`<br>`"interlaced"`<br>`"mbaff"` |
| `colour` | object | SHOULD | CICP per ITU-T H.273: `primaries`, `transfer`, `matrix` (integer CICP codes or registered names)<br>`range`: `"limited"` \| `"full"`<br>HDR: `mastering_display`, `max_cll`, `max_fall` per ITU-T H.273 / SMPTE ST 2086. |
| `language` | string | MAY | BCP 47 tag, if known. |
### 6.2 `source` object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `medium` | string | MUST | `"disc"`<br>`"iso"`<br>`"file"`<br>`"stream"` |
| `path` | string | MAY | Source path/label. |
| `title` | integer | MAY | Title/program number. |
| `playlist` | string | MAY | Playlist/PGC identifier. |
| `volume_id` | string | MAY | Disc volume identifier, if read. |
| `sector_size` | integer | SHOULD | Bytes per `src.sector` unit (e.g. `2048`). Lets readers convert `src` to an absolute byte offset. |
## 7. Picture record
One JSON object per coded picture, in coded order.
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `n` | integer | MUST | Coded-order index, 0-based, contiguous. |
| `src` | object | MUST | Provenance: `{ "file": int?, "sector": uint, "byte": uint }` — the offset of this AU's **first byte** in the source (§9). MUST be carried from demux, never reconstructed. |
| `type` | string | MUST | Coding type:<br>`"I"`<br>`"P"`<br>`"B"`<br>_ISO/IEC 13818-2 §6.3.9; H.264/H.265 slice types collapsed to frame type._ |
| `key` | boolean | MUST | `true` iff this picture is an intra (I) picture / parser-flagged decode-restart point (IDR / IRAP / I-picture).<br>_MPEG-2 open-GOP clean-RAP precision (`closed_gop`) is not currently distinguished — see note below._ |
| `gop` | boolean | SHOULD | `true` iff this picture begins a GOP / coded video sequence.<br>_Omitted when the implementation does not carry a distinct GOP-boundary signal._ |
| `pts` | integer\|null | SHOULD | Presentation timestamp in `timescale` ticks; `null` if unknown. |
| `dts` | integer\|null | MAY | Decode timestamp in `timescale` ticks. |
| `size` | integer | MAY | AU length in bytes; enables byte-range extraction with `src`. |
| `recovered` | boolean | MAY | `true` iff any byte of this AU came from a retried/marginal read (§9.1).<br>_Default `false`._ |
| codec ext | object | MAY | Codec-specific members under the codec's namespace (§8). |
The `type` and `key` members are **codec-agnostic** and MUST be populated for
every codec. `type` is the I/P/B coding type the parser decoded (collapsing
H.264/H.265 slice types to a frame type); where no per-picture coding is carried
(audio / synthetic frames), `type` is `"I"` for a key picture else `"P"`. `key`
is the picture's random-access flag as the codec parser sets it (IDR / IRAP /
I-picture). A writer MUST NOT emit a degraded record (`type:"?"` or `src:null`)
merely because a codec lacks per-picture coding info — those fallbacks are
reserved for a field that is genuinely unavailable (e.g. provenance absent on a
synthetic source).
> **Limitation (honest random-access).** `key` is set from the picture's
> intra / decode-restart flag. The per-picture coding model this index carries
> does **not** distinguish MPEG-2 open-GOP clean random-access points
> (`closed_gop`) from any other I-picture, so `key` is the parser-flagged
> decode-restart point, not a verified clean-RAP claim. A future revision MAY
> tighten `key` for codecs/profiles that carry that signal; readers MUST NOT
> assume present `key` precision beyond "intra / decode-restart point".
### 7.1 Interlace / pulldown fields
Codec-agnostic interlace/pulldown attributes, derived through the indexer's
per-picture coding accessors (MPEG-2: ISO/IEC 13818-2 §6.3.10). Emitted as
top-level members of the record, and ONLY when the codec actually measured the
signal — an OPTIONAL member that is omitted (not defaulted) when unknown:
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `field_order` | string | MAY | Display field order:<br>`"tff"` — top field first<br>`"bff"` — bottom field first<br>`"progressive"` — no field order applies<br>_Omitted when the codec did not signal it._ |
| `progressive` | boolean | MAY | `true` iff the picture is progressive.<br>_Omitted when the codec did not signal it._ |
| `nb_fields` | integer | MAY | Number of displayed field periods this picture occupies (the soft-telecine / 2:3 pulldown basis):<br>`1` for a single field picture<br>`2` for a normal frame<br>`3`/`4`/`6` for `repeat_first_field` pulldown per §6.3.10 |
Codecs that carry only a coding type (e.g. H.264 / HEVC / VC-1 through this
pipeline) omit `field_order` and `progressive` rather than guessing a default.
## 8. Codec model and extensibility
Core record members (§7) are codec-agnostic and present for every codec.
Codec-specific data is either (a) promoted to top-level members for a small,
registered set per codec profile (e.g. MPEG-2 §7.1), or (b) placed under an
`ext` object keyed by codec id for richer/optional data:
```json
{
"n": 42,
"type": "P",
"key": false,
"src": {
"sector": 17,
"byte": 924
},
"ext": {
"hevc": {
"temporal_id": 0,
"nal_type": 1
}
}
}
```
New codecs and members are added through Appendix B (codec registry) without a
breaking version bump, provided readers continue to ignore unknown members (§11).
## 9. Provenance and recovery semantics
`src` is **byte-exact** to the source as read. `src.sector` counts in
`source.sector_size`-byte units; `src.byte` is the offset within that sector of
the AU's first byte. For multi-file sources, `src.file` indexes a writer-declared
file list. Provenance MUST be the value observed at demux time; an implementation
MUST NOT recompute `src` by re-parsing — the point of FVI is to *carry* the truth.
### 9.1 Recovery
Because FVI is provenance-native, it can record reliability. A record with
`"recovered":true` indicates the AU's source bytes required retry/marginal-read
recovery. This lets downstream tools surface or quarantine pictures whose bytes
are not byte-identical to a clean read — a capability legacy index formats lack.
## 10. Time model
All `pts`/`dts` are integers in units of `1/timescale` seconds. `pts` is
presentation (display) time; `dts` is decode time. Records are in **coded**
(decode) order, so `pts` is not necessarily monotonic across records (B-pictures
reorder); `dts` is non-decreasing. Readers needing display order sort by `pts`.
## 11. Versioning and forward compatibility
- `fvi_version` is the document version; this spec defines `1`.
- **Additive** changes (new OPTIONAL members, new registered codecs) do NOT bump
`fvi_version`. Readers MUST ignore members they do not recognize.
- A change that alters the meaning of an existing member or makes a new member
REQUIRED bumps `fvi_version`.
- A reader encountering a higher `fvi_version` than it implements SHOULD process
the members it understands and MUST NOT reject the document solely for the
version being higher, unless a member it relies on is absent.
## 12. Conformance requirements (summary)
A conformant **writer** MUST: emit a Header first; emit records in coded order
with contiguous `n`; populate `src` from demux; use named/registered codec ids;
encode one JSON value per UTF-8 LF-terminated line.
A conformant **reader** MUST: accept any §4–§10 document; ignore unknown members;
not assume `picture_count`, `pts`, or `size` are present unless required above.
---
## Appendix A — JSON Schema (informative)
Header:
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["format", "fvi_version", "stream", "source", "timescale"],
"properties": {
"format": { "const": "freemkv/video-index" },
"fvi_version": { "type": "integer", "minimum": 1 },
"timescale": { "type": "integer", "minimum": 1 },
"stream": { "type": "object", "required": ["codec", "width", "height", "scan"] },
"source": { "type": "object", "required": ["medium"] }
}
}
```
Record:
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["n", "src", "type", "key"],
"properties": {
"n": { "type": "integer", "minimum": 0 },
"type": { "enum": ["I", "P", "B"] },
"key": { "type": "boolean" },
"src": {
"type": "object",
"required": ["sector", "byte"],
"properties": {
"file": { "type": "integer" },
"sector": { "type": "integer", "minimum": 0 },
"byte": { "type": "integer", "minimum": 0 }
}
}
}
}
```
## Appendix B — Registered codec identifiers
| `codec` | Bitstream | Field profile |
|---|---|---|
| `mpeg2video` | ISO/IEC 13818-2 | §7.1 (field_order/progressive/nb_fields) |
| `mpeg1video` | ISO/IEC 11172-2 | §7.1 |
| `h264` | ISO/IEC 14496-10 | core + `ext.h264` |
| `hevc` | ISO/IEC 23008-2 | core + `ext.hevc` |
| `vc1` | SMPTE 421M | core |
## Appendix C — Normative references
- RFC 2119, RFC 8174 — Requirement keywords (BCP 14).
- RFC 8259 — JSON.
- ISO/IEC 13818-2 — MPEG-2 video (picture coding, §6.3.96.3.10).
- ISO/IEC 14496-10 — H.264/AVC. ISO/IEC 23008-2 — H.265/HEVC.
- ITU-T H.273 — Coding-independent code points (colour primaries/transfer/matrix).
- SMPTE ST 2086 — Mastering display colour volume (HDR).
- BCP 47 — Language tags.
- RFC 9559 — Matroska (alignment of colour/field-order semantics).
+34
View File
@@ -0,0 +1,34 @@
# libfreemkv Documentation
Technical documentation for [libfreemkv](https://github.com/freemkv/libfreemkv), the open source optical drive library.
## Start Here
**[Disc to Rip: End-to-End Flow](disc-to-rip.md)** — How the library goes from a disc in the drive to decrypted content. Read this first.
## Reference
| Document | What it covers |
|----------|---------------|
| [Architecture](architecture.md) | Module map, design principles, error codes, platform support |
| [Drive Access](drive-access.md) | Drive, SCSI transport, profiles, unlock, why raw mode is needed |
| [Rip Recovery](rip-recovery.md) | Three-layer recovery model: Disc::patch, single-shot Drive::read, DiscStream batch halving |
| [AACS Encryption](aacs.md) | Key resolution (4 paths), content decryption, bus encryption, SCSI handshake |
| [UDF Filesystem](udf.md) | UDF 2.50 with metadata partitions, pointer chain, how files are read from disc |
| [MPLS Playlists](mpls.md) | Playlist format, play items, STN stream table, coding types |
| [CLPI Clip Info](clpi.md) | EP map (coarse + fine entries), timestamp-to-sector mapping, extent calculation |
| [API Design](api-design.md) | Stream API design, PES pipeline, input/output resolution |
## Reading Order
If you want to understand the whole library:
1. **[Disc to Rip](disc-to-rip.md)** — the big picture
2. **[Architecture](architecture.md)** — how modules fit together
3. **[Drive Access](drive-access.md)** — how we talk to hardware
4. **[UDF](udf.md)** → **[MPLS](mpls.md)** → **[CLPI](clpi.md)** — how disc content is structured
5. **[AACS](aacs.md)** — how encryption works and how we break it
## API Documentation
Generated API docs are on [docs.rs/libfreemkv](https://docs.rs/libfreemkv).
+55 -267
View File
@@ -2,206 +2,61 @@
## Overview ## Overview
AACS (Advanced Access Content System) is the encryption layer used by Blu-ray and UHD 4K discs to protect content. libfreemkv implements AACS decryption to enable transparent disc access. AACS (Advanced Access Content System) is the encryption layer used by Blu-ray
and UHD 4K discs to protect content. libfreemkv implements AACS decryption so
disc access is transparent to the application.
There are two major versions: There are two major versions:
- **AACS 1.0** -- Used by standard Blu-ray discs. Relies on a custom 160-bit elliptic curve for bus authentication and AES-128 for content encryption. Processing keys and device keys can derive the media key from the disc's Media Key Block (MKB). - **AACS 1.0** -- Used by standard Blu-ray discs.
- **AACS 2.0 / 2.1** -- Used by UHD 4K Blu-ray discs. Adds a per-sector bus
- **AACS 2.0** -- Used by UHD 4K Blu-ray discs. Adds a per-sector bus encryption layer (read_data_key) on top of the standard content encryption. Uses P-256/SHA-256 for its native handshake, though drives accept AACS 1.0 host certificates for backward compatibility. encryption layer on top of the standard content encryption. UHD drives accept
AACS 1.0 host credentials for backward compatibility.
Both versions use AES-128-CBC for content decryption with a fixed initialization vector. The fundamental key hierarchy is the same: a Volume Unique Key (VUK) decrypts per-title unit keys, which in turn decrypt the content stream.
All versions use AES-128 for content decryption. The library reads the keys it
needs from `keydb.cfg`, walks the disc's Media Key Block (MKB) to resolve the
## Architecture disc's key, and decrypts the content stream. AACS-encrypted discs therefore
require a `keydb.cfg`; CSS-protected DVDs do not (see the CSS notes in the
AACS support is split across two modules: library docs).
### `aacs.rs` -- Keys and Decryption ## How it works (feature level)
Handles everything related to key resolution and content decryption: When a disc is scanned, the library:
- KEYDB.cfg parsing (device keys, processing keys, host certificates, per-disc entries) 1. Reads the disc's AACS key-input files from the `/AACS/` directory.
- Disc hash computation (SHA-1 of `Unit_Key_RO.inf`) 2. Resolves the disc's key from `keydb.cfg` — either directly from a per-disc
- VUK resolution chain (4 paths, described below) entry, or by walking the MKB with the keys present in the keydb.
- MKB record parsing and media key derivation 3. Performs the drive-level SCSI authentication handshake needed to obtain the
- Subset-difference tree traversal (AACS-G3 key derivation) Volume ID and, for UHD, the bus-decryption key.
- Unit_Key_RO.inf parsing and unit key decryption 4. Decrypts the content stream as titles are read.
- Content Certificate parsing (AACS version detection)
- Aligned unit decryption (AES-128-CBC) A resolved key is verified against actual disc content before it is applied, so
- Bus decryption (AACS 2.0 read_data_key layer) a stale or wrong key fails loudly rather than producing silent garbage. If no
usable key is available for an AACS-encrypted disc, the library surfaces a
### `aacs_handshake.rs` -- SCSI Authentication specific error (the E70xx family) describing which part of the chain was
missing, and a missing `keydb.cfg` surfaces as `Error::KeydbLoad` with the
Handles the drive-level SCSI authentication protocol: sentinel path `<no keydb in search paths>`.
- ECDH key agreement on the AACS 160-bit curve
- ECDSA signing and verification
- Bus key derivation
- AGID management (allocate/invalidate)
- Volume ID retrieval (encrypted with bus key, verified by AES-CMAC)
- Read Data Key retrieval (for AACS 2.0 bus decryption)
- AACS LA public key certificate verification
## Key Resolution Chain
When a disc is scanned, `resolve_keys()` attempts four paths in priority order. The first path that succeeds is used.
### Path 1: KEYDB VUK Lookup (fastest)
```
Unit_Key_RO.inf --> SHA-1 --> disc_hash --> KEYDB lookup --> VUK
```
The disc hash is computed as the SHA-1 digest of the raw `Unit_Key_RO.inf` file from the disc's `/AACS/` directory. This hash is used as the lookup key in `KEYDB.cfg`. If a matching entry contains a VUK (`V` field), it is used directly.
This is the fast path and resolves the vast majority of discs in a well-maintained KEYDB.
### Path 2: KEYDB Media Key + Volume ID
```
KEYDB media_key + Volume ID (from SCSI handshake) --> VUK derivation
```
If the disc hash is not in the KEYDB but a KEYDB entry has a matching Volume ID (`I` field) and a media key (`M` field), the VUK is derived:
```
VUK = AES-128-ECB-DECRYPT(media_key, volume_id) XOR volume_id
```
Requires a successful SCSI handshake to obtain the Volume ID.
### Path 3: MKB + Processing Keys
```
MKB (from disc) + processing_keys (from KEYDB) --> media_key --> VUK
```
Processing keys are pre-computed keys that work against specific MKB versions. For each processing key, the library:
1. Parses the MKB to extract the Verify Media Key Record (`mk_dv`), subset-difference index, and conditional values (cvalues).
2. Tries each processing key against each UV/cvalue pair: `mk = AES-DEC(pk, cvalue) XOR cvalue`.
3. Validates the derived media key: `AES-ECB(mk, mk_dv)` must produce 12 leading zero bytes.
4. Derives VUK from the validated media key and Volume ID.
### Path 4: MKB + Device Keys (Subset-Difference Tree)
```
MKB + device_keys --> subset-difference tree traversal --> processing_key --> media_key --> VUK
```
The most complex path. Each device key has an associated node number, UV value, and mask parameters that position it in the AACS subset-difference tree. The library:
1. Finds the subset-difference entry in the MKB that applies to the device key's node.
2. Traverses the tree using AACS-G3 key derivation: `aesg3(key, inc) = AES-DEC(key, seed) XOR seed`, where `seed[15]` is incremented by `inc`. Each tree node produces a left child (inc=0), a processing key (inc=1), and a right child (inc=2).
3. At each level, selects left or right based on the UV bit at the current position.
4. The resulting processing key is validated against the MKB cvalue to derive the media key.
5. VUK is derived from the media key and Volume ID.
## Content Decryption
### Aligned Units
AACS encrypts content in aligned units of 6144 bytes (3 sectors of 2048 bytes each). The encryption flag is signaled by the copy_permission_indicator bits in byte 0 of the unit (`unit[0] & 0xC0 != 0`).
### Per-Unit Key Derivation
Each aligned unit has its own decryption key derived from the CPS unit key:
1. **Derive**: AES-128-ECB encrypt the first 16 bytes of the unit (plaintext TP_extra_header) with the unit key.
2. **XOR**: XOR the encrypted result with the original 16 bytes to produce the per-unit decryption key.
3. **Decrypt**: AES-128-CBC decrypt bytes 16 through 6143 using the per-unit key and the fixed AACS IV.
4. **Clear flag**: Clear the encryption indicator bits (`unit[0] &= !0xC0`).
### Fixed IV
All AES-CBC operations in AACS use the same fixed initialization vector, defined in the AACS specification.
### Verification
After decryption, the library verifies correctness by checking for MPEG-TS sync bytes (0x47) at the expected 192-byte packet boundaries within the unit. Blu-ray transport stream packets are 192 bytes: 4-byte TP_extra_header followed by a 188-byte TS packet.
## Bus Encryption
### AACS 1.0
Standard Blu-ray discs do not use bus encryption. Content is read directly from the disc and decrypted using the unit key.
### AACS 2.0
UHD 4K discs add a per-sector bus encryption layer. The drive encrypts data as it is read from the disc, and the host must decrypt it before applying AACS content decryption.
Bus encryption uses a **read_data_key** obtained during the SCSI handshake. For each 2048-byte sector within an aligned unit, bytes 16 through 2047 are AES-128-CBC encrypted with the read_data_key and the fixed AACS IV. The first 16 bytes of each sector remain plaintext.
The full decryption pipeline for AACS 2.0:
1. **Bus decrypt**: For each sector, AES-128-CBC decrypt bytes 16..2047 with the read_data_key.
2. **Content decrypt**: Standard per-unit key derivation and AES-128-CBC decryption as described above.
## SCSI Handshake
The AACS SCSI authentication handshake establishes a shared bus key between host and drive, then uses it to securely transfer the Volume ID and read data keys.
### Protocol Flow
1. **Invalidate AGIDs**: Send REPORT KEY with format 0x3F for AGIDs 0-3 to clear stale sessions.
2. **Allocate AGID**: REPORT KEY format 0x00 returns a fresh Authentication Grant ID.
3. **Send host credentials**: SEND KEY format 0x01 transmits the host nonce (20 random bytes) and host certificate (92 bytes).
4. **Receive drive credentials**: REPORT KEY format 0x01 returns the drive nonce and drive certificate.
5. **Receive drive key**: REPORT KEY format 0x02 returns the drive's ephemeral EC key point and ECDSA signature over `host_nonce || drive_key_point`.
6. **Verify drive key**: The signature is verified against the drive's public key (extracted from its certificate). AACS 1.0 certificates are verified against the AACS LA public key.
7. **Send host key**: The host generates an ephemeral key pair, signs `drive_nonce || host_key_point` with the host private key, and sends via SEND KEY format 0x02.
8. **Compute bus key**: ECDH shared secret = `host_private_key * drive_key_point`. The bus key is the low 128 bits of the shared point's x-coordinate.
### Post-Authentication Reads
- **Volume ID**: REPORT DISC STRUCTURE format 0x80. Returns 16-byte VID encrypted with the bus key, plus an AES-CMAC MAC for integrity verification.
- **Read Data Keys**: REPORT DISC STRUCTURE format 0x84. Returns the read_data_key and write_data_key, each AES-ECB encrypted with the bus key.
### Elliptic Curve
AACS 1.0 uses a custom 160-bit Weierstrass curve (`y^2 = x^3 + ax + b mod p`) with 20-byte field elements. The library implements full EC arithmetic: point addition, doubling, scalar multiplication, modular inverse, ECDSA sign/verify, and ECDH key agreement.
## AACS 2.0 Status
AACS 2.0 discs are detected via the Content Certificate file (`Content000.cer` or `Content001.cer`). A certificate type byte of 0x01 indicates AACS 2.0.
AACS 2.0 drives are identified by their drive certificate type (0x11). These drives natively use P-256/SHA-256, but accept AACS 1.0 host certificates for backward compatibility.
Current implementation status:
- AACS 2.0 detection: **implemented** (Content Certificate parsing, drive cert type check)
- AACS 1.0 handshake with AACS 2.0 drives: **implemented** (backward compatibility mode)
- Full P-256 AACS 2.0 handshake: **not yet implemented** (prepared but rarely needed since drives accept AACS 1.0 host certs)
- Bus decryption with read_data_key: **implemented**
- Content decryption: **implemented** (same as AACS 1.0)
In practice, AACS 2.0 UHD discs work through the backward-compatible AACS 1.0 handshake path, with the addition of read_data_key bus decryption.
## API Usage ## API Usage
AACS decryption is transparent to the application. The `Disc::scan()` method handles everything automatically: AACS decryption is transparent to the application. `Disc::scan()` handles
everything automatically:
```rust ```rust
use libfreemkv::{DriveSession, Disc}; use libfreemkv::{Drive, Disc};
use libfreemkv::disc::ScanOptions; use libfreemkv::disc::ScanOptions;
use std::path::Path; use std::path::Path;
let mut session = DriveSession::open(Path::new("/dev/sr0")).unwrap(); let mut drive = Drive::open(Path::new("/dev/sg4")).unwrap();
let disc = Disc::scan(&mut session, &ScanOptions::default()).unwrap(); drive.wait_ready().unwrap();
drive.init().unwrap();
let disc = Disc::scan(&mut drive, &ScanOptions::default()).unwrap();
// Check encryption state // Check encryption state
if disc.encrypted { if disc.encrypted {
if let Some(ref aacs) = disc.aacs { if let Some(ref aacs) = disc.aacs {
println!("AACS {}.0", aacs.version); println!("AACS {}.0", aacs.version);
println!("Key source: {}", aacs.key_source.name()); println!("Key source: {}", aacs.key_source.name());
println!("Disc hash: {}", aacs.disc_hash);
if let Some(mkb_ver) = aacs.mkb_version { if let Some(mkb_ver) = aacs.mkb_version {
println!("MKB version: {}", mkb_ver); println!("MKB version: {}", mkb_ver);
} }
@@ -213,108 +68,41 @@ if disc.encrypted {
// Read content -- decryption is automatic // Read content -- decryption is automatic
let mut reader = disc.open_title(&mut session, 0).unwrap(); let mut reader = disc.open_title(&mut session, 0).unwrap();
while let Some(unit) = reader.read_unit().unwrap() { while let Some(unit) = reader.read_unit().unwrap() {
// unit is 6144 bytes of decrypted content // decrypted content
} }
``` ```
The application never touches keys, never calls decryption functions, and never manages handshakes. All of that is internal to `Disc::scan()` and `ContentReader::read_unit()`. The application never touches keys, never calls decryption functions, and never
manages handshakes. All of that is internal to `Disc::scan()` and the content
reader.
### KEYDB Location ### KEYDB Location
`ScanOptions` controls where the KEYDB is loaded from. If no explicit path is set, the library checks: `ScanOptions` controls where the keydb is loaded from. If no explicit path is
set, the library checks the standard config locations. To specify an explicit
1. `~/.config/aacs/KEYDB.cfg` path:
2. `/etc/aacs/KEYDB.cfg`
To specify an explicit path:
```rust ```rust
let opts = ScanOptions::with_keydb("/path/to/KEYDB.cfg"); let opts = ScanOptions::with_keydb("/path/to/keydb.cfg");
let disc = Disc::scan(&mut session, &opts).unwrap(); let disc = Disc::scan(&mut session, &opts).unwrap();
``` ```
### AacsState ### AacsState
After a successful scan, `disc.aacs` contains an `AacsState` with: After a successful scan, `disc.aacs` contains an `AacsState`:
| Field | Type | Description | | Field | Type | Description |
|-------|------|-------------| |-------|------|-------------|
| `version` | `u8` | AACS version (1 or 2) | | `version` | `u8` | AACS version (1 or 2) |
| `bus_encryption` | `bool` | Whether bus encryption is active | | `bus_encryption` | `bool` | Whether bus encryption is active |
| `mkb_version` | `Option<u32>` | MKB version from disc | | `mkb_version` | `Option<u32>` | MKB version from disc |
| `disc_hash` | `String` | SHA-1 of Unit_Key_RO.inf (hex with 0x prefix) | | `disc_hash` | `String` | Identifier for the disc's key-input files |
| `key_source` | `KeySource` | How keys were resolved | | `key_source` | `KeySource` | How the disc's key was resolved |
| `vuk` | `[u8; 16]` | Volume Unique Key |
| `unit_keys` | `Vec<(u32, [u8; 16])>` | Decrypted unit keys (CPS unit number, key) |
| `read_data_key` | `Option<[u8; 16]>` | AACS 2.0 bus decryption key |
| `volume_id` | `[u8; 16]` | Volume ID from SCSI handshake |
### KeySource ## keydb.cfg
| Variant | Description | `keydb.cfg` is the single source of AACS key material. It is a text file (lines
|---------|-------------| starting with `;` or `#` are comments) holding the host credentials and per-disc
| `KeyDb` | VUK found directly in KEYDB by disc hash | entries the library uses to resolve a disc. autorip can auto-download and
| `KeyDbDerived` | Media key + Volume ID from KEYDB, VUK derived | refresh it from a configured URL. The library does not ship any AACS keys
| `ProcessingKey` | MKB + processing keys from KEYDB | compiled into the binary.
| `DeviceKey` | MKB + device keys, subset-difference tree traversal |
## KEYDB.cfg Format Reference
The KEYDB.cfg file contains all cryptographic material needed for AACS decryption. Lines starting with `;` or `#` are comments.
### Device Keys
```
| DK | DEVICE_KEY 0x<key> | DEVICE_NODE 0x<node> | KEY_UV 0x<uv> | KEY_U_MASK_SHIFT 0x<shift>
```
- `key`: 16-byte AES device key (hex)
- `node`: Device node number in the subset-difference tree (hex)
- `uv`: UV value for tree positioning (hex)
- `shift`: U mask shift value (hex)
### Processing Keys
```
| PK | 0x<key>
```
- `key`: 16-byte pre-computed processing key (hex)
### Host Certificate
```
| HC | HOST_PRIV_KEY 0x<privkey> | HOST_CERT 0x<cert>
```
- `privkey`: 20-byte ECDSA private key (hex)
- `cert`: 92-byte AACS host certificate (hex)
The host certificate is used for SCSI authentication. It contains the host's public key and is signed by the AACS Licensing Administrator.
### Disc Entries
```
0x<disc_hash> = <title> | D | <date> | M | 0x<media_key> | I | 0x<disc_id> | V | 0x<vuk> | U | <unit_keys>
```
- `disc_hash`: 20-byte SHA-1 of Unit_Key_RO.inf (hex)
- `title`: Human-readable disc title
- `D`: Date tag, followed by release/rip date
- `M`: Media key tag, followed by 16-byte media key (hex)
- `I`: Disc ID tag, followed by 16-byte Volume ID (hex)
- `V`: VUK tag, followed by 16-byte Volume Unique Key (hex)
- `U`: Unit keys tag, followed by space-separated `<unit_num>-0x<key>` pairs
All fields after the title are optional. A minimal entry needs only the disc hash and VUK:
```
0x<disc_hash> = <title> | V | 0x<vuk>
```
Inline comments are supported with `;`:
```
0x<disc_hash> = <title> | V | 0x<vuk> ; MKBv77
```
+234
View File
@@ -0,0 +1,234 @@
# libfreemkv API Design
## Principles
1. Lib provides building blocks. App composes them.
2. No English text in lib. Error codes only. App handles i18n.
3. No display logic in lib. App decides what to show.
4. Streams are the pipeline. Each stage wraps the next.
5. Lib fires events. App listens.
## Core API
```rust
// Open drive — explicit steps, app prints between them
let mut drive = Drive::open(path)?;
drive.wait_ready()?;
drive.init()?;
drive.probe_disc()?;
// Scan disc
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
// Browse
disc.titles // Vec<DiscTitle>
disc.format // BD / UHD / DVD
disc.capacity_gb()
```
## PES Pipeline (primary API)
The PES pipeline is the main way to move content. All streams produce/consume
PES frames. The pipeline just reads frames and writes frames.
```rust
// URL-based — any source to any destination
let opts = InputOptions::default();
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
let title = input.info().clone();
let mut output = libfreemkv::output("mkv://Movie.mkv", &title)?;
while let Ok(Some(frame)) = input.read() {
output.write(&frame)?;
}
output.finish()?;
```
The `FrameSource` and `FrameSink` traits — direction is type-checked, so
calling `read()` on a write-only sink (or `write()` on a read-only source)
is a compile error rather than a runtime fault:
```rust
pub trait FrameSource: Send {
fn read(&mut self) -> Result<Option<PesFrame>, Error>;
fn info(&self) -> &DiscTitle;
fn codec_private(&self, track: usize) -> Option<Vec<u8>> { None }
fn headers_ready(&self) -> bool { true }
}
pub trait FrameSink: Send {
fn write(&mut self, frame: &PesFrame) -> Result<(), Error>;
fn finish(self: Box<Self>) -> Result<(), Error>;
fn info(&self) -> &DiscTitle;
}
```
## Streams
All streams implement `FrameSource` (read) and/or `FrameSink` (write); the
directional split prevents runtime "wrong-direction" errors. URL-based
resolvers open any stream by string.
| Stream | Input | Output | URL | Transport |
|--------|-------|--------|-----|-----------|
| DiscStream | Yes | -- | `disc://` `disc:///dev/sg4` | Optical drive via SCSI |
| IsoStream | Yes | Yes | `iso://path.iso` | Blu-ray ISO image |
| MkvStream | Yes | Yes | `mkv://path` | Matroska container |
| M2tsStream | Yes | Yes | `m2ts://path` | BD-TS with FMKV metadata header |
| NetworkStream | Yes (listen) | Yes (connect) | `network://host:port` | TCP with FMKV metadata header |
| StdioStream | Yes (stdin) | Yes (stdout) | `stdio://` | Raw byte pipe |
| NullStream | -- | Yes | `null://` | Discard sink (byte counter) |
All URLs require a `scheme://path` format. Bare paths are rejected.
```rust
// PES pipeline (frame-level) — input() returns Box<dyn FrameSource>,
// output() returns Box<dyn FrameSink>.
let input = libfreemkv::input("disc:///dev/sg4", &opts)?; // DiscStream
let input = libfreemkv::input("iso://Movie.iso", &opts)?; // IsoStream
let output = libfreemkv::output("mkv://Movie.mkv", &title)?; // MkvOutputStream
let output = libfreemkv::output("m2ts://Movie.m2ts", &title)?; // M2tsOutputStream
let output = libfreemkv::output("network://192.0.2.10:9000", &title)?; // NetworkOutputStream
let output = libfreemkv::output("null://", &title)?; // NullOutputStream
```
### FMKV Metadata Header
M2tsStream and NetworkStream embed a JSON metadata header before the BD-TS data:
```
[8B magic "FMKV\0\0\0\0"][4B JSON length][JSON metadata][padding to 192B boundary][BD-TS data...]
```
The header carries title name, duration, codec_privates, and full stream layout
(PIDs, codecs, languages, labels). This allows the receiving end to set up
demuxing and track metadata without scanning the TS.
## Events
Lib fires events during operations. App provides a callback. No display, no text.
```rust
pub struct Event {
pub kind: EventKind,
}
pub enum EventKind {
// Init / scan
DriveOpened { device: String },
DriveReady,
InitComplete { success: bool },
ProbeComplete { success: bool },
ScanComplete { titles: usize },
// Read pipeline
BytesRead { bytes: u64, total: u64 },
ReadError { sector: u64, error: Error },
SpeedChange { speed_kbs: u16 },
ExtentStart { index: usize, start_sector: u64, sector_count: u64 },
SectorSkipped { sector: u64 },
BatchSizeChanged { new_size: u16, reason: BatchSizeReason },
Complete { bytes: u64, errors: u32 },
// Kept for forward-compat; not emitted in 0.13.6+
Retry { attempt: u32 },
SectorRecovered { sector: u64 },
}
```
Emission notes:
- `BytesRead { bytes, total }` is emitted from `DiscStream::fill_extents`
after each successful sector read. `bytes` is the cumulative running
total; `total` is the precomputed extent sum (0 if unknown).
- `SpeedChange` is emitted from the public `Drive::set_speed` API path.
It is no longer emitted from a recovery hot loop (recovery loop removed
in 0.13.6).
- `BatchSizeChanged` fires from the `DiscStream` adaptive sizer on shrink
(read failed at a larger size) and on probe-up (clean-read streak hit
the threshold). Consumers use it to display a "recovering" state
distinct from "ripping normally".
- `Retry` and `SectorRecovered` are NOT emitted in 0.13.6+. They were
tied to the inline `Drive::read` recovery phases that were removed; the
variants are kept for forward compatibility so consumers' match arms
don't need conditional compilation.
Events report what happened. App decides what to do. GUI shows a dialog. CLI
prints a line. Server logs to file.
## File Layout
```
libfreemkv/src/
├── lib.rs Public exports
├── error.rs Error codes (no English)
├── event.rs Event types for callbacks
├── halt.rs Halt cancellation token (Arc<AtomicBool> wrapper)
├── io/ Pipeline + WritebackFile primitives
│ ├── mod.rs Re-exports WritebackFile, Pipeline, Sink, Flow
│ ├── pipeline.rs Generic Pipeline<I, R> + Sink trait
│ ├── writeback_file.rs WritebackFile (was crate::io::Writer)
│ └── writeback.rs sync_file_range pipeline
├── drive/ Drive (open, init, single-shot read)
│ ├── mod.rs Drive struct, init, read (single-shot), reset, eject
│ ├── capture.rs Raw drive SCSI capture (INQUIRY/GET_CONFIG) for contribution
│ ├── linux.rs Linux drive discovery
│ ├── macos.rs macOS drive discovery
│ └── windows.rs Windows drive discovery
├── disc/ Disc (scan, titles, AACS setup, sweep, patch)
│ ├── mod.rs Disc struct, scan, titles, formats; Disc::copy + Disc::sweep (Pass 1)
│ ├── sweep.rs Pass 1 internal helpers (pub(super))
│ ├── patch.rs Disc::patch (Pass N retry over mapfile)
│ ├── mapfile.rs ddrescue-format mapfile
│ └── read_error.rs ReadCtx / ReadAction state machine
├── scsi/ SCSI transport (Linux SG_IO, macOS IOKit, Windows SPTI)
├── unlock.rs Unlocker trait + registry (pluggable unlock seam)
├── aacs/ AACS decryption (handshake, keys, keydb, decrypt)
├── css/ DVD CSS cipher
├── decrypt.rs Unified decrypt dispatcher (AACS/CSS/None)
├── pes.rs PES frame types, FrameSource / FrameSink traits
├── sector/ Sector I/O
│ ├── mod.rs SectorSource, SectorSink traits
│ ├── file.rs FileSectorSource, FileSectorSink (ISO-backed)
│ └── decrypting.rs DecryptingSectorSource decorator
├── udf.rs UDF 2.50 filesystem parser
├── mpls.rs MPLS playlist parser
├── clpi.rs CLPI clip info parser
├── ifo.rs DVD IFO parser
├── labels/ BD-J label extraction (5 format parsers)
├── keydb.rs KEYDB download, parse, save
├── identity.rs DriveId from INQUIRY
├── speed.rs DriveSpeed enum
├── mux/
│ ├── mod.rs Public mux exports
│ ├── resolve.rs URL parser + input/output (Box<dyn FrameSource/Sink>)
│ ├── meta.rs FMKV header format
│ ├── disc.rs DiscStream (optical drive → PES)
│ ├── iso.rs IsoStream (ISO image read)
│ ├── isowriter.rs ISO image writer (UDF, AVDP, multi-extent)
│ ├── mkvstream.rs MkvStream (bidirectional Matroska)
│ ├── mkvout.rs MkvOutputStream (PES → MKV)
│ ├── m2ts.rs M2tsStream (BD-TS)
│ ├── pesout.rs PES output streams (M2ts, Network, Stdio, Null)
│ ├── network.rs NetworkStream (TCP + FMKV header)
│ ├── stdio.rs StdioStream (stdin/stdout pipe)
│ ├── null.rs NullStream (discard + byte counter)
│ ├── lookahead.rs LookaheadBuffer (codec header scanning)
│ ├── ts.rs BD-TS demuxer + PAT/PMT scanner
│ ├── tsreader.rs TS reader utilities
│ ├── tsmux.rs TS muxer (PES → BD-TS packets)
│ ├── ps.rs MPEG-2 PS demuxer (DVD)
│ ├── ebml.rs EBML read/write primitives
│ ├── mkv.rs MKV muxer (tracks, clusters, cues)
│ └── codec/ Frame parsers (H.264, HEVC, MPEG-2, VC-1, AC3, EAC3, DTS, TrueHD, LPCM, PGS)
└── ...
freemkv/src/
├── main.rs CLI dispatcher (URL routing)
├── pipe.rs PES pipeline — source → dest copy
├── disc_info.rs Disc/file info display
├── info.rs Drive info + profile submission
├── strings.rs i18n string table
├── output.rs Verbosity-filtered output
└── build.rs Bundled locale code generation
```
+90 -49
View File
@@ -1,8 +1,10 @@
# libfreemkv Architecture # libfreemkv Architecture
Open source optical drive access library for 4K UHD Blu-ray, Blu-ray, and DVD. Open source optical drive access library for 4K UHD Blu-ray, Blu-ray, and DVD.
Rust library with no external dependencies at runtime -- profiles are bundled, Rust library with profiles bundled and all SCSI communication handled in-process.
AACS keys are derived internally, and all SCSI communication is handled in-process. AACS decryption requires an external `keydb.cfg` (default
`~/.config/freemkv/keydb.cfg`) — the derivation math is internal, but no AACS key
material is compiled in; DVD CSS player keys are the only compiled-in keys.
**Repository:** <https://github.com/freemkv/libfreemkv> **Repository:** <https://github.com/freemkv/libfreemkv>
**License:** AGPL-3.0-only **License:** AGPL-3.0-only
@@ -13,22 +15,27 @@ AACS keys are derived internally, and all SCSI communication is handled in-proce
1. **CLI is dumb.** All drive communication, disc parsing, AACS decryption, and 1. **CLI is dumb.** All drive communication, disc parsing, AACS decryption, and
format handling live in the library. CLI binaries are thin wrappers that call format handling live in the library. CLI binaries are thin wrappers that call
`DriveSession::open()` and `Disc::scan()`. `Drive::open()` and `Disc::scan()`.
2. **No external files.** 206 drive profiles are compiled into the binary via 2. **Firmware-clean core.** libfreemkv ships no firmware, no unlock CDBs, and no
`include_str!`. No configuration directory, no runtime file lookups for drive drive profiles. Drive-unlock logic is plugged in by an external crate through
support. the `Unlocker` trait + registry (`register_unlocker`); without one the library
still rips via the host-certificate AACS handshake.
3. **Transparent AACS.** The `ContentReader` decrypts on the fly when keys are 3. **Transparent AACS.** The `ContentReader` decrypts on the fly when keys are
available. Callers read cleartext sectors without knowing whether the disc available. Callers read cleartext sectors without knowing whether the disc
was encrypted. was encrypted.
4. **Structured errors, no English.** Every error has a numeric code (E1000-E7000). 4. **Structured errors, no English.** Every error has a numeric code (E1000-E8000).
The library never formats user-facing messages -- applications do that. The library never formats user-facing messages -- applications do that.
5. **Library-agnostic.** No concept of "supported" vs "unsupported" drives at a 5. **Library-agnostic.** No concept of "supported" vs "unsupported" drives at a
policy level. If a profile exists, the library uses it. policy level. If a profile exists, the library uses it.
6. **Streams are dumb pipes.** Streams read/write PES frames. They don't know
about encryption, transport format, or source type. Decrypt is a stream-internal
concern; the pipeline just moves frames.
--- ---
## Module Map ## Module Map
@@ -37,26 +44,40 @@ AACS keys are derived internally, and all SCSI communication is handled in-proce
libfreemkv (lib.rs) libfreemkv (lib.rs)
├── Drive Access ├── Drive Access
│ ├── drive DriveSession — open, identify, unlock, read │ ├── drive Drive — open, identify, init, single-shot read
│ ├── scsi ScsiTransport trait + SG_IO implementation │ ├── scsi ScsiTransport trait + platform backends (sg async, IOKit, SPTI)
│ ├── platform/ Platform trait — per-chipset command handlers │ ├── unlock Unlocker trait + registry — the pluggable unlock seam
│ │ └── mt1959 MediaTek MT1959 driver (LG, ASUS, hp)
│ ├── profile DriveProfile loading, matching, bundled JSON
│ ├── identity DriveId from INQUIRY + GET_CONFIG 010C │ ├── identity DriveId from INQUIRY + GET_CONFIG 010C
── speed DriveSpeed enum, SET CD SPEED CDB builder ── speed DriveSpeed enum, SET CD SPEED CDB builder
│ └── event Event system for drive status callbacks
├── Disc Scanning ├── Disc Scanning
│ ├── disc Disc::scan() — titles, streams, extents, AACS setup │ ├── disc Disc::scan() — titles, streams, extents, AACS setup
│ ├── udf UDF 2.50 filesystem reader (metadata partitions) │ ├── udf UDF 2.50 filesystem reader (metadata partitions)
│ ├── mpls MPLS playlist parser — clips, streams, STN table │ ├── mpls MPLS playlist parser — clips, streams, STN table
│ ├── clpi CLPI clip info parser — EP map, sector extents │ ├── clpi CLPI clip info parser — EP map, sector extents
── jar BD-J JAR label extraction (audio/subtitle names) ── ifo DVD IFO parser — title sets, PGC chains, cell addresses
│ └── labels/ BD-J label extraction (5 formats: Paramount, Criterion, Pixelogic, CTRM, Deluxe)
├── Encryption ├── Encryption
│ ├── aacs KEYDB parsing, VUK lookup, MKB processing, unit decryption │ ├── aacs/ AACS handshake, KEYDB, VUK lookup, MKB, unit decryption
── aacs_handshake ECDH bus authentication, Volume ID, Read Data Key ── css DVD CSS cipher — table-driven, no external keys needed
│ └── decrypt decrypt_sectors() — unified AACS/CSS/None dispatcher
── error Error enum with numeric codes E1000-E7000 ── Streaming
│ ├── mux/ Stream implementations (Disc, ISO, MKV, M2TS, Network, Stdio, Null)
│ ├── pes PES frame types; the unified pes::Stream (PesStream) read/write trait
│ └── sector/ SectorSource / SectorSink traits, FileSector{Source,Sink}, DecryptingSectorSource
├── I/O Primitives
│ ├── halt Halt cancellation token (one Arc<AtomicBool>, cloneable)
│ └── io/ Pipeline<I, R> + Sink trait + WritebackFile (bounded-cache writer)
├── Support
│ ├── keydb KEYDB.cfg download, parse, verify, save
│ └── error Error enum with numeric codes E1000-E8000
└── lib.rs Public API re-exports
``` ```
--- ---
@@ -64,27 +85,34 @@ libfreemkv (lib.rs)
## Drive Access Flow ## Drive Access Flow
``` ```
DriveSession::open("/dev/sr0") Drive::open(Path::new("/dev/sg4"))
├─ scsi::open() Open /dev/sr0 via SG_IO ├─ scsi::open() Open /dev/sg4 (async write/poll/read)
├─ DriveId::from_drive() INQUIRY + GET_CONFIG 010C ├─ DriveId::from_drive() INQUIRY + GET_CONFIG 010C
profile::find_by_drive_id() Match against 206 bundled profiles Drive ready for init/read
├─ Platform::new() Instantiate chipset driver (Mt1959)
└─ Platform::unlock() Activate raw disc access mode
``` ```
After open, the session provides: After open:
- `read_sectors(lba, count, buf)` -- raw sector reads (through platform driver) - `init()` -- routes to the matching registered unlocker (if any); otherwise
- `read_disc(lba, count, buf)` -- standard READ(10) for filesystem data a no-op and the cert handshake carries the disc
- `scsi_execute(cdb, dir, buf, timeout)` -- arbitrary SCSI commands - `probe_disc()` -- probe disc surface for optimal speeds
- `status()`, `calibrate()`, `read_config()`, `read_register()` - `read(lba, count, buf, recovery)` -- single-shot read; `recovery` only selects the per-CDB timeout (1.5 s vs. 30 s)
- `wait_ready()` -- wait for disc insertion
- `eject()` -- eject tray
Recovery is layered above `Drive::read`, not inside it. Layer 1
(`Disc::patch`) handles bad-range retry by replaying the ddrescue mapfile.
Layer 3 (`DiscStream::fill_extents` adaptive batch sizer) handles in-loop
request-size adaptation. Inline recovery (gentle retry → SCSI reset → retry)
was removed in 0.13.6 — see [`rip-recovery.md`](rip-recovery.md) and
the stop-wedge postmortem (2026-04-25).
--- ---
## Disc Scanning Flow ## Disc Scanning Flow
``` ```
Disc::scan(&mut session, &ScanOptions) Disc::scan(&mut drive, &ScanOptions)
├─ READ CAPACITY Get disc size in sectors ├─ READ CAPACITY Get disc size in sectors
├─ udf::read_filesystem() Parse UDF 2.50 (AVDP → VDS → metadata → FSD → root) ├─ udf::read_filesystem() Parse UDF 2.50 (AVDP → VDS → metadata → FSD → root)
@@ -92,14 +120,24 @@ Disc::scan(&mut session, &ScanOptions)
│ ├─ mpls::parse() Extract play items, STN streams │ ├─ mpls::parse() Extract play items, STN streams
│ └─ For each clip: │ └─ For each clip:
│ └─ clpi::parse() EP map → sector extents for the clip's time range │ └─ clpi::parse() EP map → sector extents for the clip's time range
├─ labels::detect() Parse BD-J JARs for stream labels
├─ Detect AACS Check for /AACS directory on disc ├─ Detect AACS Check for /AACS directory on disc
└─ Disc::setup_aacs() Handshake + KEYDB → VUK → unit keys (if encrypted) └─ Disc::setup_aacs() Handshake + KEYDB → VUK → unit keys (if encrypted)
``` ```
For DVD:
```
Disc::scan_dvd(&mut drive, &ScanOptions)
├─ ifo::parse() Parse VIDEO_TS.IFO — title sets, PGC chains
├─ CSS detection Check disc structure flag
└─ CSS key cracking Table-driven, no KEYDB needed
```
The result is a `Disc` with: The result is a `Disc` with:
- `titles: Vec<Title>` -- sorted by duration, each with streams and sector extents - `titles: Vec<DiscTitle>` -- sorted by duration, each with streams, sector extents, codec_privates
- `aacs: Option<AacsState>` -- decryption keys if available - `decrypt_keys()` -- DecryptKeys for content decryption
- `encrypted: bool` -- whether the disc uses AACS - `encrypted: bool` -- whether the disc uses AACS/CSS
--- ---
@@ -114,13 +152,14 @@ Four key resolution paths, tried in order:
| 3 | Processing Keys + MKB → Media Key → VUK | Medium | | 3 | Processing Keys + MKB → Media Key → VUK | Medium |
| 4 | Device Keys + MKB subset-difference tree → VUK | Slow | | 4 | Device Keys + MKB subset-difference tree → VUK | Slow |
The AACS handshake (`aacs_handshake`) performs ECDH key agreement over the The AACS handshake (`aacs/handshake`) performs ECDH key agreement over the
AACS 1.0 160-bit elliptic curve to obtain: AACS 1.0 160-bit elliptic curve to obtain:
- **Volume ID** -- needed for VUK derivation (paths 2-4) - **Volume ID** -- needed for VUK derivation (paths 2-4)
- **Read Data Key** -- needed for AACS 2.0 (UHD) bus decryption - **Read Data Key** -- needed for AACS 2.0 (UHD) bus decryption
Content decryption uses AES-128-CBC on 6144-byte aligned units. The Content decryption uses AES-128-CBC on 6144-byte aligned units. The
`ContentReader` handles this transparently. `ContentReader` handles this transparently. Streams that read sectors
(DiscStream, IsoStream) decrypt internally — the pipeline sees clean bytes.
--- ---
@@ -138,6 +177,7 @@ is baked into the library.
| E5xxx | I/O errors | `IoError` (wraps `std::io::Error`) | | E5xxx | I/O errors | `IoError` (wraps `std::io::Error`) |
| E6xxx | Disc format errors | `DiscError` (UDF, MPLS, CLPI parse failures) | | E6xxx | Disc format errors | `DiscError` (UDF, MPLS, CLPI parse failures) |
| E7xxx | AACS errors | `AacsError` (key resolution, handshake, decryption) | | E7xxx | AACS errors | `AacsError` (key resolution, handshake, decryption) |
| E8xxx | KEYDB errors | `KeydbError` (download, parse, save) |
--- ---
@@ -145,26 +185,29 @@ is baked into the library.
| Platform | Transport | Status | | Platform | Transport | Status |
|----------|-----------|--------| |----------|-----------|--------|
| Linux | SG_IO ioctl on `/dev/sr*` | Implemented | | Linux | async sg write/poll/read on `/dev/sg*` | Supported |
| macOS | IOKit SCSI passthrough | Planned | | macOS | IOKit SCSITask | Supported |
| Windows | SPTI (`IOCTL_SCSI_PASS_THROUGH_DIRECT`) | Planned | | Windows | SPTI (`IOCTL_SCSI_PASS_THROUGH_DIRECT`) | Supported |
The `ScsiTransport` trait abstracts the platform. Adding a new platform requires The `ScsiTransport` trait abstracts the platform. Adding a new platform requires
implementing `execute()` for that OS and wiring it into `scsi::open()`. implementing `execute()` for that OS and wiring it into `scsi::open()`.
--- ---
## Chipset Support ## Drive Unlock
| Chipset | Drives | Status | libfreemkv carries no drive-unlock mechanism. The `Unlocker` trait + registry
|---------|--------|--------| (`src/unlock.rs`) is the seam: an external crate implements `Unlocker` and
| MediaTek MT1959 | LG, ASUS, hp | Implemented (206 profiles) | registers it once via `register_unlocker(...)`. At drive-prep the registry is
| Renesas RS8xxx/RS9xxx | Pioneer, some HL-DT-ST | Planned | walked in order and the first unlocker whose `matches()` is true is asked to
`unlock_drive()` over the raw `ScsiTransport`. If none match, the drive is left
untouched and the host-certificate AACS handshake carries the disc.
The `Platform` trait abstracts chipset-specific commands. Each chipset implements The implementor owns everything firmware-specific — drive profiles, vendor CDBs,
10 handlers (unlock, config, register, calibrate, keepalive, status, probe, variant logic. Concrete unlockers live in the separate
read_sectors, timing). All handlers are accessed via SCSI READ BUFFER with **[freemkv-unlock](https://github.com/freemkv/freemkv-unlock)** repository, never
chipset-specific mode and buffer ID bytes. in libfreemkv. See [`drive-access.md`](drive-access.md#drive-unlock-seam) for the
trait definition and routing.
--- ---
@@ -174,7 +217,5 @@ chipset-specific mode and buffer ID bytes.
cargo build --release cargo build --release
``` ```
Linux builds produce a static library and two binaries (`freemkv-info`, Produces a Rust library crate. The `libc` dependency is unix-only (gated).
`freemkv-test`). The `libc` dependency is Linux-only. On non-Linux platforms, All three platforms build and pass CI.
the library compiles but `scsi::open()` returns a platform-not-supported error
until the IOKit/SPTI backends are implemented.
+1 -1
View File
@@ -189,7 +189,7 @@ The full ripping pipeline chains three parsers:
2. **CLPI** converts those timestamps to SPN ranges, then to sector extents. 2. **CLPI** converts those timestamps to SPN ranges, then to sector extents.
3. **UDF** provides the file's starting LBA on disc for absolute sector addressing. 3. **UDF** provides the file's starting LBA on disc for absolute sector addressing.
The `Disc::scan()` method in `src/disc.rs` orchestrates this: for each play item in each playlist, it loads the corresponding CLPI, calls `get_extents()` with the play item's in/out times, and collects the resulting sector ranges into the title's extent list. The `Disc::scan()` method in `src/disc/mod.rs` orchestrates this: for each play item in each playlist, it loads the corresponding CLPI, calls `get_extents()` with the play item's in/out times, and collects the resulting sector ranges into the title's extent list.
## References ## References
+59 -38
View File
@@ -9,12 +9,18 @@ This is the starting point for understanding the library.
Insert disc Insert disc
1. Open drive (drive.rs) 1. Open drive (drive/mod.rs)
│ INQUIRY → identify drive │ INQUIRY → identify drive (DriveId)
│ Match bundled profile → chipset, unlock parameters
2. AACS handshake (aacs_handshake.rs) — optional, separate transport 2. Init drive (drive/mod.rs → unlock seam)
│ Walk the registered-unlocker registry; first match unlocks the drive
│ (firmware/vendor handshakes are the unlocker's own business)
│ No match → drive untouched; host-cert AACS handshake carries the disc
│ Speed control → probe_disc()
3. AACS handshake (aacs/handshake.rs) — optional
│ Allocate AGID │ Allocate AGID
│ Exchange certificates + nonces (ECDH) │ Exchange certificates + nonces (ECDH)
│ Derive bus key │ Derive bus key
@@ -22,11 +28,6 @@ Insert disc
│ (fails gracefully if drive doesn't support AACS for this disc) │ (fails gracefully if drive doesn't support AACS for this disc)
3. Unlock drive (drive.rs → platform/mt1959.rs)
│ Vendor-specific command activates raw read mode
│ Required — drive firmware blocks all reads without it
4. Read UDF filesystem (udf.rs) 4. Read UDF filesystem (udf.rs)
│ Sector 256: AVDP → find Volume Descriptor Sequence │ Sector 256: AVDP → find Volume Descriptor Sequence
│ VDS: Partition Descriptor (physical start) + Logical Volume (metadata start) │ VDS: Partition Descriptor (physical start) + Logical Volume (metadata start)
@@ -35,80 +36,100 @@ Insert disc
│ → docs/udf.md │ → docs/udf.md
5. Read AACS files from disc (aacs.rs) 5. Read AACS files from disc (aacs/mod.rs)
│ AACS/Unit_Key_RO.inf → SHA1 = disc hash │ AACS/Unit_Key_RO.inf → SHA1 = disc hash
│ AACS/Content000.cer → AACS version (1.0 or 2.0), bus encryption flag │ AACS/Content000.cer → AACS version (1.0 or 2.0), bus encryption flag
│ MKB via SCSI → for key derivation fallback │ MKB via SCSI → for key derivation fallback
6. Resolve AACS keys (aacs.rs → resolve_keys) 6. Resolve encryption keys (decrypt.rs → resolve_encryption)
│ BD AACS:
│ Path 1: disc hash → KEYDB.cfg → VUK (fast, 99% of discs) │ Path 1: disc hash → KEYDB.cfg → VUK (fast, 99% of discs)
│ Path 2: KEYDB media key + Volume ID → VUK │ Path 2: KEYDB media key + Volume ID → VUK
│ Path 3: MKB + processing keys → media key → VUK │ Path 3: MKB + processing keys → media key → VUK
│ Path 4: MKB + device keys → subset-difference tree → VUK │ Path 4: MKB + device keys → subset-difference tree → VUK
│ VUK → decrypt unit keys from Unit_Key_RO.inf │ VUK → decrypt unit keys from Unit_Key_RO.inf
│ DVD CSS:
│ Table-driven cipher — no KEYDB needed
│ → docs/aacs.md │ → docs/aacs.md
7. Parse playlists (mpls.rs) 7. Parse playlists (mpls.rs) — BD/UHD only
│ BDMV/PLAYLIST/*.mpls → titles with play items │ BDMV/PLAYLIST/*.mpls → titles with play items
│ Each play item: clip ID, in/out timestamps │ Each play item: clip ID, in/out timestamps
│ STN table: video, audio, subtitle streams with codec + language │ STN table: video, audio, subtitle streams with codec + language
│ → docs/mpls.md │ → docs/mpls.md
8. Parse clip info (clpi.rs) 8. Parse clip info (clpi.rs) — BD/UHD only
│ BDMV/CLIPINF/*.clpi → EP map (timestamp → sector mapping) │ BDMV/CLIPINF/*.clpi → EP map (timestamp → sector mapping)
│ Coarse + fine entries → full PTS and SPN │ Coarse + fine entries → full PTS and SPN
│ SPN → byte offset → sector extents for reading │ SPN → byte offset → sector extents for reading
│ → docs/clpi.md │ → docs/clpi.md
9. Parse BD-J labels (jar.rs) — optional 9. Parse BD-J labels (labels/) — optional
│ BDMV/JAR/*.jar → Java class constant pool strings │ BDMV/JAR/*.jar → Java class constant pool strings
│ 5 format parsers: Paramount, Criterion, Pixelogic, CTRM, Deluxe
│ Audio track labels: "English Descriptive Audio", "French 5.1", etc. │ Audio track labels: "English Descriptive Audio", "French 5.1", etc.
10. Read + decrypt content (disc.rs → ContentReader) 10. Stream content (mux/disc.rs → DiscStream)
For each aligned unit (6144 bytes = 3 sectors): Read sectors → decrypt → TS demux → PES frames
Read 3 sectors from disc Or: read sectors → decrypt → raw bytes (for ISO output)
If AACS 2.0: bus decrypt (read_data_key, per-sector AES-CBC) Drive::read() is single-shot. DiscStream::fill_extents adapts the
If encrypted: unit decrypt (per-unit key derivation + AES-CBC) batch size on failure (halve / probe-up). Bad-range retry is layer
Output decrypted content 1 above this — Disc::patch re-runs against the mapfile.
Decrypted m2ts stream → ready for muxing/backup PES frames → output stream (MKV, M2TS, network, etc.)
``` ```
## API Summary ## API Summary
```rust ```rust
// Steps 1 + 3 (open + unlock) // Open + init drive
let mut session = DriveSession::open(Path::new("/dev/sr0"))?; let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?;
drive.init()?;
drive.probe_disc()?;
// Steps 2 + 4-9 (AACS + scan) // Scan disc (UDF + playlists + AACS — all automatic)
let disc = Disc::scan(&mut session, &ScanOptions::with_keydb("keydb.cfg"))?; let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
// Step 10 (read + decrypt) // Stream pipeline — PES frames from any source to any output.
let mut reader = disc.open_title(&mut session, 0)?; // input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
while let Some(unit) = reader.read_unit()? { // direction is type-checked, so calling .write() on an input is a compile error.
output.write_all(&unit)?; let opts = InputOptions::default();
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
let title = input.info().clone();
let mut output = libfreemkv::output("mkv://Movie.mkv", &title)?;
while let Ok(Some(frame)) = input.read() {
output.write(&frame)?;
} }
output.finish()?;
``` ```
Three lines. Everything else is internal.
## Module Reference ## Module Reference
| Module | Doc | Purpose | | Module | Doc | Purpose |
|--------|-----|---------| |--------|-----|---------|
| drive.rs | [drive-access.md](drive-access.md) | Open, identify, unlock, read | | drive/ | [drive-access.md](drive-access.md) | Open, identify, init, unlock, single-shot read |
| scsi.rs | [drive-access.md](drive-access.md) | Platform SCSI transport | | scsi/ | [drive-access.md](drive-access.md) | Platform SCSI transport (Linux, macOS, Windows) |
| udf.rs | [udf.md](udf.md) | UDF 2.50 filesystem | | udf.rs | [udf.md](udf.md) | UDF 2.50 filesystem |
| mpls.rs | [mpls.md](mpls.md) | MPLS playlists + STN streams | | mpls.rs | [mpls.md](mpls.md) | MPLS playlists + STN streams |
| clpi.rs | [clpi.md](clpi.md) | CLPI clip info + EP map | | clpi.rs | [clpi.md](clpi.md) | CLPI clip info + EP map |
| aacs.rs | [aacs.md](aacs.md) | Key resolution + content decrypt | | ifo.rs | -- | DVD IFO parser |
| aacs_handshake.rs | [aacs.md](aacs.md) | SCSI bus authentication | | aacs/ | [aacs.md](aacs.md) | Key resolution + content decrypt + bus handshake |
| disc.rs | -- | High-level scan + read API | | css/ | -- | DVD CSS cipher |
| jar.rs | -- | BD-J audio track labels | | decrypt.rs | -- | Unified decrypt dispatcher (AACS/CSS/None) |
| error.rs | -- | Error codes (E1xxx-E7xxx) | | disc/ | [rip-recovery.md](rip-recovery.md) | Disc::scan + Disc::sweep + Disc::patch + mapfile |
| labels/ | -- | BD-J stream labels (5 format parsers) |
| mux/ | -- | Stream implementations (7 stream types) |
| pes.rs | -- | PES frame types + FrameSource / FrameSink traits |
| sector/ | -- | SectorSource / SectorSink + DecryptingSectorSource decorator |
| io/ | -- | Pipeline<I, R> + Sink trait + WritebackFile |
| halt.rs | -- | Halt cancellation token |
| keydb.rs | -- | KEYDB download, parse, save |
| error.rs | -- | Error codes (E1xxx-E8xxx) |
| event.rs | -- | Drive event system |
+143 -154
View File
@@ -5,51 +5,78 @@ optical drives.
--- ---
## DriveSession ## Drive
`DriveSession` is the primary API. It owns the SCSI transport, the matched `Drive` is the primary API. It owns the SCSI transport and the drive
drive profile, and the chipset-specific platform driver. identity (`DriveId`); any drive-specific unlock logic lives behind the
pluggable [unlock seam](#drive-unlock-seam), not in `Drive` itself.
### Opening a Drive ### Opening a Drive
```rust ```rust
// Full open: identify → match profile → unlock let mut drive = Drive::open(Path::new("/dev/sg4"))?;
let mut session = DriveSession::open(Path::new("/dev/sr0"))?;
// No-unlock open: identify → match profile only
let mut session = DriveSession::open_no_unlock(Path::new("/dev/sr0"))?;
// Explicit profile (skip auto-detection)
let mut session = DriveSession::open_with_profile(Path::new("/dev/sr0"), profile)?;
``` ```
**`open()`** performs the full sequence: open device, send INQUIRY, match `open()` performs: open device send INQUIRY → build `DriveId`. The drive
profile, instantiate platform driver, and unlock. Unlock failures are silently is ready for `wait_ready()` and `init()` (which routes through the unlock
ignored (unencrypted discs do not need it). After `open()`, both raw sector seam).
reads and standard READ(10) work immediately.
**`open_no_unlock()`** skips the unlock step. This is required when AACS bus ### Drive Operations
authentication must happen before unlock. The handshake uses standard SCSI
commands that work without raw mode. After authentication completes, the caller
can invoke `session.unlock()` manually.
**`open_with_profile()`** bypasses profile auto-detection. Useful for testing
or when a custom profile is loaded from an external source.
### Session Operations
| Method | Description | | Method | Description |
|--------|-------------| |--------|-------------|
| `unlock()` | Activate raw disc access mode via platform driver | | `wait_ready()` | Wait for disc insertion (30s timeout, TUR polling) |
| `is_unlocked()` | Check if raw mode is active | | `init()` | Route to the matching registered unlocker (if any), then prepare for reads |
| `calibrate()` | Build speed lookup table for the current disc | | `probe_disc()` | Probe disc surface for optimal speeds |
| `read_sectors(lba, count, buf)` | Raw sector read (requires unlock + calibrate) | | `read(lba, count, buf, recovery)` | Read sectors. Single-shot — no inline retries or reset. |
| `read_disc(lba, count, buf)` | Standard READ(10) with 5s timeout | | `reset()` | Eject-cycle escape hatch. Caller-invoked only; not on the read path. |
| `status()` | Query drive status and feature flags | | `lock_tray()` | Prevent tray ejection during rip |
| `read_config()` | Read drive configuration block (1888 bytes) | | `unlock_tray()` | Allow tray ejection (also runs on Drop) |
| `read_register(index)` | Read 16-byte hardware register | | `eject()` | Eject disc tray |
| `probe(sub_cmd, addr, len)` | Generic READ BUFFER with caller parameters | | `drive_status()` | Query physical state (disc present, tray open, etc.) |
| `scsi_execute(cdb, dir, buf, timeout)` | Send an arbitrary SCSI CDB | | `has_profile()` | Whether a registered unlocker matches this drive |
| `close()` | Consume Drive, cleanup (also runs via Drop) |
### init() Sequence
`init()` routes drive preparation through the unlock seam:
1. Walk the registered-unlocker registry; the first whose `matches()` is true
is asked to `unlock_drive()` over the raw transport.
2. Whatever that unlocker needs (firmware upload, vendor handshakes, retries)
is the unlocker's own business — libfreemkv only forwards the transport.
3. If no unlocker matches, the drive is left untouched and the library uses
the host-certificate AACS handshake.
See [Drive Unlock Seam](#drive-unlock-seam) for the trait and registry.
### read() — single-shot
`Drive::read(lba, count, buf, recovery)` is the single read method. It issues
exactly one READ(10) CDB and returns the result. The `recovery` parameter only
selects the per-CDB timeout:
| `recovery` | Timeout | Used by |
|------------|----------|------------------------------------------|
| `false` | 1.5 s | `Disc::sweep` fast skip-forward pass, `DiscStream::fill_extents` |
| `true` | 30 s | `Disc::patch` retry pass over the mapfile |
On any SCSI failure or timeout, `read` returns `Err(DiscRead)` immediately.
There are no inline retries, no SCSI reset, no Phase 1/2/3 escalation.
Recovery is layered above `Drive::read`:
- **Layer 1 — `Disc::patch`** loops over the ddrescue mapfile and re-issues
`read(.., recovery=true)` against each non-`+` range.
- **Layer 3 — `DiscStream::fill_extents`** halves the request size on
failure, retries at the same LBA, and probes back up on a clean-read
streak.
Inline recovery (5× gentle retry → close + reset + reopen → 5× more) was
removed in 0.13.6. See the stop-wedge postmortem (2026-04-25)
for rationale: the inline reset wedged drive firmware on the LG BU40N (Initio
USB-SATA bridge) without ever recovering a sector. See
[`rip-recovery.md`](rip-recovery.md) for the full three-layer model.
--- ---
@@ -58,7 +85,7 @@ or when a custom profile is loaded from an external source.
### Trait ### Trait
```rust ```rust
pub trait ScsiTransport { pub trait ScsiTransport: Send {
fn execute( fn execute(
&mut self, &mut self,
cdb: &[u8], cdb: &[u8],
@@ -72,17 +99,43 @@ pub trait ScsiTransport {
All drive communication goes through this trait. The library never opens file All drive communication goes through this trait. The library never opens file
descriptors or calls ioctls outside of a `ScsiTransport` implementation. descriptors or calls ioctls outside of a `ScsiTransport` implementation.
### Linux: SG_IO ### Platform Backends
The `SgIoTransport` implementation: | Platform | Implementation | Device |
|----------|---------------|--------|
| Linux | `SgIoTransport` — async `write`/`poll`/`read` on `/dev/sg*` | `/dev/sg*` |
| macOS | `MacScsiTransport` — IOKit SCSITask | IOKit service |
1. Opens the device path with `O_RDWR | O_NONBLOCK`. The Linux backend uses the sg driver's asynchronous interface: `write()` submits
2. Constructs an `sg_io_hdr` struct with the CDB, data buffer, and timeout. the command, `poll()` waits with an enforceable wall-clock timeout, `read()`
3. Calls `ioctl(fd, SG_IO, &hdr)`. retrieves the result. If `poll()` times out, the fd is abandoned (closed in a
4. Returns `ScsiResult` with status, bytes transferred, and sense data. background thread) and a fresh fd opened — the kernel's USB error recovery
cannot block us. Opens with `O_RDWR | O_NONBLOCK`.
On non-zero SCSI status, the transport parses sense key, ASC, and ASCQ from the The macOS backend uses a C shim (`macos_shim.c`) for IOKit exclusive access.
sense buffer and returns `Error::ScsiError`. The shim handles:
1. `shim_open_exclusive(bsd_name)` — unmounts the target device via `diskutil`,
then walks the IOKit registry to find the `IOBDServices` matching the
requested BSD name (IOBDServices → IOBDBlockStorageDriver → IOMedia → "BSD Name"),
then creates MMCDeviceInterface → SCSITaskDeviceInterface → ObtainExclusiveAccess.
2. `shim_list_drives()` — registry-based enumeration with zero SCSI, zero exclusive
access, zero unmounts. Reads IOBDServices "Device Characteristics" for
vendor/model/firmware and child IOMedia "BSD Name" for the device path.
3. `shim_execute()` / `shim_close()` — raw CDB dispatch and cleanup.
On non-zero SCSI status, the transport parses sense key from the sense buffer
and returns `Error::ScsiError`.
`SgIoTransport::reset` (Linux) does pure userspace state cleanup: an open +
close pair to make the kernel cancel any SG_IO commands queued against a
previous fd, a 2 s sleep to let the kernel finish that cancellation, then a
fresh fd to send ALLOW MEDIUM REMOVAL to clear any stale tray lock. It does
NOT issue `SG_SCSI_RESET` or escalate via STOP+START UNIT. Both were tried
in 0.13.00.13.5 against the LG BU40N (Initio USB-SATA bridge); both failed
to recover wedged drives and made the wedge worse. The macOS reset (which
had been a no-op) was removed entirely in 0.13.6, and the top-level
`scsi::reset()` / `reset_with_timeout()` / `reset_blocking()` wrappers were
removed at the same time (no callers).
### CDB Builders ### CDB Builders
@@ -117,132 +170,68 @@ date for drives where Feature 010C is unavailable.
--- ---
## Drive Profiles ## Drive Unlock Seam
Profiles are JSON objects compiled into the binary (`profiles.json`, libfreemkv ships **no firmware, no unlock CDBs, and no drive profiles.** It
206 entries). Each profile contains: knows only the *seam*, never the *mechanism*. The seam is the `Unlocker`
trait plus a small process-wide registry (`src/unlock.rs`):
| Field | Purpose |
|-------|---------|
| `vendor_id`, `product_revision`, `vendor_specific`, `firmware_date` | Matching fields |
| `chipset` | `"mediatek"` or `"renesas"` |
| `unlock_mode`, `unlock_buf_id` | READ BUFFER CDB parameters |
| `signature` | Expected 4-byte response signature |
| `unlock_cdb` | Pre-built unlock CDB (hex-encoded) |
| `register_offsets` | Offsets for hardware register reads |
| `capabilities` | Feature flags: `bd_raw_read`, `dvd_all_regions`, etc. |
Loading:
```rust ```rust
// Bundled (compiled-in) -- no file I/O pub trait Unlocker: Send + Sync {
let profiles = profile::load_bundled()?; /// Stable, language-neutral identifier (logged).
fn name(&self) -> &str;
// External file /// True if this unlocker handles the given drive.
let profiles = profile::load_all(Path::new("/path/to/profiles.json"))?; fn matches(&self, id: &DriveId) -> bool;
/// Put the drive into extended-access mode. The one required capability.
fn unlock_drive(&self, scsi: &mut dyn ScsiTransport, id: &DriveId) -> Result<()>;
/// Read the disc Volume ID via the drive's OEM path. Default: no-op.
fn read_volume_id(&self, _scsi: &mut dyn ScsiTransport, _id: &DriveId)
-> Result<Option<[u8; 16]>> { Ok(None) }
/// Raise the drive to its maximum read speed. Default: no-op.
fn set_max_read_speed(&self, _scsi: &mut dyn ScsiTransport, _id: &DriveId)
-> Result<()> { Ok(()) }
}
``` ```
--- An unlocker is supplied by an **external crate** and registered once at
process start:
## Chipsets
### MediaTek MT1959
Covers all LG, ASUS, and hp optical drives. Two sub-variants share identical
logic with different SCSI parameters:
| Variant | READ BUFFER mode | Buffer ID |
|---------|------------------|-----------|
| MT1959-A | 0x01 | 0x44 |
| MT1959-B | 0x02 | 0x77 |
The Platform trait maps to 10 command handlers:
| Handler | Function | Description |
|---------|----------|-------------|
| 0 | `unlock()` | Send READ BUFFER, verify signature + verification bytes |
| 1 | `read_config()` | Read 1888-byte configuration block + 4-byte status |
| 2-3 | `read_register()` | Read hardware registers at profile-specified offsets |
| 4 | `calibrate()` | Probe disc surface, build 64-entry speed table |
| 5 | `keepalive()` | Periodic session maintenance |
| 6 | `status()` | Query current mode and feature flags |
| 7 | `probe()` | Generic READ BUFFER with dynamic parameters |
| 8 | `read_sectors()` | Speed lookup + SET CD SPEED + READ(10) with flag 0x08 |
| 9 | `timing()` | Timing calibration |
### Renesas (Planned)
RS8xxx/RS9xxx chipsets used in Pioneer and some HL-DT-ST drives.
Currently returns `Error::UnsupportedDrive` when a Renesas profile is matched.
---
## Why Unlock Is Needed
Optical drive firmware restricts what applications can read from disc. Without
unlock:
- **READ(10) works for unencrypted filesystem data.** UDF structures, MPLS
playlists, and CLPI clip info are readable without unlock. The `read_disc()`
method uses standard READ(10) and works on any drive.
- **READ(10) fails for encrypted content sectors.** The drive firmware returns
SCSI errors (sense key 0x05, illegal request) when an application attempts to
read sectors containing encrypted m2ts content without prior AACS
authentication via the bus key.
- **The kernel sr driver blocks block-device reads.** On Linux, the kernel's
SCSI CD-ROM driver (`sr`) refuses to expose encrypted disc content through
`/dev/sr0` as a block device. Even if you open the block device directly,
reads to encrypted regions fail.
- **Raw mode bypasses firmware restrictions.** After unlock, the drive accepts
READ(10) with the raw read flag (CDB byte 1 = 0x08) for all sectors,
regardless of encryption status. This is how raw sector ripping works.
### open() vs open_no_unlock()
AACS bus authentication uses standard MMC REPORT KEY / SEND KEY commands.
These must execute before unlock because:
1. The AACS handshake establishes a bus key via ECDH.
2. The bus key encrypts the Volume ID and Read Data Key responses.
3. The Volume ID is needed to derive the Volume Unique Key (VUK).
4. The VUK is needed to decrypt unit keys from `Unit_Key_RO.inf`.
If `open()` unlocks first, some drives reject the subsequent AACS commands.
The correct sequence for encrypted discs is:
```rust ```rust
// 1. Open without unlock libfreemkv::register_unlocker(Box::new(some_unlocker::Plugin::new()));
let mut session = DriveSession::open_no_unlock(device)?;
// 2. AACS handshake (uses standard SCSI, no unlock needed)
let auth = aacs_handshake::aacs_authenticate(&mut session, &key, &cert)?;
let vid = aacs_handshake::read_volume_id(&mut session, &mut auth)?;
// 3. Now unlock for raw reads
session.unlock()?;
session.calibrate()?;
// 4. Read and decrypt content
session.read_sectors(lba, count, &mut buf)?;
``` ```
In practice, `Disc::scan()` handles this internally. The default `open()` call The implementor owns everything about *how* a particular drive family is
unlocks immediately and is correct for most use cases -- the scan re-opens a driven — drive identification against its own profile database, firmware
second session with `open_no_unlock()` for the AACS handshake when needed. upload, vendor CDBs, variant logic. libfreemkv only hands over the raw
`ScsiTransport` and the `DriveId`.
### Routing
At drive-prep the registry is walked in registration order; the first
unlocker whose `matches()` returns true is asked to `unlock_drive()` (and,
when needed, `read_volume_id()` / `set_max_read_speed()`). If no unlocker
matches, the drive is left untouched and the library falls back to the
standard host-certificate AACS handshake (the "OEM route"). The
`register_unlocker(...)` line is the entire plug: drop it (and the unlocker
crate) and libfreemkv still compiles and rips via the cert handshake.
Concrete unlockers — including the firmware-unlock profile databases,
variant logic, and vendor CDBs that used to live in-tree — are maintained
in the separate **[freemkv-unlock](https://github.com/freemkv/freemkv-unlock)**
repository, never here.
--- ---
## Speed Control ## Speed Control
After `calibrate()`, the platform driver maintains a 64-entry speed lookup table A matching unlocker may raise the drive to its maximum read speed via
built by probing the disc surface. On each `read_sectors()` call, the driver: `set_max_read_speed()` (a no-op when no unlocker matches or the unlocker
declines). The library issues SET CD SPEED (0xBB) through the generic CDB
1. Looks up the optimal speed for the target LBA in the table. builder; the concrete speed policy lives in the unlocker.
2. Issues SET CD SPEED (0xBB) if the speed differs from current.
3. Performs the READ(10).
Available speeds: Available speeds:
+201
View File
@@ -0,0 +1,201 @@
# Rip recovery — three-layer architecture
`libfreemkv` supports a multi-stage rip model for damaged or protection-bearing
discs: a fast forward sweep that tolerates read failures, in-loop request-size
adaptation that survives transient drive trouble without bailing, and targeted
retry passes against a persistent bad-range map. The stream pipeline
(`DiscStream` + `input`/`output`) operates against the resulting ISO image, so
the mux stage never touches the drive.
Recovery is layered cleanly. Each layer has one responsibility and does not
reach into the others.
| Layer | Where it lives | What it does |
|-------|---------------|--------------|
| 1 — Bad-range retry | `Disc::patch` (one pass over the mapfile per call) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
| 2 — Single-shot primitive | `Drive::read` in `src/drive/mod.rs` | One CDB, one timeout, one result. No inline retries, no SCSI reset. |
| 3 — In-loop request adaptation | `DiscStream::fill_extents` adaptive batch sizer | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
The library exposes flat verbs; the caller drives the multipass loop. Autorip
runs `Disc::sweep` once, then loops `Disc::patch` until either the mapfile is
clean or the configured retry budget is exhausted, then hands the ISO off to
the mux pipeline. The `freemkv` CLI does the same shape with a
terminal-output progress sink. Layer 3 runs inside any consumer of
`DiscStream` (direct PES pipeline, ISO playback, etc.) without caller
involvement.
Three primitives compose the disc-side flow:
| Primitive | What it does |
|---------------------------|-----------------------------------------------------------------------|
| `Disc::sweep` | disc → ISO, one forward pass. Writes a sidecar `.mapfile`. Opt-in skip-on-error. |
| `Disc::patch` | Re-reads bad ranges from the drive. One pass per call; caller invokes N times. |
| `DiscStream` (ISO source) | Reads sectors from the ISO, feeds decrypt → demux → codec → mux. |
## Data model
### Mapfile
Format: [ddrescue](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)-compatible
plain text, greppable, tool-interoperable. Flushed to disk on every `record()`
so a crashed rip loses at most one block.
```
# Rescue Logfile. Created by libfreemkv v0.13.6
# Current pos / status / pass / pass_time
0x000000000 ? 1 0
# pos size status
0x000000000 0x12a35d000 +
0x12a35d000 0x000003000 -
0x12a360000 0x009c4a000 +
0x12d00a000 0x000064000 *
```
Status characters match ddrescue:
| Char | Meaning |
|------|----------------------------------------------------|
| `?` | Not yet attempted |
| `*` | Fast-pass failed; needs edge-trim |
| `/` | Trimmed; interior needs sector scrape |
| `-` | Unreadable this session |
| `+` | Finished (good) |
Position and size are hex byte offsets into the ISO.
### `SweepOptions` and `PatchOptions`
The library no longer dispatches between sweep and patch internally — the
caller picks the verb explicitly per pass. The two option structs are flat
and have no overlap:
```rust
SweepOptions {
decrypt: true,
resume: false,
batch_sectors: None,
skip_on_error: true, // damage-jump + zero-fill on read failure
progress: Some(&reporter),
halt: Some(flag),
}
PatchOptions {
decrypt: true,
block_sectors: None,
full_recovery: true,
reverse: true, // walk bad ranges high → low LBA
wedged_threshold: 50,
progress: Some(&reporter),
halt: Some(flag),
}
```
Caller-orchestrated dispatch (the policy `Disc::copy` used to embed):
- No mapfile → `sweep` (fresh Pass 1).
- Mapfile with `?` ranges → `sweep` with `resume: true`.
- Mapfile covers full disc, only `*` / `/` / `-` ranges → `patch`.
- Mapfile clean → done; no further pass needed.
Each consumer (autorip, `freemkv` CLI) implements the loop in roughly five
lines of `Mapfile::stats()` checks.
## Algorithm
### Pass 1 — fast sweep (`Disc::sweep`)
1. Read one ECC block (32 sectors for UHD, 16 for BD/DVD) at the current LBA.
2. On success: write data to ISO, mark `+`, advance.
3. On failure (with `multipass`): zero-fill, mark `*`, advance.
4. Track a sliding window of the last 16 ECC block results. When ≥12% are failures
**damage-jump**: skip ahead by `1024×batch×multiplier` sectors (64 MB base for
UHD). Double the multiplier on each jump (64→128→256→512 MB...). Zero-fill the gap as `*`.
5. On 16 consecutive good reads: reset jump multiplier to 1, restore max read speed.
6. Speed control: damage zone entry → minimum speed, exit → maximum speed.
7. Only transport failures (USB bridge crash) abort the pass.
Pass 1 completes when every byte has been visited (either `+` or `*`).
### Pass 2+ — patch (`Disc::patch`)
`Disc::patch` reads the mapfile and iterates every non-`+` range. Default: **reverse** mode
(walks ranges from highest LBA to lowest, within each range from end to start).
1. Issue a single-sector read with 60 s timeout (`recovery=true`). Drive firmware
does its own ECC recovery inside that window.
2. On success: write the good bytes into the ISO, mark `+`.
3. On failure with non-marginal SCSI sense: bail immediately (drive won't produce data).
4. On failure with marginal sense: mark `-`, continue.
5. Update the mapfile after every block — crash-safe resume.
6. Wedged-drive exit: 50 consecutive failures with zero recovery → bail this pass.
### In-stream — adaptive batch halving (`DiscStream::fill_extents`)
When a consumer reads a `DiscStream` directly (no ISO intermediate),
`fill_extents` runs an adaptive sizer in front of `Drive::read`:
1. Try the current preferred batch size (e.g. 32 sectors, one BD ECC block).
2. On failure: halve the batch and retry at the same LBA. Emit
`EventKind::BatchSizeChanged { reason: Shrunk }`.
3. On a clean-read streak: probe back up toward the preferred size. Emit
`EventKind::BatchSizeChanged { reason: Probed }`.
4. If a single-sector read fails: skip (zero-fill, emit
`EventKind::SectorSkipped`) when `skip_errors` is set, otherwise return
`Err(DiscRead)`.
This is layer 3. It exists so a transient single-sector glitch in a 32-sector
batch can be isolated and read individually without the caller needing to
implement retry logic.
## Design choices
**`Drive::read` is single-shot.** No inline retry phases, no SCSI reset,
no eject cycle. The `recovery` flag controls only the per-CDB timeout
(1.5 s vs. 30 s); on any failure it returns `Err(DiscRead)` immediately.
Inline recovery (5× gentle retry → close + SCSI reset + reopen → 5× more)
was removed in 0.13.6. See the stop-wedge postmortem (2026-04-25) for rationale:
the inline reset on the LG BU40N (Initio USB-SATA bridge)
wedged drive firmware below the bridge without ever recovering a sector,
and the gentle-retry phase produced long stretches of 0 KB/s with no
recoveries to show for it. Recovery responsibility is now layered: layer 1
handles ranges, layer 3 handles request size, neither touches the
wedge-prone reset path.
**No `MODE SELECT` to disable drive retries.** Neither ddrescue
nor any consumer ripper does this. Drive firmware has access to raw analog signal, laser
power control, and drive-specific ECC tuning that userspace can't replicate —
disabling it throws away recovery headroom on marginal sectors. We fail fast
via short SG_IO timeouts in pass 1 and let the firmware work the long timeout
in pass 2 / patch.
**No SCSI reset from any retry path.** `SgIoTransport::reset` (Linux) is
trimmed to a kernel SG_IO state flush plus ALLOW MEDIUM REMOVAL — the
`SG_SCSI_RESET` ioctl and STOP/START UNIT escalation were removed in 0.13.6.
The macOS reset (which had been a no-op) was removed entirely. The top-level
`scsi::reset()` / `reset_with_timeout()` / `reset_blocking()` wrappers were
also removed (no callers). The remaining `Drive::reset()` is only invoked
explicitly by callers that need an eject-cycle escape hatch — it is never
reached from a read path.
**ISO intermediate, even for single-pass.** Pass 1 always writes an ISO. The
mux stage reads the ISO via `FileSectorSource`. For single-pass (no retries),
this adds ~2-3 min (local disk mux) but gains resumability across crashes,
re-muxability without re-ripping, and a persistent forensic artifact. Callers
who need pure speed can bypass and use `DiscStream::new(Box::new(drive), …)`
directly — the lib doesn't forbid it, and layer 3 (adaptive batch halving)
still applies there.
**Mapfile in ddrescue format.** Plain text so users can `less` it, `diff` it,
or feed it to ddrescue's own tooling. Crash-safe (flush-per-record). Entries
coalesce on adjacent same-status ranges so files stay small.
**Patches target `-`, `*`, `/`, and `?` alike.** The status state machine is
ddrescue's but `patch` collapses the distinction — it just tries every
non-finished range with the long timeout. Future work can specialize (trim vs.
scrape vs. retry with direction reversal) if there's measured benefit.
## References
- [ddrescue manual, Algorithm chapter](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)
- [ddrescue optical media notes](https://www.electric-spoon.com/doc/gddrescue/html/Optical-media.html)
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/mod.rs`](../src/disc/mod.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
+8
View File
@@ -120,6 +120,14 @@ Each directory read involves two sector reads: one for the ICB, then one or more
`read_file()` reads a file by navigating the directory tree, reading the file's ICB to get its data extent, then reading the data sector by sector from the **physical partition** (partition_start + LBA, not metadata_start). `read_file()` reads a file by navigating the directory tree, reading the file's ICB to get its data extent, then reading the data sector by sector from the **physical partition** (partition_start + LBA, not metadata_start).
## Buffered Sector Reads
USB optical drives have ~500ms round-trip latency per SCSI command. Since `read_filesystem()` and `read_file()` issue one SCSI READ per sector, a full disc scan can require hundreds of commands -- taking 10+ minutes on USB.
`Disc::scan()` wraps the drive in a `BufferedSectorReader` before reading. On a single-sector read, the buffer prefetches a batch of sectors (sized from the kernel's `max_hw_sectors_kb` for the device) and caches them. Subsequent reads to nearby LBAs return from cache with zero SCSI overhead. After parsing the UDF directory structure, the entire metadata partition is pre-read into the cache, so all ICB lookups during title scanning and encryption resolution are instant.
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorSource` implementation.
### UDF Filename Encoding ### UDF Filename Encoding
UDF filenames use a compression ID as the first byte: UDF filenames use a compression ID as the first byte:
+83
View File
@@ -0,0 +1,83 @@
// Minimal ISO dumper — find exact stall point
use libfreemkv::Drive;
use std::io::{BufWriter, Write};
use std::path::Path;
use std::time::Instant;
fn main() {
let args: Vec<String> = std::env::args().collect();
if args.len() < 3 {
eprintln!("Usage: iso_dump <device> <output>");
std::process::exit(1);
}
let mut drive = Drive::open(Path::new(&args[1])).unwrap();
drive.wait_ready().unwrap();
let _ = drive.init();
let _ = drive.probe_disc();
// AACS handshake — required to read past the protected area
eprint!("Scanning disc... ");
let _ = libfreemkv::Disc::scan(&mut drive, &libfreemkv::ScanOptions::default());
eprintln!("OK");
let cap = drive.read_capacity().unwrap();
let batch = libfreemkv::disc::detect_max_batch_sectors(drive.device_path());
eprintln!("Device: {} | {} sectors | batch {}", args[1], cap, batch);
let file = std::fs::File::create(&args[2]).unwrap();
let mut w = BufWriter::with_capacity(4 * 1024 * 1024, file);
let mut buf = vec![0u8; batch as usize * 2048];
let mut lba: u32 = 0;
let start = Instant::now();
let mut last = Instant::now();
let mut bytes: u64 = 0;
let mut last_bytes: u64 = 0;
while lba < cap {
let count = ((cap - lba) as u16).min(batch);
let n = count as usize * 2048;
// Tiny yield between reads — test if pacing prevents firmware throttle
std::thread::yield_now();
let t0 = Instant::now();
let ok = drive.read(lba, count, &mut buf[..n], true).is_ok();
let read_ms = t0.elapsed().as_millis();
// Flag slow reads
if read_ms > 2000 {
eprintln!("\n SLOW READ: LBA {} took {}ms (ok={})", lba, read_ms, ok);
}
if !ok {
buf[..n].fill(0);
}
w.write_all(&buf[..n]).unwrap();
lba += count as u32;
bytes += n as u64;
if last.elapsed().as_millis() >= 1000 {
let delta = bytes - last_bytes;
let speed = delta as f64 / last.elapsed().as_secs_f64() / 1_048_576.0;
let avg = bytes as f64 / start.elapsed().as_secs_f64() / 1_048_576.0;
let pct = bytes as f64 / (cap as f64 * 2048.0) * 100.0;
eprint!(
"\r {:.1}% LBA {} | {:.0} MB/s (avg {:.0}) | {:.1} GB ",
pct,
lba,
speed,
avg,
bytes as f64 / 1e9
);
last_bytes = bytes;
last = Instant::now();
}
}
w.flush().unwrap();
eprintln!(
"\nDone: {:.1} GB in {:.0}s",
bytes as f64 / 1e9,
start.elapsed().as_secs_f64()
);
}
-3093
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -0,0 +1 @@
max_width = 100
+500
View File
@@ -0,0 +1,500 @@
//! AACS derivation "boil-down" — one public home for the key chain.
//!
//! Thin newtypes at the API boundary and three wrapper functions over the
//! existing crypto. Nothing here re-implements a primitive: every function
//! delegates to the already-audited code in [`super::keys`] and
//! [`super::variants`], so the boil-down cannot drift from production math.
//!
//! The newtypes wrap bare `[u8; 16]` ONLY at this boundary — the crypto
//! internals continue to operate on raw arrays. They exist so a caller threads
//! the chain `DK → MK → VUK → UK` without confusing one 16-byte secret for
//! another, not to refactor the resolver.
//!
//! Chain (matches `aacs::keys::resolve_keys_classical` path 1 and
//! `aacs::keys::resolve_keys_v21` path 1 byte-for-byte):
//!
//! ```text
//! mk_from_dk(device_keys, mkb, vid) → MediaKey (Km)
//! mk_from_pk(processing_keys, mkb) → MediaKey (Km)
//! vuk_from_mk(MediaKey, Vid) → Vuk (= AES-G(Km, VID))
//! uk_from_vuk(Vuk, enc_title_keys) → [UnitKey] (decrypt_unit_key each)
//! ```
//!
//! `mk_from_dk` and `mk_from_pk` are two entry points to the SAME Media Key,
//! both via the MKB's Subset-Difference cvalue tables: the device-key path
//! recovers its Processing Key at the matching SD node and walks on to the MK;
//! the processing-key path starts from a precomputed PK. Neither needs a VID
//! (the VID enters at `vuk_from_mk`).
use super::keys::{
decrypt_unit_key, derive_media_key_and_pk_from_dk, derive_media_key_from_pk, derive_vuk,
};
use super::types::DeviceKey;
/// Volume ID (16 bytes) — read from the disc via the SCSI handshake / OEM path.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Vid(pub [u8; 16]);
/// Media Key (Km, 16 bytes) — the MKB-scoped key derived from device keys.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct MediaKey(pub [u8; 16]);
/// Volume Unique Key (VUK / Kvu, 16 bytes) — derived from `MediaKey` + `Vid`,
/// decrypts the per-disc encrypted title keys in `Unit_Key_RO.inf`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Vuk(pub [u8; 16]);
/// Processing Key (Kp, 16 bytes) — an MKB Subset-Difference key that yields the
/// Media Key. A leaked/precomputed PK in the keydb, or the intermediate PK a
/// device-key walk derives at its matching SD node.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ProcessingKey(pub [u8; 16]);
/// One decrypted per-CPS-unit AACS title key.
///
/// `idx` is the POSITIONAL index of the encrypted title key within the slice
/// handed to [`uk_from_vuk`] (i.e. its order in `Unit_Key_RO.inf`'s key-storage
/// area). The CPS-unit *number* association (the `u32` in
/// `ResolvedKeys::unit_keys`) is a higher-level concern owned by
/// [`super::keys::parse_unit_key_ro`], which pairs each positional key with its
/// declared CPS unit; this primitive only does the AES, so it surfaces position.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct UnitKey {
pub idx: u32,
pub key: [u8; 16],
}
/// Derive the Volume Unique Key from a Media Key and Volume ID.
///
/// Wraps [`derive_vuk`] verbatim: `VUK = AES-128-ECB-DECRYPT(MK, VID) XOR VID`.
/// This is byte-identical to the inline `derive_vuk(&mk, ctx.volume_id)` call in
/// every classical resolver path AND to the `Kvu = AES-G(Km, VID)` step inside
/// [`derive_media_key_variant`] (AES-G and `derive_vuk` are the same math), so
/// `vuk_from_mk(mk_from_dk(..)?, vid)` reproduces the V21 variant VUK exactly.
pub fn vuk_from_mk(mk: MediaKey, vid: Vid) -> Vuk {
Vuk(derive_vuk(&mk.0, &vid.0))
}
/// Decrypt the disc's encrypted title keys with a VUK.
///
/// Wraps [`decrypt_unit_key`] (AES-128-ECB-DECRYPT) per entry, mirroring the
/// `derive_uks` closure in `resolve_keys_classical` / `resolve_keys_v21`. The
/// returned `UnitKey::idx` is the slice position; pair with CPS-unit numbers via
/// [`super::keys::parse_unit_key_ro`] when the numbering matters.
pub fn uk_from_vuk(vuk: Vuk, enc_title_keys: &[[u8; 16]]) -> Vec<UnitKey> {
enc_title_keys
.iter()
.enumerate()
.map(|(i, enc)| UnitKey {
idx: i as u32,
key: decrypt_unit_key(&vuk.0, enc),
})
.collect()
}
/// Derive the Media Key (Km) from positioned device keys via the MKB's
/// Subset-Difference tables.
///
/// Wraps [`derive_media_key_and_pk_from_dk`] — the real SD walk the resolver
/// runs: each positioned device key is placed against the MKB's subset-diff /
/// cvalue records, recovering its Processing Key at the matching node and
/// continuing to the Media Key. Reachable for real discs whenever a device key
/// applies to the MKB. No VID is involved here — it enters at [`vuk_from_mk`].
///
/// Returns [`Error::AacsMkUnavailable`] (E7018) when no supplied device key
/// resolves the MKB — the same terminal error as [`mk_from_pk`]; no numeric
/// distinction is load-bearing at this boundary.
///
/// [`Error::AacsMkUnavailable`]: crate::error::Error::AacsMkUnavailable
pub fn mk_from_dk(device_keys: &[DeviceKey], mkb: &[u8]) -> Result<MediaKey, crate::error::Error> {
// Positioned device keys drive the real Subset-Difference MKB walk
// ([`derive_media_key_and_pk_from_dk`], the same walk the resolver runs). The
// old Media-Key-Variant path needed integrator Key Correction Data absent
// in-tree, so it Err'd for EVERY real disc (dead for both consumers —
// freemkv-keysources' DK fallback and the kdb harvester). No VID is needed
// for the Media Key; it enters only at [`vuk_from_mk`].
match derive_media_key_and_pk_from_dk(mkb, device_keys) {
Some((km, _pk)) => Ok(MediaKey(km)),
None => Err(crate::error::Error::AacsMkUnavailable),
}
}
/// Derive the Media Key (Km) from one or more Processing Keys and the disc MKB.
///
/// Wraps [`derive_media_key_from_pk`] — the Subset-Difference PK→MK walk: each
/// processing key is validated (and tree-walked) against the MKB's cvalue tables
/// (records `0x04`/`0x05`) until one yields the Media Key whose verify record
/// (`0x81`/`0x86`) matches. Unlike [`mk_from_dk`] this path is reachable for
/// real discs — a leaked/precomputed AACS Processing Key in the keydb resolves
/// the Media Key directly. No VID is involved at this step; the VID enters at
/// [`vuk_from_mk`].
///
/// Returns [`Error::AacsMkUnavailable`] (E7018) when no processing key resolves
/// the MKB — the same terminal error as [`mk_from_dk`]; no numeric distinction
/// is load-bearing at this boundary.
///
/// [`Error::AacsMkUnavailable`]: crate::error::Error::AacsMkUnavailable
pub fn mk_from_pk(
processing_keys: &[[u8; 16]],
mkb: &[u8],
) -> Result<MediaKey, crate::error::Error> {
match derive_media_key_from_pk(mkb, processing_keys) {
Some(km) => Ok(MediaKey(km)),
None => Err(crate::error::Error::AacsMkUnavailable),
}
}
/// A candidate key at any rung of the AACS ladder, handed to [`resolve_candidate`].
///
/// Each variant carries the module's existing newtype for that rung (a `Dk` is a
/// POSITIONED [`DeviceKey`] — recover an unpositioned one with
/// [`super::keys::recover_dk_position`] first).
#[derive(Debug, Clone)]
pub enum KeyCandidate {
Uk(UnitKey),
Vuk(Vuk),
Mk(MediaKey),
Pk(ProcessingKey),
Dk(DeviceKey),
}
/// The AACS key chain derived from a candidate, from [`resolve_candidate`].
///
/// PURE DERIVATION — no unit sampling, no validation. `unit_keys` holds every
/// CPS-unit key the disc's `Unit_Key_RO.inf` yields from the VUK (positional
/// order); the caller runs [`super::decrypt::unit_key_validates`] to find which
/// one actually opens the disc. Rungs above the candidate are `None` (a `Vuk`
/// candidate has no `mk`/`pk`/`dk`; a `Uk` candidate has only `unit_keys`).
#[derive(Debug, Clone)]
pub struct ResolvedChain {
/// Every unit key derived from the VUK, as `(cps_unit_number, key)` — the
/// CPS-unit numbers come from `Unit_Key_RO.inf` (via `parse_unit_key_ro`), so
/// a consumer maps `UK → CPS unit` directly. Same shape as
/// [`super::keys::ResolvedKeys::unit_keys`]. A `Uk` candidate yields exactly
/// itself, keyed by its own `idx`.
pub unit_keys: Vec<(u32, [u8; 16])>,
pub vuk: Option<Vuk>,
pub mk: Option<MediaKey>,
pub pk: Option<ProcessingKey>,
/// The positioned device key (for a `Dk` candidate).
pub dk: Option<DeviceKey>,
}
/// Derive the full AACS key chain from a candidate key of ANY ladder rung.
///
/// Runs the deterministic derivation DOWNWARD to the disc's terminal unit keys:
/// `DK → MK → VUK → UKs`, `PK → MK → VUK → UKs`, `MK → VUK → UKs`,
/// `VUK → UKs`, or `UK → itself`. Composes the module's own boil steps
/// ([`mk_from_pk`], [`vuk_from_mk`], [`uk_from_vuk`]) and parses
/// `Unit_Key_RO.inf` at the version the disc's MKB declares (48-byte stride for
/// AACS-1.0, 64 for AACS-2.x), so a multi-CPS disc yields all its unit keys from
/// the one candidate.
///
/// PURE DERIVATION: no sampling, no validation, no position recovery. Every step
/// is deterministic AES, so the returned keys are only as sound as the input
/// candidate — validate `unit_keys` against a real encrypted unit with
/// [`super::decrypt::unit_key_validates`] to prove the candidate opens the disc.
///
/// Returns `None` only when derivation itself cannot proceed: a PK its MKB
/// rejects, a `Dk` the MKB can't process, a missing VID on a path that needs
/// one, or an unparseable/empty `Unit_Key_RO.inf`.
pub fn resolve_candidate(
candidate: &KeyCandidate,
mkb: &[u8],
unit_key_ro: &[u8],
vid: Option<Vid>,
) -> Option<ResolvedChain> {
use super::keys::{AacsVersion, derive_media_key_and_pk_from_dk, mkb_type, parse_unit_key_ro};
// Boil a VUK → all unit keys, each paired with its declared CPS-unit number.
// `.inf` parsing lives here: derive the stride version from the disc's own
// MKB, then defer the VUK→unit-keys step to the shared `derive_unit_keys`
// (the one place both resolvers and this path decrypt the title keys).
let boil = |vuk: Vuk| -> Option<Vec<(u32, [u8; 16])>> {
let version = mkb_type(mkb)
.map(|t| t.generation())
.unwrap_or(AacsVersion::V10);
let ukf = parse_unit_key_ro(unit_key_ro, version)?;
if ukf.encrypted_keys.is_empty() {
return None;
}
Some(super::keys::derive_unit_keys(&ukf, &vuk.0))
};
match candidate {
KeyCandidate::Uk(uk) => Some(ResolvedChain {
unit_keys: vec![(uk.idx, uk.key)],
vuk: None,
mk: None,
pk: None,
dk: None,
}),
KeyCandidate::Vuk(v) => Some(ResolvedChain {
unit_keys: boil(*v)?,
vuk: Some(*v),
mk: None,
pk: None,
dk: None,
}),
KeyCandidate::Mk(mk) => {
let vuk = vuk_from_mk(*mk, vid?);
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(*mk),
pk: None,
dk: None,
})
}
KeyCandidate::Pk(pk) => {
let mk = mk_from_pk(std::slice::from_ref(&pk.0), mkb).ok()?;
let vuk = vuk_from_mk(mk, vid?);
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(mk),
pk: Some(*pk),
dk: None,
})
}
KeyCandidate::Dk(dk) => {
let (km, pk) = derive_media_key_and_pk_from_dk(mkb, std::slice::from_ref(dk))?;
let mk = MediaKey(km);
let vuk = vuk_from_mk(mk, vid?);
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(mk),
pk: Some(ProcessingKey(pk)),
dk: Some(dk.clone()),
})
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::aacs::decrypt::aes_ecb_encrypt;
use crate::aacs::keys::{decrypt_unit_key, derive_vuk};
/// `vuk_from_mk` must equal the inline `derive_vuk` path bit-for-bit, for
/// several known (MK, VID) vectors.
#[test]
fn vuk_from_mk_matches_inline_derive_vuk() {
let cases: [([u8; 16], [u8; 16]); 3] = [
([0x5A; 16], [0xA5; 16]),
([0x11; 16], [0x22; 16]),
(
[
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C,
0x0D, 0x0E, 0x0F,
],
[
0xF0, 0xE1, 0xD2, 0xC3, 0xB4, 0xA5, 0x96, 0x87, 0x78, 0x69, 0x5A, 0x4B, 0x3C,
0x2D, 0x1E, 0x0F,
],
),
];
for (mk, vid) in cases {
let inline = derive_vuk(&mk, &vid);
let boiled = vuk_from_mk(MediaKey(mk), Vid(vid));
assert_eq!(boiled.0, inline, "vuk_from_mk must equal derive_vuk");
}
}
/// `uk_from_vuk` must equal the inline `decrypt_unit_key` path bit-for-bit
/// and carry positional indices 0..n. Built by encrypting known plaintext
/// title keys under the VUK (the same primitive the resolver inverts).
#[test]
fn uk_from_vuk_matches_inline_decrypt_unit_key() {
let vuk = [0x5Au8; 16];
let plain_keys = [[0x11u8; 16], [0x22u8; 16], [0xCDu8; 16]];
let enc: Vec<[u8; 16]> = plain_keys
.iter()
.map(|k| aes_ecb_encrypt(&vuk, k))
.collect();
let boiled = uk_from_vuk(Vuk(vuk), &enc);
assert_eq!(boiled.len(), enc.len());
for (i, uk) in boiled.iter().enumerate() {
assert_eq!(uk.idx, i as u32, "idx must be the positional index");
// Matches the inline derive_uks closure: decrypt_unit_key(vuk, enc).
assert_eq!(uk.key, decrypt_unit_key(&vuk, &enc[i]));
// And recovers the original plaintext title key.
assert_eq!(
uk.key, plain_keys[i],
"VUK roundtrip recovers the title key"
);
}
}
/// `uk_from_vuk` on an empty slice yields no keys (no panic, no phantom idx).
#[test]
fn uk_from_vuk_empty_is_empty() {
assert!(uk_from_vuk(Vuk([0u8; 16]), &[]).is_empty());
}
/// `mk_from_dk` returns `Err(AacsMkUnavailable)` when the MKB has no
/// processable Subset-Difference tables (empty MKB, or one with no
/// mk_dv/cvalues/subdiff records) — never a wrong key, never a panic.
#[test]
fn mk_from_dk_errors_on_unprocessable_mkb() {
let dk = DeviceKey {
key: [0x11; 16],
node: 1,
uv: 1,
u_mask_shift: 0,
};
// Empty MKB → no SD records to walk → Err.
let e = mk_from_dk(std::slice::from_ref(&dk), &[]);
assert!(matches!(e, Err(crate::error::Error::AacsMkUnavailable)));
// An MKB with no complete Subset-Difference tables (mk_dv / cvalues /
// subdiff) cannot yield a Media Key, so the real walk also errors —
// never silently yields a key.
let mut mkb: Vec<u8> = Vec::new();
mkb.extend_from_slice(&[0x82, 0x00, 0x00, 0x14]); // stray data record only
mkb.extend_from_slice(&[0xAB; 16]);
let e2 = mk_from_dk(&[dk], &mkb);
assert!(matches!(e2, Err(crate::error::Error::AacsMkUnavailable)));
}
/// Build a 4-byte MKB record header (type + 3-byte big-endian total length,
/// header included) and append `body`. Mirrors the MKB record framing the
/// parser expects; no crypto.
fn mkb_record(rec_type: u8, body: &[u8]) -> Vec<u8> {
let total = 4 + body.len();
let mut rec = Vec::with_capacity(total);
rec.push(rec_type);
rec.push(((total >> 16) & 0xFF) as u8);
rec.push(((total >> 8) & 0xFF) as u8);
rec.push((total & 0xFF) as u8);
rec.extend_from_slice(body);
rec
}
/// `mk_from_pk` resolves a planted Processing Key against a synthetic MKB and
/// drives the FULL boil chain PK → MK → VUK → UK. The MKB is built with the
/// same (pk, cv, mk_dv, uv) construction the production SD walk validates, so
/// this proves a PK entry yields real Unit Keys — not just an `Ok`.
#[test]
fn mk_from_pk_drives_full_chain_to_uks() {
let pk: [u8; 16] = [
0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE,
0xFF, 0x00,
];
let mk: [u8; 16] = [
0xA0, 0xA1, 0xA2, 0xA3, 0xA4, 0xA5, 0xA6, 0xA7, 0xA8, 0xA9, 0xAA, 0xAB, 0xAC, 0xAD,
0xAE, 0xAF,
];
let uv: [u8; 4] = [0x00, 0x00, 0x04, 0x00];
// cv = AES-E(pk, mk_raw), where mk_raw is mk with the last-4-bytes-uv XOR
// pre-undone, so the validate step XORs uv back in and recovers mk.
let mut mk_raw = mk;
for a in 0..4 {
mk_raw[12 + a] ^= uv[a];
}
let cv = aes_ecb_encrypt(&pk, &mk_raw);
// mk_dv = AES-E(mk, magic||pad): AES-D(mk, mk_dv) starts with the AACS
// verify sentinel.
let mut vd = [0x11u8; 16];
vd[..8].copy_from_slice(&[0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF]);
let mk_dv = aes_ecb_encrypt(&mk, &vd);
// Synthetic MKB: type/version (0x10), verify record (0x86 = mk_dv),
// one-entry SD index (0x04 = [u_mask_shift=0][uv]), one-entry cvalue
// table (0x05 = cv).
let mut sd = vec![0u8];
sd.extend_from_slice(&uv);
let mut mkb = Vec::new();
mkb.extend_from_slice(&mkb_record(0x10, &[0, 0, 0, 0x20, 0, 0, 0, 0x52]));
mkb.extend_from_slice(&mkb_record(0x86, &mk_dv));
mkb.extend_from_slice(&mkb_record(0x04, &sd));
mkb.extend_from_slice(&mkb_record(0x05, &cv));
// PK → MK.
let got_mk = mk_from_pk(std::slice::from_ref(&pk), &mkb).expect("planted PK resolves MK");
assert_eq!(got_mk, MediaKey(mk), "mk_from_pk recovers the planted MK");
// MK → VUK → UK over an encrypted title key.
let vid = Vid([0x42u8; 16]);
let plain_uk = [0x7Eu8; 16];
let vuk = vuk_from_mk(got_mk, vid);
let enc = aes_ecb_encrypt(&vuk.0, &plain_uk);
let uks = uk_from_vuk(vuk, std::slice::from_ref(&enc));
assert_eq!(uks.len(), 1);
assert_eq!(uks[0].key, plain_uk, "PK chain recovers the title key");
// A corrupt PK resolves nothing.
let mut bad = pk;
bad[0] ^= 0xFF;
assert!(matches!(
mk_from_pk(std::slice::from_ref(&bad), &mkb),
Err(crate::error::Error::AacsMkUnavailable)
));
}
/// Minimal AACS-1.0 (48-byte stride) `Unit_Key_RO.inf` with `n` encrypted
/// unit keys — `parse_unit_key_ro` numbers CPS units 1..=n.
fn synth_inf(encs: &[[u8; 16]]) -> Vec<u8> {
let uk_pos = 32usize;
let stride = 48usize;
let n = encs.len();
let total = uk_pos + 48 + n.saturating_sub(1) * stride + 16;
let mut inf = vec![0u8; total.max(20)];
inf[..4].copy_from_slice(&(uk_pos as u32).to_be_bytes());
inf[uk_pos..uk_pos + 2].copy_from_slice(&(n as u16).to_be_bytes());
for (i, k) in encs.iter().enumerate() {
let o = uk_pos + 48 + i * stride;
inf[o..o + 16].copy_from_slice(k);
}
inf
}
/// A VUK candidate boils to ALL the disc's unit keys, each paired with its
/// declared CPS-unit number, and each key equals the VUK-decrypt of its slot.
#[test]
fn resolve_candidate_vuk_returns_all_cps_units() {
let vuk = Vuk([0x33u8; 16]);
let encs = [[0x11u8; 16], [0x22u8; 16], [0x44u8; 16]];
let inf = synth_inf(&encs);
let r = resolve_candidate(&KeyCandidate::Vuk(vuk), &[], &inf, None).expect("vuk derives");
let cps: Vec<u32> = r.unit_keys.iter().map(|(c, _)| *c).collect();
assert_eq!(
cps,
vec![1, 2, 3],
"every CPS unit surfaced, numbered from the inf"
);
for ((_, key), enc) in r.unit_keys.iter().zip(encs.iter()) {
assert_eq!(
*key,
decrypt_unit_key(&vuk.0, enc),
"key = VUK-decrypt of its slot"
);
}
assert_eq!(r.vuk, Some(vuk));
assert!(r.mk.is_none() && r.pk.is_none() && r.dk.is_none());
}
/// A bare UK candidate is terminal — it returns itself keyed by its own idx.
#[test]
fn resolve_candidate_uk_is_itself() {
let uk = UnitKey {
idx: 2,
key: [0x9u8; 16],
};
let r = resolve_candidate(&KeyCandidate::Uk(uk), &[], &[], None).expect("uk is terminal");
assert_eq!(r.unit_keys, vec![(2, uk.key)]);
assert!(r.vuk.is_none() && r.mk.is_none());
}
/// MK/PK/DK paths derive the VUK from a VID; without one, derivation stops.
#[test]
fn resolve_candidate_mk_requires_vid() {
let r = resolve_candidate(&KeyCandidate::Mk(MediaKey([1u8; 16])), &[], &[], None);
assert!(r.is_none(), "MK path returns None without a VID");
}
}
+1501
View File
File diff suppressed because it is too large Load Diff
-789
View File
@@ -1,789 +0,0 @@
//! AACS bus authentication handshake — ECDH key agreement + bus key derivation.
//!
//! Implements the AACS SCSI authentication protocol to obtain:
//! - Volume ID (VID) — needed for VUK derivation
//! - Read Data Key — needed for AACS 2.0 (UHD) bus decryption
//!
//! Flow:
//! 1. Invalidate AGIDs → allocate fresh AGID
//! 2. Send host certificate + nonce
//! 3. Receive drive certificate + nonce
//! 4. Receive drive key point + signature, verify
//! 5. Sign host key point, send
//! 6. ECDH: host_priv × drive_key_point → bus key (low 128 bits of x)
//! 7. Read VID or Read Data Keys (encrypted with bus key)
//!
//! Supports:
//! - AACS 1.0: custom 160-bit curve, SHA-1, 20-byte keys
//! - AACS 2.0: drives accept AACS 1.0 host certs for backward compatibility
//! (full P-256/SHA-256 AACS 2.0 handshake prepared but rarely needed)
use crate::error::{Error, Result};
use crate::drive::DriveSession;
use crate::scsi::DataDirection;
use num_bigint::BigUint;
use num_traits::{One, Zero};
use sha1::{Sha1, Digest};
/// Execute a SCSI command that reads data from the device.
fn scsi_read(session: &mut DriveSession, cdb: &[u8], len: usize) -> Result<Vec<u8>> {
let mut buf = vec![0u8; len];
session.scsi_execute(cdb, DataDirection::FromDevice, &mut buf, 5_000)?;
Ok(buf)
}
/// Execute a SCSI command that writes data to the device.
fn scsi_write(session: &mut DriveSession, cdb: &[u8], data: &[u8]) -> Result<()> {
let mut buf = data.to_vec();
session.scsi_execute(cdb, DataDirection::ToDevice, &mut buf, 5_000)?;
Ok(())
}
// ── AACS 1.0 elliptic curve parameters (160-bit) ───────────────────────────
const EC_P: [u8; 20] = [
0x9D, 0xC9, 0xD8, 0x13, 0x55, 0xEC, 0xCE, 0xB5, 0x60, 0xBD,
0xB0, 0x9E, 0xF9, 0xEA, 0xE7, 0xC4, 0x79, 0xA7, 0xD7, 0xDF,
];
const EC_A: [u8; 20] = [
0x9D, 0xC9, 0xD8, 0x13, 0x55, 0xEC, 0xCE, 0xB5, 0x60, 0xBD,
0xB0, 0x9E, 0xF9, 0xEA, 0xE7, 0xC4, 0x79, 0xA7, 0xD7, 0xDC,
];
#[cfg(test)]
const EC_B: [u8; 20] = [
0x40, 0x2D, 0xAD, 0x3E, 0xC1, 0xCB, 0xCD, 0x16, 0x52, 0x48,
0xD6, 0x8E, 0x12, 0x45, 0xE0, 0xC4, 0xDA, 0xAC, 0xB1, 0xD8,
];
const EC_N: [u8; 20] = [
0x9D, 0xC9, 0xD8, 0x13, 0x55, 0xEC, 0xCE, 0xB5, 0x60, 0xBD,
0xC4, 0x4F, 0x54, 0x81, 0x7B, 0x2C, 0x7F, 0x5A, 0xB0, 0x17,
];
const EC_GX: [u8; 20] = [
0x2E, 0x64, 0xFC, 0x22, 0x57, 0x83, 0x51, 0xE6, 0xF4, 0xCC,
0xA7, 0xEB, 0x81, 0xD0, 0xA4, 0xBD, 0xC5, 0x4C, 0xCE, 0xC6,
];
const EC_GY: [u8; 20] = [
0x09, 0x14, 0xA2, 0x5D, 0xD0, 0x54, 0x42, 0x88, 0x9D, 0xB4,
0x55, 0xC7, 0xF2, 0x3C, 0x9A, 0x07, 0x07, 0xF5, 0xCB, 0xB9,
];
// ── AACS LA (Licensing Administrator) public key for cert verification ──────
const AACS_LA_PUB_X: [u8; 20] = [
0x01, 0xF3, 0x5D, 0xAB, 0xD8, 0xAE, 0x5F, 0x40, 0x56, 0x5E,
0x30, 0xC8, 0x8A, 0x60, 0x42, 0x82, 0x07, 0x61, 0xDF, 0x93,
];
const AACS_LA_PUB_Y: [u8; 20] = [
0x44, 0x87, 0xB5, 0xAC, 0x07, 0x10, 0x8D, 0x10, 0x5B, 0xA5,
0xB9, 0xE3, 0x2F, 0x3B, 0xBB, 0xFC, 0x0C, 0x2C, 0xBC, 0xD1,
];
// ── Elliptic curve arithmetic over GF(p) ───────────────────────────────────
#[derive(Clone, Debug)]
struct EcPoint {
x: BigUint,
y: BigUint,
infinity: bool,
}
impl EcPoint {
fn infinity() -> Self {
EcPoint { x: BigUint::zero(), y: BigUint::zero(), infinity: true }
}
fn new(x: BigUint, y: BigUint) -> Self {
EcPoint { x, y, infinity: false }
}
fn from_bytes(x_bytes: &[u8], y_bytes: &[u8]) -> Self {
EcPoint::new(BigUint::from_bytes_be(x_bytes), BigUint::from_bytes_be(y_bytes))
}
}
/// Modular inverse using extended Euclidean algorithm.
fn mod_inv(a: &BigUint, m: &BigUint) -> Option<BigUint> {
use num_bigint::BigInt;
use num_traits::Signed;
let a = BigInt::from(a.clone());
let m = BigInt::from(m.clone());
let (mut old_r, mut r) = (a, m.clone());
let (mut old_s, mut s) = (BigInt::one(), BigInt::zero());
while !r.is_zero() {
let q = &old_r / &r;
let temp_r = r.clone();
r = old_r - &q * &r;
old_r = temp_r;
let temp_s = s.clone();
s = old_s - &q * &s;
old_s = temp_s;
}
if old_r != BigInt::one() {
return None;
}
if old_s.is_negative() {
old_s += &m;
}
Some(old_s.to_biguint().unwrap())
}
/// EC point addition on curve y² = x³ + ax + b (mod p).
fn ec_add(p1: &EcPoint, p2: &EcPoint, a: &BigUint, p: &BigUint) -> EcPoint {
if p1.infinity { return p2.clone(); }
if p2.infinity { return p1.clone(); }
if p1.x == p2.x {
if p1.y == p2.y && !p1.y.is_zero() {
return ec_double(p1, a, p);
}
return EcPoint::infinity();
}
// λ = (y2 - y1) / (x2 - x1) mod p
let dy = if p2.y >= p1.y {
(&p2.y - &p1.y) % p
} else {
(p - (&p1.y - &p2.y) % p) % p
};
let dx = if p2.x >= p1.x {
(&p2.x - &p1.x) % p
} else {
(p - (&p1.x - &p2.x) % p) % p
};
let dx_inv = mod_inv(&dx, p).unwrap();
let lam = (&dy * &dx_inv) % p;
// x3 = λ² - x1 - x2 mod p
let x3 = {
let lam2 = (&lam * &lam) % p;
let sum = (&p1.x + &p2.x) % p;
if lam2 >= sum {
(lam2 - sum) % p
} else {
(p - (sum - lam2) % p) % p
}
};
// y3 = λ(x1 - x3) - y1 mod p
let y3 = {
let diff = if p1.x >= x3 {
(&p1.x - &x3) % p
} else {
(p - (&x3 - &p1.x) % p) % p
};
let prod = (&lam * &diff) % p;
if prod >= p1.y {
(prod - &p1.y) % p
} else {
(p - (&p1.y - prod) % p) % p
}
};
EcPoint::new(x3, y3)
}
/// EC point doubling.
fn ec_double(pt: &EcPoint, a: &BigUint, p: &BigUint) -> EcPoint {
if pt.infinity || pt.y.is_zero() {
return EcPoint::infinity();
}
// λ = (3x² + a) / (2y) mod p
let three = BigUint::from(3u32);
let two = BigUint::from(2u32);
let numerator = (&three * &pt.x * &pt.x + a) % p;
let denominator = (&two * &pt.y) % p;
let denom_inv = mod_inv(&denominator, p).unwrap();
let lam = (&numerator * &denom_inv) % p;
// x3 = λ² - 2x mod p
let x3 = {
let lam2 = (&lam * &lam) % p;
let two_x = (&two * &pt.x) % p;
if lam2 >= two_x {
(lam2 - two_x) % p
} else {
(p - (two_x - lam2) % p) % p
}
};
// y3 = λ(x - x3) - y mod p
let y3 = {
let diff = if pt.x >= x3 {
(&pt.x - &x3) % p
} else {
(p - (&x3 - &pt.x) % p) % p
};
let prod = (&lam * &diff) % p;
if prod >= pt.y {
(prod - &pt.y) % p
} else {
(p - (&pt.y - prod) % p) % p
}
};
EcPoint::new(x3, y3)
}
/// Scalar multiplication using double-and-add.
fn ec_mul(k: &BigUint, pt: &EcPoint, a: &BigUint, p: &BigUint) -> EcPoint {
if k.is_zero() {
return EcPoint::infinity();
}
let mut result = EcPoint::infinity();
let mut base = pt.clone();
let mut scalar = k.clone();
while !scalar.is_zero() {
if scalar.bit(0) {
result = ec_add(&result, &base, a, p);
}
base = ec_double(&base, a, p);
scalar >>= 1;
}
result
}
/// Convert BigUint to fixed-size big-endian bytes, zero-padded.
fn to_bytes_be_padded(n: &BigUint, len: usize) -> Vec<u8> {
let bytes = n.to_bytes_be();
if bytes.len() >= len {
bytes[bytes.len() - len..].to_vec()
} else {
let mut padded = vec![0u8; len - bytes.len()];
padded.extend_from_slice(&bytes);
padded
}
}
// ── ECDSA ───────────────────────────────────────────────────────────────────
/// ECDSA sign: sign SHA-1(data) with private key on AACS curve.
/// Returns (r, s) each 20 bytes.
fn ecdsa_sign(priv_key: &[u8; 20], data: &[u8]) -> ([u8; 20], [u8; 20]) {
let p = BigUint::from_bytes_be(&EC_P);
let a = BigUint::from_bytes_be(&EC_A);
let n = BigUint::from_bytes_be(&EC_N);
let g = EcPoint::from_bytes(&EC_GX, &EC_GY);
let d = BigUint::from_bytes_be(priv_key);
// Hash the data
let hash = Sha1::digest(data);
let z = BigUint::from_bytes_be(&hash);
loop {
// Generate random k
let mut k_bytes = [0u8; 20];
use rand::RngCore;
rand::thread_rng().fill_bytes(&mut k_bytes);
let k = BigUint::from_bytes_be(&k_bytes) % &n;
if k.is_zero() { continue; }
// R = k × G
let r_point = ec_mul(&k, &g, &a, &p);
let r = &r_point.x % &n;
if r.is_zero() { continue; }
// s = k⁻¹(z + r·d) mod n
let k_inv = match mod_inv(&k, &n) {
Some(v) => v,
None => continue,
};
let s = (&k_inv * ((&z + &r * &d) % &n)) % &n;
if s.is_zero() { continue; }
let r_bytes = to_bytes_be_padded(&r, 20);
let s_bytes = to_bytes_be_padded(&s, 20);
let mut r_out = [0u8; 20];
let mut s_out = [0u8; 20];
r_out.copy_from_slice(&r_bytes);
s_out.copy_from_slice(&s_bytes);
return (r_out, s_out);
}
}
/// ECDSA verify: verify signature (r, s) against SHA-1(data) using public key.
fn ecdsa_verify(pub_x: &[u8; 20], pub_y: &[u8; 20], sig_r: &[u8; 20], sig_s: &[u8; 20], data: &[u8]) -> bool {
let p = BigUint::from_bytes_be(&EC_P);
let a = BigUint::from_bytes_be(&EC_A);
let n = BigUint::from_bytes_be(&EC_N);
let g = EcPoint::from_bytes(&EC_GX, &EC_GY);
let q = EcPoint::from_bytes(pub_x, pub_y);
let r = BigUint::from_bytes_be(sig_r);
let s = BigUint::from_bytes_be(sig_s);
if r.is_zero() || r >= n || s.is_zero() || s >= n {
return false;
}
let hash = Sha1::digest(data);
let z = BigUint::from_bytes_be(&hash);
let s_inv = match mod_inv(&s, &n) {
Some(v) => v,
None => return false,
};
let u1 = (&z * &s_inv) % &n;
let u2 = (&r * &s_inv) % &n;
let p1 = ec_mul(&u1, &g, &a, &p);
let p2 = ec_mul(&u2, &q, &a, &p);
let r_point = ec_add(&p1, &p2, &a, &p);
if r_point.infinity {
return false;
}
&r_point.x % &n == r
}
// ── AACS certificate handling ───────────────────────────────────────────────
/// Verify an AACS certificate (92 bytes) against the AACS LA public key.
fn verify_cert(cert: &[u8]) -> bool {
if cert.len() < 92 { return false; }
// Certificate format: type(1) + flags(1) + padding(2) + serial(6) + pub_x(20) + pub_y(20) + sig_r(20) + sig_s(20)
// Signature is over the first 52 bytes
let mut sig_r = [0u8; 20];
let mut sig_s = [0u8; 20];
sig_r.copy_from_slice(&cert[52..72]);
sig_s.copy_from_slice(&cert[72..92]);
ecdsa_verify(&AACS_LA_PUB_X, &AACS_LA_PUB_Y, &sig_r, &sig_s, &cert[..52])
}
/// Extract public key from certificate.
fn cert_pub_key(cert: &[u8]) -> ([u8; 20], [u8; 20]) {
let mut x = [0u8; 20];
let mut y = [0u8; 20];
x.copy_from_slice(&cert[12..32]);
y.copy_from_slice(&cert[32..52]);
(x, y)
}
// ── Bus key derivation (ECDH) ───────────────────────────────────────────────
/// Compute bus key via ECDH: bus_key = low 128 bits of (host_priv × drive_key_point).x
fn compute_bus_key(host_priv: &[u8; 20], drive_key_point_x: &[u8; 20], drive_key_point_y: &[u8; 20]) -> [u8; 16] {
let p = BigUint::from_bytes_be(&EC_P);
let a = BigUint::from_bytes_be(&EC_A);
let d = BigUint::from_bytes_be(host_priv);
let dkp = EcPoint::from_bytes(drive_key_point_x, drive_key_point_y);
let shared = ec_mul(&d, &dkp, &a, &p);
// Bus key = lowest 128 bits (last 16 bytes) of x-coordinate
let x_bytes = to_bytes_be_padded(&shared.x, 20);
let mut bus_key = [0u8; 16];
bus_key.copy_from_slice(&x_bytes[4..20]); // last 16 of 20
bus_key
}
/// Generate ephemeral host key pair: (private_key, public_point_x, public_point_y).
fn generate_host_key_pair() -> ([u8; 20], [u8; 20], [u8; 20]) {
let p_mod = BigUint::from_bytes_be(&EC_P);
let a = BigUint::from_bytes_be(&EC_A);
let g = EcPoint::from_bytes(&EC_GX, &EC_GY);
let mut priv_bytes = [0u8; 20];
use rand::RngCore;
rand::thread_rng().fill_bytes(&mut priv_bytes);
let d = BigUint::from_bytes_be(&priv_bytes);
let q = ec_mul(&d, &g, &a, &p_mod);
let qx = to_bytes_be_padded(&q.x, 20);
let qy = to_bytes_be_padded(&q.y, 20);
let mut pub_x = [0u8; 20];
let mut pub_y = [0u8; 20];
pub_x.copy_from_slice(&qx);
pub_y.copy_from_slice(&qy);
(priv_bytes, pub_x, pub_y)
}
// ── AES-CMAC (for MAC verification) ────────────────────────────────────────
/// AES-128-CMAC over 16 bytes of data.
fn aes_cmac_16(data: &[u8; 16], key: &[u8; 16]) -> [u8; 16] {
use aes::Aes128;
use aes::cipher::{BlockEncrypt, KeyInit, generic_array::GenericArray};
let cipher = Aes128::new(GenericArray::from_slice(key));
// For single-block CMAC:
// 1. Generate subkey K1
let mut l = GenericArray::clone_from_slice(&[0u8; 16]);
cipher.encrypt_block(&mut l);
let mut k1 = [0u8; 16];
let carry = (l[0] >> 7) & 1;
for i in 0..15 {
k1[i] = (l[i] << 1) | (l[i + 1] >> 7);
}
k1[15] = l[15] << 1;
if carry == 1 {
k1[15] ^= 0x87; // Rb for AES-128
}
// 2. XOR data with K1, encrypt
let mut block = [0u8; 16];
for i in 0..16 {
block[i] = data[i] ^ k1[i];
}
let mut ga = GenericArray::clone_from_slice(&block);
cipher.encrypt_block(&mut ga);
let mut mac = [0u8; 16];
mac.copy_from_slice(&ga);
mac
}
// ── SCSI command builders ───────────────────────────────────────────────────
/// Build REPORT KEY CDB (0xA4).
fn cdb_report_key(agid: u8, format: u8, len: u16) -> [u8; 12] {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
cdb[7] = crate::scsi::AACS_KEY_CLASS;
cdb[8] = (len >> 8) as u8;
cdb[9] = (len & 0xFF) as u8;
cdb[10] = (agid << 6) | (format & 0x3F);
cdb
}
/// Build SEND KEY CDB (0xA3).
fn cdb_send_key(agid: u8, format: u8, len: u16) -> [u8; 12] {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_SEND_KEY;
cdb[7] = crate::scsi::AACS_KEY_CLASS;
cdb[8] = (len >> 8) as u8;
cdb[9] = (len & 0xFF) as u8;
cdb[10] = (agid << 6) | (format & 0x3F);
cdb
}
/// Build REPORT DISC STRUCTURE CDB (0xAD).
fn cdb_report_disc_structure(agid: u8, format: u8, len: u16) -> [u8; 12] {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_READ_DISC_STRUCTURE;
cdb[1] = 0x01; // Blu-ray
cdb[7] = format;
cdb[8] = (len >> 8) as u8;
cdb[9] = (len & 0xFF) as u8;
cdb[10] = agid << 6;
cdb
}
// ── High-level handshake ────────────────────────────────────────────────────
/// Result of a successful AACS authentication handshake.
#[derive(Debug)]
pub struct AacsAuth {
/// Bus key (16 bytes) — derived from ECDH
pub bus_key: [u8; 16],
/// AGID used for this session
pub agid: u8,
/// Volume ID (16 bytes) — read after auth
pub volume_id: Option<[u8; 16]>,
/// Read data key (16 bytes) — for AACS 2.0 bus decryption
pub read_data_key: Option<[u8; 16]>,
/// Drive certificate (92 bytes)
pub drive_cert: [u8; 92],
}
/// Perform the full AACS authentication handshake.
///
/// Requires a host private key (20 bytes) and host certificate (92 bytes)
/// from the KEYDB.cfg HC entry.
pub fn aacs_authenticate(
session: &mut DriveSession,
host_priv_key: &[u8; 20],
host_cert: &[u8],
) -> Result<AacsAuth> {
if host_cert.len() < 92 {
return Err(Error::AacsError { detail: "host certificate too short".into() });
}
// Step 1: Invalidate all AGIDs
for agid in 0..4u8 {
let cdb = cdb_report_key(agid, 0x3F, 2);
let _ = scsi_read(session, &cdb, 2);
}
// Step 2: Allocate AGID
let cdb = cdb_report_key(0, 0x00, 8);
let response = scsi_read(session, &cdb, 8)
.map_err(|e| Error::AacsError { detail: format!("failed to allocate AGID: {}", e) })?;
let agid = (response[7] >> 6) & 0x03;
// Step 3: Generate host nonce and ephemeral key pair
let mut host_nonce = [0u8; 20];
use rand::RngCore;
rand::thread_rng().fill_bytes(&mut host_nonce);
let (host_key, host_key_point_x, host_key_point_y) = generate_host_key_pair();
// Step 4: Send host certificate + nonce (SEND KEY format 0x01)
let mut send_buf = [0u8; 116];
send_buf[1] = 0x72; // data length
send_buf[4..24].copy_from_slice(&host_nonce);
send_buf[24..116].copy_from_slice(&host_cert[..92]);
let cdb = cdb_send_key(agid, 0x01, 116);
scsi_write(session, &cdb, &send_buf)
.map_err(|_| Error::AacsError { detail: "drive rejected host certificate".into() })?;
// Step 5: Read drive certificate + nonce (REPORT KEY format 0x01)
let cdb = cdb_report_key(agid, 0x01, 116);
let response = scsi_read(session, &cdb, 116)
.map_err(|_| Error::AacsError { detail: "failed to read drive certificate".into() })?;
let mut drive_nonce = [0u8; 20];
let mut drive_cert = [0u8; 92];
drive_nonce.copy_from_slice(&response[4..24]);
drive_cert.copy_from_slice(&response[24..116]);
// Detect AACS 2.0 drive certificate (type 0x11)
// AACS 2.0 drives use P-256/SHA-256 natively but accept AACS 1.0 host certs
// for backward compatibility. We proceed with AACS 1.0 handshake.
if drive_cert[0] == 0x11 {
// AACS 2.0 drive detected — falling back to AACS 1.0 handshake
// (full P-256 AACS 2.0 handshake not yet implemented)
// The drive should still accept our AACS 1.0 host certificate.
}
// Verify drive certificate (AACS 1.0 LA signature)
if drive_cert[0] == 0x01 && !verify_cert(&drive_cert) {
return Err(Error::AacsError { detail: "drive certificate verification failed".into() });
}
// Skip verification for AACS 2.0 certs (different LA key, P-256 curve)
// Step 6: Read drive key point + signature (REPORT KEY format 0x02)
let cdb = cdb_report_key(agid, 0x02, 84);
let response = scsi_read(session, &cdb, 84)
.map_err(|_| Error::AacsError { detail: "failed to read drive key".into() })?;
let mut drive_key_point = [0u8; 40]; // x(20) + y(20)
let mut drive_key_sig = [0u8; 40]; // r(20) + s(20)
drive_key_point.copy_from_slice(&response[4..44]);
drive_key_sig.copy_from_slice(&response[44..84]);
// Verify drive key signature: sign(drive_nonce=host_nonce || drive_key_point)
let (drive_pub_x, drive_pub_y) = cert_pub_key(&drive_cert);
let mut verify_data = [0u8; 60];
verify_data[..20].copy_from_slice(&host_nonce);
verify_data[20..60].copy_from_slice(&drive_key_point);
let mut sig_r = [0u8; 20];
let mut sig_s = [0u8; 20];
sig_r.copy_from_slice(&drive_key_sig[..20]);
sig_s.copy_from_slice(&drive_key_sig[20..40]);
if !ecdsa_verify(&drive_pub_x, &drive_pub_y, &sig_r, &sig_s, &verify_data) {
return Err(Error::AacsError { detail: "drive key signature verification failed".into() });
}
// Step 7: Sign host key point (ECDSA over drive_nonce || host_key_point)
let mut sign_data = [0u8; 60];
sign_data[..20].copy_from_slice(&drive_nonce);
sign_data[20..40].copy_from_slice(&host_key_point_x);
sign_data[40..60].copy_from_slice(&host_key_point_y);
let (host_sig_r, host_sig_s) = ecdsa_sign(host_priv_key, &sign_data);
// Step 8: Send host key point + signature (SEND KEY format 0x02)
let mut send_buf = [0u8; 84];
send_buf[1] = 0x52;
send_buf[4..24].copy_from_slice(&host_key_point_x);
send_buf[24..44].copy_from_slice(&host_key_point_y);
send_buf[44..64].copy_from_slice(&host_sig_r);
send_buf[64..84].copy_from_slice(&host_sig_s);
let cdb = cdb_send_key(agid, 0x02, 84);
scsi_write(session, &cdb, &send_buf)
.map_err(|_| Error::AacsError { detail: "drive rejected host key".into() })?;
// Step 9: Compute bus key via ECDH
let mut dkp_x = [0u8; 20];
let mut dkp_y = [0u8; 20];
dkp_x.copy_from_slice(&drive_key_point[..20]);
dkp_y.copy_from_slice(&drive_key_point[20..40]);
let bus_key = compute_bus_key(&host_key, &dkp_x, &dkp_y);
Ok(AacsAuth {
bus_key,
agid,
volume_id: None,
read_data_key: None,
drive_cert,
})
}
/// Read Volume ID after successful authentication.
pub fn read_volume_id(session: &mut DriveSession, auth: &mut AacsAuth) -> Result<[u8; 16]> {
// REPORT DISC STRUCTURE format 0x80
let cdb = cdb_report_disc_structure(auth.agid, 0x80, 36);
let response = scsi_read(session, &cdb, 36)
.map_err(|_| Error::AacsError { detail: "failed to read Volume ID".into() })?;
let mut vid = [0u8; 16];
let mut mac = [0u8; 16];
vid.copy_from_slice(&response[4..20]);
mac.copy_from_slice(&response[20..36]);
// Verify MAC: AES-CMAC(VID, bus_key) should equal mac
let calc_mac = aes_cmac_16(&vid, &auth.bus_key);
if calc_mac != mac {
return Err(Error::AacsError { detail: "VID MAC verification failed".into() });
}
auth.volume_id = Some(vid);
Ok(vid)
}
/// Read data keys after successful authentication (for AACS 2.0 bus encryption).
pub fn read_data_keys(session: &mut DriveSession, auth: &mut AacsAuth) -> Result<([u8; 16], [u8; 16])> {
// REPORT DISC STRUCTURE format 0x84
let cdb = cdb_report_disc_structure(auth.agid, 0x84, 36);
let response = scsi_read(session, &cdb, 36)
.map_err(|_| Error::AacsError { detail: "failed to read data keys".into() })?;
let mut enc_rdk = [0u8; 16];
let mut enc_wdk = [0u8; 16];
enc_rdk.copy_from_slice(&response[4..20]);
enc_wdk.copy_from_slice(&response[20..36]);
// Decrypt with bus key (AES-ECB)
let read_data_key = super::aes_ecb_decrypt(&auth.bus_key, &enc_rdk);
let write_data_key = super::aes_ecb_decrypt(&auth.bus_key, &enc_wdk);
auth.read_data_key = Some(read_data_key);
Ok((read_data_key, write_data_key))
}
// ── Tests ───────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_ec_curve_generator_on_curve() {
// Verify G is on the curve: y² = x³ + ax + b (mod p)
let p = BigUint::from_bytes_be(&EC_P);
let a = BigUint::from_bytes_be(&EC_A);
let b = BigUint::from_bytes_be(&EC_B);
let gx = BigUint::from_bytes_be(&EC_GX);
let gy = BigUint::from_bytes_be(&EC_GY);
let lhs = (&gy * &gy) % &p;
let rhs = (&gx * &gx * &gx + &a * &gx + &b) % &p;
assert_eq!(lhs, rhs, "Generator point is not on the curve");
}
#[test]
fn test_ec_mul_identity() {
let p = BigUint::from_bytes_be(&EC_P);
let a = BigUint::from_bytes_be(&EC_A);
let g = EcPoint::from_bytes(&EC_GX, &EC_GY);
// 1 × G = G
let result = ec_mul(&BigUint::one(), &g, &a, &p);
assert_eq!(result.x, g.x);
assert_eq!(result.y, g.y);
}
#[test]
fn test_ec_mul_order() {
// n × G = O (point at infinity)
let p = BigUint::from_bytes_be(&EC_P);
let a = BigUint::from_bytes_be(&EC_A);
let n = BigUint::from_bytes_be(&EC_N);
let g = EcPoint::from_bytes(&EC_GX, &EC_GY);
let result = ec_mul(&n, &g, &a, &p);
assert!(result.infinity, "n × G should be point at infinity");
}
#[test]
fn test_ecdsa_sign_verify() {
// Generate a key pair and test sign/verify
let (priv_key, pub_x, pub_y) = generate_host_key_pair();
let data = b"test data for AACS ECDSA";
let (sig_r, sig_s) = ecdsa_sign(&priv_key, data);
assert!(ecdsa_verify(&pub_x, &pub_y, &sig_r, &sig_s, data),
"ECDSA signature should verify");
// Verify with wrong data fails
assert!(!ecdsa_verify(&pub_x, &pub_y, &sig_r, &sig_s, b"wrong data"),
"ECDSA should fail with wrong data");
}
#[test]
fn test_ecdh_shared_secret() {
// Two parties should derive the same shared point
let p = BigUint::from_bytes_be(&EC_P);
let a = BigUint::from_bytes_be(&EC_A);
let g = EcPoint::from_bytes(&EC_GX, &EC_GY);
let (priv_a, pub_ax, pub_ay) = generate_host_key_pair();
let (priv_b, pub_bx, pub_by) = generate_host_key_pair();
// A computes: priv_a × pub_B
let shared_a = compute_bus_key(&priv_a, &pub_bx, &pub_by);
// B computes: priv_b × pub_A
let shared_b = compute_bus_key(&priv_b, &pub_ax, &pub_ay);
assert_eq!(shared_a, shared_b, "ECDH shared secrets should match");
}
#[test]
fn test_aes_cmac() {
// Basic CMAC test — at minimum verify it produces consistent output
let key = [0x2b, 0x7e, 0x15, 0x16, 0x28, 0xae, 0xd2, 0xa6,
0xab, 0xf7, 0x15, 0x88, 0x09, 0xcf, 0x4f, 0x3c];
let data = [0u8; 16];
let mac1 = aes_cmac_16(&data, &key);
let mac2 = aes_cmac_16(&data, &key);
assert_eq!(mac1, mac2);
assert_ne!(mac1, [0u8; 16]); // shouldn't be all zeros
}
#[test]
fn test_verify_host_cert_from_keydb() {
// Verify the host cert from our KEYDB
let keydb_path = match std::env::var("KEYDB_PATH").ok() {
Some(p) => std::path::PathBuf::from(p),
None => return, // skip if KEYDB_PATH not set
};
if !keydb_path.exists() { return; }
let db = crate::aacs::KeyDb::load(&keydb_path).unwrap();
if let Some(hc) = &db.host_cert {
let valid = verify_cert(&hc.certificate);
eprintln!("Host cert verification: {}", if valid { "PASS" } else { "FAIL" });
// Note: our cert is revoked but should still have valid LA signature
// If it doesn't verify, the LA public key might be wrong
if !valid {
eprintln!(" (cert may use different LA key or format)");
}
}
}
}
+22
View File
@@ -0,0 +1,22 @@
//! Host-certificate collection — the one libfreemkv-side concern left from the
//! old in-tree AACS handshake. The cert mutual-auth itself now lives in the
//! `freemkv-unlock` AACS unlocker; libfreemkv only gathers the certs (a
//! keysource concern) and hands them across the seam.
/// Union the host certificates a scan can offer the drive: the explicit
/// `DriveCredentials`, then each key source's `host_certs(mkb)`. Host certs are
/// keysource-served, never compiled in. `mkb` lets a source pick a
/// generation-appropriate cert (the default impl ignores it).
pub fn collect_host_certs(
opts: &crate::disc::ScanOptions,
mkb: Option<u32>,
) -> Vec<crate::aacs::HostCert> {
let mut host_certs: Vec<crate::aacs::HostCert> = Vec::new();
if let Some(c) = &opts.credentials {
host_certs.extend(c.host_certs.iter().cloned());
}
for src in &opts.key_sources {
host_certs.extend(src.host_certs(mkb));
}
host_certs
}
+3126
View File
File diff suppressed because it is too large Load Diff
+85 -1409
View File
File diff suppressed because it is too large Load Diff
+409
View File
@@ -0,0 +1,409 @@
//! Key source abstraction for the AACS resolve chain.
//!
//! libfreemkv keeps all crypto (AES-G primitives, SD-tree walking,
//! validation, MK/VUK/TK derivation) but accepts key material from
//! arbitrary backends via [`KeyProvider`].
//!
//! Methods come in two flavors:
//!
//! - **Bulk material** ([`device_keys`], [`processing_keys`],
//! [`media_keys`]) — the resolver unions (and dedups) results
//! across all providers and tries each candidate.
//! - **Disc-keyed lookup** ([`lookup_disc_by_hash`],
//! [`lookup_disc_by_vid`]) — the resolver short-circuits on the
//! first hit, so providers are queried in array order with
//! fastest/closest first.
//!
//! [`host_certs`] is a sixth method but is NOT consumed by the
//! resolver chain: the SCSI handshake reads host certs directly from
//! the caller-supplied credentials, not from the provider array. A
//! provider that overrides `host_certs` today has no effect on the
//! handshake; the method is retained as a forward-looking extension
//! point only.
//!
//! Default impls return empty / `None` so backends only override
//! the methods they actually support — an external key service might
//! implement only `lookup_disc_by_hash`, while a local file might
//! implement all six.
//!
//! Calls may block (disk I/O, network round-trips). The resolver
//! invokes each method at most a handful of times per scan; for
//! per-disc memoization, implementations should cache internally.
//!
//! [`device_keys`]: KeyProvider::device_keys
//! [`processing_keys`]: KeyProvider::processing_keys
//! [`media_keys`]: KeyProvider::media_keys
//! [`host_certs`]: KeyProvider::host_certs
//! [`lookup_disc_by_hash`]: KeyProvider::lookup_disc_by_hash
//! [`lookup_disc_by_vid`]: KeyProvider::lookup_disc_by_vid
use super::types::{DeviceKey, DiscEntry, HostCert};
/// Source of AACS key material.
///
/// Implementors return raw material only — the resolver in
/// `aacs::keys` owns all the crypto (DK→PK walking, PK validation,
/// MK→VUK→TK derivation). See module docs for method semantics.
pub trait KeyProvider: Send + Sync {
/// Device keys (top-of-tree, walked by the resolver).
fn device_keys(&self) -> Vec<DeviceKey> {
Vec::new()
}
/// Processing keys — terminal PKs or walk-input PKs. The
/// resolver tries each as a terminal first (cheap validate).
fn processing_keys(&self) -> Vec<[u8; 16]> {
Vec::new()
}
/// Every Media Key this provider holds, regardless of which disc it was
/// filed under. An MK is MKB-scoped (shared across a pressing/MKB-family),
/// so the resolver can verify each against the disc's MKB (`km_verifies`)
/// and resolve a disc whose own hash/VID isn't directly keyed.
fn media_keys(&self) -> Vec<[u8; 16]> {
Vec::new()
}
/// AACS host certificates (with their private keys) for drive
/// authentication. Multiple in case some are revoked.
///
/// NOTE: not consumed by the resolver chain — the handshake reads
/// host certs from the caller-supplied credentials directly, so
/// overriding this method has no effect on drive authentication
/// today. Retained as a forward-looking extension point.
fn host_certs(&self) -> Vec<HostCert> {
Vec::new()
}
/// Direct per-disc lookup by SHA-1 of `Unit_Key_RO.inf`. Returns
/// `Some(entry)` if this provider has pre-computed material for
/// the disc (paths 4 and 5). Short-circuits the resolver.
fn lookup_disc_by_hash(&self, _disc_hash: &[u8; 20]) -> Option<DiscEntry> {
None
}
/// Lookup by Volume ID (path 3 — pre-computed MK + matching
/// VID). Short-circuits the resolver on hit.
fn lookup_disc_by_vid(&self, _volume_id: &[u8; 16]) -> Option<DiscEntry> {
None
}
}
/// Resolver-side helpers that aggregate across a provider array.
///
/// The resolver wraps `ctx.providers` (`&[&dyn KeyProvider]`) in this
/// struct; these helpers apply the union-vs-short-circuit policy per
/// method. The bulk unions dedup so overlapping providers don't make
/// the resolver re-walk/re-validate identical material.
pub(crate) struct Providers<'a>(pub &'a [&'a dyn KeyProvider]);
impl Providers<'_> {
/// Union (deduped) — gather DKs from every provider.
pub fn device_keys(&self) -> Vec<DeviceKey> {
let mut v: Vec<DeviceKey> = self.0.iter().flat_map(|p| p.device_keys()).collect();
// DeviceKey has no Ord/Hash; dedup on the value-defining tuple.
v.sort_unstable_by_key(|d| (d.key, d.node, d.uv, d.u_mask_shift));
v.dedup_by_key(|d| (d.key, d.node, d.uv, d.u_mask_shift));
v
}
/// Union (deduped) — gather PKs from every provider.
pub fn processing_keys(&self) -> Vec<[u8; 16]> {
let mut v: Vec<[u8; 16]> = self.0.iter().flat_map(|p| p.processing_keys()).collect();
v.sort_unstable();
v.dedup();
v
}
/// Union of distinct Media Keys across every provider, for the MK-pool
/// brute (`km_verifies` against the disc's MKB).
pub fn media_keys(&self) -> Vec<[u8; 16]> {
let mut v: Vec<[u8; 16]> = self.0.iter().flat_map(|p| p.media_keys()).collect();
v.sort_unstable();
v.dedup();
v
}
/// Union — gather host certs from every provider. The SCSI handshake
/// reads host certs from the caller-supplied credentials directly and
/// does not call this, so it is currently unused by the resolver chain.
#[allow(dead_code)]
pub fn host_certs(&self) -> Vec<HostCert> {
self.0.iter().flat_map(|p| p.host_certs()).collect()
}
/// Short-circuit — query providers in array order, first hit wins.
pub fn lookup_disc_by_hash(&self, disc_hash: &[u8; 20]) -> Option<DiscEntry> {
self.0.iter().find_map(|p| p.lookup_disc_by_hash(disc_hash))
}
/// Short-circuit — query providers in array order, first hit wins.
pub fn lookup_disc_by_vid(&self, volume_id: &[u8; 16]) -> Option<DiscEntry> {
self.0.iter().find_map(|p| p.lookup_disc_by_vid(volume_id))
}
}
/// A [`KeyProvider`] backed by a single caller-supplied key's raw material —
/// the bridge for [`crate::disc::Disc::decrypt_with`].
///
/// The application's key source did the lookup and handed in material at one
/// level (DK / PK / MK / VUK). This exposes exactly that material to the
/// version-dispatched resolver, which owns ALL derivation — so a source never
/// derives, and the lib remains the single home for the AACS chain across
/// 1.0 / 2.0 / 2.1 / 2.x.
///
/// Each level fills only its own field; the rest stay empty, so the resolver
/// naturally runs the matching path (DK→…, PK→…, MK-pool brute, or a
/// disc-keyed VUK hit). `decrypt_with` already knows the disc, so the
/// `lookup_disc_by_*` hash/VID arguments are irrelevant — a present
/// `disc_entry` is returned for any query.
pub(crate) struct SuppliedKey {
pub device_keys: Vec<DeviceKey>,
pub processing_keys: Vec<[u8; 16]>,
pub media_keys: Vec<[u8; 16]>,
pub disc_entry: Option<DiscEntry>,
}
impl KeyProvider for SuppliedKey {
fn device_keys(&self) -> Vec<DeviceKey> {
self.device_keys.clone()
}
fn processing_keys(&self) -> Vec<[u8; 16]> {
self.processing_keys.clone()
}
fn media_keys(&self) -> Vec<[u8; 16]> {
self.media_keys.clone()
}
fn lookup_disc_by_hash(&self, _disc_hash: &[u8; 20]) -> Option<DiscEntry> {
self.disc_entry.clone()
}
fn lookup_disc_by_vid(&self, _volume_id: &[u8; 16]) -> Option<DiscEntry> {
self.disc_entry.clone()
}
}
#[cfg(test)]
mod tests {
use super::*;
fn entry(hash: &str, vuk: u8) -> DiscEntry {
DiscEntry {
disc_hash: hash.to_string(),
title: "t".to_string(),
media_key: None,
disc_id: None,
vuk: Some([vuk; 16]),
unit_keys: Vec::new(),
}
}
fn dk(byte: u8, node: u16) -> DeviceKey {
DeviceKey {
key: [byte; 16],
node,
uv: 1,
u_mask_shift: 0,
}
}
/// A provider that returns fixed bulk material and an optional disc entry
/// keyed unconditionally (used to test array-order short-circuiting).
#[derive(Default)]
struct Fixed {
dks: Vec<DeviceKey>,
pks: Vec<[u8; 16]>,
mks: Vec<[u8; 16]>,
hash_hit: Option<DiscEntry>,
vid_hit: Option<DiscEntry>,
}
impl KeyProvider for Fixed {
fn device_keys(&self) -> Vec<DeviceKey> {
self.dks.clone()
}
fn processing_keys(&self) -> Vec<[u8; 16]> {
self.pks.clone()
}
fn media_keys(&self) -> Vec<[u8; 16]> {
self.mks.clone()
}
fn lookup_disc_by_hash(&self, _h: &[u8; 20]) -> Option<DiscEntry> {
self.hash_hit.clone()
}
fn lookup_disc_by_vid(&self, _v: &[u8; 16]) -> Option<DiscEntry> {
self.vid_hit.clone()
}
}
// ── KeyProvider default methods all return empty ───────────────────────
#[test]
fn default_provider_methods_return_empty() {
// A bare provider that overrides nothing must yield empty material so
// the resolver simply finds nothing through it (no surprise hits).
struct Empty;
impl KeyProvider for Empty {}
let e = Empty;
assert!(e.device_keys().is_empty());
assert!(e.processing_keys().is_empty());
assert!(e.media_keys().is_empty());
assert!(e.host_certs().is_empty());
assert!(e.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(e.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
// ── Providers::processing_keys: union + dedup ──────────────────────────
#[test]
fn providers_processing_keys_union_and_dedup() {
// Two providers each carrying overlapping PKs → the aggregate is the
// deduped union (the resolver must not re-validate identical material).
let a = Fixed {
pks: vec![[0x01u8; 16], [0x02u8; 16]],
..Default::default()
};
let b = Fixed {
pks: vec![[0x02u8; 16], [0x03u8; 16]],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let mut got = Providers(arr).processing_keys();
got.sort();
assert_eq!(got, vec![[0x01u8; 16], [0x02u8; 16], [0x03u8; 16]]);
}
#[test]
fn providers_media_keys_union_and_dedup() {
let a = Fixed {
mks: vec![[0xAAu8; 16]],
..Default::default()
};
let b = Fixed {
mks: vec![[0xAAu8; 16], [0xBBu8; 16]],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let mut got = Providers(arr).media_keys();
got.sort();
assert_eq!(got, vec![[0xAAu8; 16], [0xBBu8; 16]]);
}
#[test]
fn providers_device_keys_dedup_on_value_tuple() {
// DeviceKey has no Hash/Ord; dedup keys on (key,node,uv,u_mask_shift).
// Two identical DKs across providers collapse to one; a DK differing
// only in node is kept.
let a = Fixed {
dks: vec![dk(0x11, 5), dk(0x11, 5)],
..Default::default()
};
let b = Fixed {
dks: vec![dk(0x11, 5), dk(0x11, 6)],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).device_keys();
assert_eq!(got.len(), 2, "identical DKs dedup; differing node kept");
let nodes: Vec<u16> = got.iter().map(|d| d.node).collect();
assert!(nodes.contains(&5) && nodes.contains(&6));
}
// ── Disc-keyed lookups: array-order short-circuit ──────────────────────
#[test]
fn providers_lookup_by_hash_first_hit_wins() {
// Querying providers in array order, the FIRST hit wins (closest /
// fastest first). Provider 0 hits → its entry is returned even though
// provider 1 also has one.
let a = Fixed {
hash_hit: Some(entry("first", 0x01)),
..Default::default()
};
let b = Fixed {
hash_hit: Some(entry("second", 0x02)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_hash(&[0u8; 20]).unwrap();
assert_eq!(got.disc_hash, "first");
assert_eq!(got.vuk, Some([0x01u8; 16]));
}
#[test]
fn providers_lookup_by_hash_falls_through_to_later_provider() {
// Provider 0 misses, provider 1 hits → the later provider's entry is
// used (find_map continues past None).
let a = Fixed::default(); // hash_hit None
let b = Fixed {
hash_hit: Some(entry("second", 0x02)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_hash(&[0u8; 20]).unwrap();
assert_eq!(got.disc_hash, "second");
}
#[test]
fn providers_lookup_by_vid_first_hit_wins() {
let a = Fixed {
vid_hit: Some(entry("vid-a", 0x07)),
..Default::default()
};
let b = Fixed {
vid_hit: Some(entry("vid-b", 0x08)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_vid(&[0u8; 16]).unwrap();
assert_eq!(got.disc_hash, "vid-a");
}
#[test]
fn providers_empty_array_yields_nothing() {
let arr: &[&dyn KeyProvider] = &[];
let p = Providers(arr);
assert!(p.device_keys().is_empty());
assert!(p.processing_keys().is_empty());
assert!(p.media_keys().is_empty());
assert!(p.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(p.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
// ── SuppliedKey: each level exposes only its own material ──────────────
#[test]
fn supplied_key_exposes_only_populated_fields() {
// A SuppliedKey filled at the DK level exposes DKs and nothing else,
// so the resolver runs the matching (DK→…) path and no other.
let sk = SuppliedKey {
device_keys: vec![dk(0x33, 9)],
processing_keys: Vec::new(),
media_keys: Vec::new(),
disc_entry: None,
};
assert_eq!(sk.device_keys().len(), 1);
assert!(sk.processing_keys().is_empty());
assert!(sk.media_keys().is_empty());
assert!(sk.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(sk.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
#[test]
fn supplied_key_disc_entry_returned_for_any_hash_or_vid() {
// decrypt_with already knows the disc, so a present disc_entry is
// returned regardless of the hash/VID argument (the lookup args are
// irrelevant in this bridge).
let sk = SuppliedKey {
device_keys: Vec::new(),
processing_keys: Vec::new(),
media_keys: Vec::new(),
disc_entry: Some(entry("supplied", 0x44)),
};
// Two unrelated hashes both return the same entry.
let h1 = sk.lookup_disc_by_hash(&[0x01u8; 20]).unwrap();
let h2 = sk.lookup_disc_by_hash(&[0xFFu8; 20]).unwrap();
assert_eq!(h1.disc_hash, "supplied");
assert_eq!(h2.disc_hash, "supplied");
// And by VID likewise.
assert!(sk.lookup_disc_by_vid(&[0x00u8; 16]).is_some());
}
}
+144
View File
@@ -0,0 +1,144 @@
//! Structured resolution trace — what the unlock + key-resolution attempt did.
//!
//! No user-facing English. Every step's STATE is a typed enum variant;
//! applications RENDER these into localized text (the library never does). This
//! module only DEFINES the shape and is wired through the resolve/handshake
//! return path far enough to compile.
//!
//! The `who` of each step is the source's `label()` / unlocker's `name()` — a
//! stable identifier string (a NAME, like a codec id, NOT user-facing prose),
//! carried verbatim so an app renderer never has to match an enum back to a name
//! it already has. Only the OUTCOME / path enums are structured states the app
//! maps to i18n English.
/// The full trace of a resolution attempt: the unlock phase, then the
/// key-resolution phase.
#[derive(Debug, Clone, PartialEq, Default)]
pub struct ResolutionTrace {
/// One step per unlocker consulted, in consultation order.
pub unlock: Vec<UnlockStep>,
/// One step per key source consulted, in consultation order.
pub keys: Vec<KeyStep>,
}
impl ResolutionTrace {
/// An empty trace (no steps recorded).
pub fn new() -> Self {
Self::default()
}
}
// ── Unlock phase ────────────────────────────────────────────────────────────
/// One unlocker's contribution to the unlock phase. `who` is the unlocker's
/// `name()` (a stable, product-neutral identifier), carried verbatim.
#[derive(Debug, Clone, PartialEq)]
pub struct UnlockStep {
pub who: String,
pub outcome: UnlockOutcome,
}
/// What an unlocker did.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum UnlockOutcome {
/// The drive was unlocked (or already usable) and a VID is available.
Unlocked,
/// This unlocker cannot unlock this drive's firmware.
FirmwareNotUnlockable,
/// No non-revoked host cert was usable for the auth attempt. `mkb` is the
/// disc MKB generation when known.
NoUsableHostCert { mkb: Option<u32> },
/// Every available host cert was revoked on this drive's HRL. `mkb` is the
/// disc MKB generation when known.
CertRevoked { mkb: Option<u32> },
/// The drive rejected the auth handshake (non-revocation rejection / wedge).
HandshakeRejected,
/// Auth succeeded (or was skipped) but the Volume ID could not be read.
VidUnavailable,
}
// ── Key-resolution phase ────────────────────────────────────────────────────
/// One key source's contribution to the key-resolution phase, including the
/// derivation path it walked. `who` is the source's `label()` (a stable
/// identifier, e.g. `"keydb"` / `"online"`), carried verbatim.
#[derive(Debug, Clone, PartialEq)]
pub struct KeyStep {
pub who: String,
pub path: Vec<KeyNode>,
pub outcome: KeyOutcome,
}
/// A node on the derivation path a source walked. Ordered as encountered; not
/// every path hits every node.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum KeyNode {
/// The source matched this disc (by hash / VID).
MatchedDisc,
/// The source had no entry for this disc.
NoEntry,
/// Pre-decrypted unit keys were found.
FoundUnitKeys,
/// A VUK was found.
FoundVuk,
/// A Media Key was found.
FoundMediaKey,
/// A VID is required to proceed.
NeedVid,
/// The VID came from the unlock phase.
VidFromUnlock,
/// The VID came from the keydb entry.
VidFromKeydb,
/// No VID was available.
NoVid,
/// A VUK was derived (from MK + VID).
DerivedVuk,
/// Unit keys were derived (from VUK).
DerivedUnitKeys,
}
/// The terminal outcome of a source's resolution attempt.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum KeyOutcome {
/// Usable unit keys were produced.
Resolved,
/// Derivation material existed but no VID was available to finish.
MissingVid,
/// No usable key from this source.
NoKey,
}
#[cfg(test)]
mod tests {
use super::*;
/// The trace types are constructible, derive the required traits, and an
/// empty trace round-trips. Pins the structural contract apps build against.
#[test]
fn trace_is_constructible_and_comparable() {
let t = ResolutionTrace {
unlock: vec![UnlockStep {
who: "AACS cert".to_string(),
outcome: UnlockOutcome::NoUsableHostCert { mkb: Some(68) },
}],
keys: vec![KeyStep {
who: "keydb".to_string(),
path: vec![
KeyNode::MatchedDisc,
KeyNode::FoundVuk,
KeyNode::DerivedUnitKeys,
],
outcome: KeyOutcome::Resolved,
}],
};
// Clone + PartialEq (derive contract the renderers rely on).
assert_eq!(t.clone(), t);
// `who` is the source's name carried verbatim.
assert_eq!(t.keys[0].who, "keydb");
assert_eq!(t.unlock[0].who, "AACS cert");
// Default / new is empty.
assert_eq!(ResolutionTrace::new(), ResolutionTrace::default());
assert!(ResolutionTrace::new().unlock.is_empty());
assert!(ResolutionTrace::new().keys.is_empty());
}
}
+45
View File
@@ -0,0 +1,45 @@
//! AACS primitive types shared across the resolve chain.
//!
//! These structs describe AACS key material (device keys, host
//! certificates, per-disc entries). They carry no parsing logic — the
//! keydb.cfg format lives in the `freemkv-keysources` crate. libfreemkv
//! owns only the crypto and these value types that flow through it.
/// A device key for MKB subset-difference tree processing.
#[derive(Debug, Clone)]
pub struct DeviceKey {
pub key: [u8; 16],
pub node: u16,
pub uv: u32,
pub u_mask_shift: u8,
}
/// Host certificate + private key for AACS SCSI authentication.
#[derive(Debug, Clone)]
pub struct HostCert {
/// AACS 1.0: 20 bytes. AACS 2.0: 32 bytes.
pub private_key: [u8; 20],
/// AACS 1.0: 92 bytes. AACS 2.0: 132 bytes.
pub certificate: Vec<u8>,
/// AACS 2.0 host private key (P-256, 32 bytes). None for AACS 1.0 only.
pub private_key_v2: Option<[u8; 32]>,
/// AACS 2.0 host certificate (type 0x11). None for AACS 1.0 only.
pub certificate_v2: Option<Vec<u8>>,
}
/// A per-disc entry from the key database.
#[derive(Debug, Clone)]
pub struct DiscEntry {
/// Disc hash (20 bytes, hex)
pub disc_hash: String,
/// Disc title
pub title: String,
/// Media Key (16 bytes) — from MKB processing
pub media_key: Option<[u8; 16]>,
/// Disc ID (16 bytes)
pub disc_id: Option<[u8; 16]>,
/// Volume Unique Key (16 bytes) — decrypts title keys
pub vuk: Option<[u8; 16]>,
/// Unit keys (title keys) indexed by CPS unit number
pub unit_keys: Vec<(u32, [u8; 16])>,
}
+1045
View File
File diff suppressed because it is too large Load Diff
-105
View File
@@ -1,105 +0,0 @@
//! aacs-test — Test AACS handshake against a real drive.
//!
//! Usage: aacs-test /dev/sr0 /path/to/keydb.cfg
use std::env;
use std::path::Path;
fn main() {
let args: Vec<String> = env::args().collect();
if args.len() < 3 {
eprintln!("Usage: aacs-test <device> <keydb_path>");
std::process::exit(1);
}
let device = Path::new(&args[1]);
let keydb_path = Path::new(&args[2]);
println!("aacs-test v{}", env!("CARGO_PKG_VERSION"));
println!();
// Open drive WITHOUT unlock — AACS auth must happen before raw mode
print!("Opening {} (no unlock)... ", device.display());
let mut session = match libfreemkv::DriveSession::open_no_unlock(device) {
Ok(s) => { println!("OK"); s }
Err(e) => { println!("FAILED: {}", e); std::process::exit(1); }
};
println!(" Drive: {} {}", session.profile.drive_id.trim(), session.profile.chipset.name());
// Load KEYDB
print!("Loading KEYDB... ");
let keydb = match libfreemkv::aacs::KeyDb::load(keydb_path) {
Ok(db) => {
println!("OK ({} disc entries, {} DK, {} PK)",
db.disc_entries.len(), db.device_keys.len(), db.processing_keys.len());
db
}
Err(e) => { println!("FAILED: {}", e); std::process::exit(1); }
};
let host_cert = match &keydb.host_cert {
Some(hc) => {
println!(" Host cert: {} bytes, priv_key[0]=0x{:02x}",
hc.certificate.len(), hc.private_key[0]);
hc
}
None => { println!(" No host cert in KEYDB"); std::process::exit(1); }
};
// AACS handshake
println!();
print!("AACS authenticate... ");
let mut auth = match libfreemkv::aacs::handshake::aacs_authenticate(
&mut session,
&host_cert.private_key,
&host_cert.certificate,
) {
Ok(a) => {
println!("OK");
println!(" Bus key: {:02x?}", &a.bus_key);
println!(" AGID: {}", a.agid);
println!(" Drive cert type: 0x{:02x}", a.drive_cert[0]);
a
}
Err(e) => {
println!("FAILED: {}", e);
std::process::exit(1);
}
};
// Read Volume ID
print!("Reading Volume ID... ");
match libfreemkv::aacs::handshake::read_volume_id(&mut session, &mut auth) {
Ok(vid) => {
println!("OK");
println!(" VID: {:02x?}", vid);
// Try to find matching disc in KEYDB
let matched = keydb.disc_entries.values()
.find(|e| e.disc_id == Some(vid));
if let Some(entry) = matched {
println!(" KEYDB match: {} (hash {})", entry.title, entry.disc_hash);
if let Some(vuk) = entry.vuk {
println!(" VUK: {:02x?}", vuk);
}
} else {
println!(" No exact VID match in KEYDB");
}
}
Err(e) => println!("FAILED: {}", e),
}
// Read data keys (AACS 2.0)
print!("Reading data keys... ");
match libfreemkv::aacs::handshake::read_data_keys(&mut session, &mut auth) {
Ok((rdk, wdk)) => {
println!("OK (AACS 2.0 bus encryption)");
println!(" Read data key: {:02x?}", rdk);
println!(" Write data key: {:02x?}", wdk);
}
Err(e) => println!("not available: {} (likely AACS 1.0)", e),
}
println!();
println!("Done.");
}
-159
View File
@@ -1,159 +0,0 @@
//! freemkv-info — Drive identification and compatibility checker.
//!
//! Sends standard SCSI INQUIRY and GET CONFIGURATION commands to an optical drive,
//! displays drive identity and compatibility status, and optionally outputs raw
//! response data for profile contribution.
//!
//! Usage:
//! freemkv-info /dev/sr0
//! freemkv-info /dev/sr0 --raw
//! freemkv-info /dev/sr0 --json
use std::env;
use std::path::Path;
use std::process;
fn main() {
let args: Vec<String> = env::args().collect();
if args.len() < 2 {
eprintln!("freemkv-info — Drive identification and compatibility checker");
eprintln!();
eprintln!("Usage: freemkv-info <device> [options]");
eprintln!();
eprintln!(" <device> Optical drive device (e.g. /dev/sr0)");
eprintln!(" --raw Output raw SCSI response hex (for profile contribution)");
eprintln!(" --json Output machine-readable JSON");
eprintln!(" --profiles Path to profiles directory (default: ./profiles)");
eprintln!();
eprintln!("Examples:");
eprintln!(" freemkv-info /dev/sr0");
eprintln!(" freemkv-info /dev/sr0 --raw > my_drive.txt");
process::exit(1);
}
let device = Path::new(&args[1]);
let raw_mode = args.iter().any(|a| a == "--raw");
let json_mode = args.iter().any(|a| a == "--json");
let profiles_dir = args.iter()
.position(|a| a == "--profiles")
.and_then(|i| args.get(i + 1))
.map(|s| s.as_str())
.unwrap_or("profiles");
// Open SCSI transport
let mut transport = match libfreemkv::scsi::open(device) {
Ok(t) => t,
Err(e) => {
eprintln!("Error: Cannot open {}: {}", device.display(), e);
process::exit(1);
}
};
// INQUIRY
let inquiry = match libfreemkv::scsi::inquiry(transport.as_mut()) {
Ok(i) => i,
Err(e) => {
eprintln!("Error: INQUIRY failed: {}", e);
process::exit(1);
}
};
// GET CONFIGURATION feature 0x010C
let gc_010c = libfreemkv::scsi::get_config_010c(transport.as_mut()).ok();
if json_mode {
print_json(&inquiry, &gc_010c);
} else if raw_mode {
print_raw(&inquiry, &gc_010c);
} else {
print_human(&inquiry, &gc_010c, profiles_dir);
}
}
fn print_human(
inquiry: &libfreemkv::scsi::InquiryResult,
gc_010c: &Option<Vec<u8>>,
profiles_dir: &str,
) {
println!("freemkv-info v{}", env!("CARGO_PKG_VERSION"));
println!();
println!("Drive: {} {} {}", inquiry.vendor_id, inquiry.model, inquiry.firmware);
println!("INQUIRY: additional_length=0x{:02X} ({})",
inquiry.raw.get(4).unwrap_or(&0),
inquiry.raw.get(4).unwrap_or(&0));
if let Some(gc) = gc_010c {
let data_hex: String = gc.iter().map(|b| format!("{:02x}", b)).collect();
println!("Feature 0x010C: {}", data_hex);
} else {
println!("Feature 0x010C: not available");
}
// Try to match profile
if let Ok(profiles) = libfreemkv::profile::load_all(Path::new(profiles_dir)) {
let matched = profiles.iter().find(|p| {
p.drive_id.contains(&inquiry.vendor_id)
&& p.drive_id.contains(&inquiry.model)
});
println!();
match matched {
Some(p) => {
println!("Profile: FOUND ({})", p.chipset.name());
println!("Raw Read: Supported");
}
None => {
println!("Profile: NOT FOUND");
println!("Raw Read: Unknown — run with --raw and submit a profile request");
}
}
} else {
println!();
println!("Profile: No profiles directory found at '{}'", profiles_dir);
}
}
fn print_raw(
inquiry: &libfreemkv::scsi::InquiryResult,
gc_010c: &Option<Vec<u8>>,
) {
println!("# freemkv-info raw output");
println!("# Submit this file to https://github.com/freemkv/libfreemkv/issues");
println!();
println!("vendor: {}", inquiry.vendor_id);
println!("model: {}", inquiry.model);
println!("firmware: {}", inquiry.firmware);
println!();
// Full INQUIRY hex
println!("inquiry_hex: {}", hex_encode(&inquiry.raw));
println!("inquiry_length: {}", inquiry.raw.len());
// GET CONFIG 0x010C
if let Some(gc) = gc_010c {
println!("get_config_010c_hex: {}", hex_encode(gc));
println!("get_config_010c_length: {}", gc.len());
} else {
println!("get_config_010c_hex: ERROR");
}
}
fn print_json(
inquiry: &libfreemkv::scsi::InquiryResult,
gc_010c: &Option<Vec<u8>>,
) {
let json = serde_json::json!({
"vendor": inquiry.vendor_id,
"model": inquiry.model,
"firmware": inquiry.firmware,
"inquiry_hex": hex_encode(&inquiry.raw),
"inquiry_length": inquiry.raw.len(),
"get_config_010c_hex": gc_010c.as_ref().map(|g| hex_encode(g)),
});
println!("{}", serde_json::to_string_pretty(&json).unwrap());
}
fn hex_encode(data: &[u8]) -> String {
data.iter().map(|b| format!("{:02x}", b)).collect()
}
-99
View File
@@ -1,99 +0,0 @@
//! freemkv-test — Quick verification that raw disc access works.
//!
//! Enables raw read mode, calibrates speed, reads a few test sectors.
//! Use this to verify your drive and profile are working correctly.
//!
//! Usage:
//! freemkv-test /dev/sr0
//! freemkv-test /dev/sr0 --profiles ./profiles
use std::env;
use std::path::Path;
use std::process;
fn main() {
let args: Vec<String> = env::args().collect();
if args.len() < 2 {
eprintln!("freemkv-test — Verify raw disc access works");
eprintln!();
eprintln!("Usage: freemkv-test <device> [--profiles <dir>]");
process::exit(1);
}
let device = Path::new(&args[1]);
println!("freemkv-test v{}", env!("CARGO_PKG_VERSION"));
println!();
// Open drive session (uses bundled profiles)
print!("Opening {}... ", device.display());
let mut session = match libfreemkv::DriveSession::open(device) {
Ok(s) => { println!("OK"); s }
Err(e) => { println!("FAILED: {}", e); process::exit(1); }
};
println!(" Drive ID: {}", session.profile.drive_id);
println!(" Chipset: {}", session.profile.chipset.name());
println!();
// Enable raw read mode
print!("Unlocking drive... ");
match session.unlock() {
Ok(()) => println!("OK"),
Err(e) => { println!("FAILED: {}", e); process::exit(1); }
}
// Check status
print!("Checking status... ");
match session.status() {
Ok(status) => {
if status.unlocked {
println!("OK (active)");
} else {
println!("WARNING: drive reported as locked");
}
}
Err(e) => println!("SKIP ({})", e),
}
// Calibrate speed
print!("Calibrating speed... ");
match session.calibrate() {
Ok(()) => println!("OK"),
Err(e) => println!("SKIP ({})", e),
}
// Read test sectors
let test_lbas: &[u32] = &[0, 100, 1000, 10000];
let mut buf = vec![0u8; 2048];
let mut pass = 0;
let mut fail = 0;
for &lba in test_lbas {
print!("Reading sector {}... ", lba);
match session.read_sectors(lba, 1, &mut buf) {
Ok(n) if n == 2048 => {
let nonzero = buf.iter().filter(|&&b| b != 0).count();
println!("OK ({} bytes, {} non-zero)", n, nonzero);
pass += 1;
}
Ok(n) => {
println!("PARTIAL ({} bytes)", n);
fail += 1;
}
Err(e) => {
println!("FAILED: {}", e);
fail += 1;
}
}
}
println!();
if fail == 0 {
println!("All {} checks passed. Drive is fully functional.", pass);
} else {
println!("{} passed, {} failed.", pass, fail);
process::exit(1);
}
}
+1068 -37
View File
File diff suppressed because it is too large Load Diff
+125
View File
@@ -0,0 +1,125 @@
//! Physical media constants — the single source of truth.
//!
//! Naming convention: a constant is prefixed by the **narrowest scope where it
//! is valid**. A value common to all optical media carries no prefix; a value
//! specific to a container/format/disc-type is prefixed by it
//! (`TS_`, `BD_`, …). Define each physical quantity here exactly once and import
//! it — never re-declare a bare literal or a local copy.
/// Bytes per logical sector on every optical medium freemkv reads
/// (Blu-ray, DVD-Video, CD-ROM Mode 1). Universal — hence unprefixed.
///
/// `usize` because its dominant use is buffer sizing and slice indexing, where
/// Rust *requires* `usize` (`vec![0u8; SECTOR_BYTES]`, `buf.len() < SECTOR_BYTES`).
/// For byte-offset / capacity arithmetic — which is `u64` because a disc can
/// exceed 4 GiB — use [`SECTOR_BYTES_U64`] instead of casting at each site.
pub const SECTOR_BYTES: usize = 2048;
/// [`SECTOR_BYTES`] as `u64`, for byte-offset and capacity arithmetic. The
/// single `usize → u64` boundary cast lives here, once, so offset math across
/// the workspace reads as `sectors * SECTOR_BYTES_U64` with no per-site cast.
pub const SECTOR_BYTES_U64: u64 = SECTOR_BYTES as u64;
/// Milliseconds per second. For turning a byte count ÷ bytes-per-second into a
/// movie-time figure (`bytes / bps * MILLIS_PER_SEC`) without a bare `1000.0`.
pub const MILLIS_PER_SEC: f64 = 1_000.0;
/// Bytes per MPEG-2 transport-stream packet. Common to all MPEG-TS, not just
/// Blu-ray — prefixed by the format, not a disc type.
pub const TS_PACKET_BYTES: usize = 188;
/// Bytes in an MPEG-2 transport-stream packet header: sync byte, the
/// flags/PID word, and the adaptation/continuity byte.
pub const TS_HEADER_BYTES: usize = 4;
/// Bytes in the arrival-timestamp prefix a Blu-ray M2TS prepends to each TS
/// packet to form a source packet. Same width as a TS header but a distinct
/// quantity ([`TS_HEADER_BYTES`]) — do not conflate.
pub const BD_TIMESTAMP_PREFIX_BYTES: usize = 4;
/// Bytes of payload in an MPEG-2 transport-stream packet:
/// [`TS_PACKET_BYTES`] minus the [`TS_HEADER_BYTES`] header.
pub const TS_PAYLOAD_BYTES: usize = TS_PACKET_BYTES - TS_HEADER_BYTES;
/// Bytes per Blu-ray M2TS *source packet*: a TS packet ([`TS_PACKET_BYTES`])
/// prefixed with the [`BD_TIMESTAMP_PREFIX_BYTES`] arrival-timestamp header.
/// A BDAV/M2TS construct only — DVD VOBs have no source packets — hence `BD_`.
pub const BD_SOURCE_PACKET_BYTES: usize = TS_PACKET_BYTES + BD_TIMESTAMP_PREFIX_BYTES;
/// Elementary-stream coding-type codes — the single source of truth for the
/// byte that identifies a stream's codec.
///
/// This is one registry used in two places that share the same value space:
/// the MPEG-TS PMT `stream_type` (ISO/IEC 13818-1 Table 2-34) and the Blu-ray
/// STN/CLPI `stream_coding_type` (BD-ROM Part 3). The standardized video codes
/// (`0x02`, `0x1B`, `0x24`) are ISO assignments (ISO/IEC 13818-1 Table 2-34);
/// `0xEA` (VC-1) is a BD-ROM convention in the ISO user-private range. The
/// `0x80..=0xA2` audio/graphics codes also sit in the user-private range and follow the
/// Blu-ray Disc Association / ATSC A/52 convention. Because every consumer
/// reads or writes this single byte, the family is unprefixed — the scope is
/// "any elementary stream freemkv parses or muxes".
///
/// Each constant is `u8`: the spec defines an 8-bit field and the code compares
/// it directly against a byte read from the buffer, so no casts are needed.
pub mod coding_type {
/// MPEG-2 video (ISO/IEC 13818-1 Table 2-34).
pub const MPEG2_VIDEO: u8 = 0x02;
/// H.264 / AVC video (ISO/IEC 13818-1 Table 2-34).
pub const H264: u8 = 0x1B;
/// HEVC / H.265 video (ISO/IEC 13818-1 Table 2-34, 2015 amendment).
pub const HEVC: u8 = 0x24;
/// SMPTE VC-1 video (BD-ROM convention, ISO user-private range).
pub const VC1: u8 = 0xEA;
/// LPCM audio (BD-ROM convention).
pub const LPCM: u8 = 0x80;
/// Dolby Digital (AC-3) audio (BD-ROM / ATSC A/52 convention).
pub const AC3: u8 = 0x81;
/// DTS audio (BD-ROM convention).
pub const DTS: u8 = 0x82;
/// Dolby TrueHD audio (BD-ROM convention).
pub const TRUEHD: u8 = 0x83;
/// Dolby Digital Plus (E-AC-3 / AC-3+) audio (BD-ROM convention).
pub const AC3_PLUS: u8 = 0x84;
/// DTS-HD High Resolution audio (BD-ROM Part 3-1).
pub const DTS_HD_HR: u8 = 0x85;
/// DTS-HD Master Audio (lossless) (BD-ROM Part 3-1).
pub const DTS_HD_MA: u8 = 0x86;
/// Presentation Graphics — PG subtitle stream (BD-ROM HDMV).
pub const PG: u8 = 0x90;
/// Interactive Graphics — IG / BD-J menu overlay, NOT a subtitle (BD-ROM HDMV).
pub const IG: u8 = 0x91;
/// Text subtitle stream (BD-ROM HDMV).
pub const TEXT_SUBTITLE: u8 = 0x92;
/// Secondary Dolby Digital Plus audio (BD-ROM convention).
pub const AC3_PLUS_SECONDARY: u8 = 0xA1;
/// Secondary DTS-HD audio (lossless MA, not lossy HR) (BD-ROM convention).
pub const DTS_HD_SECONDARY: u8 = 0xA2;
}
/// MPEG PES `stream_id` codes — the byte after the `00 00 01` start-code prefix
/// that identifies an elementary stream's role in a PES packet (ISO/IEC
/// 13818-1 Table 2-22). Shared by the program-stream demuxer and the TS/M2TS
/// muxers, so defined here once. Each is `u8` (matches the byte on the wire).
pub mod pes_stream_id {
/// Video stream (`110x xxxx`; freemkv emits the base id `0xE0`).
pub const VIDEO: u8 = 0xE0;
/// private_stream_1 — AC-3 / DTS / LPCM / PGS subtitle payloads.
pub const PRIVATE_STREAM_1: u8 = 0xBD;
/// padding_stream — stuffing bytes only, no payload to demux.
pub const PADDING_STREAM: u8 = 0xBE;
/// private_stream_2 — DVD navigation (PCI/DSI); carries no muxable ES.
pub const PRIVATE_STREAM_2: u8 = 0xBF;
/// Highest video stream_id — the `110x xxxx` video range tops out at 0xEF.
pub const VIDEO_MAX: u8 = 0xEF;
/// Inclusive range of every PES `stream_id` that carries demuxable payload:
/// [`PRIVATE_STREAM_1`] (0xBD) through [`VIDEO_MAX`] (0xEF) — i.e. private
/// stream 1/2, padding, MPEG audio (0xC0-0xDF) and video (0xE0-0xEF). The
/// pack (0xBA), system-header (0xBB) and program-end (0xB9) codes sit below
/// this range and are deliberately excluded: they're structural, not ES.
pub const PAYLOAD_RANGE: core::ops::RangeInclusive<u8> = PRIVATE_STREAM_1..=VIDEO_MAX;
}
+511
View File
@@ -0,0 +1,511 @@
//! CSS cipher implementation based on the Stevenson 1999 analysis.
//!
//! The CSS cipher uses two table-driven feedback circuits:
//! - LFSR1: 17-bit state (9-bit lo + 8-bit hi register, seeded from
//! key[0..2]), driven by TAB2/TAB3
//! - LFSR0: 24-bit feedback register (seeded from key[2..5] XOR seed[2..5],
//! masked to 0xFFFFFF), driven by a feedback polynomial through TAB4
//!
//! The keystream is the bytewise sum (with carry) of both LFSR outputs.
//! Content descrambling computes plain = TAB1[cipher] ^ keystream — a TAB1
//! substitution of each ciphertext byte followed by an XOR with the keystream
//! (NOT a plain XOR; the cipher is not its own inverse).
//!
//! Algorithm: Frank A. Stevenson's divide-and-conquer attack (1999).
//! Tables: CSS specification constants.
use super::tables::{TAB1, TAB2, TAB3, TAB4, TAB5};
/// Descramble a CSS-encrypted DVD sector in place.
///
/// Exact port of libdvdcss `dvdcss_unscramble` (css.c). The two content
/// LFSRs are seeded **directly** from `title_key XOR sector_seed` — there is
/// no `decrypt_key` mangling on this path (that is the disc/title-key
/// hierarchy, not the content cipher). Bytes 0x80..0x800 are recovered with
/// `*p = TAB1[*p] ^ (i_t5 & 0xff)`.
///
/// The scramble flag at byte 0x14 (bits 4-5) indicates encryption. Like
/// libdvdcss, the flag byte is NOT modified here — the caller treats a
/// nonzero `sector[0x14] & 0x30` as "needs unscrambling" and the descramble
/// is its own inverse, so re-running it on plaintext would re-scramble.
/// (freemkv historically cleared the flag; we keep clearing it so callers
/// and the existing tests can distinguish a descrambled sector. This does
/// not affect the recovered body.)
///
/// No-op (returns without modifying `sector`) in two cases:
/// - `sector.len() < 2048`: the encrypted region (0x80..0x800) is not
/// fully present. Callers chunk by 2048, so a trailing partial chunk is
/// left untouched. The `debug_assert!` flags this misuse in debug/test
/// builds; a DVD sector is always exactly 2048 bytes.
/// - scramble flags are zero: the sector is not CSS-encrypted.
///
/// Design reference: libdvdcss `dvdcss_unscramble`. The combiner mirrors
/// `css.c` line-for-line:
/// ```text
/// i_t1 = (key[0] ^ sec[0x54]) | 0x100;
/// i_t2 = key[1] ^ sec[0x55];
/// i_t3 = (key[2]|key[3]<<8|key[4]<<16) ^ (sec[0x56]|sec[0x57]<<8|sec[0x58]<<16);
/// i_t4 = i_t3 & 7; i_t3 = i_t3*2 + 8 - i_t4;
/// // per byte over 0x80..0x800:
/// i_t4 = TAB2[i_t2] ^ TAB3[i_t1];
/// i_t2 = i_t1 >> 1; i_t1 = ((i_t1 & 1) << 8) ^ i_t4; i_t4 = TAB5[i_t4];
/// i_t6 = (((((((i_t3>>3)^i_t3)>>1)^i_t3)>>8)^i_t3)>>5) & 0xff;
/// i_t3 = (i_t3 << 8) | i_t6; i_t6 = TAB4[i_t6];
/// i_t5 += i_t6 + i_t4; *p = TAB1[*p] ^ (i_t5 & 0xff); i_t5 >>= 8;
/// ```
pub fn descramble_sector(title_key: &[u8; 5], sector: &mut [u8]) {
debug_assert!(
sector.len() >= 2048,
"descramble_sector: buffer shorter than one 2048-byte sector"
);
if sector.len() < 2048 {
return;
}
// libdvdcss: `if( !(p_sec[0x14] & 0x30) ) return;`
if sector[0x14] & 0x30 == 0 {
return;
}
// LFSR1: seeded directly from (key ^ seed) — NO decrypt_key.
let mut i_t1: u32 = ((title_key[0] ^ sector[0x54]) as u32) | 0x100;
let mut i_t2: u32 = (title_key[1] ^ sector[0x55]) as u32;
// LFSR0 (i_t3): 24-bit feedback register seeded from the remaining three
// key/seed bytes, then transformed `i_t3 = i_t3*2 + 8 - (i_t3 & 7)`.
let mut i_t3: u32 = (((title_key[2] as u32)
| ((title_key[3] as u32) << 8)
| ((title_key[4] as u32) << 16))
^ ((sector[0x56] as u32) | ((sector[0x57] as u32) << 8) | ((sector[0x58] as u32) << 16)))
& 0xFF_FFFF;
let i_t4_seed = i_t3 & 7;
i_t3 = i_t3 * 2 + 8 - i_t4_seed;
let mut i_t5: u32 = 0;
for byte in sector.iter_mut().take(2048).skip(128) {
// Advance LFSR1.
let mut i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
i_t4 = TAB5[i_t4 as usize] as u32;
// Advance LFSR0 (i_t3) and fold both outputs into i_t5.
let mut i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xFF;
i_t3 = (i_t3 << 8) | i_t6;
i_t6 = TAB4[i_t6 as usize] as u32;
i_t5 += i_t6 + i_t4;
*byte = TAB1[*byte as usize] ^ (i_t5 & 0xFF) as u8;
i_t5 >>= 8;
}
// libdvdcss leaves byte 0x14 untouched; freemkv clears the scramble bits
// so downstream code and tests can tell a sector was descrambled.
sector[0x14] &= 0xCF;
}
/// Exact inverse of [`descramble_sector`]: turn a plaintext sector body into
/// CSS ciphertext under `title_key`.
///
/// Descramble computes `plain = TAB1[cipher] ^ (i_t5 & 0xff)`, so the
/// inverse is `cipher = TAB1_INV[plain ^ (i_t5 & 0xff)]` with the identical
/// LFSR keystream. The keystream derivation is byte-for-byte the same as
/// `descramble_sector` (libdvdcss `dvdcss_unscramble`); only the final
/// substitution differs. Bytes 0x80..0x800 are rewritten in place; the
/// scramble flag is set to 0x10 so a subsequent descramble runs.
///
/// Not on any production read path — it exists so the key-recovery tests
/// (and any caller that needs to produce a known CSS-encrypted sector) can
/// build genuine ciphertext rather than approximating it.
#[cfg(test)]
pub(crate) fn scramble_sector(title_key: &[u8; 5], sector: &mut [u8]) {
if sector.len() < 2048 {
return;
}
let mut i_t1: u32 = ((title_key[0] ^ sector[0x54]) as u32) | 0x100;
let mut i_t2: u32 = (title_key[1] ^ sector[0x55]) as u32;
let mut i_t3: u32 = (((title_key[2] as u32)
| ((title_key[3] as u32) << 8)
| ((title_key[4] as u32) << 16))
^ ((sector[0x56] as u32) | ((sector[0x57] as u32) << 8) | ((sector[0x58] as u32) << 16)))
& 0xFF_FFFF;
let i_t4_seed = i_t3 & 7;
i_t3 = i_t3 * 2 + 8 - i_t4_seed;
let mut i_t5: u32 = 0;
for byte in sector.iter_mut().take(2048).skip(128) {
let mut i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
i_t4 = TAB5[i_t4 as usize] as u32;
let mut i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xFF;
i_t3 = (i_t3 << 8) | i_t6;
i_t6 = TAB4[i_t6 as usize] as u32;
i_t5 += i_t6 + i_t4;
// Inverse of `*p = TAB1[*p] ^ ks`: apply ks then TAB1's inverse.
*byte = (*TAB1_INV)[(*byte ^ (i_t5 & 0xFF) as u8) as usize];
i_t5 >>= 8;
}
// Mark the sector scrambled so the descrambler will process it.
sector[0x14] = (sector[0x14] & 0xCF) | 0x10;
}
/// Inverse permutation of [`TAB1`], built at first use. `TAB1` is a
/// bijection on 0..256, so `TAB1_INV[TAB1[x]] == x`.
#[cfg(test)]
static TAB1_INV: std::sync::LazyLock<[u8; 256]> = std::sync::LazyLock::new(|| {
let mut inv = [0u8; 256];
for (i, &v) in TAB1.iter().enumerate() {
inv[v as usize] = i as u8;
}
inv
});
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn descramble_skips_unscrambled() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let mut sector = vec![0xAA; 2048];
sector[0x14] = 0x00;
let original = sector.clone();
descramble_sector(&key, &mut sector);
assert_eq!(sector, original);
}
/// Cross-check `descramble_sector` against the EXACT output of libdvdcss
/// `dvdcss_unscramble` (css.c) for a fixed sector, computed from the
/// reference C semantics with the reference tables. Pins the content
/// cipher to libdvdcss byte-for-byte.
///
/// key = 42 13 37 BE EF, seed (0x54..0x59) = DE AD BE EF 42, body = 0xAA.
#[test]
fn descramble_matches_libdvdcss_unscramble_vector() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0xAAu8; 2048];
sector[0x14] = 0x30;
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
descramble_sector(&key, &mut sector);
assert_eq!(
&sector[0x80..0x90],
&[
0x81, 0x92, 0x24, 0xA2, 0x46, 0x70, 0x3C, 0x64, 0xA6, 0x91, 0x84, 0xF5, 0x1F, 0x98,
0xA0, 0x31
],
"descramble body head must match libdvdcss dvdcss_unscramble"
);
assert_eq!(
&sector[0x7F8..0x800],
&[0x46, 0x94, 0x80, 0x0E, 0x67, 0x36, 0x65, 0xBC],
"descramble body tail must match libdvdcss dvdcss_unscramble"
);
}
#[test]
fn descramble_modifies_scrambled() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let mut sector = vec![0xAA; 2048];
sector[0x14] = 0x30; // scramble flag set
// Set a sector seed
sector[0x54..0x59].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
let original = sector.clone();
descramble_sector(&key, &mut sector);
// Header (0..128) unchanged except byte 0x14 (flag cleared)
for i in 0..128 {
if i == 0x14 {
continue;
}
assert_eq!(sector[i], original[i], "header byte {} changed", i);
}
// Encrypted region should be different
assert_ne!(&sector[128..256], &original[128..256]);
}
#[test]
fn descramble_clears_flags() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let mut sector = vec![0x00; 2048];
sector[0x14] = 0x30;
sector[0x54..0x59].copy_from_slice(&[0x00; 5]);
descramble_sector(&key, &mut sector);
assert_eq!(sector[0x14] & 0x30, 0x00);
}
/// Test 2: descramble inverts scramble over the body.
///
/// The content cipher is NOT a plain XOR involution (it applies TAB1 to
/// the ciphertext: `plain = TAB1[cipher] ^ ks`). The true inverse is
/// [`scramble_sector`]. Scrambling a plaintext body and then descrambling
/// with the same key must reproduce the original body exactly.
#[test]
fn css_descramble_inverts_scramble_over_body() {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0xAAu8; 2048];
sector[0x14] = 0x30; // scramble flag
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
let original = sector.clone();
// Scramble the plaintext body into ciphertext.
scramble_sector(&title_key, &mut sector);
// Header (0..128) unchanged except the flag byte (set by scramble).
for i in 0..128 {
if i == 0x14 {
continue;
}
assert_eq!(sector[i], original[i], "header byte {} changed", i);
}
// Encrypted region modified
assert_ne!(&sector[128..256], &original[128..256]);
// Descramble restores the plaintext body byte-for-byte.
descramble_sector(&title_key, &mut sector);
assert_eq!(sector[0x14] & 0x30, 0x00, "flag cleared after descramble");
assert_eq!(
&sector[128..2048],
&original[128..2048],
"descramble(scramble(body)) did not restore the body"
);
}
/// css_tab1_relationship
///
/// Verify the structure of TAB1: it is a substitution table used in
/// key mangling. Check that no two inputs map to the same output
/// (TAB1 is a permutation of 0..255).
#[test]
fn css_tab1_is_permutation() {
let mut seen = [false; 256];
for tab1_val in &TAB1 {
let v = *tab1_val as usize;
assert!(!seen[v], "TAB1 maps two inputs to {:#04x}", v);
seen[v] = true;
}
// Check involution property: TAB1[TAB1[x]] should map back predictably
// TAB1 is not necessarily a strict involution, but we verify the
// composition TAB1[TAB1[x]] is also a permutation
let mut seen2 = [false; 256];
for i in 0..256 {
let v = TAB1[TAB1[i] as usize] as usize;
assert!(!seen2[v], "TAB1[TAB1[x]] maps two inputs to {:#04x}", v);
seen2[v] = true;
}
}
/// css_tab4_is_bit_reversal
///
/// TAB4 reverses the bits of each byte: TAB4[0x01] = 0x80, TAB4[0x80] = 0x01, etc.
#[test]
fn css_tab4_is_bit_reversal() {
for i in 0u16..256 {
let expected = (0..8).fold(0u8, |acc, bit| acc | (((i as u8 >> bit) & 1) << (7 - bit)));
assert_eq!(
TAB4[i as usize], expected,
"TAB4[{:#04x}] = {:#04x}, expected {:#04x} (bit reversal)",
i, TAB4[i as usize], expected
);
}
// Also verify TAB4 is an involution: TAB4[TAB4[x]] == x
for i in 0..256 {
assert_eq!(
TAB4[TAB4[i] as usize], i as u8,
"TAB4 is not an involution at {:#04x}",
i
);
}
}
// ── scramble-flag detection (byte 0x14, bits 4-5) ──────────────────────
/// Only bits 4-5 of byte 0x14 are the CSS scramble flag: the code reads
/// `sector[0x14] & 0x30 == 0` (bits 6-7, i.e. 0x40/0x80, are masked out by
/// 0x30). A sector with 0x14 == 0x40 or 0x80 must therefore be treated as
/// UNSCRAMBLED and left byte-for-byte unchanged. This guards against a
/// too-wide mask silently "descrambling" (and thus corrupting) clear data.
///
/// Grounding: CSS sector header byte 0x14 — copyright/scramble bits live
/// in bits 4-5; the masked value 0 means not scrambled.
/// Mutation: widen the mask `0x30` to `0x70`/`0xF0` -> 0x40/0x80 would be
/// seen as scrambled and the body would change.
#[test]
fn descramble_treats_high_bits_of_0x14_as_clear() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
for &flag in &[0x40u8, 0x80, 0xC0, 0x0F, 0x4F, 0x8F] {
let mut sector = vec![0xAA; 2048];
sector[0x14] = flag;
sector[0x54..0x59].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
let original = sector.clone();
descramble_sector(&key, &mut sector);
assert_eq!(
sector, original,
"byte 0x14 = {flag:#04x} has flag bits 4-5 clear; sector must be untouched"
);
}
}
/// Each individual scramble bit (4 and 5) independently marks the sector
/// as encrypted: 0x10 and 0x20 must both trigger descrambling.
///
/// Grounding: `(0x10 >> 4) & 3 == 1`, `(0x20 >> 4) & 3 == 2` — both
/// nonzero.
/// Mutation: change `!= 0` early-return condition to `== 3` -> a sector
/// flagged only 0x10 or 0x20 would be skipped and left scrambled.
#[test]
fn descramble_triggers_on_either_flag_bit() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
for &flag in &[0x10u8, 0x20, 0x30] {
let mut sector = vec![0xAA; 2048];
sector[0x14] = flag;
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
let original = sector.clone();
descramble_sector(&key, &mut sector);
assert_ne!(
&sector[128..256],
&original[128..256],
"flag {flag:#04x} (bits 4-5 nonzero) must descramble the body"
);
}
}
/// After descrambling, ONLY the two scramble bits are cleared (`& 0xCF`);
/// bits 6 and 7 of byte 0x14 must be preserved. A sector with 0x14 == 0xF0
/// becomes 0xC0 (bits 6,7 kept, bits 4,5 cleared), NOT 0x00.
///
/// Grounding: code does `sector[0x14] &= 0xCF`; 0xF0 & 0xCF == 0xC0.
/// Mutation: change `&= 0xCF` to `= 0` or `&= 0x0F` -> the preserved
/// high bits assert fails.
#[test]
fn descramble_clear_preserves_high_bits_of_0x14() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let mut sector = vec![0x00; 2048];
sector[0x14] = 0xF0; // bits 4-7 set; bits 4-5 are the flag
sector[0x54..0x59].copy_from_slice(&[0x00; 5]);
descramble_sector(&key, &mut sector);
assert_eq!(
sector[0x14], 0xC0,
"scramble bits cleared, bits 6-7 preserved (0xF0 & 0xCF)"
);
}
// ── header / body boundary (encrypted region is 0x80..0x800) ───────────
/// The encrypted region is exactly bytes 0x80..0x800. Bytes 0x00..0x80
/// (the header) must NOT be modified by the keystream — except byte 0x14
/// whose flag is cleared. In particular the sector-seed bytes 0x54..0x59
/// (which live inside the header) must survive untouched, since the
/// descrambler reads them but never writes them.
///
/// Grounding: loop is `sector.iter_mut().take(2048).skip(128)` -> indices
/// 128..2048 only.
/// Mutation: change `.skip(128)` to `.skip(0)` -> header bytes (incl. the
/// seed) get XORed and this fails.
#[test]
fn descramble_leaves_header_and_seed_intact() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0x5Au8; 2048];
sector[0x14] = 0x30;
let seed = [0xDE, 0xAD, 0xBE, 0xEF, 0x42];
sector[0x54..0x59].copy_from_slice(&seed);
let original = sector.clone();
descramble_sector(&key, &mut sector);
for i in 0..0x80usize {
if i == 0x14 {
continue;
}
assert_eq!(
sector[i], original[i],
"header byte {i:#04x} must be untouched"
);
}
assert_eq!(&sector[0x54..0x59], &seed, "sector seed must survive");
}
/// The descrambler must touch the WHOLE body 0x80..0x800, not just a
/// prefix. With a constant body and constant key, the keystream is
/// non-degenerate enough that the very last sector byte (index 2047) is
/// altered. This guards the loop bound `.take(2048)` against an
/// off-by-one that would leave the final byte(s) scrambled.
///
/// Grounding: encrypted region end is 0x800 == 2048 (exclusive).
/// Mutation: change `.take(2048)` to `.take(2047)` -> last byte unchanged,
/// assert fires (keystream byte for the last position is verified nonzero
/// below by the round-trip, and this body is all-zero so any XOR shows).
#[test]
fn descramble_covers_final_body_byte() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0x00u8; 2048];
sector[0x14] = 0x30;
sector[0x54..0x59].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
descramble_sector(&key, &mut sector);
// Body was all zero; any nonzero in [0x80,0x800) is keystream. Confirm
// the keystream reaches the final byte. (If the last keystream byte
// happened to be 0 this could be a flaky test, so assert the run-end
// region as a whole differs from zero.)
assert_ne!(
&sector[2040..2048],
&[0u8; 8][..],
"the tail of the body must be descrambled (loop must reach index 2047)"
);
}
/// Descramble is keyed by `title_key XOR seed`: two different title keys
/// produce two different bodies for the same scrambled input. A cipher
/// that ignored the title key (or mixed it in wrongly) would yield
/// identical output — silent wrong-key decryption.
///
/// Grounding: per-sector key = title_key[i] ^ sector[0x54+i].
/// Mutation: in the `key` array drop the `title_key[i] ^` term -> both
/// keys give the same body, assert fires.
#[test]
fn descramble_output_depends_on_title_key() {
let seed = [0xDE, 0xAD, 0xBE, 0xEF, 0x42];
let make = |k: &[u8; 5]| {
let mut s = vec![0x00u8; 2048];
s[0x14] = 0x30;
s[0x54..0x59].copy_from_slice(&seed);
descramble_sector(k, &mut s);
s
};
let a = make(&[0x01, 0x02, 0x03, 0x04, 0x05]);
let b = make(&[0x01, 0x02, 0x03, 0x04, 0x06]); // differs in last byte
assert_ne!(
&a[128..2048],
&b[128..2048],
"different title keys must descramble differently"
);
}
/// Descramble is keyed by the sector seed too: same title key, different
/// seed -> different body. Pins that bytes 0x54..0x59 actually feed the
/// keystream (not just the per-sector XOR key).
///
/// Mutation: replace `seed` array reads with a constant -> both seeds give
/// the same body, assert fires.
#[test]
fn descramble_output_depends_on_seed() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let make = |seed: [u8; 5]| {
let mut s = vec![0x00u8; 2048];
s[0x14] = 0x30;
s[0x54..0x59].copy_from_slice(&seed);
descramble_sector(&key, &mut s);
s
};
let a = make([0x11, 0x22, 0x33, 0x44, 0x55]);
let b = make([0x11, 0x22, 0x33, 0x44, 0x56]);
assert_ne!(
&a[128..2048],
&b[128..2048],
"different seeds must descramble differently"
);
}
}
+887
View File
@@ -0,0 +1,887 @@
//! CSS (Content Scramble System) — DVD disc encryption.
//!
//! CSS uses a weak 40-bit LFSR stream cipher (broken since 1999).
//!
//! The title key is recovered keylessly: [`crack_key`] runs the Stevenson
//! known-plaintext attack (see the [`stevenson`] module) on the scrambled
//! data, needing no player keys, disc-key crack, or external key file.
//! Sectors are then decrypted with [`descramble_sector`].
//!
//! Usage:
//! ```rust,ignore
//! if let Some(state) = css::crack_key(reader, extents, batch) {
//! css::descramble_sector(&state, &mut sector);
//! }
//! ```
pub mod lfsr;
pub mod stevenson;
pub(crate) mod tables;
use crate::disc::Extent;
use crate::sector::SectorSource;
/// Consecutive CSS-locked (`05/6F/03`) reads before the crack scan early-bails.
/// The bus-auth read gate is global (all-or-nothing), so a run this long means
/// it is shut and nothing here is crackable — bail instead of grinding the full
/// 50_000-sector budget (which is what made rc5 appear to hang on a wedged USB
/// bridge). The counter resets to 0 on any readable batch.
const CSS_LOCKED_BAIL: u32 = 64;
/// CSS decryption state for a DVD title.
#[derive(Debug, Clone)]
pub struct CssState {
/// 5-byte CSS title key (from SCSI auth or the crack fallback).
pub title_key: [u8; 5],
/// LBA half-open span `[start, end)` of the extent set this key was
/// cracked from. CSS title keys are per-VTS: a key cracked from one
/// VTS does NOT descramble a title living in a different VTS. The mux
/// path checks whether the title being opened overlaps this span; if
/// not, it re-cracks from that title's own extents. `None` for keys
/// of unknown provenance (e.g. test fixtures) — treated as "applies
/// everywhere" for backward compatibility.
pub crack_span: Option<(u32, u32)>,
}
/// Recover the CSS title key with no keys, by scanning scrambled sectors and
/// running the Stevenson known-plaintext attack (see the [`stevenson`] module).
///
/// The crib comes from `AttackPattern`: a scrambled sector's cleartext region
/// (bytes 0x00..0x80) often ends in a short-period repeating run (stuffing /
/// constant fill); the attack assumes that run continues across the 0x80
/// boundary into the encrypted region, giving the known plaintext the 2^16
/// LFSR recovery needs. We scan up to 50000 sectors across the
/// extents and return the first sector that yields a key — no player keys, no
/// disc-key crack. Works on a live drive (after bus-auth unlocks reads) and on
/// disc images alike.
pub fn crack_key(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
) -> Option<CssState> {
crack_key_halt(reader, extents, batch_sectors, None)
}
/// Outcome of a CSS crack scan that distinguishes the THREE cases the bare
/// `Option<CssState>` conflated (and which caused a silent-failure bug:
/// scrambled-but-uncracked content was treated as "unencrypted" and muxed as
/// plaintext garbage at exit 0):
///
/// - [`CrackOutcome::Cracked`] — a scrambled sector yielded a title key.
/// - [`CrackOutcome::Unencrypted`] — NO scrambled sector was seen across the
/// scanned extents (`is_scrambled` never true): the content is genuinely
/// plaintext, so proceeding without a key is correct.
/// - [`CrackOutcome::ScrambledUncracked`] — scrambled sectors WERE seen but no
/// key could be recovered (the Stevenson attack found no crackable crib, or
/// the scrambled region was unreadable). The content is encrypted; muxing it
/// as plaintext would emit garbage, so callers MUST surface a hard error
/// ([`crate::error::Error::CssKeyMissing`]) instead of falling through to
/// "unencrypted".
#[derive(Debug, Clone)]
pub enum CrackOutcome {
Cracked(CssState),
Unencrypted,
ScrambledUncracked,
}
impl CrackOutcome {
/// The cracked `CssState`, if any. `None` for `Unencrypted` /
/// `ScrambledUncracked`. Lets the `Option`-returning wrappers stay thin.
pub fn into_state(self) -> Option<CssState> {
match self {
CrackOutcome::Cracked(s) => Some(s),
_ => None,
}
}
/// True when scrambled sectors were seen but no key was recovered — the
/// case callers must surface as a hard error instead of "unencrypted".
pub fn is_scrambled_uncracked(&self) -> bool {
matches!(self, CrackOutcome::ScrambledUncracked)
}
}
/// [`crack_key`] returning the full [`CrackOutcome`] (Cracked / Unencrypted /
/// ScrambledUncracked) so callers can distinguish "genuinely unencrypted" from
/// "encrypted but uncrackable" — the latter must become a hard error, never a
/// silent fall-through to plaintext.
pub fn crack_key_outcome(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
) -> CrackOutcome {
crack_key_scan(reader, extents, batch_sectors, halt, true)
}
/// [`crack_key`] with an optional cooperative-cancellation token.
///
/// "No silent hangs": the crack scans up to 50_000 sectors, which on a live
/// drive hitting bad sectors can take a long time. This variant polls `halt`
/// once per batch (the same cadence sweep/patch use) so an operator Stop or a
/// scan-level watchdog can interrupt the scan, and emits a
/// `freemkv::heartbeat` beat ("css_crack") each batch so a stuck scan is
/// visible in the log.
pub fn crack_key_halt(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
) -> Option<CssState> {
crack_key_scan(reader, extents, batch_sectors, halt, false).into_state()
}
/// The crack scan, returning the full [`CrackOutcome`]. Tracks a
/// `saw_scrambled` flag so a scrambled-but-uncracked disc is distinguished
/// from a genuinely-unencrypted one (the [`crack_key`] / [`crack_key_halt`]
/// `Option` wrappers collapse both to `None`).
fn crack_key_scan(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
// True only on the INITIAL scan: a fully CSS-locked (`05/6F/03`) result is a
// hard `ScrambledUncracked`. False on the per-VTS re-crack so a lapsed-AGID
// locked read returns None instead of killing a genuinely crackable title.
fail_on_locked: bool,
) -> CrackOutcome {
// Batch the reads: a live optical drive at 1 sector/read is glacial, and the
// crack only needs to FIND one scrambled sector whose 0x80 plaintext matches
// a known PES header. `batch_sectors` MUST be sized to the source — a drive
// rejects a READ(10) larger than its per-command max (DVD = 16) and
// `Drive::read` does not chunk, so an over-large batch fails every read and
// scans nothing. Callers pass `detect_max_batch_sectors(device_path)` for a
// live drive, a file-safe value for an image, or 1 to force per-sector.
let batch = (batch_sectors.max(1)) as u32;
// Record the LBA span the key is being cracked from so the per-title mux
// path can tell whether a later title lives in the same VTS (overlaps the
// span → key applies) or a different one (→ re-crack). Half-open [min,max).
let crack_span = extents
.iter()
.filter(|e| e.sector_count > 0)
.map(|e| (e.start_lba, e.start_lba.saturating_add(e.sector_count)))
.reduce(|(amin, amax), (bmin, bmax)| (amin.min(bmin), amax.max(bmax)));
let mut tried = 0u32;
let max_tries = 50_000u32;
let mut buf = vec![0u8; batch as usize * 2048];
let mut hb = crate::progress::Heartbeat::new("css_crack");
// Track whether ANY scrambled sector was observed. If we exhaust the scan
// budget having seen scrambled data but never recovered a key, the content
// is encrypted-but-uncrackable — a HARD failure the caller must surface,
// NOT silently treat as unencrypted (which would mux scrambled MPEG as
// plaintext → garbage at exit 0). See `CrackOutcome::ScrambledUncracked`.
let mut saw_scrambled = false;
// A read rejected with sense `05/6F/03` ("scrambled sector without
// authentication") is positive proof of CSS encryption — never collapse it
// to "unencrypted". A run of consecutive locked reads means the bus-auth
// gate is shut (it is global, so reads are all-or-nothing), so the scan
// early-bails. `consecutive_locked` resets on any readable batch, so a
// crackable title (gate open) never trips it.
let mut saw_locked = false;
let mut consecutive_locked = 0u32;
'outer: for (extent_idx, ext) in extents.iter().enumerate() {
let mut i = 0u32;
while i < ext.sector_count && tried < max_tries {
// Cooperative cancellation — poll once per batch, the same cadence
// sweep/patch use, so a Stop / watchdog can interrupt the scan.
if let Some(h) = halt {
if h.is_cancelled() {
break 'outer;
}
}
// Liveness beacon: a long scan over a damaged disc stays visible.
// The heartbeat is time-throttled; only when it actually beats do
// we emit the crack-specific context (tried/lba/extent_idx).
if hb.tick(tried as u64, max_tries as u64) {
tracing::debug!(
target: "freemkv::heartbeat",
phase = "css_crack",
tried,
lba = ext.start_lba + i,
extent_idx,
"scanning"
);
}
let n = (ext.sector_count - i).min(batch);
let want = n as usize * 2048;
match reader.read_sectors(ext.start_lba + i, n as u16, &mut buf[..want], true) {
Ok(_) => {
// A readable batch: the gate is open — reset the locked run.
consecutive_locked = 0;
for s in 0..n as usize {
tried += 1;
let sect = &buf[s * 2048..(s + 1) * 2048];
// Use the HARDENED pack-gated check (Fix 3): a clear stub
// sector with stray bits at 0x14 must NOT count as
// scramble evidence, or a genuinely-unencrypted title
// would falsely report ScrambledUncracked (a false E7023).
if is_scrambled_pack(sect) {
saw_scrambled = true;
if let Some(key) = stevenson::crack_title_key(sect) {
return CrackOutcome::Cracked(CssState {
title_key: key,
crack_span,
});
}
}
if tried >= max_tries {
break 'outer;
}
}
}
// A failed batch still counts toward the budget so a damaged
// region can't loop forever. A CSS-locked failure (`05/6F/03`)
// proves encryption and, in a long enough run, means the read
// gate is shut — track it and early-bail rather than grind.
Err(e) => {
tried += n;
if e.scsi_sense().is_some_and(|s| s.is_css_locked()) {
saw_locked = true;
consecutive_locked += 1;
if consecutive_locked >= CSS_LOCKED_BAIL {
break 'outer;
}
} else {
consecutive_locked = 0;
}
}
}
i += n;
}
}
// Budget exhausted / extents walked / early-bailed with no key recovered.
// The disc is ENCRYPTED-but-uncracked (a hard failure on the initial scan)
// when EITHER a scrambled sector was actually seen, OR — on the initial scan
// only (`fail_on_locked`) — every read was CSS-locked (`05/6F/03`), itself
// proof of scrambling. A re-crack (`fail_on_locked` false) stays soft: a
// lapsed-AGID locked read yields None, not a hard fail, so a crackable title
// in another VTS isn't killed. Only a scan that saw neither a scrambled
// sector nor a CSS-lock is genuinely unencrypted.
if saw_scrambled || (saw_locked && fail_on_locked) {
CrackOutcome::ScrambledUncracked
} else {
CrackOutcome::Unencrypted
}
}
/// Descramble a single CSS-encrypted sector in place.
pub fn descramble_sector(state: &CssState, sector: &mut [u8]) {
lfsr::descramble_sector(&state.title_key, sector);
}
/// Check if a sector has the CSS scramble flag set.
///
/// This is the RAW flag test — bits 4-5 of the sub-header byte 0x14 — used by
/// the descramble loop (`decrypt::decrypt_sectors`), which has already committed
/// to descrambling a known title's VOB data and only needs to skip the clear
/// NAV packs interleaved in it. For the CRACK SCAN's "did this disc actually
/// contain scrambled content?" decision (which must not false-positive on a
/// clear stub), use [`is_scrambled_pack`] instead.
pub fn is_scrambled(sector: &[u8]) -> bool {
sector.len() >= 2048 && (sector[0x14] >> 4) & 0x03 != 0
}
/// The 4-byte MPEG-2 Program Stream pack-start code (`00 00 01 BA`) every DVD
/// video sector opens with. CSS leaves the clear header (`0x00..0x80`)
/// untouched, so this signature survives scrambling.
pub(crate) const PACK_START: [u8; 4] = [0x00, 0x00, 0x01, 0xBA];
/// Check if a sector is a CSS-scrambled DVD **video pack** — the HARDENED test
/// the crack scan uses to set its `saw_scrambled` evidence flag (Fix 3).
///
/// [`is_scrambled`] keys solely on bits 4-5 of byte 0x14. That single byte is
/// only meaningful inside a real DVD sector — an MPEG-2 Program Stream pack,
/// which ALWAYS begins with the 32-bit pack-start code `00 00 01 BA` at offset
/// 0x00. A tiny clear / nav-only stub (a 0.5 s menu loop, an FBI-warning title)
/// can carry arbitrary bytes that happen to set bits 4-5 of byte 0x14; trusting
/// byte 0x14 alone there would flip the scan's `saw_scrambled` gate and make a
/// genuinely-UNENCRYPTED title report `ScrambledUncracked` — a false E7023.
///
/// Requiring the pack-start signature FIRST means only a sector that is
/// structurally a DVD video pack can be counted as scramble evidence. This does
/// NOT weaken the genuine "encrypted but uncrackable" hard-fail: a real
/// scrambled feature is made of valid PS packs, so its scrambled sectors still
/// pass this check and still drive `ScrambledUncracked` when no key cracks. (The
/// descramble loop keeps the looser [`is_scrambled`]: by the time it runs we
/// already know the title is CSS, and it only needs to skip interleaved clear
/// NAV packs — a wrongly-skipped or wrongly-included sector there is recoverable
/// per-sector, whereas a false scramble verdict in the scan poisons the whole
/// title's outcome.)
pub fn is_scrambled_pack(sector: &[u8]) -> bool {
sector.len() >= 2048 && sector[0x00..0x04] == PACK_START && (sector[0x14] >> 4) & 0x03 != 0
}
#[cfg(test)]
mod tests {
use super::*;
use crate::error::{Error, Result};
// ── is_scrambled ───────────────────────────────────────────────────────
/// is_scrambled returns false for any buffer shorter than one sector,
/// WITHOUT indexing byte 0x14 (which would panic on a tiny buffer). The
/// length guard is short-circuited before the flag read.
///
/// Grounding: `sector.len() >= 2048 && (sector[0x14] >> 4) & 0x03 != 0` —
/// `&&` short-circuits so a 20-byte buffer never reads index 0x14.
/// Mutation: swap the operands so the flag is read first
/// (`(sector[0x14]...) && sector.len() >= 2048`) -> panics indexing a
/// 20-byte slice; this test catches it.
#[test]
fn is_scrambled_short_buffer_is_false_no_panic() {
assert!(!is_scrambled(&[]));
assert!(!is_scrambled(&[0u8; 20])); // shorter than 0x14+1 even
assert!(!is_scrambled(&[0xFFu8; 2047])); // one byte short of a sector
}
/// is_scrambled keys on bits 4-5 of byte 0x14 (the CSS scramble field).
/// A full sector flagged 0x10/0x20/0x30 is scrambled; 0x00 and the
/// high-bit-only values 0x40/0x80 are clear.
///
/// Grounding: `(sector[0x14] >> 4) & 0x03`.
/// Mutation: widen mask to `& 0x0F` -> 0x40 reports scrambled, the 0x40
/// assert fails.
#[test]
fn is_scrambled_uses_bits_4_5_only() {
let mut s = vec![0u8; 2048];
for (flag, expected) in [
(0x00u8, false),
(0x10, true),
(0x20, true),
(0x30, true),
(0x40, false),
(0x80, false),
(0xC0, false),
(0xFF, true), // bits 4-5 set within 0xFF
] {
s[0x14] = flag;
assert_eq!(
is_scrambled(&s),
expected,
"flag byte {flag:#04x} scramble detection"
);
}
}
/// is_scrambled accepts exactly 2048 bytes as the minimum (boundary at the
/// inclusive value 2048).
///
/// Grounding: `sector.len() >= 2048`.
/// Mutation: change `>= 2048` to `> 2048` -> an exact 2048-byte scrambled
/// sector reports false; this fails.
#[test]
fn is_scrambled_exact_sector_length_accepted() {
let mut s = vec![0u8; 2048];
s[0x14] = 0x30;
assert!(is_scrambled(&s), "exactly 2048 bytes must be eligible");
}
/// Fix 3 hardening: `is_scrambled_pack` (the crack-scan evidence gate)
/// requires BOTH the MPEG-PS pack-start code at 0x00 AND the 0x14 scramble
/// bits. A clear / nav-only stub whose bytes happen to set bits 4-5 of 0x14
/// but lacks the pack-start is NOT counted as scramble evidence — without
/// this the scan flips `saw_scrambled` and a genuinely unencrypted title
/// reports `ScrambledUncracked` (the false E7023). The looser `is_scrambled`
/// (descramble gate) still reads the same sector as flagged.
///
/// Grounding: `sector[0x00..0x04] == 00 00 01 BA && (sector[0x14] >> 4)...`.
/// Mutation: drop the pack-start clause -> the 0x14-only sector counts as a
/// scrambled pack; the first assert fails.
#[test]
fn is_scrambled_pack_requires_pack_start_signature() {
let mut s = vec![0u8; 2048];
s[0x14] = 0x30; // scramble bits set, but no pack-start at 0x00
assert!(
!is_scrambled_pack(&s),
"0x14 bits without the MPEG-PS pack-start must NOT count as a scrambled pack"
);
// The looser descramble-gate check still sees the raw flag.
assert!(is_scrambled(&s), "is_scrambled keys on the 0x14 flag alone");
// A near-miss pack-start (wrong final byte) is still rejected.
s[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBB]);
assert!(
!is_scrambled_pack(&s),
"a wrong pack-start byte must not qualify"
);
// The real signature flips it to a scrambled pack.
s[0x00..0x04].copy_from_slice(&PACK_START);
assert!(
is_scrambled_pack(&s),
"valid pack-start + 0x14 bits → scrambled pack"
);
}
// ── crack_key scanning over a mock SectorSource ────────────────────────
/// Records every (lba, count) read; returns a caller-supplied flag byte at
/// 0x14 so we can drive scrambled/clear sectors, or an injected error.
struct MockSource {
reads: std::cell::RefCell<Vec<u32>>,
flag_byte: u8,
fail_all: bool,
/// Every read fails with CSS-locked sense `05/6F/03` (drive refusing
/// scrambled reads because the bus-auth gate isn't open).
lock_all: bool,
/// When set, the sector at `crackable.0` is served as a full
/// Stevenson-crackable scrambled sector (`crackable.1`, 2048 bytes)
/// instead of the uniform `flag_byte` fill. Lets the scan actually
/// reach `CrackOutcome::Cracked` from a synthetic ISO.
crackable: Option<(u32, Vec<u8>)>,
}
impl MockSource {
fn new(flag_byte: u8) -> Self {
Self {
reads: std::cell::RefCell::new(Vec::new()),
flag_byte,
fail_all: false,
lock_all: false,
crackable: None,
}
}
}
/// Build a Stevenson-crackable scrambled sector for `(title_key, seed)`:
/// the cleartext header (0x59..0x80) carries a periodic run that continues
/// across the 0x80 boundary into the encrypted region — the crib
/// `stevenson::crack_title_key` recovers a key from. Mirrors the
/// `synth_periodic_sector` fixture in the stevenson tests but built here
/// from the crate-internal `scramble_sector`.
fn crackable_sector(title_key: &[u8; 5], seed: &[u8; 5], period: usize) -> Vec<u8> {
const RUN_START: usize = 0x59;
const SEED_OFFSET: usize = 0x54;
let mut plaintext = vec![0u8; 2048];
plaintext[0x00..0x04].copy_from_slice(&PACK_START); // valid DVD pack header
plaintext[0x14] = 0x10; // scramble flag
let pat: Vec<u8> = (0..period)
.map(|k| (0xA0u8.wrapping_add(k as u8)) ^ 0x5A)
.collect();
for (i, b) in plaintext.iter_mut().enumerate().skip(RUN_START) {
*b = pat[i % period];
}
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
lfsr::scramble_sector(title_key, &mut plaintext);
plaintext
}
impl SectorSource for MockSource {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
self.reads.borrow_mut().push(lba);
if self.lock_all {
return Err(Error::DiscRead {
sector: lba as u64,
status: Some(2),
sense: Some(crate::scsi::ScsiSense {
sense_key: 0x05,
asc: 0x6F,
ascq: 0x03,
}),
});
}
if self.fail_all {
return Err(Error::DecryptFailed);
}
let n = count as usize * 2048;
let end = n.min(buf.len());
for b in buf[..end].iter_mut() {
*b = 0;
}
// Fill each sector in the batch with the uniform flag byte, EXCEPT a
// designated crackable LBA which gets the full synthetic sector.
for s in 0..count as u32 {
let sect_lba = lba + s;
let base = s as usize * 2048;
if base + 2048 > end {
break;
}
match &self.crackable {
Some((clba, sector)) if *clba == sect_lba => {
buf[base..base + 2048].copy_from_slice(sector);
}
_ => {
// Real DVD video sectors always open with the MPEG-PS
// pack-start code; `is_scrambled` (Fix 3) requires it
// before trusting the 0x14 scramble bits, so the fixture
// must include it for a `flag_byte` of 0x30 to register
// as scrambled.
buf[base..base + 4].copy_from_slice(&PACK_START);
buf[base + 0x14] = self.flag_byte;
}
}
}
Ok(n)
}
}
/// crack_key caps total scanned sectors at 50_000 even when extents are
/// far larger, and counts EVERY scanned sector (clear ones included)
/// toward the budget. With one 200_000-sector extent of clear sectors, it
/// must read exactly 50_000 sectors and return None — never run away.
///
/// Grounding: `let max_tries = 50_000; ... tried += 1` before the read,
/// loop guard `tried < max_tries`.
/// Mutation: change `50_000` to `500_000` -> read count exceeds 50_000;
/// the exact-count assert fails. Removing the `tried += 1` increment ->
/// would read all 200_000; also fails.
#[test]
fn crack_key_caps_total_tries_at_50000() {
let mut src = MockSource::new(0x00); // clear sectors, never a hit
let extents = [Extent {
start_lba: 0,
sector_count: 200_000,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none(), "clear sectors yield no key");
assert_eq!(
src.reads.borrow().len(),
50_000,
"scan must stop at the 50_000-sector budget"
);
}
// ── CrackOutcome: scrambled-but-uncracked vs genuinely unencrypted (Fix 6) ─
/// A scan over CLEAR sectors (scramble flag never set) returns
/// `Unencrypted` — the content is genuinely plaintext, so proceeding
/// without a key is correct.
#[test]
fn crack_outcome_clear_sectors_is_unencrypted() {
let mut src = MockSource::new(0x00); // never scrambled
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
matches!(outcome, CrackOutcome::Unencrypted),
"no scrambled sector seen → Unencrypted, got {outcome:?}"
);
// The Option wrapper collapses Unencrypted → None.
assert!(crack_key(&mut MockSource::new(0x00), &extents, 1).is_none());
}
/// THE Fix 6 regression: a scan that SEES scrambled sectors (flag set) but
/// recovers no key (the mock's zeroed data has no Stevenson crib) must
/// return `ScrambledUncracked` — a HARD failure — NOT `Unencrypted`. The
/// old code conflated this with "unencrypted" and muxed scrambled MPEG as
/// plaintext (garbage at exit 0).
#[test]
fn crack_outcome_scrambled_uncracked_is_hard_failure() {
let mut src = MockSource::new(0x30); // scrambled flag set, no crackable crib
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
outcome.is_scrambled_uncracked(),
"scrambled sectors seen but no key → ScrambledUncracked, got {outcome:?}"
);
// The legacy Option wrapper still collapses this to None (the callers
// that need the distinction now use crack_key_outcome instead).
assert!(crack_key(&mut MockSource::new(0x30), &extents, 1).is_none());
}
/// Even when every read FAILS, a scan that never managed to observe a
/// scrambled sector reports `Unencrypted` (we cannot prove encryption from
/// unreadable data alone — the AACS/keydb paths and the disc-level
/// `css_error` plumbing cover genuinely unreadable encrypted discs).
#[test]
fn crack_outcome_all_reads_fail_is_unencrypted() {
let mut src = MockSource::new(0x30);
src.fail_all = true; // no sector is ever inspected
let extents = [Extent {
start_lba: 0,
sector_count: 10,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
matches!(outcome, CrackOutcome::Unencrypted),
"no readable scrambled sector → Unencrypted, got {outcome:?}"
);
}
/// Fix C (rc.5.1): on the INITIAL scan, a drive that refuses every read with
/// CSS-locked sense (`05/6F/03`) is encrypted-but-locked →
/// `ScrambledUncracked` (a hard failure), NOT `Unencrypted`. This is the
/// rc4.3 bug: every VOB read came back `6F/03`, so the scan saw no scrambled
/// sector and wrongly declared the disc unencrypted → 19 KB garbage.
#[test]
fn crack_outcome_css_locked_initial_is_scrambled_uncracked() {
let mut src = MockSource::new(0x30);
src.lock_all = true; // every read → 05/6F/03
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
outcome.is_scrambled_uncracked(),
"every read 6F/03 on the initial scan → ScrambledUncracked, got {outcome:?}"
);
}
/// MISSING #1 guard: the re-crack path (the `Option`-returning `crack_key`,
/// `fail_on_locked == false`) must NOT hard-fail on a CSS-locked read — it
/// returns `None`. A lapsed-AGID re-crack of another VTS stays soft so a
/// genuinely crackable title isn't killed by a transient locked read.
#[test]
fn crack_key_recrack_locked_is_none_not_hard_fail() {
let mut src = MockSource::new(0x30);
src.lock_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
assert!(crack_key(&mut src, &extents, 1).is_none());
}
/// Fix F: a fully CSS-locked scan early-bails near `CSS_LOCKED_BAIL`
/// consecutive locked reads instead of grinding the whole 50_000-sector
/// budget (the rc5 "stuck Scanning…" hang on a wedged bridge).
#[test]
fn crack_css_locked_scan_early_bails() {
let mut src = MockSource::new(0x30);
src.lock_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 10_000,
}];
let _ = crack_key_outcome(&mut src, &extents, 1, None);
let n = src.reads.borrow().len();
assert!(
n <= (CSS_LOCKED_BAIL as usize) + 1,
"locked scan early-bails near {CSS_LOCKED_BAIL}, not 10000; read {n}"
);
}
/// The budget spans ALL extents, not per-extent: two extents summing past
/// the cap must still stop at 50_000 total reads.
///
/// Grounding: `tried` is declared outside the `for ext in extents` loop;
/// `if tried >= max_tries { break }` after each extent.
/// Mutation: move `let mut tried = 0` inside the extent loop -> each extent
/// gets its own 50_000 budget; total reads would be 80_000, this fails.
#[test]
fn crack_key_budget_is_shared_across_extents() {
let mut src = MockSource::new(0x00);
let extents = [
Extent {
start_lba: 0,
sector_count: 40_000,
},
Extent {
start_lba: 100_000,
sector_count: 40_000,
},
];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
50_000,
"the 50_000 budget is shared across all extents"
);
}
/// crack_key scans sequentially from each extent's start_lba. The first
/// reads must be at the extent's start_lba, start_lba+1, ... pinning the
/// LBA arithmetic `ext.start_lba + i`.
///
/// Grounding: `reader.read_sectors(ext.start_lba + i, 1, ...)`.
/// Mutation: change `ext.start_lba + i` to just `i` -> the recorded LBAs
/// would start at 0, not 5000; this fails.
#[test]
fn crack_key_scans_from_extent_start_lba() {
let mut src = MockSource::new(0x00);
let extents = [Extent {
start_lba: 5_000,
sector_count: 4,
}];
let _ = crack_key(&mut src, &extents, 1);
let reads = src.reads.borrow();
assert_eq!(
&reads[..],
&[5_000, 5_001, 5_002, 5_003],
"sequential scan from start_lba"
);
}
/// A read error on a sector does NOT abort the scan: crack_key keeps
/// scanning subsequent sectors (the error sector still counts toward the
/// budget). With a small failing extent, every sector is attempted and the
/// function returns None.
///
/// Grounding: `if reader.read_sectors(...).is_ok() && is_scrambled(...)` —
/// an Err simply falls through to `i += 1`.
/// Mutation: change the read-error handling to `reader.read_sectors(...)?`
/// (propagate) -> crack_key would stop after the first error and read only
/// 1 sector; this asserts all 10 were attempted.
#[test]
fn crack_key_continues_past_read_errors() {
let mut src = MockSource::new(0x30);
src.fail_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 10,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
10,
"read errors must not abort the scan"
);
}
/// Empty extents (no sectors) -> crack_key reads nothing and returns None.
/// A zero-sector extent must not read its start_lba.
///
/// Grounding: `while i < ext.sector_count` with sector_count == 0 never
/// enters.
/// Mutation: change `i < ext.sector_count` to `i <= ext.sector_count` ->
/// one spurious read at start_lba; this asserts zero reads.
#[test]
fn crack_key_empty_extent_reads_nothing() {
let mut src = MockSource::new(0x30);
let extents = [Extent {
start_lba: 42,
sector_count: 0,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
0,
"zero-sector extent reads nothing"
);
}
/// No extents at all -> immediate None, zero reads.
///
/// Grounding: `for ext in extents` over an empty slice is a no-op.
/// Mutation: any change that reads before the loop would break this.
#[test]
fn crack_key_no_extents_is_none() {
let mut src = MockSource::new(0x30);
let res = crack_key(&mut src, &[], 1);
assert!(res.is_none());
assert_eq!(src.reads.borrow().len(), 0);
}
// ── Scan-level Cracked branch + per-VTS re-crack success (audit §2 / §5 #8) ─
/// SCAN-LEVEL CRACKED (audit gap "MockSource never yields a crackable
/// sector"): drive the full `crack_key_scan` over a synthetic ISO whose
/// scan hits a Stevenson-crackable scrambled sector. The outcome must be
/// `CrackOutcome::Cracked` with a key that round-trips the sector, AND the
/// `crack_span` must be recorded as the half-open extent span (the per-VTS
/// routing key the mux path needs). Previously only the leaf crack and the
/// Uncracked/Unencrypted branches were tested — the Cracked branch and
/// `crack_span` recording were never exercised end-to-end.
#[test]
fn crack_outcome_reaches_cracked_with_span() {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let seed = [0x11, 0x22, 0x33, 0x44, 0x55];
let crackable = crackable_sector(&title_key, &seed, 8);
// The crackable sector sits a few sectors into the extent.
let mut src = MockSource::new(0x00); // surrounding sectors: clear
src.crackable = Some((1003, crackable.clone()));
let extents = [Extent {
start_lba: 1000,
sector_count: 50,
}];
let outcome = crack_key_outcome(&mut src, &extents, 4, None);
let state = match outcome {
CrackOutcome::Cracked(s) => s,
other => panic!("expected Cracked, got {other:?}"),
};
// The recovered key descrambles the crackable sector body.
let mut test = crackable.clone();
descramble_sector(&state, &mut test);
let mut plain = crackable;
lfsr::descramble_sector(&title_key, &mut plain);
assert_eq!(
&test[0x80..],
&plain[0x80..],
"recovered key must round-trip the scrambled sector body"
);
// crack_span = half-open [start, start+count) of the scanned extent.
assert_eq!(
state.crack_span,
Some((1000, 1050)),
"crack_span must record the extent LBA span for per-VTS routing"
);
}
/// CSS_ERROR WIRING (audit §2 / §5 #7): an all-locked synthetic ISO (every
/// VOB read returns CSS-locked sense `05/6F/03` across MULTIPLE extents, as a
/// real encrypted-but-unauthenticated disc image does) must produce the exact
/// outcome the scan converts into `disc.css_error = Some(Error::CssKeyMissing)`
/// — i.e. `CrackOutcome::ScrambledUncracked` / `is_scrambled_uncracked()`,
/// NOT `Unencrypted`. disc/mod.rs's `crack_key_outcome → ScrambledUncracked`
/// arm (where it stamps css_error) is driven by exactly this signal, so this
/// pins the css-layer contract that arm depends on without touching the
/// scan plumbing.
#[test]
fn all_locked_synthetic_iso_yields_css_key_missing_signal() {
let mut src = MockSource::new(0x30);
src.lock_all = true; // every read → 05/6F/03 across the whole "ISO"
let extents = [
Extent {
start_lba: 0,
sector_count: 30,
},
Extent {
start_lba: 5_000,
sector_count: 30,
},
];
let outcome = crack_key_outcome(&mut src, &extents, 16, None);
assert!(
outcome.is_scrambled_uncracked(),
"all-locked ISO → ScrambledUncracked (the css_error=CssKeyMissing \
signal), got {outcome:?}"
);
// The legacy Option wrapper still collapses it to None — callers that
// surface the hard error must use crack_key_outcome, which this proves.
let mut src2 = MockSource::new(0x30);
src2.lock_all = true;
assert!(crack_key(&mut src2, &extents, 16).is_none());
}
/// PER-VTS RE-CRACK SUCCESS (audit gap "success path missing"): the prior
/// re-crack test only covered the locked→None path. Here a re-crack
/// (`crack_key`, `fail_on_locked == false`) over a DIFFERENT VTS's extents
/// finds that VTS's own crackable sector and returns a `CssState` whose
/// `crack_span` matches the new extents — proving a key cracked for one VTS
/// is genuinely re-derived (not reused) for another.
#[test]
fn recrack_succeeds_on_other_vts_extents() {
let title_key = [0xFE, 0xDC, 0xBA, 0x98, 0x76];
let seed = [0x00, 0xFF, 0x80, 0x7F, 0x01];
let crackable = crackable_sector(&title_key, &seed, 5);
let mut src = MockSource::new(0x00);
// The second VTS lives at a disjoint LBA range; its crackable sector is
// the first one in the extent.
src.crackable = Some((9000, crackable));
let other_vts = [Extent {
start_lba: 9000,
sector_count: 20,
}];
let state = crack_key(&mut src, &other_vts, 4).expect("re-crack must recover a key");
assert_eq!(
state.crack_span,
Some((9000, 9020)),
"re-crack span must reflect the OTHER VTS extents, not a reused span"
);
}
}
+593
View File
@@ -0,0 +1,593 @@
//! CSS title-key recovery — Frank A. Stevenson's divide-and-conquer attack
//! (1999), ported exactly from libdvdcss `RecoverTitleKey` + `AttackPattern`
//! (css.c).
//!
//! Recovers the 5-byte CSS title key from a single scrambled DVD sector with
//! no player keys and no disc-key crack, using only known plaintext.
//!
//! # The cipher this attacks
//!
//! The content descrambler ([`super::lfsr::descramble_sector`], = libdvdcss
//! `dvdcss_unscramble`) seeds its two LFSRs **directly** from
//! `key = title_key XOR sector_seed` (seed = `sector[0x54..0x59]`):
//!
//! ```text
//! i_t1 = (key[0] ^ sec[0x54]) | 0x100; // LFSR1 low (9-bit)
//! i_t2 = key[1] ^ sec[0x55]; // LFSR1 high
//! i_t3 = (key[2]|key[3]<<8|key[4]<<16) ^ seed3; // LFSR0 (24-bit feedback)
//! i_t3 = i_t3*2 + 8 - (i_t3 & 7);
//! // per byte: *p = TAB1[*p] ^ (i_t5 & 0xff)
//! ```
//!
//! There is NO `decrypt_key` mangling on the content path. So the recovery
//! is a single inversion of `dvdcss_unscramble`, not the multi-stage
//! working-key inversion the previous (non-CSS) implementation used.
//!
//! # The attack
//!
//! 1. **Known plaintext → keystream.** Because the descramble applies TAB1
//! to the ciphertext, the per-byte keystream is
//! `buf[i] = TAB1[cipher[i]] ^ plain[i]` (matching libdvdcss
//! `RecoverTitleKey`'s `p_buffer`).
//! 2. **Brute the 16-bit LFSR1 seed.** For each of 2^16 seeds, run LFSR1
//! forward; for the first four steps deduce the LFSR0 output bytes from
//! the keystream (carry-tracked), reconstructing `i_t3`. For the next six
//! steps clock LFSR0 normally and check it reproduces the keystream — a
//! wrong LFSR1 seed fails fast.
//! 3. **Back-clock LFSR0.** Run four backward `i_t3` steps (each a 256-way
//! search for the byte shifted in) to reach the initial state, then undo
//! `i_t3 = i_t3*2 + 8 - (i_t3 & 7)` to recover key[2..5].
//! 4. **XOR back the seed.** `key[0..5] ^= sector_seed[0..5]` (plain XOR —
//! the descramble seeds directly, so there is no inversion).
//!
//! `AttackPattern` finds known plaintext for step 1: the longest periodic
//! run in the cleartext `sec[0x00..0x80]`, assumed to continue into the
//! encrypted region at 0x80.
use super::lfsr::descramble_sector;
use super::tables::{TAB1, TAB2, TAB3, TAB4, TAB5};
use crate::consts::SECTOR_BYTES;
const ENCRYPTED_START: usize = 0x80; // byte 128
const SEED_OFFSET: usize = 0x54; // sector seed at bytes 0x54-0x58
const FLAG_BYTE: usize = 0x14;
/// RecoverTitleKey: recover the title key from cipher + known plaintext.
///
/// Exact port of libdvdcss `RecoverTitleKey` (css.c). `crypted` is the
/// ciphertext starting at sector byte 0x80; `decrypted` is the matching
/// known plaintext; `seed` is `sector[0x54..0x59]`. On success returns the
/// recovered 5-byte title key; `None` if no LFSR seed reproduces the
/// keystream.
///
/// At least 10 bytes of `crypted`/`decrypted` are required (the cipher is
/// iterated 10 times: 4 to reconstruct LFSR0, 6 to validate).
fn recover_title_key_from_plain(
crypted: &[u8],
decrypted: &[u8],
seed: &[u8; 5],
) -> Option<[u8; 5]> {
if crypted.len() < 10 || decrypted.len() < 10 {
return None;
}
// buf[i] = TAB1[cipher[i]] ^ plain[i] — the per-byte content keystream.
let mut buffer = [0u8; 10];
for (i, b) in buffer.iter_mut().enumerate() {
*b = TAB1[crypted[i] as usize] ^ decrypted[i];
}
let mut key = [0u8; 5];
let mut found = false;
for i_try in 0u32..0x1_0000 {
let mut i_t1 = (i_try >> 8) | 0x100;
let mut i_t2 = i_try & 0xff;
let mut i_t3: u32 = 0; // not needed yet
let mut i_t5: u32 = 0;
// Iterate the cipher 4 times to reconstruct LFSR0 (i_t3).
for &b in buffer.iter().take(4) {
let i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
let i_t4 = TAB5[i_t4 as usize] as u32;
// Deduce i_t6 (LFSR0 output, pre-TAB4) and the carry.
let mut i_t6 = b as u32;
if i_t5 != 0 {
i_t6 = (i_t6 + 0xff) & 0xff;
}
if i_t6 < i_t4 {
i_t6 += 0x100;
}
i_t6 -= i_t4;
i_t5 += i_t6 + i_t4;
let i_t6 = TAB4[i_t6 as usize] as u32;
i_t3 = (i_t3 << 8) | i_t6;
i_t5 >>= 8;
}
let i_candidate = i_t3;
// Iterate 6 more times to validate the candidate.
let mut i = 4usize;
while i < 10 {
let i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
let i_t4 = TAB5[i_t4 as usize] as u32;
let mut i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xff;
i_t3 = (i_t3 << 8) | i_t6;
i_t6 = TAB4[i_t6 as usize] as u32;
i_t5 += i_t6 + i_t4;
if (i_t5 & 0xff) as u8 != buffer[i] {
break;
}
i_t5 >>= 8;
i += 1;
}
if i != 10 {
continue;
}
// Four backward steps of iterating i_t3 to deduce the initial state.
i_t3 = i_candidate;
for _ in 0..4 {
let i_t1_byte = i_t3 & 0xff;
i_t3 >>= 8;
// Brute-force the byte shifted in (top byte of the 24-bit reg).
for j in 0u32..256 {
i_t3 = (i_t3 & 0x1_ffff) | (j << 17);
let i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xff;
if i_t6 == i_t1_byte {
break;
}
}
}
// Undo `i_t3 = i_t3*2 + 8 - (i_t3 & 7)` to recover key[2..5].
let i_t4 = (i_t3 >> 1).wrapping_sub(4);
for i_t5 in 0u32..8 {
let val = i_t4.wrapping_add(i_t5);
if val.wrapping_mul(2).wrapping_add(8).wrapping_sub(val & 7) == i_t3 {
key[0] = (i_try >> 8) as u8;
key[1] = (i_try & 0xff) as u8;
key[2] = (val & 0xff) as u8;
key[3] = ((val >> 8) & 0xff) as u8;
key[4] = ((val >> 16) & 0xff) as u8;
found = true;
break;
}
}
// First fully-validated candidate wins. The 48-bit keystream constraint
// makes a second match cryptographically negligible on real sectors, but
// continuing would let a later spurious match overwrite a correct key.
if found {
break;
}
}
if found {
for (k, &s) in key.iter_mut().zip(seed.iter()) {
*k ^= s;
}
Some(key)
} else {
None
}
}
/// Recover the CSS title key from a scrambled sector using a known plaintext
/// for the encrypted region.
///
/// `plain` is the expected plaintext at byte 0x80 (at least 10 bytes).
/// Returns the recovered key only if it actually descrambles the sector back
/// to `plain` — guarding against the rare spurious LFSR-seed match.
pub fn recover_title_key(sector: &[u8], plain: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_BYTES || plain.len() < 10 {
return None;
}
if sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
let seed: [u8; 5] = [
sector[SEED_OFFSET],
sector[SEED_OFFSET + 1],
sector[SEED_OFFSET + 2],
sector[SEED_OFFSET + 3],
sector[SEED_OFFSET + 4],
];
let crypted = &sector[ENCRYPTED_START..ENCRYPTED_START + 10];
let key = recover_title_key_from_plain(crypted, plain, &seed)?;
if descramble_matches(sector, &key, plain) {
Some(key)
} else {
None
}
}
/// Verify a title key by descrambling a copy of `sector` and checking the
/// known plaintext reappears at byte 0x80.
fn descramble_matches(sector: &[u8], title: &[u8; 5], plain: &[u8]) -> bool {
let mut test = sector.to_vec();
test[FLAG_BYTE] |= 0x10; // ensure scramble flag set for the descrambler
descramble_sector(title, &mut test);
let n = plain.len().min(SECTOR_BYTES - ENCRYPTED_START);
test[ENCRYPTED_START..ENCRYPTED_START + n] == plain[..n]
}
/// AttackPattern: find a repeating pattern just before the encrypted region
/// and assume the plaintext at 0x80 continues it.
///
/// Functionally-equivalent port of libdvdcss `AttackPattern` (css.c) — finds the
/// same periodic cribs on real DVD data, though its byte-comparison anchor
/// differs from the C on phase-misaligned runs. Scans cleartext
/// `sec[0x00..0x80]` for the longest run that repeats with a cycle length in
/// 2..0x2F. If the run is long enough (`plen > 3` and at least two full
/// cycles), the known plaintext at 0x80 is taken to be the periodic run
/// continuing forward, and [`recover_title_key_from_plain`] is applied.
pub fn crack_title_key(sector: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_BYTES {
return None;
}
if sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
// Runaway guard: a single sector's crack is a bounded 2^16 LFSR search and
// should finish in well under a second on any modern CPU. If it ever
// exceeds ~2s wall-clock, something pathological is happening — log it so a
// hang is never silent.
let crack_t0 = std::time::Instant::now();
let result = crack_title_key_inner(sector);
let elapsed = crack_t0.elapsed();
if elapsed.as_secs_f64() > 2.0 {
tracing::warn!(
target: "freemkv::css",
elapsed_ms = elapsed.as_millis() as u64,
found = result.is_some(),
"css crack: single-sector recovery exceeded 2s (runaway guard)"
);
}
result
}
/// Inner body of [`crack_title_key`] — the actual AttackPattern search. Split
/// out so the public entry point can wall-clock the whole attempt for the
/// runaway guard without threading a timer through every return path.
/// AttackPattern crib: the predicted 10-byte plaintext at byte 0x80.
///
/// Scans the clear header `sec[0x00..0x80]` (never scrambled) for the longest
/// run that repeats with a cycle length in 2..0x2F. If the run is long enough
/// (`plen > 3` and at least two full cycles), the plaintext at 0x80 is taken to
/// be that periodic run continuing forward. Returns `None` for an unscrambled
/// sector or one with no usable run — such a sector can be neither cracked nor
/// key-validated, only descrambled with an externally-cached key.
///
/// The header is untouched by `descramble_sector`, so the crib is identical
/// before and after descramble: the decrypt path uses it as a per-sector
/// "did the cached key descramble correctly?" oracle (the predicted plaintext
/// must reappear at 0x80), and the cracker uses it as its known plaintext.
pub(crate) fn attack_crib(sector: &[u8]) -> Option<[u8; 10]> {
if sector.len() < SECTOR_BYTES || sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
let mut best_plen: usize = 0;
let mut best_p: usize = 0;
// For all cycle lengths from 2 to 0x2F.
for i in 2usize..0x30 {
// Count bytes that repeat with cycle length i, scanning backward from
// 0x7F. `sec[0x7F - (j % i)] == sec[0x7F - j]`.
let mut j = i + 1;
while j < 0x80 && sector[0x7f - (j % i)] == sector[0x7f - j] {
if j > best_plen {
best_plen = j;
best_p = i;
}
j += 1;
}
}
// Need at least a few repeated bytes and at least one full cycle.
if best_plen > 3 && best_p > 0 && best_plen / best_p >= 2 {
// The known plaintext is the periodic run continuing past 0x80. The
// crib starts at `0x80 - (best_plen/best_p)*best_p` and continues
// through the encrypted region; the bytes at and after 0x80 are the
// predicted plaintext (the pattern repeats with period best_p).
let cycles = best_plen / best_p;
let plain_start = 0x80 - cycles * best_p;
// Each predicted byte is the run sample one or more periods back:
// `sec[plain_start + (i % best_p)]`. For in-run offsets
// (`plain_start + i < 0x80`) the run is exactly periodic, so this
// equals `sec[plain_start + i]`; for offsets at/after 0x80 the raw
// byte is ciphertext, so we MUST wrap within the period rather than
// read it. (Reading `&sec[plain_start..+10]` directly — as before —
// pulled ciphertext into the crib whenever the run covered fewer than
// 10 bytes before 0x80, producing false-negative key recovery.)
let mut plain = [0u8; 10];
for (i, p) in plain.iter_mut().enumerate() {
*p = sector[plain_start + (i % best_p)];
}
Some(plain)
} else {
None
}
}
fn crack_title_key_inner(sector: &[u8]) -> Option<[u8; 5]> {
let plain = attack_crib(sector)?;
let seed: [u8; 5] = [
sector[SEED_OFFSET],
sector[SEED_OFFSET + 1],
sector[SEED_OFFSET + 2],
sector[SEED_OFFSET + 3],
sector[SEED_OFFSET + 4],
];
let crypted = &sector[0x80..0x80 + 10];
if let Some(key) = recover_title_key_from_plain(crypted, &plain, &seed) {
// Verify against the same predicted plaintext.
if descramble_matches(sector, &key, &plain) {
return Some(key);
}
}
None
}
#[cfg(test)]
mod tests {
use super::super::lfsr::scramble_sector;
use super::*;
/// Build a synthetic scrambled sector for a given title key and seed,
/// with `plain` placed as the plaintext at byte 0x80, scrambled with
/// EXACTLY the cipher `descramble_sector` inverts. Returns
/// (scrambled_sector, full_plaintext_body).
fn synth_sector(title_key: &[u8; 5], seed: &[u8; 5], plain: &[u8]) -> (Vec<u8>, Vec<u8>) {
let mut plaintext = vec![0u8; SECTOR_BYTES];
plaintext[0..4].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
plaintext[FLAG_BYTE] = 0x10;
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
plaintext[ENCRYPTED_START..ENCRYPTED_START + plain.len()].copy_from_slice(plain);
let body = plaintext.clone();
// scramble_sector turns the plaintext body into ciphertext and sets
// the scramble flag.
scramble_sector(title_key, &mut plaintext);
(plaintext, body)
}
/// Build a synthetic scrambled sector whose CLEARTEXT (0x00..0x80) ends
/// in a periodic run that continues into the encrypted region — the case
/// `AttackPattern` (crack_title_key) is designed to crack.
fn synth_periodic_sector(
title_key: &[u8; 5],
seed: &[u8; 5],
period: usize,
) -> (Vec<u8>, Vec<u8>) {
let mut plaintext = vec![0u8; SECTOR_BYTES];
plaintext[FLAG_BYTE] = 0x10;
// A clean periodic run occupying the tail of the cleartext header
// (RUN_START..0x80) and continuing into the encrypted region. This
// mirrors a real VOB: a periodic data run just before the scrambled
// part. The run must NOT overlap the seed bytes (0x54..0x59), or the
// AttackPattern detector would break mid-run. The phase is anchored to
// offset 0 so the run is consistent across the 0x80 boundary.
// Just above the seed (0x54..0x59); gives a 39-byte run (0x59..0x80)
// — enough for >=2 cycles of every tested period (<=19).
const RUN_START: usize = 0x59;
let pat: Vec<u8> = (0..period)
.map(|k| (0xA0u8.wrapping_add(k as u8)) ^ 0x5A)
.collect();
for (i, b) in plaintext.iter_mut().enumerate().skip(RUN_START) {
*b = pat[i % period];
}
// Seed sits below the run, undisturbed.
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
let body = plaintext.clone();
scramble_sector(title_key, &mut plaintext);
(plaintext, body)
}
#[test]
fn crack_unscrambled_returns_none() {
let sector = vec![0u8; SECTOR_BYTES];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn crack_too_short_returns_none() {
let sector = vec![0u8; 100];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn recover_needs_min_plain() {
let sector = vec![0u8; SECTOR_BYTES];
let short_plain = [0u8; 4];
assert!(recover_title_key(&sector, &short_plain).is_none());
}
/// The known plaintext used at byte 0x80 for the direct-recovery tests.
/// A realistic MPEG-2 PES header start.
const PES: [u8; 10] = [0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x80, 0x05, 0x21];
/// MANDATORY round-trip (Task C.1): synthesize a scrambled sector for a
/// known (title_key, seed), then assert recover_title_key returns a key
/// that descrambles the body back to plaintext. CSS title-key recovery is
/// well-defined up to keys that scramble identically; we assert the full
/// body round-trips (the true correctness property), and additionally
/// that the EXACT key is returned for the common case.
#[test]
fn recover_round_trips_known_keys() {
let cases: &[([u8; 5], [u8; 5])] = &[
(
[0x42, 0x13, 0x37, 0xBE, 0xEF],
[0x11, 0x22, 0x33, 0x44, 0x55],
),
(
[0x01, 0x02, 0x03, 0x04, 0x05],
[0xDE, 0xAD, 0xBE, 0xEF, 0x42],
),
(
[0xFE, 0xDC, 0xBA, 0x98, 0x76],
[0x00, 0xFF, 0x80, 0x7F, 0x01],
),
(
[0x9A, 0x78, 0x56, 0x34, 0x12],
[0xA5, 0x5A, 0x0F, 0xF0, 0xCC],
),
(
[0xFF, 0xFF, 0xFF, 0xFF, 0xFF],
[0x01, 0x01, 0x01, 0x01, 0x01],
),
];
for (title_key, seed) in cases {
let (mut sector, body) = synth_sector(title_key, seed, &PES);
let recovered =
recover_title_key(&sector, &PES).expect("recover_title_key returned None");
descramble_sector(&recovered, &mut sector);
assert_eq!(
&sector[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"recovered key did not descramble the full body for \
title={title_key:02x?} seed={seed:02x?}"
);
}
}
/// MANDATORY (Task C.1): the AttackPattern entry point crack_title_key —
/// no plaintext supplied — recovers a round-tripping key when the
/// cleartext ends in a periodic run that continues into 0x80.
#[test]
fn crack_title_key_recovers_via_attack_pattern() {
for &period in &[2usize, 3, 5, 8, 16] {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let seed = [0x11, 0x22, 0x33, 0x44, 0x55];
let (sector, body) = synth_periodic_sector(&title_key, &seed, period);
let cracked = crack_title_key(&sector)
.unwrap_or_else(|| panic!("crack_title_key returned None for period {period}"));
let mut test = sector.clone();
descramble_sector(&cracked, &mut test);
assert_eq!(
&test[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"crack_title_key key did not round-trip the body (period {period})"
);
}
}
/// recover_title_key_from_plain inverts dvdcss_unscramble exactly: scramble
/// a known body, hand back the keystream-derived key, and the recovered
/// key (XOR-back included) reproduces the plaintext.
#[test]
fn recovered_key_descrambles_back_to_plaintext() {
let cases: &[([u8; 5], [u8; 5])] = &[
(
[0x42, 0x13, 0x37, 0xBE, 0xEF],
[0x11, 0x22, 0x33, 0x44, 0x55],
),
(
[0x9A, 0x78, 0x56, 0x34, 0x12],
[0xA5, 0x5A, 0x0F, 0xF0, 0xCC],
),
(
[0xFF, 0xFF, 0xFF, 0xFF, 0xFF],
[0x01, 0x01, 0x01, 0x01, 0x01],
),
];
for (title_key, seed) in cases {
let (mut sector, body) = synth_sector(title_key, seed, &PES);
let recovered =
recover_title_key(&sector, &PES).expect("recover_title_key returned None");
descramble_sector(&recovered, &mut sector);
assert_eq!(
&sector[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"descramble with recovered key did not reproduce the body \
for title={title_key:02x?} seed={seed:02x?}"
);
}
}
// ── early-return guards ────────────────────────────────────────────────
#[test]
fn recover_rejects_sector_one_byte_short() {
let mut sector = vec![0u8; SECTOR_BYTES - 1];
sector[FLAG_BYTE] = 0x30;
assert!(recover_title_key(&sector, &PES).is_none());
}
#[test]
fn recover_rejects_unscrambled_sector() {
let sector = vec![0x00u8; SECTOR_BYTES];
assert!(recover_title_key(&sector, &PES).is_none());
}
#[test]
fn recover_high_flag_bits_are_not_scramble() {
for &flag in &[0x40u8, 0x80, 0xC0] {
let mut sector = vec![0x11u8; SECTOR_BYTES];
sector[FLAG_BYTE] = flag;
assert!(
recover_title_key(&sector, &PES).is_none(),
"flag {flag:#04x} has scramble bits clear; recover must return None"
);
}
}
#[test]
fn crack_high_flag_bits_are_not_scramble() {
for &flag in &[0x40u8, 0x80, 0xC0] {
let mut sector = vec![0x11u8; SECTOR_BYTES];
sector[FLAG_BYTE] = flag;
assert!(
crack_title_key(&sector).is_none(),
"flag {flag:#04x} clear scramble bits -> crack must return None"
);
}
}
#[test]
fn crack_rejects_sector_one_byte_short() {
let mut sector = vec![0u8; SECTOR_BYTES - 1];
if sector.len() > FLAG_BYTE {
sector[FLAG_BYTE] = 0x30;
}
assert!(crack_title_key(&sector).is_none());
}
/// crack_title_key must never panic on a fully scrambled sector with
/// arbitrary (non-periodic) content — it just returns None.
#[test]
fn crack_full_path_never_panics() {
for seed in 0u32..3 {
let mut sector = vec![0u8; SECTOR_BYTES];
sector[FLAG_BYTE] = 0x30;
let mut x = seed.wrapping_mul(2_654_435_761).wrapping_add(7);
for b in sector.iter_mut().skip(0x80) {
x = x.wrapping_mul(1_103_515_245).wrapping_add(12_345);
*b = (x >> 16) as u8;
}
for (i, b) in sector[SEED_OFFSET..SEED_OFFSET + 5].iter_mut().enumerate() {
*b = (seed.wrapping_add(i as u32) ^ 0xA5) as u8;
}
let _ = crack_title_key(&sector);
}
}
}
+285
View File
@@ -0,0 +1,285 @@
//! CSS specification tables — mathematical constants defining the cipher.
//!
//! These 5 tables are the fixed permutations and substitutions of the
//! Content Scramble System. They are mathematical constants derived from
//! the CSS specification, published in academic literature since 1999.
/// Table 1: byte substitution used in key mangling and sector seed processing.
pub const TAB1: [u8; 256] = [
0x33, 0x73, 0x3b, 0x26, 0x63, 0x23, 0x6b, 0x76, 0x3e, 0x7e, 0x36, 0x2b, 0x6e, 0x2e, 0x66, 0x7b,
0xd3, 0x93, 0xdb, 0x06, 0x43, 0x03, 0x4b, 0x96, 0xde, 0x9e, 0xd6, 0x0b, 0x4e, 0x0e, 0x46, 0x9b,
0x57, 0x17, 0x5f, 0x82, 0xc7, 0x87, 0xcf, 0x12, 0x5a, 0x1a, 0x52, 0x8f, 0xca, 0x8a, 0xc2, 0x1f,
0xd9, 0x99, 0xd1, 0x00, 0x49, 0x09, 0x41, 0x90, 0xd8, 0x98, 0xd0, 0x01, 0x48, 0x08, 0x40, 0x91,
0x3d, 0x7d, 0x35, 0x24, 0x6d, 0x2d, 0x65, 0x74, 0x3c, 0x7c, 0x34, 0x25, 0x6c, 0x2c, 0x64, 0x75,
0xdd, 0x9d, 0xd5, 0x04, 0x4d, 0x0d, 0x45, 0x94, 0xdc, 0x9c, 0xd4, 0x05, 0x4c, 0x0c, 0x44, 0x95,
0x59, 0x19, 0x51, 0x80, 0xc9, 0x89, 0xc1, 0x10, 0x58, 0x18, 0x50, 0x81, 0xc8, 0x88, 0xc0, 0x11,
0xd7, 0x97, 0xdf, 0x02, 0x47, 0x07, 0x4f, 0x92, 0xda, 0x9a, 0xd2, 0x0f, 0x4a, 0x0a, 0x42, 0x9f,
0x53, 0x13, 0x5b, 0x86, 0xc3, 0x83, 0xcb, 0x16, 0x5e, 0x1e, 0x56, 0x8b, 0xce, 0x8e, 0xc6, 0x1b,
0xb3, 0xf3, 0xbb, 0xa6, 0xe3, 0xa3, 0xeb, 0xf6, 0xbe, 0xfe, 0xb6, 0xab, 0xee, 0xae, 0xe6, 0xfb,
0x37, 0x77, 0x3f, 0x22, 0x67, 0x27, 0x6f, 0x72, 0x3a, 0x7a, 0x32, 0x2f, 0x6a, 0x2a, 0x62, 0x7f,
0xb9, 0xf9, 0xb1, 0xa0, 0xe9, 0xa9, 0xe1, 0xf0, 0xb8, 0xf8, 0xb0, 0xa1, 0xe8, 0xa8, 0xe0, 0xf1,
0x5d, 0x1d, 0x55, 0x84, 0xcd, 0x8d, 0xc5, 0x14, 0x5c, 0x1c, 0x54, 0x85, 0xcc, 0x8c, 0xc4, 0x15,
0xbd, 0xfd, 0xb5, 0xa4, 0xed, 0xad, 0xe5, 0xf4, 0xbc, 0xfc, 0xb4, 0xa5, 0xec, 0xac, 0xe4, 0xf5,
0x39, 0x79, 0x31, 0x20, 0x69, 0x29, 0x61, 0x70, 0x38, 0x78, 0x30, 0x21, 0x68, 0x28, 0x60, 0x71,
0xb7, 0xf7, 0xbf, 0xa2, 0xe7, 0xa7, 0xef, 0xf2, 0xba, 0xfa, 0xb2, 0xaf, 0xea, 0xaa, 0xe2, 0xff,
];
/// Table 2: LFSR1 high-byte feedback permutation.
///
/// Byte-identical to libdvdcss `p_css_tab2` (csstables.h).
pub const TAB2: [u8; 256] = [
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x09, 0x08, 0x0b, 0x0a, 0x0d, 0x0c, 0x0f, 0x0e,
0x12, 0x13, 0x10, 0x11, 0x16, 0x17, 0x14, 0x15, 0x1b, 0x1a, 0x19, 0x18, 0x1f, 0x1e, 0x1d, 0x1c,
0x24, 0x25, 0x26, 0x27, 0x20, 0x21, 0x22, 0x23, 0x2d, 0x2c, 0x2f, 0x2e, 0x29, 0x28, 0x2b, 0x2a,
0x36, 0x37, 0x34, 0x35, 0x32, 0x33, 0x30, 0x31, 0x3f, 0x3e, 0x3d, 0x3c, 0x3b, 0x3a, 0x39, 0x38,
0x49, 0x48, 0x4b, 0x4a, 0x4d, 0x4c, 0x4f, 0x4e, 0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47,
0x5b, 0x5a, 0x59, 0x58, 0x5f, 0x5e, 0x5d, 0x5c, 0x52, 0x53, 0x50, 0x51, 0x56, 0x57, 0x54, 0x55,
0x6d, 0x6c, 0x6f, 0x6e, 0x69, 0x68, 0x6b, 0x6a, 0x64, 0x65, 0x66, 0x67, 0x60, 0x61, 0x62, 0x63,
0x7f, 0x7e, 0x7d, 0x7c, 0x7b, 0x7a, 0x79, 0x78, 0x76, 0x77, 0x74, 0x75, 0x72, 0x73, 0x70, 0x71,
0x92, 0x93, 0x90, 0x91, 0x96, 0x97, 0x94, 0x95, 0x9b, 0x9a, 0x99, 0x98, 0x9f, 0x9e, 0x9d, 0x9c,
0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x89, 0x88, 0x8b, 0x8a, 0x8d, 0x8c, 0x8f, 0x8e,
0xb6, 0xb7, 0xb4, 0xb5, 0xb2, 0xb3, 0xb0, 0xb1, 0xbf, 0xbe, 0xbd, 0xbc, 0xbb, 0xba, 0xb9, 0xb8,
0xa4, 0xa5, 0xa6, 0xa7, 0xa0, 0xa1, 0xa2, 0xa3, 0xad, 0xac, 0xaf, 0xae, 0xa9, 0xa8, 0xab, 0xaa,
0xdb, 0xda, 0xd9, 0xd8, 0xdf, 0xde, 0xdd, 0xdc, 0xd2, 0xd3, 0xd0, 0xd1, 0xd6, 0xd7, 0xd4, 0xd5,
0xc9, 0xc8, 0xcb, 0xca, 0xcd, 0xcc, 0xcf, 0xce, 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7,
0xff, 0xfe, 0xfd, 0xfc, 0xfb, 0xfa, 0xf9, 0xf8, 0xf6, 0xf7, 0xf4, 0xf5, 0xf2, 0xf3, 0xf0, 0xf1,
0xed, 0xec, 0xef, 0xee, 0xe9, 0xe8, 0xeb, 0xea, 0xe4, 0xe5, 0xe6, 0xe7, 0xe0, 0xe1, 0xe2, 0xe3,
];
/// Table 3: LFSR1 9-bit low-word feedback table (512 entries).
///
/// Byte-identical to libdvdcss `p_css_tab3` (csstables.h): the 8-value
/// block `BASE[i & 7]` repeated 64 times. The CSS LFSR1 step indexes this
/// table with the 9-bit low register (0x100..=0x1FF), but only the low 3
/// bits select the output — the high bits are ignored, hence the constant
/// blocks. The 512-entry width simply lets the 9-bit index be used without
/// masking.
pub const TAB3: [u8; 512] = [
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
];
/// Table 4: LFSR0 byte permutation (used in initialization and output).
pub const TAB4: [u8; 256] = [
0x00, 0x80, 0x40, 0xc0, 0x20, 0xa0, 0x60, 0xe0, 0x10, 0x90, 0x50, 0xd0, 0x30, 0xb0, 0x70, 0xf0,
0x08, 0x88, 0x48, 0xc8, 0x28, 0xa8, 0x68, 0xe8, 0x18, 0x98, 0x58, 0xd8, 0x38, 0xb8, 0x78, 0xf8,
0x04, 0x84, 0x44, 0xc4, 0x24, 0xa4, 0x64, 0xe4, 0x14, 0x94, 0x54, 0xd4, 0x34, 0xb4, 0x74, 0xf4,
0x0c, 0x8c, 0x4c, 0xcc, 0x2c, 0xac, 0x6c, 0xec, 0x1c, 0x9c, 0x5c, 0xdc, 0x3c, 0xbc, 0x7c, 0xfc,
0x02, 0x82, 0x42, 0xc2, 0x22, 0xa2, 0x62, 0xe2, 0x12, 0x92, 0x52, 0xd2, 0x32, 0xb2, 0x72, 0xf2,
0x0a, 0x8a, 0x4a, 0xca, 0x2a, 0xaa, 0x6a, 0xea, 0x1a, 0x9a, 0x5a, 0xda, 0x3a, 0xba, 0x7a, 0xfa,
0x06, 0x86, 0x46, 0xc6, 0x26, 0xa6, 0x66, 0xe6, 0x16, 0x96, 0x56, 0xd6, 0x36, 0xb6, 0x76, 0xf6,
0x0e, 0x8e, 0x4e, 0xce, 0x2e, 0xae, 0x6e, 0xee, 0x1e, 0x9e, 0x5e, 0xde, 0x3e, 0xbe, 0x7e, 0xfe,
0x01, 0x81, 0x41, 0xc1, 0x21, 0xa1, 0x61, 0xe1, 0x11, 0x91, 0x51, 0xd1, 0x31, 0xb1, 0x71, 0xf1,
0x09, 0x89, 0x49, 0xc9, 0x29, 0xa9, 0x69, 0xe9, 0x19, 0x99, 0x59, 0xd9, 0x39, 0xb9, 0x79, 0xf9,
0x05, 0x85, 0x45, 0xc5, 0x25, 0xa5, 0x65, 0xe5, 0x15, 0x95, 0x55, 0xd5, 0x35, 0xb5, 0x75, 0xf5,
0x0d, 0x8d, 0x4d, 0xcd, 0x2d, 0xad, 0x6d, 0xed, 0x1d, 0x9d, 0x5d, 0xdd, 0x3d, 0xbd, 0x7d, 0xfd,
0x03, 0x83, 0x43, 0xc3, 0x23, 0xa3, 0x63, 0xe3, 0x13, 0x93, 0x53, 0xd3, 0x33, 0xb3, 0x73, 0xf3,
0x0b, 0x8b, 0x4b, 0xcb, 0x2b, 0xab, 0x6b, 0xeb, 0x1b, 0x9b, 0x5b, 0xdb, 0x3b, 0xbb, 0x7b, 0xfb,
0x07, 0x87, 0x47, 0xc7, 0x27, 0xa7, 0x67, 0xe7, 0x17, 0x97, 0x57, 0xd7, 0x37, 0xb7, 0x77, 0xf7,
0x0f, 0x8f, 0x4f, 0xcf, 0x2f, 0xaf, 0x6f, 0xef, 0x1f, 0x9f, 0x5f, 0xdf, 0x3f, 0xbf, 0x7f, 0xff,
];
/// Table 5: LFSR1 output permutation used in the keystream combiner.
/// `TAB5[i] == TAB4[i] ^ 0xFF` (bitwise complement of the TAB4 bit-reversal
/// table). Applied on the normal descramble/recrypt path (lfsr.rs) as well as
/// in the key-recovery fallback (crack.rs).
pub const TAB5: [u8; 256] = [
0xff, 0x7f, 0xbf, 0x3f, 0xdf, 0x5f, 0x9f, 0x1f, 0xef, 0x6f, 0xaf, 0x2f, 0xcf, 0x4f, 0x8f, 0x0f,
0xf7, 0x77, 0xb7, 0x37, 0xd7, 0x57, 0x97, 0x17, 0xe7, 0x67, 0xa7, 0x27, 0xc7, 0x47, 0x87, 0x07,
0xfb, 0x7b, 0xbb, 0x3b, 0xdb, 0x5b, 0x9b, 0x1b, 0xeb, 0x6b, 0xab, 0x2b, 0xcb, 0x4b, 0x8b, 0x0b,
0xf3, 0x73, 0xb3, 0x33, 0xd3, 0x53, 0x93, 0x13, 0xe3, 0x63, 0xa3, 0x23, 0xc3, 0x43, 0x83, 0x03,
0xfd, 0x7d, 0xbd, 0x3d, 0xdd, 0x5d, 0x9d, 0x1d, 0xed, 0x6d, 0xad, 0x2d, 0xcd, 0x4d, 0x8d, 0x0d,
0xf5, 0x75, 0xb5, 0x35, 0xd5, 0x55, 0x95, 0x15, 0xe5, 0x65, 0xa5, 0x25, 0xc5, 0x45, 0x85, 0x05,
0xf9, 0x79, 0xb9, 0x39, 0xd9, 0x59, 0x99, 0x19, 0xe9, 0x69, 0xa9, 0x29, 0xc9, 0x49, 0x89, 0x09,
0xf1, 0x71, 0xb1, 0x31, 0xd1, 0x51, 0x91, 0x11, 0xe1, 0x61, 0xa1, 0x21, 0xc1, 0x41, 0x81, 0x01,
0xfe, 0x7e, 0xbe, 0x3e, 0xde, 0x5e, 0x9e, 0x1e, 0xee, 0x6e, 0xae, 0x2e, 0xce, 0x4e, 0x8e, 0x0e,
0xf6, 0x76, 0xb6, 0x36, 0xd6, 0x56, 0x96, 0x16, 0xe6, 0x66, 0xa6, 0x26, 0xc6, 0x46, 0x86, 0x06,
0xfa, 0x7a, 0xba, 0x3a, 0xda, 0x5a, 0x9a, 0x1a, 0xea, 0x6a, 0xaa, 0x2a, 0xca, 0x4a, 0x8a, 0x0a,
0xf2, 0x72, 0xb2, 0x32, 0xd2, 0x52, 0x92, 0x12, 0xe2, 0x62, 0xa2, 0x22, 0xc2, 0x42, 0x82, 0x02,
0xfc, 0x7c, 0xbc, 0x3c, 0xdc, 0x5c, 0x9c, 0x1c, 0xec, 0x6c, 0xac, 0x2c, 0xcc, 0x4c, 0x8c, 0x0c,
0xf4, 0x74, 0xb4, 0x34, 0xd4, 0x54, 0x94, 0x14, 0xe4, 0x64, 0xa4, 0x24, 0xc4, 0x44, 0x84, 0x04,
0xf8, 0x78, 0xb8, 0x38, 0xd8, 0x58, 0x98, 0x18, 0xe8, 0x68, 0xa8, 0x28, 0xc8, 0x48, 0x88, 0x08,
0xf0, 0x70, 0xb0, 0x30, 0xd0, 0x50, 0x90, 0x10, 0xe0, 0x60, 0xa0, 0x20, 0xc0, 0x40, 0x80, 0x00,
];
#[cfg(test)]
mod tests {
use super::*;
/// Pins the documented relationship `TAB5[i] == TAB4[i] ^ 0xFF` so the
/// table doc cannot drift from the data.
#[test]
fn tab5_is_complement_of_tab4() {
for i in 0..256 {
assert_eq!(
TAB5[i],
TAB4[i] ^ 0xFF,
"TAB5[{i:#04x}] != TAB4[{i:#04x}] ^ 0xFF"
);
}
}
/// TAB1 is a bijection on 0..256. CSS uses it as an invertible output
/// permutation in css_DecryptKey's chained-XOR rounds; if two inputs
/// collided, the key mangling would not be invertible.
///
/// Mutation: duplicate any value (e.g. set TAB1[1] = TAB1[0]) -> the
/// "maps two inputs" assert fires.
#[test]
fn tab1_is_a_permutation() {
let mut seen = [false; 256];
for (i, &v) in TAB1.iter().enumerate() {
assert!(
!seen[v as usize],
"TAB1 maps two inputs to {v:#04x} (collision at index {i:#04x})"
);
seen[v as usize] = true;
}
}
/// TAB1's fixed structural anchors from the CSS spec table:
/// TAB1[0x00] == 0x33 and the inverse TAB1[0x33] == 0x00. These two
/// entries are the canonical first-row / inverse-lookup landmarks of the
/// published CSS TAB1 and pin the table's orientation.
///
/// Grounding: CSS specification TAB1, row 0 col 0 = 0x33; index 0x33
/// (row 3 col 3) = 0x00.
/// Mutation: change the first literal `0x33` in TAB1 -> first assert fails.
#[test]
fn tab1_known_spec_anchors() {
assert_eq!(TAB1[0x00], 0x33, "TAB1[0] is the published 0x33");
assert_eq!(TAB1[0x33], 0x00, "TAB1[0x33] is the published 0x00");
}
/// TAB2 is a permutation of 0..256 (it is the LFSR1 high-byte feedback
/// substitution). A non-bijective TAB2 would bias the LFSR1 keystream.
///
/// Mutation: set TAB2[8] = 0x00 (collides with TAB2[0]) -> assert fires.
#[test]
fn tab2_is_a_permutation() {
let mut seen = [false; 256];
for (i, &v) in TAB2.iter().enumerate() {
assert!(
!seen[v as usize],
"TAB2 maps two inputs to {v:#04x} (collision at index {i:#04x})"
);
seen[v as usize] = true;
}
}
/// TAB3 is the libdvdcss `p_css_tab3`: the 8-value feedback block
/// BASE = [0x00,0x24,0x49,0x6d,0x92,0xb6,0xdb,0xff]
/// repeated 64 times — `TAB3[i] == BASE[i & 7]`. The high bits of the
/// 9-bit index do not affect the output (libdvdcss's LFSR1 step indexes
/// with the full 9-bit low register but only `& 7` matters). This pins
/// all 512 entries to the published table.
///
/// Mutation: flip any single byte in the TAB3 literal -> the formula
/// check fails at that index.
#[test]
fn tab3_matches_lfsr1_generating_formula() {
const BASE: [u8; 8] = [0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff];
for i in 0..512usize {
let expected = BASE[i & 7];
assert_eq!(
TAB3[i], expected,
"TAB3[{i:#05x}] = {:#04x}, formula BASE[i&7] = {expected:#04x}",
TAB3[i]
);
}
}
/// TAB4 is the exact bit-reversal of each byte (CSS uses it to permute
/// LFSR0 bytes on seed and output). TAB4[b] reverses b's 8 bits MSB<->LSB.
/// Therefore it is also an involution: TAB4[TAB4[b]] == b.
///
/// Grounding: TAB4[0x01]=0x80, TAB4[0x80]=0x01, TAB4[0x00]=0x00,
/// TAB4[0xFF]=0xFF.
/// Mutation: set TAB4[1] = 0x40 (not the reversal 0x80) -> bit-reversal
/// check fails at index 1.
#[test]
fn tab4_is_exact_bit_reversal_and_involution() {
for b in 0u16..256 {
let rev = (0..8).fold(0u8, |acc, k| acc | (((b as u8 >> k) & 1) << (7 - k)));
assert_eq!(
TAB4[b as usize], rev,
"TAB4[{b:#04x}] is not the bit-reversal {rev:#04x}"
);
}
for b in 0..256usize {
assert_eq!(
TAB4[TAB4[b] as usize], b as u8,
"TAB4 not an involution at {b:#04x}"
);
}
// Spec landmark entries.
assert_eq!(TAB4[0x01], 0x80);
assert_eq!(TAB4[0x80], 0x01);
assert_eq!(TAB4[0x00], 0x00);
assert_eq!(TAB4[0xFF], 0xFF);
}
/// TAB4 is a permutation (bit-reversal is bijective). Distinct from the
/// reversal test: a table that is "reversal except two swapped entries"
/// would still be a permutation, and a table that is "reversal except one
/// duplicated entry" would fail this but might pass a sampled reversal
/// check — the two tests pin different failure modes.
///
/// Mutation: set TAB4[2] = TAB4[1] -> permutation assert fires.
#[test]
fn tab4_is_a_permutation() {
let mut seen = [false; 256];
for &v in TAB4.iter() {
assert!(!seen[v as usize], "TAB4 maps two inputs to {v:#04x}");
seen[v as usize] = true;
}
}
/// TAB5 is also a permutation (complement of a bijection is a bijection)
/// and its own self-consistency landmark: TAB5[0x00] == 0xFF (TAB4[0]^0xFF)
/// and TAB5[0xFF] == 0x00 (TAB4[0xFF]^0xFF). Pins orientation independent
/// of the complement-loop test.
///
/// Mutation: change the first TAB5 literal 0xff -> 0xfe -> the landmark
/// and permutation checks both catch it.
#[test]
fn tab5_is_permutation_with_anchors() {
let mut seen = [false; 256];
for &v in TAB5.iter() {
assert!(!seen[v as usize], "TAB5 maps two inputs to {v:#04x}");
seen[v as usize] = true;
}
assert_eq!(TAB5[0x00], 0xFF, "TAB5[0] = TAB4[0]^0xFF = 0xFF");
assert_eq!(TAB5[0xFF], 0x00, "TAB5[0xFF] = TAB4[0xFF]^0xFF = 0x00");
}
}
+1374
View File
File diff suppressed because it is too large Load Diff
+753
View File
@@ -0,0 +1,753 @@
//! Structured scan diagnostics — the `--log-level 3` self-diagnosing dump.
//!
//! A bug report log must be self-diagnosing: everything needed to explain
//! *why* freemkv made the choices it did at scan must be in the log, in a
//! compact, machine-parseable form. This module emits one terse line per row
//! (title, cell, stream, decision) under the `tracing` target
//! `freemkv::diag`, which the CLI routes to `log.txt` when `--log-level 3`
//! (debug) is set.
//!
//! Format conventions (stable, greppable):
//! - Every line is prefixed by a `tag=` so a log scraper can filter
//! (`disc`, `title`, `dvd.cell`, `dvd.vattr`, `dvd.aattr`, `bd.clip`,
//! `bd.mark`, `aacs`, `stream`, `decision`).
//! - Raw bytes are shown as `0xNN` next to their decode so a wrong decode
//! is obvious against the raw value.
//! - This module only READS already-parsed scan state — it never re-reads
//! the disc and never mutates anything.
//!
//! The DVD per-cell table (with the raw cell-category byte) is emitted from
//! the IFO scan itself ([`dump_dvd_cells`]), because the per-cell
//! `ifo::DvdCell` detail is lowered away before the `Disc` is built. The
//! `Disc`-level dump ([`dump_disc`]) covers everything that survives
//! lowering: titles, streams, the picked main feature, and AACS state.
use crate::disc::{
AudioChannels, ColorSpace, Disc, DiscTitle, FrameRate, HdrFormat, Resolution, SampleRate,
Stream,
};
use crate::ifo::{CellCategory, DvdTitle};
const DIAG: &str = "freemkv::diag";
// ── small format helpers (pure, unit-testable) ──────────────────────────────
/// Compact name for a [`Resolution`] with the interlace marker preserved.
pub fn res_str(r: Resolution) -> &'static str {
match r {
Resolution::R480i => "480i",
Resolution::R480p => "480p",
Resolution::R576i => "576i",
Resolution::R576p => "576p",
Resolution::R720p => "720p",
Resolution::R1080i => "1080i",
Resolution::R1080p => "1080p",
Resolution::R2160p => "2160p",
Resolution::R4320p => "4320p",
Resolution::Unknown => "res?",
}
}
/// Frames-per-second string for a [`FrameRate`].
pub fn fps_str(f: FrameRate) -> &'static str {
match f {
FrameRate::F23_976 => "23.976",
FrameRate::F24 => "24",
FrameRate::F25 => "25",
FrameRate::F29_97 => "29.97",
FrameRate::F30 => "30",
FrameRate::F50 => "50",
FrameRate::F59_94 => "59.94",
FrameRate::F60 => "60",
FrameRate::Unknown => "fps?",
}
}
/// PAL/NTSC field-rate family inferred from the frame rate (DVD has no
/// explicit field, so this is the colour/standard the muxer stamps).
pub fn tv_system_str(f: FrameRate) -> &'static str {
match f {
FrameRate::F25 | FrameRate::F50 => "PAL",
FrameRate::F23_976 | FrameRate::F29_97 | FrameRate::F59_94 => "NTSC",
_ => "",
}
}
/// CICP-ish short name for a [`ColorSpace`].
pub fn color_str(c: ColorSpace) -> &'static str {
match c {
ColorSpace::Bt709 => "BT.709",
ColorSpace::Bt2020 => "BT.2020",
ColorSpace::Bt470bg => "BT.470BG",
ColorSpace::Smpte170m => "SMPTE-170M",
ColorSpace::Unknown => "color?",
}
}
/// HDR format short name.
pub fn hdr_str(h: HdrFormat) -> &'static str {
match h {
HdrFormat::Sdr => "SDR",
HdrFormat::Hdr10 => "HDR10",
HdrFormat::Hdr10Plus => "HDR10+",
HdrFormat::DolbyVision => "DoVi",
HdrFormat::Hlg => "HLG",
}
}
/// Channel count from an [`AudioChannels`] layout (what lands in the MKV
/// `Channels` element).
pub fn channel_count(ch: AudioChannels) -> u8 {
match ch {
AudioChannels::Mono => 1,
AudioChannels::Stereo => 2,
AudioChannels::Stereo21 => 3,
AudioChannels::Quad => 4,
AudioChannels::Surround50 => 5,
AudioChannels::Surround51 => 6,
AudioChannels::Surround61 => 7,
AudioChannels::Surround71 => 8,
AudioChannels::Unknown => 0,
}
}
/// Sample-rate in Hz for a [`SampleRate`].
pub fn sample_rate_hz(s: SampleRate) -> u32 {
match s {
SampleRate::S44_1 => 44100,
SampleRate::S48 => 48000,
SampleRate::S88_2 => 88200,
SampleRate::S96 => 96000,
SampleRate::S176_4 => 176400,
SampleRate::S192 => 192000,
SampleRate::S48_96 => 96000,
SampleRate::S48_192 => 192000,
SampleRate::Unknown => 0,
}
}
// ── DVD cell-category dump (from the IFO scan, pre-lowering) ─────────────────
/// One formatted cell row for the DVD per-PGC cell table. Returned as a
/// string so it can be unit-tested without a logger.
///
/// Columns: `idx`, raw category (`cat=0xNN`) + decoded fields, first/last
/// sector, duration, and the keep/drop verdict from the bug-4 leading-cell
/// filter.
pub fn dvd_cell_row(idx: usize, cell: &crate::ifo::DvdCell, dropped: bool) -> String {
let c = CellCategory::decode(cell.category);
// Per-cell keep/skip REASON (self-sufficient bug log): a dropped cell is a
// leading secondary angle/interleave block piece; a kept cell is either the
// first feature cell or genuine feature content. This makes the
// leading-cell-filter decision auditable from the log without the disc.
let verdict = if dropped {
"DROP(leading-secondary-block-piece)"
} else if c.is_secondary_block_piece() {
// Kept despite being a secondary piece — only happens past the leading
// run (the filter stops at the first plain feature cell).
"keep(feature-body)"
} else {
"keep(plain-feature)"
};
format!(
"tag=dvd.cell idx={idx} cat=0x{:02X} block_mode={} block_type={} \
seamless={} ilv={} stc={} angle={} plain={} first={} last={} dur={:.1}s {}",
cell.category,
c.block_mode,
c.block_type,
c.seamless_play as u8,
c.interleaved as u8,
c.stc_discontinuity as u8,
c.seamless_angle as u8,
c.is_plain_feature() as u8,
cell.first_sector,
cell.last_sector,
cell.duration_secs,
verdict,
)
}
/// Emit the per-PGC cell table for one DVD title during the IFO scan.
///
/// `vts`/`title` identify the row group; `title` is the `DvdTitle` whose
/// cells (and bug-4 leading-cell verdict) are dumped. Called from
/// `scan_dvd_titles` while the `DvdTitle` is still in scope (the per-cell
/// category byte is lowered away before the `Disc` exists).
pub fn dump_dvd_cells(vts: u8, title_num: u16, title: &DvdTitle) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
let feature_start = title.feature_start_cell();
tracing::debug!(
target: DIAG,
"tag=dvd.pgc vts={vts} title={title_num} cells={} chapters={} \
dur={:.1}s feature_start_cell={feature_start}",
title.cells.len(),
title.chapters,
title.duration_secs,
);
for (i, cell) in title.cells.iter().enumerate() {
tracing::debug!(target: DIAG, "{}", dvd_cell_row(i, cell, i < feature_start));
}
// Chapter/PTT map (program → cumulative start time).
for (i, &t) in title.chapter_times.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.chap vts={vts} title={title_num} ch={} time={:.1}s",
i + 1,
t,
);
}
}
/// Emit the IFO `video_attr` / `audio_attr` decode for one DVD title set,
/// showing the raw bytes next to their decoded meaning. Called from the IFO
/// scan with the still-parsed `ifo::DvdTitleSet` view.
pub fn dump_dvd_attrs(ts: &crate::ifo::DvdTitleSet) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
tracing::debug!(
target: DIAG,
"tag=dvd.vobs vts={} vob_start_sector={}",
ts.vts_number,
ts.vob_start_sector,
);
let v = &ts.video;
tracing::debug!(
target: DIAG,
"tag=dvd.vattr vts={} codec={:?} res={} aspect={:?} std={:?}",
ts.vts_number,
v.codec,
res_str(v.resolution),
v.aspect,
v.standard,
);
for (i, a) in ts.audio_streams.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.aattr vts={} idx={i} codec={:?} ch={} sr={}Hz lang={:?} sub_id={:?}",
ts.vts_number,
a.codec,
a.channels,
a.sample_rate,
a.language,
a.sub_stream_id.map(|x| format!("0x{x:02X}")),
);
}
for (i, s) in ts.subtitle_streams.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.sattr vts={} idx={i} lang={:?}",
ts.vts_number,
s.language,
);
}
}
/// Emit the ACTUAL per-physical-sub-stream AC-3 channel counts read off the VOB
/// during the mux-time sub-stream probe (the Silence-of-the-Lambs wrong-stream
/// fix). This is the ground truth the IFO nibble is compared against: each row
/// is `sub_id=0x8x channels=N` for a physical `private_stream_1` AC-3 sub-stream
/// whose first frame was decoded. An empty probe (scrambled / unreadable / short
/// VOB) logs a single `probed=0` line so the absence is explicit in a bug log.
///
/// Self-sufficiency: with `tag=dvd.aattr` (the IFO's declared sub_id + claimed
/// channels) and these `tag=dvd.substream` rows (the physical reality), a bug
/// log alone shows whether the ordinal `0x80` actually carries the declared
/// channel layout — no disc needed to diagnose a wrong-substream rip.
pub fn dump_dvd_substream_probe(title_id: u16, probed: &std::collections::BTreeMap<u8, u8>) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
if probed.is_empty() {
tracing::debug!(
target: DIAG,
"tag=dvd.substream title={title_id} probed=0 (no AC-3 sync in feature head — scrambled/unreadable/none)",
);
return;
}
for (sub, ch) in probed {
tracing::debug!(
target: DIAG,
"tag=dvd.substream title={title_id} sub_id=0x{sub:02X} channels={ch} (physical acmod read from VOB)",
);
}
}
// ── MKV TrackEntry dump (the ACTUAL container elements written) ──────────────
/// `true` when the `--log-level 3` diagnostic target is enabled. Hot-path
/// callers (the opening-frame capture) check this once and skip all work when
/// off, so a normal run pays nothing.
pub fn diag_enabled() -> bool {
tracing::enabled!(target: DIAG, tracing::Level::DEBUG)
}
/// Cap on the number of codecPrivate bytes rendered to hex in a `tag=mkv.track`
/// line. The sequence header / avcC / hvcC prefix that matters for diagnosis
/// (resolution, frame rate, profile) is at the front; a multi-KB blob past this
/// is summarised as `..(+NB)` rather than flooding the log.
const CODEC_PRIVATE_HEX_CAP: usize = 64;
/// Render a track's codecPrivate as an uppercase-hex string for the diagnostic
/// line, capped at [`CODEC_PRIVATE_HEX_CAP`] bytes (`..(+NB)` suffix beyond).
/// `None` / empty → `"none"`. Pure (no logging) so it is directly unit-testable.
fn codec_private_hex(cp: Option<&[u8]>) -> String {
match cp {
Some(b) if !b.is_empty() => {
use std::fmt::Write;
let shown = b.len().min(CODEC_PRIVATE_HEX_CAP);
let mut s = String::with_capacity(shown * 2 + 8);
for byte in &b[..shown] {
let _ = write!(s, "{byte:02X}");
}
if b.len() > CODEC_PRIVATE_HEX_CAP {
let _ = write!(s, "..(+{}B)", b.len() - CODEC_PRIVATE_HEX_CAP);
}
s
}
_ => "none".to_string(),
}
}
/// Frame the raw bytes of one captured opening frame for the `.opening.bin` side
/// file: `[track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw bytes]`. Pure
/// (no I/O) so the record layout is directly unit-testable; `record` appends the
/// returned bytes to the side file.
fn frame_record(track_idx: usize, pts_ns: i64, keyframe: bool, data: &[u8]) -> Vec<u8> {
let mut rec = Vec::with_capacity(14 + data.len());
rec.push(track_idx as u8);
rec.push(keyframe as u8);
rec.extend_from_slice(&pts_ns.to_le_bytes());
rec.extend_from_slice(&(data.len() as u32).to_le_bytes());
rec.extend_from_slice(data);
rec
}
/// Emit the MKV `TrackEntry` elements the muxer is about to WRITE for one
/// track — the Windows-fps-class metadata (FlagInterlaced, FieldOrder,
/// DefaultDuration, DefaultDecodedFieldDuration, Display dims) plus the
/// codecPrivate as hex. With this row a bug log alone is enough to verify why
/// Windows Explorer reports a given frame rate for an interlaced SD track: the
/// container values that drive its fps derivation are all present, no disc and
/// no MediaInfo needed.
///
/// `track_number` is the 1-based MKV track number; `track` is the built
/// [`crate::mux::mkv::MkvTrack`] whose fields map one-to-one onto the emitted
/// elements (see `MkvMuxer::new`). No-op unless the diag target is on.
pub fn dump_mkv_track(track_number: u64, track: &crate::mux::mkv::MkvTrack) {
if !diag_enabled() {
return;
}
// codecPrivate as hex (capped so a multi-KB hvcC doesn't flood the log; the
// sequence header / avcC prefix that matters for diagnosis is at the front).
let cp = codec_private_hex(track.codec_private.as_deref());
let field_order = match track.field_order {
crate::mux::ebml::FIELD_ORDER_TFF => "TFF",
crate::mux::ebml::FIELD_ORDER_BFF => "BFF",
_ => "",
};
// FlagInterlaced is only written for video tracks (1=interlaced/2=progressive);
// report what the muxer will emit, or "—" for non-video tracks where the
// element is omitted entirely.
let interlaced = if track.track_type == crate::mux::ebml::TRACK_TYPE_VIDEO {
if track.interlaced {
"1(interlaced)"
} else {
"2(progressive)"
}
} else {
""
};
tracing::debug!(
target: DIAG,
"tag=mkv.track num={track_number} type={} codec={} flag_interlaced={interlaced} \
field_order={field_order} default_duration_ns={} field_duration_ns={} \
pixel={}x{} display={}x{} cp_len={} cp_hex={cp}",
track.track_type,
track.codec_id,
track.default_duration_ns,
track.field_duration_ns,
track.pixel_width,
track.pixel_height,
track.display_width,
track.display_height,
track.codec_private.as_ref().map_or(0, |b| b.len()),
);
}
// ── Opening-frame capture (first ~N coded frames per track → side file) ──────
/// Number of coded frames captured PER TRACK before the capture goes dormant.
/// ~100 frames covers a DVD's first few seconds of every track (the
/// opening-GOP / still-frame / menu window where mid-GOP open or PTS-floor bugs
/// show up) while bounding the side file to a few MB even for HD I-frames.
const OPENING_FRAMES_PER_TRACK: usize = 100;
/// Captures the first [`OPENING_FRAMES_PER_TRACK`] coded frames of EACH track to
/// a side file (`<output>.opening.bin`) and logs a per-frame summary line, so an
/// opening-GOP / menu / mid-GOP-open issue is diagnosable from a future log +
/// side file WITHOUT the disc. Gated to `--log-level 3`: constructed only when
/// the diag target is on, so a normal run never opens the file or records a byte.
///
/// Side-file record framing (so a reader can split it back into frames):
/// `[track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw frame bytes]`.
pub struct OpeningCapture {
file: std::fs::File,
/// Frames captured so far, per track index. Capture for a track stops once
/// its counter reaches [`OPENING_FRAMES_PER_TRACK`].
counts: Vec<usize>,
}
impl OpeningCapture {
/// Open `<output>.opening.bin` next to the MKV output. Returns `None` (no
/// capture) when the diag target is off OR the side file can't be created —
/// a diagnostic must never fail the rip. `track_count` sizes the per-track
/// counters.
pub fn new(output_path: &std::path::Path, track_count: usize) -> Option<Self> {
if !diag_enabled() {
return None;
}
let mut name = output_path.as_os_str().to_os_string();
name.push(".opening.bin");
match std::fs::File::create(&name) {
Ok(file) => {
tracing::debug!(
target: DIAG,
"tag=mkv.opening.open path={:?} per_track_cap={OPENING_FRAMES_PER_TRACK}",
std::path::Path::new(&name),
);
Some(Self {
file,
counts: vec![0; track_count],
})
}
Err(e) => {
tracing::debug!(
target: DIAG,
"tag=mkv.opening.open path={:?} failed={e} (capture disabled, rip unaffected)",
std::path::Path::new(&name),
);
None
}
}
}
/// Record one coded frame for `track_idx` if that track is still under its
/// per-track cap. Writes the framed raw bytes to the side file and logs a
/// one-line summary. A write error disables further capture for the track
/// (counter pinned to the cap) but never propagates — the rip is unaffected.
pub fn record(&mut self, track_idx: usize, pts_ns: i64, keyframe: bool, data: &[u8]) {
let Some(count) = self.counts.get_mut(track_idx) else {
return;
};
if *count >= OPENING_FRAMES_PER_TRACK {
return;
}
use std::io::Write;
let rec = frame_record(track_idx, pts_ns, keyframe, data);
if let Err(e) = self.file.write_all(&rec) {
// Stop trying on this track; a broken side file must not stall mux.
*count = OPENING_FRAMES_PER_TRACK;
tracing::debug!(
target: DIAG,
"tag=mkv.opening.frame track={track_idx} write_failed={e} (capture stopped for track)",
);
return;
}
*count += 1;
tracing::debug!(
target: DIAG,
"tag=mkv.opening.frame track={track_idx} n={count} type={} size={} pts_ns={pts_ns}",
if keyframe { "key" } else { "delta" },
data.len(),
);
}
}
// ── Disc-level dump (post-lowering: titles, streams, decisions, AACS) ────────
/// Emit the full scan diagnostic block for a built [`Disc`]. Terse, one line
/// per row, under target `freemkv::diag` at DEBUG. No-op unless that target
/// is enabled, so it costs nothing when `--log-level 3` is off.
pub fn dump_disc(disc: &Disc) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
tracing::debug!(
target: DIAG,
"tag=disc vol={:?} format={:?} content={:?} cap_sectors={} layers={} titles={} encrypted={}",
disc.volume_id,
disc.format,
disc.content_format,
disc.capacity_sectors,
disc.layers,
disc.titles.len(),
disc.encrypted,
);
dump_aacs(disc);
for (ti, title) in disc.titles.iter().enumerate() {
dump_title(ti, title);
}
// freemkv's top-level DECISION: which title is the main feature.
if let Some(main) = disc.titles.first() {
tracing::debug!(
target: DIAG,
"tag=decision pick=main_feature title_idx=0 playlist={:?} dur={:.1}s \
size={}B clips={} reason=canonical_title_order(fits-disc, fewest-clips, longest, richest-audio)",
main.playlist,
main.duration_secs,
main.size_bytes,
main.clips.len(),
);
}
}
fn dump_aacs(disc: &Disc) {
let Some(a) = disc.aacs.as_ref() else {
if disc.css.is_some() {
tracing::debug!(target: DIAG, "tag=aacs none crypto=CSS(DVD)");
} else if disc.encrypted {
tracing::debug!(target: DIAG, "tag=aacs none crypto=encrypted-no-keys");
} else {
tracing::debug!(target: DIAG, "tag=aacs none crypto=clear");
}
return;
};
// CPS-unit / unit-key counts: at scan `unit_keys` is empty (keys are
// resolved later); the unit-key count is the BE16 in the raw
// Unit_Key_RO.inf if captured. Report both: resolved count and raw len.
tracing::debug!(
target: DIAG,
"tag=aacs version={} bus_enc={} mkb_version={:?} disc_hash={} key_source={:?} \
vuk={} unit_keys_resolved={} uk_ro_bytes={} mkb_bytes={}",
a.version,
a.bus_encryption,
a.mkb_version,
a.disc_hash,
a.key_source.name(),
a.vuk.is_some(),
a.unit_keys.len(),
a.uk_ro.len(),
a.mkb.len(),
);
}
fn dump_title(ti: usize, title: &DiscTitle) {
let (mut nv, mut na, mut ns) = (0u32, 0u32, 0u32);
for s in &title.streams {
match s {
Stream::Video(_) => nv += 1,
Stream::Audio(_) => na += 1,
Stream::Subtitle(_) => ns += 1,
}
}
tracing::debug!(
target: DIAG,
"tag=title idx={ti} playlist={:?} id={} dur={:.1}s size={}B clips={} \
extents={} chapters={} v={nv} a={na} s={ns} fmt={:?}",
title.playlist,
title.playlist_id,
title.duration_secs,
title.size_bytes,
title.clips.len(),
title.extents.len(),
title.chapters.len(),
title.content_format,
);
// Per-clip rows (BD: PlayItem/CLPI; DVD has none).
for (ci, c) in title.clips.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=clip title={ti} idx={ci} id={:?} in={} out={} dur={:.1}s src_packets={}",
c.clip_id,
c.in_time,
c.out_time,
c.duration_secs,
c.source_packets,
);
}
// Per-extent rows (the sectors freemkv will actually rip — the bug-4
// decision is visible here: leading non-feature cells are already gone).
for (ei, e) in title.extents.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=extent title={ti} idx={ei} start_lba={} sectors={}",
e.start_lba,
e.sector_count,
);
}
// freemkv's per-stream DECISIONS (what the muxer will write).
for (si, s) in title.streams.iter().enumerate() {
match s {
Stream::Video(v) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=video pid=0x{:04X} codec={:?} \
res={} interlaced={} fps={} std={} color={} hdr={} aspect={:?} secondary={}",
v.pid,
v.codec,
res_str(v.resolution),
v.resolution.is_interlaced(),
fps_str(v.frame_rate),
tv_system_str(v.frame_rate),
color_str(v.color_space),
hdr_str(v.hdr),
v.display_aspect,
v.secondary,
),
Stream::Audio(a) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=audio pid=0x{:04X} codec={:?} \
channels={}({}) sr={}Hz lang={:?} secondary={}",
a.pid,
a.codec,
a.channels,
channel_count(a.channels),
sample_rate_hz(a.sample_rate),
a.language,
a.secondary,
),
Stream::Subtitle(sub) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=subtitle pid=0x{:04X} codec={:?} \
lang={:?} forced={}",
sub.pid,
sub.codec,
sub.language,
sub.forced,
),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn res_str_keeps_interlace_marker() {
assert_eq!(res_str(Resolution::R576i), "576i");
assert_eq!(res_str(Resolution::R480i), "480i");
assert_eq!(res_str(Resolution::R2160p), "2160p");
}
#[test]
fn fps_and_tv_system() {
assert_eq!(fps_str(FrameRate::F25), "25");
assert_eq!(tv_system_str(FrameRate::F25), "PAL");
assert_eq!(fps_str(FrameRate::F29_97), "29.97");
assert_eq!(tv_system_str(FrameRate::F29_97), "NTSC");
}
#[test]
fn color_and_hdr() {
assert_eq!(color_str(ColorSpace::Bt470bg), "BT.470BG");
assert_eq!(color_str(ColorSpace::Bt2020), "BT.2020");
assert_eq!(hdr_str(HdrFormat::Hdr10), "HDR10");
assert_eq!(hdr_str(HdrFormat::DolbyVision), "DoVi");
assert_eq!(hdr_str(HdrFormat::Sdr), "SDR");
}
#[test]
fn channel_count_matches_layout() {
assert_eq!(channel_count(AudioChannels::Mono), 1);
assert_eq!(channel_count(AudioChannels::Stereo), 2);
assert_eq!(channel_count(AudioChannels::Surround51), 6);
assert_eq!(channel_count(AudioChannels::Surround71), 8);
}
#[test]
fn sample_rate_hz_values() {
assert_eq!(sample_rate_hz(SampleRate::S48), 48000);
assert_eq!(sample_rate_hz(SampleRate::S96), 96000);
}
#[test]
fn codec_private_hex_renders_caps_and_handles_empty() {
// None / empty → "none" (no hex). The Windows-fps diagnosis only needs
// the seq-header prefix, so render it but cap long blobs.
assert_eq!(codec_private_hex(None), "none");
assert_eq!(codec_private_hex(Some(&[])), "none");
// Short blob: full uppercase hex, no suffix. An MPEG-2 seq header starts
// 00 00 01 B3 — exactly what a reader greps for in a bug log.
assert_eq!(
codec_private_hex(Some(&[0x00, 0x00, 0x01, 0xB3])),
"000001B3"
);
// Over the cap: first CODEC_PRIVATE_HEX_CAP bytes + a "..(+NB)" summary.
let big = vec![0xABu8; CODEC_PRIVATE_HEX_CAP + 5];
let s = codec_private_hex(Some(&big));
assert!(s.starts_with(&"AB".repeat(CODEC_PRIVATE_HEX_CAP)), "{s}");
assert!(s.ends_with("..(+5B)"), "{s}");
}
#[test]
fn frame_record_layout_is_parseable() {
// The .opening.bin record framing must round-trip so a future tool can
// split the side file back into frames without the disc:
// [track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw bytes].
let data = [0xDEu8, 0xAD, 0xBE, 0xEF];
let rec = frame_record(2, -40_000_000, true, &data);
assert_eq!(rec.len(), 14 + data.len());
assert_eq!(rec[0], 2, "track index");
assert_eq!(rec[1], 1, "keyframe flag");
assert_eq!(
i64::from_le_bytes(rec[2..10].try_into().unwrap()),
-40_000_000,
"pts_ns survives (signed — opening back-anchor can be negative)"
);
assert_eq!(
u32::from_le_bytes(rec[10..14].try_into().unwrap()),
4,
"len"
);
assert_eq!(&rec[14..], &data, "raw frame bytes follow");
// A non-keyframe records the flag as 0.
let delta = frame_record(0, 0, false, &[]);
assert_eq!(delta[1], 0);
assert_eq!(u32::from_le_bytes(delta[10..14].try_into().unwrap()), 0);
}
/// The cell row shows the raw category byte (0xNN) beside the decode, and
/// the keep/drop verdict. A plain feature cell (0x00) is "keep"; a leading
/// secondary-block cell flagged dropped reads "DROP".
#[test]
fn cell_row_shows_raw_byte_and_verdict() {
let plain = crate::ifo::DvdCell {
first_sector: 100,
last_sector: 199,
category: 0x00,
duration_secs: 12.5,
};
let row = dvd_cell_row(0, &plain, false);
assert!(row.contains("cat=0x00"), "{row}");
assert!(row.contains("block_mode=0"), "{row}");
assert!(row.contains("first=100"), "{row}");
assert!(row.contains("last=199"), "{row}");
assert!(row.contains("dur=12.5s"), "{row}");
assert!(row.contains("keep(plain-feature)"), "{row}");
assert!(!row.contains("DROP"), "{row}");
// 0x90 = in-block cell of an angle block (block_mode=2, block_type=1),
// shown dropped as a leading secondary piece.
let sec = crate::ifo::DvdCell {
first_sector: 0,
last_sector: 9,
category: 0x90,
duration_secs: 1.0,
};
let row = dvd_cell_row(0, &sec, true);
assert!(row.contains("cat=0x90"), "{row}");
assert!(row.contains("block_mode=2"), "{row}");
assert!(row.contains("block_type=1"), "{row}");
assert!(row.contains("DROP(leading-secondary-block-piece)"), "{row}");
}
}
-750
View File
@@ -1,750 +0,0 @@
//! Disc structure — scan titles, streams, and sector ranges from a Blu-ray disc.
//!
//! This is the high-level API for disc content. The CLI calls this,
//! never parses MPLS/CLPI/UDF directly.
//!
//! Usage:
//! let disc = Disc::scan(&mut session)?;
//! for title in disc.titles() { ... }
//! for stream in title.streams() { ... }
use crate::error::{Error, Result};
use crate::drive::DriveSession;
use crate::udf;
use crate::mpls;
use crate::clpi;
// ─── Public types ───────────────────────────────────────────────────────────
/// A scanned Blu-ray disc.
#[derive(Debug)]
pub struct Disc {
/// Disc capacity in sectors
pub capacity_sectors: u32,
/// Titles sorted by duration (longest first), then playlist name
pub titles: Vec<Title>,
/// AACS state — None if disc is unencrypted or keys unavailable
pub aacs: Option<AacsState>,
/// Whether this disc requires AACS decryption
pub encrypted: bool,
}
/// A title (one MPLS playlist).
#[derive(Debug, Clone)]
pub struct Title {
/// Playlist filename (e.g. "00800.mpls")
pub playlist: String,
/// Playlist number (e.g. 800)
pub playlist_id: u16,
/// Duration in seconds
pub duration_secs: f64,
/// Total size in bytes
pub size_bytes: u64,
/// Number of clips
pub clip_count: usize,
/// All streams (video, audio, subtitle, etc.)
pub streams: Vec<Stream>,
/// Sector extents for ripping (clip LBA ranges)
pub extents: Vec<Extent>,
}
/// A stream within a title.
#[derive(Debug, Clone)]
pub struct Stream {
/// Stream type
pub kind: StreamKind,
/// MPEG-TS packet ID
pub pid: u16,
/// Codec
pub codec: Codec,
/// ISO 639-2 language code (e.g. "eng", "fra")
pub language: String,
/// Video resolution (e.g. "2160p", "1080p")
pub resolution: String,
/// Frame rate (e.g. "23.976")
pub frame_rate: String,
/// Channel layout (e.g. "5.1", "7.1", "stereo")
pub channels: String,
/// Sample rate (e.g. "48kHz")
pub sample_rate: String,
/// HDR format
pub hdr: HdrFormat,
/// Color space
pub color_space: ColorSpace,
/// Whether this is a secondary/enhancement stream
pub secondary: bool,
/// Extra label (e.g. "Dolby Vision EL")
pub label: String,
}
/// Stream type.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum StreamKind {
Video,
Audio,
Subtitle,
}
/// Video/audio codec.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum Codec {
// Video
Hevc,
H264,
Vc1,
Mpeg2,
// Audio
TrueHd,
DtsHdMa,
DtsHdHr,
Dts,
Ac3,
Ac3Plus,
Lpcm,
// Subtitle
Pgs,
// Unknown
Unknown(u8),
}
/// HDR format.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum HdrFormat {
Sdr,
Hdr10,
DolbyVision,
}
/// Color space.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum ColorSpace {
Bt709,
Bt2020,
Unknown,
}
/// A contiguous range of sectors on disc.
#[derive(Debug, Clone, Copy)]
pub struct Extent {
pub start_lba: u32,
pub sector_count: u32,
}
// ─── Display helpers ────────────────────────────────────────────────────────
impl Codec {
pub fn name(&self) -> &'static str {
match self {
Codec::Hevc => "HEVC",
Codec::H264 => "H.264",
Codec::Vc1 => "VC-1",
Codec::Mpeg2 => "MPEG-2",
Codec::TrueHd => "TrueHD",
Codec::DtsHdMa => "DTS-HD MA",
Codec::DtsHdHr => "DTS-HD HR",
Codec::Dts => "DTS",
Codec::Ac3 => "AC-3",
Codec::Ac3Plus => "AC-3+",
Codec::Lpcm => "LPCM",
Codec::Pgs => "PGS",
Codec::Unknown(_) => "Unknown",
}
}
fn from_coding_type(ct: u8) -> Self {
match ct {
0x24 => Codec::Hevc,
0x1B => Codec::H264,
0xEA => Codec::Vc1,
0x02 => Codec::Mpeg2,
0x83 => Codec::TrueHd,
0x86 => Codec::DtsHdMa,
0x85 => Codec::DtsHdHr,
0x82 => Codec::Dts,
0x81 => Codec::Ac3,
0x84 | 0xA1 => Codec::Ac3Plus,
0x80 => Codec::Lpcm,
0xA2 => Codec::DtsHdHr,
0x90 | 0x91 => Codec::Pgs,
ct => Codec::Unknown(ct),
}
}
}
impl HdrFormat {
pub fn name(&self) -> &'static str {
match self {
HdrFormat::Sdr => "SDR",
HdrFormat::Hdr10 => "HDR10",
HdrFormat::DolbyVision => "Dolby Vision",
}
}
}
impl ColorSpace {
pub fn name(&self) -> &'static str {
match self {
ColorSpace::Bt709 => "BT.709",
ColorSpace::Bt2020 => "BT.2020",
ColorSpace::Unknown => "",
}
}
}
impl Title {
/// Duration formatted as "Xh Ym"
pub fn duration_display(&self) -> String {
let hrs = (self.duration_secs / 3600.0) as u32;
let mins = ((self.duration_secs % 3600.0) / 60.0) as u32;
format!("{}h {:02}m", hrs, mins)
}
/// Size in GB
pub fn size_gb(&self) -> f64 {
self.size_bytes as f64 / (1024.0 * 1024.0 * 1024.0)
}
/// Total sectors across all extents
pub fn total_sectors(&self) -> u64 {
self.extents.iter().map(|e| e.sector_count as u64).sum()
}
}
impl Stream {
/// Human-readable one-line description.
pub fn display(&self) -> String {
match self.kind {
StreamKind::Video => {
let mut parts = vec![self.codec.name().to_string()];
if !self.resolution.is_empty() { parts.push(self.resolution.clone()); }
if !self.frame_rate.is_empty() { parts.push(format!("{}fps", self.frame_rate)); }
if self.hdr != HdrFormat::Sdr { parts.push(self.hdr.name().to_string()); }
if self.color_space != ColorSpace::Unknown && self.color_space != ColorSpace::Bt709 {
parts.push(self.color_space.name().to_string());
}
if self.secondary { parts.push(format!("[{}]", self.label)); }
parts.join(" ")
}
StreamKind::Audio => {
let mut parts = vec![self.codec.name().to_string()];
if !self.channels.is_empty() { parts.push(self.channels.clone()); }
if !self.sample_rate.is_empty() { parts.push(self.sample_rate.clone()); }
if !self.language.is_empty() { parts.push(format!("({})", self.language)); }
if self.secondary { parts.push("[secondary]".to_string()); }
parts.join(" ")
}
StreamKind::Subtitle => {
let mut parts = vec![self.codec.name().to_string()];
if !self.language.is_empty() { parts.push(format!("({})", self.language)); }
parts.join(" ")
}
}
}
/// Kind as a display string
pub fn kind_name(&self) -> &'static str {
match self.kind {
StreamKind::Video => "Video",
StreamKind::Audio => "Audio",
StreamKind::Subtitle => "Subtitle",
}
}
}
// ─── AACS state ─────────────────────────────────────────────────────────────
/// AACS decryption state for a disc.
#[derive(Debug)]
pub struct AacsState {
/// AACS version (1 or 2)
pub version: u8,
/// Whether bus encryption is enabled (always true for AACS 2.0 / UHD)
pub bus_encryption: bool,
/// MKB version from disc (e.g. 68, 77)
pub mkb_version: Option<u32>,
/// Disc hash (SHA1 of Unit_Key_RO.inf) — hex string with 0x prefix
pub disc_hash: String,
/// How keys were resolved
pub key_source: KeySource,
/// Volume Unique Key (16 bytes)
pub vuk: [u8; 16],
/// Decrypted unit keys (CPS unit number, key)
pub unit_keys: Vec<(u32, [u8; 16])>,
/// Read data key for AACS 2.0 bus decryption — None for AACS 1.0
pub read_data_key: Option<[u8; 16]>,
/// Volume ID (16 bytes) — from SCSI handshake
pub volume_id: [u8; 16],
}
/// How AACS keys were resolved.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum KeySource {
/// VUK found directly in KEYDB by disc hash
KeyDb,
/// Media key + Volume ID from KEYDB → derived VUK
KeyDbDerived,
/// MKB + processing keys → media key → VUK
ProcessingKey,
/// MKB + device keys → subset-difference tree → VUK
DeviceKey,
}
impl KeySource {
pub fn name(&self) -> &'static str {
match self {
KeySource::KeyDb => "KEYDB",
KeySource::KeyDbDerived => "KEYDB (derived)",
KeySource::ProcessingKey => "MKB + processing key",
KeySource::DeviceKey => "MKB + device key",
}
}
}
// ─── Disc scanning ──────────────────────────────────────────────────────────
/// Standard KEYDB.cfg search locations (compatible with libaacs).
const KEYDB_SEARCH_PATHS: &[&str] = &[
".config/aacs/KEYDB.cfg", // relative to $HOME
];
const KEYDB_SYSTEM_PATH: &str = "/etc/aacs/KEYDB.cfg";
/// Options for disc scanning.
pub struct ScanOptions {
/// Path to KEYDB.cfg for AACS key lookup.
/// If None, searches standard locations ($HOME/.config/aacs/ and /etc/aacs/).
pub keydb_path: Option<std::path::PathBuf>,
}
impl Default for ScanOptions {
fn default() -> Self {
ScanOptions { keydb_path: None }
}
}
impl ScanOptions {
/// Create options with a specific KEYDB path.
pub fn with_keydb(path: impl Into<std::path::PathBuf>) -> Self {
ScanOptions { keydb_path: Some(path.into()) }
}
/// Resolve KEYDB path: explicit path first, then standard locations.
fn resolve_keydb(&self) -> Option<std::path::PathBuf> {
if let Some(p) = &self.keydb_path {
if p.exists() { return Some(p.clone()); }
}
if let Some(home) = std::env::var_os("HOME") {
for relative in KEYDB_SEARCH_PATHS {
let p = std::path::PathBuf::from(&home).join(relative);
if p.exists() { return Some(p); }
}
}
let p = std::path::PathBuf::from(KEYDB_SYSTEM_PATH);
if p.exists() { return Some(p); }
None
}
}
impl Disc {
/// Disc capacity in GB
pub fn capacity_gb(&self) -> f64 {
self.capacity_sectors as f64 * 2048.0 / (1024.0 * 1024.0 * 1024.0)
}
/// Scan a disc — parse filesystem, playlists, streams, and set up AACS decryption.
///
/// This is the main entry point. After scan(), the Disc is ready:
/// - titles are populated with streams
/// - AACS keys are derived (if KEYDB available)
/// - content can be read and decrypted transparently
///
/// ```no_run
/// use libfreemkv::{DriveSession, Disc};
/// use libfreemkv::disc::ScanOptions;
/// use std::path::Path;
///
/// let mut session = DriveSession::open(Path::new("/dev/sr0")).unwrap();
/// let disc = Disc::scan(&mut session, &ScanOptions::default()).unwrap();
/// for title in &disc.titles {
/// println!("{} — {} streams", title.duration_display(), title.streams.len());
/// }
/// ```
pub fn scan(session: &mut DriveSession, opts: &ScanOptions) -> Result<Self> {
// Step 1: Read capacity
let capacity = Self::read_capacity(session)?;
// Step 2: Parse UDF filesystem
let udf_fs = udf::read_filesystem(session)?;
// Step 3: Find and parse MPLS playlists
let mut titles = Vec::new();
if let Some(playlist_dir) = udf_fs.find_dir("/BDMV/PLAYLIST") {
for entry in &playlist_dir.entries {
if !entry.is_dir && entry.name.to_lowercase().ends_with(".mpls") {
let path = format!("/BDMV/PLAYLIST/{}", entry.name);
if let Ok(mpls_data) = udf_fs.read_file(session, &path) {
if let Some(title) = Self::parse_playlist(session, &udf_fs, &entry.name, &mpls_data) {
titles.push(title);
}
}
}
}
}
// Sort: longest first
titles.sort_by(|a, b| b.duration_secs.partial_cmp(&a.duration_secs).unwrap_or(std::cmp::Ordering::Equal));
// Step 4: Detect AACS encryption
let encrypted = udf_fs.find_dir("/AACS").is_some()
|| udf_fs.find_dir("/BDMV/AACS").is_some();
// Step 5: If encrypted and KEYDB available, authenticate and derive keys
let aacs = if encrypted {
if let Some(keydb_path) = opts.resolve_keydb() {
match Self::setup_aacs(session, &keydb_path) {
Ok(state) => Some(state),
Err(_) => None, // keys not found, continue without decryption
}
} else {
None
}
} else {
None
};
Ok(Disc {
capacity_sectors: capacity,
titles,
aacs,
encrypted,
})
}
/// Set up AACS decryption for this disc.
/// Call after scan() to enable transparent content decryption.
pub fn setup_aacs(
session: &mut DriveSession,
keydb_path: &std::path::Path,
) -> Result<AacsState> {
use crate::aacs::{self, KeyDb};
use crate::aacs::handshake;
// Load KEYDB
let keydb = KeyDb::load(keydb_path).map_err(|e| Error::AacsError {
detail: format!("failed to load KEYDB: {}", e),
})?;
// Step 1: Try SCSI handshake for Volume ID + read_data_key
// Open a separate transport (AACS auth must happen before raw mode).
// If handshake fails (drive doesn't support AACS layer, e.g. raw-mode drives),
// fall back to disc-hash-only KEYDB lookup.
let device_path = session.device_path().to_string();
let mut vid: Option<[u8; 16]> = None;
let mut read_data_key: Option<[u8; 16]> = None;
if !device_path.is_empty() {
if let Ok(mut aacs_session) = DriveSession::open_no_unlock(std::path::Path::new(&device_path)) {
if let Ok(hc) = keydb.host_cert.as_ref().ok_or(()) {
if let Ok(mut auth) = handshake::aacs_authenticate(
&mut aacs_session, &hc.private_key, &hc.certificate,
) {
vid = handshake::read_volume_id(&mut aacs_session, &mut auth).ok();
read_data_key = handshake::read_data_keys(&mut aacs_session, &mut auth)
.ok().map(|(rdk, _)| rdk);
}
}
}
// Handshake failure is not fatal — we can still resolve via disc hash
}
// Step 2: Read Unit_Key_RO.inf from disc via UDF (uses the unlocked main session)
let udf_fs = udf::read_filesystem(session)?;
let uk_ro_data = udf_fs.read_file(session, "/AACS/Unit_Key_RO.inf")
.or_else(|_| udf_fs.read_file(session, "/AACS/DUPLICATE/Unit_Key_RO.inf"))
.map_err(|_| Error::AacsError {
detail: "failed to read Unit_Key_RO.inf from disc".into(),
})?;
// Step 3: Read Content Certificate (optional — for AACS version detection)
let cc_data = udf_fs.read_file(session, "/AACS/Content000.cer")
.or_else(|_| udf_fs.read_file(session, "/AACS/Content001.cer"))
.ok();
// Step 4: Resolve keys
// If we have VID from handshake, use full 4-path chain.
// If no VID (handshake failed), use disc-hash-only KEYDB lookup.
let mkb_data = aacs::read_mkb_from_drive(session).ok();
let mkb_ver = mkb_data.as_deref().and_then(aacs::mkb_version);
// Use a zero VID placeholder if handshake failed — resolve_keys
// will still work via disc hash (path 1)
let vid_for_resolve = vid.unwrap_or([0u8; 16]);
let resolved = aacs::resolve_keys(
&uk_ro_data,
cc_data.as_deref(),
&vid_for_resolve,
&keydb,
mkb_data.as_deref(),
).ok_or_else(|| Error::AacsError {
detail: "failed to resolve AACS keys — disc not in KEYDB".into(),
})?;
let key_source = match resolved.key_source {
1 => KeySource::KeyDb,
2 => KeySource::KeyDbDerived,
3 => KeySource::ProcessingKey,
4 => KeySource::DeviceKey,
_ => KeySource::KeyDb,
};
Ok(AacsState {
version: if resolved.aacs2 { 2 } else { 1 },
bus_encryption: resolved.bus_encryption,
mkb_version: mkb_ver,
disc_hash: aacs::disc_hash_hex(&resolved.disc_hash),
key_source,
vuk: resolved.vuk,
unit_keys: resolved.unit_keys,
read_data_key,
volume_id: vid.unwrap_or([0u8; 16]),
})
}
// ── Internal helpers ────────────────────────────────────────────────────
fn read_capacity(session: &mut DriveSession) -> Result<u32> {
let cdb = [0x25, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00];
let mut buf = [0u8; 8];
session.scsi_execute(&cdb, crate::scsi::DataDirection::FromDevice, &mut buf, 5_000)?;
let lba = u32::from_be_bytes([buf[0], buf[1], buf[2], buf[3]]);
Ok(lba + 1)
}
fn parse_playlist(
session: &mut DriveSession,
udf_fs: &udf::UdfFs,
filename: &str,
data: &[u8],
) -> Option<Title> {
let parsed = mpls::parse(data).ok()?;
// Calculate duration from play items
let duration_ticks: u64 = parsed.play_items.iter()
.map(|pi| (pi.out_time.saturating_sub(pi.in_time)) as u64)
.sum();
let duration_secs = duration_ticks as f64 / 45000.0;
// Skip very short playlists (< 30 seconds)
if duration_secs < 30.0 {
return None;
}
// Parse each clip for EP map → sector extents
let mut extents = Vec::new();
let mut total_size: u64 = 0;
let clip_count = parsed.play_items.len();
for play_item in &parsed.play_items {
let clpi_path = format!("/BDMV/CLIPINF/{}.clpi", play_item.clip_id);
if let Ok(clpi_data) = udf_fs.read_file(session, &clpi_path) {
if let Ok(clip_info) = clpi::parse(&clpi_data) {
// Use EP map to get sector extents for this clip's time range
let clip_extents = clip_info.get_extents(play_item.in_time, play_item.out_time);
for ext in &clip_extents {
total_size += ext.sector_count as u64 * 2048;
}
extents.extend(clip_extents);
}
}
}
// Build streams from STN table
let streams: Vec<Stream> = parsed.streams.iter().map(|s| {
let kind = match s.stream_type {
1 => StreamKind::Video,
2 => StreamKind::Audio,
3 => StreamKind::Subtitle,
_ => StreamKind::Video,
};
let codec = Codec::from_coding_type(s.coding_type);
Stream {
kind,
pid: s.pid,
codec,
language: s.language.clone(),
resolution: format_resolution(s.video_format, s.video_rate),
frame_rate: format_framerate(s.video_rate),
channels: format_channels(s.audio_format),
sample_rate: format_samplerate(s.audio_rate),
hdr: HdrFormat::Sdr,
color_space: ColorSpace::Unknown,
secondary: false,
label: String::new(),
}
}).collect();
let playlist_num = filename.trim_end_matches(".mpls").trim_end_matches(".MPLS");
let playlist_id = playlist_num.parse::<u16>().unwrap_or(0);
Some(Title {
playlist: filename.to_string(),
playlist_id,
duration_secs,
size_bytes: total_size,
clip_count,
streams,
extents,
})
}
}
// ─── Decrypted reader ──────────────────────────────────────────────────────
/// A reader that reads m2ts content, decrypting transparently if needed.
pub struct ContentReader<'a> {
session: &'a mut DriveSession,
aacs: Option<&'a AacsState>,
extents: Vec<Extent>,
current_extent: usize,
current_offset: u32, // sectors into current extent
unit_key_idx: usize,
}
impl Disc {
/// Open a title for reading. Decryption is automatic — if the disc
/// is encrypted and keys were found during scan(), content is decrypted
/// on the fly. Unencrypted discs pass through unchanged.
///
/// ```no_run
/// # use libfreemkv::{DriveSession, Disc};
/// # use libfreemkv::disc::ScanOptions;
/// # use std::path::Path;
/// # let mut session = DriveSession::open(Path::new("/dev/sr0")).unwrap();
/// let disc = Disc::scan(&mut session, &ScanOptions::default()).unwrap();
/// let mut reader = disc.open_title(&mut session, 0).unwrap();
/// while let Some(unit) = reader.read_unit().unwrap() {
/// // unit is 6144 bytes of decrypted content
/// }
/// ```
pub fn open_title<'a>(&'a self, session: &'a mut DriveSession, title_idx: usize) -> Result<ContentReader<'a>> {
let title = self.titles.get(title_idx).ok_or_else(|| Error::DiscError {
detail: format!("title index {} out of range (have {})", title_idx, self.titles.len()),
})?;
Ok(ContentReader {
session,
aacs: self.aacs.as_ref(),
extents: title.extents.clone(),
current_extent: 0,
current_offset: 0,
unit_key_idx: 0,
})
}
}
impl<'a> ContentReader<'a> {
/// Read the next aligned unit (6144 bytes).
/// Automatically decrypted if AACS keys are available.
/// Returns None when all extents are exhausted.
pub fn read_unit(&mut self) -> Result<Option<Vec<u8>>> {
if self.current_extent >= self.extents.len() {
return Ok(None);
}
let extent = &self.extents[self.current_extent];
let lba = extent.start_lba + self.current_offset;
// Read 3 sectors (one aligned unit)
let mut unit = vec![0u8; crate::aacs::ALIGNED_UNIT_LEN];
for i in 0..3u32 {
let offset = (i as usize) * 2048;
let mut sector = [0u8; 2048];
session_read_sector(self.session, lba + i, &mut sector)?;
unit[offset..offset + 2048].copy_from_slice(&sector);
}
// Decrypt if needed
if let Some(aacs) = &self.aacs {
if crate::aacs::is_unit_encrypted(&unit) {
let uk = aacs.unit_keys.get(self.unit_key_idx)
.map(|(_, k)| *k)
.unwrap_or([0u8; 16]);
crate::aacs::decrypt_unit_full(
&mut unit,
&uk,
aacs.read_data_key.as_ref(),
);
}
}
// Advance position
self.current_offset += 3;
if self.current_offset >= extent.sector_count {
self.current_extent += 1;
self.current_offset = 0;
}
Ok(Some(unit))
}
}
fn session_read_sector(session: &mut DriveSession, lba: u32, buf: &mut [u8; 2048]) -> Result<()> {
let cdb = [
crate::scsi::SCSI_READ_10, 0x00,
(lba >> 24) as u8, (lba >> 16) as u8, (lba >> 8) as u8, lba as u8,
0x00, 0x00, 0x01, 0x00,
];
session.scsi_execute(&cdb, crate::scsi::DataDirection::FromDevice, buf, 10_000)?;
Ok(())
}
// ─── Format helpers ────────────────────────────────────────────────────────
fn format_resolution(video_format: u8, _video_rate: u8) -> String {
match video_format {
1 => "480i".into(),
2 => "576i".into(),
3 => "480p".into(),
4 => "1080i".into(),
5 => "720p".into(),
6 => "1080p".into(),
7 => "576p".into(),
8 => "2160p".into(),
_ => String::new(),
}
}
fn format_framerate(video_rate: u8) -> String {
match video_rate {
1 => "23.976".into(),
2 => "24".into(),
3 => "25".into(),
4 => "29.97".into(),
6 => "50".into(),
7 => "59.94".into(),
_ => String::new(),
}
}
fn format_channels(audio_format: u8) -> String {
match audio_format {
1 => "mono".into(),
3 => "stereo".into(),
6 => "5.1".into(),
12 => "7.1".into(),
_ if audio_format > 0 => format!("{}ch", audio_format),
_ => String::new(),
}
}
fn format_samplerate(audio_rate: u8) -> String {
match audio_rate {
1 => "48kHz".into(),
4 => "96kHz".into(),
5 => "192kHz".into(),
12 => "48/192kHz".into(),
14 => "48/96kHz".into(),
_ => String::new(),
}
}
+1597
View File
File diff suppressed because it is too large Load Diff
+1323
View File
File diff suppressed because it is too large Load Diff
+466
View File
@@ -0,0 +1,466 @@
//! Physical AC-3 sub-stream probing for DVD audio routing.
//!
//! ## Why this exists (Silence-of-the-Lambs wrong-substream bug)
//!
//! A DVD VTS IFO declares its audio streams in a fixed table, and freemkv's
//! scan assigns each declared stream a `private_stream_1` sub-stream id purely
//! by per-codec ordinal — the first AC-3 stream becomes `0x80`, the second
//! `0x81`, and so on (`ifo::assign_audio_sub_stream_ids`). That assumes the
//! physical sub-stream order on the wire matches the IFO declaration order.
//!
//! On some discs it does NOT. The R2 PAL "The Silence of the Lambs" feature
//! declares ONE AC-3 audio stream the IFO nibble marks as 5.1 (6 channels), but
//! the physical VOB carries the 5.1 main mix and a 2.0 down-mix on DIFFERENT
//! `0x8x` sub-stream ids, and the 2.0 is the one that happens to land at the
//! ordinal `0x80` slot. Routing the declared 5.1 stream to `0x80` by ordinal
//! therefore muxes the 2.0 down-mix while labelling it 5.1 — the wrong physical
//! track.
//!
//! The robust fix is data-driven and codec/disc agnostic: read each physical
//! AC-3 sub-stream's REAL channel count from the VOB (the `acmod`/`lfeon` of its
//! first frame after the `0x0B77` sync) and route each IFO-declared AC-3 stream
//! to the physical sub-stream whose actual channel count matches the IFO's
//! declared count — instead of trusting the ordinal. This never re-reads the
//! disc beyond a bounded head-of-feature probe and degrades to the original
//! ordinal mapping when the probe yields nothing (unreadable/short VOB).
use crate::disc::Stream;
use crate::mux::codec::ac3;
use crate::mux::ps::PsDemuxer;
use crate::sector::SectorSource;
use std::collections::BTreeMap;
/// How many 2048-byte sectors of the first feature extent to probe. The head of
/// a DVD feature opens with logos/warnings whose audio is frequently a thin 2.0
/// bed on the FIRST sub-stream only — the other physical `0x8x` sub-streams and
/// the main 5.1 mix do not appear until a sector or two further in. 512 sectors
/// (1 MiB) was too short: on Greenland it saw ONLY `0x80`, and only its opening
/// 2.0 frames. 1024 sectors (2 MiB) reliably contains at least one frame of
/// every physical AC-3 sub-stream AND enough of `0x80` to reach its 5.1 frames.
/// Still bounded so a live drive is never hammered (see the project "don't
/// hammer the live drive" rule).
const PROBE_SECTORS: u16 = 1024;
/// Decode the real per-sub-stream AC-3 channel count from a buffer of decrypted
/// MPEG-PS (DVD VOB) bytes.
///
/// Demuxes `private_stream_1` (0xBD), and for each AC-3 sub-stream id
/// (`0x80..=0x87`) records the MAXIMUM channel count seen across EVERY decodable
/// frame in the probe window (`acmod` + `lfeon` at each `0x0B77` sync). Pure and
/// unit-testable — takes the already-read bytes, never touches the disc.
///
/// ## Why the maximum, not the first frame
///
/// The first frame of a sub-stream at the head of a feature is NOT
/// representative. A DVD opens with logos/warnings, and the main `0x80`
/// sub-stream there frequently carries a thin 2.0 bed before transitioning to
/// its real 5.1 main mix a fraction of a second later (observed on Greenland:
/// `0x80`'s first frames are acmod=2 → 2 channels, then it becomes acmod=7+lfe →
/// 6 channels within the same 2 MiB window). Recording only the FIRST frame read
/// `0x80=2` and missed the 5.1 entirely, defeating the channel-match routing.
/// The 5.1 capability of a sub-stream is the *maximum* channel count any of its
/// frames carries, so we scan them all and keep the max.
///
/// Returns a map `sub_id -> max channels`. Sub-streams whose frames are all too
/// short to carry the BSI bits, or that never appear in the buffer, are absent
/// from the map.
pub fn probe_ac3_substream_channels(ps_bytes: &[u8]) -> BTreeMap<u8, u8> {
let mut found: BTreeMap<u8, u8> = BTreeMap::new();
let mut demux = PsDemuxer::new();
let mut packets = demux.feed(ps_bytes);
packets.extend(demux.flush());
for p in packets {
// Only private_stream_1 AC-3 sub-streams (0x80..=0x87).
let Some(sub) = p.sub_stream_id else { continue };
if !(0x80..=0x87).contains(&sub) {
continue;
}
// The PS demux strips the 4-byte AC-3 sub-header but does not align to a
// frame. Walk EVERY 0x0B77 sync in this sub-stream's payload, decode
// each frame's channel count, and keep the largest — the sub-stream's
// real (main-mix) channel capability. See the doc comment above for why
// the first frame alone is unreliable.
if let Some(ch) = max_substream_channels(&p.data) {
let slot = found.entry(sub).or_insert(0);
*slot = (*slot).max(ch);
}
}
found
}
/// Largest AC-3 channel count over every decodable frame in a single
/// sub-stream's payload. Returns `None` when no frame carries enough BSI bits.
///
/// Each frame is advanced by its real `ac3_frame_size` so a frame's compressed
/// body (which can contain stray `0x0B77` byte pairs) cannot be mistaken for a
/// new frame; only when a size is unmappable do we fall back to a +2 byte
/// rescan to re-lock the next genuine sync.
fn max_substream_channels(data: &[u8]) -> Option<u8> {
let mut best: Option<u8> = None;
let mut pos = 0;
while pos < data.len() {
let Some(rel) = ac3::find_ac3_sync(&data[pos..]) else {
break;
};
let start = pos + rel;
let frame = &data[start..];
if let Some(ch) = ac3::acmod_channels(frame) {
if ch > 0 {
best = Some(best.map_or(ch, |b| b.max(ch)));
}
}
// Advance past this frame by its declared size when that is mappable;
// otherwise step 2 bytes past the sync and re-scan for the next one.
let size = ac3::ac3_frame_size(frame);
pos = if (6..=8192).contains(&size) {
start + size
} else {
start + 2
};
}
best
}
/// Re-route the title's declared AC-3 audio streams onto the physical
/// sub-stream ids whose REAL channel counts match, using a probed
/// `sub_id -> channels` map.
///
/// For each declared AC-3 audio stream (in IFO order), it picks the physical
/// `0x8x` sub-stream whose probed channel count equals the stream's declared
/// channel count, never re-using a sub-stream already claimed by an earlier
/// stream. The chosen sub-stream's PID (`0xBD00 | sub_id`) is written back onto
/// the `Stream::Audio` so BOTH mux demux paths (`DiscStream` and the file-backed
/// highway) route by it.
///
/// Conservative — it only ever REASSIGNS among the physical sub-streams the
/// probe actually saw, and only when a better (exact-channel) match exists than
/// the stream's current assignment. A stream whose current sub-stream already
/// matches is left alone; a stream with no matching physical sub-stream keeps
/// its ordinal assignment. So a normal disc (physical order == IFO order) is a
/// no-op.
///
/// Returns the number of streams whose PID was changed (for diagnostics).
pub fn remap_audio_pids(streams: &mut [Stream], probed: &BTreeMap<u8, u8>) -> usize {
if probed.is_empty() {
return 0;
}
// Sub-streams already claimed by a remapped (or matching) earlier stream,
// so two declared streams never collide on one physical sub-stream.
let mut claimed: Vec<u8> = Vec::new();
let mut changed = 0usize;
for s in streams.iter_mut() {
let Stream::Audio(a) = s else { continue };
if a.codec != crate::disc::Codec::Ac3 {
continue;
}
let declared = a.channels.count();
// The sub-id this stream currently routes by (low byte of its PID).
let current_sub = (a.pid & 0x00FF) as u8;
// If the stream's current physical sub-stream already matches its
// declared channel count, keep it and claim it.
if probed.get(&current_sub) == Some(&declared) {
claimed.push(current_sub);
continue;
}
// Otherwise find an unclaimed physical sub-stream whose REAL channel
// count equals the declared count.
let pick = probed
.iter()
.find(|(sub, ch)| **ch == declared && !claimed.contains(*sub))
.map(|(sub, _)| *sub);
if let Some(sub) = pick {
let new_pid = 0xBD00 | sub as u16;
if new_pid != a.pid {
tracing::debug!(
target: "freemkv::scan",
old_pid = a.pid,
new_pid,
declared_channels = declared,
"dvd: re-routed AC-3 audio to physical sub-stream matching channel count"
);
a.pid = new_pid;
changed += 1;
}
claimed.push(sub);
} else {
// No physical match — leave the ordinal assignment, but claim its
// current sub so later streams don't steal a slot it may still use.
claimed.push(current_sub);
}
}
changed
}
/// Probe the first feature extent of a DVD title through a (decrypted) sector
/// source and re-route its AC-3 audio PIDs to the physically-correct
/// sub-streams. A bounded, best-effort scan: any read error or empty probe
/// leaves the ordinal assignment untouched.
///
/// `reader` MUST yield PLAINTEXT VOB bytes (i.e. a `DecryptingSectorSource` on a
/// CSS disc) — probing scrambled sectors yields no AC-3 syncs and is a safe
/// no-op. Returns the number of audio streams whose PID changed.
pub fn probe_and_remap<S: SectorSource + ?Sized>(
reader: &mut S,
title: &mut crate::disc::DiscTitle,
) {
// Only DVD (MPEG-PS) titles carry private_stream_1 AC-3 sub-streams.
if title.content_format != crate::disc::ContentFormat::MpegPs {
return;
}
// Nothing to disambiguate unless there is at least one AC-3 audio stream.
let has_ac3 = title
.streams
.iter()
.any(|s| matches!(s, Stream::Audio(a) if a.codec == crate::disc::Codec::Ac3));
if !has_ac3 {
return;
}
let Some(ext) = title.extents.first() else {
return;
};
let count: u16 = ext.sector_count.min(PROBE_SECTORS as u32) as u16;
if count == 0 {
return;
}
let mut buf = vec![0u8; count as usize * 2048];
// `recovery=false`: a single best-effort attempt — the probe must never
// stall the mux or hammer a marginal drive. On any error, bail to ordinal.
let n = match reader.read_sectors(ext.start_lba, count, &mut buf, false) {
Ok(n) => n,
Err(_) => return,
};
buf.truncate(n);
let probed = probe_ac3_substream_channels(&buf);
crate::diag::dump_dvd_substream_probe(title.playlist_id, &probed);
remap_audio_pids(&mut title.streams, &probed);
}
#[cfg(test)]
mod tests {
use super::*;
use crate::disc::{AudioChannels, AudioStream, Codec, LabelPurpose, SampleRate};
/// Build a single, correctly-SIZED AC-3 frame whose `acmod`/`lfeon` encode a
/// known channel count. `byte4` is `fscod=0 | frmsizecod=0`, so
/// `ac3_frame_size` reports 128 bytes and the frame is zero-padded to exactly
/// that — this lets `max_substream_channels` advance frame-by-frame over a
/// multi-frame payload exactly as it does on real VOB data. The BSI bits are
/// laid down with a writer so the test never hand-miscomputes the lfeon
/// offset, matching `acmod_channels`' reader.
fn ac3_frame(acmod: u8, lfeon: bool) -> Vec<u8> {
let mut bits: Vec<u8> = Vec::new();
let push = |val: u32, n: usize, bits: &mut Vec<u8>| {
for i in (0..n).rev() {
bits.push(((val >> i) & 1) as u8);
}
};
push(acmod as u32, 3, &mut bits);
if (acmod & 0x1) != 0 && acmod != 0x1 {
push(0, 2, &mut bits); // cmixlev
}
if (acmod & 0x4) != 0 {
push(0, 2, &mut bits); // surmixlev
}
if acmod == 0x2 {
push(0, 2, &mut bits); // dsurmod
}
push(lfeon as u32, 1, &mut bits);
// Pack the bit vector MSB-first into bytes (byte6 onward).
let mut tail = Vec::new();
let mut cur = 0u8;
for (i, b) in bits.iter().enumerate() {
cur = (cur << 1) | b;
if i % 8 == 7 {
tail.push(cur);
cur = 0;
}
}
let rem = bits.len() % 8;
if rem != 0 {
cur <<= 8 - rem;
tail.push(cur);
}
// AC-3 frame: 0x0B 0x77 crc(2) byte4(fscod=0,frmsizecod=0) bsid<<3 then BSI.
let mut frame = vec![0x0B, 0x77, 0x00, 0x00, 0x00, 8u8 << 3];
frame.extend_from_slice(&tail);
// frmsizecod=0 @ 48kHz → 64 words = 128 bytes. Pad to the real size so
// the frame-stepping in max_substream_channels lands on the next sync.
frame.resize(128, 0);
frame
}
/// Build a minimal `private_stream_1` PES carrying `frames` for `sub_id`,
/// each preceded only by the 4-byte AC-3 sub-header at the PES head. Mirrors
/// the on-disc layout the PS demux expects: PES start `0x000001BD`, length,
/// PES header (no PTS), sub-header `[sub_id, frame_count, ptr_hi, ptr_lo]`,
/// then the concatenated AC-3 frames.
fn ps_ac3_frames(sub_id: u8, frames: &[Vec<u8>]) -> Vec<u8> {
// PES sub-header for AC-3: sub_id + frame_count + 2-byte access ptr.
let mut payload = vec![sub_id, frames.len() as u8, 0x00, 0x04];
for f in frames {
payload.extend_from_slice(f);
}
// PES packet: start code 00 00 01 BD, length(2), flags(2), hdr_len(0).
let pes_payload_len = 3 + payload.len(); // flags(2)+hdrlen(1)+payload
let mut pkt = vec![0x00, 0x00, 0x01, 0xBD];
pkt.extend_from_slice(&(pes_payload_len as u16).to_be_bytes());
pkt.extend_from_slice(&[0x80, 0x00, 0x00]); // no PTS, header_data_len=0
pkt.extend_from_slice(&payload);
pkt
}
/// Single-frame `private_stream_1` PES — the common case in existing tests.
fn ps_ac3(sub_id: u8, acmod: u8, lfeon: bool) -> Vec<u8> {
ps_ac3_frames(sub_id, &[ac3_frame(acmod, lfeon)])
}
fn ac3_stream(pid: u16, channels: AudioChannels) -> Stream {
Stream::Audio(AudioStream {
pid,
codec: Codec::Ac3,
channels,
language: "en".into(),
sample_rate: SampleRate::S48,
secondary: false,
purpose: LabelPurpose::Normal,
label: String::new(),
})
}
/// The probe decodes the real channel count of each physical sub-stream.
/// 0x80 carries a 2.0 frame (acmod=2,no lfe → 2ch); 0x81 carries 5.1
/// (acmod=7 + lfe → 6ch).
#[test]
fn probe_decodes_per_substream_channels() {
let mut bytes = ps_ac3(0x80, 2, false);
bytes.extend(ps_ac3(0x81, 7, true));
let probed = probe_ac3_substream_channels(&bytes);
assert_eq!(probed.get(&0x80), Some(&2), "0x80 is the 2.0 down-mix");
assert_eq!(probed.get(&0x81), Some(&6), "0x81 is the 5.1 main mix");
}
/// GREENLAND regression — the probe must read each sub-stream's TRUE
/// (max-mix) channel count, not be poisoned by an unrepresentative head
/// frame, and must NOT cross-contaminate between sub-streams.
///
/// Mirrors the real on-disc layout that caused the mis-read: the feature
/// head carries `0x80` opening with a 2.0 frame and THEN a 5.1 frame (its
/// real main mix), interleaved with `0x81` carrying only 2.0. The old
/// first-frame probe read `0x80=2` (the logo bed) and missed the 5.1; the
/// max-over-frames probe must report `0x80=6` and `0x81=2`.
#[test]
fn probe_reads_max_channels_no_cross_contamination() {
let mut bytes = Vec::new();
// 0x80 opens with a 2.0 frame (the logo bed)...
bytes.extend(ps_ac3_frames(0x80, &[ac3_frame(2, false)]));
// ...0x81 interleaves a pure-2.0 PES (must NOT bleed 6 into 0x80)...
bytes.extend(ps_ac3_frames(
0x81,
&[ac3_frame(2, false), ac3_frame(2, false)],
));
// ...then 0x80 reaches its real 5.1 main mix (acmod=7 + lfe → 6 ch),
// with a trailing 2.0 frame in the SAME PES to prove we take the max,
// not the last frame.
bytes.extend(ps_ac3_frames(
0x80,
&[ac3_frame(7, true), ac3_frame(2, false)],
));
let probed = probe_ac3_substream_channels(&bytes);
assert_eq!(
probed.get(&0x80),
Some(&6),
"0x80's real 5.1 mix must win over its 2.0 head/tail frames"
);
assert_eq!(
probed.get(&0x81),
Some(&2),
"0x81 is a pure 2.0 stream — must not absorb 0x80's 6-channel frame"
);
}
/// SILENCE-OF-THE-LAMBS regression: the IFO declares ONE 5.1 AC-3 stream and
/// the ordinal mapping put it at 0x80, but physically 0x80 is the 2.0
/// down-mix and the 5.1 lives at 0x81. After probe+remap the declared 5.1
/// stream must route to 0x81 (PID 0xBD81), NOT the ordinal 0x80.
#[test]
fn remap_routes_declared_51_to_physical_51_substream() {
// Physical layout: 0x80 = 2.0, 0x81 = 5.1 (reversed vs ordinal).
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 2u8);
probed.insert(0x81u8, 6u8);
// Declared: one 5.1 stream, ordinally assigned 0x80 (PID 0xBD80).
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 1, "the one 5.1 stream must be re-routed");
let Stream::Audio(a) = &streams[0] else {
panic!("audio")
};
assert_eq!(
a.pid, 0xBD81,
"declared 5.1 must route to physical 0x81 (the real 5.1), not ordinal 0x80"
);
}
/// Conservative no-op: when the physical order already matches the IFO
/// order (0x80 = 5.1 as declared), remap changes nothing.
#[test]
fn remap_noop_when_physical_matches_ordinal() {
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 6u8); // 0x80 really is the 5.1
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 0, "matching physical order is a no-op");
let Stream::Audio(a) = &streams[0] else {
panic!()
};
assert_eq!(a.pid, 0xBD80);
}
/// Two declared streams (5.1 + 2.0) where the physical order is reversed:
/// 0x80=2.0, 0x81=5.1. The 5.1 declaration must claim 0x81 and the 2.0
/// declaration must claim 0x80 — no collision, both correct.
#[test]
fn remap_two_streams_no_collision() {
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 2u8);
probed.insert(0x81u8, 6u8);
// Declared order: 5.1 first (ordinal 0x80), 2.0 second (ordinal 0x81).
let mut streams = vec![
ac3_stream(0xBD80, AudioChannels::Surround51),
ac3_stream(0xBD81, AudioChannels::Stereo),
];
remap_audio_pids(&mut streams, &probed);
let pids: Vec<u16> = streams
.iter()
.filter_map(|s| match s {
Stream::Audio(a) => Some(a.pid),
_ => None,
})
.collect();
assert_eq!(
pids,
vec![0xBD81, 0xBD80],
"5.1→0x81, 2.0→0x80, no collision"
);
}
/// Empty probe (unreadable / scrambled VOB) is a no-op — the ordinal
/// assignment survives so behaviour never regresses below today's.
#[test]
fn remap_empty_probe_is_noop() {
let probed = BTreeMap::new();
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 0);
let Stream::Audio(a) = &streams[0] else {
panic!()
};
assert_eq!(a.pid, 0xBD80, "no probe data → keep ordinal");
}
}
+1101
View File
File diff suppressed because it is too large Load Diff
+1661
View File
File diff suppressed because it is too large Load Diff
+1670
View File
File diff suppressed because it is too large Load Diff
+6561
View File
File diff suppressed because it is too large Load Diff
+1705
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+232
View File
@@ -0,0 +1,232 @@
//! `Disc::sweep`'s consumer-side `Sink<WorkItem>`.
//!
//! Background: the original sweep loop runs strictly serialised —
//! SCSI read → decrypt → seek + write → mapfile.record → next iter.
//! On a healthy disc the SCSI read costs ~5-12 ms per 64 KB batch and
//! the post-read work (decrypt 1-3 ms + file write + mapfile fsync
//! 5-15 ms) adds another batch's worth of latency. The drive idles
//! during the post-read work; throughput tops out at the *sum* of
//! both costs.
//!
//! A producer/consumer split overlaps the two stages on the generic
//! [`crate::io::Pipeline`] + [`crate::io::Sink`] primitive. This module
//! is the sweep-specific `Sink` impl; the producer-side state machine
//! (read_error context, decrypt, set_speed, halt) stays in
//! `Disc::sweep` in `disc/mod.rs`.
//!
//! Correctness invariants preserved:
//! - Mapfile is single-writer (consumer-only). No locking.
//! - All `read_error::ReadCtx` state stays on the producer thread.
//! - `set_speed` calls happen on the producer thread (same thread that
//! owns the `SectorSource`). No new SCSI concurrency.
//! - Per-iteration ordering of file-write → mapfile-record is kept
//! intact in the consumer (write before record), so the on-disk
//! invariant "mapfile only marks Finished what the file has
//! received" survives a crash mid-pass.
//! - Only one SCSI command is in flight at a time; error-path timing
//! is identical and no new retry logic is introduced.
use std::io::{Seek, SeekFrom, Write};
use std::sync::mpsc::{Receiver, SyncSender, sync_channel};
use crate::error::Error;
use crate::io::{Flow, Sink};
use super::mapfile::{MapStats, Mapfile, SectorStatus};
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. 64 KB
/// matches the existing zero_gap chunk size used by the pre-split
/// sweep loop.
const ZERO_CHUNK: usize = 64 * 1024;
/// Producer → Consumer messages. The consumer applies these in FIFO
/// order; ordering of file writes and mapfile records across items is
/// preserved.
pub(super) enum WorkItem {
/// Successful batch read. Producer has already decrypted `buf` if
/// `opts.decrypt` was set. Consumer writes `buf` at `pos` and
/// records the range as `Finished`.
Good { pos: u64, buf: Vec<u8> },
/// Bisect inner-loop good single sector (already decrypted by the
/// producer). 2048 bytes.
BisectGood { pos: u64, buf: Box<[u8; 2048]> },
/// Bisect inner-loop bad single sector. Consumer writes 2048
/// zeros at `pos` and records the sector as `NonTrimmed`.
BisectBad { pos: u64 },
/// Whole-batch zero-fill (failed batch on `SkipBlock`, or the
/// failed batch portion of `JumpAhead`). Consumer streams zeros
/// across `[pos, pos+len)` and records the range as `NonTrimmed`.
SkipFill { pos: u64, len: u64 },
/// Gap fill following a `JumpAhead`. Same effect as `SkipFill`;
/// distinguished only so future logging / instrumentation can
/// tell them apart without parsing a flag.
GapFill { pos: u64, len: u64 },
/// Post-read verify downgrade. The producer's `UnitVerifier` found that the
/// just-`Finished` clip unit at `[pos, pos+len)` is confidently undecryptable
/// (a silent bad read). The consumer re-records the range as `NonTrimmed` so
/// the patch pass re-reads it — the ISO bytes (ciphertext) already written by
/// the preceding `Good` are left in place for the patch to overwrite. FIFO
/// pipe ordering guarantees this arrives AFTER the `Good` that wrote them.
MarkBad { pos: u64, len: u64 },
/// Producer wants the latest mapfile stats for the progress
/// callback. Consumer responds on `prog_tx` with a fresh
/// [`ProgressSnapshot`]. Best-effort: if the producer hasn't
/// drained the previous snapshot, the new one is silently
/// dropped — the producer's local cache stays current enough.
StatsRequest,
}
/// Snapshot the consumer sends back to the producer for the progress
/// callback.
pub(super) struct ProgressSnapshot {
pub stats: MapStats,
pub bad_ranges: Vec<(u64, u64)>,
}
/// Final summary returned by the consumer thread on shutdown — what
/// `SweepSink::close` produces, surfaced to the producer via
/// `Pipeline::finish`.
pub(super) struct ConsumerSummary {
pub stats: MapStats,
}
/// Drain any pending progress snapshots from the consumer. Returns
/// the most recent one, if any. The producer caches it and uses it
/// for subsequent progress callbacks until a fresh one arrives.
pub(super) fn try_recv_progress(rx: &Receiver<ProgressSnapshot>) -> Option<ProgressSnapshot> {
let mut latest = None;
while let Ok(snap) = rx.try_recv() {
latest = Some(snap);
}
latest
}
/// `Sink<WorkItem>` for sweep. Owns the writeback file + mapfile +
/// progress back-channel. `apply` carries the file-write +
/// mapfile.record per item; `close` drains the writeback pipeline,
/// fsyncs the ISO, and flushes the mapfile.
pub(super) struct SweepSink {
file: crate::io::WritebackFile,
map: Mapfile,
/// `sync_all`-on-failure-is-an-error iff the output is a regular
/// file. `/dev/null` and pipes always fail `sync_all`; that's not
/// a real error.
is_regular: bool,
/// Back-channel for `StatsRequest` responses. The producer caches
/// the latest snapshot and uses it for the progress callback;
/// dropped sends on a full channel are by design.
prog_tx: SyncSender<ProgressSnapshot>,
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. Held
/// in the sink so each apply call doesn't reallocate.
zero: Box<[u8; ZERO_CHUNK]>,
}
impl SweepSink {
/// Construct a new `SweepSink` plus the matching progress
/// receiver. Channel depth on the back-channel is `1` — the
/// producer's cache is the source of truth between snapshots.
pub(super) fn new(
file: crate::io::WritebackFile,
map: Mapfile,
is_regular: bool,
) -> (Self, Receiver<ProgressSnapshot>) {
let (prog_tx, prog_rx) = sync_channel::<ProgressSnapshot>(1);
let sink = SweepSink {
file,
map,
is_regular,
prog_tx,
zero: Box::new([0u8; ZERO_CHUNK]),
};
(sink, prog_rx)
}
}
impl Sink<WorkItem> for SweepSink {
type Output = ConsumerSummary;
fn apply(&mut self, item: WorkItem) -> Result<Flow, Error> {
match item {
WorkItem::Good { pos, buf } => {
// Decrypt is on the producer; consumer assumes plaintext.
let len = buf.len() as u64;
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&buf)?;
self.map.record(pos, len, SectorStatus::Finished)?;
}
WorkItem::BisectGood { pos, buf } => {
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&buf[..])?;
self.map.record(pos, 2048, SectorStatus::Finished)?;
}
WorkItem::BisectBad { pos } => {
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&self.zero[..2048])?;
self.map.record(pos, 2048, SectorStatus::NonTrimmed)?;
}
WorkItem::SkipFill { pos, len } | WorkItem::GapFill { pos, len } => {
self.file.seek(SeekFrom::Start(pos))?;
// Subsequent writes are sequential; `WritebackFile`'s
// seek-elision keeps them on the writeback pipeline path.
let mut filled = 0u64;
while filled < len {
let chunk = (len - filled).min(self.zero.len() as u64) as usize;
self.file.write_all(&self.zero[..chunk])?;
filled += chunk as u64;
}
self.map.record(pos, len, SectorStatus::NonTrimmed)?;
}
WorkItem::MarkBad { pos, len } => {
// Verify downgrade: the ISO bytes are already written by the
// preceding Good; only the mapfile status changes so patch
// re-reads this range. No file write.
self.map.record(pos, len, SectorStatus::NonTrimmed)?;
}
WorkItem::StatsRequest => {
let stats = self.map.stats();
// DAMAGE only — NOT NonTried. NonTried is the unread remainder
// ahead of the sweep head, not damage; including it made the live
// located drilldown (at-risk movie time + range count) treat the
// whole unread disc as confirmed damage, so at sweep start it
// showed ~full-movie at-risk and melted to 0 as the sweep
// progressed. Matches the one-shot progress path, which already
// excludes NonTried.
let bad_ranges = self.map.ranges_with(&[
SectorStatus::NonTrimmed,
SectorStatus::Unreadable,
SectorStatus::NonScraped,
]);
// Best-effort: drop on backpressure; producer's cache
// stays current enough.
let _ = self
.prog_tx
.try_send(ProgressSnapshot { stats, bad_ranges });
}
}
Ok(Flow::Continue)
}
fn close(mut self) -> Result<Self::Output, Error> {
// Drain the writeback pipeline + fsync the ISO, then persist
// any pending mapfile state. Same finalisation order as the
// pre-Pipeline consumer loop.
if let Err(e) = self.file.sync_all() {
if self.is_regular {
return Err(Error::IoError { source: e });
}
// Non-regular outputs (/dev/null, pipes) always fail
// sync_all; that's not a real error.
}
self.map.flush()?;
Ok(ConsumerSummary {
stats: self.map.stats(),
})
}
}
+1021
View File
File diff suppressed because it is too large Load Diff
-169
View File
@@ -1,169 +0,0 @@
//! Drive session — open, identify, unlock, and read from optical drives.
//!
//! `DriveSession` is the entry point for all drive interaction. It handles
//! device identification, profile matching, platform-specific unlock, and
//! provides both raw sector reads and standard SCSI command execution.
//!
//! Two open modes:
//! - `open()` — identify + unlock. Ready for reading immediately.
//! - `open_no_unlock()` — identify only. Used for AACS authentication
//! which must happen before the drive enters raw mode.
use std::path::Path;
use crate::error::{Error, Result};
use crate::scsi::ScsiTransport;
use crate::identity::DriveId;
use crate::profile::{self, DriveProfile, Chipset};
use crate::platform::{Platform, DriveStatus};
use crate::platform::mt1959::Mt1959;
/// A drive session with identification, platform, and SCSI transport.
///
/// Created via `DriveSession::open()` or `DriveSession::open_no_unlock()`.
/// All disc reading goes through this struct.
pub struct DriveSession {
scsi: Box<dyn ScsiTransport>,
platform: Box<dyn Platform>,
pub profile: DriveProfile,
pub drive_id: DriveId,
device_path: String,
}
impl DriveSession {
/// Open a drive, identify it, match a profile, and unlock for raw reads.
///
/// This is the standard entry point. After `open()`, the drive is ready
/// for sector reads, disc scanning, and content extraction.
pub fn open(device: &Path) -> Result<Self> {
let mut session = Self::open_no_unlock(device)?;
let _ = session.unlock(); // silently ignore — unencrypted discs don't need it
Ok(session)
}
/// Open a drive WITHOUT unlocking.
///
/// Used when AACS authentication must happen before raw mode.
/// The AACS SCSI handshake requires the drive's standard firmware
/// state — unlocking puts the drive in vendor-specific raw mode
/// which disables the AACS layer.
pub fn open_no_unlock(device: &Path) -> Result<Self> {
let mut transport = crate::scsi::open(device)?;
let profiles = profile::load_bundled()?;
let drive_id = DriveId::from_drive(transport.as_mut())?;
let profile = profile::find_by_drive_id(&profiles, &drive_id)
.cloned()
.ok_or_else(|| Error::UnsupportedDrive {
vendor_id: drive_id.vendor_id.trim().to_string(),
product_id: drive_id.product_id.trim().to_string(),
product_revision: drive_id.product_revision.trim().to_string(),
})?;
let platform = create_platform(&profile, &drive_id)?;
Ok(DriveSession {
scsi: transport,
platform,
profile,
drive_id,
device_path: device.to_string_lossy().to_string(),
})
}
/// Open with an explicit profile, skipping auto-detection.
pub fn open_with_profile(device: &Path, profile: DriveProfile) -> Result<Self> {
let mut transport = crate::scsi::open(device)?;
let drive_id = DriveId::from_drive(transport.as_mut())?;
let platform = create_platform(&profile, &drive_id)?;
Ok(DriveSession {
scsi: transport,
platform,
profile,
drive_id,
device_path: device.to_string_lossy().to_string(),
})
}
/// Device path this session was opened on.
pub fn device_path(&self) -> &str {
&self.device_path
}
/// Activate raw disc access mode (vendor-specific unlock).
pub fn unlock(&mut self) -> Result<()> {
self.platform.unlock(self.scsi.as_mut())
}
/// Check if raw disc access mode is active.
pub fn is_unlocked(&self) -> bool {
self.platform.is_unlocked()
}
/// Read drive status and feature flags.
pub fn status(&mut self) -> Result<DriveStatus> {
self.platform.status(self.scsi.as_mut())
}
/// Read drive configuration block.
pub fn read_config(&mut self) -> Result<Vec<u8>> {
self.platform.read_config(self.scsi.as_mut())
}
/// Read hardware register.
pub fn read_register(&mut self, index: u8) -> Result<[u8; 16]> {
self.platform.read_register(self.scsi.as_mut(), index)
}
/// Calibrate read speed for the current disc.
pub fn calibrate(&mut self) -> Result<()> {
self.platform.calibrate(self.scsi.as_mut())
}
/// Read raw disc sectors via platform-specific command.
pub fn read_sectors(&mut self, lba: u32, count: u16, buf: &mut [u8]) -> Result<usize> {
self.platform.read_sectors(self.scsi.as_mut(), lba, count, buf)
}
/// Platform-specific probe command.
pub fn probe(&mut self, sub_cmd: u8, address: u32, length: u32) -> Result<Vec<u8>> {
self.platform.probe(self.scsi.as_mut(), sub_cmd, address, length)
}
/// Standard SCSI READ(10) for disc filesystem data (UDF, MPLS, CLPI).
pub fn read_disc(&mut self, lba: u32, count: u16, buf: &mut [u8]) -> Result<usize> {
let cdb = [
crate::scsi::SCSI_READ_10, 0x00,
(lba >> 24) as u8, (lba >> 16) as u8, (lba >> 8) as u8, lba as u8,
0x00,
(count >> 8) as u8, count as u8,
0x00,
];
let result = self.scsi.as_mut().execute(
&cdb, crate::scsi::DataDirection::FromDevice, buf, 5_000)?;
Ok(result.bytes_transferred)
}
/// Execute a raw SCSI CDB. Used by parsers and AACS handshake.
pub fn scsi_execute(
&mut self,
cdb: &[u8],
direction: crate::scsi::DataDirection,
buf: &mut [u8],
timeout_ms: u32,
) -> Result<crate::scsi::ScsiResult> {
self.scsi.as_mut().execute(cdb, direction, buf, timeout_ms)
}
}
/// Create the platform-specific driver for a given chipset.
fn create_platform(profile: &DriveProfile, drive_id: &DriveId) -> Result<Box<dyn Platform>> {
match profile.chipset {
Chipset::MediaTek => Ok(Box::new(Mt1959::new(profile.clone()))),
Chipset::Renesas => Err(Error::UnsupportedDrive {
vendor_id: drive_id.vendor_id.trim().to_string(),
product_id: drive_id.product_id.trim().to_string(),
product_revision: "Renesas not yet implemented".to_string(),
}),
}
}
+196
View File
@@ -0,0 +1,196 @@
//! Drive data capture — read hardware information via SCSI.
use crate::drive::Drive;
use crate::error::Result;
/// Raw data captured from a drive's SCSI responses.
#[derive(Debug, Clone)]
pub struct DriveCapture {
/// Raw INQUIRY response (96 bytes)
pub inquiry: Vec<u8>,
/// Raw GET_CONFIG 010C response
pub gc_010c: Vec<u8>,
/// GET_CONFIG feature responses: (feature_code, feature_name, data)
pub features: Vec<CapturedFeature>,
/// REPORT_KEY RPC state
pub rpc_state: Option<Vec<u8>>,
/// MODE SENSE page 2A (capabilities)
pub mode_2a: Option<Vec<u8>>,
/// READ_BUFFER 0xF1 (Pioneer vendor data)
pub rb_f1: Option<Vec<u8>>,
/// READ_BUFFER mode 6 (MTK vendor data)
pub rb_mode6: Option<Vec<u8>>,
}
/// A single GET CONFIGURATION feature response from the drive.
#[derive(Debug, Clone)]
pub struct CapturedFeature {
/// MMC-6 GET CONFIGURATION feature code (e.g. `0x010D` = AACS).
pub code: u16,
/// Static human-readable label from the internal `FEATURES` table —
/// not a device-reported string.
pub name: &'static str,
/// Raw feature-descriptor payload bytes, with the 8-byte GET
/// CONFIGURATION header stripped (i.e. `buf[8..]`). Unlike
/// [`DriveCapture::gc_010c`], which retains the full header.
pub data: Vec<u8>,
}
/// Feature codes to capture.
const FEATURES: &[(u16, &str)] = &[
(0x0000, "Profile List"),
(0x0001, "Core"),
(0x0003, "Removable Medium"),
(0x0010, "Random Readable"),
(0x001D, "Multi-Read"),
(0x001E, "CD Read"),
(0x001F, "DVD Read"),
(0x0040, "BD Read"),
(0x0041, "BD Write"),
(0x0100, "Power Management"),
(0x0102, "Embedded Changer"),
(0x0107, "Real Time Streaming"),
(0x0108, "Serial Number"),
(0x010C, "Firmware Information"),
(0x010D, "AACS"),
];
/// Capture all available drive data via SCSI commands.
/// Returns raw responses — no formatting, no zipping, no presentation.
pub fn capture_drive_data(session: &mut Drive) -> Result<DriveCapture> {
let id = &session.drive_id;
// Already have INQUIRY from drive open
let inquiry = id.raw_inquiry.clone();
let gc_010c = id.raw_gc_010c.clone();
// Capture GET_CONFIG features using Drive's query methods
let mut features = Vec::new();
for &(code, name) in FEATURES {
if let Some(data) = session.get_config_feature(code) {
features.push(CapturedFeature { code, name, data });
}
}
// Vendor-specific READ_BUFFER queries
let rb_f1 = session.read_buffer(0x02, 0xF1, 48); // Pioneer
let rb_mode6 = session.read_buffer(0x06, 0x00, 32); // MTK
// Standard queries
let rpc_state = session.report_key_rpc_state();
let mode_2a = session.mode_sense_page(0x2A);
Ok(DriveCapture {
inquiry,
gc_010c,
features,
rpc_state,
mode_2a,
rb_f1,
rb_mode6,
})
}
/// Mask a string for privacy (letters->A, digits->0).
pub fn mask_string(s: &str) -> String {
s.chars()
.map(|c| {
if c.is_ascii_alphabetic() {
'A'
} else if c.is_ascii_digit() {
'0'
} else {
c
}
})
.collect()
}
/// Mask bytes for privacy.
pub fn mask_bytes(data: &[u8]) -> Vec<u8> {
data.iter()
.map(|&b| {
if b.is_ascii_alphabetic() {
b'A'
} else if b.is_ascii_digit() {
b'0'
} else {
b
}
})
.collect()
}
#[cfg(test)]
mod tests {
//! Privacy-masking + capture-orchestration tests.
//!
//! `mask_string` / `mask_bytes` redact identifying characters before
//! a drive capture leaves the machine: every ASCII letter → 'A',
//! every ASCII digit → '0', everything else (punctuation, spaces,
//! control bytes, non-ASCII) is preserved verbatim so structural
//! framing (offsets, separators) survives for diffing.
use super::*;
#[test]
fn mask_string_letters_become_a_digits_become_zero() {
// Mixed case letters all collapse to 'A'; digits to '0'.
assert_eq!(mask_string("HL-DT-ST"), "AA-AA-AA");
assert_eq!(mask_string("BU40N"), "AA00A");
}
#[test]
fn mask_string_preserves_non_alnum_punctuation_and_space() {
// Separators and spaces must be preserved so the masked output
// keeps the same shape as the original (the whole point of a
// structure-preserving redaction).
assert_eq!(mask_string("1.04"), "0.00");
assert_eq!(mask_string("a b-c.d_e"), "A A-A.A_A");
}
#[test]
fn mask_string_preserves_non_ascii_chars() {
// is_ascii_alphabetic/is_ascii_digit are false for non-ASCII, so
// multibyte chars pass through unchanged (no mojibake, no panic).
// 'c','a','f' are ASCII letters → 'A'; 'é' is non-ASCII →
// preserved; '9' → '0'.
assert_eq!(mask_string("café9"), "AAAé0");
}
#[test]
fn mask_bytes_matches_string_masking_for_ascii() {
// mask_bytes is the byte-wise analogue: letters→b'A', digits→b'0'.
assert_eq!(mask_bytes(b"HL-DT-ST"), b"AA-AA-AA".to_vec());
assert_eq!(mask_bytes(b"1.04"), b"0.00".to_vec());
}
#[test]
fn mask_bytes_preserves_non_alnum_and_high_bytes() {
// Control bytes (0x00), high bytes (0xFF), and punctuation are
// not ASCII alnum and must survive verbatim — INQUIRY payloads
// are space-padded binary and the framing must be diffable.
let input = [0x00u8, b'A', 0x20, b'7', 0xFF, b'-'];
assert_eq!(mask_bytes(&input), vec![0x00, b'A', 0x20, b'0', 0xFF, b'-']);
}
#[test]
fn feature_table_has_no_duplicate_codes() {
// capture_drive_data iterates FEATURES once per code; a duplicate
// code would silently capture the same feature twice (and bloat
// the report). Each MMC-6 feature code must be unique.
let mut seen = std::collections::HashSet::new();
for &(code, _name) in FEATURES {
assert!(seen.insert(code), "duplicate feature code {code:#06x}");
}
}
#[test]
fn feature_table_includes_aacs_010d() {
// AACS (0x010D) is the feature that gates UHD decryption capture;
// it must be in the table or AACS drives capture incompletely.
assert!(
FEATURES.iter().any(|&(c, _)| c == 0x010D),
"AACS feature 0x010D must be captured"
);
}
}
+113
View File
@@ -0,0 +1,113 @@
//! Linux drive discovery and device resolution.
use crate::drive::DeviceResolution;
use crate::error::{Error, Result};
use crate::identity::DriveId;
/// SCSI peripheral device type 5 = MMC / optical (CD/DVD/BD), held in the
/// low 5 bits of INQUIRY byte 0 (the high 3 bits are the peripheral
/// qualifier, masked off here).
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives by enumerating `/dev/sg*` SCSI-generic nodes,
/// opening each, running INQUIRY, and keeping only devices whose
/// peripheral device type is optical (MMC, type 0x05).
///
/// Devices where `scsi::open` or `DriveId::from_drive` fail are silently
/// skipped — that is intentional for enumeration (a busy or wedged node
/// shouldn't abort discovery of the others).
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
for name in enumerate_sg_names() {
let path = format!("/dev/{name}");
if !std::path::Path::new(&path).exists() {
continue;
}
if let Ok(mut transport) = crate::scsi::open(std::path::Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((path, id));
}
}
}
}
drives
}
/// Enumerate `sg*` device names. Linux assigns `/dev/sgN` sequentially
/// across *all* SCSI-generic devices (disks, tape, HBAs, optical), so a
/// fixed `sg0..15` range can miss an optical drive on a host with many
/// targets. Prefer the exact present-device list from
/// `/sys/class/scsi_generic/`; fall back to a bounded `sg0..15` probe
/// only when sysfs is unreadable (minimal containers).
fn enumerate_sg_names() -> Vec<String> {
let mut names = Vec::new();
if let Ok(entries) = std::fs::read_dir("/sys/class/scsi_generic") {
for entry in entries.flatten() {
let name = entry.file_name().to_string_lossy().to_string();
if name.starts_with("sg") {
names.push(name);
}
}
} else {
for i in 0..16 {
let name = format!("sg{i}");
if std::path::Path::new(&format!("/dev/{name}")).exists() {
names.push(name);
}
}
}
names.sort();
names
}
/// Resolve a device path to its raw `/dev/sg*` SCSI-generic node.
///
/// - `/dev/sg*` paths pass through unchanged ([`DeviceResolution::Direct`]).
/// - `/dev/sr*` block paths are matched (by vendor/product/serial) to the
/// corresponding `/dev/sg*` node ([`DeviceResolution::SrToSg`]); if no
/// match is found the original path is returned with
/// [`DeviceResolution::SrNoSgMatch`].
/// - Any other existing path passes through as [`DeviceResolution::Direct`].
#[allow(dead_code)]
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
if path.contains("/sg") {
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
return Ok((path.to_string(), DeviceResolution::Direct));
}
if path.contains("/sr") {
let mut sr_transport = crate::scsi::open(std::path::Path::new(path))?;
let sr_id = DriveId::from_drive(sr_transport.as_mut())?;
drop(sr_transport);
for (sg_path, sg_id) in find_drives() {
// Require a non-empty serial before treating vendor/product/
// serial as a unique match. serial_number falls back to an
// empty string when GET CONFIGURATION 0108h is unavailable
// (common on OEM drives); two same-model drives would then
// both compare equal and the first in enumeration order would
// win silently, resolving sr1 to sr0's sg node. An empty
// serial can't disambiguate, so fall through to the no-match
// path instead.
if !sr_id.serial_number.is_empty()
&& sg_id.vendor_id == sr_id.vendor_id
&& sg_id.product_id == sr_id.product_id
&& sg_id.serial_number == sr_id.serial_number
{
return Ok((sg_path, DeviceResolution::SrToSg));
}
}
return Ok((path.to_string(), DeviceResolution::SrNoSgMatch));
}
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
Ok((path.to_string(), DeviceResolution::Direct))
}
+55
View File
@@ -0,0 +1,55 @@
//! macOS drive discovery and device resolution.
//!
//! `find_drives` uses IOKit registry enumeration (via `scsi::list_drives`)
//! to discover optical drives without exclusive access or unmounts. Only
//! the returned paths are then opened for INQUIRY to build full `DriveId`.
use crate::drive::DeviceResolution;
use crate::error::{Error, Result};
use crate::identity::DriveId;
/// SCSI peripheral device type 5 = MMC / optical, in the low 5 bits of
/// INQUIRY byte 0.
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives via the IOKit registry (`scsi::list_drives`),
/// then open each candidate for INQUIRY to build a full `DriveId`.
///
/// Any drive where `scsi::open` or `DriveId::from_drive` fails, or whose
/// peripheral device type is not optical (MMC, type 0x05), is silently
/// skipped — the same MMC filter the Linux and Windows backends apply.
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
let discovered = crate::scsi::list_drives();
for info in discovered {
let path = std::path::Path::new(&info.path);
match crate::scsi::open(path) {
Ok(mut transport) => {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((info.path.clone(), id));
}
}
}
Err(_) => {
continue;
}
}
}
drives
}
/// Resolve a device path on macOS. There is no `sr`→`sg` style
/// substitution here (that is a Linux concern), so any existing path is
/// returned unchanged as [`DeviceResolution::Direct`]; the
/// [`DeviceResolution`] return exists for cross-platform signature parity.
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
Ok((path.to_string(), DeviceResolution::Direct))
}
+2009
View File
File diff suppressed because it is too large Load Diff
+99
View File
@@ -0,0 +1,99 @@
//! Windows drive discovery and device resolution.
use crate::drive::DeviceResolution;
use crate::error::Result;
use crate::identity::DriveId;
use std::path::Path;
/// SCSI peripheral device type 5 = MMC / optical, in the low 5 bits of
/// INQUIRY byte 0.
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives. Probes `\\.\CdRom0..15` first; only if none
/// are found does it fall back to scanning drive letters `D..Z`. Each
/// candidate is opened, INQUIRY'd, and kept only if its peripheral device
/// type is optical (MMC, type 0x05). Returns normalized `\\.\` paths.
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
// Try CdRom0..CdRom15
for i in 0..16 {
let path = format!("\\\\.\\CdRom{}", i);
if let Ok(mut transport) = crate::scsi::open(Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((path, id));
}
}
}
}
// Also try drive letters if CdRom didn't find anything
if drives.is_empty() {
for letter in b'D'..=b'Z' {
let path = format!("{}:", letter as char);
if let Ok(mut transport) = crate::scsi::open(Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
// Normalize so returned paths are consistently in
// \\.\ form regardless of which loop matched.
drives.push((normalize_path(&path), id));
}
}
}
}
}
drives
}
/// Resolve a device path to its normalized Windows `\\.\` form. Windows
/// has no `sr`→`sg` symlink-target indirection, so resolution is purely a
/// path normalization and always reports [`DeviceResolution::Direct`].
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
Ok((normalize_path(path), DeviceResolution::Direct))
}
/// Normalize a device path to Windows \\.\X: format.
///
/// Accepts: "D:", "D:\\", "\\.\D:", "\\.\CdRom0"
///
/// NOTE: A near-identical `normalize_device_path` exists in `scsi::windows`.
/// Both are kept because they live in separate `cfg(windows)` modules that
/// cannot easily share a helper without introducing cross-module coupling.
fn normalize_path(path: &str) -> String {
if path.starts_with("\\\\.\\") {
return path.to_string();
}
let trimmed = path.trim_end_matches('\\');
if trimmed.len() == 2 && trimmed.as_bytes()[1] == b':' {
return format!("\\\\.\\{}", trimmed);
}
format!("\\\\.\\{}", path)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn normalize_drive_letter() {
assert_eq!(normalize_path("D:"), "\\\\.\\D:");
assert_eq!(normalize_path("E:\\"), "\\\\.\\E:");
}
#[test]
fn normalize_already_prefixed() {
assert_eq!(normalize_path("\\\\.\\D:"), "\\\\.\\D:");
assert_eq!(normalize_path("\\\\.\\CdRom0"), "\\\\.\\CdRom0");
}
#[test]
fn normalize_cdrom() {
assert_eq!(normalize_path("CdRom0"), "\\\\.\\CdRom0");
}
}
+49
View File
@@ -0,0 +1,49 @@
//! DVD-Video navigation — read-only resolver for the **main-feature start
//! point** (issue #40). Mirrors what a DVD player's nav VM resolves: First-Play
//! → menu "Play" → title dispatch → the first cell of the feature, so the rip
//! starts at the movie rather than at raw cell 0 (e.g. skipping a leading
//! logo/warning segment when the disc's own navigation does).
//!
//! Byte layout follows the DVD-Video specification (VMGI/VTSI headers,
//! PGC/cell tables, PCI/HLI button packets); the VM command decoder is
//! verified against libdvdnav's decoder.
//!
//! Current contents: [`vmcmd`] — the VM command decoder (proven against the
//! SOTL/Greenland test discs). The IFO/PCI parsing and the navigation executor
//! that resolves the start cell build on top of this.
pub mod vmcmd;
use crate::sector::SectorSource;
/// Resolve the feature title's **true start cell** (0-based index into the
/// title PGC's cell list) by following the disc's own navigation — First-Play →
/// menu "Play" → title dispatch — the way a player reaches the movie. This is
/// what lets the rip begin at the feature instead of at raw cell 0 when the
/// disc's nav enters the title past a leading logo/warning segment (e.g. a
/// disc whose "Play" resolves to a later cell than cell 0).
///
/// Returns `None` when navigation cannot be resolved, so the caller falls back
/// to the structural leading-cell filter (today's behaviour, ≈ cell 0 / 0:00).
///
/// TODO(#40): the IFO/PCI parsing + nav executor (built on [`vmcmd`]) land
/// incrementally. Until the executor is complete this returns `None`, so wiring
/// it in is behaviour-neutral; improvements to the resolver take effect here
/// without touching the call site.
pub fn resolve_feature_start(
reader: &mut dyn SectorSource,
udf: &crate::udf::UdfFs,
vtsn: u16,
vts_ttn: u16,
) -> Option<usize> {
// `reader`/`udf` are the seam inputs the nav executor will consume to read
// VIDEO_TS.IFO + the VTS IFOs/menu VOBs. Reserved until that lands.
let _ = (reader, udf);
tracing::trace!(
target: "freemkv::dvdnav",
vtsn,
vts_ttn,
"nav start-cell resolver: unresolved — caller falls back to leading-cell filter"
);
None
}
+408
View File
@@ -0,0 +1,408 @@
//! DVD-Video VM command decoder.
//!
//! An 8-byte navigation command as found in PGC command tables (pre/post/cell)
//! and PCI button info. Decoded per the DVD-Video VM instruction set and
//! verified against libdvdnav's command decoder.
//!
//! Bit model: the 8 bytes are a big-endian 64-bit word. `byte0` bits 7-5 are the
//! command **type**; for type 1, `byte0` bit 4 selects Link (0) vs Jump (1), and
//! `byte1` bits 3-0 are the sub-command. Compare predicates live in `byte1`
//! bits 6-4 with the operands in bytes 2-5.
//!
//! This module is pure decode + a register model — no I/O, no English (numeric
//! semantics only), matching libfreemkv conventions. The navigation *executor*
//! and IFO/PCI parsing build on top of this.
/// A decoded navigation instruction. Only the variants freemkv's start-point
/// resolver needs are modelled explicitly; everything else is [`Instr::Other`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Instr {
Nop,
/// Stop executing the current command list (resume cell playback).
Break,
/// Goto command line within the same list (1-based).
Goto {
line: u8,
},
/// Leave the current domain.
Exit,
/// Jump to a VMG title (1-based TT_SRPT index).
JumpTt {
ttn: u8,
},
/// Jump to a title within the current VTS (1-based VTS title index).
JumpVtsTt {
ttn: u8,
},
/// Jump to a part-of-title (chapter) within a VTS title.
JumpVtsPtt {
ttn: u8,
pttn: u16,
},
/// Jump to the First-Play PGC.
JumpSsFp,
/// Jump to a Video-Manager menu (`menu` = menu id).
JumpSsVmgm {
menu: u8,
},
/// Jump to a Video-Title-Set menu.
JumpSsVtsm {
vts: u8,
ttn: u8,
menu: u8,
},
/// Jump to a specific VMGM menu PGC.
JumpSsVmgmPgc {
pgcn: u16,
},
/// Call a sub-domain (raw retained; resume handled by the executor).
CallSs {
sub: u8,
},
/// Link to a PGC number within the current domain.
LinkPgcn {
pgcn: u16,
},
/// Link to a part-of-title within the current PGC's title.
LinkPttn {
pttn: u16,
},
/// Link to a program number within the current PGC (1-based).
LinkPgn {
pgn: u8,
},
/// Link to a cell number within the current PGC (1-based).
LinkCn {
cn: u8,
},
/// A link "subset" op (LinkTopCell/NextPG/RSM/…); `sub` is the raw code.
LinkSub {
sub: u8,
},
/// Set a GPRM. `op` is the set-op code (1=mov, 3=add, …); value is immediate
/// (`imm`) when `immediate`, else the contents of register `src`.
SetGprm {
reg: u8,
op: u8,
immediate: bool,
imm: u16,
src: u8,
},
/// Set a system parameter / unmodelled set — executor may ignore.
SetSystem,
/// Anything not individually modelled (kept as raw bytes).
Other([u8; 8]),
}
/// A compare predicate carried by a command (`byte1` bits 6-4). `None` = always.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Compare {
/// Compare op: 1=&,2===,3=!=,4=>=,5=>,6=<=,7=<.
pub op: u8,
/// Left register index (GPRM 0-15, SPRM 128+).
pub lhs_reg: u8,
/// Right side: immediate when `immediate`, else register `rhs_reg`.
pub immediate: bool,
pub imm: u16,
pub rhs_reg: u8,
}
/// A fully decoded command: its predicate (if any) and the instruction.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Command {
pub compare: Option<Compare>,
pub instr: Instr,
}
// Command types — `byte0` bits 7-5.
const TYPE_SPECIAL: u8 = 0;
const TYPE_LINK_JUMP: u8 = 1;
const TYPE_SET_SYSTEM: u8 = 2;
const TYPE_SET_GPRM: u8 = 3;
// Special (type 0) sub-commands — `byte1` bits 3-0.
const SP_GOTO: u8 = 1;
const SP_BREAK: u8 = 2;
// Jump/Call (type 1, direct=1) sub-commands.
const JP_EXIT: u8 = 1;
const JP_JUMP_TT: u8 = 2;
const JP_JUMP_VTS_TT: u8 = 3;
const JP_JUMP_VTS_PTT: u8 = 5;
const JP_JUMP_SS: u8 = 6;
const JP_CALL_SS: u8 = 8;
// Link (type 1, direct=0) sub-commands. NOTE: sub-op 0 is NOP/no-link and 1 is
// the LinkSub form (libdvdnav `decoder.c` `eval_link_instruction`).
const LK_SUB: u8 = 1;
const LK_PGCN: u8 = 4;
const LK_PTTN: u8 = 5;
const LK_PGN: u8 = 6;
const LK_CN: u8 = 7;
// JumpSS sub-domain selector — `byte5` bits 7-6.
const SS_FP: u8 = 0;
const SS_VMGM_MENU: u8 = 1;
const SS_VTSM: u8 = 2;
// Operand field widths (spec-defined bit counts).
const MASK_TTN: u8 = 0x7F; // 7-bit title number
const MASK_PGN: u8 = 0x7F; // 7-bit program number
const MASK_LINKOP: u8 = 0x1F; // 5-bit link sub-op
const MASK_REG: u8 = 0x0F; // 4-bit GPRM index
const MASK_MENU: u8 = 0x0F; // 4-bit menu id
const MASK_PTTN: u16 = 0x03FF; // 10-bit part-of-title
const MASK_PGCN: u16 = 0x7FFF; // 15-bit PGC number
#[inline]
fn be16(b: &[u8; 8], o: usize) -> u16 {
((b[o] as u16) << 8) | b[o + 1] as u16
}
// Compare-operand layouts ("if_version"s) per libdvdnav `decoder.c`. The op
// nibble is always `byte1` bits 6-4; the immediate flag is `byte1` bit 7. The
// operand *offsets* differ by command family.
//
// v1 (special + link): lhs reg = b[3]; rhs imm = bytes4-5 / rhs reg = b[4].
// v2 (jump + system-set): lhs reg = b[6]; rhs reg = b[7] (registers only).
// v3 (set-GPRM): lhs reg = b[2]; rhs imm = bytes6-7 / rhs reg = b[6].
fn if_v1(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[3],
immediate: b[1] >> 7 != 0,
imm: be16(b, 4),
rhs_reg: b[4],
})
}
fn if_v2(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[6],
immediate: false,
imm: 0,
rhs_reg: b[7],
})
}
fn if_v3(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[2],
immediate: b[1] >> 7 != 0,
imm: be16(b, 6),
rhs_reg: b[6],
})
}
/// Decode an 8-byte VM command.
pub fn decode(b: &[u8; 8]) -> Command {
let typ = b[0] >> 5;
let direct = (b[0] >> 4) & 1;
let setop = b[0] & 0x0F;
let cmd = b[1] & 0x0F;
// Compare predicate, with the operand layout for this command family
// (libdvdnav `decoder.c` `vm_eval_command` type dispatch).
let compare = match (typ, direct) {
(TYPE_SPECIAL, _) => if_v1(b),
(TYPE_LINK_JUMP, 1) => if_v2(b), // jump
(TYPE_LINK_JUMP, 0) => if_v1(b), // link
(TYPE_SET_SYSTEM, _) => if_v2(b),
(TYPE_SET_GPRM, _) => if_v3(b),
_ => None, // 4/5/6 compound — not needed by the resolver
};
// JumpSS sub-domain selector lives in byte5 bits 7-6.
let ss_sel = b[5] >> 6;
let instr = match typ {
TYPE_LINK_JUMP if direct == 1 => match cmd {
JP_EXIT => Instr::Exit,
JP_JUMP_TT => Instr::JumpTt {
ttn: b[5] & MASK_TTN,
},
JP_JUMP_VTS_TT => Instr::JumpVtsTt {
ttn: b[5] & MASK_TTN,
},
JP_JUMP_VTS_PTT => Instr::JumpVtsPtt {
ttn: b[5] & MASK_TTN,
pttn: be16(b, 2) & MASK_PTTN,
},
JP_JUMP_SS => match ss_sel {
SS_FP => Instr::JumpSsFp,
SS_VMGM_MENU => Instr::JumpSsVmgm {
menu: b[5] & MASK_MENU,
},
SS_VTSM => Instr::JumpSsVtsm {
vts: b[4],
ttn: b[3],
menu: b[5] & MASK_MENU,
},
_ => Instr::JumpSsVmgmPgc {
pgcn: be16(b, 2) & MASK_PGCN,
},
},
JP_CALL_SS => Instr::CallSs { sub: ss_sel },
_ => Instr::Nop,
},
TYPE_LINK_JUMP => match cmd {
// direct == 0 (link). sub-op 0 = NOP/no-link.
LK_SUB => Instr::LinkSub {
sub: b[7] & MASK_LINKOP,
},
LK_PGCN => Instr::LinkPgcn {
pgcn: be16(b, 6) & MASK_PGCN,
},
LK_PTTN => Instr::LinkPttn {
pttn: be16(b, 6) & MASK_PTTN,
},
LK_PGN => Instr::LinkPgn {
pgn: b[7] & MASK_PGN,
},
LK_CN => Instr::LinkCn { cn: b[7] },
_ => Instr::Nop,
},
TYPE_SPECIAL => match cmd {
SP_GOTO => Instr::Goto { line: b[7] },
SP_BREAK => Instr::Break,
_ => Instr::Nop,
},
TYPE_SET_GPRM => Instr::SetGprm {
reg: b[3] & MASK_REG,
op: setop,
immediate: direct != 0,
imm: be16(b, 4),
src: b[5],
},
TYPE_SET_SYSTEM => Instr::SetSystem,
_ => Instr::Other(*b),
};
Command { compare, instr }
}
#[cfg(test)]
mod tests {
use super::*;
fn h(s: &str) -> [u8; 8] {
let v: Vec<u8> = (0..8)
.map(|i| u8::from_str_radix(&s[i * 2..i * 2 + 2], 16).unwrap())
.collect();
v.try_into().unwrap()
}
// KATs taken from the real SOTL / Greenland discs (decoded in the PoC).
#[test]
fn greenland_first_play_is_jumptt_1() {
let c = decode(&h("3002000000010000"));
assert_eq!(c.instr, Instr::JumpTt { ttn: 1 });
assert!(c.compare.is_none());
}
#[test]
fn sotl_first_play_is_jumpss_vtsm_root() {
// 30 06 ... byte5=0x83 -> sub 2 (VTSM), vts=byte4=1, menu=byte5&0xF=3 (root)
let c = decode(&h("3006000101830000"));
assert_eq!(
c.instr,
Instr::JumpSsVtsm {
vts: 1,
ttn: 1,
menu: 3
}
);
}
#[test]
fn sotl_title_dispatch_is_conditional_linkpgn_2() {
// 20 a6 ... CmpLink: if GPRM0 == 2 -> LinkPGN 2 (cell 2 = the 5:02 start)
let c = decode(&h("20a6000000020002"));
assert_eq!(c.instr, Instr::LinkPgn { pgn: 2 });
let cmp = c.compare.expect("conditional");
assert_eq!(cmp.op, 2); // ==
assert_eq!(cmp.lhs_reg, 0); // GPRM0
assert!(cmp.immediate);
assert_eq!(cmp.imm, 2);
}
#[test]
fn sotl_root_button_is_linkpgcn_37() {
assert_eq!(
decode(&h("2004000000000025")).instr,
Instr::LinkPgcn { pgcn: 37 }
);
}
#[test]
fn greenland_scene_button_is_linkpgn() {
assert_eq!(
decode(&h("2006000000001401")).instr,
Instr::LinkPgn { pgn: 1 }
);
}
#[test]
fn jumpvts_ptt_decodes_ttn_and_pttn() {
// synthetic: 30 05 | ptt(bytes2-3)=0x0002 | ttn(byte5)=1
let c = decode(&h("3005000200010000"));
assert_eq!(c.instr, Instr::JumpVtsPtt { ttn: 1, pttn: 2 });
}
#[test]
fn setgprm_immediate_mov() {
// SOTL First-Play pre[0]: 71 00 | reg=byte3=6 | imm(bytes4-5)=0x03e8 -> g6 = 1000
match decode(&h("7100000603e80000")).instr {
Instr::SetGprm {
reg,
op,
immediate,
imm,
..
} => {
assert_eq!(reg, 6);
assert_eq!(op, 1); // mov
assert!(immediate);
assert_eq!(imm, 1000);
}
other => panic!("expected SetGprm, got {other:?}"),
}
}
// Regression for the libdvdnav cross-check: link sub-op 0 = NOP, 1 = LinkSub.
#[test]
fn link_subop_zero_is_nop_one_is_linksub() {
assert_eq!(decode(&h("2000000000000000")).instr, Instr::Nop);
assert_eq!(
decode(&h("2001000000000010")).instr,
Instr::LinkSub { sub: 0x10 }
);
}
// if_version_1 register compare: rhs register is byte4 (not byte5).
#[test]
fn link_register_compare_rhs_is_byte4() {
// 20 26: link, cmp=EQ(2), dircmp=0(register) ; cmd=6 LinkPGN
let c = decode(&h("2026000304000002"));
assert_eq!(c.instr, Instr::LinkPgn { pgn: 2 });
let cmp = c.compare.expect("conditional");
assert!(!cmp.immediate);
assert_eq!(cmp.lhs_reg, 3);
assert_eq!(cmp.rhs_reg, 4);
}
// if_version_2 jump compare: both operands are registers in byte6 / byte7.
#[test]
fn jump_compare_uses_bytes6_and_7() {
// 30 22: jump, cmp=EQ(2) ; cmd=2 JumpTT ttn=byte5=5
let c = decode(&h("3022000000050607"));
assert_eq!(c.instr, Instr::JumpTt { ttn: 5 });
let cmp = c.compare.expect("conditional");
assert!(!cmp.immediate);
assert_eq!(cmp.lhs_reg, 6);
assert_eq!(cmp.rhs_reg, 7);
}
}
+1592 -70
View File
File diff suppressed because it is too large Load Diff
+150
View File
@@ -0,0 +1,150 @@
//! Event system for progress and status reporting.
//!
//! The lib fires events during operations like rip().
//! The app registers a callback to receive them.
//! No display logic, no English text — just data.
//!
//! ```rust,ignore
//! disc.rip(&mut session, 0, output, |event| {
//! match event.kind {
//! EventKind::BytesRead { bytes, total } => update_progress(bytes, total),
//! EventKind::SectorSkipped { sector } => log_skip(sector),
//! EventKind::BatchSizeChanged { new_size, .. } => note_recovery(new_size),
//! _ => {}
//! }
//! });
//! ```
//!
//! Note: the library currently emits only `BytesRead`, `SectorSkipped`,
//! and `BatchSizeChanged`. The other [`EventKind`] variants are part of
//! the stable event vocabulary for consumers (and future emit sites) but
//! are not produced by the library today.
use crate::error::Error;
/// An event fired by the lib during operations.
#[derive(Debug)]
pub struct Event {
pub kind: EventKind,
}
/// Types of events the lib can fire.
#[derive(Debug)]
pub enum EventKind {
// ── Init sequence events ────────────────────────────────────────
/// Drive opened successfully.
DriveOpened { device: String },
/// Drive is ready (disc spun up).
DriveReady,
/// Firmware init completed.
InitComplete { success: bool },
/// Disc probe completed.
ProbeComplete { success: bool },
/// Disc scan completed.
ScanComplete { titles: usize },
// ── Read events ─────────────────────────────────────────────────
/// Bytes successfully read and written to output.
BytesRead {
/// Bytes written so far.
bytes: u64,
/// Total bytes expected (0 if unknown).
total: u64,
},
/// A read error occurred. The lib will retry automatically.
ReadError {
/// Sector that failed.
sector: u64,
/// Error code.
error: Error,
},
/// Retrying a failed read.
Retry {
/// Current attempt number (1-based).
attempt: u32,
},
/// Drive speed changed (error recovery or restoration).
SpeedChange {
/// New speed in KB/s (0xFFFF = max).
speed_kbs: u16,
},
/// Starting a new disc extent.
ExtentStart {
/// Extent index (0-based).
index: usize,
/// First sector of extent.
start_sector: u64,
/// Number of sectors in extent.
sector_count: u64,
},
/// Sector recovered after a retry (Drive::read multi-phase recovery).
SectorRecovered { sector: u64 },
/// Sector unreadable, zero-filled (skip mode).
SectorSkipped { sector: u64 },
/// Adaptive batch sizer changed the read size.
///
/// Fires on shrink (read failed at larger size) and on probe-up
/// (enough clean reads to try larger again). Consumers use this to
/// display a "recovering" state distinct from "ripping normally".
BatchSizeChanged {
new_size: u16,
reason: BatchSizeReason,
},
/// Operation complete.
Complete {
/// Total bytes written.
bytes: u64,
/// Total read errors encountered.
errors: u32,
},
}
/// Why the adaptive batch sizer changed size.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum BatchSizeReason {
/// Read failed; sizer halved the batch.
Shrunk,
/// Clean-read streak threshold hit; sizer doubled toward preferred.
Probed,
}
/// A no-op event handler. Ignores all events.
pub fn ignore(_event: Event) {}
#[cfg(test)]
mod tests {
use super::*;
/// BatchSizeReason::Shrunk != BatchSizeReason::Probed.
/// These two variants carry distinct meanings (error vs. recovery); they
/// must not compare as equal.
/// Mutation: deriving PartialEq without proper variant discrimination
/// could make two distinct variants equal.
#[test]
fn batch_size_reason_variants_are_not_equal() {
assert_ne!(BatchSizeReason::Shrunk, BatchSizeReason::Probed);
}
/// BatchSizeReason is Clone + Copy: cloning does not move the original.
/// This is required because EventKind::BatchSizeChanged embeds it by value.
/// Mutation: removing Copy would require the caller to clone explicitly;
/// code that passes reason by value would fail to compile.
#[test]
fn batch_size_reason_is_copy() {
let r = BatchSizeReason::Shrunk;
let _r2 = r; // copy, not move
let _r3 = r; // r still usable after copy
}
}
+191
View File
@@ -0,0 +1,191 @@
//! One-bit cooperative cancellation flag.
//!
//! `Halt` is a clonable token wrapping `Arc<AtomicBool>`. Pass clones into
//! every long-running loop; the loop polls `is_cancelled()` and bails out
//! cleanly. Calling `cancel()` from any clone flips the shared flag, and
//! every other clone observes it on its next poll.
//!
//! Why: `Ordering::Relaxed` is sufficient on both load and store because
//! this flag is purely advisory — no other memory operations piggyback on
//! it for happens-before ordering. Callers that need to publish data
//! across threads do so via channels or other synchronization, not via
//! this bit.
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
/// Clonable, infallible cooperative-cancellation token.
///
/// Clones share the same underlying flag. `cancel()` is one-way; there is
/// no `reset()` by design — construct a fresh `Halt` for a fresh
/// operation.
///
/// Construct with [`Halt::new`] (or [`Halt::default`]). The `Default`
/// impl forwards to `new()` — both produce a fresh, uncancelled token.
/// The pair exists because clippy's `new_without_default` lint requires
/// `Default` whenever a public `new()` is present, even when the two
/// would do exactly the same thing.
#[derive(Clone, Debug)]
pub struct Halt(Arc<AtomicBool>);
impl Halt {
/// Construct a fresh, uncancelled token.
pub fn new() -> Self {
Self(Arc::new(AtomicBool::new(false)))
}
/// Wrap an existing `Arc<AtomicBool>` as a `Halt`. A bridge for
/// callers that already hold an `Arc<AtomicBool>` cancellation flag
/// and want to adopt the token API without allocating a new flag.
///
/// Cancelling either side flips the same bit — the wrapping `Halt`
/// and the original `Arc` are two views over one shared flag.
pub fn from_arc(flag: Arc<AtomicBool>) -> Self {
Self(flag)
}
/// Borrow the underlying `Arc<AtomicBool>`. The inverse of
/// [`from_arc`](Self::from_arc): hand the shared flag to an API that
/// still takes a raw `Arc<AtomicBool>` rather than a `Halt`.
pub fn as_arc(&self) -> &Arc<AtomicBool> {
&self.0
}
/// Flip the shared flag to cancelled. Idempotent.
pub fn cancel(&self) {
self.0.store(true, Ordering::Relaxed);
}
/// Read the shared flag.
pub fn is_cancelled(&self) -> bool {
self.0.load(Ordering::Relaxed)
}
}
impl Default for Halt {
fn default() -> Self {
Self::new()
}
}
/// Shared poll interval for halt-aware loops.
///
/// `bounded_syscall` checks the cancellation flag and the deadline
/// every [`POLL_INTERVAL`] while blocked on a worker; the same
/// cadence governs `Pipeline::send_with_halt`'s `try_send` retry.
/// 250 ms is the sweet spot between responsiveness (operator presses
/// Stop, sees it take effect within ~quarter-second) and waste
/// (atomic load + clock read is cheap but not free at thousands of
/// hertz).
///
/// Centralised here so the half-dozen halt-polling loops across `io`
/// can't drift apart silently.
pub const POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(250);
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn fresh_is_not_cancelled() {
let h = Halt::new();
assert!(!h.is_cancelled());
}
#[test]
fn cancel_flips_state() {
let h = Halt::new();
assert!(!h.is_cancelled());
h.cancel();
assert!(h.is_cancelled());
}
#[test]
fn cancel_is_idempotent() {
let h = Halt::new();
h.cancel();
h.cancel();
assert!(h.is_cancelled());
}
#[test]
fn clone_shares_state() {
let original = Halt::new();
let cloned = original.clone();
assert!(!original.is_cancelled());
assert!(!cloned.is_cancelled());
// Cancel via the clone; the original observes it.
cloned.cancel();
assert!(original.is_cancelled());
assert!(cloned.is_cancelled());
}
#[test]
fn clone_shares_state_reverse_direction() {
let original = Halt::new();
let cloned = original.clone();
// Cancel via the original; the clone observes it.
original.cancel();
assert!(cloned.is_cancelled());
}
#[test]
fn clone_shares_state_across_threads() {
let h = Halt::new();
let h2 = h.clone();
let handle = std::thread::spawn(move || {
h2.cancel();
});
handle.join().unwrap();
assert!(h.is_cancelled());
}
#[test]
fn from_arc_shares_state() {
// The 0.18 deprecation-window bridge: a Halt built from an
// existing Arc<AtomicBool> must be a *view* over the same bit,
// not a fresh copy. Cancelling either side flips both.
let arc = Arc::new(AtomicBool::new(false));
let halt = Halt::from_arc(arc.clone());
assert!(!halt.is_cancelled());
assert!(!arc.load(Ordering::Relaxed));
// Cancel via the wrapping Halt; the original Arc observes it.
halt.cancel();
assert!(arc.load(Ordering::Relaxed));
// Conversely: flip the Arc directly; the Halt view observes it.
let arc2 = Arc::new(AtomicBool::new(false));
let halt2 = Halt::from_arc(arc2.clone());
arc2.store(true, Ordering::Relaxed);
assert!(halt2.is_cancelled());
}
#[test]
fn as_arc_returns_backing_flag() {
// `as_arc()` must hand back the *same* Arc, not a clone of a
// different bit. Verified by writing through the borrowed Arc
// and observing through the Halt.
let halt = Halt::new();
let arc = halt.as_arc().clone();
assert!(!halt.is_cancelled());
arc.store(true, Ordering::Relaxed);
assert!(halt.is_cancelled());
}
// ── New comprehensive tests ────────────────────────────────────────────────
/// POLL_INTERVAL is 250ms — a specific value that the multi-thread halt
/// loops depend on for responsiveness guarantees.
/// Mutation: setting POLL_INTERVAL to 5s makes stop requests take 5s to notice.
#[test]
fn poll_interval_is_250ms() {
assert_eq!(
POLL_INTERVAL,
std::time::Duration::from_millis(250),
"POLL_INTERVAL must be 250ms for the guaranteed ~quarter-second cancel latency"
);
}
}
+106
View File
@@ -0,0 +1,106 @@
//! The single hex → bytes parser for the whole workspace.
//!
//! Key material arrives as hex from three third-party sources — the keydb, an
//! online key service, and the mapfile's `# freemkv-vid:` comment — and each
//! used to parse it slightly differently (one stripped `0x`/`0X`, one stripped
//! nothing, one stripped `0x` only). A key written with a prefix one parser
//! didn't expect was silently dropped → "can't decrypt" with no error. This is
//! the one parser they all call, so the prefix/case/validation rules live in
//! exactly one place.
//!
//! Operates on BYTES, not `&str` char indices: the inputs are untrusted, so a
//! multi-byte UTF-8 scalar must reject as malformed, never panic on a
//! mid-codepoint slice.
/// Parse a hex string into bytes. Accepts an optional `0x`/`0X` prefix
/// (case-insensitive), then requires an even run of ASCII hex digits. Any
/// non-hex byte, or an odd length, yields `None`.
pub fn parse_hex_bytes(s: &str) -> Option<Vec<u8>> {
let body = strip_prefix(s.trim());
let bytes = body.as_bytes();
// Empty → empty Vec (a legitimately-empty variable-length field); odd length
// is malformed. (`parse_hex_fixed` enforces a concrete length separately.)
if bytes.len() % 2 != 0 {
return None;
}
let mut out = Vec::with_capacity(bytes.len() / 2);
for pair in bytes.chunks_exact(2) {
out.push(byte(pair[0], pair[1])?);
}
Some(out)
}
/// Parse a hex string into a fixed `[u8; N]`. Accepts an optional `0x`/`0X`
/// prefix; requires EXACTLY `2*N` ASCII hex digits after it. `None` on any
/// non-hex byte or a length mismatch.
pub fn parse_hex_fixed<const N: usize>(s: &str) -> Option<[u8; N]> {
let body = strip_prefix(s.trim());
let bytes = body.as_bytes();
if bytes.len() != 2 * N {
return None;
}
let mut out = [0u8; N];
for (i, slot) in out.iter_mut().enumerate() {
*slot = byte(bytes[2 * i], bytes[2 * i + 1])?;
}
Some(out)
}
/// Strip a single leading `0x` / `0X` if present (case-insensitive).
fn strip_prefix(s: &str) -> &str {
s.strip_prefix("0x")
.or_else(|| s.strip_prefix("0X"))
.unwrap_or(s)
}
/// Combine two ASCII hex-digit bytes into one byte. `as char` is intentional:
/// for a non-ASCII byte it produces a Latin-1 scalar that `to_digit(16)` then
/// rejects — so non-hex (incl. `+`/`-` sign chars) and multi-byte input fail
/// cleanly rather than slipping through `from_str_radix`'s sign handling.
fn byte(hi: u8, lo: u8) -> Option<u8> {
let hi = (hi as char).to_digit(16)?;
let lo = (lo as char).to_digit(16)?;
Some((hi * 16 + lo) as u8)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn fixed_accepts_0x_0x_and_bare_same_result() {
let want = [0x00, 0x11, 0xab, 0xCD, 0xef, 0x42, 0x99, 0x00];
let bare = "0011abcdef429900";
assert_eq!(parse_hex_fixed::<8>(bare), Some(want));
assert_eq!(parse_hex_fixed::<8>(&format!("0x{bare}")), Some(want));
// The case that used to be dropped by one parser but not another.
assert_eq!(parse_hex_fixed::<8>(&format!("0X{bare}")), Some(want));
assert_eq!(parse_hex_fixed::<8>(&format!(" 0X{bare} ")), Some(want));
}
#[test]
fn fixed_rejects_wrong_length_and_non_hex_and_signs() {
assert_eq!(parse_hex_fixed::<16>("00"), None); // too short
assert_eq!(parse_hex_fixed::<2>("00112233"), None); // too long
assert_eq!(parse_hex_fixed::<2>("zz11"), None); // non-hex
assert_eq!(parse_hex_fixed::<2>("+5-A"), None); // sign chars
}
#[test]
fn does_not_panic_on_multibyte_of_exact_byte_length() {
// "中" is 3 bytes; + 29 'a' = 32 bytes → would mis-slice a &str-indexed
// parser. Must reject, not panic.
let s = "".to_string() + &"a".repeat(29);
assert_eq!(s.len(), 32);
assert_eq!(parse_hex_fixed::<16>(&s), None);
}
#[test]
fn bytes_variable_length_and_odd_rejected() {
assert_eq!(parse_hex_bytes("0xAABBCC"), Some(vec![0xAA, 0xBB, 0xCC]));
assert_eq!(parse_hex_bytes("AABBC"), None); // odd
// Empty (or prefix-only) → empty Vec: a legitimately-empty field.
assert_eq!(parse_hex_bytes(""), Some(vec![]));
assert_eq!(parse_hex_bytes("0x"), Some(vec![]));
}
}
+247 -11
View File
@@ -8,7 +8,7 @@
//! MMC-6 §5.3.10 — Feature 010Ch (Firmware Information) //! MMC-6 §5.3.10 — Feature 010Ch (Firmware Information)
use crate::error::Result; use crate::error::Result;
use crate::scsi::{ScsiTransport, DataDirection}; use crate::scsi::{DataDirection, ScsiTransport};
/// Drive identity from standard SCSI commands. /// Drive identity from standard SCSI commands.
/// ///
@@ -39,8 +39,14 @@ pub struct DriveId {
/// Format: CCYYMMDDHHMI (12 ASCII characters) /// Format: CCYYMMDDHHMI (12 ASCII characters)
pub firmware_date: String, pub firmware_date: String,
/// Drive serial number — GET CONFIGURATION Feature 0108h
pub serial_number: String,
/// Raw 96-byte INQUIRY response for additional parsing if needed. /// Raw 96-byte INQUIRY response for additional parsing if needed.
pub raw_inquiry: Vec<u8>, pub raw_inquiry: Vec<u8>,
/// Raw GET CONFIGURATION Feature 010Ch response bytes.
pub raw_gc_010c: Vec<u8>,
} }
impl DriveId { impl DriveId {
@@ -51,22 +57,71 @@ impl DriveId {
let cdb_inq = [0x12, 0x00, 0x00, 0x00, 0x60, 0x00]; let cdb_inq = [0x12, 0x00, 0x00, 0x00, 0x60, 0x00];
transport.execute(&cdb_inq, DataDirection::FromDevice, &mut inquiry, 5000)?; transport.execute(&cdb_inq, DataDirection::FromDevice, &mut inquiry, 5000)?;
// GET CONFIGURATION Feature 010Ch — MMC-6 §6.6 // GET CONFIGURATION Feature 010Ch — MMC-6 §6.6.
// Best-effort: 010Ch (Firmware Information) is an optional feature.
// A drive that lacks it may CHECK CONDITION rather than return an
// empty descriptor, so a failure here is treated as feature-absent
// (empty firmware date + empty raw bytes) instead of aborting the
// whole identity probe.
let mut gc = vec![0u8; 256]; let mut gc = vec![0u8; 256];
let cdb_gc = [0x46, 0x02, 0x01, 0x0C, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00]; let cdb_gc = [0x46, 0x02, 0x01, 0x0C, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00];
let result = transport.execute(&cdb_gc, DataDirection::FromDevice, &mut gc, 5000)?; // `bytes_transferred` is device-reported and untrusted; clamp every
// slice end to the actual buffer length before indexing.
let (firmware_date, raw_gc_010c) =
match transport.execute(&cdb_gc, DataDirection::FromDevice, &mut gc, 5000) {
Ok(result) => {
let end = result.bytes_transferred.min(gc.len());
let date = if end > 12 {
String::from_utf8_lossy(&gc[12..24.min(end)])
.trim()
.to_string()
} else {
String::new()
};
(date, gc[..end].to_vec())
}
Err(_) => (String::new(), Vec::new()),
};
let firmware_date = if result.bytes_transferred > 12 { // GET CONFIGURATION Feature 0108h — Serial Number.
String::from_utf8_lossy(&gc[12..24.min(result.bytes_transferred)]) // Best-effort, like 010Ch above: the serial-number feature is
.trim().to_string() // optional, so a drive that lacks it (CHECK CONDITION) or reports
// too few bytes deliberately yields an empty serial rather than
// failing the identity probe.
let mut gc_serial = vec![0u8; 256];
let cdb_serial = [0x46, 0x02, 0x01, 0x08, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00];
let serial_number = if let Ok(r) =
transport.execute(&cdb_serial, DataDirection::FromDevice, &mut gc_serial, 5000)
{
if r.bytes_transferred > 12 {
// `bytes_transferred` is device-reported and untrusted; clamp
// the slice end to the buffer length to avoid an out-of-range
// panic on an oversized reported count.
let end = r.bytes_transferred.min(gc_serial.len());
String::from_utf8_lossy(&gc_serial[12..end])
.trim()
.to_string()
} else {
String::new()
}
} else { } else {
String::new() String::new()
}; };
Ok(Self::from_inquiry(&inquiry, &firmware_date)) Ok(DriveId {
vendor_id: ascii_field(&inquiry, 8, 16),
product_id: ascii_field(&inquiry, 16, 32),
product_revision: ascii_field(&inquiry, 32, 36),
vendor_specific: ascii_field(&inquiry, 36, 43),
firmware_date,
serial_number,
raw_inquiry: inquiry,
raw_gc_010c,
})
} }
/// Build identity from raw INQUIRY bytes and firmware date string. /// Build identity from raw INQUIRY bytes and firmware date string.
/// Used by tests and when serial isn't available.
pub fn from_inquiry(inquiry: &[u8], firmware_date: &str) -> Self { pub fn from_inquiry(inquiry: &[u8], firmware_date: &str) -> Self {
DriveId { DriveId {
vendor_id: ascii_field(inquiry, 8, 16), vendor_id: ascii_field(inquiry, 8, 16),
@@ -74,7 +129,9 @@ impl DriveId {
product_revision: ascii_field(inquiry, 32, 36), product_revision: ascii_field(inquiry, 32, 36),
vendor_specific: ascii_field(inquiry, 36, 43), vendor_specific: ascii_field(inquiry, 36, 43),
firmware_date: firmware_date.to_string(), firmware_date: firmware_date.to_string(),
serial_number: String::new(),
raw_inquiry: inquiry.to_vec(), raw_inquiry: inquiry.to_vec(),
raw_gc_010c: Vec::new(),
} }
} }
@@ -83,21 +140,26 @@ impl DriveId {
/// Used to look up this drive in the profile database. /// Used to look up this drive in the profile database.
/// All fields trimmed for consistent matching. /// All fields trimmed for consistent matching.
pub fn match_key(&self) -> String { pub fn match_key(&self) -> String {
format!("{}|{}|{}|{}", format!(
"{}|{}|{}|{}",
self.vendor_id.trim(), self.vendor_id.trim(),
self.product_id.trim(), self.product_id.trim(),
self.product_revision.trim(), self.product_revision.trim(),
self.vendor_specific.trim()) self.vendor_specific.trim()
)
} }
} }
impl std::fmt::Display for DriveId { impl std::fmt::Display for DriveId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{} {} {} {}", write!(
f,
"{} {} {} {}",
self.vendor_id.trim(), self.vendor_id.trim(),
self.product_id.trim(), self.product_id.trim(),
self.product_revision.trim(), self.product_revision.trim(),
self.vendor_specific.trim()) self.vendor_specific.trim()
)
} }
} }
@@ -114,6 +176,49 @@ fn ascii_field(data: &[u8], start: usize, end: usize) -> String {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::scsi::{ScsiResult, ScsiTransport};
/// Transport that returns the requested data length but reports a
/// bytes_transferred larger than the caller's buffer — models a drive
/// that lies about its transfer count. The old slicing code panicked
/// on this; the clamps must keep it from indexing out of range.
struct OversizedCountTransport;
impl ScsiTransport for OversizedCountTransport {
fn execute(
&mut self,
cdb: &[u8],
_dir: DataDirection,
buf: &mut [u8],
_timeout_ms: u32,
) -> Result<ScsiResult> {
// Fill plausible ASCII so the from_utf8_lossy paths run.
for b in buf.iter_mut() {
*b = b'A';
}
// INQUIRY (0x12): honest count. GET CONFIGURATION (0x46): lie.
let bytes_transferred = if cdb.first() == Some(&0x12) {
buf.len()
} else {
buf.len() + 4096
};
Ok(ScsiResult {
status: 0,
bytes_transferred,
sense: [0u8; 32],
})
}
}
#[test]
fn from_drive_clamps_oversized_bytes_transferred() {
// Must not panic despite the transport reporting a transfer count
// far beyond the 256-byte GET CONFIGURATION buffers.
let mut t = OversizedCountTransport;
let id = DriveId::from_drive(&mut t).expect("from_drive must not error");
// raw_gc_010c is clamped to the 256-byte buffer, never the lie.
assert_eq!(id.raw_gc_010c.len(), 256);
}
#[test] #[test]
fn test_bu40n_identity() { fn test_bu40n_identity() {
@@ -149,4 +254,135 @@ mod tests {
assert_eq!(id.vendor_specific.trim(), "16/04/"); assert_eq!(id.vendor_specific.trim(), "16/04/");
assert_eq!(id.firmware_date, "201604250000"); assert_eq!(id.firmware_date, "201604250000");
} }
// ── New comprehensive tests ────────────────────────────────────────────────
/// ascii_field with a buffer shorter than `start` returns empty string
/// rather than panicking.
/// Spec: SPC-4 §6.4.2 — bytes[8:16] are vendor ID; a truncated buffer
/// (e.g. a device that reports fewer than 8 bytes) must not panic.
/// Mutation: removing the `data.len() > start` guard makes it panic on short inputs.
#[test]
fn ascii_field_short_buffer_returns_empty() {
// Buffer of length 5: start=8 is beyond the end → empty string.
let buf = vec![0u8; 5];
let result = ascii_field(&buf, 8, 16); // SPC-4 vendor ID range
assert!(result.is_empty(), "short buffer must yield empty string");
}
/// ascii_field with a buffer that covers start but not end is clamped.
/// Spec: `ascii_field` documents "clamps to data.len()".
/// Mutation: using `end` directly without `min(data.len())` panics here.
#[test]
fn ascii_field_partial_buffer_is_clamped_not_panicked() {
// Buffer of length 12: vendor_id range is [8..16], but only [8..12] present.
let mut buf = vec![0u8; 12];
buf[8..12].copy_from_slice(b"SONY");
let result = ascii_field(&buf, 8, 16);
// Must not panic; the returned string holds what we wrote.
assert_eq!(result, "SONY");
}
/// from_inquiry extracts the product_id field from INQUIRY bytes [16:32].
/// Spec: SPC-4 §6.4.2 — PRODUCT IDENTIFICATION at offset 16, length 16.
/// Mutation: shifting the product_id slice to [8:24] makes this fail.
#[test]
fn from_inquiry_extracts_product_id_at_offset_16() {
let mut inquiry = vec![0u8; 96];
// Leave vendor_id (8..16) as zeros, write product_id at 16..32.
inquiry[16..32].copy_from_slice(b"BD-RW BDR-209M");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.product_id, "BD-RW BDR-209M",
"product_id must come from INQUIRY bytes 16..32 (SPC-4 §6.4.2)"
);
}
/// from_inquiry extracts product_revision from INQUIRY bytes [32:36].
/// Spec: SPC-4 §6.4.2 — PRODUCT REVISION LEVEL at offset 32, length 4.
/// Mutation: reading revision from [36:40] produces the wrong value.
#[test]
fn from_inquiry_extracts_revision_at_offset_32() {
let mut inquiry = vec![0u8; 96];
inquiry[32..36].copy_from_slice(b"1.53");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.product_revision, "1.53",
"product_revision must come from INQUIRY bytes 32..36 (SPC-4 §6.4.2)"
);
}
/// from_inquiry extracts vendor_specific from INQUIRY bytes [36:43].
/// Spec: SPC-4 §6.4.2 — VENDOR SPECIFIC at offset 36, length 8.
/// Mutation: reading vendor_specific from [32:39] returns the revision instead.
#[test]
fn from_inquiry_extracts_vendor_specific_at_offset_36() {
let mut inquiry = vec![0u8; 96];
inquiry[36..43].copy_from_slice(b"MM01234");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.vendor_specific, "MM01234",
"vendor_specific must come from INQUIRY bytes 36..43 (SPC-4 §6.4.2)"
);
}
/// from_inquiry stores the raw inquiry bytes in raw_inquiry unchanged.
/// Mutation: copying only a slice of inquiry into raw_inquiry truncates it.
#[test]
fn from_inquiry_stores_raw_inquiry() {
let mut inquiry = vec![0u8; 96];
inquiry[8..16].copy_from_slice(b"TESTDRVR");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.raw_inquiry, inquiry,
"raw_inquiry must preserve the full 96-byte buffer"
);
}
/// GET CONFIGURATION failure (transport error) must not abort the
/// identity probe — firmware_date is empty, raw_gc_010c is empty.
/// Mutation: propagating the GET_CONFIGURATION error with `?` aborts from_drive.
#[test]
fn from_drive_gc_failure_yields_empty_firmware_date() {
struct GcFailTransport;
impl ScsiTransport for GcFailTransport {
fn execute(
&mut self,
cdb: &[u8],
_dir: DataDirection,
buf: &mut [u8],
_timeout_ms: u32,
) -> Result<ScsiResult> {
if cdb.first() == Some(&0x12) {
// INQUIRY succeeds with a plausible response.
buf[8..16].copy_from_slice(b"TESTDRV ");
buf[16..32].copy_from_slice(b"FAKE DRIVE MODEL");
buf[32..36].copy_from_slice(b"0001");
buf[36..43].copy_from_slice(b"X000001");
Ok(ScsiResult {
status: 0,
bytes_transferred: buf.len(),
sense: [0u8; 32],
})
} else {
// GET CONFIGURATION fails.
Err(crate::error::Error::ScsiError {
opcode: cdb[0],
status: crate::scsi::SCSI_STATUS_CHECK_CONDITION,
sense: None,
})
}
}
}
let mut t = GcFailTransport;
let id = DriveId::from_drive(&mut t).expect("from_drive must succeed despite GC failure");
assert!(
id.firmware_date.is_empty(),
"firmware_date must be empty when GC fails"
);
assert!(
id.raw_gc_010c.is_empty(),
"raw_gc_010c must be empty when GC fails"
);
}
} }
+1782
View File
File diff suppressed because it is too large Load Diff
+295
View File
@@ -0,0 +1,295 @@
//! Bounded-syscall primitive: run a (potentially-blocking) operation
//! on a worker thread, with a hard wall-clock deadline and an optional
//! cooperative [`Halt`] poll. The calling thread is never trapped
//! inside a kernel call.
//!
//! ## Why this exists
//!
//! [`crate::halt::Halt`] is cooperative: callers poll
//! `is_cancelled()`. It cannot reach inside a syscall the kernel
//! currently owns the thread for — `libc::sync_file_range`,
//! `libc::fsync`, `File::write` on NFS, and so on. `/api/stop` from
//! autorip therefore can't unstick a thread sitting in such a syscall.
//!
//! [`bounded_syscall`] is the escape hatch: it runs `op` on a fresh
//! worker thread, then `recv_timeout`s on a rendezvous channel for the
//! result. The wait is broken into ~250 ms slices so the calling
//! thread can poll the supplied [`Halt`] in between. If the deadline
//! elapses or the halt fires, the worker is intentionally leaked — the
//! syscall will unwind whenever the kernel decides, or at process
//! exit, but the caller is free to fall back to a degraded code path
//! (skip the sync, log loudly, etc.).
//!
//! ## Trade-offs
//!
//! - **Thread per call.** Cheap (`std::thread::spawn` is < 100 µs on
//! Linux/macOS), but not free. Use on coarse-grained finalisation
//! syscalls (`sync_all`, `sync_file_range(WAIT_AFTER)`), not on hot
//! inner-loop writes.
//! - **Leak on timeout.** A wedged syscall keeps a kernel slot and a
//! user-space thread around for the rest of the process's life.
//! Bounded by the number of independent rip/mux sessions, which is
//! one per disc. The alternative — trapping the caller forever —
//! defeats the entire purpose of `/api/stop`.
//! - **Halt granularity ~250 ms.** Halt observation is not instant;
//! it's the worst-case latency of the `recv_timeout` slice. Good
//! enough for human-driven stop requests; not suitable for hard
//! real-time deadlines.
//!
//! ## Single source of truth
//!
//! Do NOT inline this pattern. Every blocking-syscall wrapper in the
//! rip + mux pipeline calls this helper, so changes (e.g. swapping the
//! channel impl, adjusting the poll slice, adding metrics) land in one
//! place.
//!
//! ## Platform
//!
//! Pure `std::thread` + `std::sync::mpsc`. No `cfg(target_os)` needed
//! here — the helper itself is platform-agnostic. Callers that wrap
//! Linux-only syscalls (`sync_file_range`) still need their own
//! `#[cfg(target_os = "linux")]` gates; this helper does not.
use std::sync::mpsc::{RecvTimeoutError, sync_channel};
use std::thread;
use std::time::{Duration, Instant};
use crate::halt::{Halt, POLL_INTERVAL};
/// Failure outcome from a bounded syscall wrapper.
#[derive(Debug)]
pub(crate) enum BoundedError {
/// The user-visible halt token fired during the wait. The worker
/// thread is intentionally leaked — the caller should fall back to
/// a degraded code path rather than waiting on the syscall to
/// return.
Halted,
/// The deadline elapsed before the syscall returned. Same leak
/// semantics as `Halted`.
Timeout,
/// The worker thread panicked, the OS rejected the thread spawn,
/// or its sender disconnected before sending a result. Treat as a
/// benign no-op (callers usually log and continue) rather than a
/// hard error — by definition no syscall observably ran to
/// completion in this case. In the spawn-failure case no thread is
/// leaked.
WorkerLost,
}
/// Run a (potentially-blocking) operation on a worker thread with a
/// deadline and an optional cooperative halt-token poll. Returns the
/// operation's result if it completes within `timeout`; otherwise one
/// of [`BoundedError::Halted`] / [`BoundedError::Timeout`] /
/// [`BoundedError::WorkerLost`].
///
/// On `Halted` / `Timeout` the worker thread is intentionally leaked:
/// the syscall will unwind whenever the kernel decides, or when the
/// process exits. The calling thread is never trapped inside a kernel
/// call.
///
/// `halt` is polled at [`POLL_INTERVAL`] granularity. Pass `None` for
/// callers that don't (yet) have a halt token plumbed through —
/// behaviour degrades to deadline-only, matching the 0.20.5
/// `wait_after_with_timeout` shape this helper generalises.
///
/// `op` returns `R: Send + 'static`. The closure must own everything
/// it touches because it may outlive this call (timeout / halt cases).
pub(crate) fn bounded_syscall<F, R>(
halt: Option<&Halt>,
timeout: Duration,
op: F,
) -> Result<R, BoundedError>
where
F: FnOnce() -> R + Send + 'static,
R: Send + 'static,
{
// If the caller already requested halt, don't spawn (and leak) a
// worker that would run `op` to completion in the background.
if halt.is_some_and(|h| h.is_cancelled()) {
return Err(BoundedError::Halted);
}
// Rendezvous channel: the worker sends exactly one value (the
// op's return) and then exits. Capacity-0 means the send blocks
// until we receive — fine on the happy path; on the timeout /
// halt path the receiver is dropped and the worker's send
// returns Err, which the worker ignores.
let (tx, rx) = sync_channel::<R>(0);
let _ = thread::Builder::new()
.name("freemkv-bounded-syscall".into())
.spawn(move || {
// Ignore the send error: if we time out (or get halted)
// before the worker finishes, the receiver is dropped
// and `tx.send` returns Err. Either way, the worker has
// nothing more to do.
let _ = tx.send(op());
});
let deadline = Instant::now() + timeout;
loop {
let now = Instant::now();
let remaining = deadline.saturating_duration_since(now);
let slice = remaining.min(POLL_INTERVAL);
match rx.recv_timeout(slice) {
Ok(v) => return Ok(v),
Err(RecvTimeoutError::Timeout) => {
if let Some(h) = halt {
if h.is_cancelled() {
return Err(BoundedError::Halted);
}
}
if Instant::now() >= deadline {
return Err(BoundedError::Timeout);
}
// Otherwise: another slice.
}
Err(RecvTimeoutError::Disconnected) => {
// Worker thread spawn failed, or it panicked before
// sending. Caller treats this as "no syscall ran" —
// typically a no-op + log.
return Err(BoundedError::WorkerLost);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
#[test]
fn op_completes_quickly() {
let r = bounded_syscall(None, Duration::from_secs(2), || 42u32);
assert!(matches!(r, Ok(42)));
}
#[test]
fn op_exceeds_timeout() {
// Op sleeps longer than the deadline → Timeout.
let r = bounded_syscall(None, Duration::from_millis(300), || {
thread::sleep(Duration::from_secs(2));
0u32
});
assert!(matches!(r, Err(BoundedError::Timeout)));
}
#[test]
fn halt_fires_during_wait() {
let halt = Halt::new();
let halt2 = halt.clone();
// Flip the halt from a side thread after ~300 ms — long
// enough that the receive loop has rolled at least one
// 250 ms slice and is sitting in `recv_timeout` again when
// the bit flips.
thread::spawn(move || {
thread::sleep(Duration::from_millis(300));
halt2.cancel();
});
let r = bounded_syscall(Some(&halt), Duration::from_secs(5), || {
thread::sleep(Duration::from_secs(5));
0u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
}
#[test]
fn worker_panics() {
// Worker panics → sender drops without sending → recv sees
// Disconnected → WorkerLost. We use an explicit panic in the
// op closure rather than `panic!()` from inside the channel
// machinery; the spawned thread's panic is contained (no
// process abort) because we don't `.join()` it.
let r = bounded_syscall(None, Duration::from_secs(2), || -> u32 {
panic!("intentional test panic");
});
assert!(matches!(r, Err(BoundedError::WorkerLost)));
}
#[test]
fn halt_already_set_before_call_still_returns_halted() {
// Halt observed on the very first poll slice. The op blocks
// forever; we must not wait the full timeout to notice the
// halt is already set.
let halt = Halt::new();
halt.cancel();
let started = Instant::now();
let r = bounded_syscall(Some(&halt), Duration::from_secs(10), || {
thread::sleep(Duration::from_secs(10));
0u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
// Should bail out within ~1 s; allow 2 s of slack for slow
// CI hosts.
assert!(
started.elapsed() < Duration::from_secs(2),
"halt-already-set took {:?}",
started.elapsed()
);
}
#[test]
fn ok_path_takes_no_halt_token() {
// Sanity: the `None` halt path is the documented zero-config
// form (matches the 0.20.5 `wait_after_with_timeout`
// behaviour). Op returns immediately; we must observe Ok.
let flag = Arc::new(AtomicBool::new(false));
let f2 = flag.clone();
let r = bounded_syscall(None, Duration::from_secs(2), move || {
f2.store(true, Ordering::Relaxed);
"ok"
});
assert!(matches!(r, Ok("ok")));
assert!(flag.load(Ordering::Relaxed));
}
// ── Added hardening tests ───────────────────────────────────────
/// Doc contract (lines 106-110): "If the caller already requested
/// halt, don't spawn (and leak) a worker that would run `op`."
/// When halt is pre-cancelled the op closure must NEVER run — the
/// short-circuit returns Halted before spawning the worker. We
/// prove the op did not execute by checking a side-effect flag.
#[test]
fn pre_cancelled_halt_never_runs_op() {
let halt = Halt::new();
halt.cancel();
let ran = Arc::new(AtomicBool::new(false));
let r2 = ran.clone();
let r = bounded_syscall(Some(&halt), Duration::from_secs(2), move || {
r2.store(true, Ordering::SeqCst);
7u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
// The op closure must not have been scheduled at all.
assert!(
!ran.load(Ordering::SeqCst),
"op ran despite pre-cancelled halt — short-circuit at line 108 broken"
);
}
/// Timeout boundary: with a tiny deadline and an op that sleeps
/// much longer, the helper must return Timeout and must do so
/// roughly at the deadline — NOT wait for the op to finish (that
/// is the whole point of the bounded wrapper; the worker is
/// leaked). Grounds the `Instant::now() >= deadline` arm (line 141)
/// and the leak contract (doc lines 84-88).
#[test]
fn timeout_returns_near_deadline_not_after_op() {
let started = Instant::now();
let r = bounded_syscall(None, Duration::from_millis(100), || {
thread::sleep(Duration::from_secs(3));
0u32
});
let elapsed = started.elapsed();
assert!(matches!(r, Err(BoundedError::Timeout)));
// Must bail near the 100ms deadline (one POLL_INTERVAL slack at
// most), not after the 3s op. Allow generous CI slack but stay
// well under the op's 3s sleep.
assert!(
elapsed < Duration::from_millis(1500),
"timeout did not return near deadline: {elapsed:?} (op should be leaked, not awaited)"
);
}
}
+559
View File
@@ -0,0 +1,559 @@
//! `BytePrefetcher` — `std::io::Read` analogue of
//! [`crate::sector::PrefetchedSectorSource`].
//!
//! Spawns a producer thread that fills a bounded pool of `Vec<u8>`
//! chunks from the underlying reader and ships them through a
//! channel; the consumer pulls filled chunks, uses them, and sends
//! the empty `Vec<u8>` back through a recycle channel so the
//! producer can re-fill in place. Result: zero allocations and zero
//! cross-thread frees in the steady-state hot loop.
//!
//! This is the byte-stream half of the freemkv mux highway —
//! `BytePrefetcher` feeds [`crate::mux::demux_thread::DemuxThread`]
//! for `m2ts://` (the only in-tree caller today, via
//! [`crate::mux::resolve`]), and works for any stream whose source is
//! an `io::Read` rather than a `SectorSource`.
use crate::halt::{Halt, POLL_INTERVAL};
use crossbeam_channel::{Receiver, RecvTimeoutError, SendTimeoutError, Sender, bounded};
use std::io::Read;
use std::thread::JoinHandle;
/// Items flowing through the forward channel.
pub type Batch = std::io::Result<Vec<u8>>;
/// Forward channel depth — how many filled buffers the producer can
/// stay ahead by. Two is enough to absorb a moderate consumer stall
/// without piling up bytes.
const FORWARD_DEPTH: usize = 2;
/// Recycle channel depth = forward + 1 so the producer always has at
/// least one buffer to fill while the consumer holds one.
const RECYCLE_DEPTH: usize = FORWARD_DEPTH + 1;
/// Default chunk size — 16 MiB matches the ISO-mux sector batch and
/// is large enough that per-chunk overhead is amortised; small
/// enough that the in-flight memory footprint stays bounded.
pub const DEFAULT_CHUNK_BYTES: usize = 16 * 1024 * 1024;
/// Returned from [`BytePrefetcher::into_channels`]. Owns the
/// producer-thread join handle so dropping the shell joins the
/// producer.
///
/// Drop blocks the calling thread until the producer exits. To
/// guarantee a prompt exit, drop the forward receiver and the recycle
/// sender first so the producer observes channel disconnection (or
/// cancel the [`Halt`] passed to [`BytePrefetcher::new`], which the
/// producer polls at [`POLL_INTERVAL`] granularity even while parked
/// on a channel op).
pub struct PrefetchShell {
producer: Option<JoinHandle<()>>,
}
impl Drop for PrefetchShell {
fn drop(&mut self) {
if let Some(h) = self.producer.take() {
let _ = h.join();
}
}
}
/// Spawned byte prefetcher. Drop joins the producer thread.
pub struct BytePrefetcher {
rx: Option<Receiver<Batch>>,
recycle_tx: Option<Sender<Vec<u8>>>,
producer: Option<JoinHandle<()>>,
}
impl BytePrefetcher {
/// Spawn the producer thread. `reader` must be `Send` because it
/// moves into the thread. `chunk_bytes` is the size of each
/// recycled buffer; pick the natural batch size of the
/// downstream demuxer (16 MiB for the BD-TS mux pipeline).
pub fn new<R: Read + Send + 'static>(
mut reader: R,
chunk_bytes: usize,
halt: Option<Halt>,
) -> std::io::Result<Self> {
// A zero-length chunk makes every recycled buffer an empty
// slice; `reader.read(&mut [])` returns Ok(0), which the loop
// below treats as EOF — the consumer would see a clean,
// silent zero-byte stream. Callers pass the downstream
// demuxer's batch size, which is always > 0.
debug_assert!(chunk_bytes > 0, "BytePrefetcher chunk_bytes must be > 0");
let (tx, rx) = bounded::<Batch>(FORWARD_DEPTH);
let (recycle_tx, recycle_rx) = bounded::<Vec<u8>>(RECYCLE_DEPTH);
// Seed the recycle pool. Without these the first
// `recycle_rx.recv()` would block forever (no consumer has
// returned a buffer yet).
for _ in 0..RECYCLE_DEPTH {
let _ = recycle_tx.send(vec![0u8; chunk_bytes]);
}
let producer = std::thread::Builder::new()
.name("freemkv-byte-prefetch".into())
.spawn(move || {
// Wrap the feed loop in catch_unwind so a panic in the inner
// `reader.read` (e.g. a decrypt-on-read slice/arith bug) is NOT
// indistinguishable from a clean finish at the demux boundary. A
// clean exit (EOF, halt, consumer disconnect) returns and drops
// `tx` → the demux loop reads RecvError as EOF (correct). A PANIC
// sends an explicit error sentinel first so the demux loop's
// `Ok(Err(_))` arm fires and propagates a typed error instead of
// converting the dropped channel into a clean `DemuxBatch::Eof`
// that would finalize a TRUNCATED mux while reporting success.
let body = std::panic::AssertUnwindSafe(|| {
let cancelled = || halt.as_ref().map(|h| h.is_cancelled()).unwrap_or(false);
// Liveness heartbeat: the producer blocks on the recycle and
// forward channels; a stalled consumer or a wedged reader shows
// up as the beat going silent. Total is unknown, so `pos` is
// cumulative bytes read.
let mut hb = crate::progress::Heartbeat::new("byte_prefetch");
let mut produced_bytes: u64 = 0;
loop {
hb.tick(produced_bytes, 0);
if cancelled() {
return;
}
// Park on the recycle channel, but re-poll halt
// every POLL_INTERVAL: a pure-AtomicBool Halt does
// not disconnect the channel, so a blocking recv()
// would never re-reach the cancel check.
let mut buf = loop {
match recycle_rx.recv_timeout(POLL_INTERVAL) {
Ok(b) => break b,
Err(RecvTimeoutError::Timeout) => {
if cancelled() {
return;
}
}
// Consumer dropped both channels.
Err(RecvTimeoutError::Disconnected) => return,
}
};
// Re-expose the full extent. After a short read the
// prior iteration truncated to n < chunk_bytes, so
// this regrows the length back to chunk_bytes
// without reallocating (capacity was fixed at
// construction and never shrinks).
if buf.len() < chunk_bytes {
buf.resize(chunk_bytes, 0);
} else {
// SAFETY: capacity is at least chunk_bytes
// after construction.
unsafe { buf.set_len(chunk_bytes) };
}
// Read up to one full chunk. Short reads are
// valid and common — pipe `truncate` so the
// consumer sees only the bytes that arrived.
let n = match reader.read(&mut buf[..]) {
Ok(0) => return, // EOF — drop tx, consumer sees RecvError
Ok(n) => n,
Err(e) => {
let _ = tx.send(Err(e));
return;
}
};
produced_bytes += n as u64;
buf.truncate(n);
// Hand off the filled buffer, re-polling halt on
// each timeout slice so a cancel can interrupt a
// producer parked on a saturated forward channel.
let mut pending = Ok(buf);
loop {
match tx.send_timeout(pending, POLL_INTERVAL) {
Ok(()) => break,
Err(SendTimeoutError::Timeout(returned)) => {
if cancelled() {
return;
}
pending = returned;
}
// Consumer dropped.
Err(SendTimeoutError::Disconnected(_)) => return,
}
}
}
});
if std::panic::catch_unwind(body).is_err() {
// Producer panicked mid-stream — surface a typed terminal
// error so the demux thread does NOT read the dropped channel
// as a clean EOF and truncate output.
let _ = tx.send(Err(crate::error::Error::DemuxThreadPanicked.into()));
}
})?;
Ok(Self {
rx: Some(rx),
recycle_tx: Some(recycle_tx),
producer: Some(producer),
})
}
/// Peel off the channels for zero-copy pipeline consumption. The
/// caller (typically [`crate::mux::demux_thread::DemuxThread`])
/// drains `rx`, runs the demuxer in place on each filled buffer,
/// and recycles back through `recycle_tx`.
pub fn into_channels(self) -> (Receiver<Batch>, Sender<Vec<u8>>, PrefetchShell) {
// MOVE the three fields out cleanly — never clone. Each of
// `rx` and `recycle_tx` ends up with exactly ONE live copy:
// the one in the returned tuple. The pre-1.0.0 implementation
// cloned both and then `mem::forget`-ed `self`, leaking the
// originals so an extra live receiver + sender survived
// forever. That defeated the channel-disconnection shutdown:
// when the demux consumer exited early (halt, or a `tx.send`
// error in `demux_thread`), the producer's `recycle_rx.recv()`
// and `tx.send()` never saw all-peers-dropped, so the producer
// never returned and `PrefetchShell::drop`'s `join()` hung.
//
// `ManuallyDrop` + `ptr::read` reads each field out by value
// and suppresses `self`'s own `Drop` (which would otherwise
// double-`join`), leaving NO extra live endpoint behind. This
// is the panic-free equivalent of the `Option::take` approach
// and mirrors `sector::prefetched::into_channels`.
let me = std::mem::ManuallyDrop::new(self);
// SAFETY: `me` is `ManuallyDrop`, so none of these fields will
// be dropped by `me`. Each `ptr::read` performs exactly one
// bitwise move out; every field is read exactly once and never
// touched again, so there are no double-frees and no aliasing.
let producer = unsafe { std::ptr::read(&me.producer) };
// SAFETY: `rx` and `recycle_tx` are always `Some` here —
// `into_channels` is the only way to consume a live
// `BytePrefetcher`; `Drop::drop` is suppressed by `ManuallyDrop`.
let rx = unsafe { std::ptr::read(&me.rx) }.expect("rx always Some before drop");
let recycle =
unsafe { std::ptr::read(&me.recycle_tx) }.expect("recycle_tx always Some before drop");
(rx, recycle, PrefetchShell { producer })
}
}
impl Drop for BytePrefetcher {
fn drop(&mut self) {
// Drop channel endpoints BEFORE joining the producer so the
// producer observes SendTimeoutError::Disconnected (forward tx)
// or RecvTimeoutError::Disconnected (recycle rx) and exits
// promptly. Without this, a non-EOF source fills the depth-2
// forward channel and then spins in send_timeout(POLL_INTERVAL)
// forever because rx is never drained, causing join() to
// deadlock.
drop(self.rx.take());
drop(self.recycle_tx.take());
if let Some(h) = self.producer.take() {
let _ = h.join();
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Endless reader: every `read` fills the whole buffer and never
/// hits EOF, so the producer keeps trying to push batches forward
/// until the forward channel disconnects. Exactly the shape that
/// wedged the pre-1.0.0 `clone + mem::forget` `into_channels`.
struct EndlessReader;
impl Read for EndlessReader {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
buf.fill(0);
Ok(buf.len())
}
}
/// Run `f` on a helper thread and fail if it does not finish within
/// `secs`. Turns a join-deadlock into a test failure instead of a
/// hung CI run.
fn within<F: FnOnce() + Send + 'static>(secs: u64, f: F) {
let (done_tx, done_rx) = bounded::<()>(1);
std::thread::spawn(move || {
f();
let _ = done_tx.send(());
});
assert!(
done_rx
.recv_timeout(std::time::Duration::from_secs(secs))
.is_ok(),
"operation did not complete within {secs}s (deadlock)"
);
}
/// The CRITICAL regression: after `into_channels`, dropping the
/// returned forward receiver + recycle sender must let the producer
/// observe disconnection and exit, so dropping the `PrefetchShell`
/// (which joins the producer) returns promptly. With the old
/// clone+forget the leaked endpoints kept the producer blocked and
/// this join hung forever.
#[test]
fn into_channels_drop_releases_producer() {
within(10, || {
// Small chunk so the producer cycles quickly and fills the
// forward channel without allocating much.
let pf = BytePrefetcher::new(EndlessReader, 4096, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
// Consumer goes away early (halt / abort analogue): drop
// both channel endpoints without draining to EOF.
drop(rx);
drop(recycle_tx);
// Joining the producer must not hang.
drop(shell);
});
}
/// Same property via the halt path: cancel the token, then the
/// producer must exit and the shell join must complete.
#[test]
fn halt_releases_producer() {
within(10, || {
let halt = Halt::new();
let pf = BytePrefetcher::new(EndlessReader, 4096, Some(halt.clone())).expect("spawn");
let (_rx, _recycle_tx, shell) = pf.into_channels();
halt.cancel();
drop(shell);
});
}
// ── Added hardening tests ───────────────────────────────────────
use std::io::Cursor;
/// Drain the forward channel, recycling every buffer, and
/// reassemble the bytes. Returns the concatenation of every
/// delivered chunk. Stops on RecvError (producer dropped tx == EOF)
/// or on the first Err batch (which it returns separately).
fn drain_to_vec(pf: BytePrefetcher) -> (Vec<u8>, Option<std::io::Error>) {
let (rx, recycle_tx, shell) = pf.into_channels();
let mut out = Vec::new();
let mut err = None;
while let Ok(batch) = rx.recv() {
match batch {
Ok(buf) => {
out.extend_from_slice(&buf);
// Recycle so the producer can refill. Ignore send
// error (producer may have already exited at EOF).
let _ = recycle_tx.send(buf);
}
Err(e) => {
err = Some(e);
break;
}
}
}
drop(rx);
drop(recycle_tx);
drop(shell);
(out, err)
}
/// CORE CONTRACT: the prefetcher must deliver every source byte,
/// in order, exactly once — never silently truncate or duplicate.
/// Source is 5000 bytes; chunk size 1024 forces multiple chunks
/// (4 full + 1 short of 904). The reassembled stream must equal the
/// source. Mutation: replacing `buf.truncate(n)` (line 141) with a
/// no-op would over-report bytes on the final short read and this
/// fails.
#[test]
fn delivers_all_bytes_in_order_across_chunks() {
within(10, || {
let src: Vec<u8> = (0..5000u32).map(|i| (i & 0xff) as u8).collect();
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 1024, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none(), "unexpected error batch: {err:?}");
assert_eq!(got, src, "prefetcher truncated or reordered bytes");
});
}
/// Short-read truncation: a reader that returns fewer bytes than
/// requested per call must NOT leave stale tail bytes in the
/// delivered chunk. Cursor over 10 bytes with a 4096 chunk yields a
/// single 10-byte chunk; the consumer must see exactly 10 bytes,
/// not 4096. Grounds `buf.truncate(n)` at line 141. Mutation:
/// delete the truncate and the chunk would carry 4086 zero bytes of
/// padding, failing the length assert.
#[test]
fn short_read_truncates_to_actual_length() {
within(10, || {
let src = vec![0xAB; 10];
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 4096, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none());
assert_eq!(got.len(), 10, "delivered chunk padded past actual read");
assert_eq!(got, src);
});
}
/// EOF semantics: an empty source (Cursor over `[]`) yields
/// `read() == Ok(0)` on the first call, which the producer treats
/// as EOF and returns, dropping tx. The consumer sees RecvError
/// (zero batches), NOT an Err batch and NOT a zero-length Ok batch.
/// Grounds the `Ok(0) => return` arm at line 134. Mutation:
/// changing `Ok(0) => return` to `Ok(0) => continue` would spin
/// forever (within() would time out).
#[test]
fn empty_source_yields_clean_eof_no_batches() {
within(10, || {
let pf = BytePrefetcher::new(Cursor::new(Vec::<u8>::new()), 4096, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
// No Ok batch should ever arrive; first recv must be Err
// (producer dropped tx at EOF).
let first = rx.recv();
assert!(
first.is_err(),
"empty source produced a batch instead of clean EOF: {first:?}"
);
drop(rx);
drop(recycle_tx);
drop(shell);
});
}
/// Error propagation: a reader that fails mid-stream must surface
/// the io::Error as an `Err` batch on the forward channel (line
/// 137), not swallow it. We deliver one good chunk then an error.
/// The consumer must see the good bytes followed by the error.
/// Mutation: changing `let _ = tx.send(Err(e)); return;` to a plain
/// `return` would drop the error silently and this fails.
#[test]
fn read_error_is_propagated_as_err_batch() {
within(10, || {
struct OneThenError {
served: bool,
}
impl Read for OneThenError {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if !self.served {
self.served = true;
let n = buf.len().min(8);
buf[..n].fill(0x11);
Ok(n)
} else {
Err(std::io::Error::other("synthetic mid-stream read failure"))
}
}
}
let pf = BytePrefetcher::new(OneThenError { served: false }, 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert_eq!(got, vec![0x11; 8], "good chunk lost");
let err = err.expect("read error must surface as an Err batch");
assert_eq!(err.kind(), std::io::ErrorKind::Other);
});
}
/// PANIC propagation: a reader that PANICS mid-stream must NOT be read as a
/// clean EOF at the demux boundary. The producer's catch_unwind sends an
/// explicit `Err` sentinel before the thread unwinds, so the consumer sees
/// the good bytes followed by an error batch — never a silent truncation.
/// Without the catch_unwind the panic would just drop `tx`, the consumer
/// would see RecvError (== clean EOF) and the partial output would be
/// finalized as if complete.
#[test]
fn read_panic_surfaces_as_err_batch_not_clean_eof() {
within(10, || {
struct OneThenPanic {
served: bool,
}
impl Read for OneThenPanic {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if !self.served {
self.served = true;
let n = buf.len().min(8);
buf[..n].fill(0x22);
Ok(n)
} else {
panic!("synthetic mid-stream reader panic");
}
}
}
let pf = BytePrefetcher::new(OneThenPanic { served: false }, 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert_eq!(got, vec![0x22; 8], "good chunk lost before the panic");
assert!(
err.is_some(),
"a mid-stream producer PANIC must surface as an Err batch, \
not a clean EOF (which would silently truncate the mux)"
);
});
}
/// Recycle-buffer reuse must NOT leak stale bytes between chunks of
/// different lengths. After a full chunk, a short read reuses the
/// same recycled buffer; lines 123-129 regrow it to chunk_bytes
/// before reading, then line 141 truncates to the short count. We
/// verify the short chunk carries only fresh bytes by reassembling
/// the full stream. Source: 8 bytes of 0xAA + 3 bytes of 0xBB, with
/// chunk_bytes=8 → chunk0 = 8×0xAA, chunk1 = 3×0xBB.
#[test]
fn recycled_buffer_carries_no_stale_tail() {
within(10, || {
let mut src = vec![0xAA; 8];
src.extend_from_slice(&[0xBB; 3]);
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none());
assert_eq!(
got, src,
"stale bytes from recycled buffer leaked into short chunk"
);
});
}
/// Exact-multiple boundary: when the source length is an exact
/// multiple of chunk_bytes, the final non-empty chunk is followed
/// by an `Ok(0)` EOF read, NOT a spurious empty Ok batch. 12 bytes
/// with chunk_bytes=4 → three 4-byte chunks then clean EOF. Total
/// bytes must equal 12 and no zero-length batch may appear.
#[test]
fn exact_multiple_length_no_trailing_empty_batch() {
within(10, || {
let src = vec![0x42u8; 12];
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 4, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
let mut total = 0usize;
let mut batch_count = 0usize;
while let Ok(Ok(buf)) = rx.recv() {
assert!(!buf.is_empty(), "producer emitted a zero-length batch");
total += buf.len();
batch_count += 1;
let _ = recycle_tx.send(buf);
}
assert_eq!(total, 12);
assert_eq!(batch_count, 3, "expected exactly 3 full chunks");
drop(rx);
drop(recycle_tx);
drop(shell);
});
}
/// Dropping the BytePrefetcher directly (without into_channels)
/// must join the producer cleanly when the source is finite. The
/// producer reaches EOF, drops tx, and exits; Drop's join returns.
/// Grounds the BytePrefetcher Drop impl (lines 202-208). Mutation:
/// removing the `Ok(0) => return` EOF exit would hang this join.
#[test]
fn drop_finite_prefetcher_joins_cleanly() {
within(10, || {
let pf = BytePrefetcher::new(Cursor::new(vec![1u8; 100]), 4096, None).expect("spawn");
// Drop without consuming — producer fills the forward
// channel (capacity 2), reaches EOF on the third read since
// 100 < 4096 (single chunk + EOF), drops tx, exits.
drop(pf);
});
}
/// Regression: dropping a BytePrefetcher directly (without
/// into_channels) with an ENDLESS source must not deadlock. Before
/// the fix, Drop joined the producer while rx/recycle_tx were still
/// alive (sibling field drop order), so the producer filled the
/// depth-2 forward channel and then spun in send_timeout forever
/// (rx never drained, halt=None). The fix drops rx+recycle_tx
/// BEFORE the join so the producer sees SendTimeoutError::Disconnected
/// and exits.
#[test]
fn drop_endless_prefetcher_joins_cleanly() {
within(10, || {
let pf = BytePrefetcher::new(EndlessReader, 4096, None).expect("spawn");
// Drop without consuming — the old Drop deadlocked here.
drop(pf);
});
}
}
+65
View File
@@ -0,0 +1,65 @@
//! Linux read-side platform hooks: sequential-access hint at open +
//! periodic page-cache eviction during streaming reads.
//!
//! ## Why both
//!
//! `POSIX_FADV_SEQUENTIAL` at open widens the kernel's readahead window
//! so each pread aggregates into fewer NFS round-trips. `DONTNEED` on
//! the consumed window (called periodically by the caller) drops the
//! already-read pages from the page cache so an 85 GB streaming ISO
//! read doesn't fill memory and starve concurrent writes (the MKV
//! output during mux). Together they mirror the write-side
//! WritebackPipeline's policy.
//!
//! ## History
//!
//! Pre-Phase-1 (0.20.7 baseline) had both. Phase 1's introduction of
//! `FileSectorSource` silently dropped the read-side DONTNEED, and
//! 0.21.2's revert of `SEQUENTIAL` (mistakenly attributing a regression
//! to it) removed the hint. Net effect: 85 GB of ISO reads pinned in
//! the page cache + no readahead widening → mux throughput collapse
//! from 18 MB/s historical to 2.7-8 MB/s on 0.21.x. Restored in 0.21.6.
use std::fs::File;
use std::os::unix::io::AsRawFd;
pub(super) fn hint_sequential(file: &File, _len_bytes: u64) {
// Best-effort: return value ignored. A fadvise failure has no
// user-observable consequence.
unsafe {
libc::posix_fadvise(file.as_raw_fd(), 0, 0, libc::POSIX_FADV_SEQUENTIAL);
}
}
/// Drop pages in the half-open byte range `[start, start+len)` from
/// the page cache. Called periodically by `read_sectors` to bound the
/// read-side page cache pressure.
pub(super) fn drop_window(file: &File, start: u64, len: u64) {
unsafe {
libc::posix_fadvise(
file.as_raw_fd(),
start as i64,
len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
}
/// Async-prefetch `len` bytes at `offset` into the page cache. The
/// kernel `readahead(2)` syscall queues the I/O and returns
/// immediately — it does NOT wait for completion. Called right after
/// each consumed read so the next batch's I/O overlaps with the
/// caller's processing of the current batch (decrypt + demux + mux).
///
/// Without this hint, with a synchronous demux consumer running at
/// ~50 MB/s and a single-spindle disk capable of ~150 MB/s, the disk
/// sits idle ~70% of each iteration because kernel readahead alone
/// (capped at `/sys/block/<dev>/queue/read_ahead_kb`, default 128 KB)
/// can only pre-stage a tiny slice of the next batch. An explicit
/// `readahead()` of the same size as the current batch tells the
/// kernel to queue the full next-batch read now.
pub(super) fn prefetch(file: &File, offset: u64, len: u64) {
unsafe {
libc::readahead(file.as_raw_fd(), offset as i64, len as usize);
}
}
+53
View File
@@ -0,0 +1,53 @@
//! macOS: hint the kernel to prefetch a generous chunk. macOS has no
//! direct `POSIX_FADV_SEQUENTIAL` equivalent; the idiomatic hint is
//! `fcntl(F_RDADVISE, &radvisory)` describing the byte range you
//! intend to read soon. We point it at the whole file (clamped to a
//! ceiling so a multi-TB ISO doesn't ask the kernel to prefetch
//! everything at once).
use std::fs::File;
use std::os::unix::io::AsRawFd;
/// Cap on the byte length we pass to `F_RDADVISE`. Asking for a
/// multi-GB readahead window is counterproductive — the OS doesn't
/// have that much cache to throw at one fd. 64 MiB is generous for
/// our use case (sweep, mux) so the kernel's prefetch ≥ our app-level
/// pipeline depth.
const RDADVISE_MAX_BYTES: i64 = 64 * 1024 * 1024;
pub(super) fn hint_sequential(file: &File, len_bytes: u64) {
let bytes = (len_bytes as i64).min(RDADVISE_MAX_BYTES);
let mut ra = libc::radvisory {
ra_offset: 0,
ra_count: bytes as libc::c_int,
};
// Best-effort.
unsafe {
libc::fcntl(file.as_raw_fd(), libc::F_RDADVISE, &mut ra);
}
}
/// macOS has no direct `POSIX_FADV_DONTNEED` equivalent for a byte
/// range. `fcntl(F_NOCACHE)` would disable caching globally on the fd
/// (too coarse — we want the unread region to still benefit). Best
/// approximation: no-op. macOS's unified buffer cache is generally
/// less prone to the pin-everything pathology that triggers the
/// regression on Linux NFS clients.
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
/// Async-prefetch the byte range `[offset, offset+len)`. macOS uses
/// the same `fcntl(F_RDADVISE, &radvisory)` primitive as the open-
/// time sequential hint, just targeted at a moving window instead of
/// the whole file. The kernel queues I/O for the requested range and
/// returns immediately.
pub(super) fn prefetch(file: &File, offset: u64, len: u64) {
let bytes = (len as i64).min(RDADVISE_MAX_BYTES);
let mut ra = libc::radvisory {
ra_offset: offset as libc::off_t,
ra_count: bytes as libc::c_int,
};
// Best-effort — kernel hint only.
unsafe {
libc::fcntl(file.as_raw_fd(), libc::F_RDADVISE, &mut ra);
}
}
+521
View File
@@ -0,0 +1,521 @@
//! [`FileSectorSource`] — read 2048-byte sectors from an ISO file on
//! disk via direct `seek + read_exact` (`pread`-equivalent) calls,
//! letting the kernel's own readahead policy manage prefetch.
//!
//! ## Why no app-level buffer
//!
//! Pre-0.21.3 this source held a 32 MiB (later 4 MiB) read-ahead
//! buffer to amortise per-sector NFS round-trips. Empirically that
//! buffer hurt: 32 MiB refills bursted the NFS TCP connection hard
//! enough to starve the concurrent writer, and even a 4 MiB window
//! gave the kernel less freedom to pipeline reads with writes. Direct
//! pread per call lets Linux's readahead widen as it detects the
//! sequential pattern, and naturally interleaves with writeback.
//!
//! ## DONTNEED on the consumed window
//!
//! Without page-cache eviction an 85 GB streaming ISO read pins the
//! entire file in memory, starves the concurrent writer, and collapses
//! mux throughput (observed: 2.7 MB/s mux on 0.21.5 vs. 70 MB/s
//! isolated NFS reads). Every [`READ_DROP_CHUNK_BYTES_DEFAULT`] of
//! consumed bytes we call `posix_fadvise(DONTNEED)` over that window,
//! mirroring the write-side [`crate::io::writeback::WritebackPipeline`]
//! policy.
//!
//! The drop window is accounted by a monotonic forward byte counter,
//! which matches the sequential streaming pattern the mux highway
//! drives. Under random or backward access the dropped range no longer
//! lines up with the bytes actually read — but `DONTNEED` is purely an
//! advisory cache hint with no correctness impact, so this degrades to
//! a slightly imprecise hint rather than a bug.
//!
//! ## Platform open hint
//!
//! On `open()` each platform issues its "sequential access expected"
//! hint so OS-level readahead widens. The hint and the DONTNEED call
//! live in per-OS sibling modules ([`linux::hint_sequential`] et al.)
//! — no inline `#[cfg]` in this file.
//!
//! ## Read-ahead prefetch
//!
//! After every consumed read we issue an OS-level prefetch hint for
//! the next equivalent-sized window (`platform::prefetch`). The
//! kernel queues that I/O asynchronously and returns immediately, so
//! the next batch's read overlaps with the caller's processing of
//! the current batch (decrypt + demux + mux). Without this the disk
//! sits idle ~70% of each iteration because kernel SEQUENTIAL
//! readahead alone (capped at `read_ahead_kb`, default 128 KB) is
//! far smaller than our 16 MiB app-level batch.
#[cfg(target_os = "linux")]
mod linux;
#[cfg(target_os = "macos")]
mod macos;
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
mod other;
#[cfg(target_os = "windows")]
mod windows;
#[cfg(target_os = "linux")]
use linux as platform;
#[cfg(target_os = "macos")]
use macos as platform;
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
use other as platform;
#[cfg(target_os = "windows")]
use windows as platform;
use std::fs::File;
use std::io::{Read, Seek, SeekFrom};
use std::path::Path;
use crate::error::{Error, Result};
use crate::sector::SectorSource;
use crate::consts::{SECTOR_BYTES, SECTOR_BYTES_U64};
/// Bytes-read threshold per `posix_fadvise(DONTNEED)` drop on the
/// read side. Mirrors `WRITEBACK_CHUNK_BYTES` so the read-side page
/// cache stays bounded the same way the write side does.
///
/// 32 MiB is the empirically tuned value on the rip1 test bed (single
/// 7200rpm HDD via SATA): smaller windows (8 / 16 MiB) shorten the
/// kernel-readahead overlap and slow the producer; larger windows
/// (64 / 128 MiB) let the page cache pin enough of the ISO to
/// pressure concurrent writes. Override via `FREEMKV_READ_DROP_CHUNK_MIB`.
const READ_DROP_CHUNK_BYTES_DEFAULT: u64 = 32 * 1024 * 1024;
fn read_drop_chunk_bytes() -> u64 {
std::env::var("FREEMKV_READ_DROP_CHUNK_MIB")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|&n| n > 0)
.map(|n| n * 1024 * 1024)
.unwrap_or(READ_DROP_CHUNK_BYTES_DEFAULT)
}
/// SectorSource backed by a file (ISO image). Every `read_sectors`
/// call is a direct `seek + read_exact` against the underlying file
/// — kernel readahead handles prefetch, and every
/// [`READ_DROP_CHUNK_BYTES_DEFAULT`] bytes of consumed data the
/// platform's `DONTNEED` hook drops the consumed window from the
/// page cache to bound memory pressure.
pub struct FileSectorSource {
file: File,
/// Total file size in sectors. Constant after construction;
/// surfaced via [`SectorSource::capacity_sectors`].
capacity: u32,
/// Bytes read since the last DONTNEED drop. Drives the per-
/// [`read_drop_chunk_bytes`] page-cache eviction in read_sectors.
bytes_read_since_drop: u64,
/// File offset at which the current drop window starts. The next
/// DONTNEED drops from `drop_window_start` for
/// `bytes_read_since_drop` bytes. This advances monotonically with
/// the byte count, so it tracks the actual reads only under the
/// forward-sequential access the mux highway uses; under random
/// access it degrades to a harmless, imprecise advisory hint.
drop_window_start: u64,
/// Cached drop chunk size (resolved from env once at open).
drop_chunk_bytes: u64,
}
impl FileSectorSource {
/// Open an existing ISO file for reading. Capacity is derived
/// from `metadata().len() / 2048`. Returns
/// [`Error::IsoTooLarge`] if the file would exceed the 32-bit
/// LBA address space (~8 TB).
///
/// Issues the platform's "sequential access expected" hint on the
/// fd (Linux `posix_fadvise(SEQUENTIAL)`, macOS `fcntl(F_RDADVISE)`,
/// Windows no-op) so the kernel's readahead widens.
pub fn open(path: &Path) -> Result<Self> {
let file = File::open(path).map_err(|e| Error::IoError { source: e })?;
let len = file
.metadata()
.map_err(|e| Error::IoError { source: e })?
.len();
let sectors = len / SECTOR_BYTES_U64;
if sectors > u32::MAX as u64 {
return Err(Error::IsoTooLarge {
path: path.to_string_lossy().into_owned(),
});
}
let capacity = sectors as u32;
// Best-effort sequential hint. Ignored on platforms without
// an equivalent primitive (or where the API exists but the
// FS doesn't honour it).
platform::hint_sequential(&file, len);
Ok(Self {
file,
capacity,
bytes_read_since_drop: 0,
drop_window_start: 0,
drop_chunk_bytes: read_drop_chunk_bytes(),
})
}
}
impl SectorSource for FileSectorSource {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
out: &mut [u8],
_recovery: bool,
) -> Result<usize> {
let count = count as u32;
let bytes = count as usize * SECTOR_BYTES;
debug_assert!(
out.len() >= bytes,
"FileSectorSource::read_sectors: out len {} < requested {}",
out.len(),
bytes
);
if count == 0 {
return Ok(0);
}
let offset = lba as u64 * SECTOR_BYTES_U64;
self.file
.seek(SeekFrom::Start(offset))
.map_err(|e| Error::IoError { source: e })?;
self.file
.read_exact(&mut out[..bytes])
.map_err(|e| Error::IoError { source: e })?;
// Queue the next batch's read with the kernel before the
// caller starts processing what we just returned. readahead()
// is non-blocking — it queues I/O and returns, so the kernel
// pulls those pages into cache while the consumer (decrypt +
// demux + mux) runs. Next read_sectors call hits a warm cache.
platform::prefetch(&self.file, offset + bytes as u64, bytes as u64);
// Periodic page-cache eviction on the read side. Without
// this, an 85 GB streaming ISO read pins the entire file in
// the kernel page cache, which starves concurrent writes and
// collapses mux throughput. Mirrors the write-side
// WritebackPipeline's DONTNEED policy.
self.bytes_read_since_drop += bytes as u64;
if self.bytes_read_since_drop >= self.drop_chunk_bytes {
let drop_start = self.drop_window_start;
let drop_len = self.bytes_read_since_drop;
platform::drop_window(&self.file, drop_start, drop_len);
self.drop_window_start = drop_start + drop_len;
self.bytes_read_since_drop = 0;
}
Ok(bytes)
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write;
use tempfile::tempdir;
/// Build a deterministic ISO of `sectors` sectors where sector `n`
/// is filled with the byte pattern `((n & 0xff) as u8)`. Lets us
/// verify any sector by content alone.
fn make_iso(path: &std::path::Path, sectors: u32) {
let mut f = std::fs::File::create(path).unwrap();
let mut chunk = vec![0u8; SECTOR_BYTES];
for n in 0..sectors {
let b = (n & 0xff) as u8;
chunk.iter_mut().for_each(|c| *c = b);
f.write_all(&chunk).unwrap();
}
f.flush().unwrap();
}
/// Sectors used by spanning-boundary tests. Pick something that
/// exercises multi-megabyte reads without making test ISOs huge.
/// 8192 sectors = 16 MiB — large enough to cross any readahead
/// chunk size we set the kernel hint to.
const TEST_SPAN_SECTORS: u32 = 8192;
#[test]
fn sequential_reads_match_file() {
let total = TEST_SPAN_SECTORS * 2 + 17;
let dir = tempdir().unwrap();
let path = dir.path().join("seq.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), total);
let mut got = vec![0u8; SECTOR_BYTES];
for lba in 0..total {
src.read_sectors(lba, 1, &mut got, false).unwrap();
let expected = (lba & 0xff) as u8;
assert!(
got.iter().all(|b| *b == expected),
"sector {lba} content mismatch: expected 0x{expected:02x}"
);
}
}
#[test]
fn multi_sector_read_across_chunk_boundary() {
let total = TEST_SPAN_SECTORS * 2;
let dir = tempdir().unwrap();
let path = dir.path().join("span.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
let span_lba = TEST_SPAN_SECTORS - 2;
let mut buf4 = vec![0u8; SECTOR_BYTES * 4];
src.read_sectors(span_lba, 4, &mut buf4, false).unwrap();
for i in 0..4 {
let lba = span_lba + i as u32;
let expected = (lba & 0xff) as u8;
for b in &buf4[i * SECTOR_BYTES..(i + 1) * SECTOR_BYTES] {
assert_eq!(*b, expected, "byte mismatch at sub-sector {i}");
}
}
}
#[test]
fn backward_seek_reads_correct_bytes() {
// Read forward then jump back: the SectorSource contract is
// byte-correctness regardless of access pattern.
let total = TEST_SPAN_SECTORS * 2 + 5;
let dir = tempdir().unwrap();
let path = dir.path().join("back.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
let mut got = vec![0u8; SECTOR_BYTES];
src.read_sectors(TEST_SPAN_SECTORS + 1, 1, &mut got, false)
.unwrap();
src.read_sectors(0, 1, &mut got, false).unwrap();
assert!(got.iter().all(|b| *b == 0));
}
#[test]
fn read_at_eof_returns_correct_bytes() {
// File smaller than the readahead chunk — reads near EOF must
// still return correct bytes.
let total: u32 = 100;
let dir = tempdir().unwrap();
let path = dir.path().join("small.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), total);
let mut got = vec![0u8; SECTOR_BYTES];
src.read_sectors(0, 1, &mut got, false).unwrap();
src.read_sectors(total - 1, 1, &mut got, false).unwrap();
let expected = ((total - 1) & 0xff) as u8;
assert!(got.iter().all(|b| *b == expected));
}
#[test]
fn large_single_read() {
// A multi-MB single read must work — the implementation has
// no app-level chunking, so this just exercises the direct
// pread path on a larger request.
let total = TEST_SPAN_SECTORS + 100;
let dir = tempdir().unwrap();
let path = dir.path().join("big.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
let req = (TEST_SPAN_SECTORS + 1) as u16;
let req_bytes = req as usize * SECTOR_BYTES;
let mut big = vec![0u8; req_bytes];
src.read_sectors(0, req, &mut big, false).unwrap();
assert!(big[..SECTOR_BYTES].iter().all(|b| *b == 0));
let last_lba = req as u32 - 1;
let exp = (last_lba & 0xff) as u8;
let last_off = (req as usize - 1) * SECTOR_BYTES;
assert!(
big[last_off..last_off + SECTOR_BYTES]
.iter()
.all(|b| *b == exp)
);
}
#[test]
fn drop_chunk_size_env_override() {
// Explicit 8 MiB via env var.
// SAFETY: tests in this crate are single-threaded per the
// default cargo test harness, but std::env::set_var is
// declared `unsafe` since Rust 2024 (it can race with other
// threads / TLS). For a test that runs in-process before any
// FileSectorSource construction this is safe in practice.
unsafe {
std::env::set_var("FREEMKV_READ_DROP_CHUNK_MIB", "8");
}
assert_eq!(read_drop_chunk_bytes(), 8 * 1024 * 1024);
unsafe {
std::env::remove_var("FREEMKV_READ_DROP_CHUNK_MIB");
}
assert_eq!(read_drop_chunk_bytes(), READ_DROP_CHUNK_BYTES_DEFAULT);
// Garbage env value falls back to default.
unsafe {
std::env::set_var("FREEMKV_READ_DROP_CHUNK_MIB", "not-a-number");
}
assert_eq!(read_drop_chunk_bytes(), READ_DROP_CHUNK_BYTES_DEFAULT);
unsafe {
std::env::remove_var("FREEMKV_READ_DROP_CHUNK_MIB");
}
}
// ---------------------------------------------------------------
// Additional coverage.
// ---------------------------------------------------------------
/// `count == 0` must short-circuit to Ok(0) WITHOUT seeking or
/// reading, even at an out-of-range LBA — the early-return guard
/// runs before any I/O. Grounding: `if count == 0 { return Ok(0) }`.
#[test]
fn zero_count_returns_zero_no_io() {
let dir = tempdir().unwrap();
let path = dir.path().join("zc.iso");
make_iso(&path, 4);
let mut src = FileSectorSource::open(&path).unwrap();
// LBA far past EOF — must not matter because count==0 returns early.
let mut buf = [0u8; 1];
let n = src.read_sectors(1_000_000, 0, &mut buf, false).unwrap();
assert_eq!(n, 0);
}
/// Reading past EOF must ERROR (read_exact's UnexpectedEof), never
/// return a partial/short count. This is the core "never silently
/// truncate / never return fewer bytes than declared" property of
/// the SectorSource contract. Grounding: `self.file.read_exact(...)`
/// — read_exact fails if the file can't supply the full span.
#[test]
fn read_past_eof_errors_not_truncates() {
let dir = tempdir().unwrap();
let path = dir.path().join("eof.iso");
make_iso(&path, 4); // 4 sectors only
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 4);
// Request 2 sectors starting at LBA 3 → sector 4 doesn't exist.
let mut buf = vec![0u8; 2 * SECTOR_BYTES];
let r = src.read_sectors(3, 2, &mut buf, false);
let err = r.expect_err("reading past EOF must error, not short-read");
let io: std::io::Error = err.into();
assert_eq!(
io.kind(),
std::io::ErrorKind::UnexpectedEof,
"partial read at EOF must surface read_exact's UnexpectedEof"
);
}
/// On a successful full read the returned count MUST equal
/// `count * 2048` exactly — the declared byte count. Grounding:
/// `Ok(bytes)` where `bytes = count * SECTOR_BYTES`.
#[test]
fn full_read_returns_exact_declared_bytes() {
let dir = tempdir().unwrap();
let path = dir.path().join("exact.iso");
make_iso(&path, 16);
let mut src = FileSectorSource::open(&path).unwrap();
let mut buf = vec![0u8; 5 * SECTOR_BYTES];
let n = src.read_sectors(2, 5, &mut buf, false).unwrap();
assert_eq!(n, 5 * SECTOR_BYTES, "must return exactly count*2048 bytes");
}
/// Capacity is `file_len / 2048` (floor); trailing bytes that don't
/// complete a sector are NOT counted. A file of 4 sectors + 100
/// extra bytes reports capacity 4. Grounding: `len / SECTOR_BYTES`
/// integer division in `open`.
#[test]
fn capacity_floors_partial_trailing_sector() {
let dir = tempdir().unwrap();
let path = dir.path().join("partial.iso");
make_iso(&path, 4);
// Append 100 stray bytes (a torn final sector).
{
let mut f = std::fs::OpenOptions::new()
.append(true)
.open(&path)
.unwrap();
f.write_all(&[0xee; 100]).unwrap();
f.flush().unwrap();
}
let src = FileSectorSource::open(&path).unwrap();
assert_eq!(
src.capacity_sectors(),
4,
"partial trailing bytes must not inflate the sector capacity"
);
}
/// An empty file opens cleanly with capacity 0. Grounding:
/// `0 / 2048 == 0`, and the IsoTooLarge guard only fires for
/// oversize files.
#[test]
fn empty_file_capacity_zero() {
let dir = tempdir().unwrap();
let path = dir.path().join("empty.iso");
std::fs::File::create(&path).unwrap();
let src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 0);
}
/// Opening a nonexistent path returns an IoError (NotFound), not a
/// panic. Grounding: `File::open(path).map_err(...)`.
#[test]
fn open_missing_file_errors() {
let dir = tempdir().unwrap();
let path = dir.path().join("does-not-exist.iso");
let err = match FileSectorSource::open(&path) {
Ok(_) => panic!("missing file must error"),
Err(e) => e,
};
let io: std::io::Error = err.into();
assert_eq!(io.kind(), std::io::ErrorKind::NotFound);
}
/// A DONTNEED drop crossing the chunk threshold must not corrupt or
/// short subsequent reads — the eviction is a pure page-cache hint.
/// We read past the DEFAULT 32 MiB drop chunk (16384 sectors) so the
/// eviction block fires at least once, asserting every sector still
/// reads correctly. (Avoids mutating FREEMKV_READ_DROP_CHUNK_MIB to
/// sidestep a parallel-test env race with `drop_chunk_size_env_override`.)
/// Grounding: the `bytes_read_since_drop >= drop_chunk_bytes`
/// eviction block calls only `platform::drop_window` (advisory) and
/// resets counters — no data effect.
#[test]
fn dontneed_eviction_does_not_affect_data() {
// 32 MiB default chunk = 16384 sectors; read a bit past it.
let total = (READ_DROP_CHUNK_BYTES_DEFAULT / SECTOR_BYTES_U64) as u32 + 64;
let dir = tempdir().unwrap();
let path = dir.path().join("drop.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
// Read in 16-sector batches to keep the loop fast while still
// crossing the drop boundary by byte count.
let batch = 16u16;
let mut got = vec![0u8; batch as usize * SECTOR_BYTES];
let mut lba = 0u32;
while lba + batch as u32 <= total {
src.read_sectors(lba, batch, &mut got, false).unwrap();
for i in 0..batch as u32 {
let expected = ((lba + i) & 0xff) as u8;
let off = i as usize * SECTOR_BYTES;
assert!(
got[off..off + SECTOR_BYTES].iter().all(|x| *x == expected),
"DONTNEED eviction corrupted sector {}",
lba + i
);
}
lba += batch as u32;
}
}
}
+11
View File
@@ -0,0 +1,11 @@
//! Fallback for targets without a known sequential-readahead hint
//! (BSDs, illumos, etc.). No-op — reads still work, they just don't
//! get the OS-level prefetch widening.
use std::fs::File;
pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {}
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
pub(super) fn prefetch(_file: &File, _offset: u64, _len: u64) {}
+28
View File
@@ -0,0 +1,28 @@
//! Windows: the canonical sequential-access hint is
//! `FILE_FLAG_SEQUENTIAL_SCAN`, which must be passed to `CreateFile`
//! at open time and cannot be set afterward via
//! `SetFileInformationByHandle`. Since `FileSectorSource::open` uses a
//! plain `File::open`, the hints in this module are no-op stubs.
use std::fs::File;
/// No-op stub. `FILE_FLAG_SEQUENTIAL_SCAN` can only be set at
/// `CreateFile` open time, which the plain `File::open` path does not
/// do, so there is no post-open hint to issue here.
pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {
tracing::debug!(
target: "mux",
"FileSectorSource hint_sequential: windows no-op stub"
);
}
/// Windows page-cache eviction is not exposed via a posix_fadvise
/// equivalent. The kernel does its own working-set management. No-op
/// for now.
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
/// Windows async-prefetch hint. With FILE_FLAG_SEQUENTIAL_SCAN at
/// open the kernel already prefetches aggressively, so there's no
/// per-range hint we'd add on top. No-op stub for parity with the
/// posix platforms.
pub(super) fn prefetch(_file: &File, _offset: u64, _len: u64) {}
+92
View File
@@ -0,0 +1,92 @@
//! Platform-aware crash-durability primitives.
//!
//! Two flush operations need OS-specific handling to make a write survive a
//! crash / power loss:
//!
//! - [`dir`] — fsync a directory so a prior `rename(2)` into it is durable.
//! After a crash a renamed file's dirent can otherwise be lost even though
//! the rename returned, because it is still page-cache-only. This is a POSIX
//! concept: on Windows std cannot even open a directory as a `File` (it does
//! not set `FILE_FLAG_BACKUP_SEMANTICS`), and NTFS/ReFS commit the rename's
//! dirent without an explicit directory flush — so it is a no-op there
//! rather than a failed open that logs on every marker write.
//!
//! - [`file_durable`] — fsync a file's contents + metadata. Opens the file
//! **read+write**: on Windows `File::sync_all` maps to `FlushFileBuffers`,
//! which requires a handle with write access and returns
//! `ERROR_ACCESS_DENIED` (os error 5) on a read-only handle. (A read-only
//! `File::open` + `sync_all` is legal on POSIX, which is why that bug only
//! bit Windows.) The open mode is platform-uniform, so this lives here with
//! no dispatch.
//!
//! Per the crate convention (see [`crate::io::writeback_file`]), platform
//! dispatch happens once here via cfg-gated `mod` decls — callers carry no
//! inline `#[cfg(...)]`.
use std::io;
use std::path::Path;
#[cfg(not(windows))]
mod posix;
#[cfg(windows)]
mod windows;
#[cfg(not(windows))]
use posix as platform;
#[cfg(windows)]
use windows as platform;
/// fsync a directory so a prior `rename(2)` into it is durable. Best-effort:
/// failures are logged and swallowed, never propagated — the renamed file's
/// bytes are already synced and the caller's write itself succeeded. No-op on
/// Windows (see module docs).
pub fn dir(path: &Path) {
platform::fsync_dir(path)
}
/// Durably flush an existing file's contents + metadata to stable storage.
///
/// Opens the file read+write (not read-only) so the flush succeeds on every
/// platform — see the module docs for the Windows `FlushFileBuffers` rationale.
/// The file must already exist; its bytes are left intact (no create/truncate).
pub fn file_durable(path: &Path) -> io::Result<()> {
let f = std::fs::OpenOptions::new()
.read(true)
.write(true)
.open(path)?;
f.sync_all()
}
#[cfg(test)]
mod tests {
use super::*;
/// `file_durable` opens read+write (so the flush works on Windows) and
/// syncs an existing file; a missing path surfaces as `Err` so the caller
/// treats it as "not durably synced". Platform-uniform — same on
/// unix/windows.
#[test]
fn file_durable_ok_for_existing_err_for_missing() {
let td = tempfile::tempdir().unwrap();
let f = td.path().join("data.bin");
std::fs::write(&f, b"durable").unwrap();
assert!(
file_durable(&f).is_ok(),
"an existing file must open read+write and fsync cleanly"
);
assert!(
file_durable(&td.path().join("absent.bin")).is_err(),
"a missing file must surface the open failure as Err"
);
}
/// `dir` is best-effort: it must return normally for a real directory
/// (POSIX fsyncs it, Windows no-ops) and must swallow — never panic on —
/// a missing directory.
#[test]
fn dir_is_best_effort_never_panics() {
let td = tempfile::tempdir().unwrap();
dir(td.path());
dir(&td.path().join("does-not-exist"));
}
}
+18
View File
@@ -0,0 +1,18 @@
//! POSIX directory-fsync. Active on unix and any non-Windows fallback target
//! (BSD, illumos, …) — all share the same `File::open(dir).sync_all()`
//! semantics. The Windows no-op lives in the sibling `windows` module.
use std::path::Path;
pub(super) fn fsync_dir(dir: &Path) {
match std::fs::File::open(dir) {
Ok(f) => {
if let Err(e) = f.sync_all() {
tracing::warn!(path = %dir.display(), error = %e, "failed to fsync directory");
}
}
Err(e) => {
tracing::warn!(path = %dir.display(), error = %e, "could not open directory to fsync");
}
}
}
+13
View File
@@ -0,0 +1,13 @@
//! Windows directory-fsync: a no-op.
//!
//! Directory fsync is a POSIX concept. std cannot open a directory as a `File`
//! on Windows (it does not set `FILE_FLAG_BACKUP_SEMANTICS`), so the POSIX impl
//! could only ever fail the open and log a spurious warning on every marker /
//! mapfile write. NTFS/ReFS commit a rename's directory entry without an
//! explicit directory flush, so skipping it here is correct — not a durability
//! regression. (File-content durability is handled platform-uniformly by
//! [`super::file_durable`].)
use std::path::Path;
pub(super) fn fsync_dir(_dir: &Path) {}
+48
View File
@@ -0,0 +1,48 @@
//! File I/O helpers that bound kernel cache pressure on big writes.
//!
//! `WritebackFile` is a drop-in wrapper around `std::fs::File` for any
//! call site that performs large sequential writes (sweep, patch, mux,
//! etc.). It implements `Write` and `Seek` so existing code paths can
//! swap `File` for `WritebackFile` with no body changes. Internally it
//! drives a `WritebackPipeline` that, on Linux, drains dirty pages
//! continuously at 32 MB granularity to avoid the kernel's
//! accumulate-then-burst flush behaviour. macOS and Windows use a
//! no-op pipeline — their default cache policies have not been shown
//! to exhibit the same pathology for this access pattern.
//!
//! `FileSectorSource` is the read-side dual — it implements
//! [`crate::sector::SectorSource`] for an ISO file using direct
//! `pread`-equivalent calls so the kernel's own readahead policy runs
//! (which interleaves naturally with the concurrent writeback). It
//! pairs that with periodic `posix_fadvise(DONTNEED)` drops on the
//! consumed window so an 85 GB streaming ISO read doesn't fill the
//! page cache and starve the concurrent MKV write.
//!
//! `Pipeline` + `Sink` is the generic producer/consumer primitive
//! used by sweep, patch, and mux to overlap reads with writes via a
//! bounded channel + dedicated consumer thread.
//!
//! `byte_prefetcher` is the read-ahead producer feeding the mux
//! pipeline for `io::Read`-backed sources: a worker thread fills a
//! recycled pool of buffers and ships them through a channel, exposing
//! `BytePrefetcher` / `PrefetchShell`.
pub(crate) mod bounded;
pub mod byte_prefetcher;
pub mod file_sector_source;
pub mod fsync;
pub mod sink;
mod writeback;
mod writeback_file;
#[cfg(target_os = "macos")]
pub(crate) mod platform_macos;
pub mod pipeline;
pub(crate) use writeback_file::WritebackFile;
pub use pipeline::{
DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, READ_PIPELINE_DEPTH, Sink, WRITE_PIPELINE_DEPTH,
WRITE_THROUGH_DEPTH,
};
+1595
View File
File diff suppressed because it is too large Load Diff
+38
View File
@@ -0,0 +1,38 @@
//! Shared macOS `fcntl(F_PREALLOCATE)` definitions.
//!
//! The `libc` crate doesn't expose these symbols across all macOS SDK
//! versions, so we define them locally with values from
//! `/usr/include/sys/fcntl.h`. Two call sites (
//! [`crate::io::writeback_file`] and [`crate::io::sink::preallocate`])
//! need the same constants and `fstore_t` layout — keeping a single
//! source of truth here prevents the two copies from drifting.
//!
//! Module-level cfg gate lives in the parent (`io/mod.rs`); this file
//! is only compiled on macOS, so no inner `#![cfg]` is needed.
/// `fcntl(F_PREALLOCATE)` command number from `sys/fcntl.h`.
pub(crate) const F_PREALLOCATE: libc::c_int = 42;
/// Anchor preallocation at the current physical EOF.
pub(crate) const F_PEOFPOSMODE: libc::c_int = 3;
/// Prefer a contiguous allocation. Try this first; on `EINVAL` (no
/// contiguous run of that size), fall back to `F_ALLOCATEALL`.
pub(crate) const F_ALLOCATECONTIG: libc::c_uint = 0x0000_0002;
/// Allow non-contiguous allocation. Stronger guarantee than just
/// asking for `F_ALLOCATECONTIG` because the kernel will piece
/// together fragments rather than failing.
pub(crate) const F_ALLOCATEALL: libc::c_uint = 0x0000_0004;
/// `fstore_t` from `sys/fcntl.h`. `repr(C)` because we hand it to
/// `fcntl(F_PREALLOCATE)` which writes through the pointer.
#[repr(C)]
#[derive(Clone, Copy)]
pub(crate) struct Fstore {
pub fst_flags: libc::c_uint,
pub fst_posmode: libc::c_int,
pub fst_offset: libc::off_t,
pub fst_length: libc::off_t,
pub fst_bytesalloc: libc::off_t,
}
+233
View File
@@ -0,0 +1,233 @@
//! `LocalFileSink` — `BufWriter<File>` for the common local-disk case.
//!
//! Buffering: 4 MiB internal `BufWriter`. Sized to coalesce the small
//! per-PES writes that come out of the muxer into kernel-page-aligned
//! flushes without making the buffer big enough to matter for memory
//! pressure on a single concurrent rip.
//!
//! `Seek` flushes the underlying `BufWriter` first; otherwise a seek
//! could leapfrog buffered data and silently corrupt the file. This is
//! the same shape `BufWriter` itself uses when it impls `Seek` in
//! stdlib, and is necessary for MKV's seek-back operations (cluster
//! size patch, Cues index, segment header backpatch) to land on the
//! right offset.
//!
//! [`SequentialSink`](super::SequentialSink) is implemented explicitly
//! (not via a blanket impl) so its `finish()` flushes the `BufWriter`
//! and `fsync`s the file even when called through a `dyn` trait object;
//! [`RandomAccessSink`](super::RandomAccessSink) is implemented over the
//! `Seek` impl below.
use std::fs::{File, OpenOptions};
use std::io::{self, BufWriter, Seek, SeekFrom, Write};
use std::path::Path;
use super::preallocate;
use super::{RandomAccessSink, SequentialSink};
const BUFFER_BYTES: usize = 4 * 1024 * 1024;
/// Random-access write sink for local disks.
///
/// Wraps a `BufWriter<File>` with a 4 MiB internal buffer and forwards
/// `Write`/`Seek` so any call site that previously held a `File` or
/// `WritebackFile` can drop this in. `finish()` flushes the buffer and
/// runs `sync_all` on the underlying file so the caller can drop it
/// without losing data.
///
/// Construction always opens the file `create + truncate + read +
/// write`. `read` is enabled so the same handle can be reused for a
/// verification re-read after the mux (the existing
/// `FileSectorSink::create` pattern). On Linux, [`with_size_hint`]
/// additionally calls `fallocate(FALLOC_FL_KEEP_SIZE)` to pre-reserve
/// extents.
///
/// [`with_size_hint`]: Self::with_size_hint
pub struct LocalFileSink {
inner: BufWriter<File>,
}
impl LocalFileSink {
/// Open `path` for writing, truncating any existing contents.
pub fn create(path: &Path) -> io::Result<Self> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(path)?;
Ok(Self {
inner: BufWriter::with_capacity(BUFFER_BYTES, file),
})
}
/// Like [`Self::create`] but additionally calls the per-OS
/// preallocate path with `size_bytes`. On Linux this is
/// `fallocate(FALLOC_FL_KEEP_SIZE)` so the on-disk extents are
/// reserved up front (reducing fragmentation for big sequential
/// muxer output); on other OSes it is a no-op today. Failures
/// from the preallocate call are non-fatal — the file is still
/// returned, just without the size reservation.
pub fn with_size_hint(path: &Path, size_bytes: u64) -> io::Result<Self> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(path)?;
preallocate::preallocate(&file, size_bytes);
Ok(Self {
inner: BufWriter::with_capacity(BUFFER_BYTES, file),
})
}
/// Drain the internal buffer and `fsync` the underlying file.
/// Idempotent with `Drop` (the `BufWriter` also flushes on drop;
/// this call additionally surfaces fsync errors to the caller).
/// [`SequentialSink::finish`](super::SequentialSink::finish)
/// delegates here so the durable flush happens through a trait
/// object too.
pub fn sync_all(&mut self) -> io::Result<()> {
self.inner.flush()?;
self.inner.get_ref().sync_all()
}
}
impl SequentialSink for LocalFileSink {
/// Flush the 4 MiB `BufWriter` and `fsync` the file. Overriding the
/// trait default is what makes a `dyn SequentialSink` / `dyn
/// RandomAccessSink` `finish()` actually durable instead of a no-op.
fn finish(&mut self) -> io::Result<()> {
self.sync_all()
}
}
impl RandomAccessSink for LocalFileSink {}
impl Write for LocalFileSink {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.inner.write(buf)
}
fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {
self.inner.write_all(buf)
}
fn flush(&mut self) -> io::Result<()> {
self.inner.flush()
}
}
impl Seek for LocalFileSink {
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
// Flush before seeking so buffered bytes land at the offset
// they were written for, not the new one.
self.inner.flush()?;
self.inner.get_mut().seek(from)
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Read;
#[test]
fn write_seek_roundtrip() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("rt.bin");
let mut s = LocalFileSink::create(&p).unwrap();
s.write_all(b"AAAA").unwrap();
s.write_all(b"BBBB").unwrap();
// Seek back over the second word and overwrite.
s.seek(SeekFrom::Start(4)).unwrap();
s.write_all(b"CCCC").unwrap();
s.sync_all().unwrap();
drop(s);
let mut f = File::open(&p).unwrap();
let mut got = Vec::new();
f.read_to_end(&mut got).unwrap();
assert_eq!(&got[..], b"AAAACCCC");
}
#[test]
fn drop_flushes() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("drop.bin");
{
let mut s = LocalFileSink::create(&p).unwrap();
s.write_all(b"buffered").unwrap();
// No explicit flush / sync_all — BufWriter drop runs the
// flush and the file should land on disk.
}
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..], b"buffered");
}
#[test]
fn with_size_hint_creates_writable_file() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("sz.bin");
let mut s = LocalFileSink::with_size_hint(&p, 64 * 1024).unwrap();
s.write_all(b"hint-ok").unwrap();
s.sync_all().unwrap();
drop(s);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..], b"hint-ok");
}
// ── Added hardening tests ───────────────────────────────────────
/// `create` must TRUNCATE an existing file (OpenOptions
/// `.truncate(true)`, lines 52-58). Pre-seed a long file, recreate
/// it via the sink, write a shorter payload — the old tail must be
/// gone. Mutation: dropping `.truncate(true)` would leave the stale
/// tail and the length assert fails.
#[test]
fn create_truncates_existing_file() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("trunc.bin");
std::fs::write(&p, vec![0xFFu8; 4096]).unwrap();
let mut s = LocalFileSink::create(&p).unwrap();
s.write_all(b"short").unwrap();
s.sync_all().unwrap();
drop(s);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(
bytes.len(),
5,
"create must truncate the pre-existing 4096 bytes"
);
assert_eq!(&bytes, b"short");
}
/// Seek must flush the BufWriter FIRST so buffered bytes land at
/// their intended offset, not the post-seek one (lines 121-128, and
/// the module doc's silent-corruption warning). We write into the
/// buffer (no explicit flush), seek backward, write again, and
/// confirm the first write stayed at offset 0. Mutation: removing
/// the `self.inner.flush()?` in `seek` would flush the first 4
/// bytes at the seeked offset, corrupting the file.
#[test]
fn seek_flushes_buffer_before_moving() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("seek-flush.bin");
let mut s = LocalFileSink::create(&p).unwrap();
// These bytes sit in the 4 MiB BufWriter, unflushed.
s.write_all(b"HEAD").unwrap();
// Seek forward to offset 10; the buffered HEAD must be flushed
// to offset 0 BEFORE the position moves.
s.seek(SeekFrom::Start(10)).unwrap();
s.write_all(b"TAIL").unwrap();
s.sync_all().unwrap();
drop(s);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(
&bytes[0..4],
b"HEAD",
"buffered head landed at the wrong offset"
);
assert_eq!(&bytes[10..14], b"TAIL");
}
}
+239
View File
@@ -0,0 +1,239 @@
//! Output-sink trait split for the buffering architecture.
//!
//! Two traits, one for each capability axis of an output destination:
//!
//! - [`SequentialSink`] — anything you can `Write` to in order. Sockets,
//! pipes, append-only stores, plain files. Containers that don't need
//! seek (M2TS, fMP4, HEVC elementary) target this.
//! - [`RandomAccessSink`] — everything `SequentialSink` plus a working
//! `Seek`. Local files, NFS files, anything with random-write
//! semantics. Containers that need backpatch (MKV cluster sizes, Cues
//! index, MP4 moov-at-end) target this.
//!
//! `RandomAccessSink: SequentialSink` — every random-access sink is
//! also a valid sequential sink. The muxer is generic over which it
//! requires (`MkvMux<S: RandomAccessSink>`, `M2tsMux<S: SequentialSink>`)
//! so an attempt to mux MKV to a network socket is a compile error.
//!
//! Buffering policy belongs to the concrete sink, not to a wrapper at
//! the call site. `LocalFileSink` wraps a `BufWriter<File>` with a
//! 4 MiB buffer for the common local-disk case; `WritebackFile`
//! (separate module) wraps a `File` with the adaptive-chunk
//! `sync_file_range` machinery for the Linux+NFS case.
use std::io::{Seek, Write};
mod local_file;
mod preallocate;
mod socket;
pub use local_file::LocalFileSink;
pub use socket::{SocketSink, UdpSocketSink};
/// Sequential-only write destination. Sockets, pipes, append-only
/// stores. No seek. Implementations own their write buffering — the
/// trait does not impose or hide any buffering of its own.
///
/// `finish` drains any internal buffering and signals end-of-stream to
/// the underlying transport (close-write on a socket, flush + fsync on
/// a buffered file, etc.). The default impl flushes via [`Write::flush`]
/// — correct for an unbuffered destination — but every concrete sink in
/// this module overrides it to drain its own buffer and run its
/// transport-specific finalisation (socket `shutdown(Write)`, file
/// `fsync`). There is deliberately NO blanket `impl SequentialSink for
/// T`: a blanket impl would force the no-op-style default on every
/// concrete sink (a blanket impl cannot be overridden per-type without a
/// coherence conflict), so a `Box<dyn SequentialSink>` / `&mut dyn
/// SequentialSink` `finish()` call would silently skip the flush and
/// transport shutdown. With explicit per-type impls the vtable dispatches
/// `finish` to the real implementation, so flush + durable-finish
/// actually happen through a trait object.
pub trait SequentialSink: Write + Send {
fn finish(&mut self) -> std::io::Result<()> {
self.flush()
}
}
/// Random-access write destination. Local files, NFS files, anything
/// with a working `Seek`. Inherits the `SequentialSink` contract — a
/// random-access sink is always usable as a sequential sink.
pub trait RandomAccessSink: SequentialSink + Seek {}
/// Pick the right `RandomAccessSink` impl for `dest` based on its
/// filesystem type.
///
/// - Linux + NFS path → `WritebackFile` with its adaptive-chunk
/// sync_file_range machinery and (when supported) `fallocate` size
/// hint.
/// - everything else → [`LocalFileSink`] over `BufWriter<File>`. On
/// non-Linux there is no `WritebackFile` machinery to opt into, and
/// on local Linux the kernel's default writeback policy is already
/// fine.
///
/// `size_hint`, when present, is forwarded to the per-OS preallocate
/// path (`fallocate(KEEP_SIZE)` on Linux, `F_PREALLOCATE` on macOS when
/// implemented, no-op elsewhere).
///
/// Returns a boxed trait object so the call site (mux construction)
/// stays agnostic of which concrete sink got picked.
// Not yet wired into mux::resolve (follow-up commit). Kept `pub(crate)` until
// then so an unfinished signature isn't frozen into the public 1.0 API.
#[allow(dead_code)]
pub(crate) fn open_for_mkv(
dest: &std::path::Path,
size_hint: Option<u64>,
) -> std::io::Result<Box<dyn RandomAccessSink>> {
#[cfg(target_os = "linux")]
{
use crate::platform::fs_type::{FsType, detect};
if detect(dest) == FsType::Nfs {
let wf = match size_hint {
Some(n) => crate::io::WritebackFile::create_with_size_hint(dest, n)?,
None => crate::io::WritebackFile::create(dest)?,
};
return Ok(Box::new(wf));
}
}
// Only Linux differentiates the sink by filesystem type (NFS gets
// the WritebackFile machinery); every other OS always uses
// `LocalFileSink`. Reference `detect` as a value (no call, no
// `statfs` syscall) so it isn't flagged dead on non-Linux while
// still avoiding the wasted probe whose result we'd discard.
#[cfg(not(target_os = "linux"))]
let _ = crate::platform::fs_type::detect;
let sink = match size_hint {
Some(n) => LocalFileSink::with_size_hint(dest, n)?,
None => LocalFileSink::create(dest)?,
};
Ok(Box::new(sink))
}
#[cfg(test)]
mod tests {
use super::*;
// Type-level assertion: the concrete sinks satisfy the trait
// objects. These functions never run; they just have to type-check.
fn _assert_is_sequential(_: &mut dyn SequentialSink) {}
fn _assert_is_random_access(_: &mut dyn RandomAccessSink) {}
#[test]
fn concrete_sinks_satisfy_traits() {
let dir = tempfile::tempdir().unwrap();
// `LocalFileSink` is a random-access (and thus sequential) sink.
let mut s = LocalFileSink::create(&dir.path().join("b.bin")).unwrap();
_assert_is_sequential(&mut s);
_assert_is_random_access(&mut s);
// `WritebackFile` ditto, via its explicit per-type impls.
let mut wf = crate::io::WritebackFile::create(&dir.path().join("c.bin")).unwrap();
_assert_is_sequential(&mut wf);
_assert_is_random_access(&mut wf);
}
#[test]
fn open_for_mkv_returns_a_random_access_sink() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("c.bin");
let mut sink = open_for_mkv(&p, Some(64 * 1024)).unwrap();
use std::io::{Seek, SeekFrom, Write};
sink.write_all(b"hello").unwrap();
sink.seek(SeekFrom::Start(0)).unwrap();
sink.finish().unwrap();
drop(sink);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..5], b"hello");
}
/// finish() through a `dyn SequentialSink` trait object must
/// dispatch to the concrete sink's override (flush + fsync), not a
/// no-op default. This is the regression test for the silent-no-op
/// finish() bug.
#[test]
fn finish_through_trait_object_flushes_local_file() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("trait-finish.bin");
let sink = LocalFileSink::create(&p).unwrap();
// Box as the trait object the production path uses.
let mut boxed: Box<dyn SequentialSink> = Box::new(sink);
boxed.write_all(b"buffered-tail").unwrap();
// finish() through the vtable must drain the 4 MiB BufWriter and
// fsync; the bytes must be visible to a separate reader BEFORE
// we drop the sink (drop-flush must not be what saves us).
boxed.finish().unwrap();
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..], b"buffered-tail");
}
// ── Added hardening tests ───────────────────────────────────────
use std::io::{self, Write};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
/// A minimal `SequentialSink` that does NOT override `finish`, so it
/// exercises the trait's DEFAULT impl (lines 51-55), which must call
/// `Write::flush`. We record whether flush ran. This pins the
/// documented contract that the default `finish` is "correct for an
/// unbuffered destination" by flushing. Mutation: changing the
/// default `finish` body from `self.flush()` to `Ok(())` would set
/// `flushed=false` and fail.
struct FlushTracker {
flushed: Arc<AtomicBool>,
bytes: Arc<AtomicUsize>,
}
impl Write for FlushTracker {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.bytes.fetch_add(buf.len(), Ordering::SeqCst);
Ok(buf.len())
}
fn flush(&mut self) -> io::Result<()> {
self.flushed.store(true, Ordering::SeqCst);
Ok(())
}
}
// Uses the DEFAULT finish() — deliberately no override.
impl SequentialSink for FlushTracker {}
#[test]
fn default_finish_flushes() {
let flushed = Arc::new(AtomicBool::new(false));
let bytes = Arc::new(AtomicUsize::new(0));
let mut sink = FlushTracker {
flushed: flushed.clone(),
bytes: bytes.clone(),
};
sink.write_all(b"abc").unwrap();
assert!(
!flushed.load(Ordering::SeqCst),
"flush should not run before finish"
);
sink.finish().unwrap();
assert!(
flushed.load(Ordering::SeqCst),
"default SequentialSink::finish must call Write::flush"
);
assert_eq!(bytes.load(Ordering::SeqCst), 3);
}
/// `open_for_mkv` with `None` size hint must still produce a working
/// random-access sink (the `match size_hint { None => ... }` arm,
/// lines 103-106). Round-trip a seek-back patch through it to prove
/// both Write and Seek dispatch. Mutation: if the None arm returned
/// a sequential-only sink the seek would not compile / would fail.
#[test]
fn open_for_mkv_without_size_hint_is_random_access() {
use std::io::{Seek, SeekFrom};
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("nohint.bin");
let mut sink = open_for_mkv(&p, None).unwrap();
sink.write_all(b"AAAABBBB").unwrap();
sink.seek(SeekFrom::Start(4)).unwrap();
sink.write_all(b"CCCC").unwrap();
sink.finish().unwrap();
drop(sink);
assert_eq!(std::fs::read(&p).unwrap(), b"AAAACCCC");
}
}
+23
View File
@@ -0,0 +1,23 @@
//! Linux `fallocate(FALLOC_FL_KEEP_SIZE)` preallocation.
//!
//! `KEEP_SIZE` reserves extents without changing the apparent file
//! length, which matches the muxer's expectation that writes still grow
//! the file naturally.
use std::fs::File;
use std::os::unix::io::AsRawFd;
pub(super) fn preallocate_impl(file: &File, size_bytes: u64) {
let fd = file.as_raw_fd();
// Clamp to the signed `off_t` range fallocate expects; an unchecked
// `as i64` cast would wrap a >= 2^63 size to a negative length that
// fallocate rejects with EINVAL (silent no-op).
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX);
// FALLOC_FL_KEEP_SIZE = 0x01.
let rc = unsafe { libc::fallocate(fd, libc::FALLOC_FL_KEEP_SIZE, 0, len) };
tracing::debug!(
target: "mux",
"LocalFileSink fallocate size_hint={size_bytes} rc={rc} ok={}",
rc == 0
);
}
+43
View File
@@ -0,0 +1,43 @@
//! macOS `F_PREALLOCATE` extent reservation.
//!
//! `fcntl(F_PREALLOCATE)` with `F_ALLOCATECONTIG | F_ALLOCATEALL` first
//! (prefer a contiguous run but accept scattered extents to satisfy the
//! full length) and fall back to `F_ALLOCATEALL` alone on failure.
//! Reported file size is unchanged — the muxer's writes still grow it.
use std::fs::File;
use std::os::unix::io::AsRawFd;
use crate::io::platform_macos::{
F_ALLOCATEALL, F_ALLOCATECONTIG, F_PEOFPOSMODE, F_PREALLOCATE, Fstore,
};
pub(super) fn preallocate_impl(file: &File, size_bytes: u64) {
let fd = file.as_raw_fd();
// Clamp to the signed `off_t` range; an unchecked `as off_t` cast
// would wrap a >= 2^63 size to a negative length.
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX) as libc::off_t;
let mut store = Fstore {
// Prefer a contiguous run but accept scattered extents to
// satisfy the full length. Without F_ALLOCATEALL the first
// attempt is best-effort and can return rc=0 with a partial
// allocation, so the fallback below would never fire. Matches
// writeback_file/macos.rs.
fst_flags: F_ALLOCATECONTIG | F_ALLOCATEALL,
fst_posmode: F_PEOFPOSMODE,
fst_offset: 0,
fst_length: len,
fst_bytesalloc: 0,
};
let mut rc = unsafe { libc::fcntl(fd, F_PREALLOCATE, &mut store as *mut Fstore) };
if rc == -1 {
// Fall back to non-contiguous only.
store.fst_flags = F_ALLOCATEALL;
rc = unsafe { libc::fcntl(fd, F_PREALLOCATE, &mut store as *mut Fstore) };
}
tracing::debug!(
target: "mux",
"LocalFileSink F_PREALLOCATE size_hint={size_bytes} rc={rc} bytesalloc={}",
store.fst_bytesalloc
);
}
+27
View File
@@ -0,0 +1,27 @@
//! Per-OS extent preallocation. Best-effort; failures are logged at
//! debug and otherwise swallowed because the file is still usable
//! without the size reservation — only large-file fragmentation gets
//! marginally worse.
use std::fs::File;
#[cfg(target_os = "linux")]
mod linux;
#[cfg(target_os = "macos")]
mod macos;
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
mod other;
#[cfg(target_os = "linux")]
use linux::preallocate_impl;
#[cfg(target_os = "macos")]
use macos::preallocate_impl;
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
use other::preallocate_impl;
/// Reserve `size_bytes` of disk space for `file`'s on-disk extents.
/// Reported file size is unchanged — writes still grow the file
/// naturally; only the allocator's extent map is primed.
pub(super) fn preallocate(file: &File, size_bytes: u64) {
preallocate_impl(file, size_bytes);
}
+10
View File
@@ -0,0 +1,10 @@
//! Fallback preallocate impl. No-op.
use std::fs::File;
pub(super) fn preallocate_impl(_file: &File, size_bytes: u64) {
tracing::debug!(
target: "mux",
"LocalFileSink preallocate size_hint={size_bytes} skipped (no platform impl)"
);
}
+376
View File
@@ -0,0 +1,376 @@
//! TCP / UDP socket sinks (sequential-only).
//!
//! [`SocketSink`] wraps a `TcpStream` in a 1 MiB `BufWriter`. Constructor
//! tunes `SO_SNDBUF` to a caller hint when provided. `finish()` flushes
//! the buffer then `shutdown(Write)`s the socket so the peer sees clean
//! end-of-stream.
//!
//! [`UdpSocketSink`] wraps a connected `UdpSocket`. Each `write` call
//! emits exactly one datagram — the caller is responsible for packetizing
//! to a reasonable MTU (188 × 7 = 1316 bytes for MPEG-TS-over-UDP is the
//! conventional choice). `finish()` is a no-op; UDP has no end-of-stream
//! marker.
//!
//! Both types implement [`SequentialSink`] explicitly so their
//! `finish()` dispatches correctly through a `dyn SequentialSink` trait
//! object (the `SocketSink` override drains the buffer and
//! `shutdown(Write)`s; the `UdpSocketSink` override flushes only).
//! Neither implements `Seek`, so neither satisfies [`RandomAccessSink`]
//! — using one with `MkvMux` is a compile error, which is the design
//! intent.
//!
//! [`SequentialSink`]: super::SequentialSink
//! [`RandomAccessSink`]: super::RandomAccessSink
use std::io::{self, BufWriter, Write};
use std::net::{Shutdown, SocketAddr, TcpStream, ToSocketAddrs, UdpSocket};
use super::SequentialSink;
/// `BufWriter` capacity for [`SocketSink`]. 1 MiB matches the typical
/// kernel send-buffer ceiling and keeps small-write amplification from
/// containers (TS = 188-byte packets, fMP4 fragment headers = ~100 bytes)
/// from translating into syscall storms.
const TCP_BUF_CAPACITY: usize = 1024 * 1024;
/// Sequential-only sink over a TCP connection.
///
/// Wraps a `BufWriter<TcpStream>`; the inner `TcpStream` is kept as a
/// clone so [`finish`](Self::finish) can call `shutdown(Write)` after
/// flushing the buffer (the buffered writer doesn't expose the socket
/// directly).
pub struct SocketSink {
/// Buffered write half. All payload bytes go through this.
buf: BufWriter<TcpStream>,
/// Shutdown handle — clone of the socket inside `buf`. Used only by
/// `finish()` for `shutdown(Write)`; never read or written through.
shutdown_handle: TcpStream,
}
impl SocketSink {
/// Open a TCP connection to `addr` and wrap it for sequential
/// writing. `sndbuf_bytes`, when present, is forwarded to
/// `setsockopt(SO_SNDBUF)` as a kernel hint — the OS may clamp it.
///
/// `addr` accepts anything `ToSocketAddrs` does: `"192.0.2.1:1234"`,
/// `("host", 1234)`, a `SocketAddr`, etc.
pub fn connect<A: ToSocketAddrs>(addr: A, sndbuf_bytes: Option<usize>) -> io::Result<Self> {
let stream = TcpStream::connect(addr)?;
// `set_nodelay(true)` keeps small writes (TS packet trains, fMP4
// moof headers) from sitting in Nagle's algorithm until the buffer
// fills. The BufWriter already absorbs syscall overhead; Nagle
// would just add latency without coalescing more. It is a latency
// hint, not a correctness requirement, so a platform that rejects
// TCP_NODELAY must not fail the connect — demote the error.
let _ = stream.set_nodelay(true);
if let Some(n) = sndbuf_bytes {
set_send_buffer(&stream, n)?;
}
let shutdown_handle = stream.try_clone()?;
Ok(Self {
buf: BufWriter::with_capacity(TCP_BUF_CAPACITY, stream),
shutdown_handle,
})
}
}
impl Write for SocketSink {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.buf.write(buf)
}
fn flush(&mut self) -> io::Result<()> {
self.buf.flush()
}
}
impl SequentialSink for SocketSink {
/// Drain the BufWriter and `shutdown(Write)` the underlying socket
/// so the peer sees a clean EOF. Overriding the trait default is
/// what makes a `dyn SequentialSink` `finish()` send the buffered
/// tail and the EOF instead of silently dropping them.
fn finish(&mut self) -> io::Result<()> {
self.buf.flush()?;
// `shutdown(Write)` signals clean EOF to the peer. Errors here
// are non-fatal — the connection may have already been torn down
// by the peer — but we surface them so callers can log.
self.shutdown_handle.shutdown(Shutdown::Write)
}
}
/// Sequential-only sink over a connected UDP socket.
///
/// Each [`write`](Write::write) call sends exactly one datagram. The
/// caller is responsible for splitting payload at packet boundaries —
/// for MPEG-TS this means 7 × 188 = 1316 bytes per datagram, the
/// industry standard for MPEG-TS-over-UDP. No buffering happens here;
/// adding it would silently merge datagrams.
///
/// `finish()` is a no-op: UDP has no end-of-stream marker. Closing the
/// socket happens on drop.
pub struct UdpSocketSink {
socket: UdpSocket,
}
impl UdpSocketSink {
/// Bind a local UDP socket to an ephemeral port and `connect` it to
/// `peer`. `connect` doesn't open a connection — it just fixes the
/// peer address so subsequent `send` calls don't need to repeat it,
/// and so receive-side filtering rejects packets from other sources.
///
/// `sndbuf_bytes`, when present, is a hint to `SO_SNDBUF`.
pub fn connect<A: ToSocketAddrs>(peer: A, sndbuf_bytes: Option<usize>) -> io::Result<Self> {
// Resolve the peer first so the local bind matches its address
// family. Binding `0.0.0.0:0` (IPv4) and then connecting to an
// IPv6 peer fails with EAFNOSUPPORT, so pick the wildcard that
// matches the resolved family.
let peer_addr = peer
.to_socket_addrs()?
.next()
.ok_or_else(|| io::Error::from(io::ErrorKind::AddrNotAvailable))?;
let bind_addr = match peer_addr {
SocketAddr::V4(_) => "0.0.0.0:0",
SocketAddr::V6(_) => "[::]:0",
};
// Bind to the matching wildcard / any port. The kernel picks an
// ephemeral source port and the source IP at first send.
let socket = UdpSocket::bind(bind_addr)?;
socket.connect(peer_addr)?;
if let Some(n) = sndbuf_bytes {
set_udp_send_buffer(&socket, n)?;
}
Ok(Self { socket })
}
}
impl Write for UdpSocketSink {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
// `send` writes the entire datagram or fails — no partial sends
// for UDP. Match `Write::write`'s contract by reporting bytes
// accepted.
self.socket.send(buf)
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
impl SequentialSink for UdpSocketSink {
/// UDP has no end-of-stream marker, so there is nothing to shut
/// down; `write` already sent each datagram unbuffered. Flush is a
/// no-op but kept explicit so the trait-object `finish()` matches
/// the concrete behaviour.
fn finish(&mut self) -> io::Result<()> {
Ok(())
}
}
// ── Platform `SO_SNDBUF` tuning ────────────────────────────────────────────
//
// std's `TcpStream` / `UdpSocket` don't expose `SO_SNDBUF`. We drop to
// libc on Linux + macOS (the libc-dep targets in Cargo.toml). On other
// targets the hint is silently ignored — the socket still works, the
// kernel just picks its own send-buffer size.
#[cfg(any(target_os = "linux", target_os = "macos"))]
fn set_send_buffer(stream: &TcpStream, bytes: usize) -> io::Result<()> {
use std::os::unix::io::AsRawFd;
setsockopt_sndbuf(stream.as_raw_fd(), bytes)
}
#[cfg(any(target_os = "linux", target_os = "macos"))]
fn set_udp_send_buffer(socket: &UdpSocket, bytes: usize) -> io::Result<()> {
use std::os::unix::io::AsRawFd;
setsockopt_sndbuf(socket.as_raw_fd(), bytes)
}
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
fn set_send_buffer(_stream: &TcpStream, _bytes: usize) -> io::Result<()> {
// Non-Linux-non-macOS targets aren't in Cargo.toml's libc dep list;
// silently ignore the hint rather than failing the connect. Callers
// can detect via the lack of an explicit "sndbuf applied" signal
// (not provided, intentionally — this is a hint, not a guarantee).
Ok(())
}
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
fn set_udp_send_buffer(_socket: &UdpSocket, _bytes: usize) -> io::Result<()> {
Ok(())
}
#[cfg(any(target_os = "linux", target_os = "macos"))]
fn setsockopt_sndbuf(fd: std::os::unix::io::RawFd, bytes: usize) -> io::Result<()> {
// Clamp into c_int range; SO_SNDBUF takes an `int` argument.
let want: libc::c_int = bytes.try_into().unwrap_or(libc::c_int::MAX);
let ret = unsafe {
libc::setsockopt(
fd,
libc::SOL_SOCKET,
libc::SO_SNDBUF,
&want as *const _ as *const libc::c_void,
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
)
};
if ret != 0 {
return Err(io::Error::last_os_error());
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Read;
use std::net::{TcpListener, UdpSocket};
use std::thread;
/// Bind a listener, accept on a thread, return (listener_addr,
/// accepted-bytes future via JoinHandle).
#[test]
fn socket_sink_round_trips_bytes() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let accept = thread::spawn(move || {
let (mut sock, _) = listener.accept().unwrap();
let mut buf = Vec::new();
sock.read_to_end(&mut buf).unwrap();
buf
});
let mut sink = SocketSink::connect(addr, Some(256 * 1024)).unwrap();
// Write enough to overflow the BufWriter at least once, then a
// tail that lives in the buffer until `finish` flushes.
let big: Vec<u8> = (0..(2 * TCP_BUF_CAPACITY))
.map(|i| (i & 0xff) as u8)
.collect();
sink.write_all(&big).unwrap();
sink.write_all(b"tail\n").unwrap();
sink.finish().unwrap();
drop(sink);
let received = accept.join().unwrap();
assert_eq!(received.len(), big.len() + 5);
assert_eq!(&received[..big.len()], &big[..]);
assert_eq!(&received[big.len()..], b"tail\n");
}
#[test]
fn socket_sink_is_sequential_only() {
// Compile-time assertion via dyn — if this ever started
// satisfying `RandomAccessSink`, the trait split would be broken.
fn _assert_seq(_: &mut dyn super::super::SequentialSink) {}
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let _accept = thread::spawn(move || {
let _ = listener.accept();
});
let mut sink = SocketSink::connect(addr, None).unwrap();
_assert_seq(&mut sink);
// The negative is harder to assert directly (no `is_not<T>`),
// but `SocketSink` does not impl `Seek`, so it can't unify with
// `RandomAccessSink`'s super-bound. The Phase 2 blanket impl
// `impl<T: SequentialSink + Seek> RandomAccessSink for T {}` thus
// excludes it by construction.
}
#[test]
fn udp_socket_sink_delivers_datagrams() {
let receiver = UdpSocket::bind("127.0.0.1:0").unwrap();
receiver
.set_read_timeout(Some(std::time::Duration::from_secs(2)))
.unwrap();
let addr = receiver.local_addr().unwrap();
let mut sink = UdpSocketSink::connect(addr, Some(128 * 1024)).unwrap();
sink.write_all(&[1, 2, 3, 4, 5]).unwrap();
sink.write_all(&[9, 9, 9]).unwrap();
sink.finish().unwrap();
let mut buf = [0u8; 64];
let n1 = receiver.recv(&mut buf).unwrap();
assert_eq!(&buf[..n1], &[1, 2, 3, 4, 5]);
let n2 = receiver.recv(&mut buf).unwrap();
assert_eq!(&buf[..n2], &[9, 9, 9]);
}
// ── Added hardening tests ───────────────────────────────────────
/// `SocketSink::finish` must signal a clean EOF to the peer via
/// `shutdown(Write)` (lines 91-97). The receiving side's
/// `read_to_end` only returns when it observes that EOF — if
/// `finish` merely flushed without the shutdown, `read_to_end`
/// would block forever (the socket stays half-open). We assert the
/// receiver completes promptly AND sees the buffered tail.
/// Mutation: replacing the `shutdown(Write)` line with `Ok(())`
/// makes the accept thread hang and the join times out.
#[test]
fn finish_signals_eof_to_peer() {
use std::sync::mpsc;
use std::time::Duration;
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let (tx, rx) = mpsc::channel();
thread::spawn(move || {
let (mut sock, _) = listener.accept().unwrap();
let mut buf = Vec::new();
// Returns only when the peer half-closes (shutdown Write).
sock.read_to_end(&mut buf).unwrap();
let _ = tx.send(buf);
});
let mut sink = SocketSink::connect(addr, None).unwrap();
sink.write_all(b"unflushed-tail").unwrap();
sink.finish().unwrap();
// read_to_end must complete because finish() shut down writes.
let received = rx
.recv_timeout(Duration::from_secs(3))
.expect("peer never saw EOF — finish() did not shutdown(Write)");
assert_eq!(received, b"unflushed-tail");
}
/// UDP `write` must emit ONE datagram per call carrying exactly the
/// bytes passed — no buffering, no coalescing (doc lines 100-108).
/// Two writes of different lengths must arrive as two separate
/// datagrams of those exact lengths, in order. Mutation: adding a
/// BufWriter to UdpSocketSink (the doc explicitly forbids it) would
/// merge these into one datagram and the second `recv` would time
/// out.
#[test]
fn udp_write_is_one_datagram_per_call() {
let receiver = UdpSocket::bind("127.0.0.1:0").unwrap();
receiver
.set_read_timeout(Some(std::time::Duration::from_secs(2)))
.unwrap();
let addr = receiver.local_addr().unwrap();
let mut sink = UdpSocketSink::connect(addr, None).unwrap();
// Distinct lengths so a merge would be detectable.
let n_a = sink.write(&[0xAA; 10]).unwrap();
let n_b = sink.write(&[0xBB; 20]).unwrap();
assert_eq!(n_a, 10);
assert_eq!(n_b, 20);
let mut buf = [0u8; 256];
let first = receiver.recv(&mut buf).unwrap();
assert_eq!(first, 10, "first datagram must be exactly 10 bytes");
assert!(buf[..first].iter().all(|&b| b == 0xAA));
let second = receiver.recv(&mut buf).unwrap();
assert_eq!(second, 20, "second datagram must be exactly 20 bytes");
assert!(buf[..second].iter().all(|&b| b == 0xBB));
}
/// UDP `finish` is a documented no-op (lines 157-165): there is no
/// EOF marker for UDP. Calling it must not error and must not
/// affect prior datagrams. Mutation: if `finish` tried to
/// `shutdown` the UDP socket it could error or close it
/// prematurely; here it must just return Ok.
#[test]
fn udp_finish_is_noop_ok() {
let receiver = UdpSocket::bind("127.0.0.1:0").unwrap();
let addr = receiver.local_addr().unwrap();
let mut sink = UdpSocketSink::connect(addr, None).unwrap();
assert!(sink.finish().is_ok());
// A second finish is equally harmless.
assert!(sink.finish().is_ok());
}
}
+17
View File
@@ -0,0 +1,17 @@
//! Per-platform writeback pipeline. On Linux, drains dirty pages
//! continuously at chunk granularity to keep the kernel's writeback
//! queue bounded. On macOS and Windows, a no-op stub.
//!
//! The platform decision lives entirely in this file (the cfg-gated
//! `pub use` below). Callers — and `DiskWriter` itself — are
//! platform-independent.
#[cfg(target_os = "linux")]
mod linux;
#[cfg(not(target_os = "linux"))]
mod noop;
#[cfg(target_os = "linux")]
pub(super) use linux::WritebackPipeline;
#[cfg(not(target_os = "linux"))]
pub(super) use noop::WritebackPipeline;
+626
View File
@@ -0,0 +1,626 @@
//! Linux writeback pipeline using `sync_file_range` + `posix_fadvise`.
//!
//! Pathology this fixes: the kernel's default `vm.dirty_ratio` (~20 %
//! of RAM) lets dirty pages accumulate to hundreds of MB during a
//! big sequential write, then bursts a flush at 99 % disk utilisation.
//! While the burst runs, app writes block on the writeback queue —
//! observed empirically as instantaneous speed dropping from ~15 MB/s
//! to ~1 MB/s every ~30 s during a Pass 1 sweep.
//!
//! Strategy: every `chunk_bytes` of new sequential output, kick async
//! writeback (`SYNC_FILE_RANGE_WRITE`) on the just-completed chunk and
//! finalise the *previous* chunk via `WAIT_AFTER` + `posix_fadvise
//! (DONTNEED)`. By the time we finalise, that previous chunk has had
//! a full chunk's worth of work to flush — the wait is near-instant.
//! Dirty cache stays bounded at ~2 × `chunk_bytes` and writes drain
//! continuously instead of in bursts.
//!
//! The chunk size is adaptive: we measure the elapsed time of the
//! `WAIT_AFTER` call over a rolling window of the last 16 chunks and
//! resize the chunk based on the p95. Slow storage (NFS, network
//! shares, HDD) sees larger chunks to amortise per-chunk overhead;
//! fast storage (NVMe) sees smaller chunks to keep cache pressure
//! tight. Bounds: [4 MiB, 256 MiB].
//!
//! ## NFS escape hatch
//!
//! `sync_file_range(WAIT_AFTER)` on an NFS-mounted file can block
//! indefinitely waiting for the server's commit ack. If the server
//! never acks (network partition, server-side hang, slow commit), the
//! syscall never returns and the consumer thread is stuck inside the
//! kernel — `/api/stop` can't reach it because halt is cooperative.
//!
//! When `fstatfs` reports the file lives on an NFS mount
//! (`f_type == NFS_SUPER_MAGIC`), the pipeline skips the WAIT_AFTER +
//! `posix_fadvise(DONTNEED)` dance entirely. NFS clients have their
//! own buffering and commit semantics that handle dirty-page bounds
//! without us forcing the issue. The async `SYNC_FILE_RANGE_WRITE`
//! kickoff still runs (non-blocking by spec) so writeback still gets
//! a nudge.
//!
//! ## Defence in depth: WAIT_AFTER timeout
//!
//! Even on local storage, a degraded disk or odd filesystem driver
//! could in principle wedge inside WAIT_AFTER. Each WAIT_AFTER call
//! runs on a worker thread with a 30s recv_timeout on its result
//! channel. On timeout we log a loud error, set a `degraded` flag,
//! and from then on skip WAIT_AFTER + DONTNEED for the rest of the
//! pipeline's life (same shape as the NFS path). The worker thread
//! is intentionally leaked — it unwinds whenever the syscall
//! eventually returns or the process exits. The mux continues; the
//! original dirty-burst pathology re-emerges but the rip can still
//! finish instead of freezing.
use std::collections::VecDeque;
use std::fs::File;
use std::os::unix::io::{AsRawFd, RawFd};
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
const ADAPTIVE_WINDOW: usize = 16;
const CHUNK_BYTES_MIN: u64 = 4 * 1024 * 1024;
const CHUNK_BYTES_MAX: u64 = 256 * 1024 * 1024;
const ADAPTIVE_GROW_MS: u64 = 200;
const ADAPTIVE_SHRINK_MS: u64 = 20;
/// Every N chunks, emit a `debug!` snapshot of the current chunk
/// size so operators tailing the log can see where the autoscaler
/// settled.
const SIZE_LOG_INTERVAL: u64 = 32;
/// Hard upper bound on a single `sync_file_range(WAIT_AFTER)` call.
/// Beyond this we declare the pipeline degraded and stop calling
/// WAIT_AFTER for the rest of its life.
const WAIT_AFTER_TIMEOUT: Duration = Duration::from_secs(30);
pub(crate) struct WritebackPipeline {
/// Aliases the wrapping `WritebackFile::file`. Only valid for the
/// lifetime of that struct — moving the `File` independently
/// would silently UAF this fd. The pipeline is a private field of
/// `WritebackFile` and never exposed outside that wrapper, which
/// is what keeps the alias sound.
fd: RawFd,
/// An owned clone of the file descriptor, held so that any
/// leaked WAIT_AFTER worker thread retains a valid reference to
/// the underlying file description for the duration of its
/// syscall — even if the original `WritebackFile` is closed first
/// and the OS reuses its fd number. `None` only when `try_clone`
/// failed at construction (rare); the pipeline falls back to the
/// pre-clone `fd` integer in that case, which carries the original
/// fd-reuse risk but is no worse than the previous behaviour.
wait_file: Option<File>,
chunk_bytes: u64,
last_flush_pos: u64,
pending: Option<(u64, u64)>,
/// Rolling window of recent `WAIT_AFTER` elapsed_ms measurements.
wait_after_window: VecDeque<u64>,
/// Count of chunks emitted (used to space out periodic
/// `debug!` size snapshots).
chunk_count: u64,
/// True when the underlying file is on an NFS mount. NFS makes
/// WAIT_AFTER unsafe (can block forever on missing server ack), so
/// we skip it entirely and let the NFS client handle commit on
/// close.
is_nfs: bool,
/// Set the first time WAIT_AFTER exceeds [`WAIT_AFTER_TIMEOUT`].
/// Once set, behaviour matches the NFS path for the rest of the
/// pipeline's life. A plain `AtomicBool`: the flag is only ever
/// touched on the owning thread (the spawned WAIT_AFTER worker never
/// reads or writes it). `AtomicBool` over `bool` only because the
/// load/store sites read cleanly; no sharing is needed today.
degraded: AtomicBool,
}
impl WritebackPipeline {
/// Construct a pipeline aliasing `file`'s file descriptor. The
/// returned `WritebackPipeline` MUST be dropped before `file`
/// itself, or kept inside the same struct that owns `file` — the
/// alias is unchecked.
pub(crate) fn new(file: &File, start_pos: u64, chunk_bytes: u64) -> Self {
let fd = file.as_raw_fd();
let is_nfs = detect_nfs(fd);
// Clone the fd so any leaked WAIT_AFTER worker thread keeps the
// file description alive. Log but continue on clone failure.
let wait_file = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
tracing::warn!(
target: "mux",
"WritebackPipeline fd={fd}: try_clone failed ({e}), WAIT_AFTER workers \
will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
tracing::info!(
target: "mux",
"WritebackPipeline fd={fd} is_nfs={is_nfs} chunk_bytes={chunk_bytes} strategy={}",
if is_nfs { "nfs-skip-wait" } else { "wait+dontneed" }
);
Self {
fd,
wait_file,
chunk_bytes,
last_flush_pos: start_pos,
pending: None,
wait_after_window: VecDeque::with_capacity(ADAPTIVE_WINDOW),
chunk_count: 0,
is_nfs,
degraded: AtomicBool::new(false),
}
}
/// True if we should bypass the WAIT_AFTER + DONTNEED finalisation
/// step. NFS always bypasses; local storage bypasses once the
/// pipeline has flipped to degraded after a WAIT_AFTER timeout.
#[inline]
fn skip_wait(&self) -> bool {
self.is_nfs || self.degraded.load(Ordering::Relaxed)
}
/// Produce a fresh per-call `File` clone for the WAIT_AFTER worker.
///
/// Each call to `wait_after_with_timeout` needs its own owned clone
/// so the worker thread keeps the file description alive for the
/// duration of the syscall. We clone from `self.wait_file` (itself a
/// clone taken at construction) rather than from the original file.
///
/// Returns `None` only if `wait_file` is `None` (construction
/// try_clone failed) or if the second-level try_clone fails — both
/// rare; the fallback raw-fd path in `wait_after_with_timeout`
/// handles that case.
#[inline]
fn clone_for_worker(&self) -> Option<File> {
self.wait_file.as_ref().and_then(|f| f.try_clone().ok())
}
/// Caller advanced the file position to `pos`. If a chunk boundary
/// was crossed, kick async writeback for the just-completed chunk
/// and finalise the previous one.
pub(crate) fn note_progress(&mut self, pos: u64) {
if pos < self.last_flush_pos.saturating_add(self.chunk_bytes) {
return;
}
// Byte offsets are unsigned throughout; the signed cast happens
// only at the libc call boundary where the kernel ABI requires
// `i64`. `saturating_sub` documents and hardens the line-above
// guard that `pos >= last_flush_pos`.
let chunk_off: u64 = self.last_flush_pos;
let chunk_len: u64 = pos.saturating_sub(self.last_flush_pos);
let mut wait_ms: u64 = 0;
let mut fadvise_ms: u64 = 0;
// Async kickoff for the just-completed chunk runs on every
// path (NFS, degraded, normal) — it's nominally non-blocking
// by spec and gives the kernel an early hint that this range
// is ready to flush.
let kickoff_rc = unsafe {
libc::sync_file_range(
self.fd,
chunk_off as i64,
chunk_len as i64,
libc::SYNC_FILE_RANGE_WRITE,
)
};
if kickoff_rc != 0 {
// Non-fatal: the async write-out hint failed, but the data is
// still in the page cache and will be flushed by later fsync /
// kernel writeback. Surface it for diagnosability.
tracing::warn!(
target: "freemkv::io",
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
"sync_file_range(WRITE) kickoff failed"
);
}
if let Some((prev_off, prev_len)) = self.pending.take() {
if self.skip_wait() {
// NFS branch (or degraded fallback after a prior
// timeout): the WAIT_AFTER + DONTNEED dance is what
// hangs on NFS — skip it. We still advance `pending`
// so the next call has a stable cycle.
} else {
// Normal local-storage branch with belt-and-braces
// timeout. If WAIT_AFTER hangs > WAIT_AFTER_TIMEOUT
// we mark the pipeline degraded, log a loud error,
// and fall through to the skip path on subsequent
// calls.
match wait_after_with_timeout(self.clone_for_worker(), self.fd, prev_off, prev_len)
{
Some(ms) => {
wait_ms = ms;
let t_fadv = Instant::now();
unsafe {
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
fadvise_ms = t_fadv.elapsed().as_millis() as u64;
self.record_wait(wait_ms);
}
None => {
// Timeout branch: switch to NFS-style skip
// for the rest of the pipeline's life. Do
// NOT call DONTNEED — if WAIT_AFTER hasn't
// returned, the pages aren't safely flushed.
self.degraded.store(true, Ordering::Relaxed);
// Once degraded we skip DONTNEED, so every subsequent
// chunk's pages stay resident until close — the same
// page-cache exposure profile as NFS. Shrink to the
// floor so that exposure window is as small as the NFS
// path keeps it, instead of whatever the adaptive sizing
// had grown chunk_bytes to (up to 256 MiB).
self.chunk_bytes = CHUNK_BYTES_MIN;
tracing::error!(
target: "mux",
"WritebackPipeline WAIT_AFTER timed out after {}s on chunk off={} len={}, marking writeback degraded (subsequent chunks will skip WAIT_AFTER + DONTNEED, chunk_bytes lowered to floor)",
WAIT_AFTER_TIMEOUT.as_secs(),
prev_off,
prev_len
);
}
}
}
}
self.pending = Some((chunk_off, chunk_len));
self.last_flush_pos = pos;
self.chunk_count += 1;
tracing::trace!(
target: "mux",
"WritebackPipeline chunk off={} len={} wait_after_ms={wait_ms} fadvise_ms={fadvise_ms} chunk_bytes={} skip_wait={}",
chunk_off,
chunk_len,
self.chunk_bytes,
self.skip_wait(),
);
if self.chunk_count % SIZE_LOG_INTERVAL == 0 {
tracing::debug!(
target: "mux",
"WritebackPipeline chunk_bytes={} after {} chunks is_nfs={} degraded={}",
self.chunk_bytes,
self.chunk_count,
self.is_nfs,
self.degraded.load(Ordering::Relaxed),
);
}
}
/// Push a new `WAIT_AFTER` measurement into the rolling window
/// and, if the window is full, adapt `chunk_bytes` based on p95.
fn record_wait(&mut self, wait_ms: u64) {
if self.wait_after_window.len() == ADAPTIVE_WINDOW {
self.wait_after_window.pop_front();
}
self.wait_after_window.push_back(wait_ms);
if self.wait_after_window.len() < ADAPTIVE_WINDOW {
return;
}
// p95 index, derived from the window size so it stays valid if
// ADAPTIVE_WINDOW changes (a hard-coded `[14]` would panic OOB
// for a window <= 14). For the default 16 this is index 15
// (ceil(16 * 95 / 100) - 1 = 15), i.e. the top sample.
let mut sorted: Vec<u64> = self.wait_after_window.iter().copied().collect();
sorted.sort_unstable();
let p95_idx = (ADAPTIVE_WINDOW * 95).div_ceil(100).min(ADAPTIVE_WINDOW) - 1;
let p95 = sorted[p95_idx];
let old = self.chunk_bytes;
let new = if p95 > ADAPTIVE_GROW_MS && self.chunk_bytes < CHUNK_BYTES_MAX {
(self.chunk_bytes * 2).min(CHUNK_BYTES_MAX)
} else if p95 < ADAPTIVE_SHRINK_MS && self.chunk_bytes > CHUNK_BYTES_MIN {
(self.chunk_bytes / 2).max(CHUNK_BYTES_MIN)
} else {
self.chunk_bytes
};
if new != old {
self.chunk_bytes = new;
tracing::info!(
target: "mux",
"WritebackPipeline adaptive chunk_bytes {} -> {} p95_ms={p95}",
old,
new
);
}
}
/// Caller is about to seek away from the current write region.
/// Drain any in-flight chunk and reset tracking.
pub(crate) fn handle_seek(&mut self, new_pos: u64) {
self.finalize();
self.last_flush_pos = new_pos;
}
/// Drain any in-flight chunk. Idempotent. Call before `sync_all()`
/// or when discarding the pipeline.
pub(crate) fn finalize(&mut self) {
if let Some((prev_off, prev_len)) = self.pending.take() {
tracing::debug!(
target: "mux",
"WritebackPipeline finalize chunk off={prev_off} len={prev_len} skip_wait={} is_nfs={} degraded={}",
self.skip_wait(),
self.is_nfs,
self.degraded.load(Ordering::Relaxed),
);
if self.skip_wait() {
// NFS / degraded: skip WAIT_AFTER + DONTNEED. close()
// / sync_all() handle commit through their normal
// paths.
return;
}
match wait_after_with_timeout(self.clone_for_worker(), self.fd, prev_off, prev_len) {
Some(_ms) => unsafe {
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
},
None => {
self.degraded.store(true, Ordering::Relaxed);
tracing::error!(
target: "mux",
"WritebackPipeline finalize WAIT_AFTER timed out after {}s on chunk off={prev_off} len={prev_len}, marking writeback degraded",
WAIT_AFTER_TIMEOUT.as_secs(),
);
}
}
}
}
}
/// Probe whether `fd` lives on an NFS mount. Thin wrapper around
/// [`crate::platform::fs_type::detect_fd`] so writeback policy and
/// general-purpose fs-type classification stay in sync (same magic
/// numbers, same musl-vs-glibc cast handling).
///
/// Fails open: any classification other than NFS counts as "not NFS"
/// (including `Unknown` on `fstatfs` error) — better to run the
/// normal local-storage path on a misdetected NFS mount and surface
/// the freeze loudly via [`WAIT_AFTER_TIMEOUT`] than to needlessly
/// disable writeback bounding on every local file because of a
/// transient stat error.
fn detect_nfs(fd: RawFd) -> bool {
matches!(
crate::platform::fs_type::detect_fd(fd),
crate::platform::fs_type::FsType::Nfs
)
}
/// Run `sync_file_range(WAIT_AFTER)` on a worker thread and wait up
/// to [`WAIT_AFTER_TIMEOUT`] for it to return. `Some(elapsed_ms)` on
/// success; `None` on timeout. On timeout the worker thread is
/// intentionally leaked — it unwinds whenever the syscall eventually
/// returns or the process exits.
///
/// This delegates to [`crate::io::bounded::bounded_syscall`], the
/// generic worker-thread + `recv_timeout` primitive, and just adapts it
/// to the WAIT_AFTER call shape: it returns `elapsed_ms` instead of the
/// syscall's `()`, and treats `WorkerLost` as a benign no-op to match
/// the original semantics.
///
/// ## fd lifetime / fd-reuse safety
///
/// `worker_file` is an *owned* `File` (produced by `File::try_clone` at
/// pipeline construction). It is moved into the worker closure so the
/// file description stays alive for exactly as long as the worker thread
/// lives — even if the original `WritebackFile` is closed and the OS
/// reuses its fd number before the worker's syscall returns.
///
/// `fallback_fd` is used only when `worker_file` is `None` (i.e. the
/// `try_clone` at construction failed). In that case the worker captures
/// the raw fd integer, which carries the original fd-reuse risk but is
/// no worse than the pre-fix behaviour.
fn wait_after_with_timeout(
worker_file: Option<File>,
fallback_fd: RawFd,
off: u64,
len: u64,
) -> Option<u64> {
let started = Instant::now();
let result = if let Some(owned) = worker_file {
// Happy path: the closure owns a cloned File that keeps the
// file description alive until the worker drops it.
crate::io::bounded::bounded_syscall(None, WAIT_AFTER_TIMEOUT, move || unsafe {
let fd = owned.as_raw_fd();
libc::sync_file_range(fd, off as i64, len as i64, libc::SYNC_FILE_RANGE_WAIT_AFTER);
// `owned` drops here, closing the cloned fd.
})
} else {
// Fallback: try_clone failed at construction; use the raw fd.
// This carries the pre-fix fd-reuse risk on timeout, but is no
// regression from the original behaviour.
crate::io::bounded::bounded_syscall(None, WAIT_AFTER_TIMEOUT, move || unsafe {
libc::sync_file_range(
fallback_fd,
off as i64,
len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
})
};
match result {
Ok(()) => Some(started.elapsed().as_millis() as u64),
Err(crate::io::bounded::BoundedError::Timeout)
| Err(crate::io::bounded::BoundedError::Halted) => None,
Err(crate::io::bounded::BoundedError::WorkerLost) => {
// Worker thread spawn failed or panicked before sending.
// Treat as a benign success (no syscall ran) rather than
// a degrade trigger — falling through with elapsed_ms=0
// matches the no-op behaviour.
Some(0)
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::NamedTempFile;
/// Helper: build a `WritebackPipeline` over a local tempfile. On
/// every test rig (linux dev box, CI) the tempfile lives on a
/// local FS, so `is_nfs=false` and `skip_wait` returns false until
/// we explicitly mark the pipeline degraded.
fn local_pipeline(chunk_bytes: u64) -> (NamedTempFile, WritebackPipeline) {
let f = NamedTempFile::new().expect("tempfile create");
let pipeline = WritebackPipeline::new(f.as_file(), 0, chunk_bytes);
(f, pipeline)
}
#[test]
fn new_pipeline_starts_active() {
let (_f, p) = local_pipeline(32 * 1024 * 1024);
assert!(!p.is_nfs, "local tempfile must not classify as NFS");
assert!(!p.degraded.load(Ordering::Relaxed));
assert!(!p.skip_wait(), "fresh local pipeline must not skip wait");
}
#[test]
fn degraded_flag_short_circuits_wait() {
let (_f, p) = local_pipeline(32 * 1024 * 1024);
assert!(!p.skip_wait());
p.degraded.store(true, Ordering::Relaxed);
assert!(
p.skip_wait(),
"degraded flag must force the wait+dontneed bypass"
);
}
#[test]
fn record_wait_grows_chunk_on_high_p95() {
let (_f, mut p) = local_pipeline(16 * 1024 * 1024);
// Fill the window with samples above the grow threshold.
for _ in 0..ADAPTIVE_WINDOW {
p.record_wait(ADAPTIVE_GROW_MS + 50);
}
assert!(
p.chunk_bytes > 16 * 1024 * 1024,
"chunk should have grown; got {}",
p.chunk_bytes
);
assert!(p.chunk_bytes <= CHUNK_BYTES_MAX);
}
#[test]
fn record_wait_shrinks_chunk_on_low_p95() {
let (_f, mut p) = local_pipeline(64 * 1024 * 1024);
for _ in 0..ADAPTIVE_WINDOW {
p.record_wait(1); // well under ADAPTIVE_SHRINK_MS
}
assert!(
p.chunk_bytes < 64 * 1024 * 1024,
"chunk should have shrunk; got {}",
p.chunk_bytes
);
assert!(p.chunk_bytes >= CHUNK_BYTES_MIN);
}
#[test]
fn record_wait_no_op_below_window_fill() {
let (_f, mut p) = local_pipeline(16 * 1024 * 1024);
let initial = p.chunk_bytes;
// Only push a few samples; window not full → no adaptation.
for _ in 0..(ADAPTIVE_WINDOW - 1) {
p.record_wait(ADAPTIVE_GROW_MS + 100);
}
assert_eq!(
p.chunk_bytes, initial,
"chunk must not change before window is full"
);
}
#[test]
fn record_wait_clamps_to_chunk_bounds() {
// Grow past the max.
let (_f, mut p) = local_pipeline(CHUNK_BYTES_MAX);
for _ in 0..ADAPTIVE_WINDOW {
p.record_wait(ADAPTIVE_GROW_MS + 1000);
}
assert_eq!(p.chunk_bytes, CHUNK_BYTES_MAX, "must clamp to MAX");
// Shrink past the min.
let (_f, mut p) = local_pipeline(CHUNK_BYTES_MIN);
for _ in 0..ADAPTIVE_WINDOW {
p.record_wait(0);
}
assert_eq!(p.chunk_bytes, CHUNK_BYTES_MIN, "must clamp to MIN");
}
#[test]
fn detect_nfs_local_file_is_false() {
// Local tempfile must not classify as NFS. This locks in the
// consolidation through `crate::platform::fs_type::detect_fd`.
let f = NamedTempFile::new().expect("tempfile create");
use std::os::unix::io::AsRawFd;
assert!(!detect_nfs(f.as_file().as_raw_fd()));
}
#[test]
fn note_progress_below_chunk_is_noop() {
let (_f, mut p) = local_pipeline(32 * 1024 * 1024);
// No-op return before crossing the first chunk boundary.
let before = p.chunk_count;
p.note_progress(1024); // < 32 MiB
assert_eq!(p.chunk_count, before);
assert!(p.pending.is_none());
}
// ── Bug-fix regression tests ────────────────────────────────────────
/// Regression for the fd-reuse / use-after-close fix. Verifies that
/// `WritebackPipeline::new` successfully clones the fd into
/// `wait_file` (i.e. `try_clone` doesn't fail for a normal
/// tempfile) and that `clone_for_worker` returns `Some` — meaning
/// the WAIT_AFTER worker will capture an owned `File` rather than a
/// raw fd integer.
///
/// A deterministic test for the actual fd-reuse race is not clean to
/// write (it would require simultaneously closing the original File
/// and re-opening a new one to steal the fd number while the worker
/// is mid-syscall, which is inherently racy). This test instead pins
/// the structural invariant: on a normal local file, the pipeline
/// holds a valid clone and will give the worker an owned File.
#[test]
fn wait_file_clone_is_present_for_local_tempfile() {
let (_f, p) = local_pipeline(32 * 1024 * 1024);
assert!(
p.wait_file.is_some(),
"wait_file must be Some for a normal local tempfile (try_clone should not fail)"
);
// clone_for_worker must return Some — the worker will get an
// owned File, not fall through to the raw-fd fallback.
let worker_clone = p.clone_for_worker();
assert!(
worker_clone.is_some(),
"clone_for_worker must return Some when wait_file is Some"
);
}
/// Structural: the worker `File` clone returned by `clone_for_worker`
/// is a distinct file descriptor (different fd number) that refers to
/// the same underlying file. Closing the original tempfile must not
/// affect the clone's validity — the OS keeps the file description
/// alive until all file descriptors referring to it are closed.
///
/// We verify "distinct fd number" and "still usable as a raw fd"
/// without actually racing a syscall.
#[test]
fn worker_clone_has_distinct_fd_from_original() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
let pipeline = WritebackPipeline::new(f.as_file(), 0, 32 * 1024 * 1024);
let clone = pipeline
.clone_for_worker()
.expect("clone_for_worker returned None");
let clone_fd = clone.as_raw_fd();
// The clone must have a different fd number — it is a separate
// open file description (dup'd by try_clone).
assert_ne!(
clone_fd, original_fd,
"worker clone must have a distinct fd number from the original"
);
// The clone fd must be valid (non-negative on Unix).
assert!(clone_fd >= 0, "clone fd must be non-negative");
}
}
+18
View File
@@ -0,0 +1,18 @@
//! No-op writeback pipeline for non-Linux targets. macOS and Windows
//! page cache policies have not been shown to exhibit the Linux
//! accumulate-then-burst flush pathology for our access pattern.
//! If that changes, replace this stub with a real implementation
//! (e.g. `F_NOCACHE` on macOS, `FILE_FLAG_WRITE_THROUGH` on Windows).
use std::fs::File;
pub(crate) struct WritebackPipeline;
impl WritebackPipeline {
pub(crate) fn new(_file: &File, _start_pos: u64, _chunk_bytes: u64) -> Self {
Self
}
pub(crate) fn note_progress(&mut self, _pos: u64) {}
pub(crate) fn handle_seek(&mut self, _new_pos: u64) {}
pub(crate) fn finalize(&mut self) {}
}
+145
View File
@@ -0,0 +1,145 @@
//! Linux platform impl for [`super::WritebackFile`].
//!
//! - `preallocate`: `fallocate(FALLOC_FL_KEEP_SIZE)` — reserve extents
//! without growing the reported file size. Reduces extent
//! fragmentation on large sequential writes (mux output on NFS in
//! particular).
//! - `durable_sync`: `fsync` wrapped in
//! [`crate::io::bounded::bounded_syscall`] with a 60 s deadline so a
//! wedged NFS server can't trap the calling thread indefinitely.
use std::fs::File;
use std::io;
use std::os::unix::io::AsRawFd;
use std::time::Duration;
/// Pre-reserve extents for `size_bytes` of upcoming sequential writes.
/// Best-effort: a non-zero rc is logged but not propagated, since the
/// caller would just continue with the unreserved file anyway.
pub(super) fn preallocate(file: &File, size_bytes: u64) {
// FALLOC_FL_KEEP_SIZE = 0x01 — keep the reported file size at 0
// (writes grow it normally) while still pre-reserving the extents.
// Clamp to the signed `off_t` range; an unchecked `as i64` cast
// would wrap a >= 2^63 size to a negative length (EINVAL no-op).
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX);
let rc = unsafe { libc::fallocate(file.as_raw_fd(), libc::FALLOC_FL_KEEP_SIZE, 0, len) };
tracing::debug!(
target: "mux",
"WritebackFile fallocate size_hint={size_bytes} rc={rc} ok={}",
rc == 0
);
}
/// Run `fsync` on `file` with a 60 s deadline. On timeout — and
/// likewise on halt or a lost worker — we log and return `Ok(())`: the
/// kernel will still flush on close, so the data is best-effort durable.
/// The alternative (trap the thread for the rest of the rip, or return
/// an error that aborts an otherwise-complete mux) is worse, so all
/// three fallbacks return `Ok(())`. `Ok(())` from these paths is NOT a
/// durability barrier — the durable flush did not complete; only the
/// hang is bounded.
///
/// ## fd-reuse safety
///
/// The `fsync` runs on a bounded worker thread that may be leaked on
/// timeout. To avoid the leaked worker's syscall hitting a recycled fd
/// number after the original `File` is closed, we `try_clone` an owned
/// `File` and move it into the closure. The clone keeps the underlying
/// file description alive for as long as the worker thread lives.
/// On `try_clone` failure (rare) we fall back to the raw fd integer —
/// no worse than the previous behaviour.
pub(super) fn durable_sync(file: &File) -> io::Result<()> {
// Clone so a leaked worker thread retains a valid fd even after the
// original File is closed and its fd number is reused.
let owned = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
let fd = file.as_raw_fd();
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fd={fd}: try_clone failed ({e}), fsync worker will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
let fallback_fd = file.as_raw_fd();
match crate::io::bounded::bounded_syscall(
None,
Duration::from_secs(60),
move || -> io::Result<()> {
let fd = owned.as_ref().map(|f| f.as_raw_fd()).unwrap_or(fallback_fd);
let rc = unsafe { libc::fsync(fd) };
// `owned` (if Some) drops here, releasing the cloned fd.
if rc == 0 {
Ok(())
} else {
Err(io::Error::last_os_error())
}
},
) {
Ok(inner) => inner,
Err(crate::io::bounded::BoundedError::Timeout) => {
tracing::error!(
target: "mux",
"WritebackFile::sync_all fsync timed out after 60s; kernel will flush on close (best-effort)"
);
Ok(())
}
Err(crate::io::bounded::BoundedError::Halted) => {
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fsync skipped (halt requested); data not durably flushed, kernel will flush on close"
);
Ok(())
}
Err(crate::io::bounded::BoundedError::WorkerLost) => {
tracing::error!(
target: "mux",
"WritebackFile::sync_all fsync worker lost before completion; data not durably flushed, kernel will flush on close"
);
Ok(())
}
}
}
#[cfg(test)]
#[cfg(target_os = "linux")]
mod tests {
use super::*;
use tempfile::NamedTempFile;
/// Regression for the fd-reuse / use-after-close fix in `durable_sync`.
///
/// Verifies the structural invariant: `try_clone` succeeds for a normal
/// local tempfile, and the cloned `File` has a distinct fd number from
/// the original. This pins the property that a leaked fsync worker thread
/// captures an owned `File` (and thus keeps the file description alive)
/// rather than a bare fd integer that can be reused after the original
/// `File` closes.
///
/// The actual fd-reuse race is non-deterministic and not cleanly
/// testable without coordinating a simultaneous close + re-open on
/// another thread. A structural test is the accepted substitute.
#[test]
fn durable_sync_worker_uses_owned_clone_with_distinct_fd() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
// try_clone must succeed for a normal local file.
let owned = f
.as_file()
.try_clone()
.expect("try_clone must succeed for a local tempfile");
let clone_fd = owned.as_raw_fd();
// The clone must be a distinct fd (dup'd, not aliased).
assert_ne!(
clone_fd, original_fd,
"owned clone must have a distinct fd number — not an alias of the original"
);
assert!(clone_fd >= 0, "clone fd must be a valid non-negative fd");
// durable_sync must complete without error on the local tempfile.
durable_sync(f.as_file()).expect("durable_sync must return Ok on a local tempfile");
}
}
+159
View File
@@ -0,0 +1,159 @@
//! macOS platform impl for [`super::WritebackFile`].
//!
//! - `preallocate`: `fcntl(F_PREALLOCATE)` — macOS's fallocate-equiv.
//! First attempt requests `F_ALLOCATECONTIG | F_ALLOCATEALL` (prefer a
//! contiguous run but accept scattered extents to satisfy the full
//! length), falling back to `F_ALLOCATEALL` alone on failure.
//! `F_PREALLOCATE` never advances EOF regardless of the flags — only
//! `ftruncate`/writes grow the file — so the reported file size is
//! unchanged; `F_ALLOCATEALL` governs the contiguity fallback, not size.
//! - `durable_sync`: `fcntl(F_FULLFSYNC)` wrapped in
//! [`crate::io::bounded::bounded_syscall`] with a 60 s deadline.
//! F_FULLFSYNC is HFS+/APFS's true-fsync (flushes the disk's own
//! write cache) — what `fsync` should have been on macOS. Falls back
//! to plain `fsync` if F_FULLFSYNC returns ENOTSUP.
use std::fs::File;
use std::io;
use std::os::unix::io::AsRawFd;
use std::time::Duration;
use crate::io::platform_macos::{
F_ALLOCATEALL, F_ALLOCATECONTIG, F_PEOFPOSMODE, F_PREALLOCATE, Fstore,
};
/// `fcntl(F_FULLFSYNC)` opcode. Documented in `man 2 fcntl` on macOS;
/// not in the `libc` crate as a named constant.
const F_FULLFSYNC: libc::c_int = 51;
pub(super) fn preallocate(file: &File, size_bytes: u64) {
// Clamp to the signed `off_t` range; an unchecked `as off_t` cast
// would wrap a >= 2^63 size to a negative length.
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX) as libc::off_t;
let mut fst = Fstore {
fst_flags: F_ALLOCATECONTIG | F_ALLOCATEALL,
fst_posmode: F_PEOFPOSMODE,
fst_offset: 0,
fst_length: len,
fst_bytesalloc: 0,
};
// First attempt: contiguous.
let mut rc = unsafe { libc::fcntl(file.as_raw_fd(), F_PREALLOCATE, &mut fst) };
if rc == -1 {
// Fall back: drop the contiguous hint, allow scattered extents.
fst.fst_flags = F_ALLOCATEALL;
rc = unsafe { libc::fcntl(file.as_raw_fd(), F_PREALLOCATE, &mut fst) };
}
tracing::debug!(
target: "mux",
"WritebackFile F_PREALLOCATE size_hint={size_bytes} rc={rc} bytes_allocated={} ok={}",
fst.fst_bytesalloc,
rc != -1
);
}
/// ## fd-reuse safety
///
/// The F_FULLFSYNC / fsync runs on a bounded worker thread that may be
/// leaked on timeout. To avoid the leaked worker's syscall hitting a
/// recycled fd number after the original `File` is closed, we
/// `try_clone` an owned `File` and move it into the closure. The clone
/// keeps the underlying file description alive for as long as the worker
/// thread lives. On `try_clone` failure (rare) we fall back to the raw
/// fd integer — no worse than the previous behaviour.
pub(super) fn durable_sync(file: &File) -> io::Result<()> {
// Clone so a leaked worker thread retains a valid fd even after the
// original File is closed and its fd number is reused.
let owned = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
let fd = file.as_raw_fd();
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fd={fd}: try_clone failed ({e}), F_FULLFSYNC worker will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
let fallback_fd = file.as_raw_fd();
match crate::io::bounded::bounded_syscall(
None,
Duration::from_secs(60),
move || -> io::Result<()> {
let fd = owned.as_ref().map(|f| f.as_raw_fd()).unwrap_or(fallback_fd);
// Try F_FULLFSYNC first. If it isn't supported on this
// filesystem (older HFS, some network mounts) fall back to
// plain fsync — better than nothing.
let rc = unsafe { libc::fcntl(fd, F_FULLFSYNC, 0) };
if rc == 0 {
// `owned` (if Some) drops here, releasing the cloned fd.
return Ok(());
}
let err = io::Error::last_os_error();
if err.raw_os_error() == Some(libc::ENOTSUP) {
let rc = unsafe { libc::fsync(fd) };
// `owned` drops here.
if rc == 0 {
Ok(())
} else {
Err(io::Error::last_os_error())
}
} else {
// `owned` drops here.
Err(err)
}
},
) {
Ok(inner) => inner,
Err(crate::io::bounded::BoundedError::Timeout) => {
tracing::error!(
target: "mux",
"WritebackFile::sync_all F_FULLFSYNC timed out after 60s; kernel will flush on close (best-effort)"
);
Ok(())
}
Err(crate::io::bounded::BoundedError::Halted) => Ok(()),
Err(crate::io::bounded::BoundedError::WorkerLost) => Ok(()),
}
}
#[cfg(test)]
#[cfg(target_os = "macos")]
mod tests {
use super::*;
use tempfile::NamedTempFile;
/// Regression for the fd-reuse / use-after-close fix in `durable_sync`.
///
/// Verifies the structural invariant: `try_clone` succeeds for a normal
/// local tempfile, and the cloned `File` has a distinct fd number from
/// the original. This pins the property that a leaked F_FULLFSYNC/fsync
/// worker thread captures an owned `File` (keeping the file description
/// alive) rather than a bare fd integer that can be reused after the
/// original `File` closes.
///
/// The actual fd-reuse race is non-deterministic; a structural test is
/// the accepted substitute.
#[test]
fn durable_sync_worker_uses_owned_clone_with_distinct_fd() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
// try_clone must succeed for a normal local file.
let owned = f
.as_file()
.try_clone()
.expect("try_clone must succeed for a local tempfile");
let clone_fd = owned.as_raw_fd();
// The clone must be a distinct fd (dup'd, not aliased).
assert_ne!(
clone_fd, original_fd,
"owned clone must have a distinct fd number — not an alias of the original"
);
assert!(clone_fd >= 0, "clone fd must be a valid non-negative fd");
// durable_sync must complete without error on the local tempfile.
durable_sync(f.as_file()).expect("durable_sync must return Ok on a local tempfile");
}
}

Some files were not shown because too many files have changed in this diff Show More