Compare commits

...
491 Commits
Author SHA1 Message Date
Matthew Jackson 602e44fde9 v1.4.4: bump version (freemkv-unlock git-pinned for the tag) 2026-07-16 21:44:05 -07:00
Matthew Jackson 622008650c keysource: add DecodeSampleSet — a >=MIN_SAMPLE_UNITS-by-construction sample set so an online request can't be built under-sized 2026-07-16 21:43:02 -07:00
Matthew Jackson c508484451 restore freemkv-unlock path dep for local dev (post-v1.4.3) 2026-07-16 21:06:04 -07:00
Matthew Jackson cf16f66e07 v1.4.3: bump version (freemkv-unlock git-pinned for the tag) 2026-07-16 21:06:01 -07:00
Matthew Jackson adcab75409 aacs: sample only index-1 forensic segments for the online key query; hoist MIN_SAMPLE_UNITS into the base crate 2026-07-16 21:01:16 -07:00
Matthew Jackson 61e4095727 FMTS: resolve the index key map from one forensic keyserver query
The keyserver protocol now returns all 32 index keys as an array for a
forensic content sample (and a single-element array for plain content).
resolve_fmts_key_map sends one forensic batch and maps array element i to
segment index i+1, replacing the per-index blind-probe collection loop
that repeatedly hit the key service. Segment/index parsing and the
aligned-unit content classification are reworked to support this:

- rename variant_select -> index_select (per-index, not per-variant)
- content classification moves to is_clean(buf, ContentFormat) so the
  unit selector emits only units the key service accepts
- segment.rs: parse IndividualSegment.tbl index tags + SPN ranges,
  build contiguous LBA key ranges from the resolved 32-key array
- decrypt/decorator plumbing for the resolved per-index keys

Fail loud (FmtsKeyMissing) when the forensic query returns < 32 keys or
any segment index stays unresolved.
2026-07-16 19:41:44 -07:00
Matthew Jackson 540f84c7d0 restore freemkv-unlock path dep for local dev (post-v1.4.2) 2026-07-15 19:37:54 -07:00
Matthew Jackson 82c4e018d5 v1.4.2: bump version (freemkv-unlock git-pinned for the tag) 2026-07-15 19:37:51 -07:00
Matthew Jackson 08e962694f Mux decrypt/verify redesign, HD DVD first-class, MVC 3D
decrypt:
- decrypt_sectors is now a pure decrypt (apply key, leave plaintext, report
  unverified bytes); TS-structure is a separate primitive (is_clean_ts/ps) used
  only for key selection and read-verify. The mux passes decrypted bytes through
  (the demuxer drops non-conforming packets), ending the NULL-TS conceal loop and
  the per-unit key-server refetch storm. Key-proof floor replaces the 75%
  supermajority.

recovery:
- Removed the post-read decrypt-verify gate (verify.rs) that mis-aligned the
  disc-absolute unit grid against clip-anchored AACS units and false-failed good
  clips (e.g. Dunkirk's orphan-CPS clip). Bad sectors are marked by physical read
  result; decryptability is proven at scan + mux time.

HD DVD (first-class AACS):
- Role-based candidate-list file sourcing so an HD DVD's /ANY!/ files
  (MKBROM.AACS, VTKF000.AACS, CONTENT_CERT.AACS) are found with no disc-type
  branch. parse_vtkf parses VTKF000.AACS into the same UnitKeyFile as a BD
  Unit_Key_RO.inf, so the shared VUK unwrap applies unchanged. set_unit_base
  clip-anchoring. Two decrypt-axis assumptions remain UNVERIFIED-HDDVD-DECRYPT
  (no encrypted disc to test).

mux:
- MVC (Blu-ray 3D) track signals unified into one MVCDecoderConfigurationRecord;
  release-safe track_vint (3-byte VINT) and pid_index (i32) guards.

hardening:
- Container-aware is_clean / encryption detection; bytes_bad_in_title fail-safe
  on a corrupt mapfile; CSS crack gated on DiscFormat::Dvd (HD DVD excluded);
  non-vacuous CSS tests; patch NOT_READY/HARDWARE/ILLEGAL_REQUEST/ABORTED
  sense-path tests.
2026-07-15 19:35:12 -07:00
Matthew Jackson 37e563b8ef Mux: pure decrypt, policy at the caller (no null, no key-server storm)
decrypt_sectors is now a pure decrypt — apply the CPS unit key, leave the
plaintext, report how many bytes did not reach clean TS ("unverified"). It
never restores ciphertext, nulls, or re-fetches. "Did a key produce clean TS?"
is a key-selection / read-verify signal, not the verdict "did we decrypt?": a
correct key can decrypt a bad-encoded region, and broken TS is a muxer concern
(the demuxer drops the packet and resyncs).

Callers own the policy:
- mux (read > decrypt > mux): pass the decrypted bytes to the muxer, whatever
  they are; fail loud only on a genuine can't-decrypt (no key / misaligned).
- sweep/patch (reading from a disc): an unverified unit is a bad read — recover
  a fresh key and retry, or fail loud so disc-recovery re-reads it.

Removes three duplicated decisions — the decrypt-time ciphertext restore, the
mux NULL-TS conceal loop, and the per-unit key-server refetch — plus the dead
aacs_unit_still_ciphertext predicate. Key-fetch recovery now samples the on-disc
ciphertext explicitly (a pure decrypt leaves the buffer plaintext) and lives
only on the rip/verify path, never the mux.

Fixes the 30-90s/region mux stalls and key-server storm on bad-encoded UHD runs
that 1.4.1 left behind (it relaxed the gate but not the surrounding machinery).
2026-07-14 21:04:24 -07:00
Matthew Jackson e659613ab8 restore freemkv-unlock path dep for local dev (post-v1.4.1) 2026-07-14 14:45:33 -07:00
Matthew Jackson eaa7ddc683 v1.4.1: bump version (freemkv-unlock git-pinned for the tag) 2026-07-14 14:45:30 -07:00
Matthew Jackson 723c7edf45 Fix mux concealing decryptable video over a single defective packet
AACS content decryption rejected a whole 6144-byte aligned unit unless
EVERY content packet was conformant MPEG-TS. One authored-bad packet (a
pressing/encoding defect or an AACS 2.1 forensic-variant frame) made the
mux conceal the entire unit as NULL TS — destroying up to 31/32 good
packets and tallying them as loss, surfacing as false "corruption" on
otherwise-clean discs (observed across two UHD titles).

decrypt_unit now asks only "did a key OPEN this unit?" — a padding-aware
>=75% supermajority of content packets restoring their 0x47 sync, a gate
no wrong key can reach (uniform-AES noise floor) yet one that tolerates a
minority of authored-bad packets. Opened units pass through VERBATIM; a
non-conforming packet is left for the demuxer to drop on sync-loss and
resync past. TS-sync conformance is a muxer concern, never a decryption
verdict. The post-read verify/sweep gate now shares the same primitive so
it can never disagree with the mux decrypt.

Also unify the MVC (Blu-ray 3D) track signals: the mvcC CodecPrivate
extension, the BlockAdditionMapping, and each per-frame BlockAdditional
all derive from one MVCDecoderConfigurationRecord built once per track, so
a malformed dependent-view parameter set can no longer orphan a BlockAddID.
2026-07-14 14:43:06 -07:00
Matthew Jackson d9881c57c0 restore freemkv-unlock path dep for local dev (post-v1.4.0) 2026-07-13 19:28:55 -07:00
Matthew Jackson 61b4486ae9 v1.4.0: bump version (freemkv-unlock git-pinned for the tag) 2026-07-13 19:28:52 -07:00
Matthew Jackson 72d39544ac Changelog: 1.4.0 (Blu-ray 3D / MVC) 2026-07-13 12:47:03 -07:00
Matthew Jackson 1e394eaeb8 3D MVC mux: audit round 2 (converged)
Second audit round converged (severity collapsed 6 HIGH -> 1; the one
HIGH was a bounded 32-element scan, not a defect; the sole spec MEDIUM
was the same false-positive re-raised — 0xBF matches ISO/IEC 14496-15
§7.6.2 verbatim). One genuine robustness fix plus coverage:

- extract_mvc_params: skip a zero-length NAL instead of abandoning the
  scan, so a stray length prefix before the subset SPS/PPS no longer
  silently drops 3D signalling. Test proves params after a zero-length
  NAL are still found.
- Tests: parser_for_mvc_dependent routes H.264 to a passthrough parser;
  passthrough with an IDR does not re-assert param sets (the keyframe &&
  !mvc branch).
- Document the per-playlist (not per-clip) is_3d latching as a known
  limitation (real main-feature playlists are uniformly 3D).
2026-07-13 11:39:07 -07:00
Matthew Jackson 4d1aa6bbf7 Harden 3D MVC mux: robustness + tests (audit round 1)
Triage of a 10-lens code audit of the 3D branch. Fixes for real defects;
rejected three spec false-positives that matched the ISO/IEC 14496-15
§7.6.2 record verbatim.

Robustness / correctness:
- Never panic when a title's only video is the MVC dependent view: the
  base is now the first NON-dependent video, so a dependent-only title
  sets up no merge (muxed as an ordinary track) instead of hitting an
  `expect` on the skipped track slot.
- Drop a per-frame BlockAdditional (BlockAddID=2) when the track declared
  no mvcC mapping (dependent params not captured before the header) — a
  plain block keeps the file conforming instead of an orphaned add.
- A non-keyframe MVC base frame always carries a ReferenceBlock (fall back
  to a 0 offset in the pre-first-keyframe corner) so it is never mistaken
  for a seek point.
- Reference the last keyframe on the PRIMARY video track only, so a
  secondary video track's keyframe can't become a cross-track reference.
- dep_by_pts overflow: bound BEFORE inserting so the just-arrived
  dependent survives the drift-clear; count a displaced duplicate-PTS
  dependent as an orphan instead of losing it silently.

API / docs:
- Fold write_frame_with_additional into write_frame(..., Option<&[u8]>)
  per the "no foo_with_X" convention.
- Fix mvc_params doc (StereoMode is intentionally not emitted); remove a
  stale PAT/PMT comment describing an approach that was never taken.

Tests: MVCDecoderConfigurationRecord over-length guards; write_int minimal
two's-complement widths; BlockGroup/BlockAdditions/BlockAdditional +
ReferenceBlock emission; additional dropped without a mapping; h264 MVC
passthrough keeps param sets in-band; extract_mvc_params no-panic on
truncated/empty input; pairing window + dep-overflow edges; no-panic on a
dependent-only title.
2026-07-13 11:24:23 -07:00
Matthew Jackson 9ce03b4d1d Mux Blu-ray 3D (MVC) as a single MVC video track
Fold the MVC dependent (right-eye) view into the base H.264 track as a
per-frame BlockAdditional under an mvcC BlockAdditionMapping, so a 3D
title produces one MVC video track instead of two independent H.264
tracks.

- h264: MVC-passthrough parser mode keeps the dependent view's subset
  SPS/PPS in-band, so each emitted frame is a self-contained dependent
  access unit for a BlockAdditional
- resolve: route the dependent stream through the passthrough parser
- mkvstream: detect the dependent view, pair it to the base frame by
  PTS (bounded FIFO), attach it as a BlockAdditional (BlockAddID=2),
  and skip building its own track; build the mvcC
  MVCDecoderConfigurationRecord from the captured subset SPS/PPS and set
  it on the base track at activation
- mkv: emit the mvcC BlockAdditionMapping and BlockGroup/BlockAdditions,
  with a ReferenceBlock on non-keyframe base frames
- ebml: add BlockAdditions/BlockMore/BlockAdditional/BlockAddID/
  BlockAddIDValue/ReferenceBlock elements and a signed-int writer

Verified against a Blu-ray 3D ISO: ffprobe shows a single MVC track,
the mvcC mapping is present, ~144k BlockAdditionals carry the dependent
view (8.7 GB), and the base view decodes cleanly with no regression.
MVCDecoderConfigurationRecord follows ISO/IEC 14496-15 7.6.2; StereoMode
is intentionally omitted (no enum value describes MVC-in-BlockAdditional;
the mvcC mapping is the primary 3D signal per RFC 9559).
2026-07-13 11:00:04 -07:00
Matthew Jackson f099a4a31a feat(3d): read the SSIF and enumerate the MVC dependent view
Blu-ray 3D as a mux-path variant:
- Detect 3D (a clip has STREAM/SSIF/<clip>.ssif — note the SSIF/ subdir, which
  the old .ssif fallback path got wrong) and use the SSIF extents for the mux:
  one transport stream carrying both the base (left) and MVC dependent (right)
  eyes on distinct PIDs, so muxing it captures the full 3D.
- Add the dependent-view video stream. The base STN table lists only the left
  eye; the on-disc PAT/PMT are AACS-encrypted (unreadable pre-key) and the base
  STN omits the dependent view (it lives in the MPLS STN_table_SS), so use the
  BD-3D PID convention: dependent = base video PID + 1 (0x1011 -> 0x1012),
  coding 0x20 -> H.264. The demux routes its packets from the SSIF.

info now reports two video streams for a 3D title; the mux writes both eyes.
2026-07-12 23:41:47 -07:00
Matthew Jackson e606b5e3a2 feat(3d): enumerate MVC dependent view (stream_type 0x20 -> H.264 video)
from_coding_type maps 0x20 (MVC dependent view) to Codec::H264, so the existing
PAT/PMT scan surfaces the SSIF right-eye substream as a second video stream on
its own PID instead of dropping it as Unknown. No new parser: the dependent eye
rides the same demux path as any other TS video stream. Removes the throwaway
3D-structure probes (findings captured in prior commit messages).
2026-07-12 22:16:13 -07:00
Matthew Jackson de809189ec feat(3d): add MVC dependent-view coding type (0x20)
First step of Blu-ray 3D as a mux-path format variant: name stream_type 0x20
(H.264/MVC dependent view, the SSIF right-eye substream). Recognition in the
scan/demux + dependent-stream enumeration follow.
2026-07-12 22:11:01 -07:00
Matthew Jackson 253b0924e3 wip(3d): prove SSIF de-interleave = ssif−base, dependent decrypts on same UK
Empirical (300, clip 00042): base .m2ts (1567 extents, 25.85 GB) is entirely
inside the SSIF LBA span; dependent = SSIF − base = 1567 ranges, 9.59 GB
(exact vs 35.43−25.85). 200/200 sampled dependent-view units decrypt under the
base UK. => de-interleave needs no SSIF parsing (complement of the base extents
we already resolve), and the dependent eye needs no extra key.
2026-07-12 20:40:58 -07:00
Matthew Jackson 98a7892c34 wip(3d): read-only 3D-structure probe (SSIF/m2ts + MPLS ExtensionData)
Dev tool for the 3D build: uses the library UDF reader (7z cannot parse these
ISOs) to dump base .m2ts vs SSIF sizes and the MPLS ExtensionData entries
(STN_table_SS + SS sub-path). Not product code.
2026-07-12 20:38:37 -07:00
Matthew Jackson b82d21756a fix(udf): follow allocation-descriptor continuation past the AED header
read_icb_extents parsed a continuation block's allocation descriptors from
offset 0, but a continuation block begins with a 24-byte Allocation Extent
Descriptor (ECMA-167 4/14.5): 16-byte descriptor tag + prev-loc(4) +
length_of_allocation_descriptors(4); the real ADs start at offset 24. Reading
from 0 mis-parsed the AED tag as a descriptor -> one garbage extent, then an
unknown-type break, silently truncating any file whose extents spill into a
continuation block.

Few-extent files store their ADs inline and never hit this path, so it stayed
hidden; a heavily fragmented file (~1600 fragments) truncated to 113 extents +
a bogus non-unit-aligned fragment that then aborted the mux
(ExtentNotUnitAligned). Skip the 24-byte AED header, read l_ad bytes of
descriptors from offset 24. Adds a regression test with a non-zero AED tag.
2026-07-12 20:37:55 -07:00
Matthew Jackson 02c584a3d9 restore freemkv-unlock path dep for local dev (post-v1.3.2) 2026-07-10 14:26:11 -07:00
Matthew Jackson 564e66fd24 v1.3.2: bump version (freemkv-unlock git-pinned for the tag) 2026-07-10 14:26:08 -07:00
Matthew Jackson 57d40e81a5 1.3.2: AACS 2.1 FMTS variant-decode foundation
Add UnitKey.variant_number (0 = ordinary, 1..32 = forensic variant) with new/variant constructors, and aacs::variant_select — resolve a disc's single variant and classify each aligned unit (default / variant / drop foreign / conceal keyless). Correct IndividualSegment.tbl: the per-record field is the variant (cycles 1..32 on a retail disc), not a segment number — Segment.number -> Segment.variant.
2026-07-10 14:00:28 -07:00
Matthew Jackson 674819d937 changelog: trim the 1.3.1 relicense note to just the relicense fact 2026-07-10 12:53:17 -07:00
Matthew Jackson c5e8421f9f restore freemkv-unlock path dep for local dev (post-v1.3.1) 2026-07-10 12:35:12 -07:00
Matthew Jackson 7dcf9b5a0d v1.3.1: bump version (freemkv-unlock git-pinned for the tag) 2026-07-10 12:35:09 -07:00
Matthew Jackson f01c097784 1.3.1: relicense to MIT (clean-room CSS + drop copyleft-lib references)
Relicensed from AGPL-3.0 to MIT, effective 1.3.1 (<=1.3.0 remain AGPL). The CSS
content cipher and Stevenson title-key attack are attributed to their published
cryptanalysis (not libdvdcss); all libaacs/libbluray/libdvdread/libdvdnav name
references were dropped from comments while keeping the standard format/spec
descriptions. Also bumps to 1.3.1.
2026-07-10 12:31:19 -07:00
Matthew Jackson c265fa349e hddvd: authoritative title composition from the Advanced-Content playlist
Parse ADV_OBJ/VPLST000.XPL (the real HD-DVD player playlist) with roxmltree
into one DiscTitle per <Title>: its PrimaryAudioVideoClip clips in order (EVO
via the .MAP sidecar), the titleDuration, the displayName, and the ChapterList.
A layer-break split (FEATURE_1+FEATURE_2, feature/feature_Divide) composes into
ONE title with the two parts as clips, each carrying its title-time in/out
points (45kHz ticks) for seamless-join splicing. Falls back to the clip-name
heuristic when no playlist is present.

Fixes unknown runtimes (real durations), poor names (FEATURE -> 'Main Movie'),
and gives authoritative composition + chapters. Validated on ANCHORMAN (97m),
SHAUN (99m), HARRY_POTTER (152m).
2026-07-10 10:28:18 -07:00
Matthew Jackson 6a59dcb7ba ci: drop the dead crates.io publish job (libfreemkv is git-tag-only)
libfreemkv is publish=false (it git-deps the firmware crate freemkv-unlock,
never on crates.io), so the Release workflow's 'cargo publish' step failed
hard on every tag. Consumers git-tag-pin libfreemkv; the git tag is the
release artifact. Removed the publish job.
2026-07-10 08:52:57 -07:00
Matthew Jackson 0eb1817ec7 restore freemkv-unlock path dep for local dev (post-v1.3.0) 2026-07-10 08:43:47 -07:00
Matthew Jackson 21f3b6adbd v1.3.0: bump version (freemkv-unlock git-pinned for the tag) 2026-07-10 08:43:42 -07:00
Matthew Jackson b32d636c64 audit: void empty-timeline duration, cover sniff overlap
Round-11 findings from the 10-phase release audit (no real HIGH):

- When a no-declared-duration source (HD-DVD) muxes a degenerate single
  frame at tick 0 with no per-frame duration, max_block_ticks stays 0 and
  the reserved DURATION placeholder was left as a literal 0.0 (players read
  that as a zero-length file). Void the element instead, so the Segment
  omits DURATION as an unknown-duration source did before the back-patch.
- Add a regression test for the sniff_video_codec overlap fix (a
  picture_start_code whose payload begins 00 00 followed by a real start
  code) so the i+=4 marker skip can't silently regress to i+=3.
2026-07-09 20:40:54 -07:00
Matthew Jackson d877630c45 audit: lock DTS rate table, fix sniff overflow-scan, cover decrypt loss
Round-10 findings from the 10-phase release audit:

- A finder claimed the DTS SFREQ→rate table was wrong at 11/12; verified
  it against ffmpeg's avpriv_dca_sample_rates (12k/24k/48k/96k/192k at
  11-15) — the table is CORRECT. Added a test that locks the full table so
  it can't be mis-"fixed".
- sniff_video_codec advanced 3 bytes after a matched start code, re-reading
  the code byte as an overlapping start code; skip the full 4-byte marker.
- Guard the HD-DVD next_id title counter with saturating_add so a crafted
  disc with >65536 clips can't overflow (panic in debug).
- Add a test that an undecryptable unit (DecryptFailed) is zero-filled and
  counted as loss through ExtractResult (complete=false, bytes_lost>0) —
  the recovery-seam consolidation folded that bucket into bytes_unreadable.
2026-07-09 20:17:12 -07:00
Matthew Jackson 1895095f49 audit: drop dead DTS marks cap, lazy passthrough buf, doc corrections
Round-9 findings from the 10-phase release audit (no HIGH):

- Remove the MAX_PTS_MARKS backstop and its tautological test: an empty
  DTS PES returns before recording a mark, and a non-empty run is already
  bounded by the MAX_AU_BYTES buffer clear (which clears pts_marks) — so
  the deque cannot grow unbounded and the cap was dead code.
- AuAssembler::for_codec no longer reserves 256 KiB for a Passthrough
  stream (audio/subtitle, and every TS/BD stream) whose buf is never
  written; only the reassembling modes reserve.
- Correct the scan comment that claimed region is computed (it is a
  Region-free stub until region detection lands) and drop a public-repo
  reference to internal "private refactor notes" in the mkb module doc.
2026-07-09 19:50:54 -07:00
Matthew Jackson 5f8c049444 audit: clamp BD format fallback, running GOP byte counter, O(1) DTS marks
Round-8 findings from the 10-phase release audit:

- detect_disc_format's BDMV fallback passed detect_format's result
  through unchanged, so an SD bonus/menu title could tag a BD-tree disc as
  DVD (mis-sizing the ECC sweep) — violating its own "never below Blu-ray"
  invariant. Clamp anything but UHD up to Blu-ray.
- Track the MPEG-2 GOP byte total incrementally instead of re-summing the
  whole gop_buf on every pushed picture (was O(pictures²) on any MPEG-2
  disc, not just adversarial input).
- Back the DTS pts_marks deque with a VecDeque so the over-cap prune is an
  O(1) pop_front, not an O(n) Vec::remove(0).
- Add a test exercising parse_stream_id_extension's PTS/DTS skip branches
  (the real AU-opening 0xFD video PES path) — previously untested.
2026-07-09 19:27:01 -07:00
Matthew Jackson 7d427d0b17 audit: bound DTS marks, align disc-format tree order, doc/test cleanups
Round-7 findings from the 10-phase release audit (no HIGH; convergence):

- Cap DtsParser.pts_marks (MAX_PTS_MARKS): a run of zero-length timed PES
  packets grew no buffer bytes, so the drain_front mark-prune never ran —
  the deque could accumulate without bound on hostile PS input.
- detect_disc_format tested HVDVD_TS before BDMV while the title-scan
  dispatch tests BDMV first, so a disc with both trees would be classified
  HD-DVD but enumerated as Blu-ray. Align both to BDMV → HVDVD_TS →
  VIDEO_TS.
- Document why the DTS new-PES re-base can emit a locally-decreasing PTS
  (the muxer's block_ts applies the strictly-monotonic audio nudge, tested
  in mkv.rs) — this is by design, not a mux defect.
- Fix stale aacs/keys.rs comment references (functions moved to
  aacs/inf.rs / aacs::resolve/derive in the module split).
2026-07-09 19:01:56 -07:00
Matthew Jackson 0a47b22db5 audit: byte caps on GOP buffers, opener-scan resume, honest video codec
Round-6 findings from the 10-phase release audit:

- Wire the documented MAX_PENDING_BYTES byte cap into the MPEG-2 GOP
  buffer (it was dead code) and add an equivalent MAX_GOP_BYTES cap to the
  sparse-PTS reorder, so a crafted stream of few-but-huge access units
  cannot over-allocate — both were bounded only by frame count before.
- probe_evo_streams defaulted an unsniffable HD-DVD video stream to H.264,
  which mis-parses a VC-1 (or still-encrypted) clip into a corrupt track.
  Emit the video stream only when the codec is actually identified — the
  honest outcome, matching the audio path (a real clear clip always carries
  its sequence header at the head).
- Resume the AU-opener search from a cursor (like the boundary search), so
  a long unsynced junk run is O(bytes), not O(buffer) per push.
- Mark mpeg2's now-dead MAX_AU_BUFFER test-only; restore #[doc(hidden)] on
  the aacs probe harness module.
- Add regression tests: the 0xFD video-routing guard, the FMTS-is-UHD key
  state, and the GOP byte caps.
2026-07-09 18:31:47 -07:00
Matthew Jackson 2ebcb60059 audit: guard 0xFD video routing, carry frame duration, add cap tests
Round-5 findings from the 10-phase release audit:

- collect_es routed EVERY extended-stream-id (0xFD) PES into the video ES
  buffer, so a 0xFD HD-audio sub-stream (MLP/TrueHD) could pollute the
  video sample and — if it preceded the video PES — stamp the video track
  with the audio PID, losing the video. Only the VC-1 extension (0x55) is
  now treated as video; routing 0xFD audio to its own track is deferred to
  the HD-DVD program-chain follow-up.
- The sparse-PTS reorder now carries its calibrated per-frame duration onto
  each frame, so the muxer emits a BlockDuration and the back-patched
  Segment Duration covers the final frame instead of understating it.
- Add regression tests for the MAX_MARKS and MAX_VTI_HITS caps (promote
  MAX_VTI_HITS to module scope); make the differential-test factory array a
  named type; drop an identity-op in a reorder test.
2026-07-09 17:59:04 -07:00
Matthew Jackson f46d99a348 audit: fix AU mark-field loss, VTI tie determinism, and mark/perf issues
Round-4 findings from the 10-phase release audit (the first fully clean
round; it dug into the new #22/#18 refactor code):

- AuAssembler closed each AU from only the FRONT mark's fields, so when
  one PES fragment carried the source and a later fragment of the same AU
  carried the PTS, the second field was dropped — a regression vs the old
  separate pts/source mark deques. Now merge the first Some of each field
  across all in-range marks.
- parse_vti_clip_order picked the largest residue bucket with
  HashMap::into_values().max_by_key(), nondeterministic on a size tie
  (randomized HashMap iteration) — could select a different clip table
  run-to-run. Break ties by smallest offset.
- Bound the marks/disc_marks deques (MAX_MARKS): the buf-size cap prunes
  marks only when bytes accumulate, so a run of zero-length timed
  fragments could grow them without bound on hostile input.
- Add push_owned so the PS path moves the PES payload into a passthrough
  AU with no copy (MPEG-2 video + all audio), removing a per-PES
  malloc+memcpy the refactor had introduced on the DVD path.
- Back-patch the MKV duration from the block END (start + its own
  duration) so it covers the final frame instead of understating by one.
- Add direct tests for the MKB record-framing walker; drop a stale
  drain_complete_aus doc comment left on process_au.
2026-07-09 17:30:41 -07:00
Matthew Jackson 4d93d90561 docs: drop Renesas from changelog (detection-only stub) 2026-07-09 16:36:47 -07:00
Matthew Jackson 4eeafd3378 docs: soften HD-DVD changelog to partial support (PGC parse deferred) 2026-07-09 16:33:40 -07:00
Matthew Jackson a73ba98f82 mux: resume AU-boundary scans from a cursor (O(n) not O(n²))
AuAssembler::drain rescanned the whole buffered access unit from a fixed
offset on every push, so reassembling one AU split across N program-stream
PES fragments cost O(bytes²/fragment) — amplified on the HD-DVD PS path
where H.264/HEVC/VC-1 frames are large and now flow through this shared
assembler (unlike the TS path, which delivers one AU per PES).

Carry a scan_pos cursor (and, for the stateful VC-1/MPEG-2 rules, a
seen_unit flag) so each push resumes the boundary search where the last
one stopped instead of restarting. Total scan work for one AU is now
O(AU bytes). The from-scratch scanners are retained as a #[cfg(test)]
oracle; a new differential test asserts the resumable path yields
byte-identical AUs at every fragment granularity for all three modes.
2026-07-09 16:22:54 -07:00
Matthew Jackson 7676f3f2f6 audit: bound the VTI clip-table scan; fix stale aacs doc links
Round-2 findings from the 10-phase release audit:

- parse_vti_clip_order bucketed hits by residue with an O(stride*hits)
  rescan and no hit cap, so a crafted HD-DVD VTI packed with millions of
  `.EVO` tokens (up to the 64 MiB UDF read cap) could burn seconds of CPU
  on a routine scan. Bucket in a single O(hits) pass and cap collected
  hits at MAX_VTI_HITS (a real table holds a few dozen).
- Fix the stale `super::keys::…` intra-doc links left by the aacs module
  rename: the referenced fns live in `super::derive`.
2026-07-09 14:41:49 -07:00
Matthew Jackson e1cfe59dc9 audit: cap the sparse-PTS reorder buffer, FMTS key state, zero KCD
Round-1 findings from the 10-phase release audit:

- SparsePtsReorder buffered its current GOP with no bound, draining only on
  a keyframe — an open-GOP or crafted program stream that never signals one
  could hold the whole title in RAM. Force-complete the GOP at
  MAX_GOP_FRAMES, matching the MPEG-2 parser's backstop.
- inject_unit_keys labelled a 2.1 FMTS disc as AACS 1.0 / bus-encryption
  off; FMTS is UHD-family, so synthesize the UHD version + bus encryption.
- The compiled Key Correction Data was a non-zero 16-byte constant fed into
  the Media Key derivation. Per the no-compiled-keys rule it is now all-zero;
  the chain still cannot complete on a real disc (documented), so this is
  behaviour-neutral — all variant tests pass unchanged.
- Fix stale doc references (broken `super::variants` intra-doc links, and
  `aacs::keys` comments) left by the module rename.
2026-07-09 14:14:20 -07:00
Matthew Jackson c43f23a759 mux: back-patch the MKV duration from the timeline when the source has none
A title whose scan yields no duration (HD-DVD — its `.MAP` timemaps are
not parsed, so DiscTitle.duration_secs is 0) produced an MKV with no
Segment Duration element, so players/MediaInfo reported an unknown
runtime.

Reserve a DURATION placeholder when the source declares none, track the
highest block timestamp written, and back-patch the placeholder at
finish() with the real muxed runtime (also enabling the per-track BPS
tags for these titles). Gated on duration_secs == 0, so BD/UHD/DVD —
which carry a real mpls/IFO duration — write it up-front exactly as
before, unchanged.
2026-07-09 12:52:48 -07:00
Matthew Jackson 770e0ab936 docs: changelog for HD-DVD feature composition + MPEG-2 AuAssembler 2026-07-09 09:12:56 -07:00
Matthew Jackson 5557765a86 disc: compose the HD-DVD feature from its VTI clip table
HD-DVD Standard Content splits the main feature across clips at the
layer break (FEATURE_1/FEATURE_2, or feature/feature_Divide). The scanner
enumerated one title per .evo, so main-title selection picked only part 1
(e.g. Shaun's 11 GB FEATURE_1, missing the 6.8 GB FEATURE_2).

Parse the HVA*.VTI navigation file's clip table — a fixed-stride record
list naming every clip in authored order, isolated by residue-mod-stride
rather than the imprecise header pointer — and concatenate the feature
clips (matched by the feature* naming convention) into one title whose
extents run in authored order. Every other clip stays its own title.
Falls back to one-title-per-clip when the VTI is absent or unparseable,
so nothing regresses on a disc with no readable navigation.

Validated on real discs: Shaun 17.8 GB / Anchorman 20.1 GB / Harry Potter
24.4 GB now enumerate as one 2-clip FEATURE title (largest = the movie).
2026-07-09 09:12:13 -07:00
Matthew Jackson 458b04ae63 mux: reassemble MPEG-2 access units via the shared AuAssembler
The MPEG-2 parser hand-rolled its own PES reassembly — a byte buffer plus
parallel PTS / source / discontinuity mark queues keyed by absolute
offset — duplicating what AuAssembler already does for H.264/HEVC/VC-1.

Add a Mode::Mpeg2 to AuAssembler (picture 0x00 with preceding sequence
0xB3 / GOP 0xB8 headers — the same headers-precede-picture shape as the
VC-1 mode) and have the MPEG-2 parser own one via AuAssembler::mpeg2().
parse() now feeds fragments to the assembler and processes each complete
access unit; the buffer, base offset, and three mark queues are gone. The
GOP-buffered temporal_reference reorder and PTS origin-locking are
unchanged. The parser's external contract is unchanged, so all existing
MPEG-2 parser tests pass as-is; new AuAssembler tests cover the MPEG-2
boundary rule directly.
2026-07-09 08:44:01 -07:00
Matthew Jackson c31ac6b66b docs: changelog for HD-DVD VC-1, sparse-PTS, variant layout (1.3.0) 2026-07-09 08:21:09 -07:00
Matthew Jackson 3e84ecfe42 mux: reconstruct display-order PTS for sparse-PTS program streams
HD-DVD EVO (and DVD VOB) program streams timestamp video at GOP
granularity: only one access unit per GOP carries a PES PTS. The H.264 /
HEVC / VC-1 parsers collapsed a missing PTS to 0, so on such a source
every non-anchor frame landed on the same block timestamp and a decoder
reported "non monotonically increasing dts".

Add a shared SparsePtsReorder that rebuilds a display-order PTS per frame
from the coded picture type (I/P/B) plus the sparse anchor PTS, with a
per-frame duration self-calibrated from the spacing between consecutive
GOP anchors (no external frame-rate needed). Display order is derived via
the classic single-anchor-delay rule (an anchor displays only after the
previously-held anchor; a B displays immediately), exact for the
non-hierarchical GOP structures HD-DVD H.264/VC-1 use. It mirrors the
MPEG-2 parser's GOP-buffered origin-locking.

Gated to the program-stream path only: the three parsers enable it via
with_ps_reorder(is_dvd_ps), so the BD/UHD transport path (per-frame PTS)
is byte-identical and untouched.
2026-07-08 21:54:09 -07:00
Matthew Jackson 998f82dc07 aacs(2.1): pin variant record layout against reference MKBs
Correct the variant Media Key chain against two real variant MKBs
(Zombieland v70, Stand By Me v70): C for Kmp is the per-slot block of
the 0x0c cvalue table indexed by the matched subset-difference, not the
0x2d head. The 0x2d record is the VARIANTS table (leading body-16 bytes,
one big-endian u16 per subset-difference) followed by a trailing 16-byte
Nonce, with no leading header. VKD stays at 0x2f.

Route record-type selection through the named REC_* consts in mkb rather
than bare hex, and document that Key Correction Data is per-licensee: no
universal constant exists and none is compiled in, so on a real disc the
chain yields a wrong Media Key that the Verify-Media-Key gate rejects
rather than emitting a bad key. This is a key-acquisition gap, not a
code gap.
2026-07-08 21:01:57 -07:00
Matthew Jackson b80af33775 mux: HD-DVD VC-1 demux via extended stream id 0xFD
VC-1 HD-DVDs (e.g. Shaun of the Dead) carry video on MPEG-PS extended
stream id 0xFD, with the real stream selector in stream_id_extension
inside the PES extension. Parse that field so the video routes to a
distinct track (pid 0xFD00|ext) instead of being dropped.

Reframe VC-1 access units in AuAssembler with a dedicated Mode::Vc1:
an AU is delimited by the next frame BDU (0x0D) once a frame has already
been seen, so the sequence (0x0F) and entry-point (0x0E) headers that
precede an I-frame stay attached to the frame they describe. The old
single-start-code split stranded those headers on the prior AU, which
the decoder reported as bits-overconsumption and hard decode failures.

hddvd probe now tracks the video pid it detects and emits VC-1 on 0xFD.
2026-07-08 21:01:48 -07:00
Matthew Jackson 054902cefe release: prepare 1.3.0
Bump to 1.3.0 and add the 1.3.0 changelog entry (FMTS/HD-DVD formats, AACS 2.1 variant chain, recovery seam, aacs module split, main-title-by-size, and the fixes since v1.2.2). Not tagged or pushed.
2026-07-08 14:48:50 -07:00
Matthew Jackson aa277224a9 fix: assorted correctness fixes and dead-code cleanup
- aacs/resolve: a media-keys-only provider missing the VID classifies as
  VidUnavailable, not NoMaterial (an MK derives the VUK once the VID
  arrives).
- disc/bluray: mark a clip seen only after its .clpi parses, so a
  transient parse failure on the first PlayItem cannot suppress the
  clip's extents for a later PlayItem referencing it that succeeds.
- disc/patch: log rather than swallow mapfile record/flush failures on a
  reverify downgrade, so a failed persist cannot silently mismark a bad
  unit good on resume.
- mux/ts: flag a discontinuity when a partial PES is dropped, matching
  the other partial-drop paths.
- mux/demux_thread: the no-demuxer branch forwards an empty batch for
  early consumer-disconnect detection instead of reading the whole disc.
- io/pipeline: correct the send-timing log (as_secs_f64, not as_micros
  printed as ms).
- aacs/derive, aacs/variant, disc/read_error, keysource: comment/doc
  accuracy. sector/prefetched, udf: remove dead fields/functions.
- mux/disc: assert unit-aligned read counts in the test.
2026-07-08 14:44:15 -07:00
Matthew Jackson 26154b6977 sector: generic recovery seam; FMTS forensic segments as decrypt loss
Replace the AACS-specific inline key-fetch in the decrypt decorator with
a scheme-neutral recovery seam: the input stream (L3) installs a Recover
closure (none / AACS key-fetch) and the decorator (L2) runs it at the
single decrypt-miss point. FMTS (AACS 2.1) forensic-segment units that no
key opens are just undecryptable units, concealed and counted as ordinary
decrypt loss with no FMTS-specific branch ("a loss is a loss"), so the
separate bytes_undecryptable bucket collapses into one loss count.

- sector/recovery.rs: the seam (MissOutcome, none/key_fetch factories),
  naming no encryption scheme in its type.
- FMTS: segment routing primitives + BYPASS_FMTS_KEY, and an upfront
  ensure_forensic_segments_decryptable gate (Error::FmtsKeyMissing) in
  the mux input path, parallel to the unit-key gate.
- CSS descramble/rekey moves from decrypt_sectors into
  css::descramble_region: CSS self-recovers from the data itself, so it
  stays OFF the seam (which is only for external inputs).
- disc/mod.rs also: main-title selection aligned to largest physical
  size; is_regular read from the open file handle, not metadata(path),
  fixing a swallowed sync_all on a fresh-rip ISO. decrypt_threads()
  resolved once via OnceLock off the per-buffer hot path.
2026-07-08 14:44:03 -07:00
Matthew Jackson bb90991e05 labels: reader-backed detection, Criterion fix, menu-language fallback
- DetectFn now takes a SectorSource so a parser can inspect a jar's
  central directory in detect() instead of firing on "any BD-J jar".
  dbp/deluxe do the real com/<vendor>/ prefix check up front, so each
  claims only its own discs (foundational for scaling the registry).
- criterion: treat a stream-map value of 0 as unmapped and synthesize a
  real 1-based number, so a 0 can't shadow or collide with a genuine
  stream 1 (with regression tests).
- png_filenames: new Low-confidence, last-resort parser reading menu
  language from {title}_UHD01_{LANG}_Composite artwork; sits below the
  MPLS floor so a real framework parser always wins.
- vocab: add menu_lang() for 639-2/B to 639-2/T menu-token normalization.
2026-07-08 14:43:52 -07:00
Matthew Jackson 175f131008 aacs: parse FMTS SegmentKey.tbl container (16-bit variant-indexed)
New aacs::segment_key: parses the AACS 2.1 SegmentKeyNNNNN.tbl container, confirmed against a retail disc as an 8-byte header + 65536 records of 536 bytes, indexed by the 16-bit variant selector the Media Key Variant chain produces. This is the confirmed link between the two 2.1 variant layers (selector picks the device's per-segment variant). The per-record 528-byte payload layout is not yet reversed.
2026-07-07 19:18:33 -07:00
Matthew Jackson af2f750629 aacs: parse FMTS forensic segment map (IndividualSegment.tbl)
New aacs::segment: parses the AACS 2.1 IndividualSegment.tbl into the source-packet ranges of the forensic variant segments (validated against a retail disc: 792 segments, 2560 packets each). First piece of the FMTS variant decoder — the segments' variants are encrypted under segment keys, not the unit key, so a unit-key rip corrupts them (broken HEVC refs).
2026-07-07 18:59:09 -07:00
Matthew Jackson f1517f7313 disc: select main title by largest size, not clip count
canonical_title_order keyed on clip-count ascending before duration, so a short 1-clip bonus reel outranked the real feature — which is often chaptered into many clips (one per chapter). Make physical size the primary key: the main feature is the largest non-oversize title; decoy 'play-all' playlists run long but tiny (reused clips) and self-eliminate.

Validated across 23 UHD/BD discs: fixes t1 on F9, Fast Five, Fast & Furious 6, and Furious 7 (feature was ranked #13-36); no regressions on the 19 already correct.
2026-07-07 18:36:36 -07:00
Matthew Jackson 984774b068 disc: first-class FMTS + HD-DVD formats; CPI sample selection
Add DiscFormat::Fmts (AACS 2.1) and DiscFormat::HdDvd as first-class peers. Format derives from the AACS MKB generation (mkb_type().generation(): V10=BD, V20=UHD, V21=FMTS), reusing existing AACS code, and from the on-disc tree for HD-DVD/DVD. One detector (detect_disc_format) shared by the coarse DiscId probe and the full scan — no more 'default BluRay, defer to full scan'.

FMTS is a BD-tree stream variant: parse_playlist resolves the clip stream via CLIP_STREAM_EXTS (.m2ts -> .fmts -> .ssif), so the .fmts main feature yields real extents (previously silently empty). HD-DVD is a tree-level peer with its own enumerator (disc/hddvd.rs): HVDVD_TS/*.evo -> MpegPs titles with real extents (playlist/stream parsing honestly stubbed).

Sample selection for key resolution now uses the authoritative AACS CPI flag (aacs_unit_encrypted, byte-0 & 0xC0) not the ts_sync_destroyed heuristic — container-agnostic (M2TS/FMTS/EVO; TS-sync is meaningless on HD-DVD program streams) and stops the key-service '0 encrypted units' rejection.

Tests live with each format (bluray/hddvd/mod); generic UDF fixture builders extracted to a shared udf::fixture module.
2026-07-07 18:16:04 -07:00
Matthew Jackson ddc44e1105 unlock_bridge: run_features/run_bus dispatch; report LibreDrive vs Renesas
Adopt freemkv-unlock's split Unlocker trait: run_features drives the drive-prep capability, run_bus the content bus removal, each iterating unlockers until one doesn't decline (NotApplicable = try next; Ok or a real error stops).

unlocker_matrix now reports which drive-prep unlocker actually ran — LibreDrive removes the bus at the drive; Renesas unlocks features but leaves the bus to the cert. Wire product_id through to fu::DriveId.

Bump to 1.2.3.
2026-07-07 16:36:05 -07:00
Matthew Jackson b8a51ef1c7 aacs: remove boil veneer; move key newtypes to aacs::types
aacs::boil was a thin newtype veneer over aacs::derive — a duplicate layer.
Delete it: move the shared key newtypes (Vid, MediaKey, Vuk, ProcessingKey,
UnitKey) into aacs::types, and expose resolve_candidate + KeyCandidate from
aacs::derive directly. Downstream (keysource, disc::encrypt) now import from
aacs::{derive,types}. Pure API consolidation, no behaviour change; full test
suite green on Rust 1.86.
2026-07-05 12:09:39 -07:00
Matthew Jackson 08cb80aac1 aacs: keep flat re-exports for the typed key primitives
The module structure is the public API, but the typed key primitives
(MediaKey/UnitKey/Vid/Vuk, mk_from_dk/mk_from_pk/uk_from_vuk/vuk_from_mk),
derive_vuk, the aligned-unit decrypt entry points, and DeviceKey/HostCert
are load-bearing names that downstream key-source crates import through the
flat `aacs::` path. Re-export them here so those crates can track the
module refactor without a lockstep re-pin.
2026-07-04 22:19:39 -07:00
Matthew Jackson 60d0ebc8a1 aacs(2.1): variant Media Key as a clean PK -> Km primitive
Redesign derive_media_key_variant to the minimal derivation surface:

  derive_media_key_variant(mkb_records, pk) -> Km

- PK-input only. Deriving Kp from device keys (DK -> PK) is a separate
  concern (walk_processing_key); a leaked 2.1 key is a PK, and the chain
  starts at Kp. A bare PK arrives without its subset-difference slot, so
  the primitive tries it against every slot and returns the Km for the
  slot that passes the MKB's Verify-Media-Key record -- mirroring the
  classical bare-PK derive_media_key_from_pk, gated by the chain's own
  verify so an unverified key is never returned.
- VID-free: the Media Key is MKB-scoped. VUK stays the separate
  derive_vuk(Km, VID) step.
- KCD is a fixed algorithm constant compiled in, not a caller parameter
  (removes the kcd argument, the placeholder const, and KcdNotProvided).
- Soft-correction / online-challenge slots are treated as non-covering;
  surfaced over the generic miss so a disc needing those modes is
  distinguishable from a non-covering key.

resolve_keys_v21 updated to walk DK -> PK first, then call the primitive
and derive the VUK from Km + VID. Module + helper docs refreshed to the
pinned record layout; tests reworked for the PK-only signature.
2026-07-04 19:16:40 -07:00
Matthew Jackson 5dee947007 aacs(2.1): wire variants_for_uv from the 0x2d record; chain runs end-to-end
variants_for_uv previously returned None (placeholder), dead-stopping the
Media Key Variant chain at VariantsTableUnavailable on any real disc. Layout
now pinned against two real 2.1 variant MKBs (Zombieland v70, Stand By Me v70):
the 0x2d Encrypted-Media-Key-Variant-Data body is sd_count u16 VARIANTS entries
(1:1 with 0x0c cvalues / 0x04 subset-diffs) followed by the 16-byte per-disc
Nonce at the tail. variants_for_uv reads the sd_slot_index-th u16.

With this the chain runs fully: Kmp -> Kpnew(=Kmp^KCD, extracted CyberLink
constant) -> VKD(0x2f) -> Km -> Kvu. The only remaining input is a covering
2.1 Processing Key to validate against a known answer; until then the final
Verify-Media-Key (0x86) gate rejects any wrong layout pick, so a bad key can
never be emitted. Tests updated to reflect the wired lookup; 1.86 precommit
green (fmt+clippy+tests).
2026-07-04 18:26:09 -07:00
Matthew Jackson a1796e8345 aacs: point integration tests at real module paths
The facade removal (5cf4252) left tests/ calling aacs::disc_hash,
aacs::decrypt_unit, aacs::AacsVersion, etc. at the old flat paths,
so the branch did not compile its integration tests. Repoint each
to its real module (inf/derive/content/mkb). No logic change.
2026-07-04 18:25:59 -07:00
Matthew Jackson 5cf4252e44 aacs: remove the mod.rs facade — module paths are the public API
Delete the pub-use re-export facade; consumers now import from the owning
module (aacs::content::decrypt_unit, aacs::mkb::MkbType, aacs::derive::derive_vuk,
aacs::boil::mk_from_dk, aacs::resolve::resolve_keys_v2, ...). Internal callers
repointed accordingly. Path-only change; logic hash identical (95fb9924); 2210
tests green.
2026-07-04 14:19:40 -07:00
Matthew Jackson b5f9c4e426 aacs: merge media_key + volume_key into derive.rs
The volume_key module was only 34 lines and is just the tail of the same
DK/PK -> MK -> VUK -> UK derivation ladder as media_key. Fold both into one
derive module so every aacs module is a substantial, distinct responsibility
(crypto/mkb/derive/inf/content/variant/resolve). Relocation only; logic hash
identical (95fb9924); 2210 tests green.
2026-07-04 14:14:56 -07:00
Matthew Jackson 82dca307ad aacs: split keys.rs god-module into media_key/volume_key/inf/resolve
Break the 2800-line keys.rs into four responsibility-scoped modules:
- media_key.rs: DK/PK -> Media Key subset-difference walk (+ probe harness)
- volume_key.rs: VUK derivation, unit-key unwrap
- inf.rs: Unit_Key_RO.inf parsing, disc_hash, content cert, in-drive MKB read
- resolve.rs: the resolve_keys_* orchestration (keys.rs renamed)

Relocation only; the (white-box) test suite stays in resolve.rs and pulls
the moved items via glob imports. Proven byte-identical to the pre-refactor
state via the logic hash (95fb9924); 2210 tests green.
2026-07-04 14:06:57 -07:00
Matthew Jackson a9a21d8139 aacs: extract mkb.rs (MKB record format, MkbType, finders)
Relocate the shared MKB machinery into a single mkb module: the record
framing walker + MkbRecord view (from variant), the MkbType/AacsVersion
classification, the MKB-file utilities, and the record-body finders (from
keys). Fixes the inversion where the MKB parser lived in the 2.1-only
variant module. variant.rs keeps its local MkbRecord-based mkb_find_mk_dv
(name collision with the raw one; unified in the dedup follow-up).

Relocation only. Proven byte-identical to the pre-refactor state via the
function-body logic hash (imports normalized out); 2210 tests green.
2026-07-04 13:55:06 -07:00
Matthew Jackson 8dc7075356 aacs: extract crypto.rs (shared AES primitives + constants)
Relocate the shared low-level primitives into a single crypto module:
aes_ecb_encrypt/decrypt, aes_cbc_decrypt, aes_g (from content/variant) and
aesg3 + AESG3_SEED (from keys), plus AACS_IV. Fixes the scatter where AES-G
lived in the 2.1 file and AES-G3 in keys. Relocation only — no rename, no
logic change (logic-hash identical to baseline; 277 items; 2210 tests green).
2026-07-04 13:37:53 -07:00
Matthew Jackson 02218a6511 aacs: rename decrypt.rs->content.rs, variants.rs->variant.rs (no logic change)
Pure file+module-path rename. 'content' names the AACS unit-decrypt layer
(distinct from the top-level sector-decrypt driver crate::decrypt), and
'variant' (singular, spec term 'Media Key Variant') names the 2.1 chain.
Logic-hash identical to baseline; 277 items intact; tests green.
2026-07-04 13:24:26 -07:00
Matthew Jackson e32a3af898 aacs: annotate crypto with AACS spec section citations
Add [C]/[PR]/[BD]/[libaacs] §x.y provenance markers across the AACS
crypto so each primitive links to the spec section it implements, with a
source-tag legend in mod.rs. Doc-comments only — no logic, constant, or
signature changes.

Also: correct two stale record-type comments in variants.rs (0x82/0x83 →
the real 0x2d/0x2f) and document the Variant Number width (spec lsb_10 vs
the 2.1 chain's lsb_16, driven by the 65,535-entry VKD table).
2026-07-04 13:15:53 -07:00
Matthew Jackson 1253db037f restore freemkv-unlock path dep for local dev (post-v1.2.2) 2026-07-03 20:50:17 -07:00
Matthew Jackson 4b878dcc06 v1.2.2: bump version (freemkv-unlock git-pinned for the tag) 2026-07-03 20:50:14 -07:00
Matthew Jackson 7511b0b808 changelog: 1.2.2 (AACS 2.1 variant MKB, mk_from_dk fix, PK-path perf, autorip online-outage + move-error clear) 2026-07-03 20:48:04 -07:00
Matthew Jackson 707989bd68 aacs: target real AACS 2.1 Variant MKB records (0x2d/0x2f/0x0c)
The Media Key Variant scheme is detected and parsed via the actual MKB
record types found on a real variant disc:
  - 0x2d  Encrypted Media Key Variant Data (C)
  - 0x2f  Variant Key Data table (65,535 x 16)
  - 0x0c  variant cvalues (one per 0x04 subset-difference slot)

Replaces the earlier placeholder 0x82/0x83 record types, which were a
guess and appear on no real MKB. is_variant_mkb, the record finders, and
the subset-difference cvalue source (now 0x0c, falling back to 0x07/0x05)
are updated accordingly, along with the V20->V21 upgrade detection in
resolve_keys_v2 and its fixtures.

The variant chain still halts at variants_for_uv (the VARIANTS[uv] /
Nonce sub-field offsets need a covering key to confirm end-to-end
against the 0x86 verify), so a best-effort offset is never silently
trusted. All 27 variant tests pass on Rust 1.86.
2026-07-03 13:25:01 -07:00
Matthew Jackson 57b2f0c4f9 aacs: make the PK path terminal (no SD-tree walk)
A Processing Key is the key at its Subset-Difference node — one AES-G from
the Media Key — so it is tried directly against the MKB cvalue tables,
matching libaacs _calc_mk_pks (iterate PKs × cvalues). The prior code
treated every PK entry as a device-node label at unknown depth and BFS-walked
the SD tree (depth 3, capped 5), which was both wrong for terminal PKs and
~15x slower on a large UHD MKB (~181k cvalues): PK derivation on UHD dropped
from ~37s to ~2.4s.

The Subset-Difference tree walk now lives solely in the device-key path
(derive_media_key_from_dk), which owns per-node path bits; the PK path never
descends. Removed PK_WALK_MAX_DEPTH / _CAP / _walked / walk_pk_against_tables_impl;
renamed the core scan try_pk_against_tables and its probe test.

Precommit (Rust 1.86): fmt + clippy + tests green.
2026-07-02 23:17:20 -07:00
Matthew Jackson 47406a7e32 aacs: full-ladder resolve_candidate + correct mk_from_dk
resolve_candidate(candidate, mkb, unit_key_ro, vid) -> ResolvedChain: one
composed, PURE-DERIVATION boil-down for a candidate key at any ladder rung
(KeyCandidate::{Uk,Vuk,Mk,Pk,Dk}, each carrying the module's newtype). Walks
DK/PK/MK/VUK -> terminal unit keys, parsing Unit_Key_RO.inf at the version
the disc's MKB declares, and returns every CPS unit key as (cps_unit, key)
(matching ResolvedKeys) plus the intermediate chain. No sampling/validation
(that's the caller's unit_key_validates) and no position recovery (Dk is a
positioned DeviceKey; recover_dk_position first). Adds ProcessingKey newtype.
Consumers stop re-composing the ladder; every client hardens one impl.

Correct mk_from_dk: real Subset-Difference walk (derive_media_key_and_pk_from_dk)
instead of the Media-Key-Variant path, which needed integrator KCD absent
in-tree and Err'd for every real disc -- dead for both consumers. Drops the
now-unused vid arg.
2026-07-01 22:34:52 -07:00
Matthew Jackson 03cbda741a restore freemkv-unlock path dep for local dev (post-v1.2.1) 2026-07-01 19:08:56 -07:00
Matthew Jackson ec59ffe105 v1.2.1: bump version (freemkv-unlock git-pinned for the tag) 2026-07-01 19:08:53 -07:00
Matthew Jackson 02132f52b5 changelog: describe the shipped re-base DTS fix + faithful-passthrough note 2026-07-01 19:02:04 -07:00
Matthew Jackson ca8d046f2a mux/dts: re-base PTS per PES instead of a running clock (fix long-title drift)
The first cut used a global running clock (max(next, own-PES PTS) + advance),
which fixed the same-PES collision but DRIFTED: once accumulated frame
durations exceeded the PES-timestamp spacing, it never re-based, so a
feature-long DVD DTS track ran minutes past its real length (2h44 for a 2h03
film) while AC-3 from the same source stayed exact.

Match the AC-3 path: re-base to each PES's own container timestamp, and advance
by one frame duration ONLY within a run of AUs sharing one PES. Fixes the DVD
multi-frame-per-PES collision without drift; the UHD DTS-HD MA per-PES
attribution (da85f56) is preserved (each AU still takes its own core PES's PTS).
Adds new_pes_rebases_to_its_own_pts_no_drift; full mux suite green (905).
2026-07-01 18:21:25 -07:00
Matthew Jackson f33a9f6c22 v1.2.1: bump version + changelog (DVD DTS timestamp fix) 2026-07-01 17:30:25 -07:00
Matthew Jackson a9aee0d5a4 mux/dts: monotonic per-frame PTS + real frame duration (fixes DVD DTS)
DVD packs several DTS core frames into one PES; the parser stamped every
access unit with that single PES PTS and duration_ns=None, so consecutive
frames collided on one timestamp — ffmpeg rejected the output as 'non
monotonically increasing dts to muxer: X >= X' (deep-decode = corrupt,
e.g. The Punisher). The UHD DTS-HD MA path (one AU per PES, distinct PTS)
was unaffected, which is why this only surfaced on DVD.

Parse the DTS core header for samples ((NBLKS+1)*32) and sample rate
(SFREQ, 48kHz fallback) to derive each AU's duration, and stamp a running
monotonic PTS: max(next_clock, own-core-PES PTS), then advance by the
frame duration. A later PES whose PTS is ahead of the clock still wins
(preserves the UHD per-PES attribution from da85f56/571c613); frames
sharing one PES advance frame-by-frame instead of colliding.

Tests: the 3 that encoded 'same PES -> same PTS' now assert monotonic
advance; new dvd_many_cores_one_pes_are_strictly_monotonic reproduces the
Punisher bug; duration/SFREQ-fallback unit tests added.
2026-07-01 17:17:35 -07:00
Matthew Jackson e16f207efa restore freemkv-unlock path dep for local dev (post-v1.2.0) 2026-07-01 16:53:06 -07:00
Matthew Jackson e5540da6d4 v1.2.0: bump version (freemkv-unlock git-pinned for the tag) 2026-07-01 16:53:03 -07:00
Matthew Jackson 9b49beabee changelog: DVD DTS/LPCM silent-audio fix + Pass-N recovery specialists + flat scheduler 2026-07-01 16:50:08 -07:00
Matthew Jackson b0037f2db9 recovery: flat-pool bandit scheduler (FREEMKV_PATCH_FLAT, opt-in)
The tier ladder (bucket->mug->thimble) forces tier 0 to attempt ALL
ranges before tier 1, and tier 1 all before tier 2 — so on a hardened
residual (late resume, flood already gone) the marginal specialists are
unreachable for hours: you bail near-empty buckets over hundreds of dead
ranges before a thimble ever touches water.

FREEMKV_PATCH_FLAT collapses the 3 tiers into ONE flat pool of all 16
handler configs. run_handlers already sorts best-first by the live
decayed-yield scorecard, so this is a data-driven bandit: the first
ranges try them all (explore), the ranking floats whatever is actually
landing sectors to the front (exploit), re-measured per range; a handler
that doesn't fit stays last but is never dropped (floor -> can revive).
The new handlers self-limit (yield after 4 dead reads), so the flat
scheduler does NOT reintroduce the old depth-first per-range starvation.

Opt-in; unset keeps the proven tier ladder as the default for 1.2.0.
2026-07-01 16:44:29 -07:00
Matthew Jackson 90070c81bd test: widen profile_03 efficiency guard for tier-2 roster
The tier-2 marginal specialists now additionally probe permanently-bad
residual sectors before leaving them NonTrimmed, so the alternating-bad
profile's finite read count rose past the old tier-0/1 bound. Guard
still catches runaway; behavior asserts unchanged.
2026-07-01 13:55:20 -07:00
Matthew Jackson b8c2054249 recovery: wire tier 2 marginal specialists into the Pass-N chain
Add a third breadth-first tier (PATCH_TIERS 2->3) that runs the marginal
specialists on the hardened residual tiers 0-1 leave: SlowSpin (Linear
fwd+rev @ min), FuaRetry (Linear fwd+rev+Bisect @ FUA), SlowFua (Linear @
min+FUA), CachePrime, Oscillate (@ max and @ min), SpeedSweep. Every read is
a wedge-safe read_span, so they inherit wedge-abort / unproductive-yield /
deadline for free. All are new configs, so the EWMA scorecard calibrates each
once then ranks by decayed rate — a specialist that doesn't fit self-
deprioritises. Tiers 0-1 (fast scouts, slow-deep) are unchanged; this is
purely additive. Also switch the scorecard log sort to sort_by_key.

cargo test -p libfreemkv green (2200 passed).
2026-07-01 13:50:35 -07:00
Matthew Jackson c4357d9d87 recovery: CachePrime + Oscillate handlers + fixtures
CachePrime reads the good run immediately preceding a residual island to lock
the servo/PLL, then reads the island warm (boundary sectors the drive can't
cold-seek). Oscillate reads each residual sector by alternating approach —
forward-into then reverse-into — for direction-dependent tracking. Both go
through the wedge-safe read_span (primes included). FakeDisc models a
direction-dependent sector and a servo-primed boundary sector; fixtures prove
a forward/cold Linear misses each while Oscillate/CachePrime recover them.
2026-07-01 13:50:35 -07:00
Matthew Jackson 63c1d5db63 recovery: FuaRetry + SlowFua fixtures (FUA / min+FUA levers)
FuaRetry (Linear fwd+rev+Bisect at {max,fua,deep}) and SlowFua (Linear at
{min,fua,deep}) are Linear/Bisect at FUA params, no new struct. FakeDisc now
models a stochastic sector that lands only on the Nth physical (FUA) read (a
cached re-read keeps missing) and a hardest sector needing BOTH min speed and
FUA. Fixtures prove cached reads keep missing while the FUA group lands the
stochastic sector, and that neither lever alone recovers the hardest sector —
only the min+FUA combination does.
2026-07-01 13:50:35 -07:00
Matthew Jackson e91ce25db3 recovery: SpeedSweep handler + slow-spindle fixtures (SlowSpin/SpeedSweep)
SlowSpin is Linear pinned to min speed (no new struct). SpeedSweep is a new
per-sector handler that tries Max->Min until one reads (speed resonance).
FakeDisc now models a sector that reads ONLY at min speed; fixtures prove a
max-speed Linear misses it while SlowSpin and SpeedSweep recover it (and that
SpeedSweep tries fast-then-slow).
2026-07-01 13:50:35 -07:00
Matthew Jackson 2c94ce97c6 recovery: decayed (EWMA) handler scorecard so the ranking flips
Grade handlers by an exponentially-decayed recovery rate (bytes/second,
alpha=0.5) instead of a cumulative rate. The residual hardens mid-pass, so
the best technique changes: cumulative froze the early winner in the lead
forever; the EWMA re-prices continuously — a handler that stops earning
decays down, a late-starting specialist climbs. Keeps rank()'s untried->top
one-shot calibration and attempted-but-zero-time->bottom.

New flip fixture proves a handler that recovers a lot early then nothing
loses its lead to one that starts recovering later.
2026-07-01 13:50:35 -07:00
Matthew Jackson e7998a1376 recovery: parameterize the read primitive with ReadParams (speed/FUA/timeout)
Add ReadParams { speed: SpeedPref, fua: bool, timeout: TimeoutPref } and
thread it through read_span so every wedge-safe handler read can request a
spindle speed (SET CD SPEED issued only on change, restored to max when the
handler exits), set the READ(10) FUA bit, and pick the 10s vs 60s timeout.

- SectorSource gains read_sectors_fua (default ignores fua); Drive sets the
  CDB bit, DecryptingSectorSource threads fua to its inner read.
- recovery_read gains a fua param.
- Linear becomes { direction, params }; Bisect/Jump take params. Existing
  tier-0/1 instances keep identical behavior (max speed, no FUA, fast/deep).
- Scoreboard keys on the full-config String name (linear:fwd:max:fast, ...).
- FakeDisc observes speed + FUA + approach so specialist techniques are
  provably exercised in later commits.

cargo test -p libfreemkv green (2193 passed).
2026-07-01 13:50:35 -07:00
Matthew Jackson ba6ac3ccdb dvd: route private_stream_1 audio sub-ids by positional stream number
The private_stream_1 sub-stream id low nibble is the DVD audio-stream
NUMBER (0-7), shared across codecs, not a per-codec ordinal. A DTS (or
LPCM) track that isn't the disc's first audio stream got a sub-id one
too low, so the demux routing key (0xBD00 | sub_id) never matched and
every packet was dropped -> present-but-silent track. AC-3 at position
0 coincidentally worked (ordinal==position). Route by positional index.

Fixes silent DTS 5.0 track on The Punisher (AC-3 5.1 @ pos0 plays,
DTS 5.0 @ pos1 was 0xBD88, real wire id 0xBD89).
2026-07-01 13:49:55 -07:00
Matthew Jackson 1d1fe984d7 lib: remove standalone sector-health verify feature
Delete src/verify.rs (standalone 'read every sector, classify health'
module) + its lib.rs pub mod/pub use. It was old, unused code. The
decrypt-verify gate (disc::verify / UnitVerifier) is a DIFFERENT module and
stays. Relocated the pure chapter_at_offset helper (used by locate_ranges +
autorip's done-card) into disc/mod.rs as a pub fn.
2026-07-01 13:09:21 -07:00
Matthew Jackson 9298534b4b recovery: audit fixes — live at-risk excludes NonTried, latency-gated wedge, drop dead block counters
- Live located drilldown (sweep + patch progress snapshots) now excludes
  NonTried, matching the one-shot path. Including the unread remainder made
  main_at_risk_ms show ~full-movie at sweep start and melt to 0 as it
  progressed — unread is unknown, not damage.
- Wedge abort now requires the failure to be FAST (< WEDGE_FASTFAIL_MS,
  500ms) as well as wedge-family sense: a real uncorrectable sector on
  Hardware-error media spends ECC-recovery time before failing, so it no
  longer false-trips the fast-fail wedge abort. New regression test.
- Removed the always-zero blocks_attempted/read_ok/read_failed/
  unreadable_count from PatchOutcome + PatchLoopState (dead residue from
  the old grind loop; the HandlerScoreboard supersedes them) so the
  patch.done log no longer emits misleading zeros.
2026-07-01 12:07:08 -07:00
Matthew Jackson 6799890379 docs: 1.2.0 changelog — Pass-N handler chain, DVD drive-prep unlock fix, unlocker matrix 2026-07-01 11:05:31 -07:00
Matthew Jackson f36163ad02 disc: unlocker_matrix reports 'did work this rip', not 'matched'
yes now means the unlocker actually ran and did its job: LibreDrive from
the runtime firmware-unlock success, AACS host-cert only when LibreDrive
didn't do the bus (stock-drive fallback), CSS from the crack succeeding.
On a LibreDrive UHD that correctly reads LibreDrive: yes, AACS: no (LD
removed the bus, the cert route never ran) instead of the misleading
AACS: yes. Names stay registry-driven; runtime logic lives here.
2026-07-01 10:44:39 -07:00
Matthew Jackson 73ae12ed1f drive: disable FUA on READ(10) — the ~10x sequential-read regression
Forcing FUA (Force Unit Access) on EVERY READ(10) bypassed the drive's
readahead/streaming cache on the bulk Pass-1 sweep, collapsing sequential
throughput ~10x (UHD 15-25 → ~2 MB/s, DVD → ~0.5 MB/s), disc-type-
agnostic — the cache IS the streaming throughput. This was the real speed
regression (not unlock/riplock, which the earlier chase suspected). Clear
byte-1 bit 0x08. FUA will return as a dedicated Pass-N recovery handler
that sets/clears it per marginal-sector re-read, where cache-masking of a
stochastic sector actually matters — never blanket on the bulk path (#55).
2026-07-01 10:26:56 -07:00
Matthew Jackson 2f47b552e8 disc: Disc::unlocker_matrix() — registry-driven unlocker report
Returns each registered unlocker's name + whether it applies to this
drive+disc (via the unlock bridge over all_unlockers()). Disc crypto kind
derived in the library so the CLI and autorip render an identical, always-
current report with no hardcoded names. Also adds a Drive::unlocker_name()
runtime getter.
2026-07-01 10:00:59 -07:00
Matthew Jackson 89c4acdf8f drive: run drive-prep unlock for EVERY disc, incl. DVD (fix riplock regression)
The v1.0.0-rc.1 unlocker refactor added an `if disc_is_dvd() { return }`
early-return in Drive::init() that skipped the whole drive-prep unlock for
DVDs. That firmware unlock is what lifts riplock and readies max read
speed — a DRIVE-level, disc-independent feature — so skipping it left
every DVD stuck at stock/riplock speed (~0.4x, 3h ETA). UHD was unaffected
because it flows through the unlock.

Remove the skip: init() now runs the identity-keyed drive unlocker for all
discs (disc kind is Unknown at init, so only the drive unlocker matches;
the AACS host-cert handshake and CSS bus-auth still run later, gated on the
real disc kind, on top of the unlocked drive). Speed stays where it
belongs — SET CD SPEED(0xFFFF) at pass-1 start (disc/mod.rs) — not in the
unlocker. Also drop the matching probe_disc DVD skip.

Reverts the SET STREAMING stopgap (separate freemkv-unlock revert): that
treated the symptom; the real bug was the skipped unlock. Drive features
come from the firmware unlock, not a stock speed CDB.
2026-07-01 09:38:51 -07:00
Matthew Jackson 1d5a730853 recovery: fresh-eyes audit fixes (handlers + Pass-N engine + sweep)
Handlers (section_recover.rs):
- Bisect expand loops now honor ctx.halted() (were deadline-only, so a
  Stop could hang up to 60s vacuuming a readable island).
- read_span: explicit Transport arm so a bus-abort read isn't counted as
  unproductive grinding; debug_assert the sector-aligned span invariant.
- Scoreboard rank: an attempted-but-zero-time handler (e.g. returned Halted
  on its first check) now ranks BOTTOM, not top — it no longer crowds out
  proven performers.
- Document the wedge tier-size coupling + new regression test that a
  2-handler (tier-1) chain still catches a wedge via cross-section streak.

Pass-N engine (patch.rs):
- Rebuild PatchOutcome stats AFTER the post-read re-verify downgrade flush
  (was snapshotting before it, over-reporting bytes_good / recovered and
  risking a 'perfect rip' verdict on an imperfect one).
- Progress 'recovered' composes the still-bad set to MATCH work_total
  (subtract NonTried, add Unreadable) so the bar can't pin at 0 on a
  partially-swept disc or run backward on the Unreadable→NonTrimmed relabel.
- Remove dead work_done field; rewrite the stale 'adaptive batching' comment
  to describe the handler chain and mark block_sectors/full_recovery as
  informational-only.

Sweep (disc/mod.rs):
- Saturating arithmetic at the damage-jump position math (honor the
  read_error side's documented defence-in-depth guarantee).

Deferred (noted, need focused passes): fast_capture re-introduction,
Pass-1 halt-misclassified-as-jump, bytes_good display inflation, the
always-zero blocks_* telemetry, Pass-1 jump-on-first-error policy.
2026-07-01 09:14:32 -07:00
Matthew Jackson 0a0a8f1327 drive: apply drive-features for every disc, incl. DVD (fix riplock)
DVD ran the drive at ~0.4x stock speed: init() early-returned for DVD
before any speed was set, and its only speed command was the sweep's lone
legacy SET CD SPEED, which this BU40N ignores for DVD. BD/UHD got up to
speed only via the firmware unlocker's calibration — a path a DVD can't
take (it breaks stock CSS).

Call the new unlock-crate drive-features capability at init() for ALL disc
kinds, before the DVD stock-mode early-return, via a bridge shim. Stock
MMC only (SET STREAMING + SET CD SPEED), no bus unlock, so CSS is
undisturbed.
2026-07-01 08:56:19 -07:00
Matthew Jackson e396dcaab7 Pass-N: detect drive fast-fail wedge and abort instead of grinding
The 2026-07-01 overnight rip ground a wedged BU40N for 28 minutes at
0 B/s. Root cause: the handler chain's read_span split errors only into
Transport (bus gone) vs Bad (everything else), so the drive's fast-fail
wedge sense (ILLEGAL REQUEST 0x05/0x24 — it rejects every CDB in <100ms
without attempting recovery) was treated as an ordinary bad sector. The
chain hopped to the next section forever, and the existing wedge detector
in read_error.rs never saw it because the chain reads through
recovery_read directly.

Add a pass-level wedge streak: read_span counts consecutive wedge-family
(Hardware/IllegalRequest) senses; at WEDGE_ABORT_STREAK (16) it escalates
the read to Transport, which every handler already propagates as
TransportFault — aborting the whole pass and setting wedged_exit so the
caller spin-cycles. The streak is carried across sections via PatchCtx
(seeded into and read back from each per-section HandlerCtx) so a wedge is
caught even when every bad sub-range is smaller than the streak. Any Good
or non-wedge (medium-error) read resets it, so scattered bad sectors on
real media never trip it. New fixture test: a wholly-wedged 1000-sector
section aborts in <100 reads, not 1000.
2026-07-01 08:02:52 -07:00
Matthew Jackson f6f68cb9dc Bisect: halve at dead boundary + exempt boundary probes from stall
The expand loops broke out of the batch sweep on the first failing read,
overshooting the good/dead boundary by up to a batch and leaving the
readable sectors adjacent to a dead edge for the re-bisect to re-pin. On
top of that, early-yield counted those boundary-probe failures as a
stall, so the forward expand quit early AND the backward expand inherited
a poisoned unproductive counter and yielded instantly.

Now on a failed expand batch we halve the step (down to a single sector)
to recover right up to the dead edge in ~log2(batch) reads, drive the
expand loops off the deadline only (they self-terminate via halving), and
clear the unproductive streak once an island is located. patch_recovers_
good_middle_of_a_bad_range now recovers 50/50 good-middle sectors.
2026-06-30 22:06:04 -07:00
Matthew Jackson bdafec8081 section_recover: per-rip handler scorecard + Bisect-leads-scouts + proportional Jump
Scorecard: grade each handler by recovery rate (MB/s) per rip, order
best-first on later sections, log the ranking at pass end. Untried
handlers rank top so each is calibrated once before ranking narrows to
the winners. Ephemeral (reset per pass), no persistence.

Tier 0 scouts are now [Bisect, Jump, Linear-fast x2], scorecard-ordered.
Bisect leads: probing the MIDDLE of a range lands on a readable island in
one read where a linear scan grinds the dead front to reach it. Jump now
jumps to the middle of the REMAINING span (proportional) instead of a
fixed 8 MiB that leapt clean over small ranges and missed their readable
middles. Tier 1 is slow deep-recovery on the residue.
2026-06-30 21:26:35 -07:00
Matthew Jackson 71f6cb17a8 section_recover: per-rip handler scorecard + Bisect leads the scouts
Scorecard (ephemeral, reset each pass, no persistence): grades every
handler by recovery rate (bytes/sec). run_handlers orders handlers
best-first by that rate; an untried handler ranks top so each is
calibrated once, then the ranking narrows to the winners. Logged at pass
end (phase=scorecard) so the operator sees which handler is pulling the
weight on this drive/disc and which is a dud.

Tier 0 scouts are now [Bisect, Jump, Linear-fast x2], scorecard-ordered.
Bisect leads because probing a range's MIDDLE lands on a readable island
in one read, where Jump (linear from the front, big skip) can grind the
dead front or overshoot a small range entirely. The scorecard confirms
or overturns that order with real per-disc data.
2026-06-30 21:25:39 -07:00
Matthew Jackson 326868ec64 section_recover: Jump starts at 8 MiB base to clear big dead spots faster
Each dead probe read costs the drive's full ~10s timeout, so a large dead
region took a dozen escalating probes. Starting the jump at 8 MiB (vs
1 MiB) clears it in a handful; a skipped span stays bad for Bisect to
reclaim readable islands, so an over-jump loses nothing.
2026-06-30 21:12:12 -07:00
Matthew Jackson ebb3227294 patch: single Jump-scout tier 0, expand-Bisect, live in-handler progress
Tier 0 is now a single fast Jump scout: it streams the big readable
ranges back and skips dead runs in seconds, so the pass reaches every
section fast and converges to the small genuine-dead residue instead of
grinding three handlers x 60s on each dead fragment. Tier 1 (fast
mop-up + slow deep reads + Bisect) works only that residue.

Bisect now expands: on a good probe it reads outward forward and
backward in full batches until a read fails, recovering the whole
readable island in large reads; the two failing ends become smaller bad
sub-ranges it bisects again. One huge bad range becomes many precisely
located small dead clusters.

Progress heartbeat: HandlerCtx gains a throttled tick (250ms) called
from every read, pushing a fresh snapshot to the reporter DURING a
handler. The bar and speed now move continuously as recovery happens
instead of jumping once per section (the reason speed read 0 B/s and the
% looked frozen between range boundaries).
2026-06-30 20:57:36 -07:00
Matthew Jackson e8b3c719e6 patch: add Jump handler (lead fast tier) + recovery-based progress %
Jump: on sustained batch failures skip ahead an escalating distance
(1 MiB doubling to 256 MiB) to find where readable data resumes, leaving
the skipped span for Bisect to pin — mirrors the Pass-1 damage-jump. It
leads the fast tier so a large dead run is skipped in seconds instead of
the linear sweeps grinding every dead batch (10 s each) first; on a
readable range it just streams it back. Recovers readable data buried
behind a big dead front (the 192 MB Dune range).

Progress %: report bytes RECOVERED (initial-bad minus still-pending)
instead of a per-range counter that only advanced on the final tier — so
the bar reflects the readable bulk recovered during tier 0 the instant it
lands, matching the 'MB remaining' number.
2026-06-30 20:40:22 -07:00
Matthew Jackson 4e65f87b71 patch: breadth-first two-tier recovery, largest ranges first
The per-range walk was depth-first: each bad range ran the full handler
chain (fast + slow deep-recovery + bisect) before the next range was
touched. So a handful of tiny dead fragments at one end of the disc
burned the whole pass and the big NonTrimmed ranges elsewhere — usually
sweep-jump over-marks that read straight back — were never attempted.

Now recovery runs in two breadth-first tiers over ALL sections:
- Tier 0 gives every section one fast full-batch attempt (fast reads
  only), largest ranges first, so the recoverable bulk of the disc comes
  back in the first minutes.
- Tier 1 deep-recovers only the residue tier 0 could not pull.
Per-section still-bad sets persist across tiers. Largest-first ordering
means a big readable region is reached before time is spent on tiny dead
fragments.

Linear no longer collapses a failed batch to count=1 single-sector reads
(live probing: a marginal sector recovers in a large read, not a lone
one) — a failed 32-batch stays 32 and is re-attempted at full size by the
next handler/pass; Bisect salvages readable islands.

Adds a handler-start trace line so the debug log shows which handler is
running and the hand-off to the next.
2026-06-30 20:25:48 -07:00
Matthew Jackson c5e4fa7109 patch: replace grind-until-wedge loop with bounded handler chain
Pass-N recovery is now a chain of time-bounded recovery handlers instead
of one monolithic per-range loop that could grind the front of a bad
range for up to 30 min, wedge the drive, and abort the whole pass.

A bad range is a SubRanges set; recovery is an ordered list of
SectionHandlers (Linear{reverse,fast} covering back/forward x fast/slow,
and Bisect). A coordinator runs each handler with a hard per-handler
deadline: a handler recovers what it can (removing it from the still-bad
set) and hands the rest to the next handler; whatever is still bad after
the chain becomes NonTrimmed residue and we move on to the next range.

Guarantees, now structural rather than bolted-on:
- never hangs: every handler is deadline-bounded; the loop always drains
  to recovered-or-residue.
- always moves on: a range that cannot be finished leaves residue and
  advances; only a genuine transport fault or user halt ends the pass.
- extensible: a new recovery idea is one SectionHandler impl added to the
  chain; a proven-ineffective one is removed. The engine never changes.

Removes ~1.9k lines of the old inner loop (watchdogs, skip escalation,
NOT_READY grind, wedge counters) and their tests. fast_capture is now
inert (the chain supersedes it); breadth-first ordering becomes a future
scheduler concern. New module: disc/section_recover.rs (8 fixture tests,
injectable clock — bounded/never-hang proven without touching a drive).
Two A/B tests updated to the chain's strictly-better recovery counts.
2026-06-30 19:55:37 -07:00
Matthew Jackson b67eaab3a8 drive: always set FUA on READ(10) to bypass the drive cache
Every Drive::read now forces the read from physical media instead of
letting the drive satisfy it from cache. A recovery tool must not trust
the cache: on a re-read of a marginal sector the BU40N can return a
cached miss (~4 ms) instead of giving the surface a fresh physical
attempt (~40-250 ms), masking a sector that would otherwise read on a
real media hit. FUA (READ(10) byte 1 bit 3) is validated on the
BU40N/Initio bridge.
2026-06-30 18:49:14 -07:00
Matthew Jackson dfc54891c5 drive: Drive::spin_cycle() — soft un-wedge via START STOP UNIT (no eject)
Spin the disc down (START=0, LOEJ=0) then back up (START=1) to clear the
BU40N/Initio fast-fail wedge state a run of HARDWARE_ERROR reads leaves the drive
in — the non-eject power-cycle our notes say the wedge needs. The disc stays
loaded; we never eject (slot-loading — a human eject is a product failure for an
unattended service). Validated live 2026-07-01: took the drive from
failing-every-read back to reading at MB/s.
2026-06-30 17:54:51 -07:00
Matthew Jackson 4169cf7f4f disc/patch: fast-capture reads fail-fast (no deep recovery)
A fast_capture pass defers every failed block to a granular pass anyway, so
spending the drive's 60s deep-recovery timeout on it here only freezes the
breadth-first sweep on a pure-bad cluster (~25s per 32-block, incl. NOT_READY
grind). Read with recovery=false in fast_capture so a bad block fails on the
short timeout (~10s) and the sweep steps on; the granular passes
(fast_capture=false) do the deep recovery on what's left. No recovery lost —
the block stays NonTrimmed for the granular retry.
2026-06-30 17:08:26 -07:00
Matthew Jackson edd316b8a8 docs: 1.2.0 changelog — fast_capture breadth-first patch recovery 2026-06-30 16:38:56 -07:00
Matthew Jackson c57ac14ad1 disc/patch: fast_capture mode — breadth-first recovery (#50)
A PatchOptions.fast_capture pass reads each bad range ONCE at the full batch and
leaves every FAILED block NonTrimmed for a later pass — no bisect, no per-sector
grind, no retry. This lets a first retry pass grab the readable blocks (the
sweep's good skip-ahead overshoot) of EVERY section quickly, before any single
section's slow per-sector recovery — instead of grinding section 1 to exhaustion
before even touching section 2. A later pass (fast_capture = false) does the
granular bisect/retry on what's left.

Load-bearing invariant (fixture test): NO data is dropped. A failed block becomes
NonTrimmed (pending, retried by a granular pass), NEVER Unreadable. The test pins
that the readable half of a range recovers, the bad half stays NonTrimmed (not
Unreadable), and the bad block is marked in ONE batch read with zero bisection.
Disc::copy's internal patch keeps fast_capture=false (single-call full recovery).
2026-06-30 16:37:59 -07:00
Matthew Jackson 9154ac12d6 docs: 1.2.0 changelog — PassProgress mapfile-free contract + located drilldown 2026-06-30 16:13:05 -07:00
Matthew Jackson ec3d072ec7 disc: add progress_snapshot_from_mapfile one-shot + locate_ranges test
The library now offers a one-shot PassProgress snapshot built from a mapfile on
disk + title, so a client gets the fully-rendered drilldown without parsing the
mapfile itself — used for autorip's pass-boundary and mux-entry paints (no live
callback yet at those points). Adds a locate_ranges test covering in-feature vs
out-of-feature at-risk time (ported from autorip's removed from_map tests).
2026-06-30 16:09:28 -07:00
Matthew Jackson bbf04c3d51 progress: PassProgress carries the located drilldown (emit side)
Add LocatedRange + LocatedProgress to the progress contract and a 'located'
field on PassProgress, populated by the sweep + patch emitters from the
in-memory bad-range set + title. Move the range->chapter/time annotation
(locate_ranges, range_chapter, byte_offset_in_title) into the library so a
client renders the disc map + at-risk movie time straight from PassProgress
and never reads the mapfile itself — if the mapfile becomes a mapdb, this type
and its producer change, clients don't.

PassProgress is no longer Copy (located carries a Vec); it's built once per
throttled emission and passed by reference. Non-locating phases (verify,
extract) emit LocatedProgress::default(). Adds consts::MILLIS_PER_SEC.
Consumer-side wiring (autorip drops Mapfile::load) follows.
2026-06-30 15:26:49 -07:00
Matthew Jackson d4f8c86cbc disc/patch: remove scatter-recovery (ruled out by live probing); add SubRanges
Live drive probing (cold-single vs in-stream batch vs centered window,
one clean uncached read per sector) showed recovery RATE is identical
across approach: 3/5 every mode. The drive's per-sector ECC is
media-bound and partly stochastic, NOT approach-bound. So scatter
(seek-away recalibration + far-anchor re-read) does not improve recovery
and is dead weight — removed (function, read_good_sectors, SCATTER_*
consts, and its tests).

Add SubRanges: the still-bad (pos,len) sub-ranges of a section with
remove()/split, the work-list foundation for the upcoming per-section
recovery decomposition (recover_section -> stream/bisect/retry phase
helpers). Unit-tested; unused until the decomposition lands.

25 patch + 8 AB profile tests green; precommit clean on Rust 1.86.
2026-06-30 13:56:49 -07:00
Matthew Jackson a4da4dbe24 disc/patch: keep MAX speed on recovery reads + geometric batch re-grow
Live probing on the BU40N/UHD testbed (rprobe/dprobe) settled the speed
question empirically: a marginal sector reads ~12x FASTER at MAX speed
than at slow, and slow NEVER recovered one MAX didn't. Cleanly-tested
(one uncached read per sector) the recovery RATE is identical across
speed/direction/window/batch — the drive's per-sector ECC is media-bound,
not approach-bound. The only host lever that matters is read SPEED.

So the old 'drop to 0x0000 on first failure and hold it for the whole
range' only slowed the GOOD sectors of a bad range — measured ~3x slower
overall. Now the range stays at MAX; the first failure just re-attempts
once (stochastic media) and falls through. Live A/B: 11 KB/s -> 33 KB/s,
bytes_lost unchanged.

Also: batch adaptation is now halve-on-failure / double-on-success
(geometric), so a bisected batch climbs back through clean runs from any
size (the old count==1-only upscale left it stuck at mid sizes). Removed
the dead consecutive_singles_ok counter and the inert batch-retry probe.

25 patch tests green; precommit clean on Rust 1.86.
2026-06-30 13:25:07 -07:00
Matthew Jackson e6dfe54e10 disc/patch: drop cache-priming, bisect batch failures instead of count=1
Two patch-recovery speed fixes, grounded in live rip1 telemetry on the
Dune UHD bad regions.

1. Pull prime_cache. In the reverse walk it issued throwaway reads of
   lba-3..lba-1 — i.e. straight into the not-yet-visited dead L-EC core —
   grinding each ~7-9s at the fast timeout, then discarding the result.
   The live trace showed a good target read (read_duration_ms=1) taking
   9s wall-clock purely from the prime grind ahead of it. The dead-zone
   skip (Tier 1) + scatter's recalibration read (read_good_sectors)
   cover the cache-warm intent without grinding the dead zone, so prime
   is redundant harm — removed (function + CACHE_PRIME_SECTORS + call).

2. Bisect on batch failure. handle_read_failure used to collapse a
   failed count>1 batch straight to count=1 and single-walk the entire
   batch. Now it halves (count/2) and retries the same start: a good
   half recovers in BULK, the bad sector is isolated in O(log n) reads
   instead of O(n). Invariant preserved — only a count==1 failure marks
   NonTrimmed, so no good sector is lumped into a bad mark.

These are Tier 2 (bisect) of the progressive recovery model
(fast-mass-skip -> bisect -> scatter); composable so order can be tuned.
25 lib + 8 AB fixture tests green; precommit clean on Rust 1.86.
2026-06-30 11:20:44 -07:00
Matthew Jackson c68d40887a disc/patch: add scatter-recovery for stuck bad sectors
The 'reset, read good data, come back for one sector' technique. A
genuinely-damaged sector makes the drive grind its full recovery timeout
and still fail; re-reading consecutive bad LBAs at identical conditions
re-fails AND is the rapid-failure cadence that drops the BU40N into a
firmware fast-fail wedge (CLAUDE.md hard-rule #2).

scatter_recover: on a stuck single sector, make up to SCATTER_MAX_ATTEMPTS
fresh tries, each preceded by read_good_sectors() — a seek to a far
known-good anchor + a sustained read that re-seats the head/servo and
breaks the failure cadence. The fresh re-read uses the FAST timeout
(recovery=false) so a recalibrated marginal sector reads quick and a
truly-dead one fails fast instead of burning 60s per attempt. The seek +
good read IS the settle (matches ddrescue/MakeMKV) — no idle sleep.

Gated to genuine single-sector MEDIUM_ERROR (0x03): transport faults
still abort the pass, NOT_READY keeps its own retry path, wedge-family
senses are left for cooldown/eject. A scatter recovery is recorded
exactly like a normal read success (write-then-mark-Finished); nothing
marks a range good without a successful read + write.

Logs anchor_ms/reread_ms per attempt for live tuning. 5 fixture tests
(recover-marginal, give-up-dead, skip-non-medium, skip-batch, AACS
widen); 2 AB read-count guards raised for scatter's bounded extra reads.
Precommit green on Rust 1.86.
2026-06-30 10:29:55 -07:00
Matthew Jackson ad9ea4c199 disc/patch: extract PatchCtx/patch_region orchestration + unify log taxonomy
Pure structural refactor of the Pass-N recovery path, no behavior change.

- PatchCtx<R> holds the one-pass context (reader, pipe, shared, opts,
  reused read buffer, cross-region state) and owns orchestration.
- PatchCtx::run() walks the ordered bad ranges, applies the inter-range
  cooldown (armed on a grind, consumed at the next range's entry), and
  stops the pass the moment a range reports halt/wedge/transport-fault.
- PatchCtx::patch_region() is the focused single-range loop: read ->
  success/failure -> damage-skip -> per-range watchdog. Every former
  break / break 'outer is now a typed RegionOutcome (Completed,
  SkipLimit, BudgetExceeded, Wedged, Halted, TransportFault) that run()
  maps back to next-range vs stop-pass exactly as before.
- recovery_read() extracts the AACS mid-unit-window widening + plain
  read into one reusable helper (the building block the upcoming
  scatter re-read will share).
- Logging unified onto a single dotted patch.* taxonomy (patch.region.*,
  patch.read.*, patch.probe.*, patch.wedge.*, patch.batch.*,
  patch.mapfile.*, patch.skip.*, patch.stall, patch.speed); the
  freemkv::scan pass-boundary markers are left intact.

20/20 disc::patch tests green; precommit (fmt + clippy -D warnings +
tests) clean on Rust 1.86.
2026-06-30 10:04:51 -07:00
Matthew Jackson 9422909773 error: treat IoError/DeviceNotFound as SCSI transport failures (dead bus)
is_scsi_transport_failure recognized only the 0xFF SCSI sentinel, so the two
NON-SCSI dead-bus faults the Linux SG_IO transport returns — Error::IoError
(ioctl(SG_IO) == -1: ENODEV/EIO on an unplugged bridge) and Error::DeviceNotFound
(fd gone) — were classified as recoverable bad sectors. Every caller
(sweep read_error, patch, single-pass fill_extents) would then zero-fill /
skip-ahead across a wedged bus instead of aborting the pass and re-enumerating
the bridge — potentially zero-filling a huge span. Add both to the classifier so
a dead bus aborts the pass everywhere. (Companion to the unlock_bridge fix, which
maps the same two faults to 0xFF across the freemkv-unlock seam.)
2026-06-30 07:19:03 -07:00
Matthew Jackson 53e2e9a6ce unlock_bridge: map non-SCSI transport faults (IoError/DeviceNotFound) to 0xFF
Round-2 audit gap in the prior bridge fix: extract_scsi_context only recognizes
Error::ScsiError / DiscRead and collapses every other variant to (0, None). But
the Linux SG_IO transport returns genuine dead-bus faults as non-SCSI variants —
Error::IoError (ioctl(SG_IO) == -1: ENODEV/EIO on an unplugged bridge) and
Error::DeviceNotFound (fd gone) — which were crossing the seam as status 0x00, so
the unlock crate classified neither as a transport fault (needs 0xFF) nor as a
sense-bearing rejection, and kept hammering a dead bus.

Match the error variant in the adapter: a SCSI status (CHECK CONDITION or a
drive-tagged 0xFF) keeps its real status + sense; any other variant is a
non-SCSI transport/IO fault → SCSI_STATUS_TRANSPORT_FAILURE / None. Keying off
`sense.is_none()` would be wrong (a senseless CHECK CONDITION is a rejection).
Add bridge tests covering CHECK-CONDITION sense preservation, 0xFF passthrough,
and IoError/DeviceNotFound → 0xFF.
2026-06-29 22:57:09 -07:00
Matthew Jackson ab289fb5c4 mux: add the B1 drop-to-keyframe resync gate to DiscStream
DiscStream (the live-drive single-pass path) enables AACS decrypt-loss
concealment (NULL-TS fill on an undecryptable unit) but, unlike the file-backed
PipelinedPesStream, had no B1 resync gate — so after a concealed gap it forwarded
inter-coded video frames referencing the now-missing data, producing
decode-broken MKV output (dangling-reference frames).

Mirror PipelinedPesStream: add per-stream ResyncGate + is_video, and route every
codec-frame emit on the TS path (in-stream parse, demuxer EOF flush, parser EOF
flush) through gate.admit(is_video, frame.discontinuity, frame.keyframe) so a
video track drops forward to the next keyframe after a concealment event. Warn
once if a gate is still armed at EOF (a concealed gap in the final GOP with no
trailing keyframe). Audio/subtitle always admit.
2026-06-29 22:27:52 -07:00
Matthew Jackson dc3b7970e7 unlock/patch: preserve SCSI sense across the bridge; init aborts on dead bus; slow-retry first patch failure
Audit fixes (v1.1.0..HEAD regressions in the unlock migration + adaptive patch
speed):

- unlock_bridge ScsiAdapter: libfreemkv's transport returns Err on ANY non-zero
  SCSI status (a normal CHECK CONDITION), not only transport faults. The adapter
  was collapsing every such Err to { status: 0xFF, sense: None }, which discarded
  the parsed sense and defeated the AACS handshake's ILLEGAL_REQUEST wedge guard
  (so it kept hammering the drive — hard-rule #2) and inverted its
  transport-vs-rejection diagnosis. Now reconstruct status + the 32-byte sense
  buffer (sense_key@2, asc@12, ascq@13) and only emit 0xFF/None for a genuine
  transport fault.

- Drive::init: a genuine transport fault during the drive-prep unlock means the
  bus is dead — propagate it (the v1.1.0 invariant) instead of silently
  swallowing it via `if let Ok`. Other errors (no matching unlocker) still fall
  through to stock mode. SET CD SPEED max now runs only when the bus is alive.

- disc::patch: on the first read failure in a range, drop to slow recovery speed
  and RE-ATTEMPT the same position at slow speed before marking it. A
  single-sector range's first failing sector was being marked from a MAX-speed
  read it never got to recover.

- docs: lib.rs architecture diagram (handshake → host_certs) and README (stale
  pluggable-unlock-seam / register-unlocker / crates.io / docs.rs references).
2026-06-29 22:16:28 -07:00
Matthew Jackson 7cac0a1b15 disc: log adaptive patch speed transitions; issue SET CD SPEED max unconditionally
Two recovery-path fixes:
- patch: log the per-range speed transitions (INFO, phase="patch_speed") — each
  range enters at 0xFFFF (max) and drops to 0x0000 (slow recovery) on its first
  read failure. Previously the adaptive-speed behavior was invisible in the logs.
- drive init: issue the generic SET CD SPEED max UNCONDITIONALLY at drive-open,
  not only when a firmware unlocker matched. A stock-mode BD/UHD drive (no
  firmware unlock) was left riplocked because the call sat inside the
  unlocker-matched branch.
2026-06-29 21:47:50 -07:00
Matthew Jackson 5bd7da0f6f README: drop crates.io/docs.rs badges + stale plugin-model line (off crates.io now) 2026-06-29 21:17:06 -07:00
Matthew Jackson dd23f3f3c9 Cargo: publish = false — off crates.io (git-deps the firmware crate)
libfreemkv now git-deps freemkv-unlock (which carries drive firmware and can
never be on crates.io), so libfreemkv itself can only be consumed by git tag.
The freemkv-unlock dep stays an interim path dep for local cross-repo dev; the
release script swaps it to a git tag in the tagged commit.
2026-06-29 21:04:05 -07:00
Matthew Jackson c7925a293d unlock: dispatch via freemkv-unlock; delete in-tree handshake/css-auth/registry
Rewire the three unlock dispatch points through the freemkv-unlock crate via a
private `unlock_bridge`: drive-prep (kind=Unknown) at `Drive::init`, AACS cert
(kind=Aacs) at `do_handshake_cert`, CSS bus-auth (kind=Css) at scan. The bridge
news up `all_unlockers()` and runs the first matching one, mapping its
`Unlocked` result to the bus-key gate. After a successful drive unlock,
libfreemkv issues a generic SET CD SPEED (max) itself — the old per-unlocker
trait method is gone.

Delete the in-tree unlock code now owned by freemkv-unlock: the AACS cert
handshake (`aacs/handshake.rs`), the CSS bus-auth (`css/auth.rs`), and the
unlock registry (`unlock.rs`). Host-cert collection (a keysource concern) stays
in a small `aacs/host_certs.rs`. No public unlock surface remains — clients
touch libfreemkv only, oblivious to unlockers (as they are to SCSI). 2277 tests
pass.
2026-06-29 20:45:00 -07:00
Matthew Jackson 37a8d8f710 stage 4a: bridge libfreemkv to the freemkv-unlock crate
Add the freemkv-unlock dependency (path dep for local dev; re-pinned to a git
tag at release) and the consumption bridge: one generic ScsiAdapter mapping
libfreemkv's transport to the unlock contract, host-cert + DriveId mapping, and
run_unlockers() which news up all_unlockers() and runs the first matching one.
Additive — validates the dep resolves and the adapter compiles; the in-tree
registry/handshake/css path is still active and gets replaced in 4b.
2026-06-29 19:48:50 -07:00
Matthew Jackson 48bea05472 aacs: extract run_cert_handshake + collect_host_certs as shared primitives
Pull the wedge-guarded cert loop and host-cert collection out of the in-tree
AacsCertUnlocker into public aacs::handshake primitives (run_cert_handshake +
CertHandshake, collect_host_certs). The in-tree path now delegates to them, so
the external freemkv-unlock-aacs plugin runs the IDENTICAL cert handshake — one
implementation, two callers. Pure refactor of the live AACS path; the existing
handshake + collect_host_certs unit tests validate it unchanged.
2026-06-29 17:33:26 -07:00
Matthew Jackson 04ae2704f1 unlock: route_unlock returns a structured UnlockRoute; ctx carries opts
Replace route_unlock's Option<(name, Vid)> with a structured UnlockRoute
{ Unlocked(name, Unlocked) | Failed(UnlockError) | NoMatch } so a single
dispatch serves every caller: drive-prep wants "did anything unlock", and the
AACS cert route (next) needs the FAILURE REASON to render "missing keys" vs
"host cert rejected" instead of collapsing it to a bare None. Only a genuine
SCSI transport fault still returns Err (abort). UnlockCtx gains an optional
ScanOptions (the cert route's host-cert source), and read_mkb_from_drive now
takes &mut dyn ScsiTransport — both prerequisites for the cert handshake to
become an external freemkv-unlock-aacs unlocker. Drive-prep + CSS callers fold
the new outcome; no behavior change.
2026-06-29 17:18:11 -07:00
Matthew Jackson 26fc70b91b unlock: Unlocked carries the bus-key gate nuance; route_unlock returns it
Extend Unlocked with drive_unlocked + read_data_key_err so the registry result
carries everything the downstream bus-key gate needs — a firmware unlocker sets
drive_unlocked=true (bus encryption removed at the drive, no bus key), the cert
route will set read_data_key (+ read_data_key_err on a failed bus-key read).
route_unlock now returns the full Unlocked rather than just a Vid, so a single
dispatch can fold the firmware, cert, and CSS results uniformly. Prep for the
AACS cert handshake becoming a registry unlocker.
2026-06-29 16:59:04 -07:00
Matthew Jackson a061694ba0 css: CSS bus-auth becomes a uniform registry Unlocker
Convert the CSS read-unlock into a first-class registry Unlocker (CssUnlocker)
dispatched through route_unlock like every other barrier removal, instead of
a direct call in scan. libfreemkv appends the built-in CSS unlocker (and, next,
the AACS cert handshake) exactly once via ensure_builtins(), AFTER any
client-registered firmware unlocker — so the registry order is firmware → cert
→ css, owned by the lib, not the client.

Defense in depth: the unlocker does NOT trust the caller-declared DiscKind.
matches() filters on the declared kind (Css), but unlock() self-verifies
against the drive's GET CONFIGURATION profile and refuses (UnlockError::
NotApplicable, a new shared "this unlocker doesn't apply" variant) WITHOUT
issuing a single CSS CDB if the drive reports a non-DVD profile — so a
mis-routed Blu-ray is never sent CSS bus-auth. Guard the firmware unlocker the
same structural way (it matches only the drive-prep phase, kind == Unknown).

Tests: CssUnlocker matches only DiscKind::Css; a BD-profile drive yields
NotApplicable with zero CSS CDBs issued.
2026-06-29 16:49:52 -07:00
Matthew Jackson 45c0bc943c unlock: introduce UnlockCtx + DiscKind; trait keys off context
Reshape the Unlocker seam so every unlocker is dispatched at ONE place from
ONE ordered registry — the firmware, cert, and CSS routes are all "remove the
bus-encryption barrier", differing only in what they key off. matches() and
unlock() now take an UnlockCtx { drive_id, kind: DiscKind } instead of a bare
DriveId: a firmware unlocker keys off drive_id (kind irrelevant), the cert
unlocker will match DiscKind::Aacs, the CSS unlocker DiscKind::Css. UnlockCtx
is #[non_exhaustive] so a host-cert source can be added without breaking
external unlockers. Drive-prep dispatch passes DiscKind::Unknown (no disc
probed yet); the cert/CSS registry impls + the single post-probe dispatch
point follow in subsequent commits.
2026-06-29 16:27:30 -07:00
Matthew Jackson c9c64e2cbf css: auth primitives operate on &mut dyn ScsiTransport; clean up bus_auth
Thread &mut dyn ScsiTransport through unlock_css_reads and its bus-auth /
disc-key primitives instead of &mut Drive — they only ever issued SCSI via
drive.scsi_mut(), so this is mechanical and lossless, and it makes the CSS
unlock speak the same transport interface as the registry Unlocker trait
(prerequisite for CSS becoming a uniform unlocker).

Also clean up the badly-named, half-dead bus_auth:
- rename bus_auth -> establish_authenticated_session: it is run for its
  side effect (sets the drive's ASF=1, unlocking scrambled-sector reads),
  which the name now states.
- drop the derived CSS bus key from the return + computation: it had no
  consumer (descrambling is keyless via the Stevenson attack), so it was
  dead crypto computed on every DVD unlock. Return just the negotiated AGID,
  which the caller genuinely needs for the best-effort disc-key REPORT KEY.
2026-06-29 16:01:07 -07:00
Matthew Jackson 8ac18d857d aacs: handshake primitives operate on &mut dyn ScsiTransport
The AACS cert-auth primitives (aacs_authenticate, the AACS 2.0 P-256
variants, read_volume_id, read_data_keys) and their scsi_read/scsi_write
helpers touched the drive ONLY through Drive::scsi_execute — a pure
pass-through to the transport. Thread &mut dyn ScsiTransport instead of
&mut Drive so these primitives are transport-level, matching the firmware
Unlocker seam (which hands out &mut dyn ScsiTransport for testability).
Pure mechanical signature change, no logic change; the cert orchestrator
(do_handshake_cert) keeps &mut Drive for the OEM-VID shortcut and passes
session.scsi_mut() into the primitives. Step toward making the cert
handshake a uniform registry unlocker.
2026-06-29 15:47:30 -07:00
Matthew Jackson 0a006bbfa6 unlock: Unlocker::unlock returns Unlocked { vid, read_data_key }
Generalize the unlock seam toward uniform unlockers (firmware / cert / CSS).
`unlock` now reports what it LEARNED — a Volume ID and/or an AACS bus key —
instead of just a Vid; libfreemkv files those onto the disc/drive in one
place, so a plugin never touches Disc. The hardware side-effect (extended
mode) still happens inside unlock. route_unlock walks the registry in
registration order (the single ordered place that decides run order); no
priority field. Firmware route reports {vid: Some, read_data_key: None}.
2026-06-29 15:09:17 -07:00
Matthew Jackson 11554c1722 disc: credit firmware unlock in the bus-encryption gate
The bus-key gate only credited the cert handshake's read_data_key as proof
bus encryption was removed. A firmware unlocker removes it AT THE DRIVE
(serves clear content) and yields no read_data_key — so a SUCCESSFUL
firmware unlock (VID present, read_data_key None) tripped the gate and
blocked ALL key resolution, including the online source. That was the
root cause of live UHD discs reporting "missing keys" after an unlock.

Now a single predicate answers "is bus encryption gone?": never-had-it ||
file/ISO || firmware-unlocked || cert-bus-key. The gate is just
`if !bus_encryption_removed { error }` — no enumerated cases. HandshakeResult
gains `drive_unlocked`, and the read_data_key failure reason is captured so
the warn says WHY the bus key is missing.

Also: reword the first hardware-sense escalation as "fast-fail escalation"
(it is often transient — the drive recovers), reserving "wedge" for a
persistent run; and scrub the product name from core comments (it belongs
only in the unlocker crate).
2026-06-29 15:09:16 -07:00
Matthew Jackson 321442632a disc: make bus-key unavailability self-explaining in the log
When a bus-encrypted disc's handshake yields no read_data_key, the gate
logged a bare "bus_key_unavailable" with no indication of WHY — turning
every occurrence into archaeology (is the bus key not-attempted, or did
the read fail?).

Now:
- read_data_keys failure is captured (error code) instead of swallowed by
  `.ok()`, and logged at the handshake with its consequence.
- HandshakeResult carries `read_data_key_err: Option<u16>` so the gate
  distinguishes "never attempted" (None — VID-only/OEM path) from "read
  FAILED" (a code), and the bus_key_unavailable warn now reports the code
  plus whether a Volume ID was present.

No behavior change — purely diagnostic. The handshake_ok debug also now
reports has_volume_id.
2026-06-29 13:33:09 -07:00
Matthew Jackson 46797c01a3 mux: harden audio discontinuity handling (audit follow-up)
Two defensive hardenings from the post-fix audit (vs FFmpeg/GStreamer):

1. Move the `pes.discontinuity` partial-drop ABOVE the empty-data guard in
   all three audio parsers (ac3/dts/truehd), so a discontinuity signal can
   never be stranded by an empty post-gap PES. The demuxer only emits
   non-empty PES today; this is defense-in-depth for any future caller.

2. A PES with no PTS must not reset the timeline to 0. ac3 now carries
   `flush_pts_ns`, dts continues from the most recent known base; truehd
   already kept its running cadence on a None PTS. Matches OSS behavior
   (PTS rebases off the next PES that actually carries a PTS).

Adds an ac3 regression test (empty-payload discontinuity PES still drops
the stranded partial). Loss accounting was reviewed: TS-demux CC-gaps are
NOT counted toward lost_video_secs / abort (that is sector-based via
DiscStream::errors / mapfile bytes_unreadable), so a source splice never
inflates loss — no gating needed there.
2026-06-29 12:50:52 -07:00
Matthew Jackson 790a40138b mux: drop truncated partial audio frame on concealed gap
The AC-3, DTS and TrueHD parsers buffer access units across PES
boundaries. At a concealed-loss gap the buffered unit is truncated:
splicing post-gap bytes onto it manufactures a corrupt frame on top of
the real loss (FFmpeg "Failed to decode block code(s)" / "Invalid data
found" at the gap) and, for TrueHD, strands the PTS cadence into the
non-monotonic audio-DTS band seen on multi-clip titles.

The video parsers already handle this via the ResyncGate, but the
discontinuity signal was only wired into video — audio parsers ignored
pes.discontinuity and spliced across the gap.

Now, when pes.discontinuity is set, each audio parser drops the partial
(clears buf, and for DTS its PTS marks / pending base) so the post-gap
PES re-bases a fresh unit. A lost gap degrades to a clean single-frame
drop instead of a corrupt spliced frame. No effect on perfect rips: the
branch only runs when concealment inserted a discontinuity marker.

Adds a per-parser test feeding a partial frame then a discontinuity PES,
asserting the truncated partial is dropped (not spliced) and the post-gap
PTS is adopted.
2026-06-29 11:23:08 -07:00
Matthew Jackson 4e545e6e46 disc: adaptive patch speed — enter each range fast, slow on first failure
Pass N (patch) pinned the slow recovery speed (0x0000) for the whole pass.
But Pass 1's damage-jump overshoots, so most of a jumped range is clean data
the reverse-walk reads first — grinding it at slow speed wastes minutes per
gap. Now each range mirrors Pass 1's model:

- Enter at max speed (0xFFFF) + reset to the initial batch (current_batch
  carries across ranges, so a prior single-sector grind would otherwise start
  the next range slow). Reads the clean overshoot fast.
- First read failure in the range → drop to 0x0000 (once; idempotent SET CD
  SPEED) for the rest of that range — grind only the genuine damage.
- 10 s halt-responsive cooldown between ranges, gated on "this range actually
  grinded" so a many-small-range pass (100+ ranges) doesn't stall ~20 min on
  unconditional pauses.

No change to good/bad classification, the mapfile, or the abort math — purely
read speed + an inter-range cooldown. Synthetic speed-transition test to follow.
2026-06-29 10:34:18 -07:00
Matthew Jackson afe55c7871 docs: 1.2.0 changelog date → 2026-06-29 (release prep); fix neither/nor grammar in 1.1.0 entry 2026-06-29 09:57:29 -07:00
Matthew Jackson 7114e46018 mux/decrypt: audit-nit cleanup + defensive conceal fallback
Non-blocking follow-ups from the 1.2.0 audit:
- conceal loop: if decrypt reported loss but the padding-aware predicate
  matched nothing to conceal (a ~256^-31 contradiction), fall back to the
  strict predicate and conceal whatever it flags, loudly — belt-and-
  suspenders so ciphertext can never reach the mux.
- decrypt_dropped doc: reflect 1.2.0 (mux-path loss is concealed + tallied,
  not silently dropped).
- direct unit test for aacs_unit_still_ciphertext (the padding-aware
  conceal predicate): clear/all-zero/full-decrypted/full-ciphertext/
  decrypted-short-tail.
- fix three stale "v1.1.1" comment refs (the fragment-tail fix ships in
  1.2.0; there is no v1.1.1 release).
2026-06-29 09:51:28 -07:00
Matthew Jackson ccb1b16ffb mux: make B1 concealment decode-clean on every gap shape
Closes the three residual holes where a concealed/lost gap could still let
a dangling-reference frame reach the muxer (degraded/undecryptable-disc
path only; clean rips are byte-identical and untouched). Root cause: the
discontinuity signal was reconstructed from the 4-bit continuity counter
and applied per-PES, both of which are lossy.

Three coordinated changes:

1. CC-INDEPENDENT marker. fill_null_ts_unit now tags its NULL packets with
   an adaptation-field discontinuity_indicator; the demuxer recognises a
   0x1FFF packet carrying it as a concealed gap and forces a discontinuity
   on every tracked PID (the lost unit's PID is unknowable). This survives
   a loss that is an exact multiple of 16 packets (CC aliases to in-sequence
   — hole 3) and a loss at a PID's very start (no prior CC — hole 4); it
   also drops any open, potentially-truncated partial PES.

2. PUSI ATTRIBUTION. A gap landing on a PES boundary now flags the PES
   STARTING after it, not the one flushed at the boundary (hole 1) —
   stamping the pre-gap frame could arm-then-disarm the gate on a keyframe
   and admit the real post-gap inter frame.

3. PER-FRAME signal. codec::Frame gains `discontinuity`; each parser
   propagates it onto the first post-gap frame. MPEG-2 buffers whole GOPs
   asynchronously, so it associates the gap by ES OFFSET (like PTS/source),
   landing it on the exact post-gap picture mid-GOP (hole 2) — a per-PES
   flag stamped the previous picture. consume_ts (and the EOF flush drain)
   gate on frame.discontinuity.

Tests: CC-independent marker with in-sequence CC + leading-loss; PUSI
attribution flags the post-gap PES; MPEG-2 offset-mark stamps the post-gap
picture through GOP reorder, not the previous one. Existing B1 gate + EOF
tests still green (2270 lib tests).
2026-06-29 09:39:03 -07:00
Matthew Jackson a5268d15ca docs: 1.2.0 CHANGELOG — document the breaking public-API changes
Per the release decision: ship the AACS-version-stride API changes under
the 1.2.0 minor bump, with an explicit Breaking section (DiscInputs new
field, DiscInputsCtx::new arity, read_aacs_inputs* 3-tuple return).
2026-06-29 08:41:26 -07:00
Matthew Jackson accf21b03d mux: conceal only genuinely-undecryptable units (padding-aware)
The P3 concealment loop in DecryptingSectorSource::read_sectors keyed on
aacs_unit_needs_decrypt, whose sync check is the majority-vote
ts_sync_destroyed (<=16 of 32 syncs). A successfully padding-aware-
decrypted content-fragment TAIL unit (e.g. 11 content packets + 21 zero
padding) has only 11 syncs, so the majority vote called it "still
encrypted" — and when such a good unit shared a read buffer with a
genuinely-undecryptable one (dropped>0), the loop overwrote the GOOD
decrypted tail with NULL-TS, silently discarding correct video and
over-counting concealed units vs the tallied dropped bytes.

Add aacs_unit_still_ciphertext (padding-aware): encrypted AND at least
one non-zero (non-padding) 192-byte packet missing its 0x47 sync — the
same discriminator decrypt_unit uses to accept a fragment tail. The
conceal loop now uses it, so only genuinely-unrestored ciphertext is
concealed. Full and fully-ciphertext units are unchanged. Regression
test: a decrypted short-padding-tail co-resident with a failed unit is
left byte-for-byte intact while the failed unit is concealed.
2026-06-29 00:29:20 -07:00
Matthew Jackson 07396ec19d v1.2.0: bump version + reconcile CHANGELOG to [1.2.0] 2026-06-28 23:34:19 -07:00
Matthew Jackson da3859d0c5 mux: B1 drop-to-keyframe resync after a concealed gap
Pairs with A2 (read-path NULL-TS concealment). When the demux assembler
sees a TS continuity gap it now stamps `discontinuity` on the next
completed PES; the codec-parse stage carries that onto a per-track
ResyncGate. After a gap on an inter-coded video track the gate drops
forward to the next IRAP/IDR keyframe so no frame with a dangling
reference reaches the muxer (an ffmpeg deep scan would otherwise report
a missing-reference / non-existing-PPS error). Audio and subtitle tracks
have no cross-frame references, so the gate is a no-op there.

- ts.rs: PesPacket gains `discontinuity`; PesAssembler tracks a sticky
  pending_discontinuity flag set on CC gap / discontinuity_indicator and
  carried to the next completed/flushed PES.
- resync.rs (new): ResyncGate — per-track arm-on-gap, drop non-keyframes
  until the next keyframe disarms and resumes. Logs the resync + drop
  count once at the keyframe.
- pipelined_stream.rs: precompute per-track is_video, apply the gate in
  consume_ts. Out-of-range track index emits as-is (defensive).

Tests: ResyncGate unit tests; ts.rs gap-stamps-discontinuity; end-to-end
B1 video-drops-to-keyframe and audio-never-drops through PipelinedPesStream.
2026-06-28 23:01:47 -07:00
Matthew Jackson d334ab5749 1.2.0: mux loss-concealment read path (P3/Edit-2, A2 NULL-TS fill)
Decrypt-verify is a RIP gate, not a MUX gate. On the mux read path an
undecryptable content unit must never abort the mux:

- DecryptingSectorSource gains tolerate_decrypt_loss(): when set, an
  undecryptable in-content unit is tallied, overwritten with valid NULL
  TS packets (PID 0x1FFF) via aacs::fill_null_ts_unit, logged loud with
  its LBA, and the read returns Ok — the stream keeps flowing. The rip
  paths keep the fail-loud DECRYPT_VERIFY_READ decorator (re-read off the
  disc); only the mux opts in.
- Wire it into both mux read paths: the file-backed highway
  (build_iso_pipeline) and the inline DiscStream.
- NULL-TS fill keeps the demuxer byte-synced on the 192-byte stride; the
  lost video/audio PID packets surface as a CC gap the TS assembler
  already drops a partial PES on (the B1 foundation). Ciphertext is never
  passed downstream either way.
- Fix stale resolve_vid_only no-cert test: default is UHD (audit #4).

Tests: conceal-as-NULL-TS, fill well-formedness, fail-loud still holds.
2026-06-28 22:44:19 -07:00
Matthew Jackson 0b2e83e523 1.2.0: single MKB framing walker + AACS resolve hardening
- mkb_records() as the one record-framing iterator; rebuild walk_mkb,
  find_record_body, mkb_find_subdiff_records, mkb_content_len, mkb_version,
  mkb_type_raw, mkb_find_mk_dv on it (D4).
- resolve_vid_only / read_aacs_version default to UHD (+warn) on a missing
  content cert instead of BD; route MKB through bounded read_mkb_content.
- AacsVersion major()/from_major() + AACS_MAJOR_BD/UHD as the stride source;
  table + stride-discriminating regression tests.
- read_encrypted_units probes 8 evenly-spaced points per extent (off-midpoint
  scrambled content now sampled); decrypt source-zero mask uses PKT.
2026-06-28 22:28:09 -07:00
Matthew Jackson abd7af3907 1.2.0: unify hex parsing (one parser) + robust encrypted-unit sampling
- One workspace hex parser (libfreemkv::hex): the keydb / online / mapfile
  parsers had divergent prefix handling (0x vs 0X vs none) — a key written with
  a prefix one parser didn't expect was silently dropped. All three now call it.
- read_encrypted_units probes several points spread across each extent instead
  of only the midpoint-forward window, so a late-starting / sparse encrypted
  body still yields samples (empty samples make decrypt_with skip wrong-key
  validation). A read error at one probe no longer abandons the extent.
2026-06-28 22:12:06 -07:00
Matthew Jackson 7716a2cb17 1.1.1: unify AACS key-input path on Disc::inputs() + named constants
- read_aacs_inputs* now returns the AACS major version; DiscInputs carries it,
  and DiscInputsCtx parses Unit_Key_RO.inf at the disc's own stride (fixes the
  hardcoded-V20 read-time fetch for V10 discs). One source of truth, no version
  argument to drift.
- Disc::inputs() is the single complete AACS-input source (inf/MKB/VID/hash/
  version); the out-of-band duplicate readers go away.
- Named constants for AACS file paths (aacs::PATH_*) and the AACS majors
  (aacs::AACS_MAJOR_*, AacsVersion::major/from_major) replace magic strings/ints.
- push_ranges saturating (corrupt-disc panic guard).
2026-06-28 21:45:58 -07:00
Matthew Jackson 391e1ae39c 1.1.1: AACS decrypt + key-resolution hardening
- decrypt_unit: padding-aware acceptance — recover real video at content-
  fragment tails (the phantom mux-loss class) without weakening wrong-key
  rejection (a full content unit still needs all 32 TS syncs).
- scan: read the MKB via the bounded read_mkb_content so Disc::inputs()
  carries it. Online key resolution was shipping mkb=0 (a full read of the
  ~128 MiB MKB_RO allocation fails) → the decode service 404'd.
- resolve_vid_only: surface an MKB read error instead of silently emptying.
- fetch: a per-sample dry-set replaces the global fetch_spent latch, so a
  second CPS unit's key can still be fetched after the first came back empty.
- verify::push_ranges: saturating arithmetic (corrupt-disc panic guard).
- Tests for all of the above.
2026-06-28 21:12:57 -07:00
Matthew Jackson ee66db8b95 v1.1.0: bump version 2026-06-28 18:14:18 -07:00
Matthew Jackson 60df09f965 changelog: 1.1.0 (drop -beta.1 pre-release tag) 2026-06-28 17:58:46 -07:00
Matthew Jackson 46dc8720be aacs: rustfmt verify-gate files + correct cert/unit-key test fixtures to libaacs-strict layout; 1.1.0-beta.1 changelog 2026-06-28 16:49:16 -07:00
Matthew Jackson 353c8ce1fb verify: container-kind seam (HD-DVD-ready) + skip units with unread sectors
ContainerKind {Ts,Ps} + ClipLayout.container thread the post-decrypt structural check per clip; decryptability() dispatches it (TS: unit_is_clean_ts, PS: unit_is_clean_ps). New decrypt_unit_checked(unit,key,accept) decouples the container-agnostic AACS crypto from the format-specific acceptance (decrypt_unit delegates with the TS check). unit_is_clean_ps is the MPEG-2 PS pack-start check, documented UNVALIDATED for HD-DVD (.evo unit/seed/pack alignment must be confirmed on real media). clip_layouts assigns Ts today; .evo->Ps is the one-line HD-DVD hook.

reverify_iso now takes an is_finished predicate and SKIPS any unit with a non-Finished backing sector: we can't verify what wasn't read (a non-Finished sector is zero-filled because the drive read failed there), and must never waste a key lookup on a block the read already knows is bad. observe() (sweep) was already safe (only fed Good bytes).
2026-06-28 16:05:25 -07:00
Matthew Jackson 14070a2dec verify: post-read decrypt-verify gate + libaacs-strict verify + audit fixes
Post-read verify gate (new src/disc/verify.rs): UnitVerifier buffers/aligns the disc-absolute read stream into clip-file 6144-byte units, then makes one decryptability() decision per unit (CPI gate -> held keys -> key_fetch -> strict TS). POST_READ_VERIFY const kill-switch; fail-safe contract (only ever downgrades units it is confident are undecryptable; every doubt skips). Hooked into Disc::sweep (producer observes ciphertext -> WorkItem::MarkBad after the Good, FIFO-ordered) and Disc::patch (post-loop reverify_iso reads recovered units whole from the patched ISO). extract::clip_layouts enumerates AACS clips for the gate.

Standards-correct AACS verify: aacs::unit_is_clean_ts is a strict port of libaacs _verify_ts (all 32 TS syncs, not a majority vote); decrypt_unit accepts a key only on it; the majority verify_ts is removed. Deleted the Disc::verify_clips post-pass bolt-on (its primitive is absorbed by the read-path gate).

libaacs/DVD audit fixes: content-cert bus_encryption flag now read from bit 7 (was bit 0 - defeated the bus-key fail-loud gate); cc_id read from offset 14; title_cps_unit range-validated + 1->0 index-converted per libaacs. Corrected attack_crib ("functionally-equivalent" not "exact" port) and read_disc_key (READ DVD STRUCTURE 0xAD, not REPORT KEY) doc comments.

Also includes accumulated uncommitted work: key-fetch seam and TrueHD/DTS audio fix.
2026-06-28 15:03:52 -07:00
Matthew Jackson 1c8672dcaf changelog: enrich 1.1.0-beta.1 and rc.5.2 entries with concrete mechanisms (version label SoT, per-extent AACS anchor, E7017/E7022 no-key split, 3-sector unit grid) 2026-06-26 22:26:15 -07:00
Matthew Jackson b9c0a7c208 changelog: add 1.1.0-beta.1 version-stamp consistency, DTS-HD MA core-sync, and TrueHD DTS-monotonicity fixes 2026-06-26 22:16:45 -07:00
Matthew Jackson 3c6351975b docs(fvi): standardize table headers and multi-line dense cells
Use 'Semantics / reference' as the last-column header across all spec
tables, and break enum/option/note cells onto separate lines with <br>
for readability. Formatting only; no semantic changes.
2026-06-26 21:42:31 -07:00
Matthew Jackson fda771a06a mux: single source of truth for the version label (VERSION_LABEL/MUX_APP)
The CLI --version, the MKV muxing/writing-application field, and the FVI generator
all derive from one libfreemkv const, so a binary reports the exact same label it
stamps into the files it produces — no split-brain where an MKV claims one version
and the binary another.
2026-06-26 21:33:19 -07:00
Matthew Jackson 2f1697d587 docs(fvi): pretty-print JSON blocks in FVI format spec
Reformat all three JSON code blocks (section 8 ext example, Appendix A
Header schema, Appendix A Record schema) with consistent 2-space
indentation and valid syntax. No field or semantic changes.
2026-06-26 21:21:26 -07:00
Matthew Jackson fc95d55d57 mux: allow build-time version label override (FREEMKV_BUILD_LABEL)
build.rs emits FREEMKV_VERSION = FREEMKV_BUILD_LABEL when that env is set
(non-empty), else the Cargo package version. The muxing/writing-application
field and the FVI generator tag use it, so a pre-release/test build can be
stamped without bumping Cargo.toml and disturbing the tag-pinned [patch]
version matching. The git short hash is still appended either way.
2026-06-26 19:38:13 -07:00
Matthew Jackson 571c613c39 mux: fix non-monotonic audio DTS (TrueHD + DTS-HD MA) and stamp builds with git hash
TrueHD: when the PES PTS lags the access-unit cadence, resync to the PTS
but never snap the running timestamp backward, so the emitted DTS stays
monotonic across the resync (next_pts_ns = max(next_pts_ns, pts)).

DTS-HD MA: size each EXSS extension substream exactly from its header
(exss_frame_size) and skip it as a unit, so a false 0x7FFE8001 core sync
inside the lossless extension payload can no longer split the access unit
and truncate the extension. Falls back to a bounded scan when the header
is unparseable.

Provenance: build.rs bakes the git short hash into GIT_SUFFIX; the muxing/
writing-application field and the FVI generator tag now record the exact
build (e.g. "freemkv 1.1.0-beta.1 (g835cc99)"), so any output file is
traceable to the revision that produced it.
2026-06-26 19:26:24 -07:00
Matthew Jackson a12f3b6c78 Remove keydb download/save from the library
keydb I/O moves out of libfreemkv into freemkv-keysources
(KeydbSource::save / ::update). Delete src/keydb.rs entirely (save,
http_get, default_path, write_atomic, UpdateResult) and drop `pub mod
keydb;` — http_get had no real callers. The shared Keydb* Error variants
stay in error.rs (keysources raises them; the every-error-has-a-code
contract depends on them). flate2 is no longer used here, so drop it
from Cargo.toml (zip stays for labels/jar.rs).

CHANGELOG: note the keydb-I/O move; reword the rc.5.2
DefaultDecodedFieldDuration entry to state only the action taken (the
revert) rather than an unverified Windows-fps outcome.
2026-06-26 17:33:57 -07:00
Matthew Jackson a6073d6866 DVD vob_start absolute rebase + rc.5.3 audit fixes
- ifo.rs: rebase VTS title VOBS to absolute disc LBA (file_start_lba +
  vtstt_vobs); fixes DVD rips opening on the menu region instead of the
  movie (e.g. SOTL). Adds absolute-placement regression test.
- aacs/boil.rs: add mk_from_pk primitive (PK -> MK via MKB walk).
- dvdnav/: nav-VM command decoder + start-cell resolver seam, parked
  behind USE_NAV_RESOLVER (kept compiled, never executed).
- mux: FVI src.byte within-sector per spec; Unknown colour -> CICP
  unspecified (2,2,2,1); demux clear PCS -> NORMAL; ts.rs feed() base
  reset + boundary provenance fix.
- Assorted audit fixes (doc/comment/test accuracy) across the crate.
2026-06-26 17:03:58 -07:00
Matthew Jackson b4b05df97a Magic-number/taxonomy pass: central wire-format + sector + unit consts
- libfreemkv::consts: coding_type::* (ES coding-type bytes), pes_stream_id::*
  + PAYLOAD_RANGE, SECTOR_BYTES (usize) + SECTOR_BYTES_U64 (offset math)
- replace bare wire-code/sector literals across disc, mpls, clpi, labels,
  m2ts_mux, ps, tsmux, file_sector_source, extract
- remove two unreachable secondary-stream match arms in mpls parse_stream_entry
2026-06-26 13:20:21 -07:00
Matthew Jackson 52b9731453 AACS pipeline reshape + TrueHD metadata + central consts + clippy/fmt clean
- AACS: delete in-lib keydb parser (Step 3); boil-down primitives
  (mk_from_dk/vuk_from_mk/uk_from_vuk) + newtypes; KeySource->get_uk(ctx)+
  ResolveCtx; Unlocker->unlock()->Result<Vid,UnlockError> + AacsCertUnlocker;
  OEM bus-key gate (AacsBusKeyUnavailable); structured ResolutionTrace (Step 4).
- TrueHD: sample-rate from major-sync, Atmos label, 44.1k AU duration.
- consts: central media/format constants module; 17 duplicate const-defs
  centralized (sector/TS-packet/source-packet); mpls stream-entry + category
  codes named.
- clippy --all-targets -D warnings clean (1.86); fmt clean; 2199 lib tests.
2026-06-26 12:19:24 -07:00
Matthew Jackson fd8abc3684 fix(libfreemkv): rc6 hardening pass — mux timeline/colour/PCR, demux panic sentinel, parser robustness + doc accuracy
Surgical fixes (each with a regression test that fails without the change):

mux/mkv.rs, mux/demux_sink.rs: drive the clip-boundary timeline epoch
off the resolved PRIMARY VIDEO track, not the literal stream index 0.
An M2TS/PMT title can list an audio ES before video, so streams[0] may
be audio; a non-video epoch driver ratchets the frontier and inflates
the timeline. mkv cluster-opening falls back to track 0 for audio-only
titles so they still open clusters.

mux/codec/ac3.rs: correct ACMOD_CHANNELS — acmod=5 (3/1) is 4 channels,
not 3 (was undercounting a 3/1 stream); fix the A/52 Table 5.8 doc.

disc/mod.rs: HDMV coding_type 0x91 (Interactive Graphics / menus) no
longer maps to PGS subtitle — it falls through to Unknown so the PMT/STN
walker drops it instead of surfacing a bogus subtitle track.

mux/videomap.rs + mux/mkv.rs: FVI colour now mirrors the MKV muxer's CICP
precedence (measured CICP authoritative; HDR-driven PQ/HLG transfer
override) via a shared cicp_for_video helper, so the two sinks can't
disagree (HDR10 BT.2020 no longer emits SDR transfer 14).

mux/mkvstream.rs: saturating_add on cluster_ts + rel_ts so an adversarial
CLUSTER_TIMESTAMP near i64::MAX can't overflow/panic before the existing
saturating_mul.

mux/timeline.rs: tighten the tail-straggler clamp so a normal new-epoch
non-video frame leading the sparse video frontier by >3s is not demoted
into the previous clip's epoch.

mux/m2ts_mux/mod.rs: re-stamp PCR per video TS packet (mid-PES), not only
at PES boundaries, so a large UHD I-frame can't open a multi-second PCR
gap; modular 33-bit PTS rebasing so a real 90 kHz clock wrap is not
collapsed to PTS 0 (pre-base frames still floor to 0).

io/byte_prefetcher.rs, sector/prefetched.rs: wrap the producer feed loop
in catch_unwind and emit a typed error sentinel on panic, so a mid-stream
producer panic is not read as a clean EOF at the demux boundary (which
would silently truncate the mux).

mux/codec/h264.rs: extend HIGH_PROFILES to the full ISO/IEC 14496-15 set
that mandates the avcC chroma/bit-depth extension (adds 244 et al.).

Doc/comment accuracy: css/mod.rs (50000 sectors, not scrambled-sectors),
aacs/decrypt.rs (decrypt_unit already-clear path), ifo.rs (TT_SRPT at
0xC4), css/lfsr.rs (LFSR0 24-bit; TAB1-then-XOR cipher; real scramble-flag
predicate), disc/read_error.rs (for_sweep does bounded transient retries).

Skipped: keydb.rs SSRF guard (low/latent, no live caller) — a hard
loopback block breaks an existing behavioral test that exercises the
header-EOF path over a loopback server; a clean fix needs a resolver test
seam beyond this surgical pass. The sibling keydb_fetch.rs comment fix is
out of scope (freemkv crate).
2026-06-25 23:39:03 -07:00
Matthew Jackson 2307f0d79a feat(mux): emit HDR10 static metadata from HEVC SEI
Parse the two HDR10 HEVC SEI messages and emit the corresponding
Matroska Colour metadata, only when actually present in the bitstream
(SDR / no-SEI tracks omit it; nothing is fabricated).

Parse (Rec. ITU-T H.265 Annex D):
- Mastering Display Colour Volume SEI, payloadType 137 (D.2.28):
  display_primaries_x/y[3] (SEI order G,B,R), white_point_x/y
  (0.00002 units), max/min_display_mastering_luminance (0.0001 cd/m²).
- Content Light Level Info SEI, payloadType 144 (D.2.35):
  MaxCLL / MaxFALL (cd/m² integers).
HevcParser::scan_sei walks the sei_rbsp ff-extension payloadType/
payloadSize coding and de-emulates (00 00 03) before reading, reusing
the existing strip_emulation_prevention helper. Both SEI are required
before any metadata is surfaced; SEI NALs still pass through unchanged.

Carry: the measured Hdr10Metadata rides PictureInfo (the same per-coded-
picture seam FieldOrder uses), flowing through from_codec_frame onto
PesFrame.coding to the deferred-muxer activate path, where
apply_coding_to_track stamps it on the video track before the header is
written. Set only when both SEI were seen.

Emit (RFC 9559 / Matroska): new Colour children in ebml.rs
(MasteringMetadata 0x55D0, Primary R/G/B + WhitePoint chromaticity
0x55D1..0x55D8, Luminance max/min 0x55D9/0x55DA, MaxCLL 0x55BC,
MaxFALL 0x55BD). write_hdr10 converts chromaticity SEI int × 0.00002 →
Matroska float, luminance SEI int × 0.0001 → cd/m² float; MaxCLL/MaxFALL
are uints verbatim. SEI primary index 0/1/2 (G/B/R) mapped to the
Matroska R/G/B element layout. Emitted only when hdr10 is present.

Tests: SEI parse with exact raw values, requires-both-SEI, SDR omission,
and emulation-prevention stripping (hevc.rs); muxer emit with exact unit
scaling + SDR omission of MasteringMetadata/MaxCLL/MaxFALL (mkv.rs);
apply_coding_to_track HDR10 plumbing (mkvstream.rs).
2026-06-25 21:59:43 -07:00
Matthew Jackson 3d2f115691 test(disc): pin the -t 1 = main-feature contract (DVD); fix 2 test clippy nits
Owner-flagged invariant: freemkv -t 1 ALWAYS selects the main feature because the
CLI's title 1 maps to titles[0] and the list is ordered by canonical_title_order
(main feature first). Adds a DVD-shaped contract pin asserting titles[0] is the
movie after sorting (a regression there is a title-ordering bug, not a remux issue),
complementing the existing branching-UHD / normal-disc order tests.

Also clears two pre-existing test-only clippy nits surfaced under --all-targets:
unused `lba` in ClearStubReader::read_sectors, and an unneeded `mut` on the
h264 population test's closure.
2026-06-25 21:42:52 -07:00
Matthew Jackson 9db9b9bdf6 test: real-executing coverage for mux codecPrivate/DefaultDuration, sweep damage-jump, patch watchdog clock seam, and AACS CBC KAT
mux/mkv: assert emitted CODEC_PRIVATE bytes verbatim for H.264/HEVC/VC-1/
MPEG-2 (direct TrackEntry child, not nested in Video) and DefaultDuration ns
for all eight frame rates, read back out of a real MkvMuxer.

disc/sweep: end-to-end Disc::sweep against a synthetic MockReader with an
injected bad region, asserting the resulting mapfile marks the clean lead
Finished and the failed batch + zero-filled skip-ahead gap NonTrimmed,
proving the Pass-1 damage-jump engaged.

disc/patch: introduce a minimal clock seam (fn() -> Instant on the internal
PatchLoopState, defaulting to Instant::now) so the per-range and whole-pass
watchdogs are deterministically testable; public API and callers unchanged,
production behavior identical. Add tests that advance a fake clock to trip the
range budget and whole-pass stall predicate.

aacs: add an AES-128-CBC known-answer test for aes_cbc_decrypt using the
published NIST SP 800-38A F.2.2 vector (blocks 1..3 exact; block 0 via the
documented fixed-AACS-IV substitution).
2026-06-25 21:37:20 -07:00
Matthew Jackson 3f38ee6fd5 mux: reconcile fvi:// video-index sink onto rc6
Port + adapt the freemkv native per-picture video index (FVI) from the
old feat/fvi-sink branch onto rc6's codec-agnostic PictureInfo model.
This is a surgical adaptation, not a merge.

Adaptations (fvi_sink.rs, videomap.rs, tests/fvi_pipeline.rs):
- Retarget from the removed crate::mux::codec::mpeg2::PictureInfo (raw
  public fields) to rc6's authoritative crate::mux::codec::PictureInfo
  in codec/coding.rs, via its accessors.
- type from coding_type() -> CodingType{I,P,B}; emitted for ANY frame
  that carries coding (every video codec now fills it), with the
  keyframe-flag I/P fallback only when coding is absent.
- Replace the mpeg2-only tff/rff/progressive members with codec-agnostic
  members derived through the accessors: field_order (tff/bff/progressive)
  and progressive, emitted ONLY when the codec measured the signal
  (Option::Some) and omitted otherwise; plus nb_fields.
- Test fixtures rebuilt via PictureInfo::mpeg2(CodingType, Mpeg2Coding{..})
  / coding_type_only(..); added measured_cicp: None to VideoStream
  literals for rc6's struct.

Honesty decision (key / random-access):
- The codec-agnostic PictureInfo carries NO GOP-closure (no closed_gop /
  gop_start), so key is set from the frame's intra / decode-restart flag
  (frame.keyframe == coding.keyframe() for video), NOT a fabricated
  clean-RAP claim. The old gop member is honestly omitted. FVI_FORMAT.md
  is updated to document this as a limitation: key is an intra picture /
  parser-flagged decode-restart point; MPEG-2 open-GOP clean-RAP precision
  (closed_gop) is not currently distinguished. §7.1 rewritten for the
  new field_order/progressive/nb_fields members.

Wiring:
- mux/mod.rs: pub(crate) mod fvi_sink; pub(crate) mod videomap
  (#[allow(dead_code)] on videomap — the VideoMap accumulator is staged
  for side-channel reuse, sink builds records directly); pub use
  fvi_sink::FviSink.
- mux/resolve.rs: add the fvi:// output scheme to StreamUrl, parse_url,
  scheme(), path_str(), input() (write-only reject) and output()
  (constructs FviSink), mirroring the mkv:///demux:// patterns.

Provenance fix surfaced by the end-to-end test:
- pipelined_stream::consume_ps was dropping the PS demuxer's byte-exact
  source stamp (source: None) when rebuilding PesPacket, so PS/DVD-path
  frames reached the mux/index with no provenance (FVI src null). Carry
  ps.source through, matching the TS path; the real-pipeline fvi test now
  sees the stamped src sectors.

Gate: cargo +1.86 fmt + clippy --lib -D warnings clean; cargo +1.86 test
--lib (2182 passed) and --test fvi_pipeline (2 passed); precommit.sh
libfreemkv green.
2026-06-25 21:18:15 -07:00
Matthew Jackson 350df85439 mux/vc1: measure coding type from progressive PTYPE; carry source
Honest PictureInfo population for VC-1 — completes the set (MPEG-2, H.264, HEVC,
VC-1 all populate now).

- Read the advanced-profile sequence header INTERLACE flag (SMPTE 421M §6.1.1,
  bit 41), de-escaping emulation-prevention bytes as parse_vc1_resolution does.
- For a PROGRESSIVE sequence, decode the picture-layer PTYPE VLC (0=P, 10=B,
  110=I, 1110=BI→I, 1111=Skipped→P) from the first bits after the frame start
  code. For INTERLACED (FCM/FPTYPE precede PTYPE) or simple/main/unknown, decline
  → coding stays None: honestly absent, never read at a wrong bit offset.
- Set coding = coding_type_only(...) and source = pes.source; field order is not
  decoded, so field_order() is honestly None.
- Reuses the shared startcode::BitReader (no new primitive).
- Tests: progressive I/P/B from real PTYPE VLCs + source carry + field-order
  absence; interlaced honest-decline.
2026-06-25 20:50:36 -07:00
Matthew Jackson 62a013d906 mux/hevc: measure coding type from slice_type; reuse shared BitReader
Honest PictureInfo population for HEVC, on the same principle as H.264.

- Consolidate the bit reader: hevc.rs had its own BitReader (used by the SPS
  parser); h264 just gained one in startcode. Promote startcode's to the single
  shared pub(crate) reader (adds read_bits) and delete hevc's copy — one proven
  primitive, reused (SPS parse + both slice-type decoders).
- hevc: decode slice_type from the first coded slice's slice_segment_header
  (H.265 §7.3.6.1) → I/P/B (§7.4.7.1). The offset to slice_type depends on
  num_extra_slice_header_bits, which lives in the PPS — so we parse it from the
  ACTIVE PPS (§7.3.2.3) and only measure slice_type when that PPS is known.
  With no active PPS we decline rather than guess: coding stays None, honestly
  absent. Set coding = coding_type_only(...) and source = pes.source; field
  order (pic_struct SEI) is not decoded, so field_order() is honestly None.
- Tests: I/P/B from real slice headers, source carry, field-order absence, and
  the no-PPS honest-omission case.
2026-06-25 20:44:15 -07:00
Matthew Jackson 7db0a154e9 mux/h264: measure coding type from slice_type; carry source provenance
Honest PictureInfo population for H.264 — the foundation that lets the video
index claim a frame's type and be believed (a faked "P" on a B-frame would make
the index confidently wrong).

- startcode: shared minimal MSB-first BitReader with Exp-Golomb ue(v), reused by
  H.264 (and next HEVC). Documents the emulation-prevention caveat: only the
  leading slice-header fields are read, where 00 00 03 cannot intervene.
- h264: decode first_mb_in_slice + slice_type (H.264 §7.3.3) from the first
  coded slice and map to I/P/B (§7.4.3 Table 7-6; SP→P, SI→I). Set
  coding = PictureInfo::coding_type_only(...) and source = pes.source. Field
  order is NOT decoded here, so field_order() stays None — honestly absent,
  never guessed.
- Tests: I/P/B from real ue-encoded slice headers, source carry, field-order
  absence; BitReader Exp-Golomb table + truncation.
2026-06-25 20:38:09 -07:00
Matthew Jackson 2a11844cad mux: codec-agnostic PictureInfo + provenance; measure field order, never guess
Carry per-picture truth and byte-exact source provenance THROUGH the stream so
the muxer (and the upcoming video index) read MEASURED facts instead of
assuming them. Honest data in, honest data out.

- codec/coding.rs: codec-agnostic PictureInfo (CodingType / FieldOrder + the
  accessors field_order/coding_type/nb_fields/progressive/keyframe). Each codec
  folds its raw signals in; consumers use only accessors, never branch on codec.
- mpeg2: builds PictureInfo from the picture coding extension and carries it +
  SourcePos (source_marks, parallel to pts_marks) on every emitted frame.
- pes / codec::Frame: additive `coding` + `source`, forwarded through the
  highway; None for audio/subtitle and the network/stdio deserialize hop.
- mkvstream: DEFER muxer construction until the first coded picture, set the
  video track's FieldOrder from the MEASURED value, THEN write the header —
  right the first time, no guess, no seek-back. An interlaced track that arrives
  with no measured order is LOGGED loudly and left UNDETERMINED, never faked.
- mkv: MkvTrack::video no longer guesses TFF (a bitstream property the scan
  cannot know is UNDETERMINED at build). Removed VideoStream::top_field_first
  (the dead scan-time guess) crate-wide.
- Tests: parser population (every PictureInfo facet + per-PES source carry) and
  mux-stream consumption (measured -> correct; missing -> UNDETERMINED, not
  faked). Two obsolete tests updated only after confirming (their own comments)
  they existed to enforce the deleted hardcoded-TFF.
2026-06-25 20:13:55 -07:00
Matthew Jackson 051e5b5f25 mux/mkv: spec-conformance fixes (field order, CICP, VobSub idx, SeekHead, AC-3)
- FieldOrder now derives from the bitstream's measured top_field_first
  (Some(true)→TFF, Some(false)→BFF) instead of hardcoding TFF for all
  interlaced content; falls back to TFF when unmeasured. Adds
  VideoStream::top_field_first; DVD/BD scan sets None with a precise
  TODO(spec) for parser→title plumbing.
- CICP (matrix/transfer/primaries/range) now prefers measured CICP from
  the bitstream (VideoStream::measured_cicp) over the coarse ColorSpace
  enum, so the container stops assuming a colour space the stream may
  contradict. Enum remains the fallback.
- VobSub S_VOBSUB CodecPrivate now emits a `size: WxH` line ahead of the
  palette per the .idx format so players place/scale subs correctly.
- SeekHead: when zero cues are written, the CUES Seek entry is Voided
  instead of leaving a dangling pointer to the Cues offset (now Tags/EOF).
- AC-3 Channels back-patch offset is captured from the writer instead of
  the hardcoded chan_elem_pos+2 (decoupled from the VINT width choice).
- Hoisted inline CICP codes and the dvcC fourcc to named constants citing
  ITU-T H.273 / RFC 9559; fixed the stale FieldOrder comment.
- DefaultDuration vs pulldown: precise TODO(spec) left (needs the same
  parser→title channel as top_field_first).

Tests: BFF-from-measured-flag, measured-CICP-overrides-enum, VobSub size:
line present/omitted, zero-cue SeekHead Void. precommit (1.86) green.
2026-06-25 18:40:30 -07:00
Matthew Jackson 3e2740d815 wip: rc6 VFR/DVD/CSS base (held for bulletproofing + split) 2026-06-25 18:17:08 -07:00
Matthew Jackson ef5a1b9ff7 demux: solidify sink — reuse canonical primitives, fix 3 bugs
Delete re-implementations in the demux:// sink and wire to proven helpers;
keep only genuinely-new functionality.

- AnnexB reframing: delete the sink's local length_prefixed_to_annexb (it
  break'd on a zero-length NAL, dropping the rest of the access unit) and
  call the canonical append_length_prefixed_as_annex_b in mux::hevc, which
  skips just the empty NAL.
- HEVC param sets: delete hvcc_param_sets; reuse hvcc_to_annex_b.
- avcC param sets: hoist as the new canonical avcc_to_annex_b in mux::hevc,
  next to hvcc_to_annex_b (the symmetry point); the sink calls it.
- PGS .sup: emit a synthetic clear display set (empty PCS + END) at
  pts + duration_ns so subtitles time out instead of lingering to EOF.
- TimelineContinuity: move verbatim into the shared mux::timeline module
  (with the prev_offset straggler-remap intact) and use it from both the
  MKV muxer and the demux sink; delete the sink's drifted TimelineRebase
  copy (which lacked the straggler branch).
- VobSub .idx: emit the conventional 'id: <lang2>, index: 0' line mkvmerge
  reads to assign the subtitle language; palette reuse unchanged.
- output(): seed DemuxOptions.base from title.playlist when non-empty.

New constants for the PGS clear-segment framing and avcC header cite the
public HDMV PGS (BD-ROM Part 3) and ISO/IEC 14496-15 specs.

Tests: a zero-length NAL mid-frame no longer truncates the AU; a frame with
duration_ns produces a .sup clear segment; existing demux tests stay green.
2026-06-25 17:58:49 -07:00
Matthew Jackson 6c55f0cf20 mux: highway-level regression — DVD video keyframe + duration survive the pipelined path
The DVD seek-index concern is that the mux highway
(PsDemuxer -> PipelinedPesStream codec parse -> frame out -> muxer)
might drop the keyframe flag or per-frame duration that Mpeg2Parser sets
on each Frame, which would stop the muxer's cluster/cue open
(keyframe && track 0) from ever firing and leave a DVD MKV with many
clusters and zero cues.

Add dvd_highway_preserves_video_keyframe_and_duration: drives a real
Mpeg2Parser through PipelinedPesStream via DemuxBatch::Ps batches (one
PTS-stamped MPEG-2 PS video PES per GOP, decode-order I + P/B), reads the
frames back through the highway's read(), and asserts the GOP-opening
I-frames arrive as keyframes, every frame keeps its duration (BlockGroup
path), and video routes to track 0. Guards from_codec_frame's
keyframe/duration propagation across the demux-thread + recycled-buffer
highway, which the prior cue tests (codec parser straight into the muxer)
did not exercise.

Verified against the real current-VFR-build output
(Greenland-feature.mkv, today): 3365 clusters / 3365 CuePoints, all
BlockGroups, cues resolve to clusters and track the video — the highway
preserves the flags and the seek index is complete.
2026-06-25 16:58:04 -07:00
Matthew Jackson 95c3c0486b mkv: regression test — DVD VFR/BlockGroup output gets per-cluster cues
The existing cue tests (cue_count_equals_cluster_count,
cue_positions_resolve_to_clusters, cue_times_match_cluster_timestamps)
all feed frames with duration_ns=None, so they exercise only the
SimpleBlock write path (UHD/HEVC). DVD MPEG-2 video is now VFR: every
coded picture carries a per-frame duration_ns=Some(..), so it is written
as a BlockGroup, not a SimpleBlock. That cue path was untested.

Add cue_count_equals_cluster_count_blockgroup_vfr, which drives the real
Mpeg2Parser end-to-end (decode-order frames, non-monotonic B-frame
display PTS, telecine field durations) into the muxer and asserts the
output is wholly BlockGroup (no SimpleBlock) and that the Cues index has
exactly one cue per cluster, with every cue resolving to a real cluster.
Guards the DVD seek index (scrub/fast-forward) against regressing to the
chapter-seek-only, zero-cue state.
2026-06-25 16:32:32 -07:00
Matthew Jackson 73b1bb0509 demux: add demux:// per-track elementary-stream sink
New write-only pes::Stream sink that taps the per-track PesFrame stream
(the seam right before MKV muxing) and writes each track to its own
elementary-stream file, plus chapters and per-audio-track delay metadata.
Purely additive — the MKV mux path is untouched.

- mux/demux_sink.rs: DemuxSink + EsWriter dispatch. Pass-through for
  codecs whose Frame.data is already standalone ES (MPEG-2, VC-1, AC3/
  E-AC3, DTS/DTS-HD, TrueHD, LPCM). Non-trivial writers:
  - AnnexBWriter: reframes hvcC/avcC 4-byte-length-prefixed NALs to
    Annex-B and prepends VPS/SPS/PPS parsed out of the codec_private
    configuration record (HEVC .hevc / H.264 .h264).
  - PgsSupWriter: rebuilds the HDMV 'PG' segment framing the parser
    strips, with 90kHz PTS/DTS (.sup).
  - VobSubWriter: writes raw SPUs to .sub and synthesizes the .idx
    sidecar (palette + per-SPU timestamp/filepos).
  - Delay-in-filename (mkvmerge-readable 'DELAY <n>ms') + chapter XML/OGM
    export. TimelineRebase ports the MKV muxer's seamless-branch epoch
    logic so per-track ES timestamps stay continuous across clip joins.
- mux/resolve.rs: StreamUrl::Demux variant + scheme/path_str/parse_url/
  input(write-only)/output arms.
- mux/mod.rs: module + public type re-exports.

16 unit tests: Annex-B reframing, hvcC/avcC param extraction, delay
sign/rounding + mkvmerge-regex match, PGS .sup framing, VobSub .idx
synthesis, chapter XML/OGM, timeline rebase, and end-to-end file-keying
by track + track selection.
2026-06-25 16:23:37 -07:00
Matthew Jackson 86de7fe931 css: skip clear/uncrackable extra titles instead of failing the whole mux
A genuinely-clear or uncrackable extra title (a tiny menu/nav stub) no
longer poisons a multi-title rip with a false CssKeyMissing (E7023).

- decrypt_keys_for_title_checked: re-crack a non-overlapping VTS via
  crack_key_outcome and report title_is_clear when the title's own
  extents show no scrambling. A genuinely-clear stub on an otherwise-CSS
  disc needs no key.
- ensure_title_decryptable: pass a clear stub without a key; a scrambled-
  but-uncrackable title still hard-fails with CssKeyMissing.
- is_scrambled_pack: hardened scramble-evidence gate for the crack scan —
  requires the MPEG-PS pack-start signature before trusting the 0x14
  scramble bits, so a clear stub with stray 0x14 bits can't flip
  saw_scrambled. The descramble loop keeps the looser is_scrambled.
- mux/resolve: ISO per-title gate routes through the clear-aware check.
2026-06-25 13:43:04 -07:00
Matthew Jackson 0993d60e8c progress: expose bytes_retryable so 'lost' counts only failed reads
Add PassProgress::bytes_retryable_total (NonTrimmed/NonScraped — failed
and awaiting retry), distinct from bytes_pending_total which also folds
in not-yet-attempted (NonTried) bytes. Set it at every construction site
(Sweep from the snapshot, Patch from stats, 0 for sequential/placeholder
paths). The disc-level 'lost' display in the CLI can now use
unreadable+retryable instead of unreadable+pending, so a healthy
in-progress rip no longer reports its unread remainder as lost.
2026-06-25 13:04:37 -07:00
Matthew Jackson b080fb6e1b Resume plain disc->iso copy after interrupt
Disc::copy gated its entire mapfile-resume dispatch behind
`if opts.multipass`, so a plain (non-multipass) `disc:// iso://`
copy always called sweep_internal(resume=false) — wiping the
mapfile + ISO and re-sweeping from sector 0. The CLI help and
rip_iso examples promise "auto-resumes if interrupted", and the
sweep already flushes a crash-safe mapfile per block, so the only
missing piece was consulting it on re-run.

Lift the mapfile-resume dispatch out of the multipass guard:
clean->no-op, size-mismatch->fresh sweep, NonTried tail->resume
sweep now apply to both modes. The patch (Pass N) dispatch on
retryable bytes stays multipass-only; a plain copy with a
fully-attempted mapfile returns a terminal result instead of
restarting. autorip (always multipass=true) is unchanged.

Add a regression test that an interrupted plain copy resumes the
NonTried tail and does NOT re-read the Finished prefix.
2026-06-25 08:24:00 -07:00
Matthew Jackson d87defef9e libfreemkv: align E7021→E7017 in comments/test-names (keys-but-no-VID is AacsVidUnavailable) 2026-06-25 07:59:54 -07:00
Matthew Jackson 8d8f2fa463 libfreemkv: changelog — rc.6 entry (+ backfill rc.5.3, mark rc.5.2 released) 2026-06-24 23:56:41 -07:00
Matthew Jackson db64d6cea5 test: strengthen AACS resolve-reason + unlocker seam coverage (rc.6 WS3)
Targeted tests for the rc.6 surfaces, strengthening (not duplicating) the
regression tests the rc.6 commits already shipped. No production code changes.

aacs/keys.rs — resolve_keys_with_reason / classify_resolve_failure:
The E7021/E7022 split is already proven end-to-end through the
ensure_decryptable gate (disc/mod.rs). These pin the classifier directly at
the keys.rs seam for the branches the gate test does not reach:
  - processing-keys-only + zero VID -> VidUnavailable (the gate test only
    exercises the device-keys arm of has_derivation_material).
  - VID PRESENT + material -> NoMaterial: a non-zero VID must never be
    reported as VidUnavailable however much material is on hand (the has_vid
    short-circuit; the gate test only uses the zero-VID sentinel).
  - VID present + no material -> NoMaterial.
  - version dispatch: version 1 routes the V10 resolver (stamps V10), any
    other value routes the V20->V21 chain; a resolved disc returns Ok, never
    Err(ResolveFailure).

unlock.rs — Unlocker seam introspection + ordering:
  - matching_name reports the first matching unlocker without running it, and
    is None for an unsupported drive; registered_count grows after a
    registration (monotonic check — the registry is process-wide and shared
    across the unlock tests, so no exact-delta assertion).
  - route_unlock first-registered-match-wins: two unlockers matching the same
    identity, the earlier-registered one runs and the later is never consulted.
2026-06-24 23:40:36 -07:00
Matthew Jackson 7f1827565a fix(extract): anchor AACS unit base per-extent + crash-safety hardening
The dir:// extractor set the AACS unit-alignment base ONCE to the first
extent's start, then read every extent against that single base. For a
multi-extent (fragmented / Long-AD / continuation-ICB) file the second
and later extents start at arbitrary LBAs whose offset from the first
extent is generally not a multiple of 3 sectors, so the first read of
each later extent failed the decrypt-on-read gate
(is_unit_aligned(lba, unit_base)), returned DecryptFailed, and recorded
the whole extent as a zero-filled hole even though the data was readable.
Re-anchor the unit base PER extent (matching mux/disc.rs and
sector/prefetched.rs), so each clip's encrypted region gates on its own
unit grid. Same bug class as the rc.5.2 clip-anchor fix.

Also harden the extract + keydb write paths:
- finalize_file: fsync the .partial after set_len (the truncation runs
  on a second handle the content fsync never touched) and fsync the
  parent dir after rename so the new dirent is crash-durable.
- keydb write_atomic: fsync the parent dir after rename (POSIX dirent
  durability), matching the finalize_file pattern.
- AACS tail batch: document that decrypt_sectors' trailing-partial
  contract already handles the short final unit; no math change.
- decrypt-loss delta loads use Acquire (defensive happens-before if
  file extraction is ever parallelised).
- is_windows_reserved: add CONIN$/CONOUT$/CLOCK$; reserved names are
  now substituted (prefix _) instead of aborting the whole tree walk, so
  a legal Linux-authored NUL.cfg extracts.
- http_get header cap: >= MAX_HEADER_BYTES (was > , one byte over).

Regression tests: multi-extent AACS file (Δ4-sector extents) extracts
both extents with zero loss; focused per-extent alignment-arithmetic
test; reserved-name substitution assertions.
2026-06-24 23:20:15 -07:00
Matthew Jackson 787b4ff54c aacs: split no-key resolution reason into E7021 vs E7022
When key resolution had derivation material (device or processing keys)
but no Volume ID was available to derive the unit key, surface
Error::AacsVidUnavailable instead of the generic NoDiscKey. When there
was no usable key material at all, keep NoDiscKey.

resolve_keys_classical / resolve_keys_v21 still return a bare
Option<ResolvedKeys> (all existing callers unchanged); a new
resolve_keys_with_reason wrapper threads the typed ResolveFailure
(VidUnavailable | NoMaterial) out. decrypt_with uses it; the
ensure_decryptable_keys gate maps a captured AacsVidUnavailable reason to
E7021, otherwise E7022. No decryption math, key derivation, or descramble
logic changed -- only the reason reported on a resolution failure.

Adds ensure_decryptable_aacs_vid_unavailable_vs_no_key proving both
branches (device-keys + zero VID -> E7021; no keys -> E7022).
2026-06-24 22:38:54 -07:00
Matthew Jackson 75be0159e6 v1.0.0-rc.5.3: bump version 2026-06-24 21:50:00 -07:00
Matthew Jackson 8c9672819e libfreemkv: plain-English, source-agnostic AACS scan logs
Rewrite the AACS scan/VID trace lines so a reader understands them without
opening the source: name the real thing (AACS host certificate, Volume ID,
decryption key), say "key source" not "keydb", and describe what happened.
The VID flow is unchanged (unlocker OEM VID → cert handshake → continue); a
missing VID is logged, never fatal. All strings are in tracing macros (the
sanctioned debug-log channel) — no English added to any Error.
2026-06-24 21:47:37 -07:00
Matthew Jackson c9f9b1dd08 libfreemkv: keydb default location is local to the executable
default_path() now resolves <dir of current exe>/keydb.cfg with no OS-config
fallback — freemkv is a portable standalone binary, so the keydb lives next to
it. --keydb still overrides.
2026-06-24 20:46:31 -07:00
Matthew Jackson d94e54e52b libfreemkv: dir:// — decrypted file-tree extraction (Disc::extract_tree)
Sibling of Disc::copy specialized to write per-file instead of a whole ISO
image, decrypting on the way out: walk the UDF tree, read each file's extents
through the shared DecryptingSectorSource (AACS unit-aligned, CSS per-VTS),
strip AACS/, sanitize host paths per component, .partial+rename, 1-shot with
per-file loss accounting (no mapfile; recovery stays the iso:// multipass
path). Reuses UdfFs + the decrypt seam; only the per-file orchestration is new.
2026-06-24 20:46:31 -07:00
Matthew Jackson 7ace1aa3c6 libfreemkv: hard-error when decryption is needed but no key is available
Adds Disc::ensure_decryptable / ensure_decryptable_keys, the single decrypt
gate consulted before any copy or mux. When the source is encrypted and no
key resolved (and not --raw), abort with a typed error and write nothing,
instead of silently emitting ciphertext at exit 0. Unifies the prior ad-hoc
CSS/AACS checks.
2026-06-24 19:33:09 -07:00
Matthew Jackson 20c1035730 v1.0.0-rc.5.2: bump version 2026-06-24 17:14:10 -07:00
Matthew Jackson 8db1d01833 libfreemkv: rc.5.2 changelog — audio routing, AACS clip-anchor, recovery, TopGun, keysources move 2026-06-24 17:12:17 -07:00
Matthew Jackson 3bdf490fce libfreemkv: rc.5.2 SOTL video — full Windows fps, opening-GOP proof, self-sufficient log-level 3
Three Silence-of-the-Lambs (R2 PAL SD-DVD) follow-ups for rc.5.2.

SUB-TASK 1 — Windows Explorer showed 12.5 fps (half) for the 576i25 track.
Root cause: the DefaultDecodedFieldDuration (20 ms field) element rc.5.1
added to "fix" Windows fps did the opposite. With FlagInterlaced=1 +
DefaultDuration=40 ms + DefaultDecodedFieldDuration=20 ms, Explorer halved
to 12.5 fps and MediaInfo flipped to VFR. MakeMKV's correct rip omits the
field-duration element, keeps FlagInterlaced=1 + FieldOrder=TFF +
full-frame DefaultDuration (40 ms), and Explorer shows 25 fps / MediaInfo
CFR. Fix: MkvTrack::video now passes field_duration_ns == 0 so the element
is no longer written; the 1/DefaultDuration = 25 fps signal (the only one
tools trust) is the full-frame value. Interlace signalling (FlagInterlaced,
FieldOrder=TFF) is retained — MediaInfo reads scan type from the MPEG-2 ES
picture coding extension, so it still reports Interlaced / Top Field First.
Tests pin the new TrackEntry elements (element present/absent + values).

SUB-TASK 2 — opening "menu"/still-frame video. Traced the MPEG-2
opening-GOP path; the wrong/last seq header and PTS-floor-to-0 hypotheses
are RULED OUT with file:line evidence: codecPrivate is the FIRST sequence
header (read once at headers-ready, mkvstream.rs:115 + pipelined_stream.rs:289),
DVD VOBU structure guarantees each title opens on seq header + I-frame (no
mid-GOP open), the parser back-anchors leading still-frames to the disc's
real timeline (mpeg2.rs:296-303), and the muxer anchors base on the opening
keyframe's real PTS so the t=0 floor (mkv.rs:963) never corrupts it.
Regression tests pin all three (parser + muxer level).

SUB-TASK 3 — make --log-level 3 self-sufficient (diag.rs + minimal hooks).
(a) dump the ACTUAL MKV TrackEntry elements written per track
(tag=mkv.track: FlagInterlaced, FieldOrder, DefaultDuration, field duration,
Display dims, codecPrivate hex) so Windows-fps-class metadata is verifiable
from a log alone. (b) capture the first ~100 coded frames per track (raw)
to <output>.opening.bin with a per-frame summary line (tag=mkv.opening.frame:
track, key/delta, size, PTS) so opening-GOP/menu issues are diagnosable from
a future log without the disc. Both gated to log-level 3; normal runs open
no side file and record nothing.

CI gate (Rust 1.86): fmt --check, clippy -D warnings, and test --tests all
green.
2026-06-24 17:04:56 -07:00
Matthew Jackson cec217741b fix(dvd-audio-probe): read each sub-stream's true max channel count
The DVD AC-3 sub-stream probe recorded the FIRST decodable frame of each
physical 0x8x sub-stream as its channel count. A DVD feature opens with
logos/warnings whose audio is often a thin 2.0 bed on 0x80 before the
real 5.1 main mix begins a fraction of a second later. The probe locked
onto that opening 2.0 frame and reported 0x80=2, missing the 5.1
entirely (confirmed on Greenland: 0x80's head frames are acmod=2, then
acmod=7+lfe). With no 6-channel sub-stream found, channel-match routing
fell back to the ordinal map — harmless on Greenland, but on a disc where
the 5.1 lives on a non-ordinal sub-stream the wrong-substream bug stays
unfixed.

Fix: scan EVERY 0x0B77 frame of each sub-stream in the probe window and
keep the MAXIMUM channel count (the sub-stream's real main-mix
capability), advancing frame-by-frame via ac3_frame_size so a frame body
can't be mistaken for a new sync. Also bump PROBE_SECTORS 512->1024: the
1 MiB head window saw ONLY 0x80; 2 MiB reliably contains a frame of every
physical sub-stream.

Greenland tag=dvd.substream: before 0x80=2 (only); after 0x80=6, 0x81=2,
0x82=2 — matching the IFO and the decoded output.

Adds probe_reads_max_channels_no_cross_contamination regression test.
2026-06-24 16:55:34 -07:00
Matthew Jackson e4f27a0daf libfreemkv: fix rc.5.2 audit code findings
1. HEVC CRA->BLA false-trigger on 33-bit PTS wraparound
   (src/mux/codec/hevc.rs): the clip-boundary auto-detect compared the
   RAW 33-bit PES PTS against the high-water mark, so a single-clip title
   crossing 2^33->0 (~26.5h) false-armed pending_clip_boundary and rewrote
   a legitimate in-clip CRA(21)->BLA_W_LP(16), dropping valid RASL pictures
   (visible corruption) and breaking the single-clip byte-identical
   guarantee. Now unwrap the PTS onto a monotonic 64-bit timeline first
   (a near-full-period backstep is a wrap: add 2^33, update the watermark,
   do not arm). Regression test cra_after_33bit_pts_wrap_not_rewritten;
   the genuine-clip-join test still passes.

2. Single-pass recovery read bypassed the transport-failure abort
   (src/mux/disc.rs): the line-442 short-circuit only inspected the 10s
   read res. A transport failure (status 0xFF, wedged USB bridge) on the
   60s recovery read fell into the skip_errors branch and zero-filled/
   advanced, marching the disc at one bridge-recovery per probe
   (run-forever, hard rule #2). Re-check the recovery error for
   is_scsi_transport_failure() before the skip block and abort with
   Error::DiscRead. Test transport_failure_on_recovery_read_aborts_even_with_skip_errors.

3. Recovery-read SUCCESS branch had no coverage (src/mux/disc.rs tests):
   added RecoverableReader (errors when recovery=false, succeeds when
   recovery=true) and test recovery_read_success_muxes_recovered_data_no_skip
   driving fill_extents to the size-1 bottom-out and asserting the recovered
   data is muxed (counters advance, no skip).

4. TrueHD channel-correction probe omitted set_unit_base
   (src/disc/mod.rs correct_truehd_channels): the probe read via a
   DecryptingSectorSource without anchoring the AACS unit-alignment gate,
   so it degraded to absolute start_lba % 3 and returned DecryptFailed on a
   non-3-aligned extent, silently understating Atmos/7.1 as 5.1. Now call
   set_unit_base(ext.start_lba) before the probe read (no-op for CSS/None).

5. is_unit_aligned lba<unit_base latent trap (src/aacs/decrypt.rs):
   wrapping_sub mis-gated when lba < unit_base (2^32 == 1 mod 3). Switched
   to saturating_sub (clamps offset to 0, a unit boundary) and pinned the
   contract with is_unit_aligned_lba_below_base_is_well_defined plus
   is_unit_aligned_relative_to_base.

cargo +1.86 fmt --check / clippy -D warnings / test --tests all green.
2026-06-24 16:31:28 -07:00
Matthew Jackson 5b1009df71 dvd: route AC-3 audio to the physically-correct sub-stream by probed channel count
Fixes the "Silence of the Lambs" R2 PAL wrong-substream rip: the feature's
IFO declares one 5.1 AC-3 stream, but the scan assigned it the on-wire
sub-stream id 0x80 purely by per-codec ordinal (ifo::assign_audio_sub_stream_ids).
On this disc the physical 0x80 carries the 2.0 down-mix and the 5.1 main mix
lives at a different 0x8x sub-stream, so the rip muxed 2.0 while labelling it
"Dolby Digital 5.1" (the acmod fixup in mkv.rs then corrected only the Channels
element, surfacing the mismatch as the "IFO claimed 6 but acmod says 2" warning
— too late to re-route).

New src/disc/dvd_audio_probe.rs probes each physical AC-3 sub-stream's real
channel count from the head of the feature (the acmod/lfeon of its first frame
after the 0x0B77 sync) and re-routes each IFO-declared AC-3 stream onto the
physical sub-stream whose actual channel count matches the declared count,
instead of trusting the ordinal. Wired into both mux demux paths
(DiscStream::new and resolve::build_iso_pipeline) over the decrypting reader,
so it works on CSS discs and the autorip ISO-remux path alike. Bounded
512-sector best-effort read; an empty/unreadable probe degrades to the original
ordinal mapping (no regression on normal discs).

The cell selection is left unchanged: the feature's cell 0 (cat=0x02, 302.4s)
is chapter 1 of the movie (matches MakeMKV's chapter map and 1h53 duration
exactly), so it must NOT be dropped — the perceived "wrong video at the start"
was the wrong 2.0 audio over the opening, the same root cause.

Diagnostics (--log-level 3): new tag=dvd.substream rows dump the ACTUAL acmod
channel count of each physical 0x8x sub-stream read from the VOB, and the
per-cell tag=dvd.cell verdict now spells out the keep/skip reason. With the
existing tag=dvd.aattr (IFO declared sub_id + channels) a bug log alone now
shows whether the ordinal 0x80 really carries the declared layout — no disc
needed to diagnose this class.

expose ac3::find_ac3_sync as pub(crate) for the probe.
2026-06-24 16:28:21 -07:00
Matthew Jackson 1fb9463eaf mux/hevc: auto-detect non-seamless clip boundary, rewrite splice CRA→BLA
The "TopGun bug" (Top Gun 1986 UHD, DV Profile 7 dual-layer): the auditor
flags the rip `corrupt` with a flood of HEVC "Could not find ref with POC N"
decode errors (POC 114/210/228/234/240/246/252/318/336/354/372/399 — one
cluster per clip join).

Root cause: the title is multiple .m2ts clips joined at non-seamless
boundaries (mpls connection_condition 0x05/0x06), read as one concatenated
stream. Each next clip opens with a CRA whose RASL leading pictures reference
pre-join frames gone after concatenation. hevc.rs already has the spec remedy
(rewrite splice CRA_NUT 21 → BLA_W_LP 16 so a decoder sets NoRaslOutput and
drops the dangling RASL), behind mark_clip_boundary() — but nothing ever
called it: connection_condition is not plumbed through the threaded mux
pipeline, so the whole CRA→BLA mechanism was dead code.

Fix: detect the boundary inside the parser from the bitstream. Each clip
carries its own PES PTS base, so a non-seamless join is a large backward
PTS reset. HevcParser::parse now tracks a PTS high-water mark and, on a
backward step beyond 3 s (270000 ticks @ 90 kHz — mirroring the mux-side
DISCONTINUITY_BACKSTEP_NS), arms the existing CRA→BLA rewrite for the new
clip's first IRAP. Self-contained: works for every mux path, no cross-thread
plumbing. The 3 s threshold sits above any HEVC B-frame reorder dip and far
below any clip duration, so it never false-triggers in-clip; single-clip /
seamless titles never arm it and stay byte-identical.

Adds regression test cra_at_auto_detected_pts_backstep_rewritten_to_bla
(in-clip dip must not trigger; splice CRA after a backward reset must become
BLA; one-shot). hevc suite 55/55, mux suite 780/780.
2026-06-24 15:41:26 -07:00
Matthew Jackson 23414d289d libfreemkv: rc.5.2 DVD test coverage — depth-aware mux, colour codes, CSS scan
Implements the rc.5.2 quick-units list from the DVD coverage audit and
corrects the "passes-but-encodes-the-bug" tests that could not
distinguish correct from wrong behaviour.

New tests (each with the bug it guards):

mux/mkv.rs
- field_duration_is_direct_trackentry_child_not_in_video: depth-aware
  check that DefaultDecodedFieldDuration (and DefaultDuration) are direct
  TrackEntry children, NOT nested in the Video master. Replaces the flat
  find_id byte-scan that passed either way. Adds master_children /
  first_track_entry depth-walking helpers.
- pal_576i_emits_bt470bg_colour_codes / ntsc_480i_emits_smpte170m_colour_codes:
  assert the actual CICP tuples written into the MKV Colour master —
  PAL (5,5,5,1) vs NTSC (6,6,6,1) — not just stream-layer ColorSpace.
- ntsc_480i_field_order_is_tff_and_encoded: pins NTSC 480i hardcoded TFF
  and its ~33.37ms/16.68ms frame/field durations, asserting the encoded
  FlagInterlaced/FieldOrder bytes (480i was never exercised before).

ifo.rs
- video_attr_absolute_bytes_pin_real_layout: drives parse_video_attr with
  HARDCODED real DVD-Video bytes (PAL/NTSC x 4:3/16:9, plus mpeg_version
  in bits 7-6) instead of v_atr_byte, so a co-edit of the shift constants
  can't re-seed the PAL-as-NTSC bug. Anchors that permitted_df bits (1-0)
  are not read as the TV system.

disc/dvd.rs
- scan_dvd_titles_lpcm_routes_to_a0_pid_range: LPCM (coding 4) → sub-id
  0xA0 → PID 0xBDA0, disjoint from the AC-3 0xBD8x space, channels kept.
- scan_dvd_titles_multiple_vobsub_tracks_distinct_pids: three VobSub
  tracks → distinct 0x20+ordinal PIDs, per-language, shared palette.

css/mod.rs
- crack_outcome_reaches_cracked_with_span: drives the full crack scan to
  CrackOutcome::Cracked via a Stevenson-crackable synthetic sector and
  asserts crack_span recording (the Cracked branch was never exercised).
- recrack_succeeds_on_other_vts_extents: per-VTS re-crack SUCCESS path.
- all_locked_synthetic_iso_yields_css_key_missing_signal: all-locked
  multi-extent ISO → ScrambledUncracked, the signal the scan converts to
  css_error = Some(CssKeyMissing).

Corrected fixtures (passes-but-encodes-the-bug):
- scan_dvd_titles_mixed_audio_codecs_distinct_pids: real channel nibbles
  (AC-3 5.1 = 6ch, DTS 2.0 = 2ch) replacing the 1ch placeholders; asserts
  channel counts and exact canonical PIDs (0xBD80 / 0xBD88).
- ebml.rs FieldOrder comment: drop the stale "PAL DVD (576i) is
  bottom-field-first" line that contradicted the TFF-for-all code.
2026-06-24 15:41:26 -07:00
Matthew Jackson 828aed1ecb libfreemkv: clip-anchored AACS unit gate + consolidate key mechanism
The AACS unit-alignment gate measured `lba % 3` against absolute disc LBA 0,
but aligned units are anchored at each clip's encrypted-region start. A clip
whose start_lba is not 3-aligned had its readable units wrongly rejected with
"Decryption failed" (the big-title-only failure on some Blu-rays). One
canonical clip-anchored helper (`aacs::is_unit_aligned`) is now the single
source of truth for the decrypt-on-read gate; both mux read paths set the
per-extent `unit_base = start_lba` via a new `SectorSource::set_unit_base`.

Also moves key *mechanism* into the library: the encrypted sample reader
(`read_encrypted_units`) and the candidate-key resolution loop
(`resolve_and_apply`) now live here, so a key source is purely a lookup.
Regression test covers a clip based at a non-3-aligned LBA.
2026-06-24 15:40:50 -07:00
Matthew Jackson 10709fafbf libfreemkv: single-pass disc->MKV recovers marginal sectors before giving up
Single-pass disc->MKV has no Pass N, so its read bottom-out now issues one
bounded recovery read (recovery=true, ~60s ECC) before skipping or aborting,
matching the multipass patch. Fixes a transient/marginal sector surfaced as a
read failure direct-to-MKV while multipass recovered it. One read, not a loop
(hard rule #2); recovered data is used so no bogus-status hole reopens.
2026-06-24 14:58:32 -07:00
Matthew Jackson c4fda5c873 v1.0.0-rc.5.1: bump version 2026-06-24 14:36:29 -07:00
Matthew Jackson ab18f384c7 libfreemkv: rc.5.1 DVD correctness fixes
- CSS: unlock scrambled-sector reads on enforcing drives via bus-auth
  only; classify sense 6F/03 as CSS-locked; early-bail on a fully locked
  scan; gate the AACS handshake off DVD discs.
- DVD first-play menu no longer prepended to the feature: read the title
  VOBS base from vtstt_vobs (0xC4), not the menu VOBS vtsm_vobs (0xC0).
- Interlaced field-duration (DefaultDecodedFieldDuration) written as a
  direct TrackEntry child rather than inside Video, so Windows reports
  the correct frame rate.
- Audio channel count read from the AC-3 bitstream; FieldOrder set to
  TFF; per-track BPS tags.
- Structured disc diagnostics at --log-level 3; reduced per-operation
  log spam.
2026-06-24 14:34:55 -07:00
Matthew Jackson 18d1f16582 Add Contributor Covenant v2.1 Code of Conduct 2026-06-24 10:44:58 -07:00
Matthew Jackson 1aea0d81a5 v1.0.0-rc.5: bump version 2026-06-24 10:27:30 -07:00
Matthew Jackson c197b5236f Preserve interlaced scan type in label and MKV output
Interlacing is detected upstream (PAL DVD -> R576i) but was dropped in
two places: the video label hardcoded a 'p' suffix, and the muxer never
wrote any scan-type flag, so MediaInfo inferred progressive and reported
576p for a 576i source.

- Add Resolution::is_interlaced() for the R*i variants.
- generate_video_label now branches i/p for the heights that can be
  interlaced (1080, 576, 480) instead of always emitting 'p'.
- MkvTrack carries interlaced + field_order; the video serializer emits
  FlagInterlaced (0x9A; 1=interlaced, 2=progressive) and, for interlaced
  content, FieldOrder (0x9D) - bottom-field-first for PAL 576i,
  top-field-first otherwise. Adds the EBML constants.
2026-06-24 09:47:16 -07:00
Matthew Jackson d868d4e720 Fix AC-3 channel count read from wrong nibble in VTS audio attr
The VTS_AST_ATR byte 1 stores (channels - 1) in the low 3 bits, but we
were reading the high nibble. A normal 2.0 AC-3 track has 0x1 in the low
nibble and 0x0 in the high nibble, so we computed 0+1=1 and mislabeled
every stereo track as mono in the muxed Channels element.

Read the low 3 bits instead, and correct the two audio_attr tests that
had encoded (channels - 1) in the high nibble to match the buggy reader.
2026-06-24 09:39:02 -07:00
Matthew Jackson 241a592323 fix: correct stale FAIL_PAUSE_SECS doc claiming uniform sweep+patch application 2026-06-24 05:57:39 -07:00
Matthew Jackson a4a12859db fix: drop BytePrefetcher channel endpoints before join to prevent deadlock
Drop rx and recycle_tx (now Option fields) before joining the producer
thread in BytePrefetcher::Drop. Without this, a non-EOF source fills the
depth-2 forward channel then spins in send_timeout(POLL_INTERVAL) forever
because rx is never drained; join() deadlocks. Adds a regression test
(drop_endless_prefetcher_joins_cleanly) that directly exercises the path.
2026-06-24 05:50:58 -07:00
Matthew Jackson 7d3f413030 fix: correct stale doc on PatchItem::Unreadable — promotion shipped via Mapfile::record, not PatchSink 2026-06-24 05:02:24 -07:00
Matthew Jackson b993320249 fix: correct stale css::crack comment reference to css::crack_key 2026-06-24 04:57:47 -07:00
Matthew Jackson cb96550ba5 fix: remove stale sweep_pipeline.rs cross-references in patch.rs comments 2026-06-24 04:52:04 -07:00
Matthew Jackson 9d2007274c Revert "fix: discard duplicate TS packets in non-PUSI continuation path"
This reverts commit ec35b2bce1.
2026-06-24 01:59:49 -07:00
Matthew Jackson e58ce237e7 fix: enforce WEDGE_FAMILY_COOLDOWN_SECS == ZONE_ENTRY_COOLDOWN_SECS at compile time 2026-06-24 01:52:43 -07:00
Matthew Jackson fc82d98bf1 fix: correct Profile-07 comment — prime_cache(lba) reads lba-3..lba-1, not lba itself 2026-06-24 01:51:21 -07:00
Matthew Jackson d40733d0b1 fix(vc1): assemble keyframe prefix in fixed seq-then-entry order
When a keyframe AU carried an unchanged seq_header (stripped) but a
redefined entry_point (appended), the old append-then-reassert path
produced [entry_point, seq_header] — entry_point before seq_header,
violating SMPTE 421M which requires seq+entry before every RAP.

Replace the single shared prefix Vec + reassert_active() with per-type
temporaries (redefined_seq / redefined_ep) collected during the scan,
then assembled in canonical seq-then-entry order at keyframe time.
Non-keyframes still emit only genuine redefinitions, also seq-before-ep.
Removes the now-unused reassert_active() helper. Adds a regression test
covering the seq-unchanged / entry-redefined trigger case.
2026-06-24 01:47:51 -07:00
Matthew Jackson ec35b2bce1 fix: discard duplicate TS packets in non-PUSI continuation path 2026-06-24 01:45:39 -07:00
Matthew Jackson 4b6d3dda64 fix: correct comment — patch leaves middle NonTrimmed, not NonTrimmed/Unreadable 2026-06-24 01:11:39 -07:00
Matthew Jackson de3e62e6cc fix: correct doc comment in patch pipeline test to say NonTrimmed not Unreadable 2026-06-24 01:10:51 -07:00
Matthew Jackson 3cbbe148d1 fix: remove stale TDD-red comment from BytesRead emission test 2026-06-24 00:50:21 -07:00
Matthew Jackson d2cf068a60 fix: correct hevc.rs doc comments — non-seamless BD join is connection_condition 0x05/0x06 not 0x01
Comments at lines ~63 and ~102 misidentified 0x01 (first-item/seamless) as the
non-seamless trigger and labelled 0x05/0x06 as seamless — inverted vs the BD-ROM
spec and mpls.rs (which documents 1=seamless, 5/6=non-seamless). Corrected all
affected doc blocks; no logic change.
2026-06-24 00:49:22 -07:00
Matthew Jackson dc18dd9e8e fix: use trim_mkb in resolve_vid_only to avoid zeroing unrecognised MKB 2026-06-24 00:48:02 -07:00
Matthew Jackson 9e552916de fix: correct stale comment in patch.rs work-list ranges_with call
The comment at line 404 claimed "every non-Finished range" but the
immediately-following ranges_with call lists only NonTrimmed,
NonScraped, and Unreadable — deliberately omitting NonTried.
Update the comment to accurately reflect the actual status list and
explain that NonTried is excluded because it is handled by a preceding
sweep pass, not by patch.
2026-06-24 00:09:13 -07:00
Matthew Jackson cfc41086ba fix: correct module doc — only Pass 1 routes through handle_read_error, not Pass N 2026-06-23 23:34:02 -07:00
Matthew Jackson c2f6d04198 fix: update stale doc comment in ReadCtx::for_patch — sync is automatic 2026-06-23 22:51:47 -07:00
Matthew Jackson a1aca559cf fix: correct skip_sectors_for_probe doc comment (8x per index, not 2x per 3) 2026-06-23 22:50:40 -07:00
Matthew Jackson e49f21a45a fix: correct module doc watchdog constant name in patch.rs 2026-06-23 22:49:42 -07:00
Matthew Jackson 06f6d8932f Fix rc5 audit findings: keydb doc, pipeline ordering, hot-loop Arc, tests
- keydb.rs: separate default_path()/no_home_dir() doc blocks; correct the
  false XDG lock-step claim (Linux write path uses $HOME, ignores
  XDG_CONFIG_HOME; read-side search also checks XDG_CONFIG_HOME).
- io/pipeline.rs: use Release/Acquire on the abandoned flag so a leaked
  consumer reliably skips close() on weak memory models (ARM64/POWER),
  not just x86 TSO.
- mux/disc.rs: cache the decrypt-loss Arc at construction; lost_bytes()
  no longer clones an Arc per frame on the mux hot path.
- disc/dvd.rs: assert display_aspect mapping for both 16:9 (PAL test) and
  4:3 (NTSC test).
- mux/resolve.rs: extract css_error_aborts() helper and unit-test the
  scrambled-but-uncracked CSS guard (Fix 6) incl. the --raw exemption.
- aacs/keys.rs: add unit tests for mkb_type_raw/mkb_type/mkb_is_uhd and
  MkbType (Category C 2.0 UHD, prerecorded 1.0, no-0x10-record None).
- release.yml: publish job needs [verify, test] so a failing test suite
  blocks crates.io publication.
2026-06-23 19:11:09 -07:00
Matthew Jackson 15802da1db Bump to 1.0.0-rc.4.3 2026-06-23 15:49:25 -07:00
Matthew Jackson ea5ee9dc6b DVD: correct PAL/NTSC, anamorphic aspect, and SD colour
Fix three DVD video-attribute bugs surfaced by a PAL disc detected as
NTSC:

- PAL/NTSC: parse video_format from VTS_V_ATR bits 5-4, not bits 1-0
  (the old mask read permitted_df, so PAL 576i/25fps was mis-detected
  as NTSC 480i/29.97). Named consts replace the magic bit positions.
- Anamorphic aspect: write MKV DisplayWidth/Height from the disc's
  display_aspect (16:9 720x576 -> 1024x576) instead of square pixels,
  so 16:9 DVDs no longer render as 4:3.
- Colour: stamp SD colorimetry (PAL=BT.470BG, NTSC=SMPTE-170M) instead
  of BT.709 (HD).

Adds VideoStream.display_aspect (threaded through every muxer) plus
TvSystem/DvdAspect/ColorSpace plumbing, with regression tests. Removes
the deprecated Disc mux set_halt bridge (use with_halt).
2026-06-23 15:38:49 -07:00
Matthew Jackson b7a8937786 docs: changelog entry for rc.4.2 Windows fixes 2026-06-23 12:49:08 -07:00
Matthew Jackson 36012e026c v1.0.0-rc.4.2: bump version 2026-06-23 12:41:07 -07:00
Matthew Jackson 872d78d89d io: add platform-aware fsync helpers (dir + durable file sync)
Add an io::fsync module with a per-OS split (posix/windows) mirroring the
writeback_file convention, replacing two duplicated dir-fsync copies:

- dir(): POSIX directory fsync; a no-op on Windows, where std cannot open
  a directory as a File and the failed open logged a spurious warning on
  every mapfile write.
- file_durable(): opens the target read+write before sync_all so the flush
  succeeds on Windows, where FlushFileBuffers rejects a read-only handle
  with ERROR_ACCESS_DENIED.

Point the mapfile writer at the shared dir() helper.
2026-06-23 12:38:02 -07:00
Matthew Jackson cfd71eed3b v1.0.0-rc.4.1: bump version 2026-06-23 10:45:42 -07:00
Matthew Jackson b531763cae test(scsi/windows): cross-check all FFI structs + constants vs SDK headers
Audited every #[repr(C)] struct and IOCTL/flag constant in scsi/windows.rs
against the authoritative Windows SDK headers (ntddscsi.h, winioctl.h,
devioctl.h, winnt.h, fileapi.h). All correct except the already-reverted
ScsiPassThroughDirect packing. Add the missing regression guards:
- StoragePropertyQuery layout (STORAGE_PROPERTY_QUERY: 0/4/8, size 12).
- IOCTL/flag constants, with IOCTLs asserted against an independent CTL_CODE
  re-derivation (not a tautological literal) so a mistyped code is caught.
Validated compiling via cargo xwin check --target x86_64-pc-windows-msvc.
2026-06-23 10:44:21 -07:00
Matthew Jackson 2d9a82743a fix(scsi/windows): revert wrong packed(4) on ScsiPassThroughDirect (rc.4 drive-detection regression)
rc.4 added #[repr(C, packed(4))] to ScsiPassThroughDirect on the false premise
that ntddscsi.h wraps SCSI_PASS_THROUGH_DIRECT in #pragma pack(push, 4). It does
NOT — verified against the Windows SDK ntddscsi.h: the struct has no pragma pack
and uses natural alignment. On 64-bit Windows (LLP64) that puts DataBuffer at
offset 24 and the struct at 56 bytes, which bare #[repr(C)] produces and which
DeviceIoControl expects.

packed(4) instead imposed offset 20 / 48 bytes — the layout of the SDK's
SEPARATE 32-bit thunk struct SCSI_PASS_THROUGH_DIRECT32 (VOID* POINTER_32). Using
that on a 64-bit host malformed every IOCTL_SCSI_PASS_THROUGH_DIRECT, so the
INQUIRY in drive enumeration failed and autorip/CLI reported zero drives
('RC4 no longer detects my drive'). rc.3.1 (bare repr(C)) worked for the same
users; this restores that layout.

Replace the tautological packed-layout test (which asserted the same wrong
offsets the struct produced) with one cross-checked against the SDK header:
DataBuffer@24, SenseInfoOffset@32, Cdb@36, size 56. Verified compiling via
cargo xwin check for x86_64-pc-windows-msvc.
2026-06-23 10:40:32 -07:00
Matthew Jackson 23f5ae3665 ci: cache Rust builds with Swatinem/rust-cache to speed up release + CI 2026-06-23 10:01:37 -07:00
Matthew Jackson ad06f3062e v1.0.0-rc.4: bump version 2026-06-23 09:31:56 -07:00
Matthew Jackson 2fcb7d3b77 keysource: add KeySource::label() for source identification 2026-06-23 09:09:06 -07:00
Matthew Jackson 9e2f5f0109 aacs: add MkbType (BD vs UHD generation) accessor API
Expose the MKB Type field (record 0x10) as a typed MkbType enum with mkb_type()
/ mkb_type_raw() / mkb_is_uhd() helpers, so callers can distinguish AACS 1.0
(Blu-ray) from AACS 2.0/2.1 (UHD) discs without poking raw bytes.
2026-06-23 08:19:40 -07:00
Matthew Jackson b5ac505c9d CHANGELOG: add 1.0.0-rc.4 — decrypt-loss accounting, durable writes, Windows SCSI, truthful error causes 2026-06-23 07:22:05 -07:00
Matthew Jackson a7280eef85 Account for decrypt-time loss so partial AACS/CSS failures can't pass as a perfect rip
When a scrambled AACS unit fails to decrypt under every available key
(a missing/wrong CPS sub-key, or a marginal unit that fails the TS-sync
verify), decrypt_sectors restored the original encrypted bytes and
returned Ok with no signal. Those still-encrypted bytes flowed to the TS
assembler, which silently dropped the non-syncing packets with no loss
counter. The only loss accounting was DiscStream's read-error zero-fill
path, so mux reported lost_video_secs=0 for decrypt-dropped content and
the abort gate accepted the rip even under abort_on_lost_secs=0. A rip
missing real video/audio segments was published as a perfect success.

decrypt_sectors now returns the number of bytes in scrambled units that
no key could decrypt. DecryptingSectorSource accumulates that into a
shared counter exposed via decrypt_loss(); both mux pipelines fold it
into lost_bytes() — the inline DiscStream path directly, and the
file-backed highway via PipelinedPesStream sharing the producer's
counter. Restore-to-original is unchanged, so clear nav-files are never
corrupted; metadata-probe callers that don't read the counter are
unaffected. Adds regression tests at the decrypt and decorator layers.
2026-06-23 06:15:24 -07:00
Matthew Jackson ab4c66e9d4 keydb: correct read_capped_to_string doc for non-UTF-8 case
The doc claimed Error::KeydbInvalid for non-UTF-8 input, but the code
returns Error::KeydbParse (KeydbInvalid is reserved for the size-cap
violation). Correct the doc to match behavior and add a regression test
asserting non-UTF-8 yields KeydbParse.
2026-06-23 05:27:01 -07:00
Matthew Jackson 2fbdd28023 keydb: classify server-dropped connection as KeydbConnect, not KeydbParse
In http_get, when the server closes the TCP connection before the HTTP
header block completes (n == 0 on the byte-by-byte header read), or sends
a header block exceeding 64 KiB, the code returned KeydbParse (E8004).
Both are connection/protocol-level faults from the server, not parse
failures of keydb content — the keydb bytes were never received. Return
KeydbConnect (E8000) instead, which already covers TCP-level exchange
failures. A CLI user hitting a transient drop or a redirect target that
immediately closes now sees the correct 'server hung up' diagnostic
rather than 'the downloaded file was malformed'.

Add a regression test that stands up a loopback listener which accepts
then drops the connection before headers, asserting KeydbConnect.
2026-06-23 05:21:46 -07:00
Matthew Jackson aef72b920f fix(scsi/windows): pack ScsiPassThroughDirect to match ntddscsi.h layout
ntddscsi.h wraps SCSI_PASS_THROUGH_DIRECT in #pragma pack(push, 4),
forcing the PVOID DataBuffer field to 4-byte alignment even on 64-bit
hosts. The Rust struct used bare #[repr(C)], so the compiler applied
natural 8-byte pointer alignment and inserted 4 padding bytes after
TimeOutValue. That shifted DataBuffer to offset 24 (SDK: 20),
SenseInfoOffset to 32 (28), and Cdb to 36 (32), and grew the struct to
56 bytes (48). DeviceIoControl reads at the SDK offsets, so every SPTI
ioctl on 64-bit Windows either got rejected or interpreted garbage as
the CDB and DataBuffer pointer.

Add #[repr(C, packed(4))] to ScsiPassThroughDirect and the companion
SptwbDirect (so offset_of!(SptwbDirect, sense) stays correct for
SenseInfoOffset), plus a layout regression test asserting DataBuffer at
offset 20 and a 48-byte struct size.
2026-06-23 05:16:22 -07:00
Matthew Jackson 1a64ba2790 prefetched: test that event_fn fires BytesRead per batch
The prefetch producer thread fires a BytesRead event after every
batch it reads, carrying a cumulative byte count. Nothing asserted
this callback actually fired, so a consumer that passed None for the
event_fn would silently get no progress events. Add a fixture-based
regression test that drives a finite extent through new_with_events,
captures the events, and asserts the cumulative count is
non-decreasing and reaches the full extent size at EOF.

This locks the contract autorip's mux progress bar and soft-stall
watchdog depend on.
2026-06-23 04:54:40 -07:00
Matthew Jackson 81d3adbbb3 aacs: preserve transport-failure errors through the auth handshake
A SEND KEY / REPORT KEY step in the bus-auth handshake mapped every
SCSI error to a cert/key-specific code (AacsCertRejected, AacsCertRead,
AacsKeyRead, AacsKeyRejected, etc.) via map_err(|_| ...). That discarded
the underlying SCSI error, so a transport-layer wedge (bridge crash / USB
disconnect) mid-handshake was reported as 'drive rejected your host cert',
sending operators down a keydb/host-cert dead end for what is really a
replug/power-cycle situation.

Add a handshake_err() helper that keeps the original error when it is a
transport failure (is_scsi_transport_failure) and only substitutes the
handshake-specific code for genuine SCSI rejections. Apply it at every
SEND KEY / REPORT KEY / REPORT DISC STRUCTURE step in both the AACS 1.0
and AACS 2.0 paths. Add a regression test covering both branches.
2026-06-23 04:20:54 -07:00
Matthew Jackson 3ce5929fa5 disc: warn when READ CAPACITY fails instead of silently using 0 sectors
read_udf treated a READ CAPACITY SCSI failure as a 0-sector disc via
unwrap_or(0) with no diagnostic. capacity=0 then skews the layer
heuristic (always reports 1 layer, even for dual-layer discs) and the
canonical title-ordering sort, with nothing in /api/state or info to
indicate the command actually failed. Emit a tracing::warn carrying the
original error at the fallback site so a transient capacity failure is
visible. Recovery behavior is unchanged: 0 is still used as the
fallback.
2026-06-23 04:11:46 -07:00
Matthew Jackson 77fb07f954 keydb: map missing home dir to NotFound, not keydb-parse error
default_path returned Error::KeydbParse (E8004, rendered as 'failed to
parse the keydb file') when HOME/USERPROFILE was unset. That misreports
an environment failure — a process with no home directory, typically a
stripped container or CI config — as a corrupt keydb file the code never
read. Map it to an IoError(NotFound) in the I/O category instead, so no
display path blames the keydb. Add a regression test.
2026-06-23 04:05:41 -07:00
Matthew Jackson 54b79b0395 Preserve I/O error from read_aacs_inputs ISO open
Disc::read_aacs_inputs opened the ISO via FileSectorSource::open and
mapped any failure to Error::AacsNoKeys (E7000), discarding the real
Error::IoError (E5000) and its OS errno. A missing or unreadable ISO
(ENOENT/EPERM) is an I/O fault, not a key-resolution failure; callers
that dispatch on the error code would wrongly tell the user to check
their keys when the ISO simply does not exist.

Propagate the open error unchanged and add a regression test asserting
a nonexistent ISO yields E_IO_ERROR, not E_AACS_NO_KEYS.
2026-06-23 03:59:55 -07:00
Matthew Jackson 7710fccddd Fix oversized read batch on non-sysfs (Windows) optical drives
detect_max_batch_sectors() is a Linux-sysfs probe with no platform
gate. It derived the device name with rsplit('/'), which never splits a
Windows \.\CdRom0 / \.\D: path, so the whole path became the device
name, no /sys node matched, is_optical fell to false, and the function
returned the 8192-sector block default (16 MiB/request) instead of the
60-sector optical default. That value then took the Some(b) arm in
Disc::copy and bypassed the 510-sector optical clamp that lives only in
the sysfs branch, leaving every Windows rip/verify running ~16x over the
optical cap (coarser bad-sector recovery, 16 MiB UDF reads).

Gate the sysfs probe behind a new sysfs_batch_probe_supported() helper
(Linux-only, requires a '/'-delimited path) and return the optical
default for any path the probe can't handle. Add regression tests for
the \.\ device-path forms.
2026-06-23 03:51:51 -07:00
Matthew Jackson d2ff433fac scsi(windows): fix StorageAdapterDescriptor.BusType width (u8 -> u32)
STORAGE_ADAPTER_DESCRIPTOR.BusType is STORAGE_BUS_TYPE, an int-sized
(4-byte) enum, but the repr(C) struct declared it as u8. The total
size stayed 32 bytes by coincidence of alignment padding, and the two
fields actually read today (MaximumTransferLength at offset 8 and
AlignmentMask at offset 16) sit ahead of BusType, so there was no
runtime impact. But BusMajorVersion and BusMinorVersion landed at
offsets 26 and 28 instead of the SDK's 28 and 30, so any future reader
of those fields would have gotten wrong values.

Widen BusType to u32 so BusMajorVersion/BusMinorVersion fall at the
correct offsets, and add a layout regression test asserting every
field offset and the 32-byte total against the winioctl.h layout.
2026-06-23 03:45:34 -07:00
Matthew Jackson 30b2b86cad aacs: fix stale mkb_version offset doc comment
The doc comment claimed the version was a BE u32 at offset 8 of the
record body (offset 12 from pos), but the code correctly reads pos+8
(body offset 4): a 4-byte record header at pos, the Type field at body
offset 0, then the version at body offset 4. Rewrite the comment to
match the actual read so a maintainer does not 'correct' the offset and
break MKB version parsing. Clarify the matching test comment too.
2026-06-23 02:28:34 -07:00
Matthew Jackson a192e696ac disc: reject partial CPS-unit key coverage in AACS validation gate
aligned_unit_keys_validate accepted a unit-key set as soon as ONE
scrambled sample decrypted. On a multi-CPS-unit disc a set covering
CPS unit 0 but not CPS unit 1 therefore passed: decrypt_with committed
it, the sweep proceeded, and CPS-unit-1 sectors passed through as raw
encrypted bytes into the ISO/MKV with no error surfaced anywhere.

Require every scrambled sample to be descrambled by some unit key.
A sample no key covers now fails the gate, so an incomplete set is
rejected (AacsKeyRejected) and the caller falls through to the next
candidate, ultimately surfacing a key error instead of silently
writing ciphertext. Wholly-wrong-key rejection is unchanged.

Add a regression test for the partial-coverage case.
2026-06-23 01:43:36 -07:00
Matthew Jackson dc2dcdba55 disc: fix misleading bridge-degradation comment, add 04/3E regression test
The comment on the bridge-degradation branch in handle_read_error
claimed it matched the NOT_READY 04/3E sense signature, but
is_bridge_degradation() keys solely on a non-standard SCSI status byte
(anything that is not GOOD/CHECK CONDITION/TRANSPORT FAILURE) and
ignores sense_key/ASC/ASCQ. A real 04/3E bad-sector error arrives as
CHECK CONDITION (0x02), so it never took this branch — it falls through
to the generic NOT_READY retry. Rewrite the comment to describe the
status-byte condition the predicate actually detects, and drop the
parallel misleading note in the not_ready_err test helper.

Add a regression test asserting a NOT_READY 04/3E error is not
classified as bridge degradation and routes to the NOT_READY retry
(3 s pause) rather than the bridge cooldown (15 s pause).
2026-06-23 01:20:04 -07:00
Matthew Jackson fc6d9d42fd scsi(windows): only sleep on successful device reset
SptiTransport::reset() unconditionally slept 2 seconds after sending
IOCTL_STORAGE_RESET_DEVICE, even when the IOCTL failed (e.g.
ERROR_INVALID_FUNCTION on a driver that does not support the reset).
On failure no reset occurred, so there is nothing to settle and the
2-second penalty was pure waste. Gate the settle sleep on the IOCTL
return so it only fires when the drive was actually reset.
2026-06-23 00:59:26 -07:00
Matthew Jackson 03f6e076ab keydb: write keydb.cfg atomically (temp + fsync + rename)
keydb::save() overwrote the live keydb.cfg with a bare in-place
std::fs::write. keydb.cfg is the single source of AACS truth and this
path runs unattended (first-boot download + daily-refresh thread, with
a container restart on every release), so a SIGKILL, OOM-kill, power
loss, or ENOSPC mid-write could leave the file truncated with the prior
good copy already gone. A truncated keydb does not error at write time;
it surfaces later as failed key resolution on every AACS rip.

Factor the write into write_atomic(): create the parent dir, write a
unique sibling temp file, fsync, then rename (atomic within a
filesystem). On any write/fsync/rename failure the temp is removed and
the existing keydb is left untouched. Same pattern already used by the
settings and mover write paths. Add regression tests covering in-place
replacement (no stray temp) and prior-copy preservation on failure.
2026-06-23 00:24:58 -07:00
Matthew Jackson e5e8ee43ec mux: track skipped bytes for accurate loss estimation
DiscStream skips a whole AACS unit (3 sectors = 6144 bytes) per
read-error event, but only the skip-event count was exposed. Loss
estimates built from errors*2048 therefore undercounted AACS loss ~3x.

Add a lost_bytes field that accumulates the actual zero-filled byte
count at each skip, expose it via a new Stream::lost_bytes() accessor
(default 0; DiscStream and CountingStream override), so consumers can
scale lost-video time by real bytes lost rather than the event count.

Regression tests assert the AACS path records 6144 B/event (and
exceeds the errors*2048 undercount) while the align=1 path records
2048 B/event.
2026-06-23 00:12:56 -07:00
Matthew Jackson dd9be0d21d pipeline: leaked consumer must not finalise an abandoned output
When finish_with_halt's grace period expires it detaches from the
consumer thread and returns an error to the caller, but the leaked
consumer kept running to completion: once its wedged write syscall
returned it would fall through to sink.close(). For the mux writer
close() finalises the MKV (Cues block + segment-header patch), so a
leaked consumer could finalise — and keep writing to — an output file
the caller had already reported as failed, racing a fresh rip for the
same device over the same path.

Add a shared abandonment flag the consumer polls in its drain loop and
again before close(). finish_with_grace sets it before dropping the
JoinHandle, so the moment the wedged syscall returns the consumer skips
any further apply and skips close() entirely, then exits. This does not
interrupt the in-flight syscall (only its return or process exit can),
but it bounds the damage to the write already in flight instead of a
full finalise of an abandoned file.

Regression tests cover both sides: a consumer leaked past the grace
period skips close(), while one that finishes inside the grace window
still calls close() and finalises normally.
2026-06-22 23:22:57 -07:00
Matthew Jackson 835429c35d scsi/windows: surface IOCTL_STORAGE_RESET_DEVICE failures
SptiTransport::reset() discarded the DeviceIoControl return value, so a
wrong or unsupported reset IOCTL would fail with ERROR_INVALID_FUNCTION
and silently no-op while the unconditional 2s settle sleep made it look
like a reset happened. That is exactly the regression class the doc block
records for the two earlier (incorrect) code values.

Bind the result and warn (with GetLastError) when the reset fails, debug
on success. Lift IOCTL_STORAGE_RESET_DEVICE to module scope and add a
test recomputing it from the CTL_CODE formula so a wrong value can't slip
back in unnoticed.
2026-06-22 23:08:52 -07:00
Matthew Jackson 804c4cb194 mapfile: fsync parent directory after rename for durable resume checkpoint
Mapfile::flush() wrote the new state to a .tmp sibling, sync_all()'d the
temp file, then rename(2)'d it over the final mapfile path — but never
fsynced the parent directory. After the rename the new dirent lives only
in the directory's page cache, so a crash or power loss in the
rename-commit window (the wide window on NFS, the very case the temp
fsync guards) can lose it: resume then reads a stale or absent mapfile
even though the data bytes were durable, silently discarding multi-pass
recovery progress.

Add a best-effort fsync_dir() on the path's parent after the rename,
mirroring the established dirent-durability pattern in autorip's mover.
A directory that can't be opened or synced is logged and ignored rather
than failing the write, since the file bytes are already durable.

Adds a regression test exercising the parent-fsync branch against a real
subdirectory and asserting the helper is a no-op on a missing directory.
2026-06-22 23:00:04 -07:00
Matthew Jackson fa3d6dabdd release CI: parallelize test/publish/release off the critical path
Tests, crates.io publish, and the GitHub Release no longer serialize
behind each other: each job only needs the version-check (verify) gate.
Binary consumers git-tag-pin libfreemkv, so they start building the
instant the tag exists rather than waiting on the crates.io publish.
crates.io publish runs as an independent job for external consumers and
uses --no-verify (CI already compiled this commit in the test job).
2026-06-22 21:59:31 -07:00
Matthew Jackson bd3664d3c5 Demote per-read Drive::read trace event to TRACE
Drive::read fires hundreds of thousands of times per rip. Logging its
entry at DEBUG floods a diagnostic log and buries the events that
actually matter. Move it to TRACE so a level-3 (debug) bug-report log
stays readable; level-4 (trace) still captures it for deep dives.
2026-06-22 21:39:27 -07:00
Matthew Jackson a5861157de Merge branch 'feat/windows-hardening' 2026-06-22 18:08:41 -07:00
Matthew Jackson 831ca31911 v1.0.0-rc.3.1: silent-failure guards (mux empty/zero-frame, CSS crack-vs-unencrypted), Windows keydb path, AlignmentMask, English errors 2026-06-22 18:07:48 -07:00
Matthew Jackson d127783522 docs: describe the generic Unlocker seam, drop in-tree firmware specifics
The docs still documented the old in-tree firmware unlocker: the MediaTek
MT1959 variant table, the READ BUFFER unlock CDB bytes, the profiles.json
schema (unlock_mode/unlock_buf_id/unlock_cdb), the platform/mt1959 driver
listings, and the 'why unlock is needed' handshake mechanism. None of that
lives in libfreemkv anymore — the core is firmware-clean and ships only the
pluggable Unlocker trait + registry (src/unlock.rs).

Rewrite drive-access, architecture, api-design, disc-to-rip, and the README
to describe only the generic Unlocker seam: the trait, register_unlocker, the
registry routing, and the host-cert fallback when no unlocker matches. Point
readers to the freemkv-unlock repo for concrete unlockers. No source change.
2026-06-22 17:10:51 -07:00
Matthew Jackson 80bf9eba60 v1.0.0-rc.3: bump version 2026-06-22 16:13:07 -07:00
Matthew Jackson be8d555551 scsi/windows: correct IOCTL_STORAGE_RESET_DEVICE to 0x002D5004
The dual-model Windows audit (Sonnet) caught that the prior 'fix' (0x002DD000)
was also wrong: that decodes to the OBSOLETE RESET_BUS code (function 0x400,
R|W access) which class drivers reject. Canonical ntddstor.h:
  IOCTL_STORAGE_RESET_DEVICE = CTL_CODE(0x2D, 0x0401, METHOD_BUFFERED, FILE_READ_ACCESS)
  = (0x2D<<16)|(1<<14)|(0x401<<2)|0 = 0x002D5004.

Only affects the best-effort drive-reset recovery path (return value is
discarded); the normal read/unlock/rip path never calls it, so this does not
change normal Windows operation. Verified correct-by-construction; the
AlignmentMask finding is deferred to rc4 (benign on USB optical bridges,
needs a real Windows SCSI-HBA rig to validate).
2026-06-22 15:53:45 -07:00
Matthew Jackson 2cab62dc22 Fix 3 Opus-audit findings: patch transport-abort, patch AACS align, reset IOCTL
- patch (Pass N) now aborts immediately on transport failure (status=0xFF),
  symmetric with the sweep and single-pass mux. Previously a USB-bridge crash
  was treated as an ordinary bad sector and the pass hammered the crashed
  device sector-by-sector until the per-range watchdog expired. (medium)
- patch AACS recovery reads are now unit-aligned: a mid-unit single-sector
  read on an AACS disc was rejected by the decrypting reader (DecryptFailed)
  and the sector abandoned without asking the drive. The read is now widened
  to the enclosing whole 3-sector unit and the requested window copied out,
  leaving all recovery accounting (pos/block_bytes/cursor) untouched so it
  cannot desync. Only affected CLI decrypt-to-ISO --multipass re-runs. (low)
- IOCTL_STORAGE_RESET_DEVICE corrected 0x002D1004 -> 0x002DD000 (the old value
  decoded to function 0x401 with the access bits cleared, so DeviceIoControl
  would fail ERROR_INVALID_FUNCTION instead of resetting). Windows-only. (low)

Adds a transport-failure classification regression test.
2026-06-22 15:50:22 -07:00
Matthew Jackson b12c1e622d mux/disc: abort single-pass rip on transport failure (USB-bridge crash)
A direct disc://→mkv:// single-pass rip drives fill_extents in
skip_errors mode. On a read failure it shrank the batch, retried, and
once bottomed out zero-filled + skipped the unit and continued. A SCSI
transport failure (status=0xFF) is a USB-bridge crash, NOT a skippable
bad sector: the bridge is wedged and every subsequent read fails the
same way. So the loop marched the entire disc at one ~15s bridge-
recovery per probe, producing no MKV — the user-reported 'hundreds of
0x28/0xff warnings, runs forever, Movie.mkv never created'.

Fix: short-circuit to an error on transport failure before any
shrink/skip, even under skip_errors — mirroring the multipass sweep's
transport-failure rule in read_error::handle_read_error. The CLI
surfaces it so the user power-cycles the drive or switches to multipass
recovery. Regression test asserts exactly one read is issued and no skip
is counted (no infinite march).
2026-06-22 15:18:52 -07:00
Matthew Jackson 4d81affb45 Merge branch 'rc3-topgun' into rc3-integration 2026-06-22 15:08:06 -07:00
Matthew Jackson c73a3dbcb6 Fix Windows multi-drive selection, disk:// alias, and READ chunking
Three fixes for a Windows ASUS Blu-ray drive that failed/spammed errors:

- resolve.rs: accept disk:// as an alias for disc:// (identical behavior;
  empty = auto-detect, path = device). Windows users commonly type
  disk://i: after the drive-letter convention.

- drive::find_drive: prefer a drive that reports media present. Enumerate
  all optical drives, query Drive::drive_status() (GET EVENT STATUS, works
  regardless of firmware), and return the first reporting DiscPresent;
  fall back to the first enumerated drive when none report a disc so
  single-drive / quirky setups don't regress. Selection policy split into
  select_drive_with_media() for unit testing.

- READ chunking: add ScsiTransport::max_transfer_bytes() (default 1 MiB).
  Windows SPTI overrides it with the adapter MaximumTransferLength queried
  via IOCTL_STORAGE_QUERY_PROPERTY / StorageAdapterProperty, clamped to a
  64 KiB floor (fallback on query failure). Drive::read now caps each
  READ(10) to that limit: small reads take the unchanged single-CDB path,
  larger reads loop over read_one() chunks, reporting the failing chunk's
  LBA on error. This stops the 16 MiB single read that exceeded the
  adapter limit, made DeviceIoControl fail, and spammed transport-failure
  warnings with slow tiny-read fallbacks.

Tests added for the disk:// alias, media-preference selection, and READ
chunk decomposition / per-chunk error LBA.
2026-06-22 15:02:24 -07:00
Matthew Jackson 4f606ae9a3 aacs: source OEM host certs from keysource layer
Complete the OEM/AACS cert baseline so host certs are a KeySource output,
never compiled in. With an unlocker present the OEM route is unused
(unlocker_read_volume_id short-circuits); without one, the cert handshake
runs when a keysource supplies a host cert and fails gracefully when none
does.

- KeySource trait gains host_certs() (default empty), reusing the existing
  aacs::HostCert type. A source holds certs as its second kind of AACS
  material alongside decryption keys.
- ScanOptions gains key_sources so the handshake can collect certs across
  the app's keysource layer, unioned with DriveCredentials.
- do_handshake_cert collects certs via collect_host_certs (credentials +
  every key source). Zero certs from any source now returns the new
  graceful Error::AacsNoHostCert (code 7024, sentinel <no host cert>)
  instead of silently skipping; resolution still falls back to the
  path-1 disc-hash -> VUK lookup, which drops the error on a hit.
- error.rs: add E_AACS_NO_HOST_CERT / Error::AacsNoHostCert, wired into
  code(), Display, and the round-trip + sentinel tests.

HandshakeResult { volume_id, read_data_key } unchanged: the cert path
still yields both the VID and the bus key.
2026-06-22 11:23:38 -07:00
Matthew Jackson 25acd09504 unlock: finalize Unlocker 3-capability contract
Rename the trait to a generic, drive-neutral capability contract so future
unlockers don't conform to LibreDrive specifics:

  - unlock(...)   -> unlock_drive(...)        (the one required capability)
  - read_vid(...) -> read_volume_id(...)      (no-op default)
  - add set_max_read_speed(...)               (no-op default)

The trait doc now states the contract in one place: unlockers are optional
drive-capability providers; the AACS layer is the always-present baseline and
falls back to the full cert handshake when no unlocker matches. Implement only
the capabilities your drive supports.

Registry: route_unlock now calls unlock_drive; unlocker_read_vid renamed to
unlocker_read_volume_id; add unlocker_set_max_read_speed (mirrors route_unlock
resolution, first matching unlocker, no-op if none match). drive::init calls
it on a matched drive in the post-unlock path; a speed-set failure is logged
and does not fail the rip. encrypt.rs handshake updated to the new VID helper.

Tests updated for the renames; added a set_max_read_speed routing test
(match invokes, no-match is a safe no-op).
2026-06-22 11:05:21 -07:00
Matthew Jackson 159e967760 unlock: add OEM read_vid capability to Unlocker seam
An Unlocker unlocks drive functionality, not just the disc: unlock() is
one capability, OEM VID retrieval is another. Widen the Unlocker trait
with a default-no-op read_vid(), add an unlocker_read_vid registry helper
that mirrors route_unlock resolution, and consult it in do_handshake_cert
before the cert-based VID read. A matching unlocker that serves a VID via
its OEM path short-circuits the cert handshake — VID is obtained without
the host certificate + HRL (restoring the pre-refactor decoupled OEM VID
path, now living inside the unlocker). Non-matching drives, and unlockers
without an OEM VID path, fall through to cert auth unchanged.

is_unlocked() now reports the honest signal (a registered unlocker matched
this drive) instead of const false.
2026-06-22 10:50:47 -07:00
Matthew Jackson 6dc62bcd84 Extract drive unlock behind pluggable Unlocker seam
libfreemkv must stay firmware-clean for crates.io. Move ALL drive-unlock
knowledge — firmware blobs, WRITE_BUFFER/MODE SELECT upload, unlock CDBs,
the MT1959 variant-A/B handshake, the 800 KB profiles.json database, and
the DriveProfile parsing — out into the freemkv-unlock-ld crate.

libfreemkv now keeps only the seam:
  - Unlocker trait (name/matches/unlock) + a process-wide ordered registry
    (register_unlocker / route_unlock) in src/unlock.rs
  - Drive::init() walks the registry; the first unlocker whose matches(id)
    is true runs unlock(scsi, id); if none match the drive is left in
    stock mode and the host-cert AACS handshake (the OEM route) carries
    the disc.

The unlocker issues its own CDBs through the public ScsiTransport::execute,
so libfreemkv knows nothing about how unlocking happens.

Removed:
  - profiles.json
  - src/platform/mt1959/{mod,variant_a,variant_b}.rs
  - src/profile.rs (DriveProfile, ProfilesFile, find_by_drive_id, ...)
  - the PlatformDriver trait

Because the Unlocker seam reports only success/failure (no extended-access
marker), VID acquisition is now always via the cert-based handshake; the
per-drive OEM-VID-CDB shortcut and Drive::is_unlocked() (now const false)
are removed/neutralized. Disc-speed calibration moved into the unlocker's
unlock(); Drive::probe_disc() is a no-op.

git grep over src/ is firmware-blob/profiles/WRITE_BUFFER/mt1959-free.
All tests pass on Rust 1.86 (precommit green).
2026-06-22 10:31:51 -07:00
Matthew Jackson 9250f5bb30 wip: top gun EL/decrypt follow-up (in progress, rc3) 2026-06-22 10:02:53 -07:00
Matthew Jackson e960c2f1be mux/mkv: video-only timeline epochs + finer 0.1ms TimestampScale
Fixes corrupt MKV seek index on single-clip titles with many
interleaved tracks (Top Gun UHD: 2 video, 11 audio, 32 PGS).

TimelineContinuity previously shared one high_ns frontier + offset_ns
across ALL tracks. A sparse, lagging non-video frame (subtitle/audio)
ratcheted the frontier up; the next normal video frame then sat >3s
below it and was misread as a clip-boundary discontinuity, permanently
bumping offset_ns. On a one-clip title this fired thousands of times
and inflated Cue/cluster timestamps into the billions of ms, destroying
the seek index (ffmpeg then seeked to wrong positions and emitted
spurious 'Could not find ref with POC N' errors).

Now only the VIDEO track drives epoch decisions: video alone advances
the frontier and opens a new epoch on a real backward PTS jump.
Non-video tracks are remapped under the current offset and never touch
the frontier or offset. A lagging non-video tail straggler at a genuine
multi-clip boundary (old-epoch raw PTS under the new offset) is
recognised via the previous offset and remapped to the seam, so it
neither flies forward nor forces a back/forward-dated split cluster.
Genuine multi-clip seamless rebasing is preserved.

Also drop TimestampScale from 1ms to 0.1ms (100_000 ns/tick) so
23.976fps frames and 0.833ms TrueHD AUs stop colliding on a single
tick (the source of the non-monotonic-DTS warnings and the audio
cadence flattening). The finer scale shrinks the i16 block-relative
span to ~3.27s, so: cluster duration is set to 2s nominal (keeps
keyframe-driven clusters within the i16 range for typical GOPs), and
the i16-overflow cluster-split path now emits a Cue for the split
cluster so the seek index has no gaps.

Regression tests: single-clip late-subtitle must not inflate offset_ns;
non-video must not advance the frontier; non-video straggler remapped
to seam at a real boundary; every cluster (incl. i16-split) carries a
Cue. Existing tick/duration assertions updated for the new scale.
2026-06-22 09:45:45 -07:00
Matthew Jackson f74979bdb4 docs: scrub AACS internals + key-coverage claims from changelog; condense pre-1.0 history 2026-06-22 09:39:40 -07:00
Matthew Jackson dc2cac1b5f v1.0.0-rc.2: bump version 2026-06-22 09:06:54 -07:00
Matthew Jackson 337e77951c rc2: macOS cross-compile fix + security/recovery hardening
- build.rs: pass target -arch to cc so macos_shim cross-compiles (x86_64-apple-darwin)
- AACS/CSS: unit-aligned decrypting sweep; per-VTS CSS title keys (hard-fail on wrong VTS);
  reject truncated Unit_Key_RO; AACS 2.0 sig-verify skip; CSS bus-auth random nonce
- recovery: gap-filling mapfile load; sweep/copy resume reconciliation; stale-mapfile abort;
  patch wedge/damage-window range reset
- mux: TS continuity + PSI CC desync guards; HEVC numTemporalLayers clamp; MPEG-2 pending
  byte-cap; PS parse_pts marker-bit validation; HdrFormat strict parse; Unknown-variant metadata
- net/keydb: network:// SSRF parity (IPv4-mapped, CGNAT, 0.0.0.0/8, Class-E); bounded keydb
  header read + size cap + error context
- io: durable mapfile fsync; NFS writeback degrade; sync_file_range error capture;
  Windows SCSI u32 transfer guard
2026-06-22 08:58:10 -07:00
Matthew Jackson 5941c059c6 v1.0.0-rc.1
CSS keyless decrypt (Stevenson), AACS 1.0/2.0/2.1, MPEG-2 DVD, multi-OS SCSI, multipass recovery, mux highway, audit hardening
2026-06-21 21:06:07 -07:00
Matthew Jackson e8bb6225ac libfreemkv 0.31.10: parallelise recover_dk_position slot scan
The subset-difference slots are independent, so rayon find_map_any scans them
in parallel and cancels on first match. UHD MKB no-match scan ~26s -> ~4.6s on
8 cores. Bit-identical result; 58 aacs tests pass.
2026-06-17 19:56:07 -07:00
Matthew Jackson 9c80ef8245 libfreemkv 0.31.9: ~3x fewer AES ops in the subset-difference PK walk
calc_pk_from_dk derived all three children (left/pk/right) at every tree
level but used only the one it descended into; the Processing Key only
matters at the final node. Derive just the descended child per level + the
PK once at the end. Bit-for-bit identical; speeds every DK->MK derivation
(disc decryption + unpositioned-DK recovery). 60.8s -> 22.9s on a UHD
worst-case recovery scan.
2026-06-17 15:43:39 -07:00
Matthew Jackson dc87962e50 libfreemkv 0.31.8: recover unpositioned device key position from an MKB
Add recover_dk_position: boil a position-less device key down against a
disc MKB to its invariant subset-difference position (node/uv/u_mask_shift)
— zero-descent probe + ancestor walk-up, hoisted verify. Consolidate the
SD-walk surface (drop the research-only probe::walk_pk_against_tables;
make derive_media_key_from_pk_walked internal).
2026-06-17 15:29:51 -07:00
Matthew Jackson 4221cd6a86 libfreemkv 0.31.7: expose Processing Key from the DK walk 2026-06-17 13:18:34 -07:00
Matthew Jackson dda4e7482b aacs: unify the SD-walk; derive_media_key_from_dk now exposes the PK
Two byte-identical copies of the subset-difference walk lived in keys.rs
and variants.rs. Consolidate the pure helpers (aesg3, calc_v_mask,
calc_pk_from_dk) into keys.rs (pub(super)); variants.rs imports them.

Add derive_media_key_and_pk_from_dk(mkb, dks) -> Option<(mk, pk)>, which
returns the intermediate Processing Key the walk already computes;
derive_media_key_from_dk becomes a thin wrapper. This lets callers bank
the PK on a DK boil instead of re-deriving it via a second, divergent
walk (the classical-vs-variant cvalues order made that miss silently).

216 AACS tests pass (incl. a new (mk,pk) regression); precommit (1.86) green.
2026-06-17 13:11:03 -07:00
Matthew Jackson f80551f278 udf: read AACS MKB length-aware; honor inline allocation descriptors
read_file rejected the padded ~128 MiB MKB_RO.inf via the 0.31.0
MAX_FILE_BYTES cap, so read_aacs_inputs failed and the online
key-resolve path never contacted the keyserver. Read the MKB's real
record length from its header and read exactly that. Also honor
inline/embedded (AD type 3) files so small AACS .inf files read from
the ICB payload instead of being misparsed as allocation descriptors.

Release 0.31.6.
2026-06-08 19:20:41 -07:00
Matthew Jackson 41a6d89cd1 libfreemkv 0.31.5: do not force monotonic block timestamps on video
B-frame video PTS is legitimately non-monotonic in decode/storage order; the
audio-oriented monotonic nudge was clobbering it to prev+1ms, which decoders
flagged as non-monotonic DTS (thousands per title). Apply the nudge to
audio/subtitle only; video keeps its true PES PTS. + regression test.
2026-06-08 09:00:35 -07:00
Matthew Jackson f79c2a0aa9 libfreemkv 0.31.4: prune 144 vacuous tests (keep spec-grounded subset) 2026-06-08 07:28:55 -07:00
Matthew Jackson d181362460 libfreemkv 0.31.3: use RFC 5737 doc address in network URL test fixture 2026-06-07 22:42:09 -07:00
Matthew Jackson 8000bae177 libfreemkv 0.31.2: comprehensive spec-grounded test suite (~950 tests)
Test-hardening release, no runtime changes. Adds spec-grounded unit tests
across the silent-corruption surfaces — UDF/MPLS/CLPI/IFO parsing, BD/DVD
title + extent assembly, AACS/CSS key handling, TS/PS demux + codec parsers,
MKV/EBML container output, the mux pipeline, sector prefetch + decrypt
decorator, drive/SCSI sense decoding, label extraction, and core I/O. Each
test is grounded in the format spec or real on-disc behavior and verified to
fail under a targeted source mutation. No behavior changed.
2026-06-07 22:28:29 -07:00
Matthew Jackson 2a55bab3ed tests: read_aacs_inputs Long-AD coverage; scrub title names from comments
Add a precommit fixture proving Disc::read_aacs_inputs reads a Long-AD,
multi-extent /AACS/Unit_Key_RO.inf in full — the exact input the online
key-request path depends on (no disc/deploy needed). Make
read_aacs_inputs_from_reader pub(crate) for the test.

Reword internal comments/doc examples to generic descriptions instead of
specific media titles.
2026-06-07 21:25:29 -07:00
Matthew Jackson 222a596c55 libfreemkv 0.31.1: UDF Long-AD stride fix + MKB trim guard + regression tests 2026-06-07 20:49:55 -07:00
Matthew Jackson c1b4f3cbb3 tests: harden UDF allocation-descriptor + bad-sector recovery paths
Spec-grounded unit tests for the silent-corruption surfaces, each verified to
fail under a targeted source mutation (no vacuous tests).

udf (10): Extended-AD 20-byte stride + extent LBA at off+12, type-1 sparse
extents skipped not emitted, zero-length type-0 terminator, continuation-loop
bound (anti-hang), UTF-16BE and 8-bit name decoding, FID L_IU offset, parent
(..) FID skip, d-string length-byte cap. Locks the spec branches a future
allocation-descriptor refactor must not silently break.

recovery (9): Pass-N damage-skip range bounds (forward/reverse cursor stays in
range), one-quarter-of-remaining skip cap, below-threshold no-op, work-done
accounting, and bridge-degradation retry-to-budget fall-through.
2026-06-07 20:46:00 -07:00
Matthew Jackson 06c30aa466 udf: honor ICB allocation-descriptor type (Short/Long/Extended AD)
read_icb_extents hardcoded an 8-byte Short-AD stride for every file. Large
BD-ROM .m2ts streams use 16-byte Long ADs; striding them as Short ADs reads
descriptor #0 correctly (length+lba align) but lands #1 in the middle of the
first Long AD (its zero impl_use bytes). The AD-list terminator (data_len==0
=> break) then fired on that zero and stopped after the first extent, so every
multi-extent title truncated at ~1 GiB. The same reader backs read_file, so
disc AACS-input files (/AACS/*.inf) and the m2ts mux extents were both
affected.

Read the ICB Tag flags (AD type) and stride 8/16/20 bytes for Short/Long/
Extended ADs accordingly; Extended ADs carry the lba at off+12.

aacs: extract trim_mkb and restore its guard so an MKB whose content length
the parser cannot determine (mkb_content_len == 0) is returned intact instead
of truncated to empty.

Regression tests: Long-AD read_icb_extents returns all extents; Long-AD
read_file returns full content; trim_mkb never zeroes an unrecognised MKB.
2026-06-07 19:53:40 -07:00
Matthew Jackson 061f68594a 0.31.0: hardening and correctness pass across mux, codec, AACS/CSS, UDF/MPLS/CLPI, recovery, drive/SCSI, labels, and I/O
Library-wide review-and-fix pass: tightened AACS keydb/handshake/variant
handling and trailing-partial-unit policy, corrected MPLS mark offset and
added UDF allocation bounds, hardened the mux/codec framing and M2TS paths,
guarded SCSI READ CAPACITY short transfers and unified error mapping, added
overflow guards on untrusted disc input, and made prefetch shutdown
deterministic. Release profile now builds with thin LTO + single codegen unit.
2026-06-07 17:37:38 -07:00
MattJackson 5b6ea8f5c4 v0.30.7: mux correctness pass
- hevc: real chroma/bit-depth in hvcC (10-bit Main10), emit redefined
  param sets in-band; clamp oversized param sets
- h264: emit redefined SPS/PPS in-band, clamp oversized avcC
- ac3: flush final frame at EOS, per-frame PTS
- dts: attribute each access unit's PTS to its own core PES
- truehd: capture PTS only at access-unit boundary
- meta: preserve color_space round-trip
- dvd: fix subtitle/audio track PID collision
- pgs/vobsub: flush trailing subtitle at EOF; multi-PES SPU reassembly
- ts: unify codec table via from_coding_type (+LPCM), warn on unknown
2026-06-06 21:46:49 -07:00
MattJackson eeba94b21d Merge branch 'worktree-agent-a184f5a97a37e2c1a' 2026-06-06 21:40:42 -07:00
MattJackson cfc12774f2 mux: fix DVD subtitle/audio track collision, PGS/VobSub flush, unify TS codec table
Subtitle/DVD output-corruption + stream-mapping coverage fixes.

1. DVD subtitle/audio track-mapping collision (CRITICAL). The PS path
   routed 0xBD private-stream packets to a track via (sub_id & 0x1F)+1,
   so VobSub subtitle sub-id 0x20+j aliased audio track j+1: subtitle
   PES was fed to the AC-3 parser and the real subtitle track got
   nothing. Route by the canonical DVD PID instead via a new
   PsPacket::dvd_pid() that mirrors scan_dvd_titles' PID assignment
   (video 0xE0, audio 0xBD00+i, subtitle 0x20+j), then look up the
   track in pid_to_track. Fixed identically at all three sites
   (pipelined_stream consume_ps, disc.rs live feed, disc.rs EOF flush).
   Unmappable/unmapped packets now WARN instead of silently dropping.

2. PGS flush() missing. PgsParser inherited the no-op default flush, so
   the last subtitle of every PGS track (emitted only when a following
   PCS arrives) was dropped at EOF. Implemented flush() to drain the
   pending display set (duration_ns: None for the trailing block).

3. DVD VobSub multi-PES SPU not reassembled. A subpicture unit larger
   than one PES spans multiple PES (only the head carries a PTS).
   DvdSubParser is now stateful: it buffers per sub-stream until the
   leading 2-byte SPU_size is satisfied, inherits the head PTS, and
   emits one Frame. flush() drains a truncated trailing SPU at EOF.

4. One-table hygiene. scan_streams had a duplicate stream_type->Codec
   table that had drifted from Codec::from_coding_type (missing 0x80
   LPCM, 0x85 mapped to DTS-HD MA vs HR, etc.). scan_streams now uses
   from_coding_type plus a new Codec::kind()/CodecKind category split,
   so the two mappings can never diverge. Silent drops in
   scan_streams and bluray STN parsing now WARN with PID + type.

Tests: dvd_pid mapping + subtitle/audio collision regression, PGS
final-subtitle flush, VobSub multi-PES reassembly + EOF flush,
scan_streams 0x80 LPCM via from_coding_type.
2026-06-06 21:33:11 -07:00
MattJackson fd543c058b mux: rustfmt (Rust 1.86) on touched codec/meta files 2026-06-06 21:32:04 -07:00
MattJackson cd8ce708ac mux/h264: emit redefined SPS/PPS in-band + clamp oversized avcC
H.264 stored SPS/PPS in single Option slots and always stripped them
from frame data, so a mid-title parameter-set redefinition (same id,
different body) was lost — those frames decoded against the stale avcC
copy the player re-applies at each keyframe. Same defect class as the
HEVC PPS-redefinition bug. Mirror that fix: emit a changed SPS/PPS
in-band at every occurrence; strip only the first-seen / identical ones.

Also guard avcC's 16-bit NAL length fields: a param set > 65535 bytes
truncated the length while appending all bytes. Return None instead.
2026-06-06 21:24:16 -07:00
MattJackson 2eee777b8f mux/hevc: real chroma/bit-depth in hvcC + clamp oversized param sets
codec_private() hardcoded 8-bit 4:2:0 in the hvcC fixed header, wrong for
10-bit Main 10 UHD (essentially all UHD). Parse chroma_format_idc and
bit_depth_luma/chroma_minus8 from the SPS RBSP (with emulation-prevention
removal and sub-layer profile_tier_level handling) and emit the real
values; fall back to 8-bit 4:2:0 only if the SPS can't be parsed.

Also guard the 16-bit NAL length fields: a param set larger than 65535
bytes would truncate the length while appending all bytes, mis-framing
the record. Return None instead of emitting a corrupt hvcC.
2026-06-06 21:22:44 -07:00
MattJackson a5962be86c mux/truehd: capture PTS only at an access-unit boundary
A continuation PES (one that merely extends a TrueHD AU spanning PES
packets) carries its own later PTS, which the parser was adopting
mid-assembly, snapping the AU's timestamp and breaking the monotonic
per-AU cadence (A/V drift). Capture the PTS base only when the
reassembly buffer is empty, i.e. when a PES actually begins a new AU.
2026-06-06 21:19:53 -07:00
MattJackson 7306f661b9 mux/dts: attribute each access unit's PTS to its own core PES
When two access units flushed in one parse() call (a core arriving in an
earlier PES than the PES that closes the unit, the standard DTS-HD MA
core+extension-as-separate-PES layout), the second AU inherited the
latest PES's PTS instead of its own core's.

Track per-PES PTS markers keyed by buffer offset and stamp each emitted
AU with the PTS of the PES covering its first byte, rebasing markers on
every front drain. Each AU now keeps its own core's timestamp.

Extends two_cores_back_to_back to assert PTS; adds cross-PES coverage.
2026-06-06 21:19:07 -07:00
MattJackson 77f7aced83 mux/ac3: flush final frame at EOS and stamp per-frame PTS
Two defects:
- Ac3Parser inherited the no-op default flush(), so a complete final
  frame still buffered at end-of-stream was dropped (~32 ms of audio
  lost). Add a flush() that drains a complete buffered frame, mirroring
  dts.rs.
- Every frame in one parse() call was stamped with the single PES PTS,
  collapsing their timecodes and drifting A/V. Compute a base PTS once
  per call, then advance per frame by the frame's own duration (AC-3 =
  1536 samples; E-AC-3 from numblkscod), converting samples->ns at the
  stream sample rate (fscod). Each Frame now carries duration_ns.
2026-06-06 21:16:17 -07:00
MattJackson 97ae452e40 mux/meta: preserve color_space round-trip in m2ts metadata
MetaStream::Video dropped color_space on from_title/to_title, hardcoding
BT.709 on the way back. HDR titles (BT.2020) lost their color metadata.

Add a color_space field, populate it in from_title, and use it in
to_title. For pre-0.30.7 metadata that has no color_space, derive it from
the preserved hdr field (all HDR formats are BT.2020, SDR is BT.709).

Adds ColorSpace::id() + FromStr for serialization round-trip.
2026-06-06 21:13:29 -07:00
MattJackson eec0594a30 Merge branch 'main' into worktree-agent-a91cd2cf29779b84e 2026-06-06 21:11:05 -07:00
MattJackson dab6ea4359 mux/hevc: emit redefined param sets in-band at every keyframe
Fight Club redefines PPS id 0 mid-title; the parser froze the first PPS into
codecPrivate and stripped the rest, so the redefined segment decoded against
the wrong PPS (CABAC/cu_qp_delta desync, intact framing). Now any VPS/SPS/PPS
whose body differs from the codecPrivate copy is emitted in-band at every
occurrence, overriding the hvcC copy a player re-applies per keyframe. Proven:
Fight Club re-mux decode errors 320+ -> 0 across all corrupt regions.

Also adds aacs::unit_key_validates (1-block early-reject UK validation) and
ts_sync_count/ts_packet_total helpers.
2026-06-06 21:09:04 -07:00
MattJackson 9a5ed57044 mux: enforce strictly-monotonic per-track block timestamps
Fixes the non-monotonic DTS ffmpeg/players reject (observed on Fight Club audio
streams 16/18: "14061 >= 14060"). Some audio PES PTS truncate to the same
millisecond as the prior frame, or tick back 1ms from rounding. The MKV writer
now tracks the last block timecode per track and nudges a non-increasing one to
prev+1ms (sub-frame, inaudible, A/V sync unaffected at ms granularity), recorded
only for frames actually written. New helper monotonic_ts() + unit test.

Independent of the separate HEVC slice-payload corruption investigation.
2026-06-06 15:57:00 -07:00
MattJackson 7d58ba7b08 v0.30.0: DiscInputs carries the disc's volume label (title)
Add `DiscInputs.volume_label: Option<String>` — the disc's human title (UDF/ISO
volume identifier, else BDMV <di:name>), populated by Disc::inputs() from the
scan. Identity only, no secret, not used in any AACS derivation; lets a key
source forward the title so a key service can catalog disc_hash → title.
2026-06-06 09:05:15 -07:00
MattJackson 4f1dbfd042 v0.29.0: bump version 2026-06-05 20:36:19 -07:00
MattJackson 5b702a76a7 mux: write PTS (presentation), not DTS, as the MKV block timecode
The video codec parsers (HEVC, H.264, VC-1, MPEG-2) used
pes.dts.or(pes.pts) as each frame's timestamp. MKV block timecodes
are presentation timestamps; frames are stored in decode order and
the player reorders for display by timecode. Using DTS makes the
timecode monotonic in storage order, presenting B-frames in decode
order — visible motion judder / wrong frames on playback, and
PTS-based seeking lands on the wrong frame.

The compressed video was always byte-correct (verified by NAL-level
diff against a known-good demux); this was purely a timestamp defect
affecting every B-frame title. Fix: prefer PTS (pes.pts.or(pes.dts)).
Verified on a real UHD iso->mkv: emitted PTS now reorders for
B-frames identically to a reference muxer.

Update the two tests that asserted the old DTS-preferred behavior and
add an HEVC regression test pinning PTS as the block timecode.
2026-06-05 20:01:13 -07:00
MattJackson 6be5198886 libfreemkv: v1.0 hardening — codec/EBML/TS robustness + DTS parser fixes
Audit-driven fixes (rounds 1–3):
- hevc: correct hvcC profile/level SPS offsets (HEVC has a 2-byte NAL header)
- mkv: map all DTS variants to the registered A_DTS codec id; force a new
  cluster before the i16 cluster-relative timestamp can overflow
- ebml/mkvstream: bound untrusted EBML sizes (no multi-GB allocs); reject
  uint>8 (was an OOB panic) and non-{0,4,8} float widths (were a desync)
- ts: skip PES-header bytes that span a TS packet boundary; add the PMT
  section_len/prog_info_len bounds the PAT parser already had
- ac3: preserve a 0x0B77 syncword split across a PES boundary; cap buffer
- dts: validate each next-core boundary by decoded core size (a 0x7FFE8001
  pattern inside XLL payload no longer false-splits/drops the lossless
  extension); reject sub-minimum core frames; fix forced-emit PTS base
- lpcm: DVD program-stream PCM no longer double-strips the BD LPCM header
- vc1/mpeg2: do not emit a parameter-set-only PES as a standalone frame
- pgs/truehd: cap the pending reassembly buffer (parity with ac3/dts)
- aacs: ts_syncs_intact uses the exact packet count
- prefetched: capacity-guard the recycled-buffer set_len
- Cargo.toml: exclude project docs from the published crate

Convergence: a third independent audit pass found no remaining material
(CRITICAL/HIGH/MEDIUM) issues. Full precommit (fmt + clippy -D + tests,
Rust 1.86) green.
2026-06-05 16:23:39 -07:00
MattJackson e2aa9abd6d aacs: validate a resolved key against content before applying it
decrypt_with now takes the disc's encrypted content samples and, after
deriving the candidate unit keys, confirms at least one de-scrambles a real
aligned unit before committing them. A wrong key (a keydb VK that doesn't
match the disc, a stale UK) is rejected with AacsKeyRejected instead of
silently applying garbage unit keys. Conservative by design: with no samples
(resume / mapfile cache) it accepts as before, leaving those paths unchanged.

The KeySource trait becomes a stateful provider — next_key hands one candidate
at a time (the source owns the order) and reports exhaustion, replacing the
all-at-once resolve; errored() distinguishes a failed source from a clean
no-key.
2026-06-05 09:23:08 -07:00
MattJackson 3b7bee9ed4 Fix DTS-HD MA muxing as lossy core only (#10)
DTS-HD MA/HRA access units on Blu-ray are a DTS core frame (sync
0x7FFE8001) followed by one or more DTS extension substreams (sync
0x64582025) carrying the lossless audio. Ground-truthing the Dunkirk
ISO showed the m2ts demuxer hands these out as SEPARATE PES packets on
the same PID: one core PES (exactly core-sized, nothing trailing), then
the extension substreams in following PES packets with their own later
PTS.

The old DtsParser emitted one frame per PES the moment a core frame was
complete, and dropped any PES with no core sync. So every core became a
core-only (lossy) frame and the extension PES packets were discarded as
junk -- silently downgrading the track to lossy DTS core (1557 kb/s CBR,
16-bit) instead of DTS-HD MA (VBR, 24-bit lossless).

Rewrite the parser to assemble across PES boundaries: an access unit
runs from its core sync up to (but not including) the NEXT core sync, so
the core plus every following extension substream stays together. Add a
CodecParser::flush() (default empty) called at end-of-stream by both the
pipelined and inline DiscStream mux paths to drain the final buffered
unit. A 64 KiB cap guarantees forward progress and never stalls if a
boundary can't be found.

Validated on the rip1 testbed: Dunkirk eng+ger and Fight Club eng main
audio now ffprobe as profile=DTS-HD MA (Fight Club eng at 24-bit), with
VBR packet sizes (~2716-2788 B) well above the old fixed 2012 B lossy
core. Genuinely-lossy DTS dub tracks are left untouched.
2026-06-05 06:40:27 -07:00
MattJackson b7405e2d27 mux: fail iso:// with no usable AACS key instead of muxing garbage
When an AACS-encrypted ISO is muxed with decryption requested (not --raw)
but key resolution yielded no usable key, input() proceeded to mux the
still-encrypted stream — emitting ~100 MB of garbage (no TS syncs, demuxer
emits nothing) and sometimes spinning for tens of minutes.

Add a cheap result-check in resolve::input()'s Iso branch via the pure
predicate aacs_key_missing(raw, has_aacs, keys): when decryption is
requested AND the disc carries AACS state AND decrypt_keys() is None,
return new Error::NoDiscKey { disc_hash } (E7022) before muxing. The
40-hex disc hash is sourced from AacsState::disc_hash. --raw and
non-AACS (unencrypted / CSS) discs are unaffected. Unit-tested.
2026-06-05 05:06:15 -07:00
MattJackson e04d79c593 v0.28.1: bump version 2026-06-04 21:46:41 -07:00
MattJackson b34af1fa74 mux: wire TrueHD channel probe into iso:// mux setup
correct_truehd_channels() probes the first DECRYPTED access units of the
chosen title (TrueHD PIDs, bounded 8 MiB read of the first extent) and
sets AudioStream.channels from the MLP major sync — fixing the MPLS
audio_format understatement (5.1 declared on a 7.1/Atmos TrueHD track).
Regenerates the basic codec label for the corrected count; richer
editorial labels are left untouched. Wired in resolve.rs input() for
iso:// after decrypt_with (the m2ts is only decryptable post-key), using
a fresh reader so the mux reader is undisturbed. generate_audio_label
made pub(crate).
2026-06-04 20:28:37 -07:00
MattJackson a7317f8885 mux/truehd: decode true channel count from MLP major-sync format_info
Adds truehd_channels(format_info) + truehd_channels_from_stream(): the
real presentation channel count (8ch/7.1 when present, else 6ch/5.1)
from the TrueHD major sync, which the MPLS audio_format base field
understates (declares 5.1 even on a 7.1/Atmos track — the Dune case).
Per the MLP spec presentation channel-assignment masks. 3 unit tests.

This is the decoder; wiring it to correct AudioStream.channels requires
a mux-time probe of the DECRYPTED audio payload (the channel truth isn't
readable at scan — the m2ts is AACS-encrypted until the key is applied
at mux), to be added + validated on a real rip.
2026-06-04 19:51:46 -07:00
MattJackson 474273afc0 mux/mkv: signal Dolby Vision via dvcC BlockAdditionMapping
With the EL now carried (mpls PID fix), the MKV writer tags the Dolby
Vision layer so players/mediainfo recognise it: a BlockAdditionMapping
(0x41E4) with BlockAddIDType 'dvcC' and a DOVIDecoderConfigurationRecord
(profile 7, bl+el+rpu present) on the DV-layer video track. Disc
Profile 7 dual-layer is preserved losslessly as a second video track
(archival-correct, MakeMKV-equivalent); M2TS carries the EL as a native
second video PID with no extra signaling needed.

Adds dolby_vision_config() + a profile-7 dvcC unit test. End-to-end DV
recognition (mediainfo / playback) to be confirmed on a full rip.
2026-06-04 19:42:14 -07:00
MattJackson 0a2bab5789 mpls: resolve PID for sub-path / DV-EL stream entries (types 2/3/4)
parse_stream_entry only read the PID for stream-entry type 1 (a stream
in the PlayItem's own clip), returning PID 0 for every other type. The
Dolby Vision enhancement layer uses type 4 (verified on Wicked + Dune:
se_len=9 type=0x04 [00 10 15] -> PID 0x1015), so the EL fell through to
PID 0x0000 and was silently dropped by the demux/mux — Dolby Vision lost.

Now the PID offset is keyed off the entry type per the BD stream_entry()
layout: type 1 -> +2, type 2 -> +4, type 3/4 -> +3. The DV EL now
resolves to its real PID (0x1015) so it is demuxed and carried as a
stream through the PES layer to every writer (M2TS carries dual-PID DV
natively; the MKV writer's DV signaling is the format-specific piece).
2026-06-04 19:28:34 -07:00
MattJackson fa4d7ef871 mux/dts: keep DTS-HD extension substream across PES boundaries
The DTS parser emitted a core-only frame and discarded the trailing
DTS-HD extension substream whenever the extension straddled a PES
boundary (it advanced past only the core, then re-synced on the core
syncword, skipping the leftover extension bytes). That silently
downgrades DTS-HD MA / HRA to lossy DTS core.

Now: when an extension sync is visible after the core — full, or a
partial prefix at the buffer edge — wait for the full extension instead
of splitting it off; only 'nothing after the core' (final unit / EOF,
no parser flush) is taken as a genuine lossy core-only unit. Adds a
boundary-split regression test asserting core+extension is preserved.
2026-06-04 19:16:26 -07:00
MattJackson 36d1af1b7f labels+disc: codec from stream, audio-richness title tiebreak
Two validated audio-correctness fixes (proven on real discs Wicked/
Paddington/Dune/Fight Club via head-captures):

labels: apply_labels now derives the codec/channel descriptor from the
stream's OWN codec/channels unless the parser's codec_hint is BOTH
consistent with it AND richer (e.g. "Dolby Atmos" on a TrueHD stream).
A mis-bound hint ("AC-3 2.0" on a TrueHD track) is rejected and the
stream's own codec used — killing the cross-labeled shuffle (Wicked) and
the compat-core mislabel (Paddington), while keeping rich hints and
normalizing plain ones to uniform marketing names. (codec_hint_consistent
+ codec_hint_adds_detail, 5 tests.)

disc: canonical_title_order gains an audio-richness tiebreak
(lossless > channels > track-count) for titles that tie on
duration+clips — so a movie authored as a full-audio playlist plus a
stereo-only twin (Fight Club 00800 vs 00004) picks the full-audio one
instead of falling to array order.
2026-06-04 19:10:11 -07:00
MattJackson b96f6206fe 0.28.0: version bump (keydb decoupling) 2026-06-04 16:41:04 -07:00
MattJackson bb32fb993b 0.28.0 review fixes: propagate mux key-apply error; scrub key-shaped fixtures
- mux input(): propagate a failed decrypt_with rather than muxing an
  undecryptable stream silently (review M6).
- Scrub synthetic but key-SHAPED hex (HOST_PRIV_KEY/DEVICE_KEY/disc-entry) from
  the keydb parser tests to obvious zero/repeated-byte placeholders. No real or
  real-looking key material in code.
2026-06-04 16:38:58 -07:00
MattJackson c1eb74dfa5 AACS: cutover — lookup-free lib; DriveCredentials for the handshake
The library no longer loads a keydb anywhere. The scan path always captures the
disc's AACS inputs (MKB, VID, Unit_Key_RO.inf) and resolves NO key; a caller
resolves a Key from a key source and applies it via Disc::decrypt_with.

- ScanOptions loses keydb_path / unit_key / disable_keydb (and the path search);
  it now carries only optional DriveCredentials (host certs) for the live-drive
  AACS handshake. do_handshake_cert uses those instead of loading the keydb.
  An unlocked / LibreDrive drive takes the OEM Volume-ID path and needs none.
- The mux input() path takes caller-resolved unit_keys instead of a keydb_path,
  and applies them via decrypt_with.
- Deleted the now-dead inline resolve_encryption / resolve_encryption_static.

All 700+ lib tests pass.
2026-06-04 16:04:06 -07:00
MattJackson 46838c63ca AACS: trim the MKB captured at keyless scan to its record length
The VID-only scan path stashed MKB_RO/RW raw — those files are allocated to a
fixed ~128 MiB and zero-padded, so the MKB on AacsState (consumed by
Disc::inputs() and the device/processing-key decrypt_with derivation) was the
full pad, not the ~few-MB record stream. Trim to mkb_content_len, matching
read_aacs_inputs.
2026-06-04 15:32:12 -07:00
MattJackson 1e60220ff6 AACS: KeySource::needs_samples() gates the content-sample read
A source that validates server-side against ciphertext (an online key service)
needs encrypted content samples; one that keys on disc identity (keydb, mapfile)
does not. needs_samples() lets the caller skip the extra disc read unless a
configured source actually needs it. Defaults false.
2026-06-04 14:45:40 -07:00
MattJackson 080f03e8ed AACS: DiscInputs gains app-populated samples for ciphertext-validating sources
An online key service validates a candidate against real ciphertext, so it
needs a few encrypted content sample units. Add a samples field to DiscInputs;
Disc::inputs() leaves it empty (reading content needs the disc reader, which
scan does not retain) for the application to fill.
2026-06-04 14:42:11 -07:00
MattJackson 35de6101d4 AACS: add KeySource trait + DiscInputs; Media key carries a candidate pool
The library now describes the key-source abstraction it consumes, while the
concrete sources live in a companion crate. A KeySource looks a disc up from
its DiscInputs (disc hash, volume id, MKB, Unit_Key_RO.inf) and returns a Key;
the library derives down and decrypts. Sources do no derivation.

Disc::inputs() exposes those inputs from a scanned disc.

Key::Media now carries a Vec of candidate media keys rather than one: a media
key is MKB-scoped, so a source hands its whole pool and the library picks the
one that verifies against the disc's MKB. That mirrors Device/Processing and
keeps key selection (a derivation step) inside the library.
2026-06-04 14:29:32 -07:00
MattJackson 9012101573 AACS: decrypt_with accepts higher-level keys and derives down
Disc::decrypt_with now takes Device / Processing / Media / Volume keys in
addition to Unit. A caller hands in whatever level it resolved and the
library derives down the AACS chain to the per-CPS-unit keys, then
decrypts:

  Device   -> MKB walk         -> media key -> VUK -> per-CPS unit keys
  Processing -> MKB            -> media key -> VUK -> per-CPS unit keys
  Media      -> Volume ID      -> VUK              -> per-CPS unit keys
  Volume     -> Unit_Key_RO.inf, one unit key per CPS unit
  Unit       -> used directly (terminal)

Derivation stays centralized in the version-dispatched resolver
(1.0 / 2.0 / 2.1), fed by a single-key provider built from the supplied
key — no new crypto. Volume notably does NOT stop at the volume key: it
decrypts every CPS unit's key.

Scan stashes the AACS inputs (Unit_Key_RO.inf and MKB) on AacsState so an
out-of-band decrypt_with can derive without re-reading the disc.

Non-breaking: Key is #[non_exhaustive] and the existing Unit path is
unchanged. New tests cover the Volume -> per-CPS derive-down, the
missing-inputs error, and the no-units rejection.
2026-06-04 14:20:16 -07:00
MattJackson 8bc1de6c9b 0.27.5 (step 2, Phase 1): expose AACS inputs (uk_ro, mkb) on AacsState
scan now stashes the raw Unit_Key_RO.inf + MKB bytes on AacsState (via
resolve_vid_only, the disable_keydb path), so an external key-resolver can
derive unit keys from a resolved VUK without re-reading the disc — the
foundation for moving lookup/derivation out of libfreemkv. Additive: the keydb
path is untouched, all existing constructors default the new fields empty.
584 lib tests green. Builds on the KeyOrigin rename + the Key/decrypt_with API.
2026-06-04 13:20:27 -07:00
MattJackson d94a4d3444 0.27.5 (step 2, item 2): rename provenance enum KeySource -> KeyOrigin
Disambiguates the key vocabulary: Key (the input handed to decrypt_with),
key sources (the resolver's lookup list), and KeyOrigin (how a key was
resolved). Internal-only rename — no dependents import it.
2026-06-04 11:42:21 -07:00
MattJackson 1bd7e1de2a 0.27.4: lookup-free Key API (Disc::decrypt_with) + E8005 deferred-mux fix
Add libfreemkv::Key (non_exhaustive; Unit wired) + Disc::decrypt_with(Key) — the
caller resolves a key from any source and hands it in; the library decrypts with
no keydb/network of its own. inject_unit_keys now synthesizes a minimal
ExternalUk AACS state when a scan built none (no keydb at scan time), so a disc
keyed at sweep but scanned without a keydb still decrypts from its mapfile UK
instead of reporting E8005 and deferring the mux forever. Tests cover the
synthesize, replace, empty->None, and decrypt_with paths.
2026-06-04 11:08:33 -07:00
MattJackson da62ee7cf2 v0.27.3: bump version 2026-06-03 21:47:43 -07:00
MattJackson 575c76156f mapfile: persist resolved unit keys (keys XOR VID)
A keyed disc now writes its decrypted AACS unit keys to the mapfile header
(# freemkv-uk: <cps>:<hex>); an unresolved disc writes only the VID. The two are
mutually exclusive (set_unit_keys clears the VID) — unit keys are the final
answer, so deferred-mux / resume decrypts directly with no key lookup, while the
VID alone is the 'still unresolved, retry' marker. CopyOptions/SweepOptions carry
the keys (written when present, else the VID); Disc::inject_unit_keys applies
mapfile-recovered keys to a scanned disc. Round-trip test added.
2026-06-03 21:45:37 -07:00
MattJackson b518860d9c v0.27.0: detect AACS-scrambled units by raw TS sync, not flag bits
Rename is_unit_encrypted -> is_aacs_scrambled and decide encryption from the unit's MPEG-TS sync bytes (destroyed by the encrypted body) instead of the TP_extra copy-control (byte 0) or TS scrambling-control (byte 7) flags, which discs do not set reliably. One shared predicate now backs the decrypt gate and out-of-band key validation, so callers agree on what 'encrypted' means. Decryption restores the syncs, so a decrypted unit reads as clear and there is no flag to clear.
2026-06-03 07:35:50 -07:00
MattJackson c8eb42b490 v0.26.11: detect AACS unit encryption via TS scrambling-control bits
is_unit_encrypted read the TP_extra copy-control bits (byte 0), which are a
copy-permission flag, not an encryption flag. On discs whose sampled units are
clear navigation packets (PAT/PMT) those bits can be set while the unit is not
scrambled, so a correct Unit Key was used to 'decrypt' already-plaintext data,
produced garbage, and the key was wrongly treated as failing.

Read the actual flag instead: the TS transport_scrambling_control bits (top two
of TS-header byte 3 = byte 7 of the aligned unit, inside the clear seed). AACS
encrypts whole aligned units, so this one packet's TSC reflects the unit.
decrypt_unit now clears the TSC bits of every packet on the way out so the
result is valid unscrambled TS. Tests updated to the TSC flag.
2026-06-02 20:37:56 -07:00
MattJackson 48570ac065 Local resolve: MK-pool brute via km_verifies (path 2.5)
keydb stores Media Keys per-disc, but an MK is MKB-scoped (shared across a
pressing/MKB-family). A disc whose own hash/VID isn't keyed can still resolve
if any stored MK verifies against its MKB. New path 2.5 (between PK and the
VID lookup) collects the distinct MK pool from the providers, km_verifies each
against the disc MKB, and on a UNIQUE pass derives VUK (with the disc VID) then
the UK — matching the online resolver's behavior so local keydb mode resolves
the same discs (e.g. an MK present in keydb under a sibling pressing).
km_verifies is one AES-D + magic check per candidate (cheap). Adds
KeyProvider::media_keys() + a path-2.5 unit test.
2026-06-02 16:14:09 -07:00
MattJackson 08e46640fd v0.26.8: bump version 2026-06-02 15:41:38 -07:00
MattJackson a9195824ff Trim MKB to real record length; empty unit-keys means no keys
- mkb_content_len walks the MKB record stream and returns where it ends,
  so callers can drop the trailing zero padding. MKB files are allocated
  to a fixed size (~128 MiB) with records at the front; both MKB_RO and
  MKB_RW can be padded. read_aacs_inputs* now trim to the real records
  (~few MB) instead of shipping ~128 MiB of zeros.
- decrypt_keys() returns None when unit_keys is empty: an AACS state that
  carries only a VID (out-of-band resolution, pre-key) is 'encrypted, no
  keys', not a usable empty key set.
2026-06-02 15:41:27 -07:00
MattJackson 401fe23988 v0.26.7: bump version 2026-06-02 15:15:58 -07:00
MattJackson f89bce5851 Read MKB_RO not the padded MKB_RW; keep VID when keydb is disabled
- read_aacs_inputs / read_aacs_inputs_from_drive now read MKB_RO.inf first.
  MKB_RW.inf is a fixed ~128 MiB rewritable region that is mostly zero
  padding; reading it shipped 124 MiB of nothing. MKB_RO is the real,
  correctly-sized MKB (a few MB). Fall back to RW only if RO is absent.
- disable_keydb no longer drops the Volume ID. A caller resolving Unit Keys
  out-of-band needs the VID (on-disc content read during the handshake).
  New resolve_vid_only builds a keys-free AacsState carrying just the VID +
  version metadata, so the disc reports 'encrypted, no keys' (resolved
  out-of-band) instead of discarding the VID.
2026-06-02 15:15:45 -07:00
MattJackson dfccb85e15 v0.26.6: bump version 2026-06-02 14:11:13 -07:00
MattJackson ebedffb762 ScanOptions: add disable_keydb to skip all keydb lookup
A caller that resolves Unit Keys out-of-band can now set disable_keydb
so the scan consults no keydb at all — neither an explicit keydb_path
nor the standard search locations. Without it, a keydb that merely sits
in a default location ('~/.config/...') silently shadows the out-of-band
path. unit_key still takes precedence over everything.
2026-06-02 14:09:59 -07:00
MattJackson fdf63ccb7f v0.26.5: bump version 2026-06-02 13:37:32 -07:00
MattJackson c735d284da v0.26.4: bump version 2026-06-02 13:06:10 -07:00
MattJackson 94ab7bc73c scan: genericize the caller-supplied unit-key docs (no external-source naming)
The ScanOptions.unit_key path is a generic primitive — a caller-supplied Unit
Key that bypasses keydb lookup. Doc comments + a tracing log named a specific
external source; reworded to neutral 'out-of-band / external key service' so
the library makes no assumptions about where the key came from.
2026-06-02 13:00:11 -07:00
MattJackson 9f209fe066 v0.26.3: bump version 2026-06-02 10:56:56 -07:00
MattJackson 32a1a6e095 mapfile: persist AACS Volume ID as ddrescue-safe comment
Add an optional 16-byte AACS Volume ID to the Mapfile, persisted as a
'# freemkv-vid: <32-hex>' header comment so it survives to deferred-mux
and resume without touching the ISO payload. ddrescue treats the line
as a comment, so the pos/size/status data parser is unaffected.

- Mapfile gains set_vid/vid accessors; save() emits the comment in the
  header block, load() parses it back (malformed/absent -> None, never
  errors the load). Load->save round-trip preserves it (patch pass).
- Thread vid through CopyOptions and SweepOptions; sweep persists it
  when creating/opening the mapfile.
- Unit test: round-trips the VID, asserts the saved comment, and
  confirms data ranges parse identically with and without the comment.
2026-06-02 08:50:19 -07:00
MattJackson 1d3b8f5fb6 chore: gitignore scratch/ (prevent accidental commit of session artifacts) 2026-06-01 22:52:58 -07:00
MattJackson 97b0ae7be2 chore: scrub internal refs from changelog/makefile/cargo 2026-06-01 22:46:54 -07:00
MattJackson 7756f1feca aacs: restore read_aacs_inputs_from_drive (live-drive key-file extractor) 2026-06-01 21:52:27 -07:00
MattJackson 1565da610a chore: scrub non-shippable references from tests/comments 2026-06-01 21:36:57 -07:00
MattJackson e134616422 v0.26.2: bump version 2026-06-01 21:06:56 -07:00
MattJackson 8d54a3c64e aacs: KeyProvider abstraction + PK walker + external-UK key source
Two coherent additions to the AACS resolver:

KeyProvider abstraction (provider.rs) — key material comes from pluggable
backends; KeyDb implements it (device/processing keys, host certs,
disc-by-hash / disc-by-vid lookup) plus orphan-DK parsing. ResolveContext
takes a provider array. Adds the SD-tree PK walker
(derive_media_key_from_pk_walked) and a `probe` module (km_verifies MK
oracle, mkb_* record parsers) used for offline key verification. Cvalue
record selection prefers 0x05, falls back to 0x07.

External-UK key source — the second, mutually-exclusive key source for the
keyserver path. ScanOptions/InputOptions gain `unit_key`; when set,
resolve_encryption_static skips keydb entirely and uses the caller-supplied
Unit Key directly (KeySource::ExternalUk). Disc::read_aacs_inputs exposes a
disc's Unit_Key_RO.inf + MKB so a caller can fetch the UK out-of-band; the
library makes no network call itself.

CHANGELOG: redact test-disc title in historical notes.
2026-06-01 20:47:14 -07:00
MattJackson c0478e1273 labels(pixelogic): number audio from the FPL playlist, not menu segments
Pixelogic discs that authored a SEG_MainFeature menu/preview segment
containing a stray commentary token made the parser anchor on that
segment instead of the real FPL_MainFeature playlist. Combined with
'Audio Stream N' placeholder tokens not advancing the audio counter, a
lone eng_ACOM_ commentary at STN slot 4 collapsed onto slot 1 — tagging
the main feature track as Commentary (seen on Wicked: For Good).

Fix:
- Anchor the feature section on the FPL_ playlist when present; only
  fall back to SEG_MainFeature on discs with no FPL_ section.
- Count 'Audio Stream N' placeholders so editorial audio tokens get
  their true STN ordinal. Subtitle (PG Stream N) numbering is left
  unchanged — counting it regressed several corpus discs.

Extract the section/numbering loop into assign_labels with unit tests.
Verified: 11/11 labels-corpus discs still PASS; Wicked commentary now
lands on the AC-3 2.0 stereo track instead of the main TrueHD.
2026-05-31 21:16:11 -07:00
MattJackson 7d29168fec v0.26.1: AACS resolver path 5 — KEYDB unit-keys direct fallback
Adds a 5th key-resolution path that consumes pre-decrypted unit keys
directly from KEYDB when the entry has no VUK field. Covers ~4,572
entries in the public keydb (~2.5%), heavily skewed toward MKBv76+ UHD
discs where DVDFab/FindVUK can no longer extract a VUK but does extract
unit keys. Partial CPS-unit coverage is rejected so a disc is never
half-decrypted.

Resolver path order reordered root-to-leaf: DK (1) → PK (2) →
KEYDB-derived MK+VID (3) → KEYDB VUK (4) → KEYDB unit keys (5).
Previous order was leaf-first.

API:
- AacsState::vuk is now Option<[u8; 16]> (was [u8; 16])
- ResolvedKeys::vuk is now Option<[u8; 16]> (was [u8; 16])
- KeySource variants reordered + new KeyDbUnitKeys variant

3 new resolver tests (path 4 still works without VID; path 5 succeeds
with pre-decrypted unit keys; path 5 rejects partial CPS coverage).
2026-05-22 12:51:37 -07:00
MattJackson 8e14c9b850 v0.26.0: bump version 2026-05-21 15:18:39 -07:00
MattJackson e1c8343f77 aacs: OEM-driven VID retrieval — per-drive CDB from profile, cert fallback
When the drive is in extended-access state (unlocked), retrieve VID via
the per-drive `read_vid_cdb` from the bundled profile instead of the
cert-based AACS REPORT_KEY handshake. Cert handshake remains the
fallback for drives that don't enter extended-access state, or whose
profile lacks the required CDB.

Empirically verified on the BU40N (signature 999ec375) against
Barbie UHD: drive returns 36 bytes from buffer 0x44 at offset
0x10E291, VID at response[4..20]. The 16 bytes match Dune Part Two's
known VID in keydb.cfg byte-for-byte, cross-validating the path
against an independent oracle.

Architectural impact:

- Renames `Drive::is_libredrive_active()` → `Drive::is_unlocked()`.
  Internal `Mt1959::libredrive_active` becomes `Mt1959::unlocked`;
  the prior `unlocked` (init-success flag) becomes `init_complete`
  to avoid the name collision.
- `disc/encrypt.rs::Disc::read_vid` is the single entry point.
  When `is_unlocked()` is true, calls `read_vid_oem` (issues the
  per-drive CDB, validates the response signature high-3-bytes
  `00 22 00`, returns bytes [4..20]). Otherwise delegates to
  `read_vid_cert` (the existing AACS REPORT_KEY format 0x80 path).
- `DriveProfile` gains the per-drive CDB templates and identifier
  blocks extracted from each per-drive firmware payload — including
  `read_vid_cdb`, `read_disc_keys_cdb`, `drive_nominal_speed_cdb`,
  `set_speed_max_cdb`, two cache-prime canary CDBs, the buffer-0x45
  verify CDB, the firmware-upload CDB, and the unlock probe CDB.
  Variants A and B differ in which fields are populated. All optional;
  consumers fall back to the cert/handshake path when fields are
  absent.
- New error variants `Error::DriveProfileMissing` (E7020) and
  `Error::VidCdbUnavailable` (E7021). Both treated as
  "OEM unavailable → try cert path" by `read_vid`, not terminal.

Closes the v0.25.x gap where HRL-burned host certs (the public
libaacs leaked cert is on every recent drive's HRL) blocked all
post-handshake VID retrieval. With OEM-driven VID:

- AACS 1.0 BD on supported drives: rips end-to-end with our existing
  DKs walking the MKB.
- AACS 2.x UHD: fails honestly at the DK wall (E7018 "No usable DK"
  for v77+ MKBs) instead of the misleading E7017 "No Volume ID"
  the prior code surfaced. We have VID; we just don't have v77+ DK
  material — that gap is a key-acquisition problem, not a code
  problem.

Empirically verified on rip1 (BU40N + Barbie UHD, MKB v77,
2026-05-21): error code flipped from E7017 to E7018 as predicted.
The DK wall is now correctly the proximate failure for unrippable
modern UHD discs, instead of the indirect VID-retrieval wall the
v0.25.x cert-only path produced.

Renames and comment scrubs eliminate upstream-RE-vocabulary
references in the public crate per `feedback_no_breadcrumbs.md`.

674 tests pass (565 lib + 109 integration). No tradename leaks in
any modified file.
2026-05-21 15:18:21 -07:00
MattJackson a956c6ad94 v0.25.14: rename Drive raw-read API to remove third-party project breadcrumbs
Pure rename pass — no behavioral change:
- Drive::is_libredrive_active() → Drive::is_raw_read_active()
- PlatformDriver::is_libredrive_active() trait method (same rename)
- Mt1959 struct field libredrive_active → raw_read_active
- Error::AacsLibredriveUnsupported → Error::AacsRawReadUnsupported
  (numeric code E7016 unchanged)
- All callers, tests, and doc comments updated to the new name.

Old identifiers removed entirely; downstream consumers must update.
Mirrored in bdemu, freemkv, autorip, freemkv-tools.
2026-05-21 14:43:20 -07:00
MattJackson 1805d92ca4 v0.25.13: DrmScheme dispatcher + AACS 2.1 framework + libredrive cleanup
- Introduce DrmScheme enum (Css/Aacs10/Aacs20/Aacs21) + drm module with
  uniform detect/load dispatch across all four protection schemes.
- Land AACS 2.1 Media Key Variant framework in aacs::variants: chain
  derivation, MKB record types 0x82/0x83, bit-0x02 SoftKCD and bit-0x04
  online-challenge detection. Aacs21 dispatcher arm wired but commented
  out pending validation against a Variant-scheme disc.
- Replace aacs2: bool with AacsVersion enum across ContentCertificate,
  UnitKeyFile, ResolvedKeys. resolve_keys splits into _v1/_v2/_v21.
- Delete the libredrive raw-read VID shortcut from do_handshake; the
  drive enforces the AGID requirement regardless of firmware-upload
  state, so the shortcut spuriously dispatched E7017 instead of
  surfacing the real downstream walls.
2026-05-21 13:57:45 -07:00
MattJackson 823f0ad430 v0.25.12: bump version 2026-05-21 11:37:53 -07:00
MattJackson 477bdf1835 v0.25.11: bump version 2026-05-21 11:14:22 -07:00
MattJackson 4d83b69c20 aacs: libredrive raw-read VID path + revert v0.25.9 built-ins + walker fix follow-through
Three coherent threads landing for v0.25.11:

1. Libredrive raw-read VID path. When Mt1959::do_unlock sees both the
   MMkv active-mode marker at [12..16] and the LbDr mode-ID marker at
   [16..20], Drive::is_libredrive_active() returns true and
   do_handshake skips the AACS cert dance — VID is retrieved via
   READ_DISC_STRUCTURE format 0x80 with AGID=0 and bus encryption is
   already off. This unblocks UHD ripping on drives whose leaked host
   cert is on the AACS HRL.

   - platform/mt1959/mod.rs: detection + active flag + 4 unit tests.
   - platform/mod.rs: PlatformDriver::is_libredrive_active trait method.
   - drive/mod.rs: Drive::is_libredrive_active accessor.
   - disc/encrypt.rs: do_handshake branches on the flag; new
     read_volume_id_libredrive helper. Return type widened to
     (Option<HandshakeResult>, Option<Error>) so callers see which
     specific failure happened.
   - disc/mod.rs: scan_with plumbs the new tuple through and preserves
     handshake errors as disc.aacs_error.

2. Revert v0.25.9 built-in AACS keys + plugin slot. Single source of
   AACS truth: keydb.cfg. The compiled-in DKs/PKs were a slim
   convenience that didn't move the hard problem (no v77+ DKs) and
   added a maintenance surface. Plugin slot was overlapping
   functionality with the main keydb.

   - Deleted src/aacs/builtin_keys.rs (4 DKs + 3 PKs).
   - Removed KeyDb::with_builtins, load_or_builtins, merge_from,
     merge_local_plugin, local_plugin_path, internal dedup helpers.
     KeyDb::empty kept for unit-test use.
   - KeyDb::load reverts to pre-0.25.9 form: read file or return I/O
     error; no fallback.
   - disc::encrypt::resolve_encryption keydb_path back to required
     (&Path), not Option<&Path>.
   - disc::scan_with surfaces KeydbLoad { path: "<no keydb in search
     paths>" } sentinel when encrypted + no keydb — same sentinel
     autorip's message switch already handles.
   - CSS player keys in src/css/auth.rs stay compiled in; they're
     1999-era public inputs separate from AACS and pre-date the 0.25.9
     additions.

3. Walker fix follow-through (libaacs-parity validate_processing_key,
   cvalues 0x07-then-0x05 preference, path-2/3/4 short-circuit on
   zero VID) + NIST AES-CMAC KAT + VID MAC round-trip / mutation /
   zero-rejection tests.

5 new Error variants for finer-grained AACS failure reporting:
AacsHostCertRejected (E7015), AacsLibredriveUnsupported (E7016),
AacsVidUnavailable (E7017), AacsMkUnavailable (E7018),
AacsVukNotInKeydb (E7019). Lets CLIs/UIs render which piece of the
AACS chain failed instead of always saying "no keys."
2026-05-21 11:10:35 -07:00
MattJackson 7dbbfc6726 v0.25.10: bump version 2026-05-20 15:35:05 -07:00
MattJackson e635c9556f v0.25.9: built-in AACS keys + plugin slot + MKB record-type fix
Two changes that make AACS 1.0 / DVD self-sufficient:

1. MKB record-type identification bug fix. `mkb_find_mk_dv` was
   searching for type 0x10 (which is Type-and-Version, 12 bytes)
   when the Verify Media Key Record is actually type 0x81 for
   AACS 1.0 or type 0x86 for AACS 2.0/2.1. `mkb_version` had the
   inverse bug. PK and DK derivation paths therefore silently
   failed on every disc, masking how often the fallback paths
   could have worked. Fix searches the correct types; tests added
   covering both the 0x81 and 0x86 verify-record forms and the
   0x10 version record at offset 8 of the body.

2. Built-in AACS keys + operator plugin slot. Four device keys
   (covering MKB v01-v82+) and three processing keys (covering
   v63-v68) compiled directly into the library. Combined with the
   31 CSS player keys already in css/auth.rs, DVDs and Blu-rays
   (AACS 1.0) now decrypt with zero external files. New plugin
   path at ~/.config/freemkv/local_keys.cfg (same syntax as
   keydb.cfg) layered additively on top of built-ins and main
   keydb. `Disc::scan` no longer errors when keydb.cfg is absent;
   AACS 2.0 / UHD still surfaces a specific error when the disc
   needs keys none of the layers provide.

Public docstrings in project docs + README updated to describe the
three additive layers (built-ins → keydb.cfg → local_keys.cfg).
2026-05-20 09:00:32 -07:00
MattJackson d7b5c30f5d v0.25.8: bump version (unified release with autorip 0.25.8) 2026-05-19 21:45:45 -07:00
MattJackson 739a276a39 v0.25.7: BU40N firmware wedge fix in do_handshake
Pre-0.25.7 the AACS authenticate loop fired up to 16 host-cert
attempts back-to-back with no pause. Each attempt is 5-10 SCSI
REPORT_KEY/SEND_KEY exchanges, so on a disc whose host cert isn't
in our KEYDB (or one the drive rejects), the drive saw 80-160 SCSI
commands in a few hundred ms and entered a fast-fail firmware
wedge state where every subsequent CDB returns sense 05/24 until
power-cycled.

Three defences:
- MAX_CERT_ATTEMPTS capped at 3 (was 16)
- 1-second sleep between attempts
- Bail immediately on any sense_key == 0x05 (ILLEGAL_REQUEST) so
  the loop can't deepen the wedge if a regression undoes the
  attempt cap.
2026-05-19 21:18:45 -07:00
MattJackson 5f1028a62a v0.25.6: sync to autorip 0.25.6 (image diet) 2026-05-19 18:17:10 -07:00
MattJackson ea15d212de v0.25.5: sync to autorip 0.25.5 2026-05-19 18:01:14 -07:00
MattJackson 00673c8ec3 v0.25.4: sync to autorip 0.25.4 2026-05-19 17:44:23 -07:00
MattJackson bfefb4cb5b v0.25.3: sync to autorip 0.25.3 release (no behavioural changes) 2026-05-19 17:11:11 -07:00
MattJackson 1b95193517 v0.25.2: DTS-HD codec ID + PGS BlockDuration
- MkvTrack::audio emits A_DTS/MA, A_DTS/HR, A_DTS per the DTS family
  instead of mislabelling everything as A_DTS. Plex transcoder and
  strict hardware decoders reject DTS-HD MA payload under a plain
  A_DTS track.
- PgsParser is now stateful: pairs display PCS with the following
  empty PCS to compute a duration. Frame::duration_ns + PesFrame::duration_ns
  carry it through; MkvMuxer::write_frame gains a final Option<u64>
  parameter that emits BlockGroup + BlockDuration when set. Fixes
  subtitle bitmaps lingering past their intended end-time.
2026-05-19 16:11:54 -07:00
MattJackson 7dcac44136 v0.25.1: bump version 2026-05-19 14:20:44 -07:00
MattJackson 4da559e39f mux: PrefetchedSectorSource event_fn + delete DiscStream::new_pipeline
* `PrefetchedSectorSource::new_with_events` adds an optional
  `event_fn` callback that fires `BytesRead` after every successful
  batch from the producer thread. The original `new()` becomes a
  thin no-events wrapper. Lets autorip wire the highway and still
  get UI progress events without polling the consumer side.

* `build_iso_pipeline` grows an `event_fn` arg so the autorip
  multipass mux can pipe BytesRead straight through to its progress
  UI.

* Stream trait gains a default `errors() -> u64` method (= 0) so
  Box<dyn Stream> callers (autorip's mux loop) can read the
  skip-on-error counter without downcasting. `DiscStream` overrides
  to return its `errors` field.

* Delete `DiscStream::new_pipeline` and the pipeline-mode fields
  (`demux_thread`, `demux_rx`) plus the `read_pipeline` helper.
  All pipeline construction now goes through
  `PipelinedPesStream` via `build_iso_pipeline`; `DiscStream`
  becomes the single-thread-only inline path used by the drive
  single-pass read.

* `lib.rs` re-exports `build_iso_pipeline`.
2026-05-19 14:19:37 -07:00
MattJackson eeca250f69 v0.25.0: bump version 2026-05-19 13:37:24 -07:00
MattJackson c51b3181f2 mux: pipelined PES highway — read+decrypt → demux → parse on 3 threads
Introduces the freemkv mux throughput highway: a three-stage thread
pipeline that replaces the inline single-thread read path for any
file-backed source (ISO and m2ts file URLs both route through it).

  Thread A: read + decrypt  (PrefetchedSectorSource / BytePrefetcher)
  Thread B: M2TS demux      (DemuxThread)
  Thread C: codec parse     (PipelinedPesStream, on caller thread)

Each handoff uses a bounded crossbeam channel with a recycled buffer
pool — no allocations or memcpys in the steady-state hot loop.

Component map:

* io/byte_prefetcher.rs (new) — std::io::Read producer thread with
  recycled Vec<u8> pool. Pairs with PrefetchedSectorSource (sector
  side) so demux_thread::spawn_zero_copy can wire either upstream.
* sector/prefetched.rs — recycled buffer pool added; into_channels()
  peels off the rx/recycle_tx/shell triple for zero-copy demux.
* mux/demux_thread.rs (new) — owns the TsDemuxer/PsDemuxer, runs
  feed() on its thread, ships Vec<PesPacket> batches.
* mux/pipelined_stream.rs (new) — the read-side Stream impl. Pulls
  packets from the demux thread and runs codec parse on the caller.
* mux/resolve.rs — build_iso_pipeline (public) / build_m2ts_pipeline
  (private) assemble the three stages; iso:// and m2ts:// both
  return PipelinedPesStream.
* mux/m2ts.rs — collapsed to a write-only sink (Mode::Read deleted;
  the read direction lives on the highway now).
* mux/codec/h264.rs — find_start_code uses memchr SIMD memmem::find.
* mux/codec/hevc.rs — tightened frame_data initial capacity.
* mux/ts.rs — boundary-packet handling avoids the per-batch 16 MiB
  remainder copy; PesAssembler starts at 16 KiB to dodge the 64-page
  first-touch fault tax that the previous 256 KiB pre-alloc paid on
  every PES boundary.
* mux/disc.rs — gains DiscStream::new_pipeline + read_pipeline as
  the legacy autorip ingress (drive + multipass paths still need
  on_event / skip_errors before they migrate to the highway).
* io/file_sector_source/* — per-OS prefetch() syscall hook
  (Linux readahead, macOS F_RDADVISE, Windows/other no-op).
* decrypt.rs — FREEMKV_DECRYPT_THREADS renamed to FREEMKV_THREADS;
  pool sized to all cores by default.

Measured on rip1 testbed (Civil War UHD, 62 GiB ISO → null://):

  60 → 322 MB/s warm cache (old new_pipeline path)
  60 → 660 MB/s warm cache (highway path, this commit)
  60 → 126 MB/s sustained disk-bound

The IsoSectorReader baseline reader was deleted in favour of
FileSectorSource so the freemkv CLI and autorip exercise the same
read path.
2026-05-19 13:35:32 -07:00
MattJackson 2a31a47434 mux: keyframe-align MKV clusters + SeekHead; set TS RAI on keyframe PES
MKV: cluster boundaries now require a video keyframe in addition to the
5s minimum, so every cluster has a CuePoint at its start. Pre-first-
keyframe frames are dropped. Adds a SeekHead at Segment start with
fixed-width back-patched SeekPositions for Info/Tracks/Chapters/Cues.

Before this change a 2h26m UHD rip had 52 CuePoints across ~1750
clusters and a 16.5-minute gap between adjacent seek entries; players
seeking inside that gap had to scan from the prior cue. After, one
CuePoint per cluster.

TS (tsmux production path + m2ts_mux): PesFrame.keyframe is plumbed
end-to-end. Codec-private parameter sets are prepended on the first
keyframe (not the first frame); non-key video before any keyframe is
dropped. The first TS packet of a keyframe video PES carries an
adaptation field with random_access_indicator=1. m2ts_mux previously
hardcoded RAI=1 on every PCR packet; that is now gated on the current
PES being a keyframe video PES, combining correctly with PCR when both
land on the same packet.

Adds 17 tests covering keyframe alignment, cue count/position/timing,
SeekHead correctness, RAI set/clear, codec_private gating, non-key drop,
and PCR+RAI combination.
2026-05-17 15:31:16 -07:00
MattJackson b08662f95c Revert "iter14: re-engage FileSectorSource 4 MiB readahead buffer"
This reverts commit 4594196b6e.
2026-05-17 10:50:00 -07:00
MattJackson 4594196b6e iter14: re-engage FileSectorSource 4 MiB readahead buffer
iter13 strace finding: producer thread (mux ISO reader) spends 80%
of wall-clock in state D (NFS RPC wait), doing 207 preads/sec at
103 us each. Pipe is 78 MB/s isolated read; we use ~25 MB/s.

FileSectorSource was using direct per-sector pread (0.21.3 bypass).
That bypass was justified under Phase 2.5 + 0.21.7 producer-poll cap
("32 MiB refill bursts the TCP connection enough to starve the
writer thread"). Both of those conditions are gone now (iter8
baseline: no Phase 2.5, no producer poll cap).

The 4 MiB buffer infrastructure was preserved with #[allow(dead_code)]
in case re-engagement was ever wanted. iter14 just routes the hot
path through buffer_covers + refill + memcpy. Net diff: ~10 LOC of
business logic, 5 unit tests already cover the contract.

Expected: producer's effective read rate jumps from 25 → 60+ MB/s
(matches dd ceiling for 1+ MiB block reads). If consumer side keeps
up, mean mux climbs from 28.7 → 35-50 MB/s. If consumer is now the
cap, we see a clear ceiling around 30-35 and we know where to look
next.

Critically: this change ONLY affects mux-from-ISO. Disc→ISO sweep,
Pass N bad-sector retry, AACS, drive ops, mapfile, recovery — all
untouched (they use DriveSectorSource which is a separate impl).
2026-05-17 10:36:04 -07:00
MattJackson 9d6974df3e iter13 base: lock chunk back to 32 MiB after iter12 revert
iter12 revert restored chunk to 128 (iter11 value) by way of the
revert chain. Setting it back explicitly to 32 for iter13. We're
back to iter8 baseline config (no Phase 2.5, 32 MiB chunks,
DONTNEED, no app-level FileSectorSource buffer, channel depth 32).

iter13 is instrumentation: strace the mux thread during a known-dip
to see where time is actually going.
2026-05-17 09:49:03 -07:00
MattJackson 3394a5b3fe Revert "iter12: fallocate without KEEP_SIZE + truncate_at_sync; chunk back to 32 MiB"
This reverts commit e2c7e20329.
2026-05-17 09:48:35 -07:00
MattJackson e2c7e20329 iter12: fallocate without KEEP_SIZE + truncate_at_sync; chunk back to 32 MiB
iter11 (128 MiB) crashed to 16.6 mean — chunk-size sweet spot is
firmly 32 MiB. Locking in.

iter12 hypothesis: each NFS write past the file's EOF triggers a
server-side SETATTR (metadata commit) to update the file length.
With ~62 GiB of MKV output and writes at chunk boundaries, that's
thousands of SETATTRs per rip. By using plain fallocate (mode 0)
the file is pre-extended to size_bytes up front; subsequent writes
overwrite the pre-extended region with no metadata ops.

Adds high_water tracking + truncate_at_sync so we ftruncate down
to actual content size if the size hint was an overestimate.
2026-05-17 09:35:07 -07:00
MattJackson ff22385a61 iter11: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB
Prediction: ~26-27 mean (regression from iter8). Confirms chunk-size
sweet spot at 32 MiB. Then move off chunk-size as a lever entirely.
2026-05-17 09:21:00 -07:00
MattJackson d39d3933f5 Revert "Reapply "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point""
This reverts commit f7165a2020.
2026-05-17 09:19:44 -07:00
MattJackson 31dc3d35c3 iter10: chunk back to 32 MiB
iter9 (64 MiB on top of no-P2.5) regressed -1.2 from iter8. 32 MiB
remains best on the no-P2.5 path. Net iter10 = iter8 minus DONTNEED
(via the prior commit re-applying f4a881b).
2026-05-17 09:06:34 -07:00
MattJackson f7165a2020 Reapply "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point"
This reverts commit 8ec767339e.
2026-05-17 09:06:14 -07:00
MattJackson d836f8b1e7 iter9: 64 MiB chunks on top of iter8 (Phase 2.5 disabled)
iter8 (Phase 2.5 disabled, 32 MiB chunks) hit 28.7 MB/s mean
— 1.3 MB/s below R2 floor. Without Phase 2.5, WAIT_AFTER blocks the
mux thread directly once per chunk. Halving the WAIT_AFTER frequency
(32 → 64 MiB chunks) should raise the mean without changing the
underlying architecture.
2026-05-17 08:52:51 -07:00
MattJackson 8919de5b03 iter8: chunk back to 32 MiB on top of Phase-2.5 revert
iter7's 64 MiB chunks didn't help (23.1 vs iter4's 24.0). Reverting
to 32 MiB so iter8 differs from iter4 by exactly one variable:
Phase 2.5 disabled / direct passthrough mux thread.

Test question: is Phase 2.5 helping at all? If iter8 mean > iter4,
Phase 2.5 has been a net negative on this rig the whole time. If
iter8 mean < iter4, Phase 2.5 is doing what it claimed.
2026-05-17 08:39:27 -07:00
MattJackson 0dc1f12108 Revert "Reapply "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)""
This reverts commit 335e8b68cc.
2026-05-17 08:39:06 -07:00
MattJackson 6abc25a7e8 iter7: WRITEBACK_CHUNK_BYTES 32 -> 64 MiB
iter6 (8 MiB chunks) regressed mean -8.2 MB/s vs iter4 (32 MiB).
Trend: bigger = better in this workload, against the page-age
theory. Try 64 MiB to see if fewer/larger syncs raise mean further.
0.21.14 went to 128 MiB and was reverted; 64 is between.
2026-05-17 08:25:44 -07:00
MattJackson 4d22cf632f iter6 cleanup: stop tracking (agent dir)worktrees/ 2026-05-17 08:10:29 -07:00
MattJackson 611ad77586 iter6: revert depth bump + WRITEBACK_CHUNK_BYTES 32->8 MiB
iter5 (256-frame channel) regressed -1.8 MB/s vs iter4. Reverting to
32 frames.

iter4 sample pattern shows clear ~30 s oscillation (peak 45 MB/s →
dip 3 MB/s → recovery). Matches Linux vm.dirty_expire_centisecs
default (30 s). 32 MiB chunks at 25 MB/s issue WAIT_AFTER every
~1.3 s, which can't outrun the kernel's own page-age limit, so
pages buildup then flush in bursts. Smaller 8 MiB chunks
(WAIT_AFTER every ~0.33 s) should keep the dirty-page set young
and eliminate the periodic flush-burst dip.
2026-05-17 08:09:57 -07:00
MattJackson 60590193e3 iter5: READ_PIPELINE_DEPTH 32 -> 256 frames
iter4 measurement (Phase 2.5 + DONTNEED, no FileSectorSource buffer)
showed mean 24.0 MB/s but with dips to 2.8 MB/s. Channel at 32 frames
(~1.6 MiB at 50 KB/frame avg) drains in <100 ms on any producer pause,
starving the consumer.

256 frames = ~12 MiB ≈ 3-4 sec of consumer drain at the rolling-avg
floor we want to hit. Should coast through producer micro-stalls
without idle gaps.
2026-05-17 07:53:41 -07:00
MattJackson 5f24843d3c Revert "iter2: restore FileSectorSource readahead buffer (32 MiB)"
This reverts commit ecf9a4d01e.
2026-05-17 07:38:34 -07:00
MattJackson ecf9a4d01e iter2: restore FileSectorSource readahead buffer (32 MiB)
iter1 baseline (Phase 2.5 + DONTNEED restored) measured at 18.4 MB/s
mean on Civil War remux — well below the rig's 37 MB/s concurrent-r+w
ceiling. Per-sector pread is the producer-side bottleneck: ~50us per
pread on NFS = ~19k preads/sec = effective ceiling near what we see.

The 0.21.3 bypass commit cited an A/B test showing the buffer hurt
throughput on NFS bidirectional workloads. That test was taken under
the 0.21.7 producer polling cap; once the cap is gone, the cap was
the bottleneck, not the buffer. Same invalidation pattern as the
0.21.5 Phase 2.5 revert.

Restored the buffered path. DONTNEED page-cache eviction from 0.21.6
stays intact.
2026-05-16 22:46:22 -07:00
MattJackson 8ec767339e Revert "io/writeback: medium-agnostic by construction, no DONTNEED, single detection point"
This reverts commit f4a881b25d.
2026-05-16 17:02:20 -07:00
MattJackson 335e8b68cc Reapply "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)"
This reverts commit 579c94bd23.
2026-05-16 17:02:15 -07:00
MattJackson 1137b4ea1f v0.23.2: bump version 2026-05-16 12:36:28 -07:00
MattJackson 7788c560d1 v0.23.1: bump version 2026-05-16 11:35:41 -07:00
MattJackson 84e0c2ca69 v0.23.0: bump version 2026-05-16 11:17:34 -07:00
MattJackson f4a881b25d io/writeback: medium-agnostic by construction, no DONTNEED, single detection point
Three coupled changes that together meet the "stable max throughput
on any medium" bar:

1. Remove `is_nfs` from `skip_wait`. WAIT_AFTER runs on every medium
   now. The `bounded_syscall` 30s safety net (already in place) covers
   the wedged-FS case generically — no need to predict NFS-hangs at
   compile time. Re-applies the 0.21.12 fix that landed the flat
   25 MB/s on NFS in the first place.

2. Drop the `posix_fadvise(DONTNEED)` calls after WAIT_AFTER. Bounded
   *dirty* pages (the actual invariant) does not require evicting
   *clean* pages. DONTNEED was forcing read-modify-write on any
   in-window seek-then-write — exactly the pattern matroska
   cluster-size backpatches produce. Empirical 2026-05-15: write side
   of mux fed NFS at 43 MB/s while the file grew at 25 MB/s, an 18
   MB/s overhead almost certainly composed of those RMW cycles. The
   kernel reclaims clean pages under LRU when memory is actually
   needed — we don't have to ask.

3. Replace `detect_nfs(fd) -> bool` with `detect_storage_class(fd)
   -> (StorageClass, chunk_bytes_seed)`. Medium detection happens
   *once* at construction and produces *one* output: the initial
   `chunk_bytes` seed for the autotuner. NFS gets 64 MiB (commit ack
   ~10-30 ms; needs bigger chunks to amortize); other media use the
   caller's hint. No hot-path branches on medium. The autotuner drives
   all subsequent decisions from measured WAIT_AFTER p95 latency,
   identically on every (OS, FS) combination.

Module-level doc rewritten to match: no more "NFS escape hatch", no
"is_nfs" framing. The whole writeback module is now medium-agnostic
except for one labelled detection point.
2026-05-16 11:17:30 -07:00
MattJackson 1edda7fedb v0.22.1: bump version 2026-05-16 10:12:34 -07:00
MattJackson 3dd71a0650 v0.22.0: bump version 2026-05-16 09:39:44 -07:00
MattJackson 579c94bd23 Revert "io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)"
This reverts commit 2a33364253.
2026-05-16 09:30:44 -07:00
MattJackson 3b2a68fe7a Revert "io/pipeline: bump WRITE_PIPELINE_DEPTH 16 → 32"
This reverts commit d703ce439b.
2026-05-16 09:30:44 -07:00
MattJackson 5b56f112a0 Revert "io/writeback_file: coalesce consecutive Cmd::Writes in the writer thread"
This reverts commit fc13268dc6.
2026-05-16 09:30:44 -07:00
MattJackson f27e4c3088 Revert "io/writeback/linux: drop is_nfs skip — bounded cache works on every medium"
This reverts commit dffee56102.
2026-05-16 09:30:44 -07:00
MattJackson b3576f2ae6 Revert "io/writeback: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB"
This reverts commit 50867516b8.
2026-05-16 09:30:44 -07:00
MattJackson 39f2d0e992 v0.21.14: bump version 2026-05-15 12:02:43 -07:00
MattJackson 50867516b8 io/writeback: WRITEBACK_CHUNK_BYTES 32 -> 128 MiB
The pipeline's adaptive autotuner grows chunk_bytes only when the p95
WAIT_AFTER latency exceeds 200 ms. On NFS, sync_file_range(WAIT_AFTER)
translates to an NFS COMMIT RPC whose ack lands within ~10 ms — so
the autotuner never triggered and the pipeline stayed at the original
32 MiB initial value forever.

That capped sustained mux throughput by paying NFS COMMIT-RPC overhead
roughly once per second of writes. Bidirectional mountstats on rip1
2026-05-15: write side 29 MB/s (RTT 30 ms but exec_time 257 ms — 220 ms
queue/serial waiting) while concurrent dd on the same disk shows
~91 MB/s write + ~65 MB/s read available.

128 MiB initial drops COMMIT cadence 4x while keeping the bounded-
cache invariant intact (worst-case dirty pages ~2 x chunk = 256 MiB,
well under vm.dirty_ratio = 6.6 GB on the 32 GB rig). The adaptive
autotuner can still grow further (up to 256 MiB) or shrink if
WAIT_AFTER ever measures sub-20 ms p95 on faster media.
2026-05-15 12:02:40 -07:00
MattJackson 193e7c1680 v0.21.13: bump version 2026-05-15 10:20:37 -07:00
MattJackson b6463729e8 v0.21.12: bump version 2026-05-15 09:55:41 -07:00
MattJackson dffee56102 io/writeback/linux: drop is_nfs skip — bounded cache works on every medium
The WritebackPipeline's WAIT_AFTER + posix_fadvise(DONTNEED) dance
keeps dirty pages bounded at ~2 × chunk_bytes by waiting for each
chunk's writeback to commit before issuing the DONTNEED hint to drop
it from cache. The original 0.18-era design unconditionally skipped
this on NFS on the premise that "NFS clients have their own buffering
and commit semantics that handle dirty-page bounds without us forcing
the issue."

Empirically wrong. On unraid-1 NFS the kernel client buffers dirty
pages up to vm.dirty_ratio (default 20% of RAM = ~6.6 GB on the rip1
host) before the kernel forces writeback and throttles app writes.
Result on 0.21.11 mux measured 2026-05-15: mux throughput cycled
between ~45 MB/s (cache absorbing) and ~7 MB/s (cache draining under
throttle) on a ~100 s period — exactly the burst-flush pathology this
pipeline was built to fix, but disabled on the medium it actually
runs on. /proc/meminfo Dirty: column climbed lockstep with mux
write rate during the slow half of every cycle, confirming the cause.

The original safety concern — `sync_file_range(WAIT_AFTER)` hanging
indefinitely on a wedged NFS server — is already handled by
`wait_after_with_timeout`'s `bounded_syscall` wrapper (30 s deadline).
If a real WAIT_AFTER call exceeds the deadline the pipeline flips to
the `degraded` state and skips WAIT_AFTER + DONTNEED for the rest of
its life — same effect as the old NFS branch, but only triggered when
something is genuinely broken rather than as a blanket exception.

This change is medium-agnostic: every medium goes through the same
path now, every medium gets the same safety net, and the
ADAPTIVE_WINDOW chunk-size autotuner (lines 226-255 — measures p95 of
WAIT_AFTER and resizes between 4 MiB and 256 MiB) finally activates
on NFS where previously it was dead code. Slow medium auto-grows
chunks to amortise per-chunk overhead; fast medium auto-shrinks to
keep cache pressure tight; nothing in the code special-cases the
filesystem type.

`is_nfs` is still detected (for logging + observability) but no
longer keys `skip_wait`. Module doc + startup log line updated to
match.
2026-05-15 09:55:38 -07:00
MattJackson d8f27cdee0 v0.21.11: bump version 2026-05-14 22:18:33 -07:00
MattJackson fc13268dc6 io/writeback_file: coalesce consecutive Cmd::Writes in the writer thread
Pre-coalescing the Phase-2.5 writer thread issued one `file.write_all`
syscall per `Cmd::Write` dequeued. The mux side calls
`WritebackFile::write_all(buf)` per PES frame, typically 30-200 KB.
On NFS that translates to one RPC per syscall, capping per-thread
throughput at `(wsize / rtt) × inflight` — well below what the same
disk delivers under a 1 MiB `dd oflag=direct` workload (empirical
2026-05-15: dd 71 MB/s vs mux ~25 MB/s sustained, with instantaneous
samples bursting 7→108 MB/s as the kernel page cache filled and
drained on its own cadence).

Coalesce instead: dequeue drains consecutive `Cmd::Write` items off
the ring up to a 1 MiB byte budget, returns them as
`DequeuedWork::Writes(Vec<Vec<u8>>)`, and the run loop concatenates
into one contiguous buffer and issues a single `file.write_all`.
Non-write commands (Seek, Flush, SyncAll, Finish) break the run and
are returned one at a time as `DequeuedWork::Other`, preserving their
ordering relative to the writes.

Single-buffer fast path avoids the concat allocation when only one
write is in the queue at dequeue time. A single oversize write (e.g.
the rare matroska cluster larger than 1 MiB) is admitted alone so it
still makes progress — the kernel splits internally.

This is generic across mediums: bigger app writes are at-least-as-
good on local SSD, HDD, or NFS. On fast storage the ring rarely fills
so coalescing is mostly a no-op; on slow storage with significant
per-RPC overhead it materially improves throughput.

Tests in `tests/` (write_then_drop_persists_bytes, sync_all_drains,
seek_then_patch_roundtrip, flush_is_observed_in_order) still pass —
ordering and durability semantics are unchanged.
2026-05-14 22:18:23 -07:00
MattJackson d9b55f02d0 v0.21.10: bump version 2026-05-14 20:43:21 -07:00
MattJackson 35d66d2059 v0.21.9: bump version 2026-05-14 20:21:36 -07:00
MattJackson d703ce439b io/pipeline: bump WRITE_PIPELINE_DEPTH 16 → 32
The depth was conservative because pre-0.21.8 a full sync_file_range
stall on NFS could traverse this channel and pin the producer. With
0.21.8's restored Phase 2.5 writer thread + 128 MiB byte-bounded ring
inside WritebackFile, every blocking syscall happens downstream of
this channel — never on it. The original "smaller buffer reduces
backpressure risk" rationale no longer applies.

Empirical (2026-05-15 Civil War UHD remux on 0.21.8): 30.9 MB/s
sustained but instantaneous samples spanning 9-53 MB/s, stdev 9.3.
Distribution clusters 52% of samples in 25-35 MB/s but has a long
9-15 MB/s tail. The tail corresponds to brief stalls in the matroska
builder when the sink momentarily lags — exactly the case a deeper
inter-thread channel covers. 32 frames at PES-frame sizes is still
well under a megabyte of additional memory, so the cost is zero.
2026-05-14 20:21:20 -07:00
MattJackson 6112a26d15 v0.21.8: bump version 2026-05-14 18:54:10 -07:00
MattJackson 2a33364253 io/writeback_file: restore Phase 2.5 (writer thread + bounded ring)
Reverts 523af46. That revert was made on the premise that Phase 2.5
caused a ~60% mux throughput regression on NFS bidirectional workloads.
The premise was wrong: at the time of measurement the producer was
capped at ~8 MB/s by a 50 ms thread::sleep poll in
Pipeline::send_with_halt (fixed in v0.21.7's io/pipeline change), so
the comparison was measuring the polling cap on both sides.

With the polling cap removed, direct passthrough exposes the kernel's
default dirty-page writeback pathology on NFS: writes accumulate, the
kernel periodically bursts a flush, app writes block for the burst.
Observed empirically on Civil War UHD remux 2026-05-14: 5-45 MB/s
spiking around a ~21 MB/s sustained mean, dominated by burst-flush
back-pressure cycles.

Phase 2.5 decouples the mux thread from the file syscall:
  * mux writes complete instantly into a 128 MiB byte-bounded SPSC ring,
  * a dedicated writer thread executes the real File writes, seeks,
    and sync_file_range calls; can sit in a kernel burst without
    blocking the mux pipeline,
  * backpressure via Condvar notify/wait, no polling primitive,
  * the ActiveClusterBuffer fast-path preserves the original MKV
    cluster-backpatch optimisation so in-window seeks don't drain
    the current writeback chunk.

Halt-safety is preserved: every blocking writeback syscall on the
writer thread still routes through bounded_syscall with a 60 s
deadline. A wedged NFS server cannot trap the writer indefinitely;
the muxer keeps queueing into the ring; the kernel page cache and
the ring together absorb the stall.
2026-05-14 16:42:00 -07:00
MattJackson fa3872ddcc v0.21.7: bump version 2026-05-14 11:10:57 -07:00
MattJackson c427389f36 io/pipeline: replace polling-send with kernel-wakeup channel
The halt-aware send loop polled try_send on a 50 ms sleep slice when
the channel was full. That capped producer throughput at 1 / 50 ms =
20 frames/sec ≈ 1 MB/s at typical PES frame sizes — way below NFS,
let alone local SSD/NVMe. Multi-day diagnostic 2026-05-13/14 surfaced
it as the root cause of the 18 → 3 MB/s mux throughput regression on
0.21.x.

Replaced std::sync::mpsc::sync_channel with crossbeam_channel::bounded
and switched send_with_halt to send_timeout. Producer now BLOCKS on
consumer drain (kernel-wakeup) instead of polling — the timeout slice
(250 ms) only fires when the producer is waiting for stop signal
observation, never on the happy path.

Result: no throughput cap from this primitive at any medium speed.
Mux is bounded by actual storage / network bandwidth, not by our
channel implementation.

Same fix needs to land in autorip's mux.rs producer-side
sync_channel (separate sibling commit).

Documented in (internal)/memory/
feedback_send_with_halt_poll_throttle.md.
2026-05-14 11:10:39 -07:00
MattJackson 6980f562df v0.21.6: bump version 2026-05-14 09:16:34 -07:00
MattJackson e3b2c9d850 file_sector_source: restore read-side DONTNEED + SEQUENTIAL (the actual fix)
Empirical: isolated NFS read 70 MB/s + write 93 MB/s on the rip1 setup
right now, but mux throughput pinned at 2.7 MB/s on 0.21.5. NOT
environmental — code regression.

Root cause: Phase 1 silently dropped the read-side
posix_fadvise(POSIX_FADV_DONTNEED) eviction that the pre-Phase-1 (0.20.7)
hot path had. Without it, an 85 GB streaming ISO read pins the entire
file in the kernel page cache, starving concurrent MKV writeback. 0.21.2
then also dropped the POSIX_FADV_SEQUENTIAL hint on the same theory,
compounding the regression.

Restored both, per-OS split:
- linux: posix_fadvise(SEQUENTIAL) at open + posix_fadvise(DONTNEED)
  on consumed 32 MiB windows
- macos: F_RDADVISE hint at open (kept); drop_window no-op (macOS unified
  buffer cache less prone to the pin pathology)
- windows / other: both no-op stubs

Target mux speed restored to 20+ MB/s (per concurrent-NFS math:
70/2 read × 0.73 MKV/ISO ratio ≈ 25 MB/s achievable).
2026-05-14 09:16:23 -07:00
MattJackson a383200ef1 v0.21.5: bump version 2026-05-14 01:46:59 -07:00
MattJackson 51d15f551e file_sector_source: allow(dead_code) on residual buffer fields post-0.21.3 bypass 2026-05-14 01:46:51 -07:00
MattJackson 523af461a5 io/writeback_file: revert write path to direct passthrough (0.20.7 baseline)
The Phase 2.5 writer-thread + bounded ring + ActiveClusterBuffer
architecture introduced a ~60% mux throughput regression on NFS
bidirectional workloads (18 MB/s -> 7-8 MB/s in 0.21.x). Other
candidates (FileSectorSource readahead buffer, per-OS sink split)
ruled out empirically across 0.21.1-0.21.4.

Reverts WritebackFile's hot path to direct File passthrough matching
the 0.20.7 baseline:
  * Write/write_all/flush call straight through to self.file.
  * Seek calls through to self.file and notifies the pipeline.
  * sync_all runs pipeline.finalize() then per-OS durable_sync.

Kept (intentional): the per-OS file split established in earlier work
(writeback_file/{linux,macos,windows,other}.rs). create_with_size_hint
still dispatches to platform::preallocate and sync_all still routes
through platform::durable_sync (bounded_syscall 60s deadline on
Linux/macOS), so halt-safety on a wedged NFS fsync is preserved.

Removed:
  * Cmd / RingState / Shared / WriterState / writer_thread_main
  * ActiveClusterBuffer (active-cluster window for in-window seeks)
  * push_command / dequeue / publish_error / mark_writer_gone
  * RING_CAPACITY_BYTES, ACTIVE_CLUSTER_WINDOW_BYTES,
    MAX_WRITE_CHUNK_BYTES, WRITER_THREAD_NAME constants
  * writer JoinHandle field + Drop join logic
  * Tests that asserted internal writer-thread state
    (backpressure_blocks_when_ring_full, in/out_of_window_seek_*,
    active_cluster_buffer_*, writer_thread_panic_surfaces_on_drop)

Kept tests (correctness-of-output only):
  * write_then_drop_persists_bytes
  * sync_all_drains_and_flushes (renamed from sync_all_blocks_until_ring_drains)
  * seek_then_patch_roundtrip (replaces in_window_seek_then_patch_roundtrip)
  * flush_is_observed_in_order

mod.rs: 1026 -> 303 lines. linux/macos/windows/other.rs untouched.
2026-05-14 01:40:37 -07:00
MattJackson 695b65149e v0.21.4: bump version 2026-05-14 00:09:28 -07:00
MattJackson d3f9560689 test/file_sector_source: drop buf-state assertions after 0.21.3 bypass
The three tests (multi_sector_read_spanning_buffer_boundary,
backward_seek_rebuffers, partial_buffer_at_eof) were asserting
internal buf_start_lba / buf_len_sectors state. With 0.21.3's
read-path bypass, those fields are no longer mutated. The
byte-level contract assertions (read returns correct bytes for
every scenario the tests cover) remain intact.
2026-05-14 00:09:21 -07:00
MattJackson 1fa5a7d27f v0.21.3: bump version 2026-05-13 23:57:53 -07:00
MattJackson 55219070f1 io/file_sector_source: bypass app-level buffer — direct pread per call
The 32 MiB readahead window (0.21.0–0.21.1) regressed mux throughput
on NFS bidirectional workloads vs the pre-Phase-1 0.20.7 baseline
(18 -> 7-8 MB/s). The 0.21.2 4 MiB shrink made it worse (5-6 MB/s).
Both signs point at the application-level buffer itself, not the size.

This commit bypasses the buffer entirely on the read path — every
read_sectors call seeks and pread()s direct to the file. That matches
0.20.7's hot path. Kernel readahead handles the policy; on NFS that
interleaves naturally with concurrent writes on the same TCP
connection.

Buffer state fields and refill/buffer_covers are kept so the
structure is preserved for a future per-source-type policy (e.g. a
local-disk source where batched reads ARE beneficial), and so the
existing tests still exercise that machinery.
2026-05-13 23:57:46 -07:00
MattJackson a34c419521 v0.21.2: bump version 2026-05-13 22:28:09 -07:00
MattJackson b179846f5d io/file_sector_source: throttle readahead for NFS bidirectional workloads
Empirical regression observed during 0.21.1 mux test on rip1/unraid-1:
historical 0.20.7 baseline averaged ~18 MB/s mux throughput; 0.21.1
dropped to ~7-8 MB/s flat. Same NFS source + destination, same disc.

Suspect: 32 MiB FileSectorSource readahead + posix_fadvise(SEQUENTIAL)
together saturate the TCP connection on read bursts, starving the
writer thread's concurrent NFS writes (mux reads the source ISO and
writes the MKV over the same connection).

- READAHEAD_BUF_BYTES: 32 MiB -> 4 MiB. Matches NFS rsize=1 MiB * 4
  round-trips per refill, interleaves cleanly with writes.
- linux/hint_sequential: now no-op. Kernel's default ~128 KiB
  readahead is what we want on NFS-backed ISOs (the dominant case).
  Per-OS file stays so we can re-enable a hint cleanly later if a
  different path benefits.
2026-05-13 22:27:57 -07:00
MattJackson e110e80e6e lint: silence clippy::unnecessary_cast on glibc + fix doc list indent
CI's lint workflow runs clippy on linux target where:
- platform/fs_type/linux.rs and io/writeback/linux.rs: the i64 cast
  on buf.f_type / NFS_SUPER_MAGIC is unnecessary on glibc x86_64 (both
  already i64) but required on musl (c_ulong); silence the lint via
  inline allow with explanatory comment.
- mux/m2ts_mux/packet.rs: doc comment continuation across lines was
  parsed as an unindented list item. Reworded to a single flowing
  sentence.
2026-05-13 20:53:26 -07:00
MattJackson 597fa34099 v0.21.1: bump version 2026-05-13 20:30:09 -07:00
MattJackson 52eeb949e3 mux: Rust 1.86 compat + fmt
- m2ts_mux: replace u64::is_multiple_of (stable in 1.87+) with %.
  CI on the pinned 1.86 toolchain rejected the unstable feature use.
- mux/{fmp4,hevc,m2ts_mux}: rustfmt drift cleanup (test-code wrapping).
2026-05-13 20:30:01 -07:00
MattJackson 6ca62b8411 v0.21.0: bump version 2026-05-13 20:17:47 -07:00
MattJackson 04195c27d8 io+mux: phase 3 — streaming sinks + sequential container muxers
SocketSink + UdpSocketSink (`src/io/sink/socket.rs`) — sequential-only
TCP/UDP write destinations. SocketSink wraps BufWriter<TcpStream> with
1 MiB capacity, tunes SO_SNDBUF on construction, calls shutdown(Write)
on finish(). UdpSocketSink emits one datagram per write — caller
packetizes. Both impl Write+Send and thus satisfy SequentialSink via
the Phase 2 blanket; neither impls Seek, so RandomAccessSink is
correctly inaccessible (compile error to mux MKV onto a socket).

New sequential container muxers in src/mux/:

  - hevc/ — raw HEVC Annex B elementary stream. Length-prefixed NALU
    → 00 00 00 01 NALU. hvcC parsing emits VPS/SPS/PPS once at stream
    head. Fully ships.

  - m2ts_mux/ — standard MPEG-TS (188-byte packets). Single program,
    HEVC video on PID 0x100, optional AC3/TrueHD audio on PID 0x101.
    PAT+PMT re-emitted every 250 packets; PCR stamped on video every
    40 packets. Hand-rolled, no new deps. Distinct from the existing
    BD-TS (192-byte) `mux::m2ts::M2tsStream` — that path stays as-is.

  - fmp4/ — fragmented MP4. STUB: ftyp + minimal moov skeleton with
    one HEVC video trak + mvex/trex. Media fragments (moof+mdat) are
    TODO for v0.22.0 — write_video accumulates frames into a pending
    buffer that finish() clears. Init segment is well-formed enough
    that init_segment_starts_with_ftyp_then_moov asserts the box
    chain.

17 new unit tests added (socket round-trip, HEVC Annex B conversion,
M2TS packet alignment + PAT/PMT cadence + per-PID CC, fMP4 box chain).
All 514 lib tests + 17 new = pass on Rust 1.86 (fmt + clippy + test
via (internal)/scripts/precommit.sh libfreemkv).

No new dependencies. No version bump. Don't-touch list clean.
2026-05-13 20:17:33 -07:00
MattJackson 5a8f8e54e1 io: phase 2.5 — writer thread + bounded ring + active-cluster buffer
WritebackFile now offloads all File I/O to a dedicated writer thread.
Muxer's Write/Seek/sync_all push into a bounded byte ring; the writer
drains the ring and runs the syscalls.

- Commands: Write(Vec<u8>), Seek(SeekFrom), SyncAll(oneshot),
  Finish(oneshot). One ring carries all four; ordering preserved.
- ActiveClusterBuffer mirrors the last ACTIVE_CLUSTER_WINDOW_BYTES
  written. Seek-back within window → in-memory patch + re-emit
  (no forced drain). Seek-back outside window → real drain+seek.
  Wins for MKV: cluster size patches almost always land inside the
  active window; only the end-of-mux Cues + segment header backpatch
  fall outside.
- Ring capacity RING_CAPACITY_BYTES = 128 MiB. Backpressure on full
  blocks the muxer (correct semantics for archival workflows).
- All syscalls in writer thread wrapped in bounded_syscall(60s) so
  a wedged NFS doesn't pin the thread forever.
- Drop drains via Finish + joins the writer thread; sync_all blocks
  until ring drained AND underlying fsync completes.
2026-05-13 20:17:29 -07:00
MattJackson 1a2c830cf6 fmt: rustfmt reorder of cfg-gated use/mod decls in sink + fs_type 2026-05-13 20:10:57 -07:00
MattJackson 637c2bfe36 v0.20.10: bump version 2026-05-13 19:59:07 -07:00
MattJackson 0d28357b4d io+platform: phase 2 — sink trait split + fs_type detection
Introduces the SequentialSink / RandomAccessSink trait pair under
io::sink and an open_for_mkv dispatch helper that picks WritebackFile
on Linux+NFS and LocalFileSink everywhere else. LocalFileSink wraps
BufWriter<File> with a 4 MiB buffer and exposes a per-OS preallocate
path (fallocate on Linux, F_PREALLOCATE on macOS, no-op fallback).

Adds platform::fs_type::detect with a per-OS split (statfs on Linux /
macOS, UNC heuristic on Windows, Unknown elsewhere) so construction-
site dispatch has a single primitive to call.

Blanket impls cover the common shapes: any Write+Send is a
SequentialSink, and any SequentialSink+Seek is a RandomAccessSink.
WritebackFile satisfies the random-access trait via the blanket impl
without needing an explicit per-type impl. No callers wired yet — the
mux::resolve construction sites stay on WritebackFile pending Phase 3.

Tests: 5 new sink/preallocate tests + 3 fs_type tests (1 ignored,
needs a real NFS mount). cargo +1.86 fmt + clippy + tests all green.
2026-05-13 19:58:54 -07:00
MattJackson 5495332f07 v0.20.9: bump version 2026-05-13 19:52:48 -07:00
MattJackson e22fc6fd47 io: phase 1 buffering — read-side flatness
Three changes targeting 0.20.9's "muxer never read-stalls on NFS read
latency" invariant:

A. FileSectorSource gets a 32 MiB internal read-ahead buffer
   (READAHEAD_BUF_BYTES). Splits out from src/sector/file.rs into
   src/io/file_sector_source/ with per-OS open hints (Linux
   posix_fadvise(SEQUENTIAL), macOS fcntl(F_RDADVISE) with 64 MiB
   cap, Windows TODO stub, BSD/illumos no-op). Backward seeks
   rebuffer; partial reads at EOF return only the bytes that exist;
   oversize-request bypass for count > BUF_SECTORS.

B. WritebackFile inline #[cfg(target_os = "linux")] blocks split
   into per-OS files under src/io/writeback_file/. Linux unchanged
   (fallocate KEEP_SIZE, fsync via bounded_syscall). macOS gets a
   real F_PREALLOCATE + F_FULLFSYNC impl (was a "skipped (non-linux)"
   debug log before). Windows is a stub (FlushFileBuffers via
   std sync_all; TODO for SetFileValidData). BSDs/illumos fall back
   to std sync_all.

C. New byte_channel module — byte-bounded producer/consumer wrapping
   std sync_channel with Mutex/Condvar byte accounting. Sender blocks
   when used_bytes + item.byte_size() > capacity. HasByteSize impl
   for PesFrame. Default cap BYTE_CHANNEL_DEFAULT_CAPACITY = 64 MiB,
   sized to absorb worst-case NFS read p99 (~2 s × UHD peak compressed
   ~15 MB/s). The mux call site lives in autorip (out of scope here);
   this lands the primitive in libfreemkv for autorip to adopt.

Test counts: byte_channel +6, file_sector_source +5, sector::file
round-trip suite (3) preserved. passn_handler_ab.rs A/B fixture
(8 profiles) still green.

precommit.sh libfreemkv: fmt + clippy + test all green on Rust 1.86.

No version bump; no Cargo.lock changes; no forbidden-file edits
(disc/patch.rs, disc/read_error.rs, io/pipeline.rs,
tests/passn_handler_ab.rs).
2026-05-13 19:48:23 -07:00
MattJackson 5b98c13e47 v0.20.8: bump version 2026-05-13 19:18:51 -07:00
MattJackson 8f8f1a62a2 io+disc: bundle 0.20.8 dev work
- io/pipeline.rs: add send_with_halt + finish_with_halt for cooperative
  halt during blocking producer-consumer handoffs; 5 new tests
- disc/patch.rs: split Disc::patch body (1168 -> 316 LOC) into named
  helpers (compute_initial_state, prime_cache, check_range_watchdog,
  handle_skip_limit, compute_damage_skip, handle_read_success,
  handle_read_failure, report_patch_progress, build_outcome) with
  PatchLoopState / RangeFrame structs; references shared
  PATCH_DAMAGE_THRESHOLD_PCT constant
- disc/read_error.rs: add pub const PATCH_DAMAGE_THRESHOLD_PCT = 6;
  ReadCtx::for_patch() now references the shared constant (was a
  latent 12 / 6 inconsistency)
- tests/passn_handler_ab.rs: 8-profile A/B fixture locking current
  patch-side recovery behavior (clean / all-medium / alternating /
  edge-bad-good-middle / single-bad / deep-pit / medium-then-good /
  batch-fail). Goldens captured pre-unification; will catch any
  future refactor that breaks the size-aware skip cap.
2026-05-13 19:15:48 -07:00
MattJackson e3cfd27942 v0.20.7: version bump for unified release (no source changes)
The 0.20.7 work lives downstream in autorip — process-level safety net
(hard watchdog 5-min mux escalation + std::process::exit, restart-loop
counter + auto-quarantine after 3 attempts, partial-state preservation,
failure_reason surfaced in /api/state).

Bumping libfreemkv to 0.20.7 keeps the 4 crates on the unified release
train per project docs.
2026-05-13 14:29:40 -07:00
MattJackson aa12bdad62 v0.20.6: io::bounded — halt-safe wrapper for blocking syscalls
Generalizes 0.20.5's hand-written wait_after_with_timeout into a
reusable primitive. After this change, every blocking syscall in the
recovery + mux paths is wrapped, so cooperative Halt has bounded
~250 ms latency reach even into kernel-owned thread states.

New module src/io/bounded.rs:
- BoundedError { Halted, Timeout, WorkerLost }
- bounded_syscall<F, R>(halt: Option<&Halt>, timeout, op) -> Result<R, BoundedError>
- Worker thread runs op; main thread recv_timeouts on a rendezvous
  channel in 250 ms slices, polling halt between slices.
- Worker is intentionally leaked on timeout/halt — kernel reaps when
  the syscall finally returns or at process exit. Calling thread is
  NEVER trapped inside a kernel call.
- 6 unit tests cover the happy path + each error variant.

Refactored callsites:
- src/io/writeback/linux.rs::wait_after_with_timeout now delegates
  to bounded_syscall. ~30 LOC of duplicated channel/thread plumbing
  deleted. Same semantics, cleaner.
- src/io/writeback_file.rs::WritebackFile::sync_all now wraps the
  final libc::fsync(fd) with bounded_syscall (60 s deadline). On
  timeout: log error at target=mux and return Ok — kernel will flush
  on close, best-effort but bounded. Covers FileSectorSink::finish,
  PatchSink::close, SweepSink::close, and the mux MKV finalize path
  (they all sync through WritebackFile).

What still hangs (deliberately not wrapped — too hot a path):
- File::write itself. Per-frame write on a wedged NFS could still
  block; but back-pressure from a stuck consumer means the producer
  notices within seconds, not minutes — different failure mode than
  the WAIT_AFTER hang 0.20.5/0.20.6 fix.
2026-05-13 14:22:24 -07:00
MattJackson ef3895cdc5 v0.20.5: NFS-aware writeback + bounded sync_file_range timeout
Targets the recurring mux hang on NFS dest where the consumer thread
sits indefinitely inside libc::sync_file_range(SYNC_FILE_RANGE_WAIT_AFTER)
because the NFS server never returns a commit ack. The whole rip
wedges; halt is cooperative and can't reach inside a kernel syscall.

A. NFS detection at WritebackPipeline construction (fstatfs f_type ==
NFS_SUPER_MAGIC 0x6969). When NFS:
- Skip SYNC_FILE_RANGE_WAIT_AFTER entirely.
- Skip posix_fadvise(DONTNEED) — NFS client handles its own buffering.
- Still issue async SYNC_FILE_RANGE_WRITE (harmless hint).
Cannot hang on a syscall not made. fstatfs failure fails open (assume
local). Logged at info on construction so operators see which strategy
is active. The whole hang vector is removed for NFS deployments.

B. Hard timeout on WAIT_AFTER for non-NFS (defense in depth, since
even a degraded local disk could in principle hang the syscall).
Each WAIT_AFTER runs on a worker thread; main thread waits on a
sync_channel rendezvous with 30s deadline. On timeout: log error,
set per-pipeline 'degraded' Arc<AtomicBool>, downgrade to NFS-style
skip for the rest of the pipeline's life. Worker thread leaks
intentionally — it'll unwind when the syscall eventually returns or
the process exits. Converts indefinite freeze into 'log loud +
downgrade + keep ripping'.

C. Diagnostic logging for the 73%-of-this-movie reproduction:
- WritebackFile::seek logs every non-trivial seek (from, to, signed
  delta) at target=mux so we can see if MkvMuxer seeks back before
  a stall.
- WritebackPipeline::finalize logs the chunk being finalised before
  any WAIT_AFTER call, so a hung chunk is identifiable by offset.

No new dependencies. macOS / Windows noop stubs unchanged. Net
+198 LOC libfreemkv (mostly writeback/linux.rs).
2026-05-13 14:09:55 -07:00
MattJackson 2dcf969ac8 v0.20.4: mux performance + observability — universal across storage
Four targeted changes to maximize mux throughput regardless of storage
backend (local SSD, local HDD, NFS, network share) and surface enough
log data to diagnose 'mux slow' reports without a re-rip:

1. POSIX_FADV_SEQUENTIAL on FileSectorSource::open (Linux only).
   Widens the kernel readahead window for sequential ISO reads. One
   syscall at open, free on every storage type.

2. POSIX_FADV_DONTNEED on the ISO read side after every 32 MiB chunk.
   Mirrors the writeback DONTNEED that already runs on the write
   side. Keeps the read-side page cache bounded during multi-GB ISO
   reads — eliminates the OOM-pressure / eviction-storm risk on
   long mux runs. Linux only; per-drop trace at target="mux".

3. WritebackFile::create_with_size_hint(path, size_bytes) calls
   fallocate(FALLOC_FL_KEEP_SIZE) on Linux to pre-reserve extents
   for the output. Reported file size stays 0 (writes grow it
   naturally) but the on-disk extent allocation is contiguous —
   reduces extent fragmentation for big sequential muxes. Wired
   into mkv:// and m2ts:// output paths via DiscTitle::size_bytes.
   No-op on macOS/Windows; old create() kept with #[allow(dead_code)]
   for callers without a size hint.

4. Adaptive WRITEBACK_CHUNK_BYTES in the Linux writeback pipeline.
   Tracks sync_file_range(WAIT_AFTER) elapsed_ms in a rolling
   16-sample window. p95 > 200 ms → double chunk size (cap 256 MiB).
   p95 < 20 ms → halve (floor 4 MiB). One algorithm, both
   fast-storage (small chunks, responsive) and slow-storage (big
   chunks, fewer commit round-trips) optimized. Per-chunk trace +
   per-32-chunk debug snapshot + info-on-resize so an operator can
   see where the autoscaler settled.

All four are universal — no storage-type detection, no env vars to
flip, no per-deploy tuning required. Total +201/-6 across four files.
2026-05-13 13:50:44 -07:00
MattJackson 005f887bf9 v0.20.3: add halt check to Disc::patch backtrack inner loop
WO-5 (partial): the patch backtrack inner loop ('while bt_pos <
backtrack_end' in disc/patch.rs) issues per-sector reads to fill the
gap created by a damage-window skip. A long backtrack span can run
minutes; without an inline halt poll, the outer halt only takes
effect when control returns to the per-range loop. Adds a halt poll
at the top of each iteration so cancellation propagates inside the
backtrack span.

Per-sector read failures inside the backtrack already drop through
to the main fail path; this only changes the cancellation latency
between an /api/stop call and the producer actually unwinding. Drops
worst-case unwind from 'whole backtrack span × per-sector recovery
timeout' (minutes) to 'one in-flight SCSI command' (seconds).

The broader Arc<AtomicBool> → Halt migration on CopyOptions /
SweepOptions / PatchOptions / Drive::halt and Pipeline::send halt-
awareness is deferred — separate cycle, larger API impact.
2026-05-13 11:55:52 -07:00
MattJackson d7243a6044 v0.20.2: delete dead retry block + wedge-family cooldown in Disc::patch
WO-3c: Delete dead non-NOT_READY retry block (~100 LOC). The block
declared retry_count = 0 inside the per-iteration Err arm, so the
'MAX_NON_NOT_READY_RETRIES=3' budget actually fired exactly once
(1s pause + 1 retry) before falling through to NonTrimmed. The
'exponential backoff: 2s, 4s, 8s' comment was wrong by construction.
Cross-pass NonTrimmed retry (each pass gives the same sectors another
shot) already covers the recovery case, and gives the drive minutes
between attempts instead of 1-8 seconds — empirically what stochastic
recovery on the BU40N actually needs.

WO-4 (targeted slice): Add wedge-family cooldown on HARDWARE_ERROR /
ILLEGAL_REQUEST senses. These are what the BU40N's firmware fast-fail
state returns; every subsequent read in that state comes back in
<100ms. Pre-fix patch hammered the drive: mark NonTrimmed, sleep 1s,
advance, hit next wedge, mark, sleep 1s — exactly the rapid-retry
cadence the firmware is sensitive to. Now a wedge-family sense triggers
WEDGE_FAMILY_COOLDOWN_SECS=30 cooldown (matches read_error.rs's
ZONE_ENTRY_COOLDOWN_SECS), and WEDGE_ABORT_THRESHOLD=16 consecutive
wedges aborts the pass for autorip eject+reload. Any non-wedge read
clears the counter.

Also drops the duplicate NonTrimmed dispatch (Mapfile::record is
idempotent so it wasn't a correctness bug, but it doubled per-failure
consumer work).
2026-05-13 11:53:57 -07:00
MattJackson f1926c38dc v0.20.1: delete SectorReader, extract Disc::patch, doc/stub cleanup
WO-2 (delete SectorReader trait):
- The 0.18 trait split into SectorSource (read-only) and SectorSink
  (write-only) is final; the legacy SectorReader alias was a bridge.
- Renames every internal &mut dyn SectorReader (~25 sites) to
  &mut dyn SectorSource. The trait method capacity() becomes
  capacity_sectors() with a default of 0 (preserves SectorReader's
  default-0 behavior).
- Deletes the SectorReader trait, its blanket-to-Source bridge, and
  the FileSectorReader type alias. Adds explicit forwarding impls
  for Box<dyn SectorSource> and &mut dyn SectorSource so generic
  decorators like DecryptingSectorSource<S: SectorSource> compose.

WO-3a (extract Disc::patch):
- Moves Disc::patch (1230 lines) and bytes_bad_in_title from
  disc/mod.rs into disc/patch.rs as a split inherent impl. Zero
  behavior change — pure mechanical relocation. disc/mod.rs drops
  from 3,945 to 2,714 LOC.

WO-6 (partial):
- Deletes src/labels/png_filenames.rs — was a 72-LOC stub with
  detect() returning false, never wired into the PARSERS registry.

project docs doc drift fixes (audited 2026-05-13):
- JUMP_BASE_SECTORS: 256→1024 (64 MB base for UHD, not 8 MB)
- PASSN_DAMAGE_THRESHOLD_PCT: 12→6
- PASSN_SKIP_SECTORS_BASE: 64→32
- MAX_RANGE_SECS=180: replaced by proportional range_sectors × 25,
  capped at RANGE_BUDGET_CAP_SECS=1800.
2026-05-13 11:36:55 -07:00
MattJackson 4709a73c80 v0.20.0: delete FrameSource/FrameSink, keep single Stream trait
The 0.18 trait split into FrameSource (read-only) and FrameSink
(write-only) was an over-engineered API. Consumers don't think
"frame source backed by MKV" — they think "open MKV for reading".
The split paid a real API-complexity cost (two trait names, two
re-exports, dual impls per bidirectional type, deprecation bridge)
for one marginal property: compile-time direction-safety at the
trait-object boundary. The runtime error path on a wrong-direction
call (StreamReadOnly / StreamWriteOnly) is unambiguous and rare in
practice.

Deletions:
- pes::Stream is no longer #[deprecated]
- pes::FrameSource trait + its blanket-from-Stream bridge
- pes::FrameSink trait + the trampoline impls on every concrete type
- The compile-time-direction-safety test scaffolding
- Crate-root FrameSource / FrameSink re-exports

Additions:
- Stream is now Send-bounded (Stream: Send supertrait). Every
  concrete impl was already Send-compliant — Box<dyn Read + Send>
  and Box<dyn Write + Send> were already in place on the trait
  objects MkvStream / M2tsStream / etc hold internally. Promoting
  Send into the trait makes Box<dyn Stream> Send too, which lets
  autorip drop its SendStream unsafe newtype.

The public API is now: one Stream trait, one concrete type per
format, two constructors (open/create or input/output). Bidirectional
types route through internal Mode { Read | Write } discriminants.

Net: -347 lines libfreemkv, -38 lines autorip, -5 lines freemkv.
2026-05-13 08:42:14 -07:00
MattJackson 55bd1ee868 v0.19.1: repair Cargo.toml after botched 0.19.0 bump
v0.19.0 was tagged with a search-and-replace gone wrong:
rust-version, serde, and zip all had their version strings
replaced with "0.19.0". Edition 2024 rejected rust-version
0.19.0 (< 1.85), failing every CI build. No artifacts shipped
to crates.io.

Repair:
- rust-version: 0.19.0 → 1.86 (CI pin)
- serde: 0.19.0 → 1
- zip: 0.19.0 → 2

Also drops an unused start_lba binding in mux/disc.rs that
clippy 1.86 catches.
2026-05-12 21:29:07 -07:00
MattJackson 5285e8b6ec v0.19.0: bump version 2026-05-12 19:24:06 -07:00
MattJackson c5af61e46b v0.18.25: bump version 2026-05-12 16:27:28 -07:00
MattJackson ed73c1ab68 v0.18.25: add mux debug logging for reader type and stall detection 2026-05-12 16:27:16 -07:00
MattJackson 5ce2df0888 v0.18.24: bump version 2026-05-12 15:46:47 -07:00
MattJackson ea591a36a5 v0.18.23: bump version 2026-05-12 10:59:45 -07:00
MattJackson 368e10486e v0.18.22: bump version 2026-05-12 09:33:00 -07:00
MattJackson a99c4f8487 v0.18.21: format code 2026-05-12 09:18:52 -07:00
MattJackson 4df470e572 v0.18.21: fix borrow checker, make io public 2026-05-12 09:01:52 -07:00
MattJackson a0b13941fe v0.18.21: fix borrow checker error in debug logging 2026-05-12 08:49:33 -07:00
MattJackson 275c9da009 v0.18.21: verbose debug logging for pipeline stalls (fixed) 2026-05-12 08:37:42 -07:00
MattJackson da183722fa v0.18.20: bump version 2026-05-11 22:33:15 -07:00
MattJackson 4a5424eabe v0.18.20: bump version 2026-05-11 22:28:57 -07:00
MattJackson 293d89c586 v0.18.20: separate read/write pipeline depths 2026-05-11 22:18:56 -07:00
200 changed files with 91741 additions and 16488 deletions
+10 -1
View File
@@ -13,6 +13,7 @@ jobs:
- uses: dtolnay/rust-toolchain@1.86.0
with:
components: clippy, rustfmt
- uses: Swatinem/rust-cache@v2
- run: cargo fmt --check
# libfreemkv is a library — Cargo.lock is gitignored. --locked
# would always fail on a fresh runner because there's no committed
@@ -25,6 +26,7 @@ jobs:
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
- run: cargo test --tests
check-macos:
@@ -32,6 +34,7 @@ jobs:
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
- run: cargo check
check-windows:
@@ -39,4 +42,10 @@ jobs:
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- run: cargo check
- uses: Swatinem/rust-cache@v2
# Build the tests (not just `cargo check`): catches errors in test
# code and forces full codegen of the Windows-only SPTI transport
# (src/scsi/windows.rs), which never compiles on the Linux/macOS dev
# hosts. We don't `cargo test` here — the suite needs no drive but the
# extra build is the value; running tests is covered by the Linux job.
- run: cargo build --tests
+35
View File
@@ -0,0 +1,35 @@
name: leak-guard
# Self-contained public-repo leak gate. Public CI cannot reach the private
# tooling, so this encodes only the generic net: internal-infra references,
# tracked CLAUDE.md/.claude paths, and AI-attribution in commit messages.
# No project-specific reverse-engineering vocabulary lives here.
on: [push, pull_request]
jobs:
leak-guard:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Compute commit range
id: range
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
base="${{ github.event.pull_request.base.sha }}"
head="${{ github.event.pull_request.head.sha }}"
echo "range=$base..$head" >> "$GITHUB_OUTPUT"
else
before="${{ github.event.before }}"
after="${{ github.sha }}"
# New branch / first push: github.event.before is all-zeros.
if [ -z "$before" ] || [ "$before" = "0000000000000000000000000000000000000000" ]; then
echo "range=$after" >> "$GITHUB_OUTPUT"
else
echo "range=$before..$after" >> "$GITHUB_OUTPUT"
fi
fi
- name: Run leak-guard
run: bash ci/leak-guard.sh "${{ steps.range.outputs.range }}"
+15 -11
View File
@@ -22,29 +22,33 @@ jobs:
fi
echo "Version match: $CARGO_VER"
# Tests run as a PARALLEL TRIPWIRE: they fail the run if they fail, but the
# publish/release jobs do NOT `needs:` this job. The tag decision was already
# gated by the local precommit (same Rust 1.86, same commit). Binary consumers
# (freemkv/autorip/bdemu) git-tag-pin libfreemkv and therefore start building
# the instant this tag exists — so this test job and the crates.io publish
# below must NOT sit on their critical path.
test:
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: Swatinem/rust-cache@v2
# libfreemkv is a library — Cargo.lock isn't tracked, so --locked
# would always fail (no lockfile to lock against on a fresh runner).
- run: cargo test
publish:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- name: Publish to crates.io
run: cargo publish
env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
# NOTE: there is no crates.io publish job. libfreemkv is git-tag-only
# (`package.publish = false` — it git-deps the firmware crate freemkv-unlock,
# which never ships to crates.io). Every consumer git-tag-pins libfreemkv via
# a committed [patch.crates-io]; the git tag itself IS the release artifact.
# A `cargo publish` here fails hard on `publish = false`, so it was removed.
release:
needs: test
# Only needs `verify`; the GitHub Release can be cut as soon as the version
# check passes, in parallel with test + publish.
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
+10
View File
@@ -4,3 +4,13 @@ Cargo.lock
*.swo
.DS_Store
.cargo/
# session scratch — never track (may contain RE breadcrumbs)
scratch/
# stray local build artifact
/rust_out
# internal agent context — never publish (path AND dir; leak-guard blocks both)
CLAUDE.md
.claude/
+896 -2021
View File
File diff suppressed because it is too large Load Diff
+83
View File
@@ -0,0 +1,83 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at owner@example.invalid. All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of actions.
**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
+1 -1
View File
@@ -24,4 +24,4 @@ cargo test
## License
By contributing, you agree your code will be licensed under AGPL-3.0.
By contributing, you agree your code will be licensed under MIT.
+38 -4
View File
@@ -1,13 +1,22 @@
[package]
name = "libfreemkv"
version = "0.18.18"
version = "1.4.4"
edition = "2024"
rust-version = "1.86"
license = "AGPL-3.0-only"
license = "MIT"
description = "Open source raw disc access library for optical drives"
repository = "https://github.com/freemkv/libfreemkv"
keywords = ["bluray", "uhd", "optical", "scsi", "disc"]
categories = ["hardware-support", "multimedia"]
# Keep internal AI-instruction / private notes out of the published crate.
exclude = ["CLAUDE.md"]
# OFF crates.io: libfreemkv git-deps freemkv-unlock (firmware, never published),
# so libfreemkv itself can only be consumed by git tag. Clients git-tag-pin it.
publish = false
[profile.release]
lto = "thin"
codegen-units = 1
[dependencies]
serde = { version = "1", features = ["derive"] }
@@ -16,7 +25,10 @@ sha1 = "0.10"
sha2 = "0.10"
aes = "0.8"
cbc = "0.1"
flate2 = "1"
# Interim path dep for local cross-repo dev; the release script re-pins this to
# `{ git = ".../freemkv-unlock", tag = "vX.Y.Z" }` before tagging libfreemkv (so
# the released tag resolves freemkv-unlock from git, not a sibling path).
freemkv-unlock = { git = "https://github.com/freemkv/freemkv-unlock", tag = "v1.4.4" }
num-bigint = "0.4"
num-traits = "0.2"
num-integer = "0.1"
@@ -24,10 +36,32 @@ rand = "0.8"
cmac = "0.7"
zip = { version = "2", default-features = false, features = ["deflate"] }
base64 = "0.22.1"
# Read-only XML DOM parser (pure Rust, forbid(unsafe_code), entity-expansion
# bounded). Parses the HD-DVD Advanced-Content playlist `ADV_OBJ/VPLST000.XPL`
# — untrusted disc bytes — into authoritative titles/clips/chapters. A real
# parser, not a hand-rolled scanner: the XPL is genuine XML (comments, varied
# attribute order, self-closing tags).
roxmltree = "0.20"
# Trace-level instrumentation for Disc::copy + SgIoTransport::execute. Permitted
# under project docs ("Acceptable strings: debug/trace logging"). Consumers (autorip)
# under CLAUDE.md ("Acceptable strings: debug/trace logging"). Consumers (autorip)
# wire a tracing subscriber and pipe events into the JSONL debug log.
tracing = "0.1"
# Bounded MPSC channel with kernel-wakeup send_timeout. Used by `io::pipeline`
# so the halt-aware send/finish loops can BLOCK on consumer drain instead of
# polling — the 50 ms poll cadence of the previous mpsc-based impl capped mux
# throughput at ~1 MB/s (0.21.7).
crossbeam-channel = "0.5"
# Persistent work-stealing thread pool for parallel AACS unit
# decryption. Per-call std::thread::scope spawned fresh OS threads
# and that overhead dominated for typical batch sizes (60 units).
# rayon's global pool initialises once on first use.
rayon = "1"
# SIMD-accelerated bytestring search. Drives the HEVC/H.264 start-code
# scan in `mux::codec::h264::find_start_code` — naive byte-by-byte
# walk is ~500 MB/s single-thread on x86_64; memchr's vectorised
# `memmem::find` for the 3-byte `00 00 01` needle hits ~5 GB/s on
# AVX2-capable hosts.
memchr = "2"
[target.'cfg(target_os = "linux")'.dependencies]
libc = "0.2"
+17 -12
View File
@@ -1,16 +1,21 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
MIT License
Copyright (C) 2026 FreeMKV Contributors
Copyright (c) 2026 Matthew Jackson & Contributors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, version 3 of the License.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+1 -2
View File
@@ -1,6 +1,5 @@
# libfreemkv — local dev helper.
# Mirrors the cross-crate scripts in (internal)/scripts/test-all.sh
# but scoped to this single crate.
# Mirrors the workspace-wide CI checks but scoped to this single crate.
.PHONY: test build check ci clean
+27 -22
View File
@@ -1,24 +1,26 @@
[![Crates.io](https://img.shields.io/crates/v/libfreemkv)](https://crates.io/crates/libfreemkv)
[![docs.rs](https://img.shields.io/docsrs/libfreemkv)](https://docs.rs/libfreemkv)
[![License: AGPL-3.0](https://img.shields.io/badge/license-AGPL--3.0-blue)](LICENSE)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
# libfreemkv
Rust library for 4K UHD / Blu-ray / DVD optical drives. Drive access, disc scanning, stream labels, AACS decryption, CSS decryption, KEYDB updates, and content reading in one crate. Bundled drive profiles — no external files needed.
Rust library for 4K UHD / Blu-ray / DVD optical drives. Drive access, disc scanning, stream labels, AACS decryption, CSS decryption, KEYDB updates, and content reading in one crate. Drive-level unlocking is handled internally; consumers work with disc access and decryption only.
**12+ MB/s** sustained read speeds on BD. Full init: unlock, firmware upload, speed calibration — all from pure Rust.
DVDs (CSS) decrypt out of the box. Blu-ray and UHD (AACS) require a `keydb.cfg` (default `~/.config/freemkv/keydb.cfg`) supplying disc-specific volume unique keys; no AACS key material is compiled in.
**12+ MB/s** sustained read speeds on BD. Drive prep (`init()`) handles unlocking internally via the `freemkv-unlock` crate — clients never see it; when no drive unlock applies, the library rips via the host-certificate AACS handshake.
Multi-lingual by design — the library outputs structured data and numeric error codes, never English text. Build any UI or localization on top.
**[API Documentation](https://docs.rs/libfreemkv)** · **[Technical Docs](docs/)**
**[Source & API](https://github.com/freemkv/libfreemkv)** · **[Technical Docs](docs/)**
Part of the [freemkv](https://github.com/freemkv) project.
## Install
Consumed by git tag (not published to crates.io):
```toml
[dependencies]
libfreemkv = "0.18"
libfreemkv = { git = "https://github.com/freemkv/libfreemkv", tag = "vX.Y.Z" }
```
## Quick Start
@@ -27,10 +29,10 @@ libfreemkv = "0.18"
use libfreemkv::{Drive, Disc, ScanOptions};
use std::path::Path;
// Open drive — profiles are bundled, auto-identified
// Open drive — identified via INQUIRY
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?; // wait for disc
drive.init()?; // unlock + firmware upload
drive.init()?; // unlock + prep (handled internally)
drive.probe_disc()?; // probe disc surface for optimal speeds
// Scan disc — UDF, playlists, streams, AACS (all automatic)
@@ -56,8 +58,7 @@ output.finish()?;
For damaged discs the library exposes two flat verbs — `Disc::sweep` for the
forward Pass 1 and `Disc::patch` for retrying bad ranges. The library never
loops; the multipass policy is the caller's job. See
[`docs/rip-recovery.md`](docs/rip-recovery.md) and the design notes in
`(internal)/memory/0_18_redesign.md`.
[`docs/rip-recovery.md`](docs/rip-recovery.md).
```rust
use libfreemkv::{SweepOptions, PatchOptions};
@@ -99,7 +100,7 @@ loop {
## What It Does
- **Drive access** — open, identify, unlock, firmware upload, speed calibration, eject
- **Drive access** — open, identify, internal unlock + prep, speed control, eject
- **12+ MB/s reads** — auto-detects kernel transfer limits, sustained full speed
- **Disc scanning** — UDF 2.50 filesystem, MPLS playlists, CLPI clip info
- **Stream labels** — 5 BD-J format parsers (Paramount, Criterion, Pixelogic, CTRM, Deluxe)
@@ -120,17 +121,21 @@ loop {
| StdioStream | Yes (stdin) | Yes (stdout) | Raw byte pipe |
| NullStream | -- | Yes | Discard sink (byte counter for benchmarks) |
Streams implement `FrameSource` (read) and/or `FrameSink` (write); direction is type-checked. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
Streams implement a single unified `pes::Stream` trait (re-exported as `PesStream`) exposing `read()` and `write()` on one type. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
AACS decryption requires a KEYDB.cfg file. If available at `~/.config/aacs/KEYDB.cfg` or passed via `ScanOptions`, the library handles everything — handshake, key derivation, and per-sector decryption — without the application needing to know anything about encryption.
### Keys
DVDs (CSS) decrypt out of the box, with no external key file needed.
Blu-rays and UHD (AACS) require a `keydb.cfg` at `~/.config/freemkv/keydb.cfg` (or passed via `ScanOptions`). No AACS key material is compiled into the binary.
## Architecture
```text
Drive — open, identify, init, unlock, single-shot read
Drive — open, identify, init, single-shot read
├── ScsiTransport — SG_IO (Linux), IOKit (macOS), SPTI (Windows)
├── DriveProfile — per-drive unlock parameters (bundled)
└── PlatformDriver — MediaTek (supported), Renesas (planned)
└── unlock_bridge — private seam to the freemkv-unlock crate
(firmware / AACS cert / CSS bus-auth unlockers)
Disc — scan titles, streams, AACS/CSS state
├── UDF reader — Blu-ray UDF 2.50 with metadata partitions
@@ -139,12 +144,11 @@ Disc — scan titles, streams, AACS/CSS state
├── IFO parser — DVD title sets, PGC chains, cell addresses
├── Labels — 5 BD-J format parsers (detect + parse)
├── AACS — key resolution + content decryption
├── CSS — DVD CSS cipher (table-driven, no keys needed)
├── CSS — DVD CSS (bus auth → player-key disc crack → known-plaintext title-key attack)
└── KEYDB — download + verify + save
Streams — unified PES pipeline
├── FrameSource — read() PES frames (direction-typed)
├── FrameSink — write() PES frames (direction-typed)
├── PesStream — pes::Stream: one trait, read()/write() PES frames
├── DiscStream — sectors → decrypt → TS demux → PES
├── IsoStream — ISO file → decrypt → TS demux → PES
├── MkvStream — MKV mux/demux
@@ -170,6 +174,7 @@ All errors are structured with numeric codes. No user-facing English text — ap
| E6xxx | Disc format errors |
| E7xxx | AACS errors |
| E8xxx | KEYDB update errors |
| E9xxx | Stream / mux errors (URL, PES, ISO, pipeline, demux) |
## Platform Support
@@ -181,8 +186,8 @@ All errors are structured with numeric codes. No user-facing English text — ap
## Contributing
Run `freemkv info disc:// --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to contribute your drive's profile.
Run `freemkv info disc:// --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to capture your drive's identity for contribution. Drive-unlock profiles are maintained in the [freemkv-unlock](https://github.com/freemkv/freemkv-unlock) repository.
## License
AGPL-3.0-only
MIT
+61
View File
@@ -1,4 +1,6 @@
fn main() {
emit_git_suffix();
let target = std::env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
if target == "macos" {
println!("cargo:rustc-link-lib=framework=IOKit");
@@ -8,8 +10,22 @@ fn main() {
let obj = format!("{out_dir}/macos_shim.o");
let lib = format!("{out_dir}/libmacos_scsi.a");
// Build the shim for the TARGET arch, not the host's. A bare `cc` on an
// Apple-Silicon CI runner defaults to arm64, so cross-building to
// x86_64-apple-darwin would link a host-arch object against x86_64 Rust
// code → "Undefined symbols for architecture x86_64". (Still raw `cc`,
// not the `cc` crate, which breaks IOKit exclusive access.)
let target_arch = std::env::var("CARGO_CFG_TARGET_ARCH").unwrap_or_default();
let clang_arch: &str = if target_arch == "aarch64" {
"arm64"
} else {
&target_arch // x86_64 → x86_64
};
std::process::Command::new("cc")
.args([
"-arch",
clang_arch,
"-c",
"src/scsi/macos_shim.c",
"-o",
@@ -34,3 +50,48 @@ fn main() {
println!("cargo:rerun-if-changed=src/scsi/macos_shim.c");
}
}
/// Bake the git short hash into the build as `GIT_SUFFIX` so any muxed MKV or
/// FVI index is traceable to the exact source revision (e.g. ` (g835cc99)`).
/// Empty when git or the repo is unavailable (e.g. a crates.io tarball build),
/// leaving just the package version. Always emitted so `env!("GIT_SUFFIX")`
/// resolves on every target.
fn emit_git_suffix() {
// Version label for the muxing-app / FVI generator tag. `FREEMKV_BUILD_LABEL`
// overrides the Cargo package version when set (non-empty) — used to stamp a
// pre-release/test build without bumping Cargo.toml and disturbing the
// tag-pinned [patch] version matching. Unset → the package version.
let version = std::env::var("FREEMKV_BUILD_LABEL")
.ok()
.filter(|s| !s.trim().is_empty())
.or_else(|| std::env::var("CARGO_PKG_VERSION").ok())
.unwrap_or_default();
println!("cargo:rustc-env=FREEMKV_VERSION={version}");
println!("cargo:rerun-if-env-changed=FREEMKV_BUILD_LABEL");
let suffix = git_short_hash()
.map(|h| format!(" (g{h})"))
.unwrap_or_default();
println!("cargo:rustc-env=GIT_SUFFIX={suffix}");
// Re-run when HEAD (or the branch it points at) moves so the stamp stays
// current without a clean rebuild.
println!("cargo:rerun-if-changed=.git/HEAD");
if let Ok(head) = std::fs::read_to_string(".git/HEAD") {
if let Some(ref_path) = head.strip_prefix("ref: ") {
println!("cargo:rerun-if-changed=.git/{}", ref_path.trim());
}
}
}
fn git_short_hash() -> Option<String> {
let out = std::process::Command::new("git")
.args(["rev-parse", "--short=7", "HEAD"])
.output()
.ok()?;
if !out.status.success() {
return None;
}
let h = String::from_utf8(out.stdout).ok()?.trim().to_string();
if h.is_empty() { None } else { Some(h) }
}
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
#
# leak-guard.sh — self-contained public-repo leak gate.
#
# This is the LAST line of defense in CI. It is intentionally self-contained:
# public CI cannot reach the private tooling, so this script encodes ONLY the
# generic net — internal infrastructure references, agent-context files, and
# AI-attribution in commit messages. It deliberately contains NO project-
# specific reverse-engineering vocabulary (those words would themselves be a
# leak). The richer private scanner stays private.
#
# Fails (exit 1) if any of the following appear in the repo:
# 1. a tracked CLAUDE.md or .claude/ path (agent context — never public),
# 2. tracked file content matching the internal-infra net,
# 3. a commit message (in the given range) with AI attribution.
#
# Usage:
# leak-guard.sh [<commit-range>]
# <commit-range> optional git rev-list range to scan commit messages
# (e.g. "abc..def"). If omitted, commit-message scan is
# skipped (path + content checks always run).
set -euo pipefail
# Absolute path to this script, resolved before any cd, so we can exclude it
# from the content scan (it necessarily contains the detection patterns).
SELF_ABS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
REPO="$(git rev-parse --show-toplevel)"
cd "$REPO"
fail=0
note() { printf ' ✗ %s\n' "$1"; fail=1; }
# Internal-infra net — GENERIC ONLY. This script ships in the public repo, so
# the patterns themselves must not name any org-specific identifier (doing so
# would itself leak the infra they guard). We catch the leak *class*:
# - RFC1918 private IPv4 ranges (10/8, 172.16/12, 192.168/16),
# - private/internal/non-routable TLDs (.internal/.local/.lan/.corp/.invalid),
# - docker.internal.
# The full org-specific net (literal hostnames, service names, repo paths,
# vendor tooling, …) lives ONLY in the private scanner and never ships here.
INFRA_RE='\b10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|\b172\.(1[6-9]|2[0-9]|3[01])\.[0-9]{1,3}\.[0-9]{1,3}|\b192\.168\.[0-9]{1,3}\.[0-9]{1,3}|\.internal\b|\.local\b|\.lan\b|\.corp\b|\.invalid\b|docker\.internal'
# Home-path net — GENERIC ONLY. Catches an absolute developer home path
# committed into a tracked file (a macOS /Users/<user>/… or Linux /home/<user>/…
# path). This names NO specific user — it matches the leak *class* (any home
# path), so the pattern itself reveals nothing org- or person-specific. A real
# leak (e.g. /Users/alice/Developer/x slipping into a public RELEASE.md) trips
# this regardless of whose machine it came from. The username segment is a
# literal-username class ([A-Za-z0-9._-]) so dynamic/templated paths that build
# the user at runtime — shell `/home/$USER/`, doc `/home/<rip>/`, Rust
# `/home/{user}/` — do NOT false-positive; only a baked-in literal home leaks.
HOMEPATH_RE='/Users/[A-Za-z0-9._-]+/|/home/[A-Za-z0-9._-]+/'
# AI-attribution net (case-insensitive). "claude" matches only as a standalone
# word — NOT preceded by a dot/slash/alnum and NOT followed by .md — so legit
# mentions of CLAUDE.md / .claude/ in a commit message don't false-positive.
ATTR_RE='co-authored-by|generated with|🤖|(?<![.\/A-Za-z0-9])claude(?!\.md)'
echo "── leak-guard: tracked agent-context paths ──"
while IFS= read -r f; do
case "$f" in
CLAUDE.md|*/CLAUDE.md|.claude|.claude/*|*/.claude|*/.claude/*)
note "tracked agent-context file: $f (CLAUDE.md/.claude must never be tracked in a public repo)" ;;
esac
done < <(git ls-files)
# Match a PCRE against a file, emitting "LINE: MATCH". The pattern is passed as
# an argument (not interpolated into a //) so metacharacters like the "/" in a
# path-style token can't break the regex. Reads raw bytes so non-UTF-8 blobs
# don't abort the scan.
pcre_matches() {
perl -e '
my ($file, $re) = @ARGV;
open(my $fh, "<:raw", $file) or exit 0;
my $rx; eval { $rx = qr/$re/i }; exit 0 if $@;
while (my $l = <$fh>) { if ($l =~ /$rx/) { print "$.: $&\n"; } }
' "$1" "$2" 2>/dev/null
}
# This script's own source necessarily contains the detection patterns (e.g.
# the regex tokens in INFRA_RE), so scanning it would always self-flag. Skip it.
SELF="$(git ls-files --full-name -- "$SELF_ABS" 2>/dev/null | head -1)"
echo "── leak-guard: internal-infra references in tracked files ──"
while IFS= read -r f; do
case "$f" in *.png|*.jpg|*.jpeg|*.ico|*.gif|*.bin|*.crate|*.gz|*.zip|*.pdf) continue ;; esac
[ -n "$SELF" ] && [ "$f" = "$SELF" ] && continue
[ -f "$f" ] || continue
while IFS= read -r hit; do
[ -z "$hit" ] && continue
note "internal-infra reference: $f:$hit"
done < <(pcre_matches "$f" "$INFRA_RE")
while IFS= read -r hit; do
[ -z "$hit" ] && continue
note "[HOME-PATH] absolute home path: $f:$hit (no local home path may be committed to a public repo)"
done < <(pcre_matches "$f" "$HOMEPATH_RE")
done < <(git ls-files)
RANGE="${1:-}"
if [ -n "$RANGE" ]; then
echo "── leak-guard: AI-attribution in commit messages ($RANGE) ──"
while IFS= read -r sha; do
[ -z "$sha" ] && continue
msg="$(git log -1 --format='%B' "$sha" 2>/dev/null || true)"
# Pass the pattern as an argument (not interpolated into a //) so the
# lookbehind char class and "/" don't break the regex.
hit="$(printf '%s' "$msg" | perl -e '
my $re = $ARGV[0]; my $rx = qr/$re/i;
while (my $l = <STDIN>) { if ($l =~ /($rx)/) { print "$1\n"; last; } }
' "$ATTR_RE" | head -1 || true)"
[ -n "$hit" ] && note "commit ${sha:0:12}: message contains \"$hit\" (owner rule: zero AI attribution, ever)"
done < <(git rev-list "$RANGE" 2>/dev/null || true)
fi
echo
if [ "$fail" -ne 0 ]; then
echo "✗ leak-guard: blocking finding(s) above — DO NOT MERGE/PUBLISH"
exit 1
fi
echo "✓ leak-guard: clean"
+302
View File
@@ -0,0 +1,302 @@
# FVI — Freemkv Video Index Format
**Specification version:** 1.0 (DRAFT)\
**File extension:** `.fvi`\
**Media type:** `application/vnd.freemkv.fvi+jsonl`\
**Status:** Draft for review. This document is the normative reference for the FVI
format; implementations and downstream tools cite it by section.
---
## 1. Scope and purpose
FVI is an open, codec-agnostic, byte-exact **index of the coded pictures** in a
video bitstream, together with **provenance** back to the source medium.
An FVI document answers, for every picture in a stream, three questions:
1. **Where is it?** — the byte-exact offset of its first byte in the *source*
(the disc/ISO/file), so a reader can extract or seek to any picture without
re-parsing the whole bitstream.
2. **What is it?** — coding type, random-access capability, GOP boundary, and
(where the codec defines them) field/pulldown attributes.
3. **When is it?** — decode and presentation timestamps on a declared timescale.
FVI is **not** a container, a codec, or a copy of the bitstream. It indexes; it
never stores coded samples. It is the serialized form of an indexer's per-picture
truth — carried from the demuxer, **never reconstructed** (§9).
## 2. Conformance
The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**,
**SHOULD**, **SHOULD NOT**, **MAY**, and **OPTIONAL** are to be interpreted as
described in BCP 14 (RFC 2119, RFC 8174) when, and only when, they appear in all
capitals.
A **conformant writer** MUST emit a document that satisfies §4–§10. A
**conformant reader** MUST accept any such document and MUST ignore unknown
object members (§11) so that forward-compatible extensions do not break it.
## 3. Terminology
- **Picture** — one coded video frame (or pair of fields coded as a frame). The
unit FVI indexes.
- **Access unit (AU)** — the set of bitstream bytes that decode to exactly one
picture (ISO/IEC 14496-10 §3; ISO/IEC 23008-2 §3).
- **Coded order** — the order pictures appear in the bitstream. FVI records are
emitted in coded order.
- **GOP / coded video sequence** — a self-contained run beginning at a
random-access point.
- **Provenance** — the mapping from an AU back to the exact bytes of the physical
source it was read from (§9).
- **Source position (`src`)** — `{ file, sector, byte }`, the provenance anchor of
an AU.
## 4. Encoding
An FVI document is a sequence of **UTF-8** text lines separated by a single LF
(`U+000A`). Each non-empty line is exactly one JSON value (RFC 8259), forming a
**JSON Lines / NDJSON** stream. A writer MUST NOT emit a UTF-8 BOM. A writer MUST
NOT pretty-print: each JSON value occupies exactly one line.
The first line MUST be the **Header** object (§6). Each subsequent line is one
**Picture record** (§7), in coded order.
Rationale: line-delimited JSON is streamable (a writer appends as it indexes; a
reader processes without loading the whole file), line-addressable (picture *n*
is near line *n+1*), append-safe, and parseable by every language without a
custom grammar — while remaining a precisely specified format, not an ad-hoc dump.
A document MAY be concatenated for multiple elementary streams: each stream is its
own header line followed by its records. Readers MUST treat a Header line as the
start of a new stream section.
## 5. Document structure
```
<header> line 1 (exactly one Header object)
<record> line 2 .. N (one Picture record per picture, coded order)
[<header> <record>…] (OPTIONAL further stream sections)
```
## 6. Header object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `format` | string | MUST | Constant `"freemkv/video-index"`. Signature: a document begins with these bytes. |
| `fvi_version` | integer | MUST | Document format version. This spec defines `1`. |
| `generator` | string | SHOULD | Producing tool + version, e.g. `"freemkv/1.0.0-rc.6"`. |
| `stream` | object | MUST | The indexed elementary stream (§6.1). |
| `source` | object | MUST | Provenance root (§6.2). |
| `timescale` | integer | MUST | Ticks per second for all `pts`/`dts` (§10). E.g. `90000`. |
| `picture_count` | integer | MAY | Total pictures, if known at header time; OMITTED when streaming. |
### 6.1 `stream` object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `codec` | string | MUST | Registered codec id (Appendix B), e.g. `"mpeg2video"`, `"hevc"`. |
| `width`,`height` | integer | MUST | Coded luma dimensions in pixels. |
| `dar` | `[int,int]` | SHOULD | Display aspect ratio as `[num,den]`. |
| `frame_rate` | `[int,int]` | SHOULD | Nominal rate as exact rational `[num,den]` (e.g. `[24000,1001]`). |
| `scan` | string | MUST | `"progressive"`<br>`"interlaced"`<br>`"mbaff"` |
| `colour` | object | SHOULD | CICP per ITU-T H.273: `primaries`, `transfer`, `matrix` (integer CICP codes or registered names)<br>`range`: `"limited"` \| `"full"`<br>HDR: `mastering_display`, `max_cll`, `max_fall` per ITU-T H.273 / SMPTE ST 2086. |
| `language` | string | MAY | BCP 47 tag, if known. |
### 6.2 `source` object
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `medium` | string | MUST | `"disc"`<br>`"iso"`<br>`"file"`<br>`"stream"` |
| `path` | string | MAY | Source path/label. |
| `title` | integer | MAY | Title/program number. |
| `playlist` | string | MAY | Playlist/PGC identifier. |
| `volume_id` | string | MAY | Disc volume identifier, if read. |
| `sector_size` | integer | SHOULD | Bytes per `src.sector` unit (e.g. `2048`). Lets readers convert `src` to an absolute byte offset. |
## 7. Picture record
One JSON object per coded picture, in coded order.
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `n` | integer | MUST | Coded-order index, 0-based, contiguous. |
| `src` | object | MUST | Provenance: `{ "file": int?, "sector": uint, "byte": uint }` — the offset of this AU's **first byte** in the source (§9). MUST be carried from demux, never reconstructed. |
| `type` | string | MUST | Coding type:<br>`"I"`<br>`"P"`<br>`"B"`<br>_ISO/IEC 13818-2 §6.3.9; H.264/H.265 slice types collapsed to frame type._ |
| `key` | boolean | MUST | `true` iff this picture is an intra (I) picture / parser-flagged decode-restart point (IDR / IRAP / I-picture).<br>_MPEG-2 open-GOP clean-RAP precision (`closed_gop`) is not currently distinguished — see note below._ |
| `gop` | boolean | SHOULD | `true` iff this picture begins a GOP / coded video sequence.<br>_Omitted when the implementation does not carry a distinct GOP-boundary signal._ |
| `pts` | integer\|null | SHOULD | Presentation timestamp in `timescale` ticks; `null` if unknown. |
| `dts` | integer\|null | MAY | Decode timestamp in `timescale` ticks. |
| `size` | integer | MAY | AU length in bytes; enables byte-range extraction with `src`. |
| `recovered` | boolean | MAY | `true` iff any byte of this AU came from a retried/marginal read (§9.1).<br>_Default `false`._ |
| codec ext | object | MAY | Codec-specific members under the codec's namespace (§8). |
The `type` and `key` members are **codec-agnostic** and MUST be populated for
every codec. `type` is the I/P/B coding type the parser decoded (collapsing
H.264/H.265 slice types to a frame type); where no per-picture coding is carried
(audio / synthetic frames), `type` is `"I"` for a key picture else `"P"`. `key`
is the picture's random-access flag as the codec parser sets it (IDR / IRAP /
I-picture). A writer MUST NOT emit a degraded record (`type:"?"` or `src:null`)
merely because a codec lacks per-picture coding info — those fallbacks are
reserved for a field that is genuinely unavailable (e.g. provenance absent on a
synthetic source).
> **Limitation (honest random-access).** `key` is set from the picture's
> intra / decode-restart flag. The per-picture coding model this index carries
> does **not** distinguish MPEG-2 open-GOP clean random-access points
> (`closed_gop`) from any other I-picture, so `key` is the parser-flagged
> decode-restart point, not a verified clean-RAP claim. A future revision MAY
> tighten `key` for codecs/profiles that carry that signal; readers MUST NOT
> assume present `key` precision beyond "intra / decode-restart point".
### 7.1 Interlace / pulldown fields
Codec-agnostic interlace/pulldown attributes, derived through the indexer's
per-picture coding accessors (MPEG-2: ISO/IEC 13818-2 §6.3.10). Emitted as
top-level members of the record, and ONLY when the codec actually measured the
signal — an OPTIONAL member that is omitted (not defaulted) when unknown:
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `field_order` | string | MAY | Display field order:<br>`"tff"` — top field first<br>`"bff"` — bottom field first<br>`"progressive"` — no field order applies<br>_Omitted when the codec did not signal it._ |
| `progressive` | boolean | MAY | `true` iff the picture is progressive.<br>_Omitted when the codec did not signal it._ |
| `nb_fields` | integer | MAY | Number of displayed field periods this picture occupies (the soft-telecine / 2:3 pulldown basis):<br>`1` for a single field picture<br>`2` for a normal frame<br>`3`/`4`/`6` for `repeat_first_field` pulldown per §6.3.10 |
Codecs that carry only a coding type (e.g. H.264 / HEVC / VC-1 through this
pipeline) omit `field_order` and `progressive` rather than guessing a default.
## 8. Codec model and extensibility
Core record members (§7) are codec-agnostic and present for every codec.
Codec-specific data is either (a) promoted to top-level members for a small,
registered set per codec profile (e.g. MPEG-2 §7.1), or (b) placed under an
`ext` object keyed by codec id for richer/optional data:
```json
{
"n": 42,
"type": "P",
"key": false,
"src": {
"sector": 17,
"byte": 924
},
"ext": {
"hevc": {
"temporal_id": 0,
"nal_type": 1
}
}
}
```
New codecs and members are added through Appendix B (codec registry) without a
breaking version bump, provided readers continue to ignore unknown members (§11).
## 9. Provenance and recovery semantics
`src` is **byte-exact** to the source as read. `src.sector` counts in
`source.sector_size`-byte units; `src.byte` is the offset within that sector of
the AU's first byte. For multi-file sources, `src.file` indexes a writer-declared
file list. Provenance MUST be the value observed at demux time; an implementation
MUST NOT recompute `src` by re-parsing — the point of FVI is to *carry* the truth.
### 9.1 Recovery
Because FVI is provenance-native, it can record reliability. A record with
`"recovered":true` indicates the AU's source bytes required retry/marginal-read
recovery. This lets downstream tools surface or quarantine pictures whose bytes
are not byte-identical to a clean read — a capability legacy index formats lack.
## 10. Time model
All `pts`/`dts` are integers in units of `1/timescale` seconds. `pts` is
presentation (display) time; `dts` is decode time. Records are in **coded**
(decode) order, so `pts` is not necessarily monotonic across records (B-pictures
reorder); `dts` is non-decreasing. Readers needing display order sort by `pts`.
## 11. Versioning and forward compatibility
- `fvi_version` is the document version; this spec defines `1`.
- **Additive** changes (new OPTIONAL members, new registered codecs) do NOT bump
`fvi_version`. Readers MUST ignore members they do not recognize.
- A change that alters the meaning of an existing member or makes a new member
REQUIRED bumps `fvi_version`.
- A reader encountering a higher `fvi_version` than it implements SHOULD process
the members it understands and MUST NOT reject the document solely for the
version being higher, unless a member it relies on is absent.
## 12. Conformance requirements (summary)
A conformant **writer** MUST: emit a Header first; emit records in coded order
with contiguous `n`; populate `src` from demux; use named/registered codec ids;
encode one JSON value per UTF-8 LF-terminated line.
A conformant **reader** MUST: accept any §4–§10 document; ignore unknown members;
not assume `picture_count`, `pts`, or `size` are present unless required above.
---
## Appendix A — JSON Schema (informative)
Header:
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["format", "fvi_version", "stream", "source", "timescale"],
"properties": {
"format": { "const": "freemkv/video-index" },
"fvi_version": { "type": "integer", "minimum": 1 },
"timescale": { "type": "integer", "minimum": 1 },
"stream": { "type": "object", "required": ["codec", "width", "height", "scan"] },
"source": { "type": "object", "required": ["medium"] }
}
}
```
Record:
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["n", "src", "type", "key"],
"properties": {
"n": { "type": "integer", "minimum": 0 },
"type": { "enum": ["I", "P", "B"] },
"key": { "type": "boolean" },
"src": {
"type": "object",
"required": ["sector", "byte"],
"properties": {
"file": { "type": "integer" },
"sector": { "type": "integer", "minimum": 0 },
"byte": { "type": "integer", "minimum": 0 }
}
}
}
}
```
## Appendix B — Registered codec identifiers
| `codec` | Bitstream | Field profile |
|---|---|---|
| `mpeg2video` | ISO/IEC 13818-2 | §7.1 (field_order/progressive/nb_fields) |
| `mpeg1video` | ISO/IEC 11172-2 | §7.1 |
| `h264` | ISO/IEC 14496-10 | core + `ext.h264` |
| `hevc` | ISO/IEC 23008-2 | core + `ext.hevc` |
| `vc1` | SMPTE 421M | core |
## Appendix C — Normative references
- RFC 2119, RFC 8174 — Requirement keywords (BCP 14).
- RFC 8259 — JSON.
- ISO/IEC 13818-2 — MPEG-2 video (picture coding, §6.3.9–6.3.10).
- ISO/IEC 14496-10 — H.264/AVC. ISO/IEC 23008-2 — H.265/HEVC.
- ITU-T H.273 — Coding-independent code points (colour primaries/transfer/matrix).
- SMPTE ST 2086 — Mastering display colour volume (HDR).
- BCP 47 — Language tags.
- RFC 9559 — Matroska (alignment of colour/field-order semantics).
+50 -264
View File
@@ -2,191 +2,45 @@
## Overview
AACS (Advanced Access Content System) is the encryption layer used by Blu-ray and UHD 4K discs to protect content. libfreemkv implements AACS decryption to enable transparent disc access.
AACS (Advanced Access Content System) is the encryption layer used by Blu-ray
and UHD 4K discs to protect content. libfreemkv implements AACS decryption so
disc access is transparent to the application.
There are two major versions:
- **AACS 1.0** -- Used by standard Blu-ray discs. Relies on a custom 160-bit elliptic curve for bus authentication and AES-128 for content encryption. Processing keys and device keys can derive the media key from the disc's Media Key Block (MKB).
- **AACS 2.0** -- Used by UHD 4K Blu-ray discs. Adds a per-sector bus encryption layer (read_data_key) on top of the standard content encryption. Uses P-256/SHA-256 for its native handshake, though drives accept AACS 1.0 host certificates for backward compatibility.
Both versions use AES-128-CBC for content decryption with a fixed initialization vector. The fundamental key hierarchy is the same: a Volume Unique Key (VUK) decrypts per-title unit keys, which in turn decrypt the content stream.
## Architecture
AACS support is split across two modules:
### `aacs.rs` -- Keys and Decryption
Handles everything related to key resolution and content decryption:
- KEYDB.cfg parsing (device keys, processing keys, host certificates, per-disc entries)
- Disc hash computation (SHA-1 of `Unit_Key_RO.inf`)
- VUK resolution chain (4 paths, described below)
- MKB record parsing and media key derivation
- Subset-difference tree traversal (AACS-G3 key derivation)
- Unit_Key_RO.inf parsing and unit key decryption
- Content Certificate parsing (AACS version detection)
- Aligned unit decryption (AES-128-CBC)
- Bus decryption (AACS 2.0 read_data_key layer)
### `aacs_handshake.rs` -- SCSI Authentication
Handles the drive-level SCSI authentication protocol:
- ECDH key agreement on the AACS 160-bit curve
- ECDSA signing and verification
- Bus key derivation
- AGID management (allocate/invalidate)
- Volume ID retrieval (encrypted with bus key, verified by AES-CMAC)
- Read Data Key retrieval (for AACS 2.0 bus decryption)
- AACS LA public key certificate verification
## Key Resolution Chain
When a disc is scanned, `resolve_keys()` attempts four paths in priority order. The first path that succeeds is used.
### Path 1: KEYDB VUK Lookup (fastest)
```
Unit_Key_RO.inf --> SHA-1 --> disc_hash --> KEYDB lookup --> VUK
```
The disc hash is computed as the SHA-1 digest of the raw `Unit_Key_RO.inf` file from the disc's `/AACS/` directory. This hash is used as the lookup key in `KEYDB.cfg`. If a matching entry contains a VUK (`V` field), it is used directly.
This is the fast path and resolves the vast majority of discs in a well-maintained KEYDB.
### Path 2: KEYDB Media Key + Volume ID
```
KEYDB media_key + Volume ID (from SCSI handshake) --> VUK derivation
```
If the disc hash is not in the KEYDB but a KEYDB entry has a matching Volume ID (`I` field) and a media key (`M` field), the VUK is derived:
```
VUK = AES-128-ECB-DECRYPT(media_key, volume_id) XOR volume_id
```
Requires a successful SCSI handshake to obtain the Volume ID.
### Path 3: MKB + Processing Keys
```
MKB (from disc) + processing_keys (from KEYDB) --> media_key --> VUK
```
Processing keys are pre-computed keys that work against specific MKB versions. For each processing key, the library:
1. Parses the MKB to extract the Verify Media Key Record (`mk_dv`), subset-difference index, and conditional values (cvalues).
2. Tries each processing key against each UV/cvalue pair: `mk = AES-DEC(pk, cvalue) XOR cvalue`.
3. Validates the derived media key: `AES-ECB(mk, mk_dv)` must produce 12 leading zero bytes.
4. Derives VUK from the validated media key and Volume ID.
### Path 4: MKB + Device Keys (Subset-Difference Tree)
```
MKB + device_keys --> subset-difference tree traversal --> processing_key --> media_key --> VUK
```
The most complex path. Each device key has an associated node number, UV value, and mask parameters that position it in the AACS subset-difference tree. The library:
1. Finds the subset-difference entry in the MKB that applies to the device key's node.
2. Traverses the tree using AACS-G3 key derivation: `aesg3(key, inc) = AES-DEC(key, seed) XOR seed`, where `seed[15]` is incremented by `inc`. Each tree node produces a left child (inc=0), a processing key (inc=1), and a right child (inc=2).
3. At each level, selects left or right based on the UV bit at the current position.
4. The resulting processing key is validated against the MKB cvalue to derive the media key.
5. VUK is derived from the media key and Volume ID.
## Content Decryption
### Aligned Units
AACS encrypts content in aligned units of 6144 bytes (3 sectors of 2048 bytes each). The encryption flag is signaled by the copy_permission_indicator bits in byte 0 of the unit (`unit[0] & 0xC0 != 0`).
### Per-Unit Key Derivation
Each aligned unit has its own decryption key derived from the CPS unit key:
1. **Derive**: AES-128-ECB encrypt the first 16 bytes of the unit (plaintext TP_extra_header) with the unit key.
2. **XOR**: XOR the encrypted result with the original 16 bytes to produce the per-unit decryption key.
3. **Decrypt**: AES-128-CBC decrypt bytes 16 through 6143 using the per-unit key and the fixed AACS IV.
4. **Clear flag**: Clear the encryption indicator bits (`unit[0] &= !0xC0`).
### Fixed IV
All AES-CBC operations in AACS use the same fixed initialization vector, defined in the AACS specification.
### Verification
After decryption, the library verifies correctness by checking for MPEG-TS sync bytes (0x47) at the expected 192-byte packet boundaries within the unit. Blu-ray transport stream packets are 192 bytes: 4-byte TP_extra_header followed by a 188-byte TS packet.
## Bus Encryption
### AACS 1.0
Standard Blu-ray discs do not use bus encryption. Content is read directly from the disc and decrypted using the unit key.
### AACS 2.0
UHD 4K discs add a per-sector bus encryption layer. The drive encrypts data as it is read from the disc, and the host must decrypt it before applying AACS content decryption.
Bus encryption uses a **read_data_key** obtained during the SCSI handshake. For each 2048-byte sector within an aligned unit, bytes 16 through 2047 are AES-128-CBC encrypted with the read_data_key and the fixed AACS IV. The first 16 bytes of each sector remain plaintext.
The full decryption pipeline for AACS 2.0:
1. **Bus decrypt**: For each sector, AES-128-CBC decrypt bytes 16..2047 with the read_data_key.
2. **Content decrypt**: Standard per-unit key derivation and AES-128-CBC decryption as described above.
## SCSI Handshake
The AACS SCSI authentication handshake establishes a shared bus key between host and drive, then uses it to securely transfer the Volume ID and read data keys.
### Protocol Flow
1. **Invalidate AGIDs**: Send REPORT KEY with format 0x3F for AGIDs 0-3 to clear stale sessions.
2. **Allocate AGID**: REPORT KEY format 0x00 returns a fresh Authentication Grant ID.
3. **Send host credentials**: SEND KEY format 0x01 transmits the host nonce (20 random bytes) and host certificate (92 bytes).
4. **Receive drive credentials**: REPORT KEY format 0x01 returns the drive nonce and drive certificate.
5. **Receive drive key**: REPORT KEY format 0x02 returns the drive's ephemeral EC key point and ECDSA signature over `host_nonce || drive_key_point`.
6. **Verify drive key**: The signature is verified against the drive's public key (extracted from its certificate). AACS 1.0 certificates are verified against the AACS LA public key.
7. **Send host key**: The host generates an ephemeral key pair, signs `drive_nonce || host_key_point` with the host private key, and sends via SEND KEY format 0x02.
8. **Compute bus key**: ECDH shared secret = `host_private_key * drive_key_point`. The bus key is the low 128 bits of the shared point's x-coordinate.
### Post-Authentication Reads
- **Volume ID**: REPORT DISC STRUCTURE format 0x80. Returns 16-byte VID encrypted with the bus key, plus an AES-CMAC MAC for integrity verification.
- **Read Data Keys**: REPORT DISC STRUCTURE format 0x84. Returns the read_data_key and write_data_key, each AES-ECB encrypted with the bus key.
### Elliptic Curve
AACS 1.0 uses a custom 160-bit Weierstrass curve (`y^2 = x^3 + ax + b mod p`) with 20-byte field elements. The library implements full EC arithmetic: point addition, doubling, scalar multiplication, modular inverse, ECDSA sign/verify, and ECDH key agreement.
## AACS 2.0 Status
AACS 2.0 discs are detected via the Content Certificate file (`Content000.cer` or `Content001.cer`). A certificate type byte of 0x01 indicates AACS 2.0.
AACS 2.0 drives are identified by their drive certificate type (0x11). These drives natively use P-256/SHA-256, but accept AACS 1.0 host certificates for backward compatibility.
Current implementation status:
- AACS 2.0 detection: **implemented** (Content Certificate parsing, drive cert type check)
- AACS 1.0 handshake with AACS 2.0 drives: **implemented** (backward compatibility mode)
- Full P-256 AACS 2.0 handshake: **not yet implemented** (prepared but rarely needed since drives accept AACS 1.0 host certs)
- Bus decryption with read_data_key: **implemented**
- Content decryption: **implemented** (same as AACS 1.0)
In practice, AACS 2.0 UHD discs work through the backward-compatible AACS 1.0 handshake path, with the addition of read_data_key bus decryption.
- **AACS 1.0** -- Used by standard Blu-ray discs.
- **AACS 2.0 / 2.1** -- Used by UHD 4K Blu-ray discs. Adds a per-sector bus
encryption layer on top of the standard content encryption. UHD drives accept
AACS 1.0 host credentials for backward compatibility.
All versions use AES-128 for content decryption. The library reads the keys it
needs from `keydb.cfg`, walks the disc's Media Key Block (MKB) to resolve the
disc's key, and decrypts the content stream. AACS-encrypted discs therefore
require a `keydb.cfg`; CSS-protected DVDs do not (see the CSS notes in the
library docs).
## How it works (feature level)
When a disc is scanned, the library:
1. Reads the disc's AACS key-input files from the `/AACS/` directory.
2. Resolves the disc's key from `keydb.cfg` — either directly from a per-disc
entry, or by walking the MKB with the keys present in the keydb.
3. Performs the drive-level SCSI authentication handshake needed to obtain the
Volume ID and, for UHD, the bus-decryption key.
4. Decrypts the content stream as titles are read.
A resolved key is verified against actual disc content before it is applied, so
a stale or wrong key fails loudly rather than producing silent garbage. If no
usable key is available for an AACS-encrypted disc, the library surfaces a
specific error (the E70xx family) describing which part of the chain was
missing, and a missing `keydb.cfg` surfaces as `Error::KeydbLoad` with the
sentinel path `<no keydb in search paths>`.
## API Usage
AACS decryption is transparent to the application. The `Disc::scan()` method handles everything automatically:
AACS decryption is transparent to the application. `Disc::scan()` handles
everything automatically:
```rust
use libfreemkv::{Drive, Disc};
@@ -203,7 +57,6 @@ if disc.encrypted {
if let Some(ref aacs) = disc.aacs {
println!("AACS {}.0", aacs.version);
println!("Key source: {}", aacs.key_source.name());
println!("Disc hash: {}", aacs.disc_hash);
if let Some(mkb_ver) = aacs.mkb_version {
println!("MKB version: {}", mkb_ver);
}
@@ -215,108 +68,41 @@ if disc.encrypted {
// Read content -- decryption is automatic
let mut reader = disc.open_title(&mut session, 0).unwrap();
while let Some(unit) = reader.read_unit().unwrap() {
// unit is 6144 bytes of decrypted content
// decrypted content
}
```
The application never touches keys, never calls decryption functions, and never manages handshakes. All of that is internal to `Disc::scan()` and `ContentReader::read_unit()`.
The application never touches keys, never calls decryption functions, and never
manages handshakes. All of that is internal to `Disc::scan()` and the content
reader.
### KEYDB Location
`ScanOptions` controls where the KEYDB is loaded from. If no explicit path is set, the library checks:
1. `~/.config/aacs/KEYDB.cfg`
2. `/etc/aacs/KEYDB.cfg`
To specify an explicit path:
`ScanOptions` controls where the keydb is loaded from. If no explicit path is
set, the library checks the standard config locations. To specify an explicit
path:
```rust
let opts = ScanOptions::with_keydb("/path/to/KEYDB.cfg");
let opts = ScanOptions::with_keydb("/path/to/keydb.cfg");
let disc = Disc::scan(&mut session, &opts).unwrap();
```
### AacsState
After a successful scan, `disc.aacs` contains an `AacsState` with:
After a successful scan, `disc.aacs` contains an `AacsState`:
| Field | Type | Description |
|-------|------|-------------|
| `version` | `u8` | AACS version (1 or 2) |
| `bus_encryption` | `bool` | Whether bus encryption is active |
| `mkb_version` | `Option<u32>` | MKB version from disc |
| `disc_hash` | `String` | SHA-1 of Unit_Key_RO.inf (hex with 0x prefix) |
| `key_source` | `KeySource` | How keys were resolved |
| `vuk` | `[u8; 16]` | Volume Unique Key |
| `unit_keys` | `Vec<(u32, [u8; 16])>` | Decrypted unit keys (CPS unit number, key) |
| `read_data_key` | `Option<[u8; 16]>` | AACS 2.0 bus decryption key |
| `volume_id` | `[u8; 16]` | Volume ID from SCSI handshake |
| `disc_hash` | `String` | Identifier for the disc's key-input files |
| `key_source` | `KeySource` | How the disc's key was resolved |
### KeySource
## keydb.cfg
| Variant | Description |
|---------|-------------|
| `KeyDb` | VUK found directly in KEYDB by disc hash |
| `KeyDbDerived` | Media key + Volume ID from KEYDB, VUK derived |
| `ProcessingKey` | MKB + processing keys from KEYDB |
| `DeviceKey` | MKB + device keys, subset-difference tree traversal |
## KEYDB.cfg Format Reference
The KEYDB.cfg file contains all cryptographic material needed for AACS decryption. Lines starting with `;` or `#` are comments.
### Device Keys
```
| DK | DEVICE_KEY 0x<key> | DEVICE_NODE 0x<node> | KEY_UV 0x<uv> | KEY_U_MASK_SHIFT 0x<shift>
```
- `key`: 16-byte AES device key (hex)
- `node`: Device node number in the subset-difference tree (hex)
- `uv`: UV value for tree positioning (hex)
- `shift`: U mask shift value (hex)
### Processing Keys
```
| PK | 0x<key>
```
- `key`: 16-byte pre-computed processing key (hex)
### Host Certificate
```
| HC | HOST_PRIV_KEY 0x<privkey> | HOST_CERT 0x<cert>
```
- `privkey`: 20-byte ECDSA private key (hex)
- `cert`: 92-byte AACS host certificate (hex)
The host certificate is used for SCSI authentication. It contains the host's public key and is signed by the AACS Licensing Administrator.
### Disc Entries
```
0x<disc_hash> = <title> | D | <date> | M | 0x<media_key> | I | 0x<disc_id> | V | 0x<vuk> | U | <unit_keys>
```
- `disc_hash`: 20-byte SHA-1 of Unit_Key_RO.inf (hex)
- `title`: Human-readable disc title
- `D`: Date tag, followed by release/rip date
- `M`: Media key tag, followed by 16-byte media key (hex)
- `I`: Disc ID tag, followed by 16-byte Volume ID (hex)
- `V`: VUK tag, followed by 16-byte Volume Unique Key (hex)
- `U`: Unit keys tag, followed by space-separated `<unit_num>-0x<key>` pairs
All fields after the title are optional. A minimal entry needs only the disc hash and VUK:
```
0x<disc_hash> = <title> | V | 0x<vuk>
```
Inline comments are supported with `;`:
```
0x<disc_hash> = <title> | V | 0x<vuk> ; MKBv77
```
`keydb.cfg` is the single source of AACS key material. It is a text file (lines
starting with `;` or `#` are comments) holding the host credentials and per-disc
entries the library uses to resolve a disc. autorip can auto-download and
refresh it from a configured URL. The library does not ship any AACS keys
compiled into the binary.
+9 -10
View File
@@ -85,10 +85,10 @@ All URLs require a `scheme://path` format. Bare paths are rejected.
// PES pipeline (frame-level) — input() returns Box<dyn FrameSource>,
// output() returns Box<dyn FrameSink>.
let input = libfreemkv::input("disc:///dev/sg4", &opts)?; // DiscStream
let input = libfreemkv::input("iso://Dune.iso", &opts)?; // IsoStream
let output = libfreemkv::output("mkv://Dune.mkv", &title)?; // MkvOutputStream
let output = libfreemkv::output("m2ts://Dune.m2ts", &title)?; // M2tsOutputStream
let output = libfreemkv::output("network://10.0.0.1:9000", &title)?; // NetworkOutputStream
let input = libfreemkv::input("iso://Movie.iso", &opts)?; // IsoStream
let output = libfreemkv::output("mkv://Movie.mkv", &title)?; // MkvOutputStream
let output = libfreemkv::output("m2ts://Movie.m2ts", &title)?; // M2tsOutputStream
let output = libfreemkv::output("network://192.0.2.10:9000", &title)?; // NetworkOutputStream
let output = libfreemkv::output("null://", &title)?; // NullOutputStream
```
@@ -171,23 +171,23 @@ libfreemkv/src/
│ └── writeback.rs sync_file_range pipeline
├── drive/ Drive (open, init, single-shot read)
│ ├── mod.rs Drive struct, init, read (single-shot), reset, eject
│ ├── capture.rs Drive profile capture for contribution
│ ├── capture.rs Raw drive SCSI capture (INQUIRY/GET_CONFIG) for contribution
│ ├── linux.rs Linux drive discovery
│ ├── macos.rs macOS drive discovery
│ └── windows.rs Windows drive discovery
├── disc/ Disc (scan, titles, AACS setup, sweep, patch)
│ ├── mod.rs Disc struct, scan, titles, formats
│ ├── sweep.rs Disc::sweep (Pass 1 forward sweep)
│ ├── mod.rs Disc struct, scan, titles, formats; Disc::copy + Disc::sweep (Pass 1)
│ ├── sweep.rs Pass 1 internal helpers (pub(super))
│ ├── patch.rs Disc::patch (Pass N retry over mapfile)
│ ├── mapfile.rs ddrescue-format mapfile
│ └── read_error.rs ReadCtx / ReadAction state machine
├── scsi/ SCSI transport (Linux SG_IO, macOS IOKit, Windows SPTI)
├── platform/ Drive unlock (MT1959 A/B)
├── unlock.rs Unlocker trait + registry (pluggable unlock seam)
├── aacs/ AACS decryption (handshake, keys, keydb, decrypt)
├── css/ DVD CSS cipher
├── decrypt.rs Unified decrypt dispatcher (AACS/CSS/None)
├── pes.rs PES frame types, FrameSource / FrameSink traits
├── sector/ Sector I/O (was sector.rs in 0.17)
├── sector/ Sector I/O
│ ├── mod.rs SectorSource, SectorSink traits
│ ├── file.rs FileSectorSource, FileSectorSink (ISO-backed)
│ └── decrypting.rs DecryptingSectorSource decorator
@@ -198,7 +198,6 @@ libfreemkv/src/
├── labels/ BD-J label extraction (5 format parsers)
├── keydb.rs KEYDB download, parse, save
├── identity.rs DriveId from INQUIRY
├── profile.rs Bundled drive profiles
├── speed.rs DriveSpeed enum
├── mux/
│ ├── mod.rs Public mux exports
+28 -26
View File
@@ -1,11 +1,13 @@
# libfreemkv Architecture
Open source optical drive access library for 4K UHD Blu-ray, Blu-ray, and DVD.
Rust library with no external dependencies at runtime -- profiles are bundled,
AACS keys are derived internally, and all SCSI communication is handled in-process.
Rust library with profiles bundled and all SCSI communication handled in-process.
AACS decryption requires an external `keydb.cfg` (default
`~/.config/freemkv/keydb.cfg`) — the derivation math is internal, but no AACS key
material is compiled in; DVD CSS player keys are the only compiled-in keys.
**Repository:** <https://github.com/freemkv/libfreemkv>
**License:** AGPL-3.0-only
**License:** MIT
---
@@ -15,9 +17,10 @@ AACS keys are derived internally, and all SCSI communication is handled in-proce
format handling live in the library. CLI binaries are thin wrappers that call
`Drive::open()` and `Disc::scan()`.
2. **No external files.** Bundled drive profiles are compiled into the binary via
`include_str!`. No configuration directory, no runtime file lookups for drive
support.
2. **Firmware-clean core.** libfreemkv ships no firmware, no unlock CDBs, and no
drive profiles. Drive-unlock logic is plugged in by an external crate through
the `Unlocker` trait + registry (`register_unlocker`); without one the library
still rips via the host-certificate AACS handshake.
3. **Transparent AACS.** The `ContentReader` decrypts on the fly when keys are
available. Callers read cleartext sectors without knowing whether the disc
@@ -41,11 +44,9 @@ AACS keys are derived internally, and all SCSI communication is handled in-proce
libfreemkv (lib.rs)
│
├── Drive Access
│ ├── drive Drive — open, identify, init, unlock, single-shot read
│ ├── drive Drive — open, identify, init, single-shot read
│ ├── scsi ScsiTransport trait + platform backends (sg async, IOKit, SPTI)
│ ├── platform/ Platform trait — per-chipset command handlers
│ │ └── mt1959 MediaTek MT1959 driver (LG, ASUS, HP)
│ ├── profile DriveProfile loading, matching, bundled JSON
│ ├── unlock Unlocker trait + registry — the pluggable unlock seam
│ ├── identity DriveId from INQUIRY + GET_CONFIG 010C
│ ├── speed DriveSpeed enum, SET CD SPEED CDB builder
│ └── event Event system for drive status callbacks
@@ -65,7 +66,7 @@ libfreemkv (lib.rs)
│
├── Streaming
│ ├── mux/ Stream implementations (Disc, ISO, MKV, M2TS, Network, Stdio, Null)
│ ├── pes PES frame types; FrameSource / FrameSink direction-typed traits
│ ├── pes PES frame types; the unified pes::Stream (PesStream) read/write trait
│ └── sector/ SectorSource / SectorSink traits, FileSector{Source,Sink}, DecryptingSectorSource
│
├── I/O Primitives
@@ -74,8 +75,7 @@ libfreemkv (lib.rs)
│
├── Support
│ ├── keydb KEYDB.cfg download, parse, verify, save
│ ├── error Error enum with numeric codes E1000-E8000
│ └── profile Bundled drive profiles
│ └── error Error enum with numeric codes E1000-E8000
│
└── lib.rs Public API re-exports
```
@@ -89,13 +89,12 @@ Drive::open(Path::new("/dev/sg4"))
│
├─ scsi::open() Open /dev/sg4 (async write/poll/read)
├─ DriveId::from_drive() INQUIRY + GET_CONFIG 010C
├─ profile::find_by_drive_id() Match against bundled profiles
├─ Platform::new() Instantiate chipset driver (Mt1959)
└─ Drive ready for init/unlock/read
└─ Drive ready for init/read
```
After open:
- `init()` -- unlock + firmware upload + speed calibration
- `init()` -- routes to the matching registered unlocker (if any); otherwise
a no-op and the cert handshake carries the disc
- `probe_disc()` -- probe disc surface for optimal speeds
- `read(lba, count, buf, recovery)` -- single-shot read; `recovery` only selects the per-CDB timeout (1.5 s vs. 30 s)
- `wait_ready()` -- wait for disc insertion
@@ -195,17 +194,20 @@ implementing `execute()` for that OS and wiring it into `scsi::open()`.
---
## Chipset Support
## Drive Unlock
| Chipset | Drives | Status |
|---------|--------|--------|
| MediaTek MT1959 | LG, ASUS, HP | Supported (bundled profiles) |
| Renesas RS8xxx/RS9xxx | Pioneer, some HL-DT-ST | Planned |
libfreemkv carries no drive-unlock mechanism. The `Unlocker` trait + registry
(`src/unlock.rs`) is the seam: an external crate implements `Unlocker` and
registers it once via `register_unlocker(...)`. At drive-prep the registry is
walked in order and the first unlocker whose `matches()` is true is asked to
`unlock_drive()` over the raw `ScsiTransport`. If none match, the drive is left
untouched and the host-certificate AACS handshake carries the disc.
The `Platform` trait abstracts chipset-specific commands. Each chipset implements
handlers (unlock, config, register, calibrate, keepalive, status, probe,
read_sectors, timing). All handlers are accessed via SCSI READ BUFFER with
chipset-specific mode and buffer ID bytes.
The implementor owns everything firmware-specific — drive profiles, vendor CDBs,
variant logic. Concrete unlockers live in the separate
**[freemkv-unlock](https://github.com/freemkv/freemkv-unlock)** repository, never
in libfreemkv. See [`drive-access.md`](drive-access.md#drive-unlock-seam) for the
trait definition and routing.
---
+1 -1
View File
@@ -189,7 +189,7 @@ The full ripping pipeline chains three parsers:
2. **CLPI** converts those timestamps to SPN ranges, then to sector extents.
3. **UDF** provides the file's starting LBA on disc for absolute sector addressing.
The `Disc::scan()` method in `src/disc.rs` orchestrates this: for each play item in each playlist, it loads the corresponding CLPI, calls `get_extents()` with the play item's in/out times, and collects the resulting sector ranges into the title's extent list.
The `Disc::scan()` method in `src/disc/mod.rs` orchestrates this: for each play item in each playlist, it loads the corresponding CLPI, calls `get_extents()` with the play item's in/out times, and collects the resulting sector ranges into the title's extent list.
## References
+7 -7
View File
@@ -10,14 +10,14 @@ Insert disc
│
▼
1. Open drive (drive/mod.rs)
│ INQUIRY → identify drive
│ Match bundled profile → chipset, unlock parameters
│ INQUIRY → identify drive (DriveId)
│
▼
2. Init drive (drive/mod.rs → platform/mt1959)
│ Firmware upload (if needed, 10s recovery wait)
│ Unlock → vendor-specific command activates raw read mode
│ Speed calibration → probe_disc()
2. Init drive (drive/mod.rs → unlock seam)
│ Walk the registered-unlocker registry; first match unlocks the drive
│ (firmware/vendor handshakes are the unlocker's own business)
│ No match → drive untouched; host-cert AACS handshake carries the disc
│ Speed control → probe_disc()
│
▼
3. AACS handshake (aacs/handshake.rs) — optional
@@ -98,7 +98,7 @@ drive.probe_disc()?;
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
// Stream pipeline — PES frames from any source to any output.
// 0.18: input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
// input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
// direction is type-checked, so calling .write() on an input is a compile error.
let opts = InputOptions::default();
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
+63 -89
View File
@@ -7,8 +7,9 @@ optical drives.
## Drive
`Drive` is the primary API. It owns the SCSI transport, the matched
drive profile, and the chipset-specific platform driver.
`Drive` is the primary API. It owns the SCSI transport and the drive
identity (`DriveId`); any drive-specific unlock logic lives behind the
pluggable [unlock seam](#drive-unlock-seam), not in `Drive` itself.
### Opening a Drive
@@ -16,15 +17,16 @@ drive profile, and the chipset-specific platform driver.
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
```
`open()` performs: open device → send INQUIRY → match profile → instantiate
platform driver. The drive is ready for `wait_ready()` and `init()`.
`open()` performs: open device → send INQUIRY → build `DriveId`. The drive
is ready for `wait_ready()` and `init()` (which routes through the unlock
seam).
### Drive Operations
| Method | Description |
|--------|-------------|
| `wait_ready()` | Wait for disc insertion (30s timeout, TUR polling) |
| `init()` | Firmware upload + unlock + speed calibration |
| `init()` | Route to the matching registered unlocker (if any), then prepare for reads |
| `probe_disc()` | Probe disc surface for optimal speeds |
| `read(lba, count, buf, recovery)` | Read sectors. Single-shot — no inline retries or reset. |
| `reset()` | Eject-cycle escape hatch. Caller-invoked only; not on the read path. |
@@ -32,17 +34,21 @@ platform driver. The drive is ready for `wait_ready()` and `init()`.
| `unlock_tray()` | Allow tray ejection (also runs on Drop) |
| `eject()` | Eject disc tray |
| `drive_status()` | Query physical state (disc present, tray open, etc.) |
| `has_profile()` | Whether a bundled profile matched |
| `has_profile()` | Whether a registered unlocker matches this drive |
| `close()` | Consume Drive, cleanup (also runs via Drop) |
### init() Sequence
`init()` orchestrates the full drive unlock:
`init()` routes drive preparation through the unlock seam:
1. Platform driver `run_init()` — sends vendor-specific SCSI commands
2. If firmware upload needed: upload, wait 10s for drive reset, retry
3. Speed calibration after unlock
4. Max 3 attempts before giving up
1. Walk the registered-unlocker registry; the first whose `matches()` is true
is asked to `unlock_drive()` over the raw transport.
2. Whatever that unlocker needs (firmware upload, vendor handshakes, retries)
is the unlocker's own business — libfreemkv only forwards the transport.
3. If no unlocker matches, the drive is left untouched and the library uses
the host-certificate AACS handshake.
See [Drive Unlock Seam](#drive-unlock-seam) for the trait and registry.
### read() — single-shot
@@ -164,100 +170,68 @@ date for drives where Feature 010C is unavailable.
---
## Drive Profiles
## Drive Unlock Seam
Profiles are JSON objects compiled into the binary (`profiles.json`).
Each profile contains:
| Field | Purpose |
|-------|---------|
| `vendor_id`, `product_revision`, `vendor_specific`, `firmware_date` | Matching fields |
| `chipset` | `"mediatek"` or `"renesas"` |
| `unlock_mode`, `unlock_buf_id` | READ BUFFER CDB parameters |
| `signature` | Expected 4-byte response signature |
| `unlock_cdb` | Pre-built unlock CDB (hex-encoded) |
| `register_offsets` | Offsets for hardware register reads |
| `capabilities` | Feature flags: `bd_raw_read`, `dvd_all_regions`, etc. |
Loading:
libfreemkv ships **no firmware, no unlock CDBs, and no drive profiles.** It
knows only the *seam*, never the *mechanism*. The seam is the `Unlocker`
trait plus a small process-wide registry (`src/unlock.rs`):
```rust
// Bundled (compiled-in) -- no file I/O
let profiles = profile::load_bundled()?;
pub trait Unlocker: Send + Sync {
/// Stable, language-neutral identifier (logged).
fn name(&self) -> &str;
// External file
let profiles = profile::load_all(Path::new("/path/to/profiles.json"))?;
/// True if this unlocker handles the given drive.
fn matches(&self, id: &DriveId) -> bool;
/// Put the drive into extended-access mode. The one required capability.
fn unlock_drive(&self, scsi: &mut dyn ScsiTransport, id: &DriveId) -> Result<()>;
/// Read the disc Volume ID via the drive's OEM path. Default: no-op.
fn read_volume_id(&self, _scsi: &mut dyn ScsiTransport, _id: &DriveId)
-> Result<Option<[u8; 16]>> { Ok(None) }
/// Raise the drive to its maximum read speed. Default: no-op.
fn set_max_read_speed(&self, _scsi: &mut dyn ScsiTransport, _id: &DriveId)
-> Result<()> { Ok(()) }
}
```
---
An unlocker is supplied by an **external crate** and registered once at
process start:
## Chipsets
```rust
libfreemkv::register_unlocker(Box::new(some_unlocker::Plugin::new()));
```
### MediaTek MT1959
The implementor owns everything about *how* a particular drive family is
driven — drive identification against its own profile database, firmware
upload, vendor CDBs, variant logic. libfreemkv only hands over the raw
`ScsiTransport` and the `DriveId`.
Covers all LG, ASUS, and HP optical drives. Two sub-variants share identical
logic with different SCSI parameters:
### Routing
| Variant | READ BUFFER mode | Buffer ID |
|---------|------------------|-----------|
| MT1959-A | 0x01 | 0x44 |
| MT1959-B | 0x02 | 0x77 |
At drive-prep the registry is walked in registration order; the first
unlocker whose `matches()` returns true is asked to `unlock_drive()` (and,
when needed, `read_volume_id()` / `set_max_read_speed()`). If no unlocker
matches, the drive is left untouched and the library falls back to the
standard host-certificate AACS handshake (the "OEM route"). The
`register_unlocker(...)` line is the entire plug: drop it (and the unlocker
crate) and libfreemkv still compiles and rips via the cert handshake.
The Platform trait maps to command handlers:
| Handler | Function | Description |
|---------|----------|-------------|
| 0 | `unlock()` | Send READ BUFFER, verify signature + verification bytes |
| 1 | `read_config()` | Read 1888-byte configuration block + 4-byte status |
| 2-3 | `read_register()` | Read hardware registers at profile-specified offsets |
| 4 | `calibrate()` | Probe disc surface, build 64-entry speed table |
| 5 | `keepalive()` | Periodic session maintenance |
| 6 | `status()` | Query current mode and feature flags |
| 7 | `probe()` | Generic READ BUFFER with dynamic parameters |
| 8 | `read_sectors()` | Speed lookup + SET CD SPEED + READ(10) with flag 0x08 |
| 9 | `timing()` | Timing calibration |
### Renesas (Planned)
RS8xxx/RS9xxx chipsets used in Pioneer and some HL-DT-ST drives.
Currently returns `Error::UnsupportedDrive` when a Renesas profile is matched.
---
## Why Unlock Is Needed
Optical drive firmware restricts what applications can read from disc. Without
unlock:
- **READ(10) works for unencrypted filesystem data.** UDF structures, MPLS
playlists, and CLPI clip info are readable without unlock. Standard READ(10)
works on any drive.
- **READ(10) fails for encrypted content sectors.** The drive firmware returns
SCSI errors (sense key 0x05, illegal request) when an application attempts to
read sectors containing encrypted m2ts content without prior AACS
authentication via the bus key.
- **Raw mode bypasses firmware restrictions.** After unlock, the drive accepts
READ(10) with the raw read flag (CDB byte 1 = 0x08) for all sectors,
regardless of encryption status.
### AACS Before Unlock
AACS bus authentication uses standard MMC REPORT KEY / SEND KEY commands.
On some drives these must execute before unlock. The `Disc::scan()` handles
this internally — it manages the handshake/unlock ordering automatically.
Concrete unlockers — including the firmware-unlock profile databases,
variant logic, and vendor CDBs that used to live in-tree — are maintained
in the separate **[freemkv-unlock](https://github.com/freemkv/freemkv-unlock)**
repository, never here.
---
## Speed Control
After `probe_disc()`, the platform driver maintains a speed lookup table
built by probing the disc surface. On each `read()` call, the driver:
1. Looks up the optimal speed for the target LBA.
2. Issues SET CD SPEED (0xBB) if the speed differs from current.
3. Performs the READ(10).
A matching unlocker may raise the drive to its maximum read speed via
`set_max_read_speed()` (a no-op when no unlocker matches or the unlocker
declines). The library issues SET CD SPEED (0xBB) through the generic CDB
builder; the concrete speed policy lives in the unlocker.
Available speeds:
+6 -6
View File
@@ -108,8 +108,8 @@ lines of `Mapfile::stats()` checks.
2. On success: write data to ISO, mark `+`, advance.
3. On failure (with `multipass`): zero-fill, mark `*`, advance.
4. Track a sliding window of the last 16 ECC block results. When ≥12% are failures
→ **damage-jump**: skip ahead by `256×batch×multiplier` sectors (8 MB base for
UHD). Double the multiplier on each jump (8→16→32→64 MB...). Zero-fill the gap as `*`.
→ **damage-jump**: skip ahead by `1024×batch×multiplier` sectors (64 MB base for
UHD). Double the multiplier on each jump (64→128→256→512 MB...). Zero-fill the gap as `*`.
5. On 16 consecutive good reads: reset jump multiplier to 1, restore max read speed.
6. Speed control: damage zone entry → minimum speed, exit → maximum speed.
7. Only transport failures (USB bridge crash) abort the pass.
@@ -161,8 +161,8 @@ recoveries to show for it. Recovery responsibility is now layered: layer 1
handles ranges, layer 3 handles request size, neither touches the
wedge-prone reset path.
**No `MODE SELECT` to disable drive retries.** Research showed neither ddrescue
nor MakeMKV does this. Drive firmware has access to raw analog signal, laser
**No `MODE SELECT` to disable drive retries.** Neither ddrescue
nor any consumer ripper does this. Drive firmware has access to raw analog signal, laser
power control, and drive-specific ECC tuning that userspace can't replicate —
disabling it throws away recovery headroom on marginal sectors. We fail fast
via short SG_IO timeouts in pass 1 and let the firmware work the long timeout
@@ -178,7 +178,7 @@ explicitly by callers that need an eject-cycle escape hatch — it is never
reached from a read path.
**ISO intermediate, even for single-pass.** Pass 1 always writes an ISO. The
mux stage reads the ISO via `IsoSectorReader`. For single-pass (no retries),
mux stage reads the ISO via `FileSectorSource`. For single-pass (no retries),
this adds ~2-3 min (local disk mux) but gains resumability across crashes,
re-muxability without re-ripping, and a persistent forensic artifact. Callers
who need pure speed can bypass and use `DiscStream::new(Box::new(drive), …)`
@@ -198,4 +198,4 @@ scrape vs. retry with direction reversal) if there's measured benefit.
- [ddrescue manual, Algorithm chapter](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)
- [ddrescue optical media notes](https://www.electric-spoon.com/doc/gddrescue/html/Optical-media.html)
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/sweep.rs`](../src/disc/sweep.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/mod.rs`](../src/disc/mod.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
-2067
View File
File diff suppressed because it is too large Load Diff
+1334
View File
File diff suppressed because it is too large Load Diff
+102
View File
@@ -0,0 +1,102 @@
//! AACS common cryptographic primitives — [C] Chapter 2 / §3.2.2.
//!
//! Source: `[C]` = AACS Introduction and Common Cryptographic Elements Book,
//! Rev 0.953. The shared low-level building blocks — AES-128 ECB E/D, AES-G,
//! the AES-G3 Triple Generator, AES-CBC decrypt — and their fixed constants
//! (`iv0`, `s0`). Used by every AACS generation; relocated here so the
//! primitives live in one place instead of being scattered across the
//! content / keys / variant modules.
use aes::Aes128;
use aes::cipher::{BlockDecrypt, BlockEncrypt, KeyInit, generic_array::GenericArray};
/// Fixed IV used by AACS for all AES-CBC operations. [C] §2.1.2 (default CBC IV, `iv0`).
pub(crate) const AACS_IV: [u8; 16] = [
0x0B, 0xA0, 0xF8, 0xDD, 0xFE, 0xA6, 0x1F, 0xB3, 0xD8, 0xDF, 0x9F, 0x56, 0x6A, 0x05, 0x0F, 0x78,
];
/// AES-128-ECB encrypt a single 16-byte block. [C] §2.1.1 (`AES-128E`).
pub(crate) fn aes_ecb_encrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut block = GenericArray::clone_from_slice(data);
cipher.encrypt_block(&mut block);
let mut out = [0u8; 16];
out.copy_from_slice(&block);
out
}
/// AES-128-ECB decrypt a single 16-byte block. [C] §2.1.1 (`AES-128D`).
pub(crate) fn aes_ecb_decrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut block = GenericArray::clone_from_slice(data);
cipher.decrypt_block(&mut block);
let mut out = [0u8; 16];
out.copy_from_slice(&block);
out
}
/// AES-128-CBC decrypt in-place with the fixed AACS IV. [C] §2.1.2 (`AES-128CBCD`).
///
/// Precondition: `data.len()` is a multiple of 16. Any trailing partial
/// block is silently ignored; all callers pass aligned regions (6128 and
/// 2032 bytes), and the assert documents/enforces that contract.
pub(crate) fn aes_cbc_decrypt(key: &[u8; 16], data: &mut [u8]) {
debug_assert!(
data.len() % 16 == 0,
"aes_cbc_decrypt requires a block-aligned slice"
);
let cipher = Aes128::new(GenericArray::from_slice(key));
let num_blocks = data.len() / 16;
// Process blocks in reverse to avoid clobbering ciphertext needed for XOR
for i in (0..num_blocks).rev() {
let offset = i * 16;
let prev = if i == 0 {
AACS_IV
} else {
let mut p = [0u8; 16];
p.copy_from_slice(&data[(i - 1) * 16..i * 16]);
p
};
let mut block = GenericArray::clone_from_slice(&data[offset..offset + 16]);
cipher.decrypt_block(&mut block);
for j in 0..16 {
data[offset + j] = block[j] ^ prev[j];
}
}
}
/// AES-G(x1, x2) = AES-128D(x1, x2) XOR x2. [C] §2.1.3 (note: uses AES-128**D**).
///
/// The Media Key Variant chain uses AES-G to derive both the variant
/// number (`Kvn = AES-G(Kp, Nonce)`) and the Volume Unique Key
/// (`Kvu = AES-G(Km, VID)`). See [`super::derive::derive_vuk`] for the
/// classical VUK form — the math is identical, this exposes it as a
/// neutral primitive for the variant chain.
pub(crate) fn aes_g(x1: &[u8; 16], x2: &[u8; 16]) -> [u8; 16] {
let mut out = aes_ecb_decrypt(x1, x2);
for i in 0..16 {
out[i] ^= x2[i];
}
out
}
/// AACS-G3 seed constant (`s0`). [C] §3.2.2.
pub(crate) const AESG3_SEED: [u8; 16] = [
0x7B, 0x10, 0x3C, 0x5D, 0xCB, 0x08, 0xC4, 0xE5, 0x1A, 0x27, 0xB0, 0x17, 0x99, 0x05, 0x3B, 0xD9,
];
/// AACS-G3: derive a subkey from a parent key. [C] §3.2.2 (Triple AES Generator:
/// left=`D(k,s0)⊕s0` inc 0, pk=`D(k,s0+1)⊕(s0+1)` inc 1, right=`D(k,s0+2)⊕(s0+2)` inc 2).
/// seed[15] += inc, then AES-DEC(key, seed) XOR seed.
///
/// Shared with [`super::variant`] (its variant chain runs the same SD
/// tree); a single definition keeps the two walks byte-identical.
pub(crate) fn aesg3(key: &[u8; 16], inc: u8) -> [u8; 16] {
let mut seed = AESG3_SEED;
seed[15] = seed[15].wrapping_add(inc);
let mut out = aes_ecb_decrypt(key, &seed);
for i in 0..16 {
out[i] ^= seed[i];
}
out
}
-299
View File
@@ -1,299 +0,0 @@
//! AACS content decryption — AES primitives, unit decryption, bus encryption.
use aes::Aes128;
use aes::cipher::{BlockDecrypt, BlockEncrypt, KeyInit, generic_array::GenericArray};
// ── AACS constants ──────────────────────────────────────────────────────────
/// Fixed IV used by AACS for all AES-CBC operations.
pub(crate) const AACS_IV: [u8; 16] = [
0x0B, 0xA0, 0xF8, 0xDD, 0xFE, 0xA6, 0x1F, 0xB3, 0xD8, 0xDF, 0x9F, 0x56, 0x6A, 0x05, 0x0F, 0x78,
];
/// Size of an AACS aligned unit (3 × 2048-byte sectors).
pub const ALIGNED_UNIT_LEN: usize = 6144;
/// Size of one sector.
const SECTOR_LEN: usize = 2048;
/// Transport stream packet spacing in Blu-ray m2ts (192 bytes = 4 TP_extra + 188 TS).
const TS_PACKET_LEN: usize = 192;
/// TS sync byte.
const TS_SYNC: u8 = 0x47;
// ── AES primitives ──────────────────────────────────────────────────────────
/// AES-128-ECB encrypt a single 16-byte block.
pub(crate) fn aes_ecb_encrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut block = GenericArray::clone_from_slice(data);
cipher.encrypt_block(&mut block);
let mut out = [0u8; 16];
out.copy_from_slice(&block);
out
}
/// AES-128-ECB decrypt a single 16-byte block.
pub(crate) fn aes_ecb_decrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut block = GenericArray::clone_from_slice(data);
cipher.decrypt_block(&mut block);
let mut out = [0u8; 16];
out.copy_from_slice(&block);
out
}
/// AES-128-CBC decrypt in-place with the fixed AACS IV.
/// AES-128-CBC decrypt in-place with the fixed AACS IV.
pub(crate) fn aes_cbc_decrypt(key: &[u8; 16], data: &mut [u8]) {
let cipher = Aes128::new(GenericArray::from_slice(key));
let num_blocks = data.len() / 16;
// Process blocks in reverse to avoid clobbering ciphertext needed for XOR
for i in (0..num_blocks).rev() {
let offset = i * 16;
let prev = if i == 0 {
AACS_IV
} else {
let mut p = [0u8; 16];
p.copy_from_slice(&data[(i - 1) * 16..i * 16]);
p
};
let mut block = GenericArray::clone_from_slice(&data[offset..offset + 16]);
cipher.decrypt_block(&mut block);
for j in 0..16 {
data[offset + j] = block[j] ^ prev[j];
}
}
}
// ── Content decryption ──────────────────────────────────────────────────────
/// Check if a 6144-byte aligned unit is encrypted (copy_permission_indicator bits).
pub fn is_unit_encrypted(unit: &[u8]) -> bool {
unit.len() >= ALIGNED_UNIT_LEN && (unit[0] & 0xC0) != 0
}
/// Verify decrypted unit by checking TS sync bytes at expected offsets.
fn verify_ts(unit: &[u8]) -> bool {
// In a 6144-byte unit, TS packets start at byte 0 with 4-byte TP_extra_header
// then 188-byte TS packet, repeating every 192 bytes.
// Sync byte 0x47 should appear at offset 4, 196, 388, ...
let mut count = 0;
let mut offset = 4;
while offset < unit.len() {
if unit[offset] == TS_SYNC {
count += 1;
}
offset += TS_PACKET_LEN;
}
// Expect at least most packets to have sync bytes
let total = (unit.len() - 4) / TS_PACKET_LEN + 1;
count > total / 2
}
/// Decrypt one AACS aligned unit (6144 bytes) in-place.
/// Returns true if decryption succeeded (verified by TS sync bytes).
///
/// Algorithm:
/// 1. AES-128-ECB encrypt first 16 bytes with unit_key → derived
/// 2. XOR derived with original 16 bytes → unit_decrypt_key
/// 3. AES-128-CBC decrypt bytes 16..6143 with unit_decrypt_key and AACS IV
/// 4. Clear encryption flag bits
pub fn decrypt_unit(unit: &mut [u8], unit_key: &[u8; 16]) -> bool {
if unit.len() < ALIGNED_UNIT_LEN {
return false;
}
if !is_unit_encrypted(unit) {
return true; // not encrypted
}
// Save original first 16 bytes (they're plaintext TP_extra_header)
let mut header = [0u8; 16];
header.copy_from_slice(&unit[..16]);
// Step 1: Encrypt header with unit key to derive per-unit key
let derived = aes_ecb_encrypt(unit_key, &header);
// Step 2: XOR to get the actual decryption key
let mut decrypt_key = [0u8; 16];
for i in 0..16 {
decrypt_key[i] = derived[i] ^ header[i];
}
// Step 3: Decrypt bytes 16..6143 with AES-CBC
aes_cbc_decrypt(&decrypt_key, &mut unit[16..ALIGNED_UNIT_LEN]);
// Step 4: Clear encryption flag
unit[0] &= !0xC0;
// Verify
verify_ts(unit)
}
/// Decrypt one aligned unit trying multiple unit keys. Returns the key index that worked.
pub fn decrypt_unit_try_keys(unit: &mut [u8], unit_keys: &[[u8; 16]]) -> Option<usize> {
if !is_unit_encrypted(unit) {
return Some(0);
}
// Save original for retry
let original = unit[..ALIGNED_UNIT_LEN].to_vec();
for (i, key) in unit_keys.iter().enumerate() {
unit[..ALIGNED_UNIT_LEN].copy_from_slice(&original);
if decrypt_unit(unit, key) {
return Some(i);
}
}
// Restore original on failure
unit[..ALIGNED_UNIT_LEN].copy_from_slice(&original);
None
}
/// Remove bus encryption from an aligned unit (AACS 2.0 / UHD).
/// Bus encryption uses read_data_key, decrypting bytes 16..2047 of each 2048-byte sector.
pub fn decrypt_bus(unit: &mut [u8], read_data_key: &[u8; 16]) {
for sector_start in (0..ALIGNED_UNIT_LEN).step_by(SECTOR_LEN) {
if sector_start + SECTOR_LEN > unit.len() {
break;
}
// First 16 bytes of each sector are plaintext
aes_cbc_decrypt(
read_data_key,
&mut unit[sector_start + 16..sector_start + SECTOR_LEN],
);
}
}
/// Full decrypt of an aligned unit: bus decrypt (if needed) then AACS decrypt.
pub fn decrypt_unit_full(
unit: &mut [u8],
unit_key: &[u8; 16],
read_data_key: Option<&[u8; 16]>,
) -> bool {
if !is_unit_encrypted(unit) {
return true;
}
if let Some(rdk) = read_data_key {
decrypt_bus(unit, rdk);
}
decrypt_unit(unit, unit_key)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_aes_ecb_roundtrip() {
let key = [
0x15u8, 0x66, 0x5F, 0x98, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A,
0x0B, 0x0C,
];
let plain = [0x41u8; 16];
let enc = aes_ecb_encrypt(&key, &plain);
let dec = aes_ecb_decrypt(&key, &enc);
assert_eq!(dec, plain);
}
#[test]
fn test_decrypt_unit_unencrypted() {
// Unit with 0xC0 bits clear should pass through unchanged
let mut unit = vec![0u8; ALIGNED_UNIT_LEN];
unit[0] = 0x00; // not encrypted
let key = [0u8; 16];
assert!(decrypt_unit(&mut unit, &key));
}
#[test]
fn test_aes_cbc_roundtrip() {
let key = [
0x11u8, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE,
0xFF, 0x00,
];
let original = vec![0x42u8; 128]; // 8 blocks
let mut data = original.clone();
// Encrypt with CBC manually (forward direction)
fn aes_cbc_encrypt(key: &[u8; 16], data: &mut [u8]) {
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut prev = super::AACS_IV;
let num_blocks = data.len() / 16;
for i in 0..num_blocks {
let offset = i * 16;
for j in 0..16 {
data[offset + j] ^= prev[j];
}
let mut block = GenericArray::clone_from_slice(&data[offset..offset + 16]);
cipher.encrypt_block(&mut block);
data[offset..offset + 16].copy_from_slice(&block);
prev.copy_from_slice(&data[offset..offset + 16]);
}
}
aes_cbc_encrypt(&key, &mut data);
assert_ne!(data, original); // should be different after encrypt
super::aes_cbc_decrypt(&key, &mut data);
assert_eq!(data, original); // should match after roundtrip
}
#[test]
fn test_decrypt_unit_synthetic() {
// Build a fake 6144-byte aligned unit with known TS sync pattern,
// encrypt it with the AACS algorithm, then decrypt and verify.
let unit_key = [0xAAu8; 16];
// Build plaintext unit with TS sync bytes every 192 bytes starting at offset 4
let mut plain = vec![0u8; ALIGNED_UNIT_LEN];
let mut offset = 4;
while offset < ALIGNED_UNIT_LEN {
plain[offset] = TS_SYNC;
offset += TS_PACKET_LEN;
}
// Set encryption flag
plain[0] |= 0xC0;
// Now encrypt bytes 16..6143 using the AACS algorithm (reverse of decrypt)
let header: [u8; 16] = plain[..16].try_into().unwrap();
let derived = aes_ecb_encrypt(&unit_key, &header);
let mut encrypt_key = [0u8; 16];
for i in 0..16 {
encrypt_key[i] = derived[i] ^ header[i];
}
// CBC encrypt bytes 16..6143
let cipher = Aes128::new(GenericArray::from_slice(&encrypt_key));
let mut prev = AACS_IV;
let num_blocks = (ALIGNED_UNIT_LEN - 16) / 16;
for i in 0..num_blocks {
let off = 16 + i * 16;
for j in 0..16 {
plain[off + j] ^= prev[j];
}
let mut block = GenericArray::clone_from_slice(&plain[off..off + 16]);
cipher.encrypt_block(&mut block);
plain[off..off + 16].copy_from_slice(&block);
prev.copy_from_slice(&plain[off..off + 16]);
}
// Now plain contains encrypted data. Decrypt it.
let mut unit = plain;
assert!(is_unit_encrypted(&unit));
assert!(decrypt_unit(&mut unit, &unit_key));
assert!(!is_unit_encrypted(&unit)); // flag should be cleared
// Verify TS sync bytes
let mut count = 0;
let mut off = 4;
while off < ALIGNED_UNIT_LEN {
if unit[off] == TS_SYNC {
count += 1;
}
off += TS_PACKET_LEN;
}
assert_eq!(count, (ALIGNED_UNIT_LEN - 4) / TS_PACKET_LEN + 1);
}
}
+688
View File
@@ -0,0 +1,688 @@
//! Media-key derivation: DK/PK → Media Key via the subset-difference tree.
//! [C] §3.2.2–§3.2.5.
use super::crypto::*;
use super::inf::*;
use super::mkb::*;
use super::types::*;
/// Derive Media Key from MKB data using processing keys.
///
/// A Processing Key is **terminal**: it is the key at its Subset-Difference
/// node, one `AES-G` from the Media Key. So this is the fast path — each PK is
/// tried *directly* against the MKB cvalue tables (no tree descent) — the
/// direct PK × cvalue iteration. On a large AACS 2.x UHD MKB
/// (~181k cvalues) this is ~15x faster than treating a PK as a device-node
/// label and walking the tree.
///
/// If you hold a **device-node label** at unknown tree depth (not a terminal
/// PK), derive its Media Key through the device-key path
/// ([`derive_media_key_from_dk`]) — that path owns the Subset-Difference tree
/// walk; the PK path never descends.
///
/// MKB format:
/// Record type 0x10 = Type and Version Record (has MKB version)
/// Record type 0x81 = Verify Media Key Record, AACS 1.0 (has mk_dv)
/// Record type 0x86 = Verify Media Key Record, AACS 2.0/2.1 (has mk_dv)
/// Record type 0x04 = Subset-Difference Index (has UVS entries)
/// Record type 0x05 = Media Key Data Record (cvalues, 1:1 with 0x04)
/// Record type 0x07 = Explicit Subset-Difference Record (NOT cvalues)
pub fn derive_media_key_from_pk(mkb: &[u8], processing_keys: &[[u8; 16]]) -> Option<[u8; 16]> {
let mk_dv = mkb_find_mk_dv(mkb)?;
let uvs = mkb_find_subdiff_records(mkb)?;
let cvalues = mkb_find_cvalues(mkb)?;
try_pk_against_tables(processing_keys, &uvs, &cvalues, &mk_dv)
}
/// Core terminal-PK table scan over explicit record bodies. Each processing
/// key is tried **directly** against every `(uv, cvalue)` pair — no tree
/// descent. Reached in production via [`derive_media_key_from_pk`]; factored
/// out so reproduction harnesses can drive it with explicit tables.
pub(crate) fn try_pk_against_tables(
processing_keys: &[[u8; 16]],
uvs: &[u8],
cvalues: &[u8],
mk_dv: &[u8; 16],
) -> Option<[u8; 16]> {
let num_uvs = uvs
.chunks(5)
.take_while(|c| c.len() == 5 && (c[0] & 0xC0) == 0)
.count();
for pk in processing_keys {
for i in 0..num_uvs {
if (i + 1) * 16 > cvalues.len() {
continue;
}
let record_start = i * 5;
if record_start + 5 > uvs.len() {
continue;
}
let uv = &uvs[record_start + 1..record_start + 5];
let cv = &cvalues[i * 16..(i + 1) * 16];
if let Some(mk) = validate_processing_key(pk, cv, uv, mk_dv) {
return Some(mk);
}
}
}
None
}
/// Validate a processing key against a cvalue/UV pair.
/// Returns the Media Key if valid.
///
/// Steps (media key: [C] §3.2.4; verify relation: [C] §3.2.5.1.4):
/// 1. `mk = AES-128D(pk, cvalue)` [C] §3.2.4
/// 2. `mk[12..16] ^= uv` (4 bytes XOR into the last 4 bytes only) [C] §3.2.4
/// 3. `dec_vd = AES-128D(mk, mk_dv)` [C] §3.2.5.1.4
/// 4. If `dec_vd[0..8] == 01 23 45 67 89 AB CD EF` → valid. [C] §3.2.5.1.4
pub(crate) fn validate_processing_key(
pk: &[u8; 16],
cvalue: &[u8],
uv: &[u8],
mk_dv: &[u8; 16],
) -> Option<[u8; 16]> {
if cvalue.len() < 16 || uv.len() < 4 {
return None;
}
// Step 1: mk = AES-128D(pk, cvalue)
let mut cv = [0u8; 16];
cv.copy_from_slice(&cvalue[..16]);
let mut mk = aes_ecb_decrypt(pk, &cv);
// Step 2: XOR uv into the last 4 bytes of mk (mk[12..16]).
for a in 0..4 {
mk[12 + a] ^= uv[a];
}
// Step 3 + 4: dec_vd = AES-128D(mk, mk_dv); verify magic.
let dec_vd = aes_ecb_decrypt(&mk, mk_dv);
const VERIFY_MAGIC: [u8; 8] = [0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF];
if dec_vd[..8] == VERIFY_MAGIC {
return Some(mk);
}
None
}
/// Compute v_mask from a UV value. [C] §3.2.3. Shared with [`super::variant`].
pub(super) fn calc_v_mask(uv: u32) -> u32 {
let mut v_mask: u32 = 0xFFFF_FFFF;
while (uv & !v_mask) == 0 && v_mask != 0 {
v_mask <<= 1;
}
v_mask
}
/// Derive processing key from device key using subset-difference tree traversal.
/// [C] §3.2.4 (device-tree descent, MSB-branch, terminal PK). Shared with [`super::variant`].
pub(super) fn calc_pk_from_dk(
dk: &[u8; 16],
uv: u32,
v_mask: u32,
dev_key_v_mask: u32,
) -> [u8; 16] {
// Descend from the device node to the record node, following the record's
// `uv` bits. At each level only the child we descend INTO is needed (the
// sibling is computed but never used), and the Processing Key is the
// `aesg3(.,1)` of the FINAL node — so we derive ONE child per level and the
// PK once at the end, instead of left/pk/right at every level. Identical
// result, ~3x fewer block ops. (left child = `aesg3(node,0)`, right = `,2`.)
let mut node = *dk;
let mut current_v_mask = dev_key_v_mask;
// The subset-difference tree is at most 32 levels deep (u32 mask), so the
// walk must converge in <= 32 steps. The arithmetic `>> 1` sign-extends
// current_v_mask, so a v_mask coarser than dev_key_v_mask (reachable from
// a crafted/corrupt MKB) would otherwise saturate at 0xFFFF_FFFF and spin
// forever — bound the loop to keep a bad disc from hanging the rip thread.
let mut steps = 0u32;
while current_v_mask != v_mask {
if steps >= 32 {
break;
}
steps += 1;
// Find the highest unset bit in current_v_mask
let mut bit_pos: i32 = -1;
for i in (0..32).rev() {
if (current_v_mask & (1u32 << i)) == 0 {
bit_pos = i;
break;
}
}
let inc = if bit_pos < 0 || (uv & (1u32 << bit_pos as u32)) == 0 {
0 // left child
} else {
2 // right child
};
node = aesg3(&node, inc);
current_v_mask = ((current_v_mask as i32) >> 1) as u32;
}
aesg3(&node, 1)
}
/// Derive Media Key from MKB using device keys (subset-difference tree).
///
/// Thin wrapper over [`derive_media_key_and_pk_from_dk`] that drops the
/// intermediate Processing Key. Callers that need the PK lineage (e.g.
/// the key service banking DK·PK·MK) should call the `_and_pk_` form.
pub fn derive_media_key_from_dk(mkb: &[u8], device_keys: &[DeviceKey]) -> Option<[u8; 16]> {
derive_media_key_and_pk_from_dk(mkb, device_keys).map(|(mk, _pk)| mk)
}
/// Derive both the Media Key and the intermediate Processing Key from an
/// MKB using device keys (subset-difference tree).
///
/// Identical walk to [`derive_media_key_from_dk`]; this form additionally
/// returns the Processing Key `Kp` derived at the matching subset-difference
/// node — the value `calc_pk_from_dk` produces immediately before it
/// validates into the Media Key. Returns `Some((mk, pk))` for the first DK
/// that walks a uv slot whose Processing Key validates against the MKB.
pub fn derive_media_key_and_pk_from_dk(
mkb: &[u8],
device_keys: &[DeviceKey],
) -> Option<([u8; 16], [u8; 16])> {
let mk_dv = mkb_find_mk_dv(mkb)?;
let uvs = mkb_find_subdiff_records(mkb)?;
let cvalues = mkb_find_cvalues(mkb)?;
// Count UV entries
let num_uvs = uvs
.chunks(5)
.take_while(|c| c.len() == 5 && (c[0] & 0xC0) == 0)
.count();
for dk in device_keys {
let device_number = dk.node as u32;
// Find applying subset-difference for this device
for uvs_idx in 0..num_uvs {
let p_uv = &uvs[1 + 5 * uvs_idx..];
let u_mask_shift = uvs[5 * uvs_idx]; // byte before the UV value
// `num_uvs` was computed via `take_while(.. c[0] & 0xC0 == 0)`, so
// every iterated slot already has its revoked-marker bits clear — no
// inner `& 0xC0` re-check is needed (it would be unreachable).
//
// Shifts of 32..=63 (0x20..=0x3F) have those bits clear but would
// panic in debug / wrap to a wrong mask in release. The MKB byte is
// disc-controlled, so a crafted/corrupt MKB must not crash the ripper:
// skip an out-of-range slot rather than `<<` it.
if u_mask_shift >= 32 {
continue;
}
let uv = u32::from_be_bytes([p_uv[0], p_uv[1], p_uv[2], p_uv[3]]);
if uv == 0 {
continue;
}
// u-mask = shift count of low-order 0 bits ([C] §3.2.5.1.5); v-mask [C] §3.2.3.
let u_mask: u32 = 0xFFFF_FFFF << u_mask_shift;
let v_mask = calc_v_mask(uv);
// Subset-difference applies iff (d&mu)==(uv&mu) && (d&mv)!=(uv&mv). [C] §3.2.4.
if ((device_number & u_mask) == (uv & u_mask))
&& ((device_number & v_mask) != (uv & v_mask))
{
// Found matching subset-difference — find the right device key.
// dk.u_mask_shift is a u8 from keydb with no range check;
// guard the shift the same way as the MKB byte above.
if dk.u_mask_shift >= 32 {
continue;
}
let dev_key_v_mask = calc_v_mask(dk.uv);
let dev_key_u_mask: u32 = 0xFFFF_FFFF << dk.u_mask_shift;
if u_mask == dev_key_u_mask && (uv & dev_key_v_mask) == (dk.uv & dev_key_v_mask) {
// Derive processing key via tree traversal
let pk = calc_pk_from_dk(&dk.key, uv, v_mask, dev_key_v_mask);
// Validate and derive media key
if uvs_idx < cvalues.len() / 16 {
let cv = &cvalues[uvs_idx * 16..(uvs_idx + 1) * 16];
if let Some(mk) =
validate_processing_key(&pk, cv, &uvs[1 + uvs_idx * 5..], &mk_dv)
{
return Some((mk, pk));
}
}
}
}
}
}
None
}
/// Recover the subset-difference position (`node`, `uv`, `u_mask_shift`) of an
/// UNPOSITIONED device key by scanning a disc MKB. A device key alone (just the
/// 16 bytes) cannot be walked — the walk needs its tree node. This finds that
/// node empirically: for each MKB subset-difference record, it tries the device
/// at the record's node AND at every ancestor v-position (the device may sit one
/// or more levels ABOVE the record, descending via AES-G to reach it), deriving
/// the candidate Processing Key DIRECTLY (one [`calc_pk_from_dk`] per candidate,
/// no full re-walk) and checking it validates against that record's cvalue.
///
/// On the first verifying candidate it pins `(uv, u_mask_shift)` — invariant for
/// the key across all discs — and resolves a gate-passing `node` (a one-time
/// ≤32-try search at the single hit). Returns a [`DeviceKey`] ready to bank and
/// reuse on every future disc via [`derive_media_key_from_dk`]. `None` if the
/// key does not apply to this MKB.
///
/// Cost is `O(slots × tree_depth)` — linear in the MKB's subset-difference
/// index, not the quartic cost of re-deriving per candidate.
pub fn recover_dk_position(mkb: &[u8], key: &[u8; 16]) -> Option<DeviceKey> {
let mk_dv = mkb_find_mk_dv(mkb)?;
let uvs = mkb_find_subdiff_records(mkb)?;
let cvalues = mkb_find_cvalues(mkb)?;
let num_uvs = uvs
.chunks(5)
.take_while(|c| c.len() == 5 && (c[0] & 0xC0) == 0)
.count();
let n_cv = cvalues.len() / 16;
// Hoisted ONCE for the whole scan: the Processing Key the device produces if
// it sits EXACTLY at a record (zero descent) is `AES-G3(key, 1)` — it does
// not depend on the record, so the zero-descent probe of every slot reuses
// this single value instead of re-deriving it per slot.
let pk_zero_descent = aesg3(key, 1);
// The slots are independent, so the scan parallelises — a UHD MKB has ~181k
// slots (~26s single-threaded). `find_map_any` returns the first matching
// node found by any thread and cancels the rest; a valid MKB has exactly one
// matching subset-difference, so which thread finds it is immaterial.
use rayon::prelude::*;
let found = (0..num_uvs.min(n_cv)).into_par_iter().find_map_any(|i| {
let u_mask_shift = uvs[5 * i];
if u_mask_shift >= 32 {
return None;
}
let p_uv = &uvs[1 + 5 * i..];
let uv_r = u32::from_be_bytes([p_uv[0], p_uv[1], p_uv[2], p_uv[3]]);
if uv_r == 0 {
return None;
}
let v_mask = calc_v_mask(uv_r);
let cv = &cvalues[i * 16..(i + 1) * 16];
let uv_bytes = &uvs[1 + i * 5..];
// Zero descent (device sits at this slot's node): cheapest, most common.
if validate_processing_key(&pk_zero_descent, cv, uv_bytes, &mk_dv).is_some() {
return Some((uv_r, u_mask_shift));
}
// Descent: device is an ANCESTOR of the slot. Walk the depth bit up from
// the slot's lowest set bit; each level descends to the slot's node.
let p = uv_r.trailing_zeros();
for k in (p + 1)..32 {
let uv_d = if k + 1 >= 32 {
1u32 << k
} else {
(uv_r & (0xFFFF_FFFFu32 << (k + 1))) | (1u32 << k)
};
let pk = calc_pk_from_dk(key, uv_r, v_mask, calc_v_mask(uv_d));
if validate_processing_key(&pk, cv, uv_bytes, &mk_dv).is_some() {
return Some((uv_d, u_mask_shift));
}
}
None
});
found.and_then(|(uv, mask)| resolve_dk_node(mkb, key, uv, mask))
}
/// Resolve a positioned [`DeviceKey`] for an orphan `key` known to sit at
/// `(uv, u_mask_shift)`: find a `device_number` (node) that passes the walk's
/// subset-difference gate on `mkb`. The derived key is independent of the exact
/// node (it only gates), so any gating node yields the same Media Key — a
/// one-time ≤32-try search, run only once at the recovered position.
pub(crate) fn resolve_dk_node(
mkb: &[u8],
key: &[u8; 16],
uv: u32,
u_mask_shift: u8,
) -> Option<DeviceKey> {
for b in 0..u_mask_shift {
let dk = DeviceKey {
key: *key,
node: ((uv ^ (1u32 << b)) & 0xFFFF) as u16,
uv,
u_mask_shift,
};
if derive_media_key_from_dk(mkb, std::slice::from_ref(&dk)).is_some() {
return Some(dk);
}
}
// Degenerate MKB (no gating bit): fall back to the node itself.
Some(DeviceKey {
key: *key,
node: (uv & 0xFFFF) as u16,
uv,
u_mask_shift,
})
}
/// Public, side-effect-free accessors over the MKB record helpers, exposed so
/// independent reproduction harnesses (e.g. `examples/prove_hkd_aacs.rs`) can
/// exercise the exact same parser + verify primitives the production walk uses.
/// These are thin wrappers — no new logic.
#[doc(hidden)]
pub mod probe {
use super::super::crypto::aes_ecb_decrypt;
/// `mk_dv` from the MKB's Verify-Media-Key record (type 0x81 / 0x86).
pub fn mkb_mk_dv(mkb: &[u8]) -> Option<[u8; 16]> {
super::mkb_find_mk_dv(mkb)
}
/// Body of the MKB's Subset-Difference Index record (type 0x04).
pub fn mkb_subdiff(mkb: &[u8]) -> Option<Vec<u8>> {
super::mkb_find_subdiff_records(mkb)
}
/// Body of the MKB's Media-Key-Data (cvalues) record. Selects record
/// `0x05` (the large cvalue table, 1:1 with the `0x04` Subset-Difference
/// index on AACS 2.x UHD MKBs), falling back to `0x07` only when `0x05`
/// is absent.
pub fn mkb_cvalues(mkb: &[u8]) -> Option<Vec<u8>> {
super::mkb_find_cvalues(mkb)
}
/// Body (header stripped) of the first MKB record of `rec_type`. Lets a
/// harness pin an exact record type for cross-checking the production
/// cvalue selection (e.g. compare record `0x05` vs `0x07` sizes).
pub fn mkb_record_body(mkb: &[u8], rec_type: u8) -> Option<Vec<u8>> {
super::find_record_body(mkb, rec_type)
}
/// AES-128-ECB single-block decrypt (the AACS verify primitive).
pub fn aes_dec(key: &[u8; 16], block: &[u8; 16]) -> [u8; 16] {
aes_ecb_decrypt(key, block)
}
/// Does `km` satisfy the MKB's Verify-Media-Key relation?
/// `AES-D(km, mk_dv)[0..8] == 01 23 45 67 89 AB CD EF`.
pub fn km_verifies(mkb: &[u8], km: &[u8; 16]) -> bool {
match super::mkb_find_mk_dv(mkb) {
Some(mk_dv) => {
aes_ecb_decrypt(km, &mk_dv)[..8] == [0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF]
}
None => false,
}
}
}
// ── Volume key: Media Key + Volume ID → VUK → unit keys ──────────────────────
/// Derive VUK from Media Key and Volume ID. [PR] §3.3 / [BD] §3.3
/// (`Kvu = AES-G(Km, IDv)`; AES-G uses AES-128D):
/// VUK = AES-128-ECB-DECRYPT(media_key, volume_id) XOR volume_id
pub fn derive_vuk(media_key: &[u8; 16], volume_id: &[u8; 16]) -> [u8; 16] {
let mut vuk = aes_ecb_decrypt(media_key, volume_id);
for i in 0..16 {
vuk[i] ^= volume_id[i];
}
vuk
}
/// Decrypt an encrypted unit key using the VUK (AES-128-ECB). [PR] §3.5
/// (Title Key unwrap `Kt = AES-128D(Ku, Kte)`); the BD "CPS Unit Key" synonym is [BD] §3.9.3.
pub fn decrypt_unit_key(vuk: &[u8; 16], encrypted_uk: &[u8; 16]) -> [u8; 16] {
aes_ecb_decrypt(vuk, encrypted_uk)
}
/// Decrypt every encrypted unit key in a parsed `Unit_Key_RO.inf` with a VUK,
/// paired with its declared CPS-unit number. THE single VUK→unit-keys step:
/// both classical/v21 resolvers and [`resolve_candidate`] call this, so the
/// map cannot drift between the player and harvest paths.
pub(crate) fn derive_unit_keys(uk_file: &UnitKeyFile, vuk: &[u8; 16]) -> Vec<(u32, [u8; 16])> {
uk_file
.encrypted_keys
.iter()
.map(|(num, enc_key)| (*num, decrypt_unit_key(vuk, enc_key)))
.collect()
}
/// A candidate key at any rung of the AACS ladder, handed to [`resolve_candidate`].
///
/// Each variant carries the [`super::types`] newtype for that rung (a `Dk` is a
/// POSITIONED [`DeviceKey`] — recover an unpositioned one with
/// [`recover_dk_position`] first).
#[derive(Debug, Clone)]
pub enum KeyCandidate {
Uk(UnitKey),
Vuk(Vuk),
Mk(MediaKey),
Pk(ProcessingKey),
Dk(DeviceKey),
}
/// The AACS key chain derived from a candidate, from [`resolve_candidate`].
///
/// PURE DERIVATION — no unit sampling, no validation. `unit_keys` holds every
/// CPS-unit key the disc's `Unit_Key_RO.inf` yields from the VUK (paired with
/// its declared CPS-unit number); the caller runs
/// `decrypt_unit` + `is_clean_ts` to find which one actually opens the
/// disc. Rungs above the candidate are `None`.
#[derive(Debug, Clone)]
pub struct ResolvedChain {
pub unit_keys: Vec<(u32, [u8; 16])>,
pub vuk: Option<Vuk>,
pub mk: Option<MediaKey>,
pub pk: Option<ProcessingKey>,
/// The positioned device key (for a `Dk` candidate).
pub dk: Option<DeviceKey>,
}
/// Derive the full AACS key chain from a candidate key of ANY ladder rung.
///
/// Runs the deterministic derivation DOWNWARD to the disc's terminal unit keys:
/// `DK → MK → VUK → UKs`, `PK → MK → VUK → UKs`, `MK → VUK → UKs`,
/// `VUK → UKs`, or `UK → itself`. Composes the raw derivation primitives
/// ([`derive_media_key_from_pk`], [`derive_media_key_and_pk_from_dk`],
/// [`derive_vuk`], [`derive_unit_keys`]) and parses `Unit_Key_RO.inf` at the
/// version the disc's MKB declares, so a multi-CPS disc yields all its unit
/// keys from the one candidate.
///
/// PURE DERIVATION: no sampling, no validation, no position recovery. Validate
/// `unit_keys` against a real encrypted unit with
/// `decrypt_unit` + `is_clean_ts` to prove the candidate opens the disc.
///
/// Returns `None` only when derivation itself cannot proceed: a PK its MKB
/// rejects, a `Dk` the MKB can't process, a missing VID on a path that needs
/// one, or an unparseable/empty `Unit_Key_RO.inf`.
pub fn resolve_candidate(
candidate: &KeyCandidate,
mkb: &[u8],
unit_key_ro: &[u8],
vid: Option<Vid>,
) -> Option<ResolvedChain> {
// Boil a VUK → all unit keys, each paired with its declared CPS-unit number.
// Derive the stride version from the disc's own MKB, then defer to the shared
// `derive_unit_keys` (the one place both resolvers and this path decrypt).
let boil = |vuk: Vuk| -> Option<Vec<(u32, [u8; 16])>> {
let version = mkb_type(mkb)
.map(|t| t.generation())
.unwrap_or(AacsVersion::V10);
// BD/UHD Unit_Key_RO.inf or HD DVD VTKF000.AACS — dispatched by magic.
let ukf = parse_title_keys(unit_key_ro, version)?;
if ukf.encrypted_keys.is_empty() {
return None;
}
Some(derive_unit_keys(&ukf, &vuk.0))
};
match candidate {
KeyCandidate::Uk(uk) => Some(ResolvedChain {
unit_keys: vec![(uk.idx, uk.key)],
vuk: None,
mk: None,
pk: None,
dk: None,
}),
KeyCandidate::Vuk(v) => Some(ResolvedChain {
unit_keys: boil(*v)?,
vuk: Some(*v),
mk: None,
pk: None,
dk: None,
}),
KeyCandidate::Mk(mk) => {
let vuk = Vuk(derive_vuk(&mk.0, &vid?.0));
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(*mk),
pk: None,
dk: None,
})
}
KeyCandidate::Pk(pk) => {
let km = derive_media_key_from_pk(mkb, std::slice::from_ref(&pk.0))?;
let vuk = Vuk(derive_vuk(&km, &vid?.0));
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(MediaKey(km)),
pk: Some(*pk),
dk: None,
})
}
KeyCandidate::Dk(dk) => {
let (km, pk) = derive_media_key_and_pk_from_dk(mkb, std::slice::from_ref(dk))?;
let vuk = Vuk(derive_vuk(&km, &vid?.0));
Some(ResolvedChain {
unit_keys: boil(vuk)?,
vuk: Some(vuk),
mk: Some(MediaKey(km)),
pk: Some(ProcessingKey(pk)),
dk: Some(dk.clone()),
})
}
}
}
#[cfg(test)]
mod resolve_candidate_tests {
use super::*;
use crate::aacs::crypto::aes_ecb_encrypt;
/// Minimal AACS-1.0 (48-byte stride) `Unit_Key_RO.inf` with `n` encrypted
/// unit keys — `parse_unit_key_ro` numbers CPS units 1..=n.
fn synth_inf(encs: &[[u8; 16]]) -> Vec<u8> {
let uk_pos = 32usize;
let stride = 48usize;
let n = encs.len();
let total = uk_pos + 48 + n.saturating_sub(1) * stride + 16;
let mut inf = vec![0u8; total.max(20)];
inf[..4].copy_from_slice(&(uk_pos as u32).to_be_bytes());
inf[uk_pos..uk_pos + 2].copy_from_slice(&(n as u16).to_be_bytes());
for (i, k) in encs.iter().enumerate() {
let o = uk_pos + 48 + i * stride;
inf[o..o + 16].copy_from_slice(k);
}
inf
}
/// A VUK candidate boils to ALL the disc's unit keys, each paired with its
/// declared CPS-unit number, and each key equals the VUK-decrypt of its slot.
#[test]
fn resolve_candidate_vuk_returns_all_cps_units() {
let vuk = Vuk([0x33u8; 16]);
let encs = [[0x11u8; 16], [0x22u8; 16], [0x44u8; 16]];
let inf = synth_inf(&encs);
let r = resolve_candidate(&KeyCandidate::Vuk(vuk), &[], &inf, None).expect("vuk derives");
let cps: Vec<u32> = r.unit_keys.iter().map(|(c, _)| *c).collect();
assert_eq!(
cps,
vec![1, 2, 3],
"every CPS unit surfaced, numbered from the inf"
);
for ((_, key), enc) in r.unit_keys.iter().zip(encs.iter()) {
assert_eq!(
*key,
decrypt_unit_key(&vuk.0, enc),
"key = VUK-decrypt of its slot"
);
}
assert_eq!(r.vuk, Some(vuk));
assert!(r.mk.is_none() && r.pk.is_none() && r.dk.is_none());
}
/// A bare UK candidate is terminal — it returns itself keyed by its own idx.
#[test]
fn resolve_candidate_uk_is_itself() {
let uk = UnitKey::new(2, [0x9u8; 16]);
let r = resolve_candidate(&KeyCandidate::Uk(uk), &[], &[], None).expect("uk is terminal");
assert_eq!(r.unit_keys, vec![(2, uk.key)]);
assert!(r.vuk.is_none() && r.mk.is_none());
}
/// MK/PK/DK paths derive the VUK from a VID; without one, derivation stops.
#[test]
fn resolve_candidate_mk_requires_vid() {
let r = resolve_candidate(&KeyCandidate::Mk(MediaKey([1u8; 16])), &[], &[], None);
assert!(r.is_none(), "MK path returns None without a VID");
}
/// A planted Processing Key resolves against a synthetic MKB and drives the
/// FULL chain PK → MK → VUK → UK — proving a PK candidate yields real keys.
#[test]
fn resolve_candidate_pk_drives_full_chain() {
let pk: [u8; 16] = [
0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE,
0xFF, 0x00,
];
let mk: [u8; 16] = [
0xA0, 0xA1, 0xA2, 0xA3, 0xA4, 0xA5, 0xA6, 0xA7, 0xA8, 0xA9, 0xAA, 0xAB, 0xAC, 0xAD,
0xAE, 0xAF,
];
let uv: [u8; 4] = [0x00, 0x00, 0x04, 0x00];
let mut mk_raw = mk;
for a in 0..4 {
mk_raw[12 + a] ^= uv[a];
}
let cv = aes_ecb_encrypt(&pk, &mk_raw);
let mut vd = [0x11u8; 16];
vd[..8].copy_from_slice(&[0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF]);
let mk_dv = aes_ecb_encrypt(&mk, &vd);
// 4-byte record header (type + BE24 total length) + body.
let rec = |t: u8, body: &[u8]| -> Vec<u8> {
let total = 4 + body.len();
let mut r = vec![
t,
((total >> 16) & 0xFF) as u8,
((total >> 8) & 0xFF) as u8,
(total & 0xFF) as u8,
];
r.extend_from_slice(body);
r
};
let mut sd = vec![0u8];
sd.extend_from_slice(&uv);
let mut mkb = Vec::new();
mkb.extend_from_slice(&rec(0x10, &[0, 0, 0, 0x20, 0, 0, 0, 0x52]));
mkb.extend_from_slice(&rec(0x86, &mk_dv));
mkb.extend_from_slice(&rec(0x04, &sd));
mkb.extend_from_slice(&rec(0x05, &cv));
let vid = Vid([0x42u8; 16]);
let plain_uk = [0x7Eu8; 16];
let vuk = derive_vuk(&mk, &vid.0);
let enc = aes_ecb_encrypt(&vuk, &plain_uk);
let inf = synth_inf(std::slice::from_ref(&enc));
let r = resolve_candidate(&KeyCandidate::Pk(ProcessingKey(pk)), &mkb, &inf, Some(vid))
.expect("planted PK resolves the full chain");
assert_eq!(r.mk, Some(MediaKey(mk)), "PK recovers the planted MK");
assert_eq!(r.unit_keys.len(), 1);
assert_eq!(
r.unit_keys[0].1, plain_uk,
"PK chain recovers the title key"
);
}
}
File diff suppressed because it is too large Load Diff
+22
View File
@@ -0,0 +1,22 @@
//! Host-certificate collection — the one libfreemkv-side concern left from the
//! old in-tree AACS handshake. The cert mutual-auth itself now lives in the
//! `freemkv-unlock` AACS unlocker; libfreemkv only gathers the certs (a
//! keysource concern) and hands them across the seam.
/// Union the host certificates a scan can offer the drive: the explicit
/// `DriveCredentials`, then each key source's `host_certs(mkb)`. Host certs are
/// keysource-served, never compiled in. `mkb` lets a source pick a
/// generation-appropriate cert (the default impl ignores it).
pub fn collect_host_certs(
opts: &crate::disc::ScanOptions,
mkb: Option<u32>,
) -> Vec<crate::aacs::types::HostCert> {
let mut host_certs: Vec<crate::aacs::types::HostCert> = Vec::new();
if let Some(c) = &opts.credentials {
host_certs.extend(c.host_certs.iter().cloned());
}
for src in &opts.key_sources {
host_certs.extend(src.host_certs(mkb));
}
host_certs
}
+178
View File
@@ -0,0 +1,178 @@
//! FMTS index selection — the pure decode-time decision for a 2.1 disc.
//!
//! A 2.1 disc resolves to exactly one forensic index (1..=32) for a given
//! rip. `IndividualSegment.tbl` tags each forensic segment with an index (see
//! [`super::segment`]); the decode keeps the segments matching our index,
//! drops the other 31, and treats everything outside a segment as ordinary
//! (index-0) content. This module owns that classification and nothing else —
//! no I/O, no keys, no cipher — so it is fully testable in isolation. The
//! decrypt pipeline consumes the [`UnitDisposition`] it returns.
//!
//! Where the resolved index comes from is a separate concern
//! ([`resolve_disc_index`]): today it is read off the index keys the key
//! source handed us; when Processing Keys are available it will come from the
//! VK derivation instead. Either way the disposition logic below is identical.
use super::segment::{Segment, segment_for_unit};
use super::types::UnitKey;
/// What the decode should do with one AACS aligned unit.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum UnitDisposition {
/// Outside every forensic segment: ordinary content, decrypt with the
/// default (index-0) unit key.
Default,
/// Inside a forensic segment tagged with OUR resolved index: decrypt with
/// that index's key.
Index(u8),
/// Inside a forensic segment tagged with a DIFFERENT index: not our
/// watermark, so it is not part of our output — drop it.
DropForeignIndex(u8),
/// Inside a forensic segment but no index key is held (the disc's index
/// was never resolved): the segment cannot be decoded, so it is concealed
/// as loss. Carries the segment's index for diagnostics.
ForensicNoKey(u8),
}
/// Resolve the disc's single forensic index from the keys we hold.
///
/// Scans for an index key (`index_number` in `1..=32`) and returns its
/// index. `None` when only default (index-0) keys are held — i.e. no
/// index source answered, so forensic segments are not decodable. A disc has
/// exactly one index, so the first non-zero key decides; if several distinct
/// index keys were somehow supplied the lowest wins (deterministic), which is
/// only a defensive tiebreak — the probe/derivation yields one.
pub fn resolve_disc_index(unit_keys: &[UnitKey]) -> Option<u8> {
unit_keys
.iter()
.map(|k| k.index_number)
.filter(|&v| v != 0)
.min()
}
/// Classify the AACS aligned unit at `unit_offset` (clip-relative bytes) given
/// the forensic segment map and the disc's resolved index (`None` if no
/// index key is held).
pub fn unit_disposition(
unit_offset: u64,
segments: &[Segment],
disc_index: Option<u8>,
) -> UnitDisposition {
match segment_for_unit(segments, unit_offset) {
// Not in any forensic segment → ordinary content.
None => UnitDisposition::Default,
// In a forensic segment → decide by whether it is our index.
Some(seg) => {
let seg_index = seg.index as u8;
match disc_index {
Some(v) if v == seg_index => UnitDisposition::Index(v),
Some(_) => UnitDisposition::DropForeignIndex(seg_index),
None => UnitDisposition::ForensicNoKey(seg_index),
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::aacs::content::ALIGNED_UNIT_LEN;
use crate::aacs::segment::{SOURCE_PACKET_LEN, parse_individual_segments};
/// Build a one-record segment table (index, start_spn, end_spn).
fn tbl(recs: &[(u16, u32, u32)]) -> Vec<Segment> {
let mut v = Vec::new();
v.extend_from_slice(&0x0100_0000u32.to_be_bytes());
v.extend_from_slice(&(recs.len() as u16).to_be_bytes());
v.extend_from_slice(&16u16.to_be_bytes());
for &(n, s, e) in recs {
v.extend_from_slice(&0x0100_0000u32.to_be_bytes());
v.extend_from_slice(&n.to_be_bytes());
v.extend_from_slice(&1u16.to_be_bytes());
v.extend_from_slice(&s.to_be_bytes());
v.extend_from_slice(&e.to_be_bytes());
}
parse_individual_segments(&v).expect("parse")
}
fn uk(idx: u32, index: u8) -> UnitKey {
if index == 0 {
UnitKey::new(idx, [0u8; 16])
} else {
UnitKey::forensic(idx, [index; 16], index)
}
}
#[test]
fn resolve_picks_the_single_index_key() {
// Default keys only → no index resolved.
assert_eq!(resolve_disc_index(&[uk(0, 0)]), None);
assert_eq!(resolve_disc_index(&[]), None);
// One index key among defaults → that index.
assert_eq!(resolve_disc_index(&[uk(0, 0), uk(1, 7)]), Some(7));
// Defensive: lowest of several distinct indexes (deterministic).
assert_eq!(resolve_disc_index(&[uk(0, 9), uk(1, 3)]), Some(3));
}
#[test]
fn unit_outside_segments_is_default() {
let segs = tbl(&[(1, 343680, 346239)]);
let off = 1000u64 * SOURCE_PACKET_LEN; // well before the segment
assert_eq!(
unit_disposition(off, &segs, Some(1)),
UnitDisposition::Default
);
// With no segments at all (1.0 / 2.0), everything is Default.
assert_eq!(
unit_disposition(off, &[], Some(1)),
UnitDisposition::Default
);
}
#[test]
fn unit_in_our_index_decrypts() {
let segs = tbl(&[(7, 100, 200)]);
let off = 120u64 * SOURCE_PACKET_LEN;
assert_eq!(
unit_disposition(off, &segs, Some(7)),
UnitDisposition::Index(7)
);
}
#[test]
fn unit_in_foreign_index_drops() {
// Segment tagged index 7, but our disc index is 3 → drop it.
let segs = tbl(&[(7, 100, 200)]);
let off = 120u64 * SOURCE_PACKET_LEN;
assert_eq!(
unit_disposition(off, &segs, Some(3)),
UnitDisposition::DropForeignIndex(7)
);
}
#[test]
fn forensic_unit_with_no_key_is_concealed() {
// A forensic segment but we never resolved an index → conceal as loss.
let segs = tbl(&[(7, 100, 200)]);
let off = 120u64 * SOURCE_PACKET_LEN;
assert_eq!(
unit_disposition(off, &segs, None),
UnitDisposition::ForensicNoKey(7)
);
}
#[test]
fn straddling_unit_still_classified_as_its_segment() {
// A unit whose 32-packet span only tails into the segment still routes
// to the segment (matches segment_for_unit's span test).
let segs = tbl(&[(5, 100, 200)]);
let unit_packets = (ALIGNED_UNIT_LEN as u64 / SOURCE_PACKET_LEN) as u32; // 32
// Start so the unit covers [80, 80+31] = [80, 111]: overlaps at 100.
let off = 80u64 * SOURCE_PACKET_LEN;
assert!(80 + unit_packets - 1 >= 100, "sanity: unit tails into seg");
assert_eq!(
unit_disposition(off, &segs, Some(5)),
UnitDisposition::Index(5)
);
}
}
+457
View File
@@ -0,0 +1,457 @@
//! AACS on-disc key-input files: `Unit_Key_RO.inf` parsing, the disc-hash
//! keydb lookup key, the Content Certificate, and the in-drive MKB read.
//! These turn raw disc files into the structures the key paths consume.
use super::mkb::*;
/// Parsed Unit_Key_RO.inf file.
#[derive(Debug)]
pub struct UnitKeyFile {
/// Disc hash (SHA1 of the entire file) — used as KEYDB lookup key
pub disc_hash: [u8; 20],
/// Application type (1 = BD-ROM)
pub app_type: u8,
/// Number of BDMV directories
pub num_bdmv_dir: u8,
/// Whether SKB MKB is used
pub use_skb_mkb: bool,
/// AACS generation this file's stride matches
pub version: AacsVersion,
/// Encrypted unit keys (CPS unit number, encrypted key)
pub encrypted_keys: Vec<(u32, [u8; 16])>,
/// Title → CPS unit index mapping (title_idx → unit_key_idx)
pub title_cps_unit: Vec<u16>,
}
/// Compute disc hash (SHA1 of Unit_Key_RO.inf content).
pub fn disc_hash(data: &[u8]) -> [u8; 20] {
use sha1::{Digest, Sha1};
let hash = Sha1::digest(data);
let mut out = [0u8; 20];
out.copy_from_slice(&hash);
out
}
/// Format disc hash as hex string with 0x prefix (for KEYDB lookup).
pub fn disc_hash_hex(hash: &[u8; 20]) -> String {
let mut s = String::with_capacity(42);
s.push_str("0x");
for b in hash {
s.push_str(&format!("{b:02X}"));
}
s
}
/// Parse Unit_Key_RO.inf from raw bytes.
///
/// Format (from AACS spec):
/// [0..4] BE32: offset to key storage area (uk_pos)
/// [16] app_type (1 = BD-ROM)
/// [17] num_bdmv_dir
/// [18] bit 7: use_skb_mkb
/// [20..22] BE16: first_play CPS unit
/// [22..24] BE16: top_menu CPS unit
/// [24..26] BE16: num_titles
/// [26..] title entries: 2 bytes padding + 2 bytes CPS unit, × num_titles
///
/// Key storage at uk_pos:
/// [uk_pos..uk_pos+2] BE16: num_unit_keys
/// [uk_pos+48..] encrypted keys, 16 bytes each
/// AACS 1.0: 48-byte stride
/// AACS 2.0 / 2.1: 64-byte stride (48 + 16 extra)
pub fn parse_unit_key_ro(data: &[u8], version: AacsVersion) -> Option<UnitKeyFile> {
if data.len() < 20 {
return None;
}
let hash = disc_hash(data);
// Header
let app_type = data[16];
let num_bdmv_dir = data[17];
let use_skb_mkb = (data[18] >> 7) & 1 == 1;
// Key storage offset
let uk_pos = u32::from_be_bytes([data[0], data[1], data[2], data[3]]) as usize;
if uk_pos + 2 > data.len() {
return None;
}
// Number of unit keys
let num_uk = u16::from_be_bytes([data[uk_pos], data[uk_pos + 1]]) as usize;
if num_uk == 0 {
return Some(UnitKeyFile {
disc_hash: hash,
app_type,
num_bdmv_dir,
use_skb_mkb,
version,
encrypted_keys: Vec::new(),
title_cps_unit: Vec::new(),
});
}
// Stride between keys
let stride = version.unit_key_stride();
// Validate size
let keys_start = uk_pos + 48; // first key at uk_pos + 48
if keys_start + 16 > data.len() {
return None;
}
// Extract encrypted keys
let mut encrypted_keys = Vec::with_capacity(num_uk);
let mut pos = keys_start;
for i in 0..num_uk {
if pos + 16 > data.len() {
break;
}
let mut key = [0u8; 16];
key.copy_from_slice(&data[pos..pos + 16]);
encrypted_keys.push(((i + 1) as u32, key));
pos += stride;
}
// The loop above `break`s if the buffer runs out mid-key. A short list
// means the .inf is malformed/truncated — reject it rather than silently
// accepting fewer keys than the header declared, which would later map
// title CPS units to nonexistent keys.
if encrypted_keys.len() != num_uk {
return None;
}
// Title → CPS unit mapping (AACS Unit_Key_RO format): each on-disc CPS
// value is in `1..=num_uk` (else zeroes it) and converts the 1-based on-disc
// index to a 0-based key index. We mirror that so the stored value is a safe,
// ready-to-use key index rather than a raw 1-based number.
let to_key_idx = |cps: u16| -> u16 {
if cps >= 1 && cps as usize <= num_uk {
cps - 1
} else {
0
}
};
let mut title_cps_unit = Vec::new();
if data.len() >= 26 {
let first_play = u16::from_be_bytes([data[20], data[21]]);
let top_menu = u16::from_be_bytes([data[22], data[23]]);
let num_titles = u16::from_be_bytes([data[24], data[25]]) as usize;
title_cps_unit.push(to_key_idx(first_play));
title_cps_unit.push(to_key_idx(top_menu));
for i in 0..num_titles {
let off = 26 + i * 4 + 2; // 2 bytes padding + 2 bytes CPS unit
if off + 2 <= data.len() {
let cps = u16::from_be_bytes([data[off], data[off + 1]]);
title_cps_unit.push(to_key_idx(cps));
}
}
}
Some(UnitKeyFile {
disc_hash: hash,
app_type,
num_bdmv_dir,
use_skb_mkb,
version,
encrypted_keys,
title_cps_unit,
})
}
/// HD DVD Video Title Key File (`VTKF000.AACS`) magic — "DVD HD Video TKF".
pub const VTKF_MAGIC: &[u8; 12] = b"DVD_HD_V_TKF";
/// Fixed header length before the first title-key entry.
const VTKF_HEADER_LEN: usize = 0x80;
/// Each title-key entry: BE32 flag + 16-byte encrypted key + 12-byte 0xFF pad.
const VTKF_ENTRY_LEN: usize = 0x20;
/// Parse an HD DVD `VTKF000.AACS` into the SAME [`UnitKeyFile`] a BD/UHD
/// `Unit_Key_RO.inf` yields — so the shared AACS crypto (`derive_unit_keys` →
/// `decrypt_unit_key(vuk, …)`) unwraps HD DVD title keys with no change. Only
/// the on-disc CONTAINER differs between BD and HD DVD; the title-key unwrap is
/// the identical AES-128 VUK step (`Kt = AES-128D(Kvu, Kte)`).
///
/// Layout (grounded in real discs — Shaun of the Dead, Anchorman, Harry Potter):
/// ```text
/// [0x00..0x0C] magic "DVD_HD_V_TKF"
/// [0x0C..0x10] BE32 total file length
/// [0x10..0x1C] associated playlist name ("VPLST000.XPL")
/// [0x1C..0x80] reserved (zero)
/// [0x80..] 32-byte entries: BE32 flag | 16-byte ENCRYPTED title key | 12-byte 0xFF pad
/// flag bit 31 (0x8000_0000) set = present; a cleared flag ends the table
/// [tail] 16-byte signature/MAC (never a key — the cleared-flag stop guards it)
/// ```
/// Entries number 1..=N as CPS units, matching `Unit_Key_RO`'s 1-based CPS
/// numbering, so a title's CPS unit indexes this list identically. The
/// title→CPS mapping itself is playlist-driven (`VPLST000.XPL`) and owned by the
/// HD DVD enumerator, so `title_cps_unit` is left empty here.
pub fn parse_vtkf(data: &[u8]) -> Option<UnitKeyFile> {
if data.len() < VTKF_HEADER_LEN || &data[..12] != VTKF_MAGIC {
return None;
}
// SHA1 of the WHOLE file — the KEYDB lookup key. BackupHDDVD-family key
// databases index an HD DVD disc by SHA1(VTKF000.AACS), the same role the
// BD disc_hash plays for `Unit_Key_RO.inf`.
let hash = disc_hash(data);
let mut encrypted_keys = Vec::new();
let mut pos = VTKF_HEADER_LEN;
let mut cps: u32 = 1;
while pos + VTKF_ENTRY_LEN <= data.len() {
let flag = u32::from_be_bytes([data[pos], data[pos + 1], data[pos + 2], data[pos + 3]]);
// A cleared present-bit terminates the key table. The file's trailing
// 16-byte signature then follows and must NOT be read as a key.
if flag & 0x8000_0000 == 0 {
break;
}
let mut key = [0u8; 16];
key.copy_from_slice(&data[pos + 4..pos + 20]);
encrypted_keys.push((cps, key));
cps += 1;
pos += VTKF_ENTRY_LEN;
}
if encrypted_keys.is_empty() {
return None;
}
Some(UnitKeyFile {
disc_hash: hash,
app_type: 0, // HD DVD VTKF carries no BD-ROM app_type
num_bdmv_dir: 0, // BD-only concept
use_skb_mkb: false,
version: AacsVersion::V10, // HD DVD is always AACS 1.0
encrypted_keys,
title_cps_unit: Vec::new(),
})
}
/// Parse a disc's title-key file, dispatching on the self-describing magic:
/// an HD DVD `VTKF000.AACS` (`DVD_HD_V_TKF`) → [`parse_vtkf`]; anything else is a
/// BD/UHD `Unit_Key_RO.inf` → [`parse_unit_key_ro`]. Both return the same
/// [`UnitKeyFile`], so every downstream AACS derivation stays container-agnostic
/// — the single seam where BD-vs-HD-DVD key layout is resolved (mirrors the key
/// service, which classifies HD DVD by the very same magic).
pub fn parse_title_keys(data: &[u8], version: AacsVersion) -> Option<UnitKeyFile> {
if data.len() >= 12 && &data[..12] == VTKF_MAGIC {
parse_vtkf(data)
} else {
parse_unit_key_ro(data, version)
}
}
/// MKB disc structure format code.
const MKB_DISC_STRUCTURE_FORMAT: u8 = 0x83;
/// MKB pack buffer size.
const MKB_PACK_SIZE: usize = 32772;
/// Read MKB from drive via SCSI (REPORT DISC STRUCTURE format 0x83).
/// Returns the concatenated MKB data from all packs.
pub fn read_mkb_from_drive(
session: &mut dyn crate::scsi::ScsiTransport,
) -> crate::error::Result<Vec<u8>> {
use crate::scsi::{DataDirection, SCSI_READ_DISC_STRUCTURE};
let cdb = [
SCSI_READ_DISC_STRUCTURE,
0x01,
0x00,
0x00,
0x00,
0x00,
0x00,
MKB_DISC_STRUCTURE_FORMAT,
(MKB_PACK_SIZE >> 8) as u8,
(MKB_PACK_SIZE & 0xFF) as u8,
0x00,
0x00,
];
let mut buf = vec![0u8; 32772];
session.execute(&cdb, DataDirection::FromDevice, &mut buf, 10_000)?;
let data_len = u16::from_be_bytes([buf[0], buf[1]]) as usize;
if data_len < 2 {
return Ok(Vec::new());
}
let len = data_len - 2;
let num_packs = buf[3] as usize;
let mut mkb = Vec::with_capacity(32768 * num_packs.max(1));
if len > 0 && len <= 32768 {
mkb.extend_from_slice(&buf[4..4 + len]);
}
// Read remaining packs
for pack in 1..num_packs {
let mut cdb = [
SCSI_READ_DISC_STRUCTURE,
0x01,
0x00,
0x00,
0x00,
0x00,
0x00,
MKB_DISC_STRUCTURE_FORMAT,
(MKB_PACK_SIZE >> 8) as u8,
(MKB_PACK_SIZE & 0xFF) as u8,
0x00,
0x00,
];
// Pack number goes in address field
cdb[2] = ((pack >> 24) & 0xFF) as u8;
cdb[3] = ((pack >> 16) & 0xFF) as u8;
cdb[4] = ((pack >> 8) & 0xFF) as u8;
cdb[5] = (pack & 0xFF) as u8;
let mut buf = vec![0u8; 32772];
if session
.execute(&cdb, DataDirection::FromDevice, &mut buf, 10_000)
.is_ok()
{
let len = u16::from_be_bytes([buf[0], buf[1]]) as usize;
if len > 2 && len - 2 <= 32768 {
mkb.extend_from_slice(&buf[4..4 + len - 2]);
}
}
}
Ok(mkb)
}
/// AACS Content Certificate — identifies disc AACS version and features.
#[derive(Debug)]
pub struct ContentCert {
/// Bus encryption enabled flag
pub bus_encryption: bool,
/// Content Certificate ID (6 bytes)
pub cc_id: [u8; 6],
/// AACS generation indicated by the certificate type byte.
///
/// Cert type `0x00` → [`AacsVersion::V10`]; any other value →
/// [`AacsVersion::V20`]. The certificate alone cannot distinguish
/// V20 from V21 — Variant detection happens after the MKB walk.
pub version: AacsVersion,
}
/// Parse a Content Certificate (ContentXXX.cer) file.
pub fn parse_content_cert(data: &[u8]) -> Option<ContentCert> {
if data.len() < 20 {
return None;
}
// Content Certificate layout (per the AACS content-cert format):
// [0] certificate type (0x00 = AACS1, 0x10 = AACS2)
// [1] bit7 bus_encryption_enabled_flag (`p[1] >> 7`)
// [14..20] cc_id (6 bytes) (`p + 14`)
let version = if data[0] == 0x00 {
AacsVersion::V10
} else {
AacsVersion::V20
};
// The flag is bit 7 of byte 1, NOT bit 0. Reading bit 0 (the prior bug) made
// a bus-encrypted cert (byte1=0x80) read as `false`, defeating the
// AacsBusKeyUnavailable fail-loud gate in disc/encrypt.rs.
let bus_encryption = (data[1] >> 7) & 1 == 1;
let mut cc_id = [0u8; 6];
cc_id.copy_from_slice(&data[14..20]);
Some(ContentCert {
bus_encryption,
cc_id,
version,
})
}
#[cfg(test)]
mod vtkf_tests {
use super::*;
/// Build a synthetic `VTKF000.AACS` matching the real on-disc layout
/// (Shaun of the Dead / Anchorman): magic, BE32 size, playlist name,
/// reserved to 0x80, then 32-byte present-flagged entries, a cleared-flag
/// terminator, and a 16-byte trailer.
fn synth_vtkf(keys: &[[u8; 16]]) -> Vec<u8> {
let mut v = Vec::new();
v.extend_from_slice(VTKF_MAGIC); // 0x00
v.extend_from_slice(&0u32.to_be_bytes()); // 0x0C size (patched below)
v.extend_from_slice(b"VPLST000.XPL"); // 0x10
v.resize(0x80, 0); // reserve to first entry
for k in keys {
v.extend_from_slice(&0x8000_0000u32.to_be_bytes()); // present flag
v.extend_from_slice(k); // 16-byte encrypted title key
v.extend_from_slice(&[0xFFu8; 12]); // 0xFF pad → 32-byte entry
}
// Cleared-flag terminator entry (must NOT be read as a key).
v.extend_from_slice(&[0u8; VTKF_ENTRY_LEN]);
// 16-byte trailing signature (must NOT be read as a key).
v.extend_from_slice(&[0xABu8; 16]);
let len = v.len() as u32;
v[0x0C..0x10].copy_from_slice(&len.to_be_bytes());
v
}
#[test]
fn parse_vtkf_extracts_present_entries_and_stops_at_terminator() {
let k1 = [0x11u8; 16];
let k2 = [0x22u8; 16];
let k3 = [0x33u8; 16];
let data = synth_vtkf(&[k1, k2, k3]);
let ukf = parse_vtkf(&data).expect("valid VTKF must parse");
// Exactly the three present entries — the cleared-flag terminator and
// the 16-byte trailer are NOT mistaken for keys.
assert_eq!(ukf.encrypted_keys.len(), 3, "must stop at the cleared flag");
assert_eq!(ukf.encrypted_keys[0], (1, k1), "CPS units number 1..=N");
assert_eq!(ukf.encrypted_keys[1], (2, k2));
assert_eq!(ukf.encrypted_keys[2], (3, k3));
assert_eq!(ukf.version, AacsVersion::V10, "HD DVD is AACS 1.0");
// disc_hash is SHA1 of the whole file (the KEYDB lookup key).
assert_eq!(ukf.disc_hash, disc_hash(&data));
}
#[test]
fn parse_vtkf_rejects_non_magic() {
let mut data = synth_vtkf(&[[0x11u8; 16]]);
data[0] = b'X'; // corrupt magic
assert!(
parse_vtkf(&data).is_none(),
"non-VTKF magic must be rejected"
);
assert!(
parse_vtkf(&[0u8; 4]).is_none(),
"too short must be rejected"
);
}
#[test]
fn parse_title_keys_dispatches_by_magic() {
// VTKF magic → parse_vtkf.
let data = synth_vtkf(&[[0x44u8; 16], [0x55u8; 16]]);
let ukf = parse_title_keys(&data, AacsVersion::V10).expect("VTKF dispatch");
assert_eq!(ukf.encrypted_keys.len(), 2);
// Non-VTKF → parse_unit_key_ro (a 2-byte buffer is not a valid inf, so
// this proves it ROUTED to the BD parser rather than parse_vtkf).
assert!(
parse_title_keys(&[0x00, 0x00], AacsVersion::V10).is_none(),
"non-magic input must route to parse_unit_key_ro"
);
}
/// The whole point of the seam: a parsed VTKF feeds the SHARED VUK→title-key
/// crypto (`decrypt_unit_key`) exactly like a BD `Unit_Key_RO.inf` would —
/// no HD-DVD-specific crypto path.
#[test]
fn vtkf_encrypted_keys_feed_shared_vuk_unwrap() {
let enc = [0x9Au8; 16];
let data = synth_vtkf(&[enc]);
let ukf = parse_vtkf(&data).unwrap();
let vuk = [0x5Cu8; 16];
let derived = super::super::derive::decrypt_unit_key(&vuk, &ukf.encrypted_keys[0].1);
// Same as applying the shared unwrap directly to the stored enc key.
assert_eq!(derived, super::super::derive::decrypt_unit_key(&vuk, &enc));
}
}
-425
View File
@@ -1,425 +0,0 @@
//! AACS Key Database parsing — KEYDB.cfg format.
use std::collections::HashMap;
/// Parsed AACS key database.
#[derive(Debug)]
pub struct KeyDb {
/// Device keys for MKB processing
pub device_keys: Vec<DeviceKey>,
/// Processing keys (pre-computed media keys for specific MKB versions)
pub processing_keys: Vec<[u8; 16]>,
/// Host certificate + private key for SCSI authentication
pub host_certs: Vec<HostCert>,
/// Per-disc VUK entries indexed by disc hash (hex lowercase)
pub disc_entries: HashMap<String, DiscEntry>,
}
/// A device key for MKB subset-difference tree processing.
#[derive(Debug, Clone)]
pub struct DeviceKey {
pub key: [u8; 16],
pub node: u16,
pub uv: u32,
pub u_mask_shift: u8,
}
/// Host certificate + private key for AACS SCSI authentication.
#[derive(Debug, Clone)]
pub struct HostCert {
/// AACS 1.0: 20 bytes. AACS 2.0: 32 bytes.
pub private_key: [u8; 20],
/// AACS 1.0: 92 bytes. AACS 2.0: 132 bytes.
pub certificate: Vec<u8>,
/// AACS 2.0 host private key (P-256, 32 bytes). None for AACS 1.0 only.
pub private_key_v2: Option<[u8; 32]>,
/// AACS 2.0 host certificate (type 0x11). None for AACS 1.0 only.
pub certificate_v2: Option<Vec<u8>>,
}
/// A per-disc entry from the key database.
#[derive(Debug, Clone)]
pub struct DiscEntry {
/// Disc hash (20 bytes, hex)
pub disc_hash: String,
/// Disc title
pub title: String,
/// Media Key (16 bytes) — from MKB processing
pub media_key: Option<[u8; 16]>,
/// Disc ID (16 bytes)
pub disc_id: Option<[u8; 16]>,
/// Volume Unique Key (16 bytes) — decrypts title keys
pub vuk: Option<[u8; 16]>,
/// Unit keys (title keys) indexed by CPS unit number
pub unit_keys: Vec<(u32, [u8; 16])>,
}
/// Parse a hex string like "0xABCD..." into bytes.
pub(crate) fn parse_hex(s: &str) -> Option<Vec<u8>> {
let s = s.trim().trim_start_matches("0x").trim_start_matches("0X");
if s.len() % 2 != 0 {
return None;
}
let mut out = Vec::with_capacity(s.len() / 2);
for i in (0..s.len()).step_by(2) {
out.push(u8::from_str_radix(&s[i..i + 2], 16).ok()?);
}
Some(out)
}
/// Parse hex into a fixed-size array.
pub(crate) fn parse_hex16(s: &str) -> Option<[u8; 16]> {
let v = parse_hex(s)?;
if v.len() != 16 {
return None;
}
let mut out = [0u8; 16];
out.copy_from_slice(&v);
Some(out)
}
pub(crate) fn parse_hex20(s: &str) -> Option<[u8; 20]> {
let v = parse_hex(s)?;
if v.len() != 20 {
return None;
}
let mut out = [0u8; 20];
out.copy_from_slice(&v);
Some(out)
}
impl KeyDb {
/// Parse a KEYDB.cfg file from a string.
pub fn parse(data: &str) -> Self {
let mut db = KeyDb {
device_keys: Vec::new(),
processing_keys: Vec::new(),
host_certs: Vec::new(),
disc_entries: HashMap::new(),
};
for line in data.lines() {
let line = line.trim();
// Skip comments and empty lines
if line.is_empty() || line.starts_with(';') || line.starts_with('#') {
continue;
}
// Device Key
if line.starts_with("| DK") {
if let Some(dk) = Self::parse_device_key(line) {
db.device_keys.push(dk);
}
continue;
}
// Processing Key
if line.starts_with("| PK") {
if let Some(pk) = Self::parse_processing_key(line) {
db.processing_keys.push(pk);
}
continue;
}
// Host Certificate (AACS 2.0)
if line.starts_with("| HC2") {
if let Some(hc) = db.host_certs.last_mut() {
if let Some((pk, cert)) = Self::parse_host_cert_v2(line) {
hc.private_key_v2 = Some(pk);
hc.certificate_v2 = Some(cert);
}
}
continue;
}
// Host Certificate (AACS 1.0)
if line.starts_with("| HC") {
if let Some(hc) = Self::parse_host_cert(line) {
db.host_certs.push(hc);
}
continue;
}
// Disc entry: starts with 0x
if line.starts_with("0x") && line.contains(" = ") {
if let Some(entry) = Self::parse_disc_entry(line) {
db.disc_entries.insert(entry.disc_hash.clone(), entry);
}
}
}
db
}
/// Load KEYDB.cfg from a file path.
pub fn load(path: &std::path::Path) -> std::io::Result<Self> {
let data = std::fs::read_to_string(path)?;
Ok(Self::parse(&data))
}
/// Look up a disc by its hash. Returns the VUK if found.
pub fn find_vuk(&self, disc_hash: &str) -> Option<[u8; 16]> {
let hash = disc_hash
.trim()
.to_lowercase()
.trim_start_matches("0x")
.to_string();
// Try with 0x prefix and without
self.disc_entries
.get(&format!("0x{hash}"))
.or_else(|| self.disc_entries.get(&hash))
.and_then(|e| e.vuk)
}
/// Look up a disc by its hash. Returns the full entry.
pub fn find_disc(&self, disc_hash: &str) -> Option<&DiscEntry> {
let hash = disc_hash
.trim()
.to_lowercase()
.trim_start_matches("0x")
.to_string();
self.disc_entries
.get(&format!("0x{hash}"))
.or_else(|| self.disc_entries.get(&hash))
}
// ── Parsers ─────────────────────────────────────────────────────────────
fn parse_device_key(line: &str) -> Option<DeviceKey> {
// | DK | DEVICE_KEY 0x... | DEVICE_NODE 0x... | KEY_UV 0x... | KEY_U_MASK_SHIFT 0x...
let key_str = line.split("DEVICE_KEY").nth(1)?.split('|').next()?.trim();
let node_str = line.split("DEVICE_NODE").nth(1)?.split('|').next()?.trim();
let uv_str = line.split("KEY_UV").nth(1)?.split('|').next()?.trim();
let shift_str = line
.split("KEY_U_MASK_SHIFT")
.nth(1)?
.split(';')
.next()?
.split('|')
.next()?
.trim();
Some(DeviceKey {
key: parse_hex16(key_str)?,
node: u16::from_str_radix(node_str.trim_start_matches("0x"), 16).ok()?,
uv: u32::from_str_radix(uv_str.trim_start_matches("0x"), 16).ok()?,
u_mask_shift: u8::from_str_radix(shift_str.trim_start_matches("0x"), 16).ok()?,
})
}
fn parse_processing_key(line: &str) -> Option<[u8; 16]> {
// | PK | 0x...
let parts: Vec<&str> = line.split('|').collect();
if parts.len() >= 3 {
let key_str = parts[2].split(';').next()?.trim();
return parse_hex16(key_str);
}
None
}
fn parse_host_cert(line: &str) -> Option<HostCert> {
// | HC | HOST_PRIV_KEY 0x... | HOST_CERT 0x...
let priv_str = line
.split("HOST_PRIV_KEY")
.nth(1)?
.split('|')
.next()?
.trim();
let cert_str = line
.split("HOST_CERT")
.nth(1)?
.split(';')
.next()?
.split('|')
.next()?
.trim();
Some(HostCert {
private_key: parse_hex20(priv_str)?,
certificate: parse_hex(cert_str)?,
private_key_v2: None,
certificate_v2: None,
})
}
/// Parse AACS 2.0 host cert: `| HC2 | HOST_PRIV_KEY 0x... | HOST_CERT 0x...`
fn parse_host_cert_v2(line: &str) -> Option<([u8; 32], Vec<u8>)> {
let priv_str = line
.split("HOST_PRIV_KEY")
.nth(1)?
.split('|')
.next()?
.trim();
let cert_str = line
.split("HOST_CERT")
.nth(1)?
.split(';')
.next()?
.split('|')
.next()?
.trim();
let priv_bytes = parse_hex(priv_str)?;
if priv_bytes.len() != 32 {
return None;
}
let mut pk = [0u8; 32];
pk.copy_from_slice(&priv_bytes);
let cert = parse_hex(cert_str)?;
if cert.len() < 132 {
return None;
}
Some((pk, cert))
}
fn parse_disc_entry(line: &str) -> Option<DiscEntry> {
// 0x<hash> = <title> | D | <date> | M | 0x<mk> | I | 0x<id> | V | 0x<vuk> | U | <unit_keys>
let (hash_part, rest) = line.split_once(" = ")?;
let disc_hash = hash_part.trim().to_lowercase();
// Extract title (before first |)
let title_part = rest.split(" | ").next().unwrap_or("").trim();
// Clean title: "TITLE_NAME (Display Title)" → use display title if present
let title = if let Some(start) = title_part.find('(') {
if let Some(end) = title_part.rfind(')') {
title_part[start + 1..end].to_string()
} else {
title_part.to_string()
}
} else {
title_part.to_string()
};
// Parse fields by tag
let mut media_key = None;
let mut disc_id = None;
let mut vuk = None;
let mut unit_keys = Vec::new();
let parts: Vec<&str> = rest.split(" | ").collect();
let mut i = 0;
while i < parts.len() {
match parts[i].trim() {
"M" => {
if i + 1 < parts.len() {
media_key = parse_hex16(parts[i + 1].trim());
i += 1;
}
}
"I" => {
if i + 1 < parts.len() {
disc_id = parse_hex16(parts[i + 1].trim());
i += 1;
}
}
"V" => {
if i + 1 < parts.len() {
vuk = parse_hex16(parts[i + 1].trim());
i += 1;
}
}
"U" => {
if i + 1 < parts.len() {
// Unit keys: "1-0xKEY" or "1-0xKEY ; comment"
let uk_str = parts[i + 1].split(';').next().unwrap_or("").trim();
for uk in uk_str.split(' ') {
let uk = uk.trim();
if let Some((num, key)) = uk.split_once('-') {
if let Ok(n) = num.parse::<u32>() {
if let Some(k) = parse_hex16(key) {
unit_keys.push((n, k));
}
}
}
}
i += 1;
}
}
_ => {}
}
i += 1;
}
Some(DiscEntry {
disc_hash,
title,
media_key,
disc_id,
vuk,
unit_keys,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Get KEYDB path from KEYDB_PATH environment variable. Returns None if not set or not found.
fn keydb_path() -> Option<std::path::PathBuf> {
let path = std::path::PathBuf::from(std::env::var("KEYDB_PATH").ok()?);
if path.exists() { Some(path) } else { None }
}
#[test]
fn test_parse_disc_entry() {
let line = r#"***REMOVED*** = DUNE_PART_TWO (Dune: Part Two) | D | 2024-04-02 | M | ***REMOVED*** | I | ***REMOVED*** | V | ***REMOVED*** | U | 1-***REMOVED*** ; MKBv77"#;
let entry = KeyDb::parse_disc_entry(line).unwrap();
assert_eq!(entry.title, "Dune: Part Two");
assert!(entry.media_key.is_some());
assert!(entry.vuk.is_some());
assert_eq!(entry.unit_keys.len(), 1);
assert_eq!(entry.unit_keys[0].0, 1);
}
#[test]
fn test_parse_device_key() {
let line = "| DK | DEVICE_KEY ***REMOVED*** | DEVICE_NODE 0x0800 | KEY_UV 0x00000400 | KEY_U_MASK_SHIFT 0x17 ; MKBv01-MKBv48";
let dk = KeyDb::parse_device_key(line).unwrap();
assert_eq!(dk.node, 0x0800);
assert_eq!(dk.u_mask_shift, 0x17);
}
#[test]
fn test_parse_host_cert() {
let line = "| HC | HOST_PRIV_KEY ***REMOVED*** | HOST_CERT ***REMOVED*** ; Revoked";
let hc = KeyDb::parse_host_cert(line).unwrap();
assert_eq!(hc.private_key[0], 0x90);
assert_eq!(hc.certificate.len(), 92);
}
#[test]
fn test_parse_full_keydb() {
let path = match keydb_path() {
Some(p) => p,
None => return,
}; // skip if not available
let db = KeyDb::load(&path).unwrap();
assert_eq!(db.device_keys.len(), 4);
assert_eq!(db.processing_keys.len(), 3);
assert!(!db.host_certs.is_empty());
assert!(db.disc_entries.len() > 170000);
// Look up Dune: Part Two
let dune = db
.disc_entries
.values()
.find(|e| e.title.contains("Dune: Part Two") && e.vuk.is_some())
.expect("Dune: Part Two not found");
assert!(dune.media_key.is_some());
assert!(dune.vuk.is_some());
assert!(!dune.unit_keys.is_empty());
eprintln!(
"Parsed {} disc entries, {} DK, {} PK",
db.disc_entries.len(),
db.device_keys.len(),
db.processing_keys.len()
);
}
}
-1007
View File
File diff suppressed because it is too large Load Diff
+435
View File
@@ -0,0 +1,435 @@
//! AACS Media Key Block — [C] Chapter 3.
//!
//! The MKB record format (framing walker, the `MkbRecord` view, record-body
//! finders), the MKBType / AACS-generation classification, and MKB-file
//! utilities (content length, trimming, version). Consolidated here so the one
//! place that understands MKB bytes is `mkb`. Some duplicate record finders
//! still live side by side pending a follow-up that collapses them.
// ── MKB record types ([C] Chapter 3) ──────────────────────────────────────
// The ONE canonical set. Every record-type comparison in the `aacs` module
// references these, so a type byte is never a bare literal scattered across
// files (the `0x0c` variant-data record in particular used to appear in several
// hand-rolled forms).
/// Type-and-Version — carries the 32-bit MKBType / AACS generation.
pub(crate) const REC_TYPE_AND_VERSION: u8 = 0x10;
/// Subset-Difference index — the per-slot `(u_mask_shift, uv)` table.
pub(crate) const REC_SUBSET_DIFFERENCE: u8 = 0x04;
/// Media Key Data — the classical (1.0 / 2.0) per-subset cvalue table.
pub(crate) const REC_MEDIA_KEY_DATA: u8 = 0x05;
/// Explicit Subset-Difference — the smaller cvalue table some MKBs use.
pub(crate) const REC_EXPLICIT_SUBSET_DIFF: u8 = 0x07;
/// Media Key Variant Data (AACS 2.1) — the per-subset-difference `C` table
/// (one 16-byte C per slot); the `Kmp` step reads C from HERE, not `0x2d`.
pub(crate) const REC_MEDIA_KEY_VARIANT_DATA: u8 = 0x0c;
/// Variant Data + Nonce (AACS 2.1) — the `VARIANTS[uv]` table (leading bytes)
/// with the 16-byte `Kvn` Nonce at the tail.
pub(crate) const REC_VARIANT_DATA_AND_NONCE: u8 = 0x2d;
/// Variant Key Data table (AACS 2.1) — 65,535×16, indexed by the resolved VKD index.
pub(crate) const REC_VKD_TABLE: u8 = 0x2f;
/// Verify-Media-Key — AACS 1.0.
pub(crate) const REC_VERIFY_MEDIA_KEY_V1: u8 = 0x81;
/// Verify-Media-Key — AACS 2.x.
pub(crate) const REC_VERIFY_MEDIA_KEY_V2: u8 = 0x86;
/// A single MKB record produced by [`walk_mkb`].
#[derive(Debug, Clone)]
pub struct MkbRecord {
/// Byte offset of the record within the MKB.
pub offset: usize,
/// Record type byte.
pub rec_type: u8,
/// Record length in bytes (includes the 4-byte header).
pub rec_len: usize,
/// Record body (the bytes after the 4-byte header).
pub body: Vec<u8>,
}
/// Walk an MKB into a flat list of records.
///
/// MKB record framing per AACS: 1 byte type, 3 bytes BE length
/// INCLUDING the 4-byte header, followed by payload. The walker stops
/// at the first `(type=0, len=0)` end marker or at end of buffer.
pub fn walk_mkb(mkb: &[u8]) -> Vec<MkbRecord> {
mkb_records(mkb)
.map(|(offset, rec_type, rec_len)| MkbRecord {
offset,
rec_type,
rec_len,
body: mkb[offset + 4..offset + rec_len].to_vec(),
})
.collect()
}
/// THE single MKB record-framing walker: yields `(offset, rec_type, rec_len)`
/// for each record — a 4-byte header (type byte + big-endian 24-bit length)
/// then the body — stopping at the `00 000000` end marker or a
/// malformed/out-of-bounds length. Lazy (no body clone), so a find-one-record
/// caller never materialises the multi-MB cvalue table. [`walk_mkb`] and every
/// MKB record walk in `aacs::resolve`/`aacs::derive` are built on this, so the framing rules — and
/// any future fix to them — live in exactly one place (they had drifted across
/// six hand-rolled copies).
pub(crate) fn mkb_records(mkb: &[u8]) -> impl Iterator<Item = (usize, u8, usize)> + '_ {
let mut pos = 0usize;
std::iter::from_fn(move || {
if pos + 4 > mkb.len() {
return None;
}
let rec_type = mkb[pos];
let rec_len = ((mkb[pos + 1] as usize) << 16)
| ((mkb[pos + 2] as usize) << 8)
| (mkb[pos + 3] as usize);
if rec_type == 0 && rec_len == 0 {
return None;
}
if rec_len < 4 || pos + rec_len > mkb.len() {
return None;
}
let here = pos;
pos += rec_len;
Some((here, rec_type, rec_len))
})
}
pub(crate) fn mkb_find_body(records: &[MkbRecord], rec_type: u8) -> Option<&[u8]> {
records
.iter()
.find(|r| r.rec_type == rec_type && !r.body.is_empty())
.map(|r| r.body.as_slice())
}
/// AACS protection generation a disc carries.
///
/// The content cert byte distinguishes V10 (`0x00`) from V20 (`0x01`). V21
/// cannot be detected from the cert alone — a V21 disc carries a V20 cert
/// and is upgraded to `V21` only after the MKB walk turns up the real Variant
/// records `0x2d` / `0x2f` (Encrypted Media Key Variant Data and the Variant
/// Key Data table).
///
/// Key-storage stride in `Unit_Key_RO.inf` is 48 bytes for V10 and 64
/// bytes for V20 / V21.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AacsVersion {
/// AACS 1.0 — original BD-ROM.
V10,
/// AACS 2.0 — UHD-BD, classical Media Key derivation.
V20,
/// AACS 2.1 — UHD-BD with Media Key Variant chain on top of V20.
V21,
}
/// AACS major version as the small integer threaded through the scan / key
/// paths (`AacsState.version`, `DiscInputs.version`, `DiscInputsCtx::new`):
/// 1 = AACS 1.0 (BD), 2 = AACS 2.x (UHD). Centralised so the bare `1`/`2` — and
/// the V10-vs-else stride choice it drives — lives in exactly one place.
pub const AACS_MAJOR_BD: u8 = 1;
pub const AACS_MAJOR_UHD: u8 = 2;
impl AacsVersion {
/// Stride (in bytes) between successive encrypted unit keys in
/// `Unit_Key_RO.inf`.
pub(crate) fn unit_key_stride(self) -> usize {
match self {
AacsVersion::V10 => 48,
AacsVersion::V20 | AacsVersion::V21 => 64,
}
}
/// This version as the major integer ([`AACS_MAJOR_BD`] / [`AACS_MAJOR_UHD`]).
pub fn major(self) -> u8 {
match self {
AacsVersion::V10 => AACS_MAJOR_BD,
AacsVersion::V20 | AacsVersion::V21 => AACS_MAJOR_UHD,
}
}
/// The version a bare major integer selects for stride purposes: only the
/// BD major is V10; every other value takes the V20/V21 64-byte stride.
pub fn from_major(major: u8) -> Self {
if major == AACS_MAJOR_BD {
AacsVersion::V10
} else {
AacsVersion::V20
}
}
}
/// Find Verify Media Key Record (type 0x81 for AACS 1.0, 0x86 for AACS 2.0/2.1) in MKB.
/// 0x81: [C] §3.2.5.1.4. 0x86 (AACS 2.x): [RE] — not in the public spec (from real 2.x MKBs).
pub(crate) fn mkb_find_mk_dv(mkb: &[u8]) -> Option<[u8; 16]> {
// Verify-Media-Key record (0x81 for AACS 1.0, 0x86 for AACS 2.x): mk_dv is
// the 16 bytes at record offset 4 (body offset 0). Needs rec_len >= 20.
let found = mkb_records(mkb).find(|&(_, rt, len)| {
(rt == REC_VERIFY_MEDIA_KEY_V1 || rt == REC_VERIFY_MEDIA_KEY_V2) && len >= 20
});
match found {
Some((o, rec_type, rec_len)) => {
let mut dv = [0u8; 16];
dv.copy_from_slice(&mkb[o + 4..o + 20]);
tracing::debug!(
target: "freemkv::disc",
phase = "mkb_mk_dv_found",
rec_type,
pos = o,
rec_len,
"mk_dv extracted from MKB"
);
Some(dv)
}
None => {
tracing::warn!(
target: "freemkv::disc",
phase = "mkb_mk_dv_not_found",
"no 0x81/0x86 record with rec_len>=20 found"
);
None
}
}
}
/// Find Subset-Difference records (type 0x04) in MKB. [C] §3.2.5.1.5.
pub(crate) fn mkb_find_subdiff_records(mkb: &[u8]) -> Option<Vec<u8>> {
find_record_body(mkb, 0x04)
}
/// Find the Media Key Data Record (cvalues table) in an MKB. [C] §3.2.4 / §3.2.5.1.7.
///
/// The cvalue table is record type `0x05` (Media Key Data) on BOTH AACS
/// 1.0 and AACS 2.x MKBs — its 16-byte cvalue entries are 1:1 with the
/// 5-byte Subset-Difference index entries in record `0x04` — the standard AACS
/// MKB layout (`0x05` cvalues 1:1 with the `0x04` subset-difference index).
///
/// On AACS 2.x in-drive UHD MKBs the `0x05` table is large (the full
/// subset-difference cvalue set: ~181k entries on a retail MKB, 1:1 with
/// the giant `0x04` index), while record `0x07` (Explicit
/// Subset-Difference Record) is a much smaller structure (~96 entries) and
/// is NOT the cvalue table. An earlier version of this function preferred
/// `0x07`, which under-tested the Subset-Difference walk on UHD discs and
/// prevented the DK→walk path from ever finding the matching uv. The
/// selection MUST therefore be `0x05`-first; `0x07` is only a fallback for
/// malformed/legacy MKBs that somehow lack a `0x05` record.
pub(crate) fn mkb_find_cvalues(mkb: &[u8]) -> Option<Vec<u8>> {
if let Some(body) = find_record_body(mkb, 0x05) {
return Some(body);
}
find_record_body(mkb, 0x07)
}
/// Walk an MKB and return the payload (header stripped) of the first
/// record matching `rec_type`. Returns `None` if no such record exists or
/// the record is empty.
pub(crate) fn find_record_body(mkb: &[u8], rec_type_wanted: u8) -> Option<Vec<u8>> {
mkb_records(mkb)
.find(|&(_, rt, len)| rt == rec_type_wanted && len > 4)
.map(|(o, _, len)| mkb[o + 4..o + len].to_vec())
}
/// Real content length of an MKB: the byte offset where the record stream
/// ends. MKB files (especially `MKB_RW.inf`, but `MKB_RO.inf` too on some
/// discs) are allocated to a fixed size — often ~128 MiB — with the records at
/// the front and the rest zero padding. Walking records (type+len) and stopping
/// at the first padding byte (`type == 0` / zero-length / overrun) gives the
/// actual size so callers can trim off megabytes of zeros before sending or
/// archiving. Returns `mkb.len()` only if the whole buffer parsed as records.
pub fn mkb_content_len(mkb: &[u8]) -> usize {
// End of the last framed record = where the fixed-region zero padding begins.
// (The `00 000000` terminator / overrun stops the walk; real MKBs pad with
// zeros, so this matches the prior "stop at the first padding byte".)
mkb_records(mkb)
.last()
.map(|(o, _, len)| o + len)
.unwrap_or(0)
}
/// Trim an MKB's trailing fixed-region padding to its real content length —
/// but ONLY when [`mkb_content_len`] actually found one. It returns 0 for an
/// MKB whose first record cannot be parsed; truncating to 0 in that case would
/// hand downstream consumers (and the online key service) an EMPTY MKB that can
/// never resolve. So a 0 (or a length that isn't strictly inside the buffer)
/// leaves the MKB untouched. A 0.31.0 regression dropped this guard and
/// `truncate`-d unconditionally, zeroing unrecognised MKBs.
pub fn trim_mkb(mut mkb: Vec<u8>) -> Vec<u8> {
let n = mkb_content_len(&mkb);
if n > 0 && n < mkb.len() {
mkb.truncate(n);
}
mkb
}
/// Get MKB version from Type and Version Record (type 0x10).
/// Layout: 4-byte record header at `pos` (type + BE24 length), then the
/// record body starts at `pos + 4`. The body holds the BE u32 Type field at
/// body offset 0 (`pos + 4`), then the BE u32 version at body offset 4
/// (`pos + 8`).
pub fn mkb_version(mkb: &[u8]) -> Option<u32> {
// Type-and-Version record (0x10): version is the BE u32 at body offset 4
// (record offset 8). Needs rec_len >= 12 (4 header + 4 type + 4 version).
mkb_records(mkb)
.find(|&(_, rt, len)| rt == REC_TYPE_AND_VERSION && len >= 12)
.map(|(o, _, _)| u32::from_be_bytes([mkb[o + 8], mkb[o + 9], mkb[o + 10], mkb[o + 11]]))
}
/// `0x00031003` — recordable media MKB (Class I & II compute Km directly).
pub const MKB_TYPE_3_RECORDABLE: u32 = 0x0003_1003;
/// `0x00041003` — AACS 1.0 pre-recorded content MKB (KCD-based). Standard BD.
pub const MKB_TYPE_4_PRERECORDED: u32 = 0x0004_1003;
/// `0x000A1003` — Class II / Unified MKB (Sequence-Key-Block functionality).
pub const MKB_TYPE_10_CLASS_II: u32 = 0x000A_1003;
/// `0x48141003` — AACS 2.0 Category C (UHD content) MKB type value.
pub const MKB_20_CATEGORY_C: u32 = 0x4814_1003;
/// `0x48151003` — AACS 2.1 Category C (UHD content) MKB type value.
pub const MKB_21_CATEGORY_C: u32 = 0x4815_1003;
/// The AACS MKB Type field, decoded.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MkbType {
/// Type 3 — recordable media.
Recordable,
/// Type 4 — AACS 1.0 pre-recorded content (KCD). Standard Blu-ray.
Prerecorded,
/// Type 10 — Class II / Unified (SKB).
ClassII,
/// AACS 2.0 Category C — UHD content.
CategoryC20,
/// AACS 2.1 Category C — UHD content.
CategoryC21,
/// Unrecognized MKBType value (raw field preserved).
Other(u32),
}
impl MkbType {
pub(crate) fn from_raw(raw: u32) -> Self {
match raw {
MKB_TYPE_3_RECORDABLE => MkbType::Recordable,
MKB_TYPE_4_PRERECORDED => MkbType::Prerecorded,
MKB_TYPE_10_CLASS_II => MkbType::ClassII,
MKB_20_CATEGORY_C => MkbType::CategoryC20,
MKB_21_CATEGORY_C => MkbType::CategoryC21,
other => MkbType::Other(other),
}
}
/// AACS generation this MKB belongs to (Category C → 2.0/2.1, else 1.0).
pub fn generation(self) -> AacsVersion {
match self {
MkbType::CategoryC21 => AacsVersion::V21,
MkbType::CategoryC20 => AacsVersion::V20,
_ => AacsVersion::V10,
}
}
/// `true` for UHD (AACS 2.x Category C); `false` for Blu-ray (AACS 1.x).
pub fn is_uhd(self) -> bool {
matches!(self, MkbType::CategoryC20 | MkbType::CategoryC21)
}
}
/// The raw 32-bit MKBType field from the Type-and-Version record (0x10), bytes
/// 4-7. `None` if no 0x10 record is present. [C] §3.2.5.1.1 Table 3-2.
pub fn mkb_type_raw(mkb: &[u8]) -> Option<u32> {
// Type-and-Version record (0x10): the 32-bit MKBType is bytes 4-7 (body
// offset 0). Needs rec_len >= 8 (4 header + 4 type).
mkb_records(mkb)
.find(|&(_, rt, len)| rt == REC_TYPE_AND_VERSION && len >= 8)
.map(|(o, _, _)| u32::from_be_bytes([mkb[o + 4], mkb[o + 5], mkb[o + 6], mkb[o + 7]]))
}
/// Decode an MKB's Type field. `None` if no Type-and-Version record is present.
pub fn mkb_type(mkb: &[u8]) -> Option<MkbType> {
mkb_type_raw(mkb).map(MkbType::from_raw)
}
/// `Some(true)` if this MKB is a UHD (AACS 2.x Category C) block, `Some(false)`
/// for Blu-ray (AACS 1.x), `None` if the Type record is absent.
pub fn mkb_is_uhd(mkb: &[u8]) -> Option<bool> {
mkb_type(mkb).map(MkbType::is_uhd)
}
#[cfg(test)]
mod tests {
use super::*;
/// One MKB record: 1 type byte + big-endian 24-bit total length + body.
fn rec(rec_type: u8, body: &[u8]) -> Vec<u8> {
let len = 4 + body.len();
let mut v = vec![rec_type, (len >> 16) as u8, (len >> 8) as u8, len as u8];
v.extend_from_slice(body);
v
}
/// Type-and-Version record (0x10): body = 4-byte MKBType + 4-byte version.
fn type_and_version(mkb_type: u32, version: u32) -> Vec<u8> {
let mut body = mkb_type.to_be_bytes().to_vec();
body.extend_from_slice(&version.to_be_bytes());
rec(REC_TYPE_AND_VERSION, &body)
}
#[test]
fn walker_frames_records_and_stops_at_end_marker() {
let mut mkb = type_and_version(MKB_20_CATEGORY_C, 77);
mkb.extend(rec(REC_VKD_TABLE, &[0xAA; 16]));
mkb.extend([0x00, 0x00, 0x00, 0x00]); // end marker
mkb.extend(rec(0x99, &[0xFF; 8])); // must NOT be walked (past the marker)
let recs = walk_mkb(&mkb);
assert_eq!(recs.len(), 2, "walk stops at the 00 000000 end marker");
assert_eq!(recs[0].rec_type, REC_TYPE_AND_VERSION);
assert_eq!(recs[1].rec_type, REC_VKD_TABLE);
assert_eq!(recs[1].body, vec![0xAA; 16]);
}
#[test]
fn walker_stops_on_malformed_or_out_of_bounds_length() {
// A record whose declared length runs past the buffer end must terminate
// the walk rather than panic or read OOB.
let mkb = vec![REC_VKD_TABLE, 0x00, 0xFF, 0xFF, 0x01, 0x02]; // len=0xFFFF, only 6 bytes
assert!(
walk_mkb(&mkb).is_empty(),
"over-long record yields no records"
);
// A sub-4 length (shorter than the header itself) is also rejected.
let short = vec![REC_VKD_TABLE, 0x00, 0x00, 0x02];
assert!(walk_mkb(&short).is_empty(), "sub-4 length is rejected");
// A truncated header (< 4 bytes) yields nothing.
assert!(walk_mkb(&[0x10, 0x00]).is_empty());
}
#[test]
fn mkb_type_and_version_decode_from_the_type_record() {
let mut mkb = type_and_version(MKB_21_CATEGORY_C, 100);
mkb.extend([0x00, 0x00, 0x00, 0x00]);
assert_eq!(mkb_type_raw(&mkb), Some(MKB_21_CATEGORY_C));
assert_eq!(mkb_version(&mkb), Some(100));
assert_eq!(mkb_is_uhd(&mkb), Some(true), "2.1 Category C is UHD");
let bd = type_and_version(MKB_TYPE_4_PRERECORDED, 68);
assert_eq!(
mkb_is_uhd(&bd),
Some(false),
"AACS 1.0 prerecorded is not UHD"
);
// No Type record → None (not a panic, not a fabricated value).
assert_eq!(mkb_version(&rec(REC_VKD_TABLE, &[0; 16])), None);
assert_eq!(mkb_type_raw(&[]), None);
}
#[test]
fn trim_mkb_keeps_only_the_framed_records() {
let mut mkb = type_and_version(MKB_20_CATEGORY_C, 1);
let content_len = mkb.len(); // the single framed record, no end marker
mkb.extend([0x00, 0x00, 0x00, 0x00]); // end marker
mkb.extend([0xDE; 4096]); // trailing padding past the end marker
let trimmed = trim_mkb(mkb);
assert_eq!(
trimmed.len(),
content_len,
"trim keeps the framed records, dropping the end marker and padding"
);
}
}
+148 -16
View File
@@ -8,25 +8,157 @@
//! | DK | DEVICE_KEY 0x... | DEVICE_NODE 0x... | KEY_UV 0x... | KEY_U_MASK_SHIFT 0x...
//! | PK | 0x...
//! | HC | HOST_PRIV_KEY 0x... | HOST_CERT 0x...
//! | HC2 | HOST_PRIV_KEY 0x... | HOST_CERT 0x...
//! 0x<disc_hash> = <title> | D | <date> | M | 0x<media_key> | I | 0x<disc_id> | V | 0x<vuk> | U | <unit_keys>
//!
//! The VUK decrypts title keys from AACS/Unit_Key_RO.inf on disc.
//! Title keys decrypt m2ts stream content (AES-128-CBC).
//!
//! ## Spec provenance
//!
//! The crypto below carries `[TAG] §x.y` citations back to the published AACS
//! specification (Final Rev 0.953), so each primitive links to the section it
//! implements:
//! - `[C]` — AACS Introduction and Common Cryptographic Elements Book (primitives, MKB/key-management).
//! - `[PR]` — AACS Pre-recorded Video Book (Volume/Title Key layer).
//! - `[BD]` — AACS Blu-ray Disc Pre-recorded Book (CPS Unit Key, Aligned Unit, Block Key).
//! - `[RE]` — reverse-engineered from real discs, cited only where the public
//! spec is silent (the `0x86` verify record and the Category-C MKB type values).
pub mod decrypt;
pub mod handshake;
pub mod keydb;
pub mod keys;
pub mod content;
pub mod crypto;
pub mod derive;
pub mod host_certs;
pub mod index_select;
pub mod inf;
pub mod mkb;
pub mod provider;
pub mod resolve;
pub mod segment;
pub mod segment_key;
pub mod trace;
pub mod types;
pub mod variant;
// Explicit re-exports — only items needed by external consumers and sibling crate modules.
// AES primitives (aes_ecb_encrypt, aes_ecb_decrypt, aes_cbc_decrypt) are pub(crate) in decrypt.rs.
pub use decrypt::{
ALIGNED_UNIT_LEN, decrypt_bus, decrypt_unit, decrypt_unit_full, decrypt_unit_try_keys,
is_unit_encrypted,
};
pub use keydb::{DeviceKey, DiscEntry, HostCert, KeyDb};
pub use keys::{
ContentCert, ResolvedKeys, UnitKeyFile, decrypt_unit_key, derive_media_key_from_dk,
derive_media_key_from_pk, derive_vuk, disc_hash, disc_hash_hex, mkb_version,
parse_content_cert, parse_unit_key_ro, read_mkb_from_drive, resolve_keys,
};
/// On-disc UDF paths to the AACS key-input files.
///
/// BD and UHD keep their key material under `/AACS/…`; HD DVD keeps the
/// equivalents under `/ANY!/…` with different names (`VTKF000.AACS` is the
/// title-key file — magic `DVD_HD_V_TKF`; `MKBROM.AACS` is the MKB). The
/// container difference is expressed here purely as DATA: each ROLE
/// ([`UNIT_KEY_RO_PATHS`], [`MKB_PATHS`], [`CONTENT_CERT_PATHS`]) is an ordered
/// candidate list, and every reader walks it with [`read_first`] taking the
/// first that reads. No reader ever branches on disc type — a BD/UHD disc has
/// the `/AACS/` files so those win; an HD DVD has neither, so it falls through
/// to the `/ANY!/` entry. Centralised so `resolve_vid_only`, `read_aacs_inputs`,
/// `read_mkb_content`, and `read_aacs_version` can never silently diverge the
/// disc_hash / MKB / VID that another reader feeds a key service.
pub const PATH_UNIT_KEY_RO: &str = "/AACS/Unit_Key_RO.inf";
pub const PATH_UNIT_KEY_RO_DUPLICATE: &str = "/AACS/DUPLICATE/Unit_Key_RO.inf";
pub const PATH_MKB_RO: &str = "/AACS/MKB_RO.inf";
pub const PATH_MKB_RW: &str = "/AACS/MKB_RW.inf";
pub const PATH_CONTENT_CERT: &str = "/AACS/Content000.cer";
pub const PATH_CONTENT_CERT_ALT: &str = "/AACS/Content001.cer";
/// HD DVD title-key file (`/ANY!/`), forwarded as `inf_b64`; the key service
/// recognises it by its `DVD_HD_V_TKF` magic.
pub const PATH_VTKF_HDDVD: &str = "/ANY!/VTKF000.AACS";
/// HD DVD Media Key Block (`/ANY!/`), forwarded as `mkb_b64`.
pub const PATH_MKBROM_HDDVD: &str = "/ANY!/MKBROM.AACS";
/// HD DVD content certificate (`/ANY!/`); byte 0 gives the AACS major (0x00 → V10).
pub const PATH_CONTENT_CERT_HDDVD: &str = "/ANY!/CONTENT_CERT.AACS";
/// Title-key / `Unit_Key_RO.inf` role, in resolution order (BD/UHD, then HD DVD).
pub const UNIT_KEY_RO_PATHS: &[&str] = &[
PATH_UNIT_KEY_RO,
PATH_UNIT_KEY_RO_DUPLICATE,
PATH_VTKF_HDDVD,
];
/// MKB role, in resolution order (BD/UHD RO then RW, then HD DVD).
pub const MKB_PATHS: &[&str] = &[PATH_MKB_RO, PATH_MKB_RW, PATH_MKBROM_HDDVD];
/// Content-certificate role, in resolution order (BD/UHD, then HD DVD).
pub const CONTENT_CERT_PATHS: &[&str] = &[
PATH_CONTENT_CERT,
PATH_CONTENT_CERT_ALT,
PATH_CONTENT_CERT_HDDVD,
];
/// Walk an AACS role's candidate paths and return the first that reads.
///
/// `read` performs the actual per-path read (full file or bounded prefix), so
/// callers share the same first-present walk regardless of read style. Returns
/// [`Error::AacsNoKeys`] if no candidate is present. This is the single place
/// the `/AACS/` (BD/UHD) vs `/ANY!/` (HD DVD) layout difference is resolved.
pub(crate) fn read_first<F>(candidates: &[&str], mut read: F) -> crate::error::Result<Vec<u8>>
where
F: FnMut(&str) -> crate::error::Result<Vec<u8>>,
{
for path in candidates {
if let Ok(buf) = read(path) {
return Ok(buf);
}
}
Err(crate::error::Error::AacsNoKeys)
}
// The module structure IS the public API — consumers import from the owning
// module directly (e.g. `aacs::content::decrypt_unit`, `aacs::mkb::MkbType`,
// `aacs::derive::{derive_vuk, resolve_candidate}`, `aacs::resolve::resolve_keys_v2`).
// The `derive::probe` reproduction harness stays reachable via its module path.
//
// A small set of flat re-exports is kept for the typed key primitives and the
// content-decrypt entry points that downstream key-source crates import through
// the `aacs::` path. These are the stable, load-bearing names; keeping them here
// lets those crates track the module refactor without a lockstep re-pin.
pub use content::ALIGNED_UNIT_LEN;
pub use derive::derive_vuk;
pub use types::{DeviceKey, HostCert, MediaKey, ProcessingKey, UnitKey, Vid, Vuk};
#[cfg(test)]
mod tests {
//! Surface guards. The public API is the module tree itself (no facade).
//! Touching one representative item per module keeps these as a
//! compile-time contract that the module paths stay stable.
use super::content::ALIGNED_UNIT_LEN;
use super::inf::{disc_hash, disc_hash_hex};
use super::mkb::{AacsVersion, mkb_content_len, walk_mkb};
use super::variant::is_variant_mkb;
#[test]
fn aligned_unit_len_is_three_2048_byte_sectors() {
// ALIGNED_UNIT_LEN is the AACS aligned-unit size: 3 × 2048 = 6144.
// Re-exported from decrypt; pin the value here so the public constant
// and the spec stay in lockstep.
assert_eq!(ALIGNED_UNIT_LEN, 6144);
assert_eq!(ALIGNED_UNIT_LEN, 3 * 2048);
}
#[test]
fn version_strides_are_reexported_and_distinct() {
// The three AACS generations are part of the public surface, and the
// V10 (48) vs V20/V21 (64) stride distinction is the load-bearing
// difference. Confirm the enum re-export is usable and the variants
// are distinct values.
assert_ne!(AacsVersion::V10, AacsVersion::V20);
assert_ne!(AacsVersion::V20, AacsVersion::V21);
}
#[test]
fn public_helpers_are_callable_by_module_path() {
// Touch a representative function from each module so a dropped/renamed
// item fails to compile. Smoke calls, not behavioural assertions.
let _ = !crate::aacs::content::is_clean(
&[0u8; ALIGNED_UNIT_LEN],
crate::disc::ContentFormat::BdTs,
);
let _ = mkb_content_len(&[]);
let _ = is_variant_mkb(&walk_mkb(&[]));
let _ = disc_hash_hex(&disc_hash(b"x"));
let _ = super::derive::resolve_candidate(
&super::derive::KeyCandidate::Uk(super::types::UnitKey::new(0, [0u8; 16])),
&[],
&[],
None,
);
}
}
+409
View File
@@ -0,0 +1,409 @@
//! Key source abstraction for the AACS resolve chain.
//!
//! libfreemkv keeps all crypto (AES-G primitives, SD-tree walking,
//! validation, MK/VUK/TK derivation) but accepts key material from
//! arbitrary backends via [`KeyProvider`].
//!
//! Methods come in two flavors:
//!
//! - **Bulk material** ([`device_keys`], [`processing_keys`],
//! [`media_keys`]) — the resolver unions (and dedups) results
//! across all providers and tries each candidate.
//! - **Disc-keyed lookup** ([`lookup_disc_by_hash`],
//! [`lookup_disc_by_vid`]) — the resolver short-circuits on the
//! first hit, so providers are queried in array order with
//! fastest/closest first.
//!
//! [`host_certs`] is a sixth method but is NOT consumed by the
//! resolver chain: the SCSI handshake reads host certs directly from
//! the caller-supplied credentials, not from the provider array. A
//! provider that overrides `host_certs` today has no effect on the
//! handshake; the method is retained as a forward-looking extension
//! point only.
//!
//! Default impls return empty / `None` so backends only override
//! the methods they actually support — an external key service might
//! implement only `lookup_disc_by_hash`, while a local file might
//! implement all six.
//!
//! Calls may block (disk I/O, network round-trips). The resolver
//! invokes each method at most a handful of times per scan; for
//! per-disc memoization, implementations should cache internally.
//!
//! [`device_keys`]: KeyProvider::device_keys
//! [`processing_keys`]: KeyProvider::processing_keys
//! [`media_keys`]: KeyProvider::media_keys
//! [`host_certs`]: KeyProvider::host_certs
//! [`lookup_disc_by_hash`]: KeyProvider::lookup_disc_by_hash
//! [`lookup_disc_by_vid`]: KeyProvider::lookup_disc_by_vid
use super::types::{DeviceKey, DiscEntry, HostCert};
/// Source of AACS key material.
///
/// Implementors return raw material only — the resolver in
/// `aacs::resolve` and `aacs::derive` own the crypto (DK→PK walking, PK validation,
/// MK→VUK→TK derivation). See module docs for method semantics.
pub trait KeyProvider: Send + Sync {
/// Device keys (top-of-tree, walked by the resolver).
fn device_keys(&self) -> Vec<DeviceKey> {
Vec::new()
}
/// Processing keys — terminal PKs or walk-input PKs. The
/// resolver tries each as a terminal first (cheap validate).
fn processing_keys(&self) -> Vec<[u8; 16]> {
Vec::new()
}
/// Every Media Key this provider holds, regardless of which disc it was
/// filed under. An MK is MKB-scoped (shared across a pressing/MKB-family),
/// so the resolver can verify each against the disc's MKB (`km_verifies`)
/// and resolve a disc whose own hash/VID isn't directly keyed.
fn media_keys(&self) -> Vec<[u8; 16]> {
Vec::new()
}
/// AACS host certificates (with their private keys) for drive
/// authentication. Multiple in case some are revoked.
///
/// NOTE: not consumed by the resolver chain — the handshake reads
/// host certs from the caller-supplied credentials directly, so
/// overriding this method has no effect on drive authentication
/// today. Retained as a forward-looking extension point.
fn host_certs(&self) -> Vec<HostCert> {
Vec::new()
}
/// Direct per-disc lookup by SHA-1 of `Unit_Key_RO.inf`. Returns
/// `Some(entry)` if this provider has pre-computed material for
/// the disc (paths 4 and 5). Short-circuits the resolver.
fn lookup_disc_by_hash(&self, _disc_hash: &[u8; 20]) -> Option<DiscEntry> {
None
}
/// Lookup by Volume ID (path 3 — pre-computed MK + matching
/// VID). Short-circuits the resolver on hit.
fn lookup_disc_by_vid(&self, _volume_id: &[u8; 16]) -> Option<DiscEntry> {
None
}
}
/// Resolver-side helpers that aggregate across a provider array.
///
/// The resolver wraps `ctx.providers` (`&[&dyn KeyProvider]`) in this
/// struct; these helpers apply the union-vs-short-circuit policy per
/// method. The bulk unions dedup so overlapping providers don't make
/// the resolver re-walk/re-validate identical material.
pub(crate) struct Providers<'a>(pub &'a [&'a dyn KeyProvider]);
impl Providers<'_> {
/// Union (deduped) — gather DKs from every provider.
pub fn device_keys(&self) -> Vec<DeviceKey> {
let mut v: Vec<DeviceKey> = self.0.iter().flat_map(|p| p.device_keys()).collect();
// DeviceKey has no Ord/Hash; dedup on the value-defining tuple.
v.sort_unstable_by_key(|d| (d.key, d.node, d.uv, d.u_mask_shift));
v.dedup_by_key(|d| (d.key, d.node, d.uv, d.u_mask_shift));
v
}
/// Union (deduped) — gather PKs from every provider.
pub fn processing_keys(&self) -> Vec<[u8; 16]> {
let mut v: Vec<[u8; 16]> = self.0.iter().flat_map(|p| p.processing_keys()).collect();
v.sort_unstable();
v.dedup();
v
}
/// Union of distinct Media Keys across every provider, for the MK-pool
/// brute (`km_verifies` against the disc's MKB).
pub fn media_keys(&self) -> Vec<[u8; 16]> {
let mut v: Vec<[u8; 16]> = self.0.iter().flat_map(|p| p.media_keys()).collect();
v.sort_unstable();
v.dedup();
v
}
/// Union — gather host certs from every provider. The SCSI handshake
/// reads host certs from the caller-supplied credentials directly and
/// does not call this, so it is currently unused by the resolver chain.
#[allow(dead_code)]
pub fn host_certs(&self) -> Vec<HostCert> {
self.0.iter().flat_map(|p| p.host_certs()).collect()
}
/// Short-circuit — query providers in array order, first hit wins.
pub fn lookup_disc_by_hash(&self, disc_hash: &[u8; 20]) -> Option<DiscEntry> {
self.0.iter().find_map(|p| p.lookup_disc_by_hash(disc_hash))
}
/// Short-circuit — query providers in array order, first hit wins.
pub fn lookup_disc_by_vid(&self, volume_id: &[u8; 16]) -> Option<DiscEntry> {
self.0.iter().find_map(|p| p.lookup_disc_by_vid(volume_id))
}
}
/// A [`KeyProvider`] backed by a single caller-supplied key's raw material —
/// the bridge for [`crate::disc::Disc::decrypt_with`].
///
/// The application's key source did the lookup and handed in material at one
/// level (DK / PK / MK / VUK). This exposes exactly that material to the
/// version-dispatched resolver, which owns ALL derivation — so a source never
/// derives, and the lib remains the single home for the AACS chain across
/// 1.0 / 2.0 / 2.1 / 2.x.
///
/// Each level fills only its own field; the rest stay empty, so the resolver
/// naturally runs the matching path (DK→…, PK→…, MK-pool brute, or a
/// disc-keyed VUK hit). `decrypt_with` already knows the disc, so the
/// `lookup_disc_by_*` hash/VID arguments are irrelevant — a present
/// `disc_entry` is returned for any query.
pub(crate) struct SuppliedKey {
pub device_keys: Vec<DeviceKey>,
pub processing_keys: Vec<[u8; 16]>,
pub media_keys: Vec<[u8; 16]>,
pub disc_entry: Option<DiscEntry>,
}
impl KeyProvider for SuppliedKey {
fn device_keys(&self) -> Vec<DeviceKey> {
self.device_keys.clone()
}
fn processing_keys(&self) -> Vec<[u8; 16]> {
self.processing_keys.clone()
}
fn media_keys(&self) -> Vec<[u8; 16]> {
self.media_keys.clone()
}
fn lookup_disc_by_hash(&self, _disc_hash: &[u8; 20]) -> Option<DiscEntry> {
self.disc_entry.clone()
}
fn lookup_disc_by_vid(&self, _volume_id: &[u8; 16]) -> Option<DiscEntry> {
self.disc_entry.clone()
}
}
#[cfg(test)]
mod tests {
use super::*;
fn entry(hash: &str, vuk: u8) -> DiscEntry {
DiscEntry {
disc_hash: hash.to_string(),
title: "t".to_string(),
media_key: None,
disc_id: None,
vuk: Some([vuk; 16]),
unit_keys: Vec::new(),
}
}
fn dk(byte: u8, node: u16) -> DeviceKey {
DeviceKey {
key: [byte; 16],
node,
uv: 1,
u_mask_shift: 0,
}
}
/// A provider that returns fixed bulk material and an optional disc entry
/// keyed unconditionally (used to test array-order short-circuiting).
#[derive(Default)]
struct Fixed {
dks: Vec<DeviceKey>,
pks: Vec<[u8; 16]>,
mks: Vec<[u8; 16]>,
hash_hit: Option<DiscEntry>,
vid_hit: Option<DiscEntry>,
}
impl KeyProvider for Fixed {
fn device_keys(&self) -> Vec<DeviceKey> {
self.dks.clone()
}
fn processing_keys(&self) -> Vec<[u8; 16]> {
self.pks.clone()
}
fn media_keys(&self) -> Vec<[u8; 16]> {
self.mks.clone()
}
fn lookup_disc_by_hash(&self, _h: &[u8; 20]) -> Option<DiscEntry> {
self.hash_hit.clone()
}
fn lookup_disc_by_vid(&self, _v: &[u8; 16]) -> Option<DiscEntry> {
self.vid_hit.clone()
}
}
// ── KeyProvider default methods all return empty ───────────────────────
#[test]
fn default_provider_methods_return_empty() {
// A bare provider that overrides nothing must yield empty material so
// the resolver simply finds nothing through it (no surprise hits).
struct Empty;
impl KeyProvider for Empty {}
let e = Empty;
assert!(e.device_keys().is_empty());
assert!(e.processing_keys().is_empty());
assert!(e.media_keys().is_empty());
assert!(e.host_certs().is_empty());
assert!(e.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(e.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
// ── Providers::processing_keys: union + dedup ──────────────────────────
#[test]
fn providers_processing_keys_union_and_dedup() {
// Two providers each carrying overlapping PKs → the aggregate is the
// deduped union (the resolver must not re-validate identical material).
let a = Fixed {
pks: vec![[0x01u8; 16], [0x02u8; 16]],
..Default::default()
};
let b = Fixed {
pks: vec![[0x02u8; 16], [0x03u8; 16]],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let mut got = Providers(arr).processing_keys();
got.sort();
assert_eq!(got, vec![[0x01u8; 16], [0x02u8; 16], [0x03u8; 16]]);
}
#[test]
fn providers_media_keys_union_and_dedup() {
let a = Fixed {
mks: vec![[0xAAu8; 16]],
..Default::default()
};
let b = Fixed {
mks: vec![[0xAAu8; 16], [0xBBu8; 16]],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let mut got = Providers(arr).media_keys();
got.sort();
assert_eq!(got, vec![[0xAAu8; 16], [0xBBu8; 16]]);
}
#[test]
fn providers_device_keys_dedup_on_value_tuple() {
// DeviceKey has no Hash/Ord; dedup keys on (key,node,uv,u_mask_shift).
// Two identical DKs across providers collapse to one; a DK differing
// only in node is kept.
let a = Fixed {
dks: vec![dk(0x11, 5), dk(0x11, 5)],
..Default::default()
};
let b = Fixed {
dks: vec![dk(0x11, 5), dk(0x11, 6)],
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).device_keys();
assert_eq!(got.len(), 2, "identical DKs dedup; differing node kept");
let nodes: Vec<u16> = got.iter().map(|d| d.node).collect();
assert!(nodes.contains(&5) && nodes.contains(&6));
}
// ── Disc-keyed lookups: array-order short-circuit ──────────────────────
#[test]
fn providers_lookup_by_hash_first_hit_wins() {
// Querying providers in array order, the FIRST hit wins (closest /
// fastest first). Provider 0 hits → its entry is returned even though
// provider 1 also has one.
let a = Fixed {
hash_hit: Some(entry("first", 0x01)),
..Default::default()
};
let b = Fixed {
hash_hit: Some(entry("second", 0x02)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_hash(&[0u8; 20]).unwrap();
assert_eq!(got.disc_hash, "first");
assert_eq!(got.vuk, Some([0x01u8; 16]));
}
#[test]
fn providers_lookup_by_hash_falls_through_to_later_provider() {
// Provider 0 misses, provider 1 hits → the later provider's entry is
// used (find_map continues past None).
let a = Fixed::default(); // hash_hit None
let b = Fixed {
hash_hit: Some(entry("second", 0x02)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_hash(&[0u8; 20]).unwrap();
assert_eq!(got.disc_hash, "second");
}
#[test]
fn providers_lookup_by_vid_first_hit_wins() {
let a = Fixed {
vid_hit: Some(entry("vid-a", 0x07)),
..Default::default()
};
let b = Fixed {
vid_hit: Some(entry("vid-b", 0x08)),
..Default::default()
};
let arr: &[&dyn KeyProvider] = &[&a, &b];
let got = Providers(arr).lookup_disc_by_vid(&[0u8; 16]).unwrap();
assert_eq!(got.disc_hash, "vid-a");
}
#[test]
fn providers_empty_array_yields_nothing() {
let arr: &[&dyn KeyProvider] = &[];
let p = Providers(arr);
assert!(p.device_keys().is_empty());
assert!(p.processing_keys().is_empty());
assert!(p.media_keys().is_empty());
assert!(p.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(p.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
// ── SuppliedKey: each level exposes only its own material ──────────────
#[test]
fn supplied_key_exposes_only_populated_fields() {
// A SuppliedKey filled at the DK level exposes DKs and nothing else,
// so the resolver runs the matching (DK→…) path and no other.
let sk = SuppliedKey {
device_keys: vec![dk(0x33, 9)],
processing_keys: Vec::new(),
media_keys: Vec::new(),
disc_entry: None,
};
assert_eq!(sk.device_keys().len(), 1);
assert!(sk.processing_keys().is_empty());
assert!(sk.media_keys().is_empty());
assert!(sk.lookup_disc_by_hash(&[0u8; 20]).is_none());
assert!(sk.lookup_disc_by_vid(&[0u8; 16]).is_none());
}
#[test]
fn supplied_key_disc_entry_returned_for_any_hash_or_vid() {
// decrypt_with already knows the disc, so a present disc_entry is
// returned regardless of the hash/VID argument (the lookup args are
// irrelevant in this bridge).
let sk = SuppliedKey {
device_keys: Vec::new(),
processing_keys: Vec::new(),
media_keys: Vec::new(),
disc_entry: Some(entry("supplied", 0x44)),
};
// Two unrelated hashes both return the same entry.
let h1 = sk.lookup_disc_by_hash(&[0x01u8; 20]).unwrap();
let h2 = sk.lookup_disc_by_hash(&[0xFFu8; 20]).unwrap();
assert_eq!(h1.disc_hash, "supplied");
assert_eq!(h2.disc_hash, "supplied");
// And by VID likewise.
assert!(sk.lookup_disc_by_vid(&[0x00u8; 16]).is_some());
}
}
+2084
View File
File diff suppressed because it is too large Load Diff
+438
View File
@@ -0,0 +1,438 @@
//! AACS 2.1 FMTS forensic segment map — `AACS/IndividualSegment.tbl`.
//!
//! An FMTS main feature interleaves short forensic **segments** — the sequence-key
//! / forensic-watermark mechanism. Each segment carries an **index** (1..32): a
//! tag in `IndividualSegment.tbl` that selects which of the 32 forensic **index
//! keys** decrypts that segment's units, in place of the ordinary CPS Unit Key.
//!
//! Terminology (see the project AACS reference): the **index** here is NOT the
//! AACS 2.1 *Media Key Variant* — that is the 65536-value device selector in the
//! MKB that decides *which set* of index keys a device receives, a layer this
//! module does not deal with. All the index keys belong to one variant, whose
//! number is unknown and irrelevant to the segment map. Decrypting a segment with
//! the Unit Key yields garbage — broken HEVC reference frames (empirically:
//! `Could not find ref with POC …` on a plain unit-key rip).
//!
//! This table says WHERE the segments live and which index each carries, so a
//! decoder can decrypt them with the matching index key instead of muxing
//! unit-key garbage.
//!
//! Format (validated against a retail AACS 2.1 disc):
//! ```text
//! header (8 bytes): u32 type | u16 count | u16 record_size (= 16)
//! record[count] (16 bytes each):
//! u32 marker (= 0x01000000) | u16 index | u16 flag (= 1)
//! u32 start_spn | u32 end_spn (source-packet numbers, inclusive)
//! ```
//! `index` is the 1..32 forensic index tag, NOT a sequential segment id: measured
//! on a retail 2.1 disc (Zombieland) it cycles 1,2,…,32,1,2,… across records in
//! file order — 24 full cycles of 32 plus a final partial cycle of 24 = 792
//! records. Source-packet numbers are the 192-byte BDAV packet index: byte offset
//! = `spn * 192`. Each segment is ~2560 packets (~480 KB) = 80 aligned units,
//! spread across the entire 54 GB feature (one roughly every 67 MB). Inside a
//! segment the 80 units interleave in two stride-2 halves: applying the segment's
//! index key decrypts ~40 of them to clean TS and garbles the other ~40 (a second
//! interleaved half, unidentified), which the demux then drops — leaving one
//! coherent stream. Confirmed by decoding a retail disc with a full set of 32
//! index keys.
/// Fixed size of one `IndividualSegment.tbl` record.
pub const SEGMENT_RECORD_LEN: usize = 16;
/// Bytes per BDAV source packet (188-byte TS + 4-byte arrival-time header).
pub const SOURCE_PACKET_LEN: u64 = 192;
/// Whether a 2.1 (FMTS) disc may rip WITHOUT the forensic index keys.
///
/// `true` (today): the forensic segments are skipped as expected loss
/// and the bulk of the title decodes with the unit key, so a 2.1 disc rips
/// mostly-complete. A unit key (VUK) is still required, exactly as for any AACS
/// disc. `false`: the absence of a segment-key source is a hard, UPFRONT failure
/// ([`Error::FmtsKeyMissing`]) — the same policy as a missing unit key, so a
/// forensic-holed rip is refused rather than produced. No segment-key source
/// exists yet, so `true` is the only value under which a 2.1 disc rips at all;
/// flip to `false` once segment keys can be sourced and a partial rip should be
/// refused. Hardcoded on purpose — not a user setting.
///
/// [`Error::FmtsKeyMissing`]: crate::error::Error::FmtsKeyMissing
pub const BYPASS_FMTS_KEY: bool = false;
/// One forensic segment: the inclusive source-packet range it occupies in the
/// FMTS clip.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Segment {
/// Forensic index tag, 1..=32 (field@4 of the record). Cycles across the
/// table rather than counting up — it selects WHICH of the 32 index keys
/// decrypts this range. (`0` is not used here; the default/non-forensic
/// content carries no segment record at all.)
pub index: u16,
/// First source packet of the segment (inclusive).
pub start_spn: u32,
/// Last source packet of the segment (inclusive).
pub end_spn: u32,
}
impl Segment {
/// Source-packet count in this (inclusive) segment.
pub fn packet_count(&self) -> u32 {
self.end_spn
.saturating_sub(self.start_spn)
.saturating_add(1)
}
/// Byte offset of the segment start within the clip (`start_spn * 192`).
pub fn start_byte(&self) -> u64 {
self.start_spn as u64 * SOURCE_PACKET_LEN
}
/// Byte length of the segment (`packet_count * 192`).
pub fn byte_len(&self) -> u64 {
self.packet_count() as u64 * SOURCE_PACKET_LEN
}
/// True when source packet `spn` falls inside this segment.
pub fn contains_spn(&self, spn: u32) -> bool {
spn >= self.start_spn && spn <= self.end_spn
}
/// True when the inclusive source-packet span `[first, last]` overlaps this
/// segment. Used to decide whether an aligned unit (which spans several
/// packets) touches the segment at all, not just whether one packet does.
pub fn overlaps_spn(&self, first: u32, last: u32) -> bool {
first <= self.end_spn && last >= self.start_spn
}
}
/// Source packets spanned by one AACS aligned unit: `6144 / 192 = 32`.
pub const PACKETS_PER_UNIT: u32 =
(crate::aacs::content::ALIGNED_UNIT_LEN as u64 / SOURCE_PACKET_LEN) as u32;
/// Byte offset within the clip of a clip-relative 2048-byte sector `lba`. The
/// FMTS decode reads the clip file directly, so `lba` 0 is the clip's first
/// byte and this offset lines up with the source-packet grid the segment map
/// uses.
pub fn lba_byte_offset(lba: u32) -> u64 {
lba as u64 * 2048
}
/// The forensic segment an AACS aligned unit belongs to, if any, given the
/// unit's clip-relative byte offset.
///
/// This is the routing decision behind a 2.1 decrypt-miss: a unit that
/// overlaps a forensic segment must be opened with that segment's **index key**
/// (selected by the segment's `index`), not the CPS Unit Key. Opening it with
/// the Unit Key is exactly what yields the broken-reference-frame garbage a
/// plain unit-key rip produces. A unit outside every segment is ordinary
/// content and a miss on it is a Unit-Key miss, so this returns `None` and the
/// caller falls back to the normal unit-key fetch.
///
/// The unit is tested as a packet *span* (`[off/192, (off+6144-1)/192]`) so a
/// unit that only partly overlaps a segment edge is still classified as
/// forensic; on the observed disc segments are unit-aligned, but the span test
/// does not rely on that.
pub fn segment_for_unit(segments: &[Segment], unit_offset: u64) -> Option<&Segment> {
let unit_len = crate::aacs::content::ALIGNED_UNIT_LEN as u64;
let first = (unit_offset / SOURCE_PACKET_LEN) as u32;
let last = ((unit_offset + unit_len - 1) / SOURCE_PACKET_LEN) as u32;
segments.iter().find(|s| s.overlaps_spn(first, last))
}
/// Parse `IndividualSegment.tbl` into its forensic segments, in table
/// order. Returns `None` when the header is malformed, the record size is not
/// [`SEGMENT_RECORD_LEN`], or the declared record count overruns the buffer —
/// so a truncated / foreign table degrades to "no segment map" rather than
/// yielding bogus ranges.
pub fn parse_individual_segments(tbl: &[u8]) -> Option<Vec<Segment>> {
if tbl.len() < 8 {
return None;
}
let count = u16::from_be_bytes([tbl[4], tbl[5]]) as usize;
let record_size = u16::from_be_bytes([tbl[6], tbl[7]]) as usize;
if record_size != SEGMENT_RECORD_LEN {
return None;
}
if 8usize.checked_add(count.checked_mul(record_size)?)? > tbl.len() {
return None;
}
let mut segments = Vec::with_capacity(count);
for i in 0..count {
let o = 8 + i * record_size;
// o+4..o+8 = index (u16, 1..32) + flag (u16); o+8..o+16 = start/end SPN.
let index = u16::from_be_bytes([tbl[o + 4], tbl[o + 5]]);
let start_spn = u32::from_be_bytes([tbl[o + 8], tbl[o + 9], tbl[o + 10], tbl[o + 11]]);
let end_spn = u32::from_be_bytes([tbl[o + 12], tbl[o + 13], tbl[o + 14], tbl[o + 15]]);
segments.push(Segment {
index,
start_spn,
end_spn,
});
}
Some(segments)
}
/// Map a clip-relative byte offset to the absolute LBA that holds it, by walking
/// the title's extents (the `.fmts` clip's sectors in file order). Segment
/// offsets in [`Segment`] are clip-relative source-packet numbers, so this is how
/// a segment's `spn` range becomes disc LBAs. `None` if the offset is past the
/// clip.
pub fn clip_byte_to_lba(extents: &[crate::disc::Extent], clip_byte: u64) -> Option<u32> {
let mut cum = 0u64;
for e in extents {
let len = e.sector_count as u64 * crate::consts::SECTOR_BYTES as u64;
if clip_byte < cum + len {
let sector_in_ext = ((clip_byte - cum) / crate::consts::SECTOR_BYTES as u64) as u32;
return Some(e.start_lba.saturating_add(sector_in_ext));
}
cum += len;
}
None
}
/// Build the `[start_lba, end_lba) → key_idx` ranges for an FMTS forensic key map.
///
/// Each forensic segment's clip-relative source-packet span becomes an absolute
/// LBA range tagged with the key its `index` selects (via `index_to_key_idx`,
/// e.g. `|i| i as usize` when the pool is `[base, idx1, idx2, …]`). Applying that
/// one key across the whole segment decodes the ~40 units of its interleave half
/// to clean TS and garbles the other ~40 (the second interleaved half), which the
/// demux then drops — yielding one coherent stream. Ranges outside every segment
/// are left for the map's default (the ordinary Unit Key). A segment that straddles
/// a UDF extent boundary is emitted as one range per whole-sector slice it covers.
///
/// The result feeds [`AacsKeyMap::from_ranges`](crate::decrypt::AacsKeyMap::from_ranges)
/// with the Unit-Key index as the default — the same structure the CPS map uses,
/// only finer-grained.
pub fn fmts_key_ranges(
segments: &[Segment],
extents: &[crate::disc::Extent],
index_to_key_idx: &dyn Fn(u16) -> usize,
) -> Vec<(u32, u32, usize)> {
let mut ranges = Vec::new();
for s in segments {
let start_byte = s.start_spn as u64 * SOURCE_PACKET_LEN;
let end_byte = (s.end_spn as u64 + 1) * SOURCE_PACKET_LEN; // exclusive
// A segment is unit-aligned and contiguous in clip bytes; map its first
// and last sector to LBAs. Segments are ~480 KB and extents are GB-sized,
// so a segment almost never crosses an extent boundary — but if the two
// ends land in different extents (non-contiguous LBAs), skip rather than
// emit a wrong span; the units there fall to the Unit Key (garble+drop),
// never a mis-decrypt.
let (Some(a), Some(b)) = (
clip_byte_to_lba(extents, start_byte),
clip_byte_to_lba(extents, end_byte - 1),
) else {
continue;
};
if b >= a
&& (b - a) as u64 == (end_byte - 1 - start_byte) / crate::consts::SECTOR_BYTES as u64
{
ranges.push((a, b + 1, index_to_key_idx(s.index)));
}
}
ranges
}
#[cfg(test)]
mod tests {
use super::*;
/// Build a table with the real on-disc layout: 8-byte header + N 16-byte
/// records. `recs` are `(index, start_spn, end_spn)`.
fn build_tbl(recs: &[(u16, u32, u32)]) -> Vec<u8> {
let mut v = Vec::new();
v.extend_from_slice(&0x0100_0000u32.to_be_bytes()); // type
v.extend_from_slice(&(recs.len() as u16).to_be_bytes()); // count
v.extend_from_slice(&(SEGMENT_RECORD_LEN as u16).to_be_bytes()); // record_size
for &(n, s, e) in recs {
v.extend_from_slice(&0x0100_0000u32.to_be_bytes()); // marker
v.extend_from_slice(&n.to_be_bytes());
v.extend_from_slice(&1u16.to_be_bytes()); // flag
v.extend_from_slice(&s.to_be_bytes());
v.extend_from_slice(&e.to_be_bytes());
}
v
}
#[test]
fn fmts_key_ranges_maps_segments_to_lba_by_index() {
use crate::disc::Extent;
// One big clip extent starting at LBA 1000. Clip byte B lives at
// LBA 1000 + B/2048.
let extents = vec![Extent {
start_lba: 1000,
sector_count: 1_000_000,
}];
// Two segments, indexes 5 and 7 (spn ranges as on a real disc).
let segs = vec![
Segment {
index: 5,
start_spn: 100,
end_spn: 199,
},
Segment {
index: 7,
start_spn: 10_000,
end_spn: 10_099,
},
];
// Pool layout [base, idx1, idx2, …] → index N uses key slot N.
let ranges = fmts_key_ranges(&segs, &extents, &|v| v as usize);
assert_eq!(ranges.len(), 2, "one LBA range per segment");
// Segment 0: spn 100..=199 → clip bytes [19200, 38400) → sectors 9..=18
// → LBA 1009..1019, key index 5.
assert_eq!(ranges[0], (1009, 1019, 5));
// Segment 1: spn 10000..=10099 → bytes [1_920_000, 1_939_200) →
// sectors 937..=946 → LBA 1937..1947, key index 7.
assert_eq!(ranges[1], (1937, 1947, 7));
// The ranges drive an AacsKeyMap with the Unit Key (index 0) as default.
let map = crate::decrypt::AacsKeyMap::from_ranges(ranges, 0);
assert_eq!(map.key_idx_for(500), 0, "outside any segment → Unit Key");
assert_eq!(map.key_idx_for(1012), 5, "inside index-5 segment → key 5");
assert_eq!(map.key_idx_for(1940), 7, "inside index-7 segment → key 7");
assert_eq!(
map.key_idx_for(1019),
0,
"segment end is exclusive → Unit Key"
);
}
#[test]
fn clip_byte_to_lba_walks_extents() {
use crate::disc::Extent;
let extents = vec![
Extent {
start_lba: 100,
sector_count: 10,
}, // clip bytes [0, 20480)
Extent {
start_lba: 500,
sector_count: 10,
}, // clip bytes [20480, 40960)
];
assert_eq!(clip_byte_to_lba(&extents, 0), Some(100));
assert_eq!(clip_byte_to_lba(&extents, 2048), Some(101));
assert_eq!(clip_byte_to_lba(&extents, 20480), Some(500)); // second extent
assert_eq!(clip_byte_to_lba(&extents, 22528), Some(501));
assert_eq!(clip_byte_to_lba(&extents, 40960), None); // past the clip
}
#[test]
fn parses_real_disc_layout() {
// First three records observed on retail 2.1 (Zombieland): the variant
// field counts 1,2,3,… (it wraps at 32 further into the table — see
// `index_field_cycles_one_to_thirty_two`), segments are 2560 packets.
let tbl = build_tbl(&[
(1, 343680, 346239),
(2, 695616, 698175),
(3, 1051840, 1054399),
]);
let segs = parse_individual_segments(&tbl).expect("parse");
assert_eq!(segs.len(), 3);
assert_eq!(segs[0].index, 1);
assert_eq!(segs[1].index, 2);
assert_eq!(segs[2].index, 3);
assert_eq!(segs[0].start_spn, 343680);
assert_eq!(segs[0].end_spn, 346239);
assert_eq!(segs[0].packet_count(), 2560);
assert_eq!(segs[0].byte_len(), 2560 * 192);
assert_eq!(segs[0].start_byte(), 343680 * 192);
assert!(segs[0].contains_spn(345000));
assert!(!segs[0].contains_spn(343679));
assert!(!segs[0].contains_spn(346240));
}
#[test]
fn rejects_wrong_record_size() {
let mut tbl = build_tbl(&[(1, 0, 10)]);
tbl[6..8].copy_from_slice(&20u16.to_be_bytes()); // record_size != 16
assert!(parse_individual_segments(&tbl).is_none());
}
#[test]
fn rejects_truncated_and_overrun() {
assert!(parse_individual_segments(&[0u8; 4]).is_none()); // < header
let mut tbl = build_tbl(&[(1, 0, 10)]);
tbl[4..6].copy_from_slice(&99u16.to_be_bytes()); // claims 99 recs, has 1
assert!(parse_individual_segments(&tbl).is_none());
}
#[test]
fn empty_table_is_empty_not_none() {
let tbl = build_tbl(&[]);
assert_eq!(parse_individual_segments(&tbl), Some(Vec::new()));
}
#[test]
fn packets_per_unit_is_thirty_two() {
// 6144-byte aligned unit / 192-byte source packet.
assert_eq!(PACKETS_PER_UNIT, 32);
}
#[test]
fn unit_inside_segment_routes_to_index() {
// A real first-record segment: packets [343680, 346239].
let segs = parse_individual_segments(&build_tbl(&[(1, 343680, 346239)])).unwrap();
// A unit sitting squarely inside: start at packet 344000 → byte 344000*192.
let off = 344000u64 * SOURCE_PACKET_LEN;
let hit = segment_for_unit(&segs, off).expect("inside the segment");
assert_eq!(hit.index, 1);
}
#[test]
fn index_field_cycles_one_to_thirty_two() {
// Reality on Zombieland: field@4 is the index, cycling 1..=32 in file
// order (NOT a sequential segment id). Reproduce one-and-a-bit cycles.
let mut recs = Vec::new();
let mut spn = 1000u32;
for row in 0..2 {
for v in 1..=32u16 {
recs.push((v, spn, spn + 2559));
spn += 50_000; // ~one segment every ~67 MB
}
let _ = row;
}
let segs = parse_individual_segments(&build_tbl(&recs)).unwrap();
assert_eq!(segs.len(), 64);
assert_eq!(segs[31].index, 32); // end of first cycle
assert_eq!(segs[32].index, 1); // wraps, does not become 33
assert!(segs.iter().all(|s| (1..=32).contains(&s.index)));
}
#[test]
fn unit_outside_every_segment_is_unit_key_miss() {
let segs = parse_individual_segments(&build_tbl(&[(1, 343680, 346239)])).unwrap();
// A unit well before the segment is ordinary content → None (unit-key path).
let off = 1000u64 * SOURCE_PACKET_LEN;
assert!(segment_for_unit(&segs, off).is_none());
}
#[test]
fn unit_straddling_a_segment_edge_counts_as_forensic() {
// Segment starts at packet 100. A unit that ENDS just inside it (its 32
// packets straddle the boundary) must still route to the index key,
// because part of its ciphertext is forensic-encrypted.
let segs = parse_individual_segments(&build_tbl(&[(7, 100, 200)])).unwrap();
// Unit covering packets [80, 111]: overlaps [100,200] at the tail.
let off = 80u64 * SOURCE_PACKET_LEN;
let hit = segment_for_unit(&segs, off).expect("straddles the start edge");
assert_eq!(hit.index, 7);
// A unit ending exactly at packet 99 (offset s.t. last = 99) does NOT overlap.
let before = 68u64 * SOURCE_PACKET_LEN; // [68, 99]
assert!(segment_for_unit(&segs, before).is_none());
}
#[test]
fn no_segments_never_routes_to_index() {
// The 1.0 / 2.0 case: no forensic map, so every miss is a unit-key miss.
assert!(segment_for_unit(&[], lba_byte_offset(0)).is_none());
assert!(segment_for_unit(&[], lba_byte_offset(9_999_999)).is_none());
}
#[test]
fn lba_maps_to_the_packet_grid() {
// A unit is 3 sectors (6144 bytes) = 32 packets. Clip-relative LBA 3 is
// the second aligned unit, which starts at packet 32.
let off = lba_byte_offset(3);
assert_eq!(off / SOURCE_PACKET_LEN, 32);
}
}
+165
View File
@@ -0,0 +1,165 @@
//! AACS 2.1 FMTS forensic segment keys, `AACS/SegmentKeyNNNNN.tbl`.
//!
//! One file per CPS unit (`SegmentKey00001.tbl`, ...). It is the on-disc key
//! store for the forensic variant segments mapped by [`super::segment`]. A
//! device does not read a segment key directly. It derives a **16-bit variant
//! selector** from the Media Key Variant chain (see [`super::variant`]) and uses
//! that selector to index this table, which is how the device's position in the
//! key tree decides which variant it can decrypt (the traitor-tracing link).
//!
//! Container format (confirmed against a retail AACS 2.1 disc):
//! ```text
//! header (8 bytes): u32 tag | u16 index_space | u16 record_size
//! record[index_space] (record_size bytes each)
//! ```
//! On the reference disc: `index_space` = `0xffff` (the full 16-bit selector
//! space, 65536 records), `record_size` = `0x0218` = 536. Total
//! `8 + 65536 * 536 = 35,127,304` bytes, which matches the file exactly. Each
//! record begins with an 8-byte sub-header, then 528 bytes of encrypted key
//! material.
//!
//! **Not yet reversed:** the internal layout of a record's 528-byte payload, and
//! how it maps onto the segments of [`super::segment`]. One numeric coincidence
//! worth noting for whoever cracks it: the reference disc has 792 segments and
//! `528 = 33 * 16`, with `792 = 24 * 33`, so `33` appears on both sides. Until
//! the mapping and the key derivation are pinned, this module exposes only the
//! confirmed container: locate the record for a given 16-bit selector.
/// Bytes of the fixed file header.
pub const HEADER_LEN: usize = 8;
/// The on-disc segment-key table container. Borrows the file bytes; a record is
/// looked up by the 16-bit variant selector.
#[derive(Debug, Clone, Copy)]
pub struct SegmentKeyTable<'a> {
data: &'a [u8],
/// Number of records (the selector index space, e.g. 65536).
count: usize,
/// Bytes per record (e.g. 536).
record_size: usize,
}
impl<'a> SegmentKeyTable<'a> {
/// Parse and validate the container header against the buffer length.
///
/// Returns `None` when the buffer is too small, or the declared
/// `count * record_size` (plus header) does not match the buffer, so a
/// truncated or foreign table degrades to "no segment keys" rather than
/// handing back bogus records. `index_space` of `0xffff` is read as the full
/// 65536-entry space (a device selector is a full 16-bit value).
pub fn parse(data: &'a [u8]) -> Option<Self> {
if data.len() < HEADER_LEN {
return None;
}
let index_space = u16::from_be_bytes([data[4], data[5]]);
let record_size = u16::from_be_bytes([data[6], data[7]]) as usize;
// 0xffff means the full 16-bit selector space (65536 records).
let count = if index_space == 0xffff {
0x1_0000
} else {
index_space as usize
};
if record_size == 0 {
return None;
}
let body = count.checked_mul(record_size)?;
if HEADER_LEN.checked_add(body)? != data.len() {
return None;
}
Some(Self {
data,
count,
record_size,
})
}
/// Number of records (the selector index space).
pub fn record_count(&self) -> usize {
self.count
}
/// Bytes per record.
pub fn record_size(&self) -> usize {
self.record_size
}
/// The raw record for a 16-bit variant `selector`, including its 8-byte
/// sub-header. `None` if the selector is past the table (only possible when
/// `index_space` was not the full 16-bit space).
pub fn record(&self, selector: u16) -> Option<&'a [u8]> {
let idx = selector as usize;
if idx >= self.count {
return None;
}
let start = HEADER_LEN + idx * self.record_size;
self.data.get(start..start + self.record_size)
}
/// The encrypted key payload for a selector: the record with its 8-byte
/// sub-header stripped. The internal layout of these bytes is not yet
/// reversed (see module docs).
pub fn record_payload(&self, selector: u16) -> Option<&'a [u8]> {
self.record(selector).and_then(|r| r.get(HEADER_LEN..))
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Build a container with `record_size` and the given `index_space`, filling
/// each record with a distinguishable byte so lookups can be checked.
fn build(index_space: u16, record_size: u16) -> Vec<u8> {
let count = if index_space == 0xffff {
0x1_0000
} else {
index_space as usize
};
let mut v = Vec::with_capacity(HEADER_LEN + count * record_size as usize);
v.extend_from_slice(&0x0100_0000u32.to_be_bytes()); // tag
v.extend_from_slice(&index_space.to_be_bytes());
v.extend_from_slice(&record_size.to_be_bytes());
for i in 0..count {
let mut rec = vec![(i & 0xff) as u8; record_size as usize];
// sub-header, as seen on disc
rec[..8].copy_from_slice(&[0x01, 0x00, 0x00, 0x00, 0x00, 0x20, 0x01, 0x02]);
v.extend_from_slice(&rec);
}
v
}
#[test]
fn parses_retail_container_geometry() {
// The real disc: 0xffff index space, 536-byte records, 35,127,304 total.
let data = build(0xffff, 536);
assert_eq!(
data.len(),
35_127_304,
"matches the retail file size exactly"
);
let t = SegmentKeyTable::parse(&data).expect("parse");
assert_eq!(t.record_count(), 65_536);
assert_eq!(t.record_size(), 536);
let rec = t.record(0x1234).expect("record");
assert_eq!(rec.len(), 536);
assert_eq!(&rec[..8], &[0x01, 0x00, 0x00, 0x00, 0x00, 0x20, 0x01, 0x02]);
assert_eq!(t.record_payload(0x1234).unwrap().len(), 528);
}
#[test]
fn small_index_space_bounds_lookups() {
let data = build(4, 32);
let t = SegmentKeyTable::parse(&data).expect("parse");
assert_eq!(t.record_count(), 4);
assert!(t.record(3).is_some());
assert!(t.record(4).is_none(), "selector past the table is None");
}
#[test]
fn rejects_size_mismatch_and_truncation() {
assert!(SegmentKeyTable::parse(&[0u8; 4]).is_none());
let mut data = build(4, 32);
data.truncate(data.len() - 1); // body no longer matches header
assert!(SegmentKeyTable::parse(&data).is_none());
}
}
+144
View File
@@ -0,0 +1,144 @@
//! Structured resolution trace — what the unlock + key-resolution attempt did.
//!
//! No user-facing English. Every step's STATE is a typed enum variant;
//! applications RENDER these into localized text (the library never does). This
//! module only DEFINES the shape and is wired through the resolve/handshake
//! return path far enough to compile.
//!
//! The `who` of each step is the source's `label()` / unlocker's `name()` — a
//! stable identifier string (a NAME, like a codec id, NOT user-facing prose),
//! carried verbatim so an app renderer never has to match an enum back to a name
//! it already has. Only the OUTCOME / path enums are structured states the app
//! maps to i18n English.
/// The full trace of a resolution attempt: the unlock phase, then the
/// key-resolution phase.
#[derive(Debug, Clone, PartialEq, Default)]
pub struct ResolutionTrace {
/// One step per unlocker consulted, in consultation order.
pub unlock: Vec<UnlockStep>,
/// One step per key source consulted, in consultation order.
pub keys: Vec<KeyStep>,
}
impl ResolutionTrace {
/// An empty trace (no steps recorded).
pub fn new() -> Self {
Self::default()
}
}
// ── Unlock phase ────────────────────────────────────────────────────────────
/// One unlocker's contribution to the unlock phase. `who` is the unlocker's
/// `name()` (a stable, product-neutral identifier), carried verbatim.
#[derive(Debug, Clone, PartialEq)]
pub struct UnlockStep {
pub who: String,
pub outcome: UnlockOutcome,
}
/// What an unlocker did.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum UnlockOutcome {
/// The drive was unlocked (or already usable) and a VID is available.
Unlocked,
/// This unlocker cannot unlock this drive's firmware.
FirmwareNotUnlockable,
/// No non-revoked host cert was usable for the auth attempt. `mkb` is the
/// disc MKB generation when known.
NoUsableHostCert { mkb: Option<u32> },
/// Every available host cert was revoked on this drive's HRL. `mkb` is the
/// disc MKB generation when known.
CertRevoked { mkb: Option<u32> },
/// The drive rejected the auth handshake (non-revocation rejection / wedge).
HandshakeRejected,
/// Auth succeeded (or was skipped) but the Volume ID could not be read.
VidUnavailable,
}
// ── Key-resolution phase ────────────────────────────────────────────────────
/// One key source's contribution to the key-resolution phase, including the
/// derivation path it walked. `who` is the source's `label()` (a stable
/// identifier, e.g. `"keydb"` / `"online"`), carried verbatim.
#[derive(Debug, Clone, PartialEq)]
pub struct KeyStep {
pub who: String,
pub path: Vec<KeyNode>,
pub outcome: KeyOutcome,
}
/// A node on the derivation path a source walked. Ordered as encountered; not
/// every path hits every node.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum KeyNode {
/// The source matched this disc (by hash / VID).
MatchedDisc,
/// The source had no entry for this disc.
NoEntry,
/// Pre-decrypted unit keys were found.
FoundUnitKeys,
/// A VUK was found.
FoundVuk,
/// A Media Key was found.
FoundMediaKey,
/// A VID is required to proceed.
NeedVid,
/// The VID came from the unlock phase.
VidFromUnlock,
/// The VID came from the keydb entry.
VidFromKeydb,
/// No VID was available.
NoVid,
/// A VUK was derived (from MK + VID).
DerivedVuk,
/// Unit keys were derived (from VUK).
DerivedUnitKeys,
}
/// The terminal outcome of a source's resolution attempt.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum KeyOutcome {
/// Usable unit keys were produced.
Resolved,
/// Derivation material existed but no VID was available to finish.
MissingVid,
/// No usable key from this source.
NoKey,
}
#[cfg(test)]
mod tests {
use super::*;
/// The trace types are constructible, derive the required traits, and an
/// empty trace round-trips. Pins the structural contract apps build against.
#[test]
fn trace_is_constructible_and_comparable() {
let t = ResolutionTrace {
unlock: vec![UnlockStep {
who: "AACS cert".to_string(),
outcome: UnlockOutcome::NoUsableHostCert { mkb: Some(68) },
}],
keys: vec![KeyStep {
who: "keydb".to_string(),
path: vec![
KeyNode::MatchedDisc,
KeyNode::FoundVuk,
KeyNode::DerivedUnitKeys,
],
outcome: KeyOutcome::Resolved,
}],
};
// Clone + PartialEq (derive contract the renderers rely on).
assert_eq!(t.clone(), t);
// `who` is the source's name carried verbatim.
assert_eq!(t.keys[0].who, "keydb");
assert_eq!(t.unlock[0].who, "AACS cert");
// Default / new is empty.
assert_eq!(ResolutionTrace::new(), ResolutionTrace::default());
assert!(ResolutionTrace::new().unlock.is_empty());
assert!(ResolutionTrace::new().keys.is_empty());
}
}
+112
View File
@@ -0,0 +1,112 @@
//! AACS primitive types shared across the resolve chain.
//!
//! These structs describe AACS key material (device keys, host
//! certificates, per-disc entries). They carry no parsing logic — the
//! keydb.cfg format lives in the `freemkv-keysources` crate. libfreemkv
//! owns only the crypto and these value types that flow through it.
/// A device key for MKB subset-difference tree processing.
#[derive(Debug, Clone)]
pub struct DeviceKey {
pub key: [u8; 16],
pub node: u16,
pub uv: u32,
pub u_mask_shift: u8,
}
/// Host certificate + private key for AACS SCSI authentication.
#[derive(Debug, Clone)]
pub struct HostCert {
/// AACS 1.0: 20 bytes. AACS 2.0: 32 bytes.
pub private_key: [u8; 20],
/// AACS 1.0: 92 bytes. AACS 2.0: 132 bytes.
pub certificate: Vec<u8>,
/// AACS 2.0 host private key (P-256, 32 bytes). None for AACS 1.0 only.
pub private_key_v2: Option<[u8; 32]>,
/// AACS 2.0 host certificate (type 0x11). None for AACS 1.0 only.
pub certificate_v2: Option<Vec<u8>>,
}
/// Volume ID (16 bytes) — read from the disc via the SCSI handshake / OEM path.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Vid(pub [u8; 16]);
/// Media Key (Km, 16 bytes) — the MKB-scoped key derived from device keys.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct MediaKey(pub [u8; 16]);
/// Volume Unique Key (VUK / Kvu, 16 bytes) — derived from `MediaKey` + `Vid`,
/// decrypts the per-disc encrypted title keys in `Unit_Key_RO.inf`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Vuk(pub [u8; 16]);
/// Processing Key (Kp, 16 bytes) — an MKB Subset-Difference key that yields the
/// Media Key. A leaked/precomputed PK in the keydb, or the intermediate PK a
/// device-key walk derives at its matching SD node.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ProcessingKey(pub [u8; 16]);
/// One decrypted per-CPS-unit AACS title key.
///
/// `idx` is the POSITIONAL index of the encrypted title key within the slice
/// handed to the VUK→UK step (i.e. its order in `Unit_Key_RO.inf`'s key-storage
/// area). The CPS-unit *number* association is a higher-level concern owned by
/// [`super::inf::parse_unit_key_ro`], which pairs each positional key with its
/// declared CPS unit; this primitive only does the AES, so it surfaces position.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct UnitKey {
pub idx: u32,
pub key: [u8; 16],
/// AACS 2.1 (FMTS) forensic **index** tag (see [`crate::aacs::segment`]).
///
/// `0` = ordinary (non-forensic) content — the value for every 1.0 / 2.0
/// key and for the bulk of a 2.1 title. `1..=32` = a forensic index key that
/// decrypts the `IndividualSegment.tbl` segments tagged with that same index.
/// This is the per-segment index (1..32), NOT the AACS 2.1 Media Key Variant
/// (the 65536-value device selector), which is a separate MKB-layer concern.
pub index_number: u8,
}
impl UnitKey {
/// An ordinary (non-forensic) unit key: `index_number == 0`. The value
/// for every AACS 1.0 / 2.0 key and the bulk of a 2.1 title.
pub const fn new(idx: u32, key: [u8; 16]) -> Self {
Self {
idx,
key,
index_number: 0,
}
}
/// A forensic index key: `index_number` in `1..=32`, decrypting the
/// `IndividualSegment.tbl` segments tagged with that index.
pub const fn forensic(idx: u32, key: [u8; 16], index_number: u8) -> Self {
Self {
idx,
key,
index_number,
}
}
/// Whether this key decrypts ordinary (non-forensic) content (index 0).
pub const fn is_default_index(&self) -> bool {
self.index_number == 0
}
}
/// A per-disc entry from the key database.
#[derive(Debug, Clone)]
pub struct DiscEntry {
/// Disc hash (20 bytes, hex)
pub disc_hash: String,
/// Disc title
pub title: String,
/// Media Key (16 bytes) — from MKB processing
pub media_key: Option<[u8; 16]>,
/// Disc ID (16 bytes)
pub disc_id: Option<[u8; 16]>,
/// Volume Unique Key (16 bytes) — decrypts title keys
pub vuk: Option<[u8; 16]>,
/// Unit keys (title keys) indexed by CPS unit number
pub unit_keys: Vec<(u32, [u8; 16])>,
}
+1173
View File
File diff suppressed because it is too large Load Diff
+612 -42
View File
@@ -6,22 +6,28 @@
//!
//! Reference: https://github.com/lw/BluRay/wiki/CLPI
use crate::consts::{BD_SOURCE_PACKET_BYTES, SECTOR_BYTES_U64};
use crate::disc::Extent;
use crate::error::{Error, Result};
/// Parsed CLPI clip info.
#[derive(Debug)]
#[allow(dead_code)]
pub struct ClipInfo {
pub(crate) struct ClipInfo {
/// CLPI version string. Parsed for completeness; not yet consumed.
#[allow(dead_code)]
pub version: String,
/// Total source packets in the m2ts (each 192 bytes)
pub source_packet_count: u32,
/// Coarse EP entries for the primary video stream
/// Coarse EP entries for the primary video stream. Populated for the
/// EP-map → sector-extent lookup (`get_extents`), which is exercised by
/// tests and reserved for the timestamp-range read path.
#[allow(dead_code)]
pub ep_coarse: Vec<EpCoarse>,
/// Fine EP entries for the primary video stream
/// Fine EP entries for the primary video stream (see `ep_coarse`).
#[allow(dead_code)]
pub ep_fine: Vec<EpFine>,
/// Per-stream metadata from the ProgramInfo section (BD spec).
/// Cross-validates the MPLS STN view — see `labels/clpi.rs`.
/// Cross-validates the MPLS STN view — see `labels/clpi_audit.rs`.
/// Empty when program_info is missing or malformed.
pub streams: Vec<ClpiStream>,
}
@@ -30,57 +36,82 @@ pub struct ClipInfo {
/// table. Mirrors the same fields the MPLS STN table carries — see
/// `mpls::StreamEntry` for the playlist-side equivalent.
#[derive(Debug, Clone)]
#[allow(dead_code)]
pub struct ClpiStream {
pub(crate) struct ClpiStream {
/// PID of the stream in the MPEG-TS (matches MPLS).
pub pid: u16,
/// SCSI/BD coding type byte (0x80 LPCM, 0x83 TrueHD, 0x86 DTS-HD MA,
/// BD stream coding type byte (0x80 LPCM, 0x83 TrueHD, 0x86 DTS-HD MA,
/// 0x90 PG, etc.). See `labels::mpls_universal::coding_type_to_codec_hint`.
pub coding_type: u8,
/// ISO 639-2 3-char language code. Empty for video streams.
pub language: String,
// The CLPI cross-validation consumer (labels/clpi_audit.rs) reads only
// pid/coding_type/language. The codec sub-fields below are parsed from
// the BD stream_coding_info for completeness but have no reader yet.
/// Audio format byte (1=mono, 3=stereo, 6=5.1, 12=7.1).
/// Zero for non-audio streams.
#[allow(dead_code)]
pub audio_format: u8,
/// Audio sample rate (1=48kHz, 4=96kHz, 5=192kHz). Zero for non-audio.
#[allow(dead_code)]
pub audio_rate: u8,
/// Video format byte (1=480i, 4=1080i, 5=720p, 6=1080p, 8=2160p).
/// Zero for non-video.
#[allow(dead_code)]
pub video_format: u8,
/// Video rate (1=23.976, 2=24, 3=25, 4=29.97, 6=50, 7=59.94).
#[allow(dead_code)]
pub video_rate: u8,
}
/// Coarse EP-map entry. Fields feed the EP-map resolution used by
/// `get_extents` (test-exercised; reserved for the timestamp-range path).
#[derive(Debug, Clone)]
#[allow(dead_code)]
pub struct EpCoarse {
pub(crate) struct EpCoarse {
pub ref_to_fine_id: u32,
pub pts_coarse: u32,
pub spn_coarse: u32,
}
/// Fine EP-map entry (see `EpCoarse`).
#[derive(Debug, Clone)]
#[allow(dead_code)]
pub struct EpFine {
pub(crate) struct EpFine {
pub pts_fine: u32,
pub spn_fine: u32,
}
// EP-map → sector-extent resolution. Exercised by the unit tests and
// reserved for the timestamp-range read path; no production caller yet.
#[allow(dead_code)]
impl ClipInfo {
/// Reconstruct full PTS from coarse + fine entry.
pub fn full_pts(coarse: &EpCoarse, fine: &EpFine) -> u32 {
(coarse.pts_coarse << 19) + (fine.pts_fine << 8)
///
/// The BD spec PTS is 33-bit: `pts_coarse` is 14 bits (max 16383) and
/// `16383 << 19` exceeds `u32::MAX`, so the result must be `u64` to
/// avoid overflow (panic in debug, silent wrap in release).
pub fn full_pts(coarse: &EpCoarse, fine: &EpFine) -> u64 {
((coarse.pts_coarse as u64) << 19) + ((fine.pts_fine as u64) << 8)
}
/// Reconstruct full SPN from coarse + fine entry.
pub fn full_spn(coarse: &EpCoarse, fine: &EpFine) -> u32 {
(coarse.spn_coarse & 0xFFFE_0000) + fine.spn_fine
// The two operands occupy non-overlapping bit ranges (coarse holds
// the high bits, fine the low 17), so OR expresses intent and is
// robust to a hand-constructed EpFine.
debug_assert!(fine.spn_fine <= 0x1_FFFF);
(coarse.spn_coarse & 0xFFFE_0000) | fine.spn_fine
}
/// Get all EP entries as (PTS, SPN) pairs, fully resolved.
pub fn resolved_ep_map(&self) -> Vec<(u32, u32)> {
let mut entries = Vec::new();
///
/// PTS resets at each coarse-group boundary on disc, so the raw
/// concatenation is not globally monotonic. The returned vector is
/// sorted by PTS so callers (e.g. [`get_extents`]) can binary-search it.
///
/// [`get_extents`]: ClipInfo::get_extents
pub fn resolved_ep_map(&self) -> Vec<(u64, u32)> {
let mut entries = Vec::with_capacity(self.ep_fine.len());
for (ci, coarse) in self.ep_coarse.iter().enumerate() {
let fine_start = coarse.ref_to_fine_id as usize;
@@ -98,6 +129,12 @@ impl ClipInfo {
}
}
// get_extents binary-searches by PTS, so the map must be ordered.
// Real discs have globally increasing PTS in coarse order; sort by
// (pts, spn) so a cross-group PTS collision can't leave the search
// landing on the wrong group's SPN.
entries.sort_by_key(|&(pts, spn)| (pts, spn));
entries
}
@@ -105,7 +142,9 @@ impl ClipInfo {
///
/// Converts PTS timestamps to SPN ranges, then SPN to LBA
/// using the file's starting LBA on disc.
pub fn get_extents(&self, in_time: u32, out_time: u32) -> Vec<Extent> {
pub fn get_extents(&self, in_time: u64, out_time: u64) -> Vec<Extent> {
// resolved_ep_map() returns entries sorted by PTS, so binary search
// is valid here.
let ep_map = self.resolved_ep_map();
if ep_map.is_empty() {
return Vec::new();
@@ -122,20 +161,22 @@ impl ClipInfo {
let end_spn = match ep_map.binary_search_by_key(&out_time, |(pts, _)| *pts) {
Ok(i) => ep_map[i].1,
Err(i) if i < ep_map.len() => ep_map[i].1,
_ => ep_map.last().unwrap().1 + 1,
_ => ep_map.last().unwrap().1.saturating_add(1),
};
if end_spn <= start_spn {
return Vec::new();
}
// SPN → byte offset: spn × 192
// Byte offset → sectors: offset / 2048
// Note: the caller needs to add the file's starting LBA from UDF
let start_byte = start_spn as u64 * 192;
let end_byte = end_spn as u64 * 192;
let start_sector = (start_byte / 2048) as u32;
let end_sector = end_byte.div_ceil(2048) as u32;
// SPN → byte offset → sector range. Note: the caller adds the file's
// starting LBA from UDF. The start sector FLOORS (the extent begins in
// whichever sector contains its first byte) and the end sector CEILS
// (the extent must cover through the sector holding its last byte), so
// a sub-sector-aligned range still spans every sector it touches.
let start_byte = start_spn as u64 * BD_SOURCE_PACKET_BYTES as u64;
let end_byte = end_spn as u64 * BD_SOURCE_PACKET_BYTES as u64;
let start_sector = (start_byte / SECTOR_BYTES_U64) as u32;
let end_sector = end_byte.div_ceil(SECTOR_BYTES_U64) as u32;
vec![Extent {
start_lba: start_sector, // relative to m2ts file start
@@ -162,7 +203,7 @@ pub fn parse(data: &[u8]) -> Result<ClipInfo> {
// ClipInfo section at offset 40
// source_packet_count at offset 40 + 4(len) + 2(reserved) + 1(stream_type) + 1(app_type) + 4(reserved) + 4(ts_rate)
let source_packet_count = if data.len() > 56 {
let source_packet_count = if data.len() >= 60 {
u32::from_be_bytes([data[56], data[57], data[58], data[59]])
} else {
0
@@ -195,7 +236,7 @@ pub fn parse(data: &[u8]) -> Result<ClipInfo> {
}
/// Parse the ProgramInfo section: per-stream (pid, coding_type,
/// language, codec sub-fields). Layout per BD spec / libbluray
/// language, codec sub-fields). Layout per the BD CLPI spec
/// clpi_parse.c:
///
/// ```text
@@ -220,6 +261,7 @@ pub fn parse(data: &[u8]) -> Result<ClipInfo> {
/// errors because the EP map is the primary CLPI output, and a corrupt
/// program_info shouldn't break sector-range lookups.
fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
use crate::consts::coding_type as c;
let mut out = Vec::new();
if data.len() < 6 {
return out;
@@ -258,16 +300,15 @@ fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
let mut language = String::new();
match coding_type {
// Video — MPEG-2 (0x02), H.264 (0x1B), HEVC (0x24)
0x02 | 0x1B | 0x24 => {
// Video — MPEG-2, H.264, HEVC
c::MPEG2_VIDEO | c::H264 | c::HEVC => {
if sci.len() >= 2 {
video_format = (sci[1] >> 4) & 0x0F;
video_rate = sci[1] & 0x0F;
}
}
// Primary audio — LPCM(0x80), AC-3(0x81), DTS(0x82),
// TrueHD(0x83), AC-3+(0x84), DTS-HD(0x85), DTS-HD MA(0x86)
0x80..=0x86 => {
// Primary audio — LPCM, AC-3, DTS, TrueHD, AC-3+, DTS-HD HR, DTS-HD MA
c::LPCM..=c::DTS_HD_MA => {
if sci.len() >= 2 {
audio_format = (sci[1] >> 4) & 0x0F;
audio_rate = sci[1] & 0x0F;
@@ -276,8 +317,8 @@ fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
language = String::from_utf8_lossy(&sci[2..5]).to_string();
}
}
// Secondary audio (0xA1 AC-3+, 0xA2 DTS-HD)
0xA1 | 0xA2 => {
// Secondary audio (AC-3+ secondary, DTS-HD secondary)
c::AC3_PLUS_SECONDARY | c::DTS_HD_SECONDARY => {
if sci.len() >= 2 {
audio_format = (sci[1] >> 4) & 0x0F;
audio_rate = sci[1] & 0x0F;
@@ -286,8 +327,8 @@ fn parse_program_info(data: &[u8]) -> Vec<ClpiStream> {
language = String::from_utf8_lossy(&sci[2..5]).to_string();
}
}
// PG (0x90), IG (0x91): coding_type + 3-byte language [+ char_code for PG]
0x90 | 0x91 => {
// PG, IG: coding_type + 3-byte language [+ char_code for PG]
c::PG | c::IG => {
if sci.len() >= 4 {
language = String::from_utf8_lossy(&sci[1..4]).to_string();
}
@@ -322,8 +363,17 @@ fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
return Ok((Vec::new(), Vec::new()));
}
// Bound all EP-map reads to this CPI section. The length field counts
// bytes after itself, so the section spans data[..cpi_length + 4]. A
// bogus ep_map_offset within data.len() but past the CPI section would
// otherwise read into an adjacent CLPI section; clamp first.
let data = &data[..(cpi_length + 4).min(data.len())];
// CPI type at bits 44-47 (byte 5, lower 4 bits)
// Skip to EP map: offset 4 (after length) + 2 (reserved/type)
if data.len() < 6 {
return Ok((Vec::new(), Vec::new()));
}
let ep_map = &data[6..];
if ep_map.len() < 4 {
return Ok((Vec::new(), Vec::new()));
@@ -344,16 +394,13 @@ fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
return Ok((Vec::new(), Vec::new()));
}
// Stream PID entry — bit-packed per BD spec (libbluray clpi_parse.c):
// Stream PID entry — bit-packed per the BD CLPI spec:
// stream_PID: 16 bits → ep_map[2..4]
// reserved: 10 bits ┐
// EP_stream_type: 4 bits │ ep_map[4..14] = 80 bits
// num_EP_coarse: 16 bits │ (10+4+16+18+32 = 80)
// num_EP_fine: 18 bits │
// EP_map_start_address: 32 bits ┘
if ep_map.len() < 16 {
return Ok((Vec::new(), Vec::new()));
}
let _stream_pid = u16::from_be_bytes([ep_map[2], ep_map[3]]);
// Read 10 bytes (80 bits) from ep_map[4..14] for bit extraction
@@ -389,7 +436,10 @@ fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
// Coarse entries start at offset 4, 8 bytes each
let coarse_data = &stream_ep[4..];
let mut ep_coarse = Vec::with_capacity(num_coarse);
// Cap the pre-reservation by what the slice can actually hold:
// num_coarse is a 16-bit disc field, so a hostile value would
// otherwise reserve up to ~0.5 MB for an entry table that doesn't exist.
let mut ep_coarse = Vec::with_capacity(num_coarse.min(coarse_data.len() / 8));
for i in 0..num_coarse {
let off = i * 8;
if off + 8 > coarse_data.len() {
@@ -419,7 +469,13 @@ fn parse_cpi(data: &[u8]) -> Result<(Vec<EpCoarse>, Vec<EpFine>)> {
}
// Fine entries at fine_start, 4 bytes each
let mut ep_fine = Vec::with_capacity(num_fine);
// Cap the pre-reservation: num_fine is an 18-bit disc field (max
// 262143), so reserve only what the slice can actually hold.
let mut ep_fine = if fine_start < stream_ep.len() {
Vec::with_capacity(num_fine.min((stream_ep.len() - fine_start) / 4))
} else {
Vec::new()
};
if fine_start < stream_ep.len() {
let fine_data = &stream_ep[fine_start..];
for i in 0..num_fine {
@@ -643,10 +699,49 @@ mod tests {
};
// full_pts = (100 << 19) + (50 << 8) = 52_428_800 + 12_800 = 52_441_600
let pts = ClipInfo::full_pts(&coarse, &fine);
assert_eq!(pts, (100 << 19) + (50 << 8));
assert_eq!(pts, (100u64 << 19) + (50u64 << 8));
assert_eq!(pts, 52_441_600);
}
#[test]
fn full_pts_no_u32_overflow() {
// pts_coarse is a 14-bit field (max 0x3FFF = 16383); 16383 << 19
// overflows u32, so full_pts must use u64.
let coarse = EpCoarse {
ref_to_fine_id: 0,
pts_coarse: 0x3FFF,
spn_coarse: 0,
};
let fine = EpFine {
pts_fine: 0x7FF,
spn_fine: 0,
};
let pts = ClipInfo::full_pts(&coarse, &fine);
assert_eq!(pts, (0x3FFFu64 << 19) + (0x7FFu64 << 8));
assert!(pts > u32::MAX as u64);
}
#[test]
fn resolved_ep_map_sorted_for_binary_search() {
// Two coarse groups whose fine PTS reset across the boundary
// (50,100 then 25,75) produce a non-monotonic raw concatenation.
// resolved_ep_map must sort so get_extents' binary search is valid.
let cpi = build_cpi(
0x1011,
&[(0, 0, 0x00020000), (2, 0, 0x00040000)],
&[(50, 1024), (100, 2048), (25, 512), (75, 1536)],
);
let data = build_clpi(1_000_000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let resolved = clip.resolved_ep_map();
assert_eq!(resolved.len(), 4);
// Strictly sorted by PTS.
for w in resolved.windows(2) {
assert!(w[0].0 <= w[1].0, "ep_map not sorted: {resolved:?}");
}
}
#[test]
fn full_spn_calculation() {
let coarse = EpCoarse {
@@ -674,6 +769,22 @@ mod tests {
assert_eq!(spn2, 0x00FE0000 + 0x1234);
}
#[test]
fn parse_truncated_clipinfo_no_panic() {
// 57/58/59-byte CLPI with valid magic: passes the data.len() < 40
// guard but data[56..60] needs 60 bytes. Must not panic.
for len in 40..60usize {
let mut data = vec![0u8; len];
data[0..4].copy_from_slice(b"HDMV");
if len >= 8 {
data[4..8].copy_from_slice(b"0200");
}
let clip = parse(&data).expect("short CLPI should parse, not panic");
// source_packet_count is unreadable below 60 bytes → 0.
assert_eq!(clip.source_packet_count, 0);
}
}
#[test]
fn parse_invalid_magic() {
let mut data = build_clpi(1000, None);
@@ -707,4 +818,463 @@ mod tests {
assert!(clip2.ep_coarse.is_empty());
assert!(clip2.ep_fine.is_empty());
}
// ─────────────────────────────────────────────────────────────────────
// Added hardening tests. Grounded in the BD-ROM CLPI spec
// (https://github.com/lw/BluRay/wiki/CLPI).
// ─────────────────────────────────────────────────────────────────────
/// Build a ProgramInfo section. `streams` = Vec<(pid, sci_bytes)>.
/// Layout per source doc: length(4)+reserved(1)+num_programs(1)+
/// per program [spn(4)+pmt_pid(2)+num_streams(1)+num_groups(1)] then
/// per stream [pid(2)+sci_len(1)+sci].
fn build_program_info(streams: &[(u16, Vec<u8>)]) -> Vec<u8> {
let mut body = Vec::new();
body.push(0); // reserved (offset 4)
body.push(1); // num_programs = 1 (offset 5)
// program 0 header (8 bytes)
body.extend_from_slice(&0u32.to_be_bytes()); // spn_program_sequence_start
body.extend_from_slice(&0u16.to_be_bytes()); // program_map_pid
body.push(streams.len() as u8); // num_streams
body.push(0); // num_groups
for (pid, sci) in streams {
body.extend_from_slice(&pid.to_be_bytes());
body.push(sci.len() as u8);
body.extend_from_slice(sci);
}
// Prepend length(4) = bytes after the length field.
let mut out = Vec::new();
out.extend_from_slice(&(body.len() as u32).to_be_bytes());
out.extend_from_slice(&body);
out
}
/// Build a CLPI with a ProgramInfo section. prog_info_start is placed
/// right after the 60-byte header; cpi (if any) follows program_info.
fn build_clpi_with_proginfo(
source_packet_count: u32,
prog_info: &[u8],
cpi_data: Option<&[u8]>,
) -> Vec<u8> {
let mut buf = vec![0u8; 60];
buf[0..4].copy_from_slice(b"HDMV");
buf[4..8].copy_from_slice(b"0200");
let prog_info_start: u32 = 60;
buf[12..16].copy_from_slice(&prog_info_start.to_be_bytes());
let cpi_start: u32 = if cpi_data.is_some() {
(60 + prog_info.len()) as u32
} else {
0
};
buf[16..20].copy_from_slice(&cpi_start.to_be_bytes());
buf[56..60].copy_from_slice(&source_packet_count.to_be_bytes());
buf.extend_from_slice(prog_info);
if let Some(cpi) = cpi_data {
buf.extend_from_slice(cpi);
}
buf
}
/// source_packet_count is a big-endian u32 at offset [56..60]. Verify
/// BE decode of a value with all four bytes distinct (not LE / wrong
/// offset).
#[test]
fn source_packet_count_big_endian_offset_56() {
let data = build_clpi(0x01020304, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.source_packet_count, 0x01020304);
}
/// Magic must be exactly "HDMV" at [0..4]. Anything else → ClpiParse.
/// Spec: CLPI files begin with the type_indicator "HDMV".
#[test]
fn wrong_magic_rejected() {
let mut data = build_clpi(1000, None);
data[0..4].copy_from_slice(b"INDX");
assert!(parse(&data).is_err());
}
/// Under-40-byte input is rejected before any field read
/// (`data.len() < 40` guard).
#[test]
fn under_40_bytes_rejected() {
assert!(parse(&[0u8; 39]).is_err());
assert!(parse(b"HDMV0200").is_err());
assert!(parse(&[]).is_err());
}
/// ProgramInfo: a video stream (coding 0x1B = H.264) carries
/// format/rate in sci[1] nibbles and NO language. Verify the video
/// arm: format hi-nibble, rate lo-nibble, language stays empty.
#[test]
fn program_info_video_stream() {
// sci = coding_type(0x1B) + format_rate(0x61 → fmt 6, rate 1)
let sci = vec![0x1Bu8, 0x61];
let pi = build_program_info(&[(0x1011, sci)]);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.streams.len(), 1);
assert_eq!(clip.streams[0].pid, 0x1011);
assert_eq!(clip.streams[0].coding_type, 0x1B);
assert_eq!(clip.streams[0].video_format, 6);
assert_eq!(clip.streams[0].video_rate, 1);
assert_eq!(clip.streams[0].language, "");
}
/// ProgramInfo primary-audio (coding 0x80..=0x86): sci[1] = format/rate
/// nibbles, sci[2..5] = ISO 639 language. Verify TrueHD (0x83) at
/// offset, 5.1 / 48kHz, language "eng".
#[test]
fn program_info_audio_stream_lang_offset() {
// sci = 0x83 + 0x61 (fmt 6, rate 1) + "eng"
let sci = vec![0x83u8, 0x61, b'e', b'n', b'g'];
let pi = build_program_info(&[(0x1100, sci)]);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.streams[0].coding_type, 0x83);
assert_eq!(clip.streams[0].audio_format, 6);
assert_eq!(clip.streams[0].audio_rate, 1);
assert_eq!(clip.streams[0].language, "eng");
}
/// ProgramInfo PG (0x90)/IG (0x91): layout is coding_type(1)+lang(3),
/// so language is at sci[1..4] (NOT sci[2..5] like audio). Verify the
/// PG arm reads from the right offset.
#[test]
fn program_info_pg_lang_offset() {
// sci = 0x90 + "fra" (lang directly after coding_type)
let sci = vec![0x90u8, b'f', b'r', b'a'];
let pi = build_program_info(&[(0x1200, sci)]);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.streams[0].coding_type, 0x90);
assert_eq!(clip.streams[0].language, "fra");
// Audio nibbles must NOT be populated for a PG stream.
assert_eq!(clip.streams[0].audio_format, 0);
}
/// ProgramInfo with multiple streams: PID and coding for each must be
/// read from the correct per-stream offset (pid(2)+sci_len(1)+sci).
/// Three mixed streams must all parse with distinct PIDs in order.
#[test]
fn program_info_multiple_streams_advance_correctly() {
let v = (0x1011u16, vec![0x24u8, 0x81]); // HEVC video
let a = (0x1100u16, vec![0x86u8, 0x61, b'e', b'n', b'g']); // DTS-HD MA
let s = (0x1200u16, vec![0x90u8, b'j', b'p', b'n']); // PG
let pi = build_program_info(&[v, a, s]);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert_eq!(clip.streams.len(), 3);
assert_eq!(clip.streams[0].pid, 0x1011);
assert_eq!(clip.streams[0].coding_type, 0x24);
assert_eq!(clip.streams[1].pid, 0x1100);
assert_eq!(clip.streams[1].coding_type, 0x86);
assert_eq!(clip.streams[1].language, "eng");
assert_eq!(clip.streams[2].pid, 0x1200);
assert_eq!(clip.streams[2].language, "jpn");
}
/// parse_program_info is best-effort: a stream whose declared sci_len
/// runs past the section (`sci_end > data.len()`) makes it return the
/// streams collected so far (here: none), never panic. Source returns
/// `out` early on the overflow.
#[test]
fn program_info_truncated_sci_no_panic() {
// One stream claiming sci_len = 200 but with no body.
let mut body = Vec::new();
body.push(0); // reserved
body.push(1); // num_programs
body.extend_from_slice(&0u32.to_be_bytes());
body.extend_from_slice(&0u16.to_be_bytes());
body.push(1); // num_streams
body.push(0); // num_groups
body.extend_from_slice(&0x1011u16.to_be_bytes()); // pid
body.push(200); // sci_len = 200, no body follows
let mut pi = Vec::new();
pi.extend_from_slice(&(body.len() as u32).to_be_bytes());
pi.extend_from_slice(&body);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should not panic");
assert!(clip.streams.is_empty());
}
/// parse_program_info rejects sci_len == 0 (`sci_len < 1` → return).
/// A zero-length stream_coding_info is unusable.
#[test]
fn program_info_zero_sci_len_yields_no_stream() {
let mut body = Vec::new();
body.push(0);
body.push(1);
body.extend_from_slice(&0u32.to_be_bytes());
body.extend_from_slice(&0u16.to_be_bytes());
body.push(1);
body.push(0);
body.extend_from_slice(&0x1011u16.to_be_bytes());
body.push(0); // sci_len = 0
let mut pi = Vec::new();
pi.extend_from_slice(&(body.len() as u32).to_be_bytes());
pi.extend_from_slice(&body);
let data = build_clpi_with_proginfo(100, &pi, None);
let clip = parse(&data).expect("should parse");
assert!(clip.streams.is_empty());
}
/// pts_coarse field is 14 bits: dword0 = ref_to_fine_id<<14 | pts_coarse.
/// A pts_coarse of 0x3FFF (max) with ref_to_fine_id 5 must decode both
/// without bleed. Verify the >>14 and &0x3FFF split.
#[test]
fn coarse_pts_14bit_split() {
let cpi = build_cpi(0x1011, &[(5, 0x3FFF, 0x12340000)], &[(0, 0)]);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
assert_eq!(clip.ep_coarse[0].ref_to_fine_id, 5);
assert_eq!(clip.ep_coarse[0].pts_coarse, 0x3FFF);
assert_eq!(clip.ep_coarse[0].spn_coarse, 0x12340000);
}
/// Fine entry: dword = is_angle(1)+i_end_offset(3)+pts_fine(11)+
/// spn_fine(17). pts_fine occupies bits 17..28 (>>17 & 0x7FF), spn_fine
/// the low 17 bits (& 0x1FFFF). Set high bits (is_angle/i_end_offset)
/// and verify they do NOT bleed into pts_fine.
#[test]
fn fine_entry_bit_layout_isolates_pts_and_spn() {
// Construct a raw fine dword with is_angle=1, i_end_offset=0b111,
// pts_fine=0x5AA, spn_fine=0x1AAAA, then verify decode.
let is_angle: u32 = 1;
let i_end: u32 = 0b111;
let pts_f: u32 = 0x5AA; // 11-bit
let spn_f: u32 = 0x1AAAA; // 17-bit
let dword: u32 = (is_angle << 31) | (i_end << 28) | (pts_f << 17) | spn_f;
// Build the CPI by hand with this raw fine dword.
let mut stream_ep = Vec::new();
let fine_start: u32 = 4; // no coarse entries → fine right after header
stream_ep.extend_from_slice(&fine_start.to_be_bytes());
stream_ep.extend_from_slice(&dword.to_be_bytes());
let num_coarse: u32 = 0;
let num_fine: u32 = 1;
let ep_map_start: u32 = 14;
let ep_stream_type: u32 = 1;
let packed: u128 = ((ep_stream_type as u128) << 66)
| ((num_coarse as u128) << 50)
| ((num_fine as u128) << 32)
| (ep_map_start as u128);
let packed_bytes = packed.to_be_bytes();
let stream_header_bits = &packed_bytes[6..16];
let mut ep_map = Vec::new();
ep_map.push(0);
ep_map.push(1);
ep_map.extend_from_slice(&0x1011u16.to_be_bytes());
ep_map.extend_from_slice(stream_header_bits);
ep_map.extend_from_slice(&stream_ep);
let mut cpi = Vec::new();
cpi.extend_from_slice(&((2 + ep_map.len()) as u32).to_be_bytes());
cpi.extend_from_slice(&[0u8; 2]);
cpi.extend_from_slice(&ep_map);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
assert_eq!(clip.ep_fine.len(), 1);
assert_eq!(clip.ep_fine[0].pts_fine, 0x5AA); // high bits stripped
assert_eq!(clip.ep_fine[0].spn_fine, 0x1AAAA);
}
/// resolved_ep_map assigns fine entries to coarse groups via
/// [ref_to_fine_id .. next coarse's ref_to_fine_id). full_pts combines
/// coarse<<19 + fine<<8 and full_spn ORs masked coarse with fine.
/// Verify the first resolved entry's (pts, spn) for a known fixture.
#[test]
fn resolved_ep_map_combines_coarse_and_fine() {
// coarse 0: ref_to_fine_id=0, pts_coarse=10, spn_coarse=0x00020000
// fine 0: pts_fine=3, spn_fine=0x100
let cpi = build_cpi(0x1011, &[(0, 10, 0x00020000)], &[(3, 0x100)]);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let resolved = clip.resolved_ep_map();
assert_eq!(resolved.len(), 1);
let expected_pts = (10u64 << 19) + (3u64 << 8);
let expected_spn = (0x00020000u32 & 0xFFFE_0000) | 0x100;
assert_eq!(resolved[0].0, expected_pts);
assert_eq!(resolved[0].1, expected_spn);
}
/// get_extents converts an in/out PTS range to a single sector Extent.
/// SPN→byte = spn×192, byte→sector = /2048 (start floored, end ceiled),
/// relative to m2ts file start. Verify the math for a known fixture.
#[test]
fn get_extents_spn_to_sector_math() {
// Two EP points: PTS p0 → SPN 0, PTS p1 → SPN big_spn.
// full_spn ORs (spn_coarse & 0xFFFE0000) with spn_fine, so the SPN
// must be coarse-aligned (low 17 bits clear) to survive intact.
// 0x20000 (131072) is the smallest non-zero coarse-aligned SPN.
let big_spn: u32 = 0x20000;
let cpi = build_cpi(0x1011, &[(0, 0, 0), (1, 100, big_spn)], &[(0, 0), (0, 0)]);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let p0 = 0u64; // PTS of first EP
let p1 = 100u64 << 19; // PTS of second EP
let extents = clip.get_extents(p0, p1);
assert_eq!(extents.len(), 1);
// Mirror production: SPN→byte ×packet, byte→sector with start FLOORed
// and end CEILed (same constants as get_extents).
let start_spn: u64 = 0;
let end_spn = big_spn as u64;
let start_byte = start_spn * BD_SOURCE_PACKET_BYTES as u64;
let end_byte = end_spn * BD_SOURCE_PACKET_BYTES as u64;
let start_sector = (start_byte / SECTOR_BYTES_U64) as u32;
let end_sector = end_byte.div_ceil(SECTOR_BYTES_U64) as u32;
assert_eq!(extents[0].start_lba, start_sector);
assert_eq!(extents[0].sector_count, end_sector - start_sector);
// Concretely: 0x20000 × 192 / 2048 = 12288 sectors.
assert_eq!(extents[0].sector_count, 12288);
}
/// get_extents returns an empty Vec when the EP map is empty (no CPI),
/// since there is no SPN to resolve. Documented early return.
#[test]
fn get_extents_empty_when_no_ep_map() {
let data = build_clpi(1000, None);
let clip = parse(&data).expect("should parse");
assert!(clip.get_extents(0, 1_000_000).is_empty());
}
/// get_extents returns empty when end_spn <= start_spn (degenerate or
/// inverted range). Source has an explicit `if end_spn <= start_spn`
/// guard. Use in_time == out_time on a single-point map.
#[test]
fn get_extents_empty_on_degenerate_range() {
let cpi = build_cpi(0x1011, &[(0, 50, 0x1000)], &[(0, 0)]);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let p = 50u64 << 19;
// in == out → start_spn == end_spn → empty.
assert!(clip.get_extents(p, p).is_empty());
}
/// full_spn masks the LOW 17 bits of spn_coarse (& 0xFFFE0000) before
/// OR-ing fine. A spn_coarse with low bits set must have them cleared,
/// then replaced by spn_fine. Independent of parse, exercises the
/// reconstruction directly with a hostile low-bit pattern.
#[test]
fn full_spn_clears_coarse_low_17_bits() {
let coarse = EpCoarse {
ref_to_fine_id: 0,
pts_coarse: 0,
spn_coarse: 0x0006_FFFF, // low 17 bits all set
};
let fine = EpFine {
pts_fine: 0,
spn_fine: 0x5,
};
// 0x0006_FFFF & 0xFFFE_0000 = 0x0006_0000; | 0x5 = 0x0006_0005.
assert_eq!(ClipInfo::full_spn(&coarse, &fine), 0x0006_0005);
}
/// CPI guard: cpi_length < 4 short-circuits to empty maps (the length
/// field counts bytes after itself, and the EP map needs ≥4). A
/// cpi_length of 0/1/2/3 must yield empty EP maps, not panic.
#[test]
fn cpi_length_below_4_yields_empty() {
for bad_len in 0u32..4 {
let mut cpi = Vec::new();
cpi.extend_from_slice(&bad_len.to_be_bytes());
cpi.extend_from_slice(&[0u8; 20]); // padding so the slice exists
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
assert!(clip.ep_coarse.is_empty(), "len={bad_len}");
assert!(clip.ep_fine.is_empty(), "len={bad_len}");
}
}
/// ep_map_offset that points past the EP map (`ep_map_offset + 4 >
/// ep_map.len()`) → empty maps (bounds guard), not panic. Patch the
/// EP_map_start field to a huge value.
#[test]
fn ep_map_offset_out_of_bounds_yields_empty() {
let cpi = build_cpi(0x1011, &[(0, 10, 0x20000)], &[(5, 100)]);
let mut data = build_clpi(1000, Some(&cpi));
// EP_map_start is the low 32 bits of the 80-bit stream header at
// ep_map[4..14]. In the file: header(60) + cpi_length(4) +
// reserved(2) + ep_map reserved(1) + num_streams(1) + pid(2) = 70,
// then 10 header bytes [70..80]; EP_map_start is the last 4 [76..80].
let off = 60 + 4 + 2 + 1 + 1 + 2 + 6; // = 76
data[off..off + 4].copy_from_slice(&0xFFFF_FFFFu32.to_be_bytes());
let clip = parse(&data).expect("should not panic");
assert!(clip.ep_coarse.is_empty());
assert!(clip.ep_fine.is_empty());
}
/// num_coarse declares more entries than the CPI section holds. The
/// loop must stop at `off + 8 > coarse_data.len()` (break), not read
/// out of bounds. Patch num_coarse to a large value while supplying 1
/// coarse entry's worth of bytes.
#[test]
fn coarse_count_overshoot_truncates_safely() {
let cpi = build_cpi(0x1011, &[(0, 10, 0x20000)], &[(5, 100)]);
let mut data = build_clpi(1000, Some(&cpi));
// num_coarse is bits 14..30 of the 80-bit header. Rather than
// bit-surgery, rebuild with a hand-set num_coarse=255 but only 1
// coarse entry of bytes — done below directly.
let _ = &mut data;
let num_coarse_decl: u32 = 255;
let num_fine: u32 = 1;
let ep_map_start: u32 = 14;
let ep_stream_type: u32 = 1;
let packed: u128 = ((ep_stream_type as u128) << 66)
| ((num_coarse_decl as u128) << 50)
| ((num_fine as u128) << 32)
| (ep_map_start as u128);
let packed_bytes = packed.to_be_bytes();
let stream_header_bits = &packed_bytes[6..16];
// stream EP data: fine_start points past the 1 coarse entry.
let fine_start: u32 = 4 + 8; // 4-byte header + 1 coarse entry x 8 bytes
let mut stream_ep = Vec::new();
stream_ep.extend_from_slice(&fine_start.to_be_bytes());
// exactly ONE coarse entry (8 bytes), though header claims 255.
stream_ep.extend_from_slice(&10u32.to_be_bytes());
stream_ep.extend_from_slice(&0x20000u32.to_be_bytes());
// one fine entry (4 bytes)
stream_ep.extend_from_slice(&(((5u32 & 0x7FF) << 17) | 100).to_be_bytes());
let mut ep_map = Vec::new();
ep_map.push(0);
ep_map.push(1);
ep_map.extend_from_slice(&0x1011u16.to_be_bytes());
ep_map.extend_from_slice(stream_header_bits);
ep_map.extend_from_slice(&stream_ep);
let mut cpi2 = Vec::new();
cpi2.extend_from_slice(&((2 + ep_map.len()) as u32).to_be_bytes());
cpi2.extend_from_slice(&[0u8; 2]);
cpi2.extend_from_slice(&ep_map);
let data2 = build_clpi(1000, Some(&cpi2));
let clip = parse(&data2).expect("should not panic on coarse overshoot");
// Only the 1 real coarse entry was readable.
assert_eq!(clip.ep_coarse.len(), 1);
assert_eq!(clip.ep_coarse[0].pts_coarse, 10);
}
/// resolved_ep_map: the LAST coarse group's fine range extends to
/// ep_fine.len() (no "next coarse" bound). Verify all trailing fine
/// entries are assigned to the final coarse group.
#[test]
fn resolved_ep_map_last_group_to_end() {
// coarse 0 ref_to_fine_id=0, coarse 1 ref_to_fine_id=1.
// 3 fine entries: fine 0 → coarse 0; fine 1,2 → coarse 1.
let cpi = build_cpi(
0x1011,
&[(0, 0, 0), (1, 100, 0)],
&[(0, 10), (0, 20), (0, 30)],
);
let data = build_clpi(1000, Some(&cpi));
let clip = parse(&data).expect("should parse");
let resolved = clip.resolved_ep_map();
// All 3 fine entries resolved (last group picks up fine 1 and 2).
assert_eq!(resolved.len(), 3);
}
}
+129
View File
@@ -0,0 +1,129 @@
//! Physical media constants — the single source of truth.
//!
//! Naming convention: a constant is prefixed by the **narrowest scope where it
//! is valid**. A value common to all optical media carries no prefix; a value
//! specific to a container/format/disc-type is prefixed by it
//! (`TS_`, `BD_`, …). Define each physical quantity here exactly once and import
//! it — never re-declare a bare literal or a local copy.
/// Bytes per logical sector on every optical medium freemkv reads
/// (Blu-ray, DVD-Video, CD-ROM Mode 1). Universal — hence unprefixed.
///
/// `usize` because its dominant use is buffer sizing and slice indexing, where
/// Rust *requires* `usize` (`vec![0u8; SECTOR_BYTES]`, `buf.len() < SECTOR_BYTES`).
/// For byte-offset / capacity arithmetic — which is `u64` because a disc can
/// exceed 4 GiB — use [`SECTOR_BYTES_U64`] instead of casting at each site.
pub const SECTOR_BYTES: usize = 2048;
/// [`SECTOR_BYTES`] as `u64`, for byte-offset and capacity arithmetic. The
/// single `usize → u64` boundary cast lives here, once, so offset math across
/// the workspace reads as `sectors * SECTOR_BYTES_U64` with no per-site cast.
pub const SECTOR_BYTES_U64: u64 = SECTOR_BYTES as u64;
/// Milliseconds per second. For turning a byte count ÷ bytes-per-second into a
/// movie-time figure (`bytes / bps * MILLIS_PER_SEC`) without a bare `1000.0`.
pub const MILLIS_PER_SEC: f64 = 1_000.0;
/// Bytes per MPEG-2 transport-stream packet. Common to all MPEG-TS, not just
/// Blu-ray — prefixed by the format, not a disc type.
pub const TS_PACKET_BYTES: usize = 188;
/// Bytes in an MPEG-2 transport-stream packet header: sync byte, the
/// flags/PID word, and the adaptation/continuity byte.
pub const TS_HEADER_BYTES: usize = 4;
/// Bytes in the arrival-timestamp prefix a Blu-ray M2TS prepends to each TS
/// packet to form a source packet. Same width as a TS header but a distinct
/// quantity ([`TS_HEADER_BYTES`]) — do not conflate.
pub const BD_TIMESTAMP_PREFIX_BYTES: usize = 4;
/// Bytes of payload in an MPEG-2 transport-stream packet:
/// [`TS_PACKET_BYTES`] minus the [`TS_HEADER_BYTES`] header.
pub const TS_PAYLOAD_BYTES: usize = TS_PACKET_BYTES - TS_HEADER_BYTES;
/// Bytes per Blu-ray M2TS *source packet*: a TS packet ([`TS_PACKET_BYTES`])
/// prefixed with the [`BD_TIMESTAMP_PREFIX_BYTES`] arrival-timestamp header.
/// A BDAV/M2TS construct only — DVD VOBs have no source packets — hence `BD_`.
pub const BD_SOURCE_PACKET_BYTES: usize = TS_PACKET_BYTES + BD_TIMESTAMP_PREFIX_BYTES;
/// Elementary-stream coding-type codes — the single source of truth for the
/// byte that identifies a stream's codec.
///
/// This is one registry used in two places that share the same value space:
/// the MPEG-TS PMT `stream_type` (ISO/IEC 13818-1 Table 2-34) and the Blu-ray
/// STN/CLPI `stream_coding_type` (BD-ROM Part 3). The standardized video codes
/// (`0x02`, `0x1B`, `0x24`) are ISO assignments (ISO/IEC 13818-1 Table 2-34);
/// `0xEA` (VC-1) is a BD-ROM convention in the ISO user-private range. The
/// `0x80..=0xA2` audio/graphics codes also sit in the user-private range and follow the
/// Blu-ray Disc Association / ATSC A/52 convention. Because every consumer
/// reads or writes this single byte, the family is unprefixed — the scope is
/// "any elementary stream freemkv parses or muxes".
///
/// Each constant is `u8`: the spec defines an 8-bit field and the code compares
/// it directly against a byte read from the buffer, so no casts are needed.
pub mod coding_type {
/// MPEG-2 video (ISO/IEC 13818-1 Table 2-34).
pub const MPEG2_VIDEO: u8 = 0x02;
/// H.264 / AVC video (ISO/IEC 13818-1 Table 2-34).
pub const H264: u8 = 0x1B;
/// H.264 / MVC dependent view (Blu-ray 3D right-eye substream). Carried in
/// the SSIF interleaved stream under its own PID; the base view is [`H264`].
/// ISO/IEC 13818-1 stream_type 0x20 (MVC video sub-bitstream).
pub const H264_MVC: u8 = 0x20;
/// HEVC / H.265 video (ISO/IEC 13818-1 Table 2-34, 2015 amendment).
pub const HEVC: u8 = 0x24;
/// SMPTE VC-1 video (BD-ROM convention, ISO user-private range).
pub const VC1: u8 = 0xEA;
/// LPCM audio (BD-ROM convention).
pub const LPCM: u8 = 0x80;
/// Dolby Digital (AC-3) audio (BD-ROM / ATSC A/52 convention).
pub const AC3: u8 = 0x81;
/// DTS audio (BD-ROM convention).
pub const DTS: u8 = 0x82;
/// Dolby TrueHD audio (BD-ROM convention).
pub const TRUEHD: u8 = 0x83;
/// Dolby Digital Plus (E-AC-3 / AC-3+) audio (BD-ROM convention).
pub const AC3_PLUS: u8 = 0x84;
/// DTS-HD High Resolution audio (BD-ROM Part 3-1).
pub const DTS_HD_HR: u8 = 0x85;
/// DTS-HD Master Audio (lossless) (BD-ROM Part 3-1).
pub const DTS_HD_MA: u8 = 0x86;
/// Presentation Graphics — PG subtitle stream (BD-ROM HDMV).
pub const PG: u8 = 0x90;
/// Interactive Graphics — IG / BD-J menu overlay, NOT a subtitle (BD-ROM HDMV).
pub const IG: u8 = 0x91;
/// Text subtitle stream (BD-ROM HDMV).
pub const TEXT_SUBTITLE: u8 = 0x92;
/// Secondary Dolby Digital Plus audio (BD-ROM convention).
pub const AC3_PLUS_SECONDARY: u8 = 0xA1;
/// Secondary DTS-HD audio (lossless MA, not lossy HR) (BD-ROM convention).
pub const DTS_HD_SECONDARY: u8 = 0xA2;
}
/// MPEG PES `stream_id` codes — the byte after the `00 00 01` start-code prefix
/// that identifies an elementary stream's role in a PES packet (ISO/IEC
/// 13818-1 Table 2-22). Shared by the program-stream demuxer and the TS/M2TS
/// muxers, so defined here once. Each is `u8` (matches the byte on the wire).
pub mod pes_stream_id {
/// Video stream (`110x xxxx`; freemkv emits the base id `0xE0`).
pub const VIDEO: u8 = 0xE0;
/// private_stream_1 — AC-3 / DTS / LPCM / PGS subtitle payloads.
pub const PRIVATE_STREAM_1: u8 = 0xBD;
/// padding_stream — stuffing bytes only, no payload to demux.
pub const PADDING_STREAM: u8 = 0xBE;
/// private_stream_2 — DVD navigation (PCI/DSI); carries no muxable ES.
pub const PRIVATE_STREAM_2: u8 = 0xBF;
/// Highest video stream_id — the `110x xxxx` video range tops out at 0xEF.
pub const VIDEO_MAX: u8 = 0xEF;
/// Inclusive range of every PES `stream_id` that carries demuxable payload:
/// [`PRIVATE_STREAM_1`] (0xBD) through [`VIDEO_MAX`] (0xEF) — i.e. private
/// stream 1/2, padding, MPEG audio (0xC0-0xDF) and video (0xE0-0xEF). The
/// pack (0xBA), system-header (0xBB) and program-end (0xB9) codes sit below
/// this range and are deliberately excluded: they're structural, not ES.
pub const PAYLOAD_RANGE: core::ops::RangeInclusive<u8> = PRIVATE_STREAM_1..=VIDEO_MAX;
}
-638
View File
@@ -1,638 +0,0 @@
//! CSS drive authentication — full key hierarchy.
//!
//! Protocol:
//! 1. Bus authentication (challenge-response) → bus key
//! 2. Read disc key block (READ DVD STRUCTURE) → XOR with bus key → decrypt with player keys → disc key
//! 3. Read title key (REPORT KEY format 0x04) → XOR with bus key → decrypt with disc key → title key
//!
//! Based on libdvdcss (VideoLAN) and Stevenson 1999 analysis.
use crate::drive::Drive;
use crate::error::{Error, Result};
// ── Player keys (from libdvdcss, Stevenson's PlayerKey cracker) ───────────
const PLAYER_KEYS: [[u8; 5]; 31] = [
[0x01, 0xaf, 0xe3, 0x12, 0x80],
[0x12, 0x11, 0xca, 0x04, 0x3b],
[0x14, 0x0c, 0x9e, 0xd0, 0x09],
[0x14, 0x71, 0x35, 0xba, 0xe2],
[0x1a, 0xa4, 0x33, 0x21, 0xa6],
[0x26, 0xec, 0xc4, 0xa7, 0x4e],
[0x2c, 0xb2, 0xc1, 0x09, 0xee],
[0x2f, 0x25, 0x9e, 0x96, 0xdd],
[0x33, 0x2f, 0x49, 0x6c, 0xe0],
[0x35, 0x5b, 0xc1, 0x31, 0x0f],
[0x36, 0x67, 0xb2, 0xe3, 0x85],
[0x39, 0x3d, 0xf1, 0xf1, 0xbd],
[0x3b, 0x31, 0x34, 0x0d, 0x91],
[0x45, 0xed, 0x28, 0xeb, 0xd3],
[0x48, 0xb7, 0x6c, 0xce, 0x69],
[0x4b, 0x65, 0x0d, 0xc1, 0xee],
[0x4c, 0xbb, 0xf5, 0x5b, 0x23],
[0x51, 0x67, 0x67, 0xc5, 0xe0],
[0x53, 0x94, 0xe1, 0x75, 0xbf],
[0x57, 0x2c, 0x8b, 0x31, 0xae],
[0x63, 0xdb, 0x4c, 0x5b, 0x4a],
[0x7b, 0x1e, 0x5e, 0x2b, 0x57],
[0x85, 0xf3, 0x85, 0xa0, 0xe0],
[0xab, 0x1e, 0xe7, 0x7b, 0x72],
[0xab, 0x36, 0xe3, 0xeb, 0x76],
[0xb1, 0xb8, 0xf9, 0x38, 0x03],
[0xb8, 0x5d, 0xd8, 0x53, 0xbd],
[0xbf, 0x92, 0xc3, 0xb0, 0xe2],
[0xcf, 0x1a, 0xb2, 0xf8, 0x0a],
[0xec, 0xa0, 0xcf, 0xb3, 0xff],
[0xfc, 0x95, 0xa9, 0x87, 0x35],
];
// ── CryptKey tables ───────────────────────────────────────────────────────
const CRYPT_TAB0: [u8; 256] = [
0xB7, 0xF4, 0x82, 0x57, 0xDA, 0x4D, 0xDB, 0xE2, 0x2F, 0x52, 0x1A, 0xA8, 0x68, 0x5A, 0x8A, 0xFF,
0xFB, 0x0E, 0x6D, 0x35, 0xF7, 0x5C, 0x76, 0x12, 0xCE, 0x25, 0x79, 0x29, 0x39, 0x62, 0x08, 0x24,
0xA5, 0x85, 0x7B, 0x56, 0x01, 0x23, 0x68, 0xCF, 0x0A, 0xE2, 0x5A, 0xED, 0x3D, 0x59, 0xB0, 0xA9,
0xB0, 0x2C, 0xF2, 0xB8, 0xEF, 0x32, 0xA9, 0x40, 0x80, 0x71, 0xAF, 0x1E, 0xDE, 0x8F, 0x58, 0x88,
0xB8, 0x3A, 0xD0, 0xFC, 0xC4, 0x1E, 0xB5, 0xA0, 0xBB, 0x3B, 0x0F, 0x01, 0x7E, 0x1F, 0x9F, 0xD9,
0xAA, 0xB8, 0x3D, 0x9D, 0x74, 0x1E, 0x25, 0xDB, 0x37, 0x56, 0x8F, 0x16, 0xBA, 0x49, 0x2B, 0xAC,
0xD0, 0xBD, 0x95, 0x20, 0xBE, 0x7A, 0x28, 0xD0, 0x51, 0x64, 0x63, 0x1C, 0x7F, 0x66, 0x10, 0xBB,
0xC4, 0x56, 0x1A, 0x04, 0x6E, 0x0A, 0xEC, 0x9C, 0xD6, 0xE8, 0x9A, 0x7A, 0xCF, 0x8C, 0xDB, 0xB1,
0xEF, 0x71, 0xDE, 0x31, 0xFF, 0x54, 0x3E, 0x5E, 0x07, 0x69, 0x96, 0xB0, 0xCF, 0xDD, 0x9E, 0x47,
0xC7, 0x96, 0x8F, 0xE4, 0x2B, 0x59, 0xC6, 0xEE, 0xB9, 0x86, 0x9A, 0x64, 0x84, 0x72, 0xE2, 0x5B,
0xA2, 0x96, 0x58, 0x99, 0x50, 0x03, 0xF5, 0x38, 0x4D, 0x02, 0x7D, 0xE7, 0x7D, 0x75, 0xA7, 0xB8,
0x67, 0x87, 0x84, 0x3F, 0x1D, 0x11, 0xE5, 0xFC, 0x1E, 0xD3, 0x83, 0x16, 0xA5, 0x29, 0xF6, 0xC7,
0x15, 0x61, 0x29, 0x1A, 0x43, 0x4F, 0x9B, 0xAF, 0xC5, 0x87, 0x34, 0x6C, 0x0F, 0x3B, 0xA8, 0x1D,
0x45, 0x58, 0x25, 0xDC, 0xA8, 0xA3, 0x3B, 0xD1, 0x79, 0x1B, 0x48, 0xF2, 0xE9, 0x93, 0x1F, 0xFC,
0xDB, 0x2A, 0x90, 0xA9, 0x8A, 0x3D, 0x39, 0x18, 0xA3, 0x8E, 0x58, 0x6C, 0xE0, 0x12, 0xBB, 0x25,
0xCD, 0x71, 0x22, 0xA2, 0x64, 0xC6, 0xE7, 0xFB, 0xAD, 0x94, 0x77, 0x04, 0x9A, 0x39, 0xCF, 0x7C,
];
const CRYPT_TAB1: [u8; 256] = [
0x8C, 0x47, 0xB0, 0xE1, 0xEB, 0xFC, 0xEB, 0x56, 0x10, 0xE5, 0x2C, 0x1A, 0x5D, 0xEF, 0xBE, 0x4F,
0x08, 0x75, 0x97, 0x4B, 0x0E, 0x25, 0x8E, 0x6E, 0x39, 0x5A, 0x87, 0x53, 0xC4, 0x1F, 0xF4, 0x5C,
0x4E, 0xE6, 0x99, 0x30, 0xE0, 0x42, 0x88, 0xAB, 0xE5, 0x85, 0xBC, 0x8F, 0xD8, 0x3C, 0x54, 0xC9,
0x53, 0x47, 0x18, 0xD6, 0x06, 0x5B, 0x41, 0x2C, 0x67, 0x1E, 0x41, 0x74, 0x33, 0xE2, 0xB4, 0xE0,
0x23, 0x29, 0x42, 0xEA, 0x55, 0x0F, 0x25, 0xB4, 0x24, 0x2C, 0x99, 0x13, 0xEB, 0x0A, 0x0B, 0xC9,
0xF9, 0x63, 0x67, 0x43, 0x2D, 0xC7, 0x7D, 0x07, 0x60, 0x89, 0xD1, 0xCC, 0xE7, 0x94, 0x77, 0x74,
0x9B, 0x7E, 0xD7, 0xE6, 0xFF, 0xBB, 0x68, 0x14, 0x1E, 0xA3, 0x25, 0xDE, 0x3A, 0xA3, 0x54, 0x7B,
0x87, 0x9D, 0x50, 0xCA, 0x27, 0xC3, 0xA4, 0x50, 0x91, 0x27, 0xD4, 0xB0, 0x82, 0x41, 0x97, 0x79,
0x94, 0x82, 0xAC, 0xC7, 0x8E, 0xA5, 0x4E, 0xAA, 0x78, 0x9E, 0xE0, 0x42, 0xBA, 0x28, 0xEA, 0xB7,
0x74, 0xAD, 0x35, 0xDA, 0x92, 0x60, 0x7E, 0xD2, 0x0E, 0xB9, 0x24, 0x5E, 0x39, 0x4F, 0x5E, 0x63,
0x09, 0xB5, 0xFA, 0xBF, 0xF1, 0x22, 0x55, 0x1C, 0xE2, 0x25, 0xDB, 0xC5, 0xD8, 0x50, 0x03, 0x98,
0xC4, 0xAC, 0x2E, 0x11, 0xB4, 0x38, 0x4D, 0xD0, 0xB9, 0xFC, 0x2D, 0x3C, 0x08, 0x04, 0x5A, 0xEF,
0xCE, 0x32, 0xFB, 0x4C, 0x92, 0x1E, 0x4B, 0xFB, 0x1A, 0xD0, 0xE2, 0x3E, 0xDA, 0x6E, 0x7C, 0x4D,
0x56, 0xC3, 0x3F, 0x42, 0xB1, 0x3A, 0x23, 0x4D, 0x6E, 0x84, 0x56, 0x68, 0xF4, 0x0E, 0x03, 0x64,
0xD0, 0xA9, 0x92, 0x2F, 0x8B, 0xBC, 0x39, 0x9C, 0xAC, 0x09, 0x5E, 0xEE, 0xE5, 0x97, 0xBF, 0xA5,
0xCE, 0xFA, 0x28, 0x2C, 0x6D, 0x4F, 0xEF, 0x77, 0xAA, 0x1B, 0x79, 0x8E, 0x97, 0xB4, 0xC3, 0xF4,
];
const CRYPT_TAB2: [u8; 256] = [
0xB7, 0x75, 0x81, 0xD5, 0xDC, 0xCA, 0xDE, 0x66, 0x23, 0xDF, 0x15, 0x26, 0x62, 0xD1, 0x83, 0x77,
0xE3, 0x97, 0x76, 0xAF, 0xE9, 0xC3, 0x6B, 0x8E, 0xDA, 0xB0, 0x6E, 0xBF, 0x2B, 0xF1, 0x19, 0xB4,
0x95, 0x34, 0x48, 0xE4, 0x37, 0x94, 0x5D, 0x7B, 0x36, 0x5F, 0x65, 0x53, 0x07, 0xE2, 0x89, 0x11,
0x98, 0x85, 0xD9, 0x12, 0xC1, 0x9D, 0x84, 0xEC, 0xA4, 0xD4, 0x88, 0xB8, 0xFC, 0x2C, 0x79, 0x28,
0xD8, 0xDB, 0xB3, 0x1E, 0xA2, 0xF9, 0xD0, 0x44, 0xD7, 0xD6, 0x60, 0xEF, 0x14, 0xF4, 0xF6, 0x31,
0xD2, 0x41, 0x46, 0x67, 0x0A, 0xE1, 0x58, 0x27, 0x43, 0xA3, 0xF8, 0xE0, 0xC8, 0xBA, 0x5A, 0x5C,
0x80, 0x6C, 0xC6, 0xF2, 0xE8, 0xAD, 0x7D, 0x04, 0x0D, 0xB9, 0x3C, 0xC2, 0x25, 0xBD, 0x49, 0x63,
0x8C, 0x9F, 0x51, 0xCE, 0x20, 0xC5, 0xA1, 0x50, 0x92, 0x2D, 0xDD, 0xBC, 0x8D, 0x4F, 0x9A, 0x71,
0x2F, 0x30, 0x1D, 0x73, 0x39, 0x13, 0xFB, 0x1A, 0xCB, 0x24, 0x59, 0xFE, 0x05, 0x96, 0x57, 0x0F,
0x1F, 0xCF, 0x54, 0xBE, 0xF5, 0x06, 0x1B, 0xB2, 0x6D, 0xD3, 0x4D, 0x32, 0x56, 0x21, 0x33, 0x0B,
0x52, 0xE7, 0xAB, 0xEB, 0xA6, 0x74, 0x00, 0x4C, 0xB1, 0x7F, 0x82, 0x99, 0x87, 0x0E, 0x5E, 0xC0,
0x8F, 0xEE, 0x6F, 0x55, 0xF3, 0x7E, 0x08, 0x90, 0xFA, 0xB6, 0x64, 0x70, 0x47, 0x4A, 0x17, 0xA7,
0xB5, 0x40, 0x8A, 0x38, 0xE5, 0x68, 0x3E, 0x8B, 0x69, 0xAA, 0x9B, 0x42, 0xA5, 0x10, 0x01, 0x35,
0xFD, 0x61, 0x9E, 0xE6, 0x16, 0x9C, 0x86, 0xED, 0xCD, 0x2E, 0xFF, 0xC4, 0x5B, 0xA0, 0xAE, 0xCC,
0x4B, 0x3B, 0x03, 0xBB, 0x1C, 0x2A, 0xAC, 0x0C, 0x3F, 0x93, 0xC7, 0x72, 0x7A, 0x09, 0x22, 0x3D,
0x45, 0x78, 0xA9, 0xA8, 0xEA, 0xC9, 0x6A, 0xF7, 0x29, 0x91, 0xF0, 0x02, 0x18, 0x3A, 0x4E, 0x7C,
];
const CRYPT_TAB3: [u8; 288] = [
0x73, 0x51, 0x95, 0xE1, 0x12, 0xE4, 0xC0, 0x58, 0xEE, 0xF2, 0x08, 0x1B, 0xA9, 0xFA, 0x98, 0x4C,
0xA7, 0x33, 0xE2, 0x1B, 0xA7, 0x6D, 0xF5, 0x30, 0x97, 0x1D, 0xF3, 0x02, 0x60, 0x5A, 0x82, 0x0F,
0x91, 0xD0, 0x9C, 0x10, 0x39, 0x7A, 0x83, 0x85, 0x3B, 0xB2, 0xB8, 0xAE, 0x0C, 0x09, 0x52, 0xEA,
0x1C, 0xE1, 0x8D, 0x66, 0x4F, 0xF3, 0xDA, 0x92, 0x29, 0xB9, 0xD5, 0xC5, 0x77, 0x47, 0x22, 0x53,
0x14, 0xF7, 0xAF, 0x22, 0x64, 0xDF, 0xC6, 0x72, 0x12, 0xF3, 0x75, 0xDA, 0xD7, 0xD7, 0xE5, 0x02,
0x9E, 0xED, 0xDA, 0xDB, 0x4C, 0x47, 0xCE, 0x91, 0x06, 0x06, 0x6D, 0x55, 0x8B, 0x19, 0xC9, 0xEF,
0x8C, 0x80, 0x1A, 0x0E, 0xEE, 0x4B, 0xAB, 0xF2, 0x08, 0x5C, 0xE9, 0x37, 0x26, 0x5E, 0x9A, 0x90,
0x00, 0xF3, 0x0D, 0xB2, 0xA6, 0xA3, 0xF7, 0x26, 0x17, 0x48, 0x88, 0xC9, 0x0E, 0x2C, 0xC9, 0x02,
0xE7, 0x18, 0x05, 0x4B, 0xF3, 0x39, 0xE1, 0x20, 0x02, 0x0D, 0x40, 0xC7, 0xCA, 0xB9, 0x48, 0x30,
0x57, 0x67, 0xCC, 0x06, 0xBF, 0xAC, 0x81, 0x08, 0x24, 0x7A, 0xD4, 0x8B, 0x19, 0x8E, 0xAC, 0xB4,
0x5A, 0x0F, 0x73, 0x13, 0xAC, 0x9E, 0xDA, 0xB6, 0xB8, 0x96, 0x5B, 0x60, 0x88, 0xE1, 0x81, 0x3F,
0x07, 0x86, 0x37, 0x2D, 0x79, 0x14, 0x52, 0xEA, 0x73, 0xDF, 0x3D, 0x09, 0xC8, 0x25, 0x48, 0xD8,
0x75, 0x60, 0x9A, 0x08, 0x27, 0x4A, 0x2C, 0xB9, 0xA8, 0x8B, 0x8A, 0x73, 0x62, 0x37, 0x16, 0x02,
0xBD, 0xC1, 0x0E, 0x56, 0x54, 0x3E, 0x14, 0x5F, 0x8C, 0x8F, 0x6E, 0x75, 0x1C, 0x07, 0x39, 0x7B,
0x4B, 0xDB, 0xD3, 0x4B, 0x1E, 0xC8, 0x7E, 0xFE, 0x3E, 0x72, 0x16, 0x83, 0x7D, 0xEE, 0xF5, 0xCA,
0xC5, 0x18, 0xF9, 0xD8, 0x68, 0xAB, 0x38, 0x85, 0xA8, 0xF0, 0xA1, 0x73, 0x9F, 0x5D, 0x19, 0x0B,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x33, 0x72, 0x39, 0x25, 0x67, 0x26, 0x6D, 0x71,
0x36, 0x77, 0x3C, 0x20, 0x62, 0x23, 0x68, 0x74, 0xC3, 0x82, 0xC9, 0x15, 0x57, 0x16, 0x5D, 0x81,
];
const VARIANTS: [u8; 32] = [
0xB7, 0x74, 0x85, 0xD0, 0xCC, 0xDB, 0xCA, 0x73, 0x03, 0xFE, 0x31, 0x03, 0x52, 0xE0, 0xB7, 0x42,
0x63, 0x16, 0xF2, 0x2A, 0x79, 0x52, 0xFF, 0x1B, 0x7A, 0x11, 0xCA, 0x1A, 0x9B, 0x40, 0xAD, 0x01,
];
const SECRET: [u8; 5] = [0x55, 0xD6, 0xC4, 0xC5, 0x28];
const PERM_CHALLENGE: [[usize; 10]; 3] = [
[1, 3, 0, 7, 5, 2, 9, 6, 4, 8],
[6, 1, 9, 3, 8, 5, 7, 4, 0, 2],
[4, 0, 3, 5, 7, 2, 8, 6, 1, 9],
];
const PERM_VARIANT: [[u8; 32]; 2] = [
[
0x0A, 0x08, 0x0E, 0x0C, 0x0B, 0x09, 0x0F, 0x0D, 0x1A, 0x18, 0x1E, 0x1C, 0x1B, 0x19, 0x1F,
0x1D, 0x02, 0x00, 0x06, 0x04, 0x03, 0x01, 0x07, 0x05, 0x12, 0x10, 0x16, 0x14, 0x13, 0x11,
0x17, 0x15,
],
[
0x12, 0x1A, 0x16, 0x1E, 0x02, 0x0A, 0x06, 0x0E, 0x10, 0x18, 0x14, 0x1C, 0x00, 0x08, 0x04,
0x0C, 0x13, 0x1B, 0x17, 0x1F, 0x03, 0x0B, 0x07, 0x0F, 0x11, 0x19, 0x15, 0x1D, 0x01, 0x09,
0x05, 0x0D,
],
];
// ── SCSI constants ────────────────────────────────────────────────────────
const SCSI_READ_DVD_STRUCTURE: u8 = 0xAD;
// ── Public API ────────────────────────────────────────────────────────────
/// Perform CSS bus authentication only.
pub fn authenticate(drive: &mut Drive) -> Result<()> {
let (_, _) = bus_auth(drive)?;
Ok(())
}
/// Full CSS key extraction: bus auth → disc key → title key.
pub fn authenticate_and_read_title_key(drive: &mut Drive, lba: u32) -> Result<[u8; 5]> {
// Session 1: bus auth → disc key (AGID consumed by READ_DVD_STRUCTURE)
let (agid, bus_key) = bus_auth(drive)?;
let disc_key = read_disc_key(drive, agid, &bus_key)?;
// Session 2: fresh bus auth → title key (needs separate AGID)
let (agid2, bus_key2) = bus_auth(drive)?;
let encrypted_title = read_raw_title_key(drive, agid2, lba)?;
// Decrypt title key: XOR with bus key, then decrypt with disc key
let mut title_key = [0u8; 5];
for i in 0..5 {
title_key[i] = encrypted_title[i] ^ bus_key2[i];
}
if title_key == [0u8; 5] {
return Ok(title_key);
}
let title_key = super::lfsr::decrypt_key(0xFF, &disc_key, &title_key);
Ok(title_key)
}
// ── Step 1: Bus Authentication ────────────────────────────────────────────
fn bus_auth(drive: &mut Drive) -> Result<(u8, [u8; 5])> {
let scsi = drive.scsi_mut();
// Invalidate all AGIDs via REPORT KEY format 0x3F
for agid in 0..4u8 {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
// alloc_len = 0 (no data transfer)
cdb[10] = (agid << 6) | 0x3F;
let mut buf = [0u8; 8];
let _ = scsi.execute(
&cdb,
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
);
}
// Allocate AGID
let mut buf = [0u8; 8];
scsi.execute(
&report_key_cdb(0, 0x00, 8),
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
let agid = (buf[7] >> 6) & 0x03;
// Host sends challenge
let host_challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
let mut hc_buf = [0u8; 16];
hc_buf[0] = 0x00;
hc_buf[1] = 0x0E;
for i in 0..10 {
hc_buf[4 + i] = host_challenge[9 - i];
}
scsi.execute(
&send_key_cdb(agid, 0x01, 16),
crate::scsi::DataDirection::ToDevice,
&mut hc_buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
// Get Key1 from drive
let mut dk_buf = [0u8; 12];
scsi.execute(
&report_key_cdb(agid, 0x02, 12),
crate::scsi::DataDirection::FromDevice,
&mut dk_buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
let mut key1 = [0u8; 5];
for i in 0..5 {
key1[i] = dk_buf[4 + (4 - i)];
}
// Brute-force variant (0-31)
let mut variant: Option<u8> = None;
for v in 0..32u8 {
if crypt_key(0, v, &host_challenge) == key1 {
variant = Some(v);
break;
}
}
let variant = variant.ok_or(Error::CssAuthFailed)?;
// Get drive challenge
let mut dc_buf = [0u8; 16];
scsi.execute(
&report_key_cdb(agid, 0x01, 16),
crate::scsi::DataDirection::FromDevice,
&mut dc_buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
let mut drive_challenge = [0u8; 10];
for i in 0..10 {
drive_challenge[i] = dc_buf[4 + (9 - i)];
}
// Compute Key2 and send it
let key2 = crypt_key(1, variant, &drive_challenge);
let mut hk_buf = [0u8; 12];
hk_buf[0] = 0x00;
hk_buf[1] = 0x0A;
for i in 0..5 {
hk_buf[4 + i] = key2[4 - i];
}
scsi.execute(
&send_key_cdb(agid, 0x03, 12),
crate::scsi::DataDirection::ToDevice,
&mut hk_buf,
5_000,
)
.map_err(|_| Error::CssAuthFailed)?;
// Bus key = CryptKey(2, variant, key1 || key2)
let mut combined = [0u8; 10];
combined[..5].copy_from_slice(&key1);
combined[5..].copy_from_slice(&key2);
let bus_key = crypt_key(2, variant, &combined);
Ok((agid, bus_key))
}
// ── Step 2: Disc Key ──────────────────────────────────────────────────────
fn read_disc_key(drive: &mut Drive, agid: u8, bus_key: &[u8; 5]) -> Result<[u8; 5]> {
let scsi = drive.scsi_mut();
// READ DVD STRUCTURE, format 0x02 (disc key), 2048+4 bytes
let alloc_len: u16 = 2048 + 4;
let mut cdb = [0u8; 12];
cdb[0] = SCSI_READ_DVD_STRUCTURE;
// bytes 2-5: address = 0
cdb[6] = 0; // layer
cdb[7] = 0x02; // format = disc key
cdb[8] = (alloc_len >> 8) as u8;
cdb[9] = alloc_len as u8;
cdb[10] = agid << 6;
let mut buf = vec![0u8; alloc_len as usize];
let dvd_result = scsi.execute(
&cdb,
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
);
dvd_result.map_err(|_| Error::CssAuthFailed)?;
// Disc key block starts at offset 4 (skip 4-byte header)
let disc_key_block = &mut buf[4..4 + 2048];
// XOR with reversed bus key (per libdvdcss)
for (i, byte) in disc_key_block.iter_mut().enumerate() {
*byte ^= bus_key[4 - (i % 5)];
}
// Try each player key against each of 408 disc key entries.
// Each entry in the block is the disc key encrypted with a specific player key.
// We try all known player keys and verify by checking that two different
// entries produce the same disc key.
let mut candidates: Vec<([u8; 5], usize, usize)> = Vec::new(); // (disc_key, pk_idx, pos)
for (pk_idx, player_key) in PLAYER_KEYS.iter().enumerate() {
for pos in 0..408 {
let offset = pos * 5;
if offset + 5 > disc_key_block.len() {
break;
}
let mut enc = [0u8; 5];
enc.copy_from_slice(&disc_key_block[offset..offset + 5]);
let candidate = super::lfsr::decrypt_key(0x00, player_key, &enc);
// Check if any previous candidate matches (same disc key from different entry/pk)
for (prev, _, _) in &candidates {
if *prev == candidate {
return Ok(candidate);
}
}
candidates.push((candidate, pk_idx, pos));
}
}
Err(Error::CssAuthFailed)
}
// ── Step 3: Title Key ─────────────────────────────────────────────────────
/// Read the raw (bus-encrypted) title key bytes from the drive.
fn read_raw_title_key(drive: &mut Drive, agid: u8, lba: u32) -> Result<[u8; 5]> {
let scsi = drive.scsi_mut();
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
cdb[2] = (lba >> 24) as u8;
cdb[3] = (lba >> 16) as u8;
cdb[4] = (lba >> 8) as u8;
cdb[5] = lba as u8;
cdb[8] = 0x00;
cdb[9] = 0x0C;
cdb[10] = (agid << 6) | 0x04;
let mut buf = [0u8; 12];
let result = scsi.execute(
&cdb,
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
);
result.map_err(|_| Error::CssAuthFailed)?;
let mut key = [0u8; 5];
for i in 0..5 {
key[i] = buf[5 + (4 - i)];
}
Ok(key)
}
#[allow(dead_code)]
fn read_title_key(
drive: &mut Drive,
agid: u8,
lba: u32,
bus_key: &[u8; 5],
disc_key: &[u8; 5],
) -> Result<[u8; 5]> {
let scsi = drive.scsi_mut();
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
cdb[2] = (lba >> 24) as u8;
cdb[3] = (lba >> 16) as u8;
cdb[4] = (lba >> 8) as u8;
cdb[5] = lba as u8;
cdb[8] = 0x00;
cdb[9] = 0x0C;
cdb[10] = (agid << 6) | 0x04;
let mut buf = [0u8; 12];
let tk_result = scsi.execute(
&cdb,
crate::scsi::DataDirection::FromDevice,
&mut buf,
5_000,
);
tk_result.map_err(|_| Error::CssAuthFailed)?;
// Title key at bytes 5..10, byte-reversed
let mut title_key = [0u8; 5];
for i in 0..5 {
title_key[i] = buf[5 + (4 - i)];
}
// XOR with reversed bus key (same pattern as disc key block)
for i in 0..5 {
title_key[i] ^= bus_key[4 - i];
}
// Check for null key (title not encrypted)
if title_key == [0u8; 5] {
return Ok(title_key);
}
// Decrypt with disc key (invert=0xFF for title keys)
let title_key = super::lfsr::decrypt_key(0xFF, disc_key, &title_key);
Ok(title_key)
}
// ── CSSCryptKey ───────────────────────────────────────────────────────────
/// Exposed for testing only.
pub fn test_crypt_key(key_type: usize, variant: u8, challenge: &[u8; 10]) -> [u8; 5] {
crypt_key(key_type, variant, challenge)
}
fn crypt_key(key_type: usize, variant: u8, challenge: &[u8; 10]) -> [u8; 5] {
let perm = &PERM_CHALLENGE[key_type];
let mut scratch = [0u8; 10];
for i in 0..10 {
scratch[i] = challenge[perm[i]];
}
let css_variant = match key_type {
0 => variant as usize,
1 => PERM_VARIANT[0][variant as usize] as usize,
_ => PERM_VARIANT[1][variant as usize] as usize,
};
let cse = VARIANTS[css_variant] ^ CRYPT_TAB2[css_variant];
let mut tmp1 = [0u8; 5];
for i in 0..5 {
tmp1[i] = scratch[5 + i] ^ SECRET[i] ^ CRYPT_TAB2[i];
}
let mut lfsr0: u32 = ((tmp1[0] as u32) << 17)
| ((tmp1[1] as u32) << 9)
| (((tmp1[2] as u32) & !7) << 1)
| 8
| (tmp1[2] as u32 & 7);
let mut lfsr1: u32 = ((tmp1[3] as u32) << 9) | 0x100 | (tmp1[4] as u32);
let mut bits = [0u8; 30];
let mut carry: u32 = 0;
for idx in (0..30).rev() {
let mut val: u8 = 0;
for bit in 0..8u8 {
let lfsr0_out = ((lfsr0 >> 24) ^ (lfsr0 >> 21) ^ (lfsr0 >> 20) ^ (lfsr0 >> 12)) & 1;
lfsr0 = ((lfsr0 << 1) | lfsr0_out) & 0x1FFFFFF;
let lfsr1_out = ((lfsr1 >> 16) ^ (lfsr1 >> 2)) & 1;
lfsr1 = ((lfsr1 << 1) | lfsr1_out) & 0x1FFFF;
let combined = ((!lfsr1_out) & 1) + carry + ((!lfsr0_out) & 1);
carry = (combined >> 1) & 1;
val |= ((combined & 1) as u8) << bit;
}
bits[idx] = val;
}
let mut tmp1 = [scratch[0], scratch[1], scratch[2], scratch[3], scratch[4]];
let mut tmp2 = [0u8; 5];
// Round 1: bits[25..29] ^ scratch -> tmp1 (term from original scratch)
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[25 + i] ^ tmp1[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
tmp1[i] = CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term;
term = scratch[i]; // original challenge, NOT modified tmp1
}
tmp1[4] ^= tmp1[0];
}
// Round 2
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[20 + i] ^ tmp1[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
tmp2[i] = CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term;
term = tmp1[i];
}
tmp2[4] ^= tmp2[0];
}
// Round 3 (uses CRYPT_TAB0)
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[15 + i] ^ tmp2[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
let idx3 = (CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term) as usize;
tmp1[i] = CRYPT_TAB0[idx3] ^ CRYPT_TAB2[idx3];
term = tmp2[i];
}
tmp1[4] ^= tmp1[0];
}
// Round 4 (uses CRYPT_TAB0)
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[10 + i] ^ tmp1[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
let idx3 = (CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term) as usize;
tmp2[i] = CRYPT_TAB0[idx3] ^ CRYPT_TAB2[idx3];
term = tmp1[i];
}
tmp2[4] ^= tmp2[0];
}
// Round 5
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[5 + i] ^ tmp2[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
tmp1[i] = CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term;
term = tmp2[i];
}
tmp1[4] ^= tmp1[0];
}
// Round 6
let mut key = [0u8; 5];
{
let mut term: u8 = 0;
for i in (0..5usize).rev() {
let idx = (bits[i] ^ tmp1[i]) as usize;
let idx2 = (CRYPT_TAB1[idx] ^ (!CRYPT_TAB2[idx]) ^ cse) as usize;
key[i] = CRYPT_TAB2[idx2] ^ CRYPT_TAB3[idx2] ^ term;
term = tmp1[i];
}
}
key
}
// ── SCSI CDB builders ────────────────────────────────────────────────────
fn report_key_cdb(agid: u8, format: u8, alloc_len: u16) -> [u8; 12] {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_REPORT_KEY;
cdb[8] = (alloc_len >> 8) as u8;
cdb[9] = alloc_len as u8;
cdb[10] = (agid << 6) | (format & 0x3F);
cdb
}
fn send_key_cdb(agid: u8, format: u8, param_len: u16) -> [u8; 12] {
let mut cdb = [0u8; 12];
cdb[0] = crate::scsi::SCSI_SEND_KEY;
cdb[8] = (param_len >> 8) as u8;
cdb[9] = param_len as u8;
cdb[10] = (agid << 6) | (format & 0x3F);
cdb
}
// ── Tests ─────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn crypt_key_is_deterministic() {
let challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
for v in 0..32u8 {
let r1 = crypt_key(0, v, &challenge);
let r2 = crypt_key(0, v, &challenge);
assert_eq!(r1, r2);
}
}
#[test]
fn crypt_key_varies_by_variant() {
let challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
assert_ne!(crypt_key(0, 0, &challenge), crypt_key(0, 1, &challenge));
}
#[test]
fn crypt_key_varies_by_type() {
let challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
assert_ne!(crypt_key(0, 5, &challenge), crypt_key(1, 5, &challenge));
}
#[test]
fn crypt_key_nonzero() {
let challenge: [u8; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
for v in 0..32u8 {
assert_ne!(crypt_key(0, v, &challenge), [0u8; 5]);
}
}
#[test]
fn player_keys_count() {
assert_eq!(PLAYER_KEYS.len(), 31);
}
}
-398
View File
@@ -1,398 +0,0 @@
//! CSS title key recovery — Stevenson's divide-and-conquer attack (1999).
//!
//! Given a scrambled DVD sector with known plaintext (MPEG-2 PES headers),
//! recovers the 5-byte title key by:
//!
//! 1. XORing ciphertext with TAB1[ciphertext] to cancel the mangling
//! 2. Iterating all 2^16 LFSR1 states
//! 3. For each: deducing what LFSR0 must produce, then verifying
//!
//! Total work: ~65536 iterations with 10-byte validation = instant.
//!
//! Algorithm: Frank A. Stevenson, "Divide and conquer attack" (1999).
use super::tables::{TAB1, TAB2, TAB3, TAB4, TAB5};
/// Sector layout constants.
const SECTOR_SIZE: usize = 2048;
const ENCRYPTED_START: usize = 0x80; // byte 128
const SEED_OFFSET: usize = 0x54; // sector seed at bytes 0x54-0x58
const FLAG_BYTE: usize = 0x14;
/// Recover the CSS title key from a scrambled sector using known plaintext.
///
/// The `plain` slice should contain the expected plaintext of the encrypted
/// region (bytes 0x80+). For MPEG-2 sectors, the first bytes are typically
/// a PES header: `00 00 01 [stream_id] ...`
///
/// Returns the recovered 5-byte title key, or None if recovery fails.
pub fn recover_title_key(sector: &[u8], plain: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_SIZE || plain.len() < 10 {
return None;
}
let flags = (sector[FLAG_BYTE] >> 4) & 0x03;
if flags == 0 {
return None;
}
let crypted = &sector[ENCRYPTED_START..];
let seed = &sector[SEED_OFFSET..SEED_OFFSET + 5];
// Phase 1: Cancel the TAB1 mangling layer
// The CSS cipher applies TAB1 as an output permutation.
// XORing ciphertext with TAB1[ciphertext] and plaintext removes it,
// leaving the raw LFSR combination output.
let mut buf = [0u8; 10];
for i in 0..10 {
if i >= crypted.len() || i >= plain.len() {
return None;
}
buf[i] = TAB1[crypted[i] as usize] ^ plain[i];
}
// Phase 2: Stevenson attack — iterate all 2^16 LFSR1 initial states
let mut result_key = [0u8; 5];
let mut found = false;
'outer: for i_try in 0u32..0x10000 {
let mut t1 = (i_try >> 8) | 0x100;
let mut t2 = i_try & 0xFF;
let mut t5: u32 = 0;
// Clock LFSR1 forward 4 steps to reconstruct LFSR0 state
let mut t3: u32 = 0;
for &buf_byte in buf.iter().take(4) {
// Advance LFSR1
let t4 = TAB2[t2 as usize] ^ TAB3[t1 as usize];
t2 = t1 >> 1;
t1 = ((t1 & 1) << 8) ^ t4 as u32;
let t4_perm = TAB5[t4 as usize];
// Deduce LFSR0 output from the buffer and LFSR1 output
let mut t6 = buf_byte as u32;
if t5 > 0 {
t6 = (t6 + 0xFF) & 0xFF;
}
if t6 < t4_perm as u32 {
t6 += 0x100;
}
t6 -= t4_perm as u32;
t5 += t6 + t4_perm as u32;
let t6_inv = TAB4[t6 as usize & 0xFF];
// Build LFSR0 candidate from deduced output bytes
t3 = (t3 << 8) | t6_inv as u32;
t5 >>= 8;
}
let candidate = t3;
// Phase 3: Validate — clock 6 more steps and check against buffer
let mut valid = true;
for &buf_byte in buf.iter().skip(4) {
let t4 = TAB2[t2 as usize] ^ TAB3[t1 as usize];
t2 = t1 >> 1;
t1 = ((t1 & 1) << 8) ^ t4 as u32;
let t4_perm = TAB5[t4 as usize];
// Clock LFSR0 forward
let t6 = ((((((t3 >> 8) ^ t3) >> 1) ^ t3) >> 3) ^ t3) >> 7;
t3 = (t3 << 8) | (t6 & 0xFF);
let t6_perm = TAB4[(t6 & 0xFF) as usize];
t5 += t6_perm as u32 + t4_perm as u32;
if (t5 & 0xFF) as u8 != buf_byte {
valid = false;
break;
}
t5 >>= 8;
}
if !valid {
continue;
}
// Phase 4: Recover the initial LFSR0 state from the candidate
t3 = candidate;
let mut recovery_ok = true;
for _ in 0..4 {
let t1_byte = t3 & 0xFF;
t3 >>= 8;
// Brute-force the byte that was shifted in
let mut found_j = false;
for j in 0u32..256 {
t3 = (t3 & 0x1FFFF) | (j << 17);
let t6 = ((((((t3 >> 8) ^ t3) >> 1) ^ t3) >> 3) ^ t3) >> 7;
if (t6 & 0xFF) == t1_byte {
found_j = true;
break;
}
}
if !found_j {
recovery_ok = false;
break;
}
}
if !recovery_ok {
continue 'outer;
}
// Convert LFSR0 initial state back to key bytes
let t4 = (t3 >> 1).wrapping_sub(4);
for t5_off in 0u32..8 {
let val = t4.wrapping_add(t5_off);
if (val * 2 + 8 - (val & 7)) == t3 {
result_key[0] = (i_try >> 8) as u8;
result_key[1] = (i_try & 0xFF) as u8;
result_key[2] = (val & 0xFF) as u8;
result_key[3] = ((val >> 8) & 0xFF) as u8;
result_key[4] = ((val >> 16) & 0xFF) as u8;
found = true;
break;
}
}
if found {
break;
}
}
if !found {
return None;
}
// XOR with sector seed to get the actual title key
result_key[0] ^= seed[0];
result_key[1] ^= seed[1];
result_key[2] ^= seed[2];
result_key[3] ^= seed[3];
result_key[4] ^= seed[4];
Some(result_key)
}
/// Crack the CSS title key from an encrypted sector using MPEG-2 pattern attack.
///
/// Detects the PES header pattern at byte 0x80 and uses it as known plaintext.
pub fn crack_title_key(sector: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_SIZE {
return None;
}
let flags = (sector[FLAG_BYTE] >> 4) & 0x03;
if flags == 0 {
return None;
}
// The PES header at byte 0x80 typically starts with 00 00 01 [stream_id].
// The next bytes are PES length and flags. We need at least 10 bytes of
// known plaintext for the Stevenson attack.
//
// Strategy: try common PES patterns. The first 3 bytes are always 00 00 01.
// The stream_id varies. Bytes 4-9 depend on PES header structure.
//
// For a standard PES with PTS:
// 00 00 01 [id] [len_hi] [len_lo] [flags] [flags2] [hdr_len] [PTS...]
//
// We try multiple stream IDs and use zeros for unknown bytes (most common).
// Try many PES header patterns at byte 0x80.
// Structure: 00 00 01 [stream_id] [len_hi] [len_lo] [flags1] [flags2] [hdr_len] [data]
let mut patterns: Vec<[u8; 10]> = Vec::with_capacity(128);
// Padding stream (0xBE): payload is 0xFF bytes, various lengths
for len_hi in 0u8..8 {
for len_lo_top in [0x00u8, 0x80, 0xFF] {
patterns.push([
0x00, 0x00, 0x01, 0xBE, len_hi, len_lo_top, 0xFF, 0xFF, 0xFF, 0xFF,
]);
}
}
// Video (0xE0) and audio (0xBD, 0xC0) with typical PES headers
for &sid in &[0xE0u8, 0xBD, 0xC0] {
for &flags1 in &[0x80u8, 0x81, 0x84, 0x85, 0x8C, 0x8D] {
for &flags2 in &[0x00u8, 0x05, 0x80, 0xC0] {
let hdr_len = if flags2 & 0x80 != 0 { 0x05u8 } else { 0x00 };
let pts0 = if flags2 & 0x80 != 0 { 0x21u8 } else { 0x00 };
// Try with several PES lengths
for &len_hi in &[0x00u8, 0x07] {
patterns.push([
0x00, 0x00, 0x01, sid, len_hi, 0x00, flags1, flags2, hdr_len, pts0,
]);
}
}
}
}
// Navigation pack system header (0xBB)
patterns.push([0x00, 0x00, 0x01, 0xBB, 0x00, 0x12, 0x80, 0xC4, 0xE1, 0x04]);
for pattern in &patterns {
if let Some(key) = recover_title_key(sector, pattern) {
let mut test = sector.to_vec();
super::lfsr::descramble_sector(&key, &mut test);
if test[0x80] == 0x00 && test[0x81] == 0x00 && test[0x82] == 0x01 {
return Some(key);
}
}
}
None
}
/// Crack CSS key from multiple sectors.
pub fn crack_from_sectors(sectors: &[Vec<u8>]) -> Option<[u8; 5]> {
for sector in sectors {
if sector.len() < SECTOR_SIZE {
continue;
}
let flags = (sector[FLAG_BYTE] >> 4) & 0x03;
if flags == 0 {
continue;
}
if let Some(key) = crack_title_key(sector) {
return Some(key);
}
}
None
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn crack_unscrambled_returns_none() {
let sector = vec![0u8; 2048];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn crack_too_short_returns_none() {
let sector = vec![0u8; 100];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn recover_needs_10_bytes_plain() {
let sector = vec![0u8; 2048];
let short_plain = [0u8; 5];
assert!(recover_title_key(&sector, &short_plain).is_none());
}
/// Test 3: css_crack_recovers_key_from_scrambled_sector
///
/// Build a plaintext sector with known MPEG-2 PES headers, scramble it
/// with a known title key, then run crack_title_key() on the scrambled
/// sector. If the Stevenson attack succeeds, verify that descrambling
/// with the recovered key produces the original plaintext at bytes 128..132.
#[test]
fn css_crack_recovers_key_from_scrambled_sector() {
use super::super::lfsr::descramble_sector;
let title_key: [u8; 5] = [0x42, 0x13, 0x37, 0xBE, 0xEF];
// Build a plaintext MPEG-2 sector
let mut plaintext = vec![0x00u8; SECTOR_SIZE];
// Pack header at byte 0: 00 00 01 BA
plaintext[0] = 0x00;
plaintext[1] = 0x00;
plaintext[2] = 0x01;
plaintext[3] = 0xBA;
// Scramble flag at byte 0x14
plaintext[FLAG_BYTE] = 0x30;
// Sector seed at bytes 0x54-0x58
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
// PES header at byte 0x80: 00 00 01 E0 (video stream)
// Then typical PES header bytes for a stream with PTS
plaintext[0x80] = 0x00;
plaintext[0x81] = 0x00;
plaintext[0x82] = 0x01;
plaintext[0x83] = 0xE0;
plaintext[0x84] = 0x00; // PES length hi
plaintext[0x85] = 0x00; // PES length lo
plaintext[0x86] = 0x80; // flags: data_alignment, copyright
plaintext[0x87] = 0x80; // PTS flag
plaintext[0x88] = 0x05; // PES header data length
plaintext[0x89] = 0x21; // PTS byte 1
let original_plaintext = plaintext.clone();
// "Scramble" the sector by calling descramble (which XORs the keystream)
// on the plaintext. This produces a scrambled sector.
descramble_sector(&title_key, &mut plaintext);
// The scramble flag was cleared by descramble_sector. Restore it so
// the cracker sees it as encrypted.
plaintext[FLAG_BYTE] = 0x30;
// Now we have a scrambled sector. Try to crack the title key.
let cracked_key = crack_title_key(&plaintext);
match cracked_key {
Some(key) => {
// Verify: descramble with the cracked key should recover plaintext
let mut test = plaintext.clone();
descramble_sector(&key, &mut test);
// Check that the PES header is recovered
assert_eq!(test[0x80], 0x00, "PES byte 0 mismatch");
assert_eq!(test[0x81], 0x00, "PES byte 1 mismatch");
assert_eq!(test[0x82], 0x01, "PES byte 2 mismatch");
assert_eq!(test[0x83], 0xE0, "PES byte 3 mismatch");
// Also verify the rest of the encrypted region matches original
assert_eq!(
&test[0x80..SECTOR_SIZE],
&original_plaintext[0x80..SECTOR_SIZE],
"Decrypted content does not match original plaintext"
);
eprintln!(
"Stevenson attack succeeded: cracked key = {:02X?}, original = {:02X?}",
key, title_key
);
}
None => {
// The Stevenson attack may not always find a key for all title keys
// and sector seeds. This is expected for some combinations where the
// known plaintext pattern doesn't match what crack_title_key tries.
eprintln!(
"Stevenson attack did not find key for title_key={:02X?} seed={:02X?}. \
This can happen when the cipher output doesn't match the tried patterns. \
Testing with recover_title_key directly with exact plaintext.",
title_key,
&[0x11u8, 0x22, 0x33, 0x44, 0x55],
);
// Try with exact known plaintext instead of guessing
let exact_plain: [u8; 10] =
[0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x80, 0x05, 0x21];
let recovered = recover_title_key(&plaintext, &exact_plain);
if let Some(key) = recovered {
let mut test = plaintext.clone();
descramble_sector(&key, &mut test);
assert_eq!(test[0x80], 0x00);
assert_eq!(test[0x81], 0x00);
assert_eq!(test[0x82], 0x01);
eprintln!(
"recover_title_key with exact plaintext succeeded: {:02X?}",
key
);
} else {
eprintln!(
"recover_title_key also returned None. The attack may not converge \
for this particular key/seed combination. This is a known limitation \
of the brute-force LFSR0 recovery phase."
);
}
}
}
}
}
+352 -161
View File
@@ -1,135 +1,161 @@
//! CSS cipher implementation based on the Stevenson 1999 analysis.
//! CSS content cipher — an independent implementation of the publicly
//! documented Content Scramble System stream cipher.
//!
//! The CSS cipher uses two table-driven feedback circuits:
//! - LFSR1: 9-bit state (two halves), driven by TAB2/TAB3
//! - LFSR0: 32-bit state, driven by a feedback polynomial through TAB4
//! The algorithm is the one recovered and published in Frank A. Stevenson's
//! 1999 cryptanalysis ("Cryptanalysis of Contents Scrambling System") and
//! described in the open CSS literature. It is implemented here from that public
//! description; its constants (see [`super::tables`]) are the cipher's own
//! defined values. Nothing in this file is copied or translated from any
//! particular CSS software.
//!
//! The keystream is the bytewise sum (with carry) of both LFSR outputs.
//! Content descrambling XORs this keystream with the encrypted sector data.
//! The cipher uses two table-driven linear-feedback circuits:
//! - **LFSR1** — a 17-bit register (a 9-bit and an 8-bit half seeded from
//! `key[0..2] XOR seed[0..2]`), stepped through `TAB2`/`TAB3`/`TAB5`.
//! - **LFSR0** — a 24-bit feedback register (seeded from `key[2..5] XOR
//! seed[2..5]`), stepped through a feedback polynomial and `TAB4`.
//!
//! Algorithm: Frank A. Stevenson's divide-and-conquer attack (1999).
//! Tables: CSS specification constants.
//! Each output byte is the sum-with-carry of the two register outputs. A body
//! byte is recovered as `plain = TAB1[cipher] ^ keystream` — a `TAB1`
//! substitution of the ciphertext byte followed by an XOR with the keystream
//! (so the cipher is deliberately not its own inverse).
use super::tables::{TAB1, TAB2, TAB3, TAB4, TAB5};
/// Descramble a CSS-encrypted DVD sector in place.
///
/// The sector seed (bytes 0x54-0x58) is XORed with the title key to produce
/// the per-sector key. Bytes 0x80..0x800 (128..2048) are then decrypted
/// using the two-LFSR keystream.
/// The two feedback registers are seeded **directly** from
/// `title_key XOR sector_seed` (bytes `0x54..0x59`) — there is no title-key
/// mangling on the content path (that belongs to the disc/title-key hierarchy,
/// not the sector cipher). Only the body, bytes `0x80..0x800`, is transformed:
/// `body[i] = TAB1[body[i]] ^ (keystream & 0xff)`.
///
/// The scramble flag at byte 0x14 (bits 4-5) indicates encryption.
/// After descrambling, the flag is cleared.
/// The scramble flag at byte `0x14` (bits 4-5) marks an encrypted sector. This
/// routine CLEARS that flag after unscrambling, so a descrambled sector reads as
/// `sector[0x14] & 0x30 == 0`; callers and tests use that to tell it from
/// ciphertext, and re-running descramble on an already-cleared sector is a no-op
/// (the flag guard below skips it). Clearing does not affect the recovered body.
///
/// No-op (returns without modifying `sector`) in two cases:
/// - `sector.len() < 2048`: the encrypted region (`0x80..0x800`) is not fully
/// present. Callers chunk by 2048, so a trailing partial chunk is left
/// untouched. The `debug_assert!` flags this misuse in debug/test builds; a
/// DVD sector is always exactly 2048 bytes.
/// - scramble flags are zero: the sector is not CSS-encrypted.
pub fn descramble_sector(title_key: &[u8; 5], sector: &mut [u8]) {
debug_assert!(
sector.len() >= 2048,
"descramble_sector: buffer shorter than one 2048-byte sector"
);
if sector.len() < 2048 {
return;
}
let flags = (sector[0x14] >> 4) & 0x03;
if flags == 0 {
// Not scrambled (flag bits 4-5 clear) → nothing to do.
if sector[0x14] & 0x30 == 0 {
return;
}
// Per-sector key = title_key XOR sector_seed (bytes 0x54-0x58)
let key = [
title_key[0] ^ sector[0x54],
title_key[1] ^ sector[0x55],
title_key[2] ^ sector[0x56],
title_key[3] ^ sector[0x57],
title_key[4] ^ sector[0x58],
];
// LFSR1 halves, seeded from (key ^ seed) bytes 0-1. The 9-bit half carries a
// set bit 8 (`| 0x100`) as its running marker.
let mut r1a: u32 = ((title_key[0] ^ sector[0x54]) as u32) | 0x100;
let mut r1b: u32 = (title_key[1] ^ sector[0x55]) as u32;
// Decrypt the key through the CSS mangling function to get the working key
let working_key = decrypt_key(0xFF, &key, &sector[0x54..0x59]);
// LFSR0 (24-bit), seeded from the remaining three key/seed bytes, then
// pre-conditioned `r0 = r0*2 + 8 - (r0 & 7)`.
let mut r0: u32 = (((title_key[2] as u32)
| ((title_key[3] as u32) << 8)
| ((title_key[4] as u32) << 16))
^ ((sector[0x56] as u32) | ((sector[0x57] as u32) << 8) | ((sector[0x58] as u32) << 16)))
& 0xFF_FFFF;
r0 = r0 * 2 + 8 - (r0 & 7);
// Generate keystream and XOR with encrypted region
let mut lfsr1_lo: u32 = working_key[0] as u32 | 0x100;
let mut lfsr1_hi: u32 = working_key[1] as u32;
// Keystream accumulator; the low byte is the current keystream byte and the
// high bits carry into the next iteration.
let mut acc: u32 = 0;
let mut lfsr0: u32 = ((working_key[4] as u32) << 17)
| ((working_key[3] as u32) << 9)
| (((working_key[2] as u32) << 1) + 8 - (working_key[2] as u32 & 7));
lfsr0 = (TAB4[(lfsr0 & 0xFF) as usize] as u32) << 24
| (TAB4[((lfsr0 >> 8) & 0xFF) as usize] as u32) << 16
| (TAB4[((lfsr0 >> 16) & 0xFF) as usize] as u32) << 8
| TAB4[((lfsr0 >> 24) & 0xFF) as usize] as u32;
let mut combined: u32 = 0;
// Generate 1920 keystream bytes (for sector bytes 128..2048)
// Per libdvdcss css_unscramble: TAB1 permutation on ciphertext, no invert on LFSR0
for byte in sector.iter_mut().take(2048).skip(128) {
let o_lfsr1 = TAB2[lfsr1_hi as usize] ^ TAB3[lfsr1_lo as usize];
lfsr1_hi = lfsr1_lo >> 1;
lfsr1_lo = ((lfsr1_lo & 1) << 8) ^ o_lfsr1 as u32;
// Step LFSR1: its output byte `o1`.
let mut o1 = (TAB2[r1b as usize] ^ TAB3[r1a as usize]) as u32;
r1b = r1a >> 1;
r1a = ((r1a & 1) << 8) ^ o1;
o1 = TAB5[o1 as usize] as u32;
let o_lfsr0 = (((((((lfsr0 >> 8) ^ lfsr0) >> 1) ^ lfsr0) >> 3) ^ lfsr0) >> 7) as u8;
lfsr0 = (lfsr0 >> 8) | ((o_lfsr0 as u32) << 24);
// Step LFSR0: its output byte `o0`.
let mut o0 = (((((((r0 >> 3) ^ r0) >> 1) ^ r0) >> 8) ^ r0) >> 5) & 0xFF;
r0 = (r0 << 8) | o0;
o0 = TAB4[o0 as usize] as u32;
combined += TAB5[o_lfsr1 as usize] as u32 + TAB4[o_lfsr0 as usize] as u32;
*byte ^= (combined & 0xFF) as u8;
combined >>= 8;
// Combine (sum with carry) and recover the plaintext byte.
acc += o0 + o1;
*byte = TAB1[*byte as usize] ^ (acc & 0xFF) as u8;
acc >>= 8;
}
// Clear scramble flags
// Clear the scramble bits so downstream code and tests can tell a sector was
// descrambled; bits 6-7 of byte 0x14 are preserved.
sector[0x14] &= 0xCF;
}
/// CSS key decryption / mangling function.
/// Exact inverse of [`descramble_sector`]: turn a plaintext sector body into
/// CSS ciphertext under `title_key`.
///
/// Decrypts `p_crypted` using `p_key` with the CSS two-LFSR cipher.
/// The `invert` parameter controls the XOR applied to LFSR0 output
/// (0x00 for disc key decryption, 0xFF for title key / sector key).
pub(crate) fn decrypt_key(invert: u8, p_key: &[u8; 5], p_crypted: &[u8]) -> [u8; 5] {
if p_crypted.len() < 5 {
return *p_key;
/// Descramble computes `plain = TAB1[cipher] ^ (keystream & 0xff)`, so the
/// inverse is `cipher = TAB1_INV[plain ^ (keystream & 0xff)]` with the identical
/// keystream. The keystream derivation is the same as [`descramble_sector`];
/// only the final substitution differs. Bytes `0x80..0x800` are rewritten in
/// place; the scramble flag is set to `0x10` so a subsequent descramble runs.
///
/// Not on any production read path — it exists so the key-recovery tests (and
/// any caller that needs a known CSS-encrypted sector) can build genuine
/// ciphertext rather than approximating it.
#[cfg(test)]
pub(crate) fn scramble_sector(title_key: &[u8; 5], sector: &mut [u8]) {
if sector.len() < 2048 {
return;
}
let mut lfsr1_lo: u32 = p_key[0] as u32 | 0x100;
let mut lfsr1_hi: u32 = p_key[1] as u32;
let mut r1a: u32 = ((title_key[0] ^ sector[0x54]) as u32) | 0x100;
let mut r1b: u32 = (title_key[1] ^ sector[0x55]) as u32;
let mut r0: u32 = (((title_key[2] as u32)
| ((title_key[3] as u32) << 8)
| ((title_key[4] as u32) << 16))
^ ((sector[0x56] as u32) | ((sector[0x57] as u32) << 8) | ((sector[0x58] as u32) << 16)))
& 0xFF_FFFF;
r0 = r0 * 2 + 8 - (r0 & 7);
let mut lfsr0: u32 = ((p_key[4] as u32) << 17)
| ((p_key[3] as u32) << 9)
| (((p_key[2] as u32) << 1) + 8 - (p_key[2] as u32 & 7));
lfsr0 = (TAB4[(lfsr0 & 0xFF) as usize] as u32) << 24
| (TAB4[((lfsr0 >> 8) & 0xFF) as usize] as u32) << 16
| (TAB4[((lfsr0 >> 16) & 0xFF) as usize] as u32) << 8
| TAB4[((lfsr0 >> 24) & 0xFF) as usize] as u32;
let mut acc: u32 = 0;
let mut combined: u32 = 0;
let mut k = [0u8; 5];
for byte in sector.iter_mut().take(2048).skip(128) {
let mut o1 = (TAB2[r1b as usize] ^ TAB3[r1a as usize]) as u32;
r1b = r1a >> 1;
r1a = ((r1a & 1) << 8) ^ o1;
o1 = TAB5[o1 as usize] as u32;
for byte in &mut k {
let o_lfsr1 = TAB2[lfsr1_hi as usize] ^ TAB3[lfsr1_lo as usize];
lfsr1_hi = lfsr1_lo >> 1;
lfsr1_lo = ((lfsr1_lo & 1) << 8) ^ o_lfsr1 as u32;
let mut o0 = (((((((r0 >> 3) ^ r0) >> 1) ^ r0) >> 8) ^ r0) >> 5) & 0xFF;
r0 = (r0 << 8) | o0;
o0 = TAB4[o0 as usize] as u32;
acc += o0 + o1;
let o_lfsr0 = (((((((lfsr0 >> 8) ^ lfsr0) >> 1) ^ lfsr0) >> 3) ^ lfsr0) >> 7) as u8;
lfsr0 = (lfsr0 >> 8) | ((o_lfsr0 as u32) << 24);
// TAB5 for LFSR1 output, TAB4 for LFSR0^invert (per libdvdcss css_DecryptKey)
combined += TAB5[o_lfsr1 as usize] as u32 + TAB4[(o_lfsr0 ^ invert) as usize] as u32;
*byte = (combined & 0xFF) as u8;
combined >>= 8;
// Inverse of `*p = TAB1[*p] ^ ks`: apply ks then TAB1's inverse.
*byte = (*TAB1_INV)[(*byte ^ (acc & 0xFF) as u8) as usize];
acc >>= 8;
}
// Two rounds of chained XOR through TAB1
let mut result = [0u8; 5];
result[4] = k[4] ^ TAB1[p_crypted[4] as usize] ^ p_crypted[3];
result[3] = k[3] ^ TAB1[p_crypted[3] as usize] ^ p_crypted[2];
result[2] = k[2] ^ TAB1[p_crypted[2] as usize] ^ p_crypted[1];
result[1] = k[1] ^ TAB1[p_crypted[1] as usize] ^ p_crypted[0];
result[0] = k[0] ^ TAB1[p_crypted[0] as usize] ^ result[4];
result[4] = k[4] ^ TAB1[result[4] as usize] ^ result[3];
result[3] = k[3] ^ TAB1[result[3] as usize] ^ result[2];
result[2] = k[2] ^ TAB1[result[2] as usize] ^ result[1];
result[1] = k[1] ^ TAB1[result[1] as usize] ^ result[0];
result[0] = k[0] ^ TAB1[result[0] as usize];
result
// Mark the sector scrambled so the descrambler will process it.
sector[0x14] = (sector[0x14] & 0xCF) | 0x10;
}
/// Inverse permutation of [`TAB1`], built at first use. `TAB1` is a bijection on
/// `0..256`, so `TAB1_INV[TAB1[x]] == x`.
#[cfg(test)]
static TAB1_INV: std::sync::LazyLock<[u8; 256]> = std::sync::LazyLock::new(|| {
let mut inv = [0u8; 256];
for (i, &v) in TAB1.iter().enumerate() {
inv[v as usize] = i as u8;
}
inv
});
#[cfg(test)]
mod tests {
use super::*;
@@ -144,6 +170,35 @@ mod tests {
assert_eq!(sector, original);
}
/// Regression vector: the deterministic output of the CSS content cipher for
/// a fixed key/seed/body. The value is generated by this implementation and
/// is self-consistent with the scramble/descramble round-trip below — any
/// correct CSS descrambler yields the same bytes, since the cipher is
/// deterministic. Pins the implementation against accidental change.
///
/// key = 42 13 37 BE EF, seed (0x54..0x59) = DE AD BE EF 42, body = 0xAA.
#[test]
fn descramble_produces_the_reference_css_vector() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0xAAu8; 2048];
sector[0x14] = 0x30;
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
descramble_sector(&key, &mut sector);
assert_eq!(
&sector[0x80..0x90],
&[
0x81, 0x92, 0x24, 0xA2, 0x46, 0x70, 0x3C, 0x64, 0xA6, 0x91, 0x84, 0xF5, 0x1F, 0x98,
0xA0, 0x31
],
"descramble body head must match the reference CSS vector"
);
assert_eq!(
&sector[0x7F8..0x800],
&[0x46, 0x94, 0x80, 0x0E, 0x67, 0x36, 0x65, 0xBC],
"descramble body tail must match the reference CSS vector"
);
}
#[test]
fn descramble_modifies_scrambled() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
@@ -174,67 +229,14 @@ mod tests {
assert_eq!(sector[0x14] & 0x30, 0x00);
}
#[test]
fn decrypt_key_produces_output() {
let key = [0x12, 0x34, 0x56, 0x78, 0x9A];
let crypted = [0xAB, 0xCD, 0xEF, 0x01, 0x23];
let result = decrypt_key(0xFF, &key, &crypted);
// Should produce a 5-byte result different from input
assert_ne!(result, key);
assert_ne!(result, [0u8; 5]);
}
/// Test 1: css_decrypt_key_roundtrip
/// Test 2: descramble inverts scramble over the body.
///
/// decrypt_key is not a simple encrypt/decrypt pair — it is a one-way mangling
/// function. However, we can verify consistency: calling it twice with the same
/// parameters produces the same output, and varying the invert byte changes
/// the LFSR0 contribution predictably.
/// The content cipher is NOT a plain XOR involution (it applies TAB1 to the
/// ciphertext: `plain = TAB1[cipher] ^ ks`). The true inverse is
/// [`scramble_sector`]. Scrambling a plaintext body and then descrambling
/// with the same key must reproduce the original body exactly.
#[test]
fn css_decrypt_key_roundtrip() {
let keys: &[[u8; 5]] = &[
[0x12, 0x34, 0x56, 0x78, 0x9A],
[0x00, 0x00, 0x00, 0x00, 0x00],
[0xFF, 0xFF, 0xFF, 0xFF, 0xFF],
[0xAB, 0xCD, 0xEF, 0x01, 0x23],
];
let crypted_inputs: &[[u8; 5]] = &[
[0x11, 0x22, 0x33, 0x44, 0x55],
[0xAA, 0xBB, 0xCC, 0xDD, 0xEE],
[0x00, 0x00, 0x00, 0x00, 0x00],
];
for key in keys {
for crypted in crypted_inputs {
// decrypt_key with invert=0x00 and invert=0xFF should give different results
let r0 = decrypt_key(0x00, key, crypted);
let rff = decrypt_key(0xFF, key, crypted);
// The two results differ because the invert byte XORs the LFSR0 output
// They should not be equal (except by extreme coincidence)
// More importantly, both should be deterministic
let r0_again = decrypt_key(0x00, key, crypted);
let rff_again = decrypt_key(0xFF, key, crypted);
assert_eq!(r0, r0_again, "decrypt_key(0x00) not deterministic");
assert_eq!(rff, rff_again, "decrypt_key(0xFF) not deterministic");
// With different invert values, the keystream differs
assert_ne!(
r0, rff,
"invert=0x00 and 0xFF gave same result for key {:?}",
key
);
}
}
}
/// Test 2: css_descramble_produces_valid_mpeg2
///
/// descramble_sector XORs a keystream into bytes 128..2048. Calling it
/// twice with the same key and restored scramble flag should roundtrip,
/// since XOR is its own inverse.
#[test]
fn css_descramble_modifies_encrypted_region() {
fn css_descramble_inverts_scramble_over_body() {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0xAAu8; 2048];
@@ -242,11 +244,10 @@ mod tests {
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
let original = sector.clone();
descramble_sector(&title_key, &mut sector);
// Flag cleared
assert_eq!(sector[0x14] & 0x30, 0x00);
// Header (0..128) unchanged except flag byte
// Scramble the plaintext body into ciphertext.
scramble_sector(&title_key, &mut sector);
// Header (0..128) unchanged except the flag byte (set by scramble).
for i in 0..128 {
if i == 0x14 {
continue;
@@ -255,13 +256,22 @@ mod tests {
}
// Encrypted region modified
assert_ne!(&sector[128..256], &original[128..256]);
// Descramble restores the plaintext body byte-for-byte.
descramble_sector(&title_key, &mut sector);
assert_eq!(sector[0x14] & 0x30, 0x00, "flag cleared after descramble");
assert_eq!(
&sector[128..2048],
&original[128..2048],
"descramble(scramble(body)) did not restore the body"
);
}
/// Test 4: css_tab1_relationship
/// css_tab1_relationship
///
/// Verify the structure of TAB1: it is a substitution table used in
/// key mangling. Check that no two inputs map to the same output
/// (TAB1 is a permutation of 0..255).
/// Verify the structure of TAB1: it is a substitution table used in key
/// mangling. Check that no two inputs map to the same output (TAB1 is a
/// permutation of 0..255).
#[test]
fn css_tab1_is_permutation() {
let mut seen = [false; 256];
@@ -281,7 +291,7 @@ mod tests {
}
}
/// Test 5: css_tab4_is_bit_reversal
/// css_tab4_is_bit_reversal
///
/// TAB4 reverses the bits of each byte: TAB4[0x01] = 0x80, TAB4[0x80] = 0x01, etc.
#[test]
@@ -303,4 +313,185 @@ mod tests {
);
}
}
// ── scramble-flag detection (byte 0x14, bits 4-5) ──────────────────────
/// Only bits 4-5 of byte 0x14 are the CSS scramble flag: the code reads
/// `sector[0x14] & 0x30 == 0` (bits 6-7, i.e. 0x40/0x80, are masked out by
/// 0x30). A sector with 0x14 == 0x40 or 0x80 must therefore be treated as
/// UNSCRAMBLED and left byte-for-byte unchanged. This guards against a
/// too-wide mask silently "descrambling" (and thus corrupting) clear data.
///
/// Grounding: CSS sector header byte 0x14 — copyright/scramble bits live in
/// bits 4-5; the masked value 0 means not scrambled.
/// Mutation: widen the mask `0x30` to `0x70`/`0xF0` -> 0x40/0x80 would be
/// seen as scrambled and the body would change.
#[test]
fn descramble_treats_high_bits_of_0x14_as_clear() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
for &flag in &[0x40u8, 0x80, 0xC0, 0x0F, 0x4F, 0x8F] {
let mut sector = vec![0xAA; 2048];
sector[0x14] = flag;
sector[0x54..0x59].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
let original = sector.clone();
descramble_sector(&key, &mut sector);
assert_eq!(
sector, original,
"byte 0x14 = {flag:#04x} has flag bits 4-5 clear; sector must be untouched"
);
}
}
/// Each individual scramble bit (4 and 5) independently marks the sector as
/// encrypted: 0x10 and 0x20 must both trigger descrambling.
///
/// Grounding: `(0x10 >> 4) & 3 == 1`, `(0x20 >> 4) & 3 == 2` — both nonzero.
/// Mutation: change `!= 0` early-return condition to `== 3` -> a sector
/// flagged only 0x10 or 0x20 would be skipped and left scrambled.
#[test]
fn descramble_triggers_on_either_flag_bit() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
for &flag in &[0x10u8, 0x20, 0x30] {
let mut sector = vec![0xAA; 2048];
sector[0x14] = flag;
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
let original = sector.clone();
descramble_sector(&key, &mut sector);
assert_ne!(
&sector[128..256],
&original[128..256],
"flag {flag:#04x} (bits 4-5 nonzero) must descramble the body"
);
}
}
/// After descrambling, ONLY the two scramble bits are cleared (`& 0xCF`);
/// bits 6 and 7 of byte 0x14 must be preserved. A sector with 0x14 == 0xF0
/// becomes 0xC0 (bits 6,7 kept, bits 4,5 cleared), NOT 0x00.
///
/// Grounding: code does `sector[0x14] &= 0xCF`; 0xF0 & 0xCF == 0xC0.
/// Mutation: change `&= 0xCF` to `= 0` or `&= 0x0F` -> the preserved high
/// bits assert fails.
#[test]
fn descramble_clear_preserves_high_bits_of_0x14() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let mut sector = vec![0x00; 2048];
sector[0x14] = 0xF0; // bits 4-7 set; bits 4-5 are the flag
sector[0x54..0x59].copy_from_slice(&[0x00; 5]);
descramble_sector(&key, &mut sector);
assert_eq!(
sector[0x14], 0xC0,
"scramble bits cleared, bits 6-7 preserved (0xF0 & 0xCF)"
);
}
// ── header / body boundary (encrypted region is 0x80..0x800) ───────────
/// The encrypted region is exactly bytes 0x80..0x800. Bytes 0x00..0x80 (the
/// header) must NOT be modified by the keystream — except byte 0x14 whose
/// flag is cleared. In particular the sector-seed bytes 0x54..0x59 (which
/// live inside the header) must survive untouched, since the descrambler
/// reads them but never writes them.
///
/// Grounding: loop is `sector.iter_mut().take(2048).skip(128)` -> indices
/// 128..2048 only.
/// Mutation: change `.skip(128)` to `.skip(0)` -> header bytes (incl. the
/// seed) get XORed and this fails.
#[test]
fn descramble_leaves_header_and_seed_intact() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0x5Au8; 2048];
sector[0x14] = 0x30;
let seed = [0xDE, 0xAD, 0xBE, 0xEF, 0x42];
sector[0x54..0x59].copy_from_slice(&seed);
let original = sector.clone();
descramble_sector(&key, &mut sector);
for i in 0..0x80usize {
if i == 0x14 {
continue;
}
assert_eq!(
sector[i], original[i],
"header byte {i:#04x} must be untouched"
);
}
assert_eq!(&sector[0x54..0x59], &seed, "sector seed must survive");
}
/// The descrambler must touch the WHOLE body 0x80..0x800, not just a prefix.
/// With a constant body and constant key, the keystream is non-degenerate
/// enough that the very last sector byte (index 2047) is altered. This guards
/// the loop bound `.take(2048)` against an off-by-one that would leave the
/// final byte(s) scrambled.
///
/// Grounding: encrypted region end is 0x800 == 2048 (exclusive).
/// Mutation: change `.take(2048)` to `.take(2047)` -> last byte unchanged,
/// assert fires (this body is all-zero so any keystream XOR shows).
#[test]
fn descramble_covers_final_body_byte() {
let key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let mut sector = vec![0x00u8; 2048];
sector[0x14] = 0x30;
sector[0x54..0x59].copy_from_slice(&[0x11, 0x22, 0x33, 0x44, 0x55]);
descramble_sector(&key, &mut sector);
// Body was all zero; any nonzero in [0x80,0x800) is keystream. Confirm
// the keystream reaches the final byte.
assert_ne!(
&sector[2040..2048],
&[0u8; 8][..],
"the tail of the body must be descrambled (loop must reach index 2047)"
);
}
/// Descramble is keyed by `title_key XOR seed`: two different title keys
/// produce two different bodies for the same scrambled input. A cipher that
/// ignored the title key (or mixed it in wrongly) would yield identical
/// output — silent wrong-key decryption.
///
/// Grounding: per-sector key = title_key[i] ^ sector[0x54+i].
/// Mutation: in the `key` array drop the `title_key[i] ^` term -> both keys
/// give the same body, assert fires.
#[test]
fn descramble_output_depends_on_title_key() {
let seed = [0xDE, 0xAD, 0xBE, 0xEF, 0x42];
let make = |k: &[u8; 5]| {
let mut s = vec![0x00u8; 2048];
s[0x14] = 0x30;
s[0x54..0x59].copy_from_slice(&seed);
descramble_sector(k, &mut s);
s
};
let a = make(&[0x01, 0x02, 0x03, 0x04, 0x05]);
let b = make(&[0x01, 0x02, 0x03, 0x04, 0x06]); // differs in last byte
assert_ne!(
&a[128..2048],
&b[128..2048],
"different title keys must descramble differently"
);
}
/// Descramble is keyed by the sector seed too: same title key, different
/// seed -> different body. Pins that bytes 0x54..0x59 actually feed the
/// keystream (not just the per-sector XOR key).
///
/// Mutation: replace `seed` array reads with a constant -> both seeds give
/// the same body, assert fires.
#[test]
fn descramble_output_depends_on_seed() {
let key = [0x01, 0x02, 0x03, 0x04, 0x05];
let make = |seed: [u8; 5]| {
let mut s = vec![0x00u8; 2048];
s[0x14] = 0x30;
s[0x54..0x59].copy_from_slice(&seed);
descramble_sector(&key, &mut s);
s
};
let a = make([0x11, 0x22, 0x33, 0x44, 0x55]);
let b = make([0x11, 0x22, 0x33, 0x44, 0x56]);
assert_ne!(
&a[128..2048],
&b[128..2048],
"different seeds must descramble differently"
);
}
}
+898 -38
View File
@@ -1,65 +1,269 @@
//! CSS (Content Scramble System) — DVD disc encryption.
//!
//! CSS uses a weak 40-bit LFSR stream cipher (broken since 1999).
//! No keys needed — the title key is cracked from encrypted content
//! using a known-plaintext attack on MPEG-2 PES headers.
//!
//! The title key is recovered keylessly: [`crack_key`] runs the Stevenson
//! known-plaintext attack (see the [`stevenson`] module) on the scrambled
//! data, needing no player keys, disc-key crack, or external key file.
//! Sectors are then decrypted with [`descramble_sector`].
//!
//! Usage:
//! ```rust,ignore
//! let key = css::crack_key(reader, &extents)?;
//! css::descramble_sector(&key, &mut sector);
//! if let Some(state) = css::crack_key(reader, extents, batch) {
//! css::descramble_sector(&state, &mut sector);
//! }
//! ```
pub mod auth;
pub mod crack;
pub mod lfsr;
pub mod stevenson;
pub(crate) mod tables;
use crate::disc::Extent;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
/// Consecutive CSS-locked (`05/6F/03`) reads before the crack scan early-bails.
/// The bus-auth read gate is global (all-or-nothing), so a run this long means
/// it is shut and nothing here is crackable — bail instead of grinding the full
/// 50_000-sector budget (which is what made rc5 appear to hang on a wedged USB
/// bridge). The counter resets to 0 on any readable batch.
const CSS_LOCKED_BAIL: u32 = 64;
/// CSS decryption state for a DVD title.
#[derive(Debug, Clone)]
pub struct CssState {
/// Cracked 5-byte title key
/// 5-byte CSS title key (from SCSI auth or the crack fallback).
pub title_key: [u8; 5],
/// LBA half-open span `[start, end)` of the extent set this key was
/// cracked from. CSS title keys are per-VTS: a key cracked from one
/// VTS does NOT descramble a title living in a different VTS. The mux
/// path checks whether the title being opened overlaps this span; if
/// not, it re-cracks from that title's own extents. `None` for keys
/// of unknown provenance (e.g. test fixtures) — treated as "applies
/// everywhere" for backward compatibility.
pub crack_span: Option<(u32, u32)>,
}
/// Crack the CSS title key by reading encrypted sectors and applying
/// a known-plaintext attack on MPEG-2 headers.
/// Recover the CSS title key with no keys, by scanning scrambled sectors and
/// running the Stevenson known-plaintext attack (see the [`stevenson`] module).
///
/// Crack the CSS title key by scanning scrambled sectors across extents.
///
/// The Stevenson attack needs a sector where a PES header starts at byte
/// 0x80 (start of the encrypted region). This only happens when a new PES
/// packet begins at exactly sector offset 128. We scan up to 50000
/// scrambled sectors sequentially across all extents.
pub fn crack_key(reader: &mut dyn SectorReader, extents: &[Extent]) -> Option<CssState> {
let mut tried = 0u32;
let max_tries = 50_000;
/// The crib comes from the periodic-run detector: a scrambled sector's cleartext region
/// (bytes 0x00..0x80) often ends in a short-period repeating run (stuffing /
/// constant fill); the attack assumes that run continues across the 0x80
/// boundary into the encrypted region, giving the known plaintext the 2^16
/// LFSR recovery needs. We scan up to 50000 sectors across the
/// extents and return the first sector that yields a key — no player keys, no
/// disc-key crack. Works on a live drive (after bus-auth unlocks reads) and on
/// disc images alike.
pub fn crack_key(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
) -> Option<CssState> {
crack_key_halt(reader, extents, batch_sectors, None)
}
for ext in extents {
let mut i = 0;
while i < ext.sector_count && tried < max_tries {
let mut buf = vec![0u8; 2048];
if reader
.read_sectors(ext.start_lba + i, 1, &mut buf, true)
.is_ok()
&& is_scrambled(&buf)
{
if let Some(key) = crack::crack_title_key(&buf) {
return Some(CssState { title_key: key });
}
tried += 1;
}
i += 1;
}
if tried >= max_tries {
break;
/// Outcome of a CSS crack scan that distinguishes the THREE cases the bare
/// `Option<CssState>` conflated (and which caused a silent-failure bug:
/// scrambled-but-uncracked content was treated as "unencrypted" and muxed as
/// plaintext garbage at exit 0):
///
/// - [`CrackOutcome::Cracked`] — a scrambled sector yielded a title key.
/// - [`CrackOutcome::Unencrypted`] — NO scrambled sector was seen across the
/// scanned extents (`is_scrambled` never true): the content is genuinely
/// plaintext, so proceeding without a key is correct.
/// - [`CrackOutcome::ScrambledUncracked`] — scrambled sectors WERE seen but no
/// key could be recovered (the Stevenson attack found no crackable crib, or
/// the scrambled region was unreadable). The content is encrypted; muxing it
/// as plaintext would emit garbage, so callers MUST surface a hard error
/// ([`crate::error::Error::CssKeyMissing`]) instead of falling through to
/// "unencrypted".
#[derive(Debug, Clone)]
pub enum CrackOutcome {
Cracked(CssState),
Unencrypted,
ScrambledUncracked,
}
impl CrackOutcome {
/// The cracked `CssState`, if any. `None` for `Unencrypted` /
/// `ScrambledUncracked`. Lets the `Option`-returning wrappers stay thin.
pub fn into_state(self) -> Option<CssState> {
match self {
CrackOutcome::Cracked(s) => Some(s),
_ => None,
}
}
None
/// True when scrambled sectors were seen but no key was recovered — the
/// case callers must surface as a hard error instead of "unencrypted".
pub fn is_scrambled_uncracked(&self) -> bool {
matches!(self, CrackOutcome::ScrambledUncracked)
}
}
/// [`crack_key`] returning the full [`CrackOutcome`] (Cracked / Unencrypted /
/// ScrambledUncracked) so callers can distinguish "genuinely unencrypted" from
/// "encrypted but uncrackable" — the latter must become a hard error, never a
/// silent fall-through to plaintext.
pub fn crack_key_outcome(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
) -> CrackOutcome {
crack_key_scan(reader, extents, batch_sectors, halt, true)
}
/// [`crack_key`] with an optional cooperative-cancellation token.
///
/// "No silent hangs": the crack scans up to 50_000 sectors, which on a live
/// drive hitting bad sectors can take a long time. This variant polls `halt`
/// once per batch (the same cadence sweep/patch use) so an operator Stop or a
/// scan-level watchdog can interrupt the scan, and emits a
/// `freemkv::heartbeat` beat ("css_crack") each batch so a stuck scan is
/// visible in the log.
pub fn crack_key_halt(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
) -> Option<CssState> {
crack_key_scan(reader, extents, batch_sectors, halt, false).into_state()
}
/// The crack scan, returning the full [`CrackOutcome`]. Tracks a
/// `saw_scrambled` flag so a scrambled-but-uncracked disc is distinguished
/// from a genuinely-unencrypted one (the [`crack_key`] / [`crack_key_halt`]
/// `Option` wrappers collapse both to `None`).
fn crack_key_scan(
reader: &mut dyn SectorSource,
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
// True only on the INITIAL scan: a fully CSS-locked (`05/6F/03`) result is a
// hard `ScrambledUncracked`. False on the per-VTS re-crack so a lapsed-AGID
// locked read returns None instead of killing a genuinely crackable title.
fail_on_locked: bool,
) -> CrackOutcome {
// Batch the reads: a live optical drive at 1 sector/read is glacial, and the
// crack only needs to FIND one scrambled sector whose 0x80 plaintext matches
// a known PES header. `batch_sectors` MUST be sized to the source — a drive
// rejects a READ(10) larger than its per-command max (DVD = 16) and
// `Drive::read` does not chunk, so an over-large batch fails every read and
// scans nothing. Callers pass `detect_max_batch_sectors(device_path)` for a
// live drive, a file-safe value for an image, or 1 to force per-sector.
let batch = (batch_sectors.max(1)) as u32;
// Record the LBA span the key is being cracked from so the per-title mux
// path can tell whether a later title lives in the same VTS (overlaps the
// span → key applies) or a different one (→ re-crack). Half-open [min,max).
let crack_span = extents
.iter()
.filter(|e| e.sector_count > 0)
.map(|e| (e.start_lba, e.start_lba.saturating_add(e.sector_count)))
.reduce(|(amin, amax), (bmin, bmax)| (amin.min(bmin), amax.max(bmax)));
let mut tried = 0u32;
let max_tries = 50_000u32;
let mut buf = vec![0u8; batch as usize * 2048];
let mut hb = crate::progress::Heartbeat::new("css_crack");
// Track whether ANY scrambled sector was observed. If we exhaust the scan
// budget having seen scrambled data but never recovered a key, the content
// is encrypted-but-uncrackable — a HARD failure the caller must surface,
// NOT silently treat as unencrypted (which would mux scrambled MPEG as
// plaintext → garbage at exit 0). See `CrackOutcome::ScrambledUncracked`.
let mut saw_scrambled = false;
// A read rejected with sense `05/6F/03` ("scrambled sector without
// authentication") is positive proof of CSS encryption — never collapse it
// to "unencrypted". A run of consecutive locked reads means the bus-auth
// gate is shut (it is global, so reads are all-or-nothing), so the scan
// early-bails. `consecutive_locked` resets on any readable batch, so a
// crackable title (gate open) never trips it.
let mut saw_locked = false;
let mut consecutive_locked = 0u32;
'outer: for (extent_idx, ext) in extents.iter().enumerate() {
let mut i = 0u32;
while i < ext.sector_count && tried < max_tries {
// Cooperative cancellation — poll once per batch, the same cadence
// sweep/patch use, so a Stop / watchdog can interrupt the scan.
if let Some(h) = halt {
if h.is_cancelled() {
break 'outer;
}
}
// Liveness beacon: a long scan over a damaged disc stays visible.
// The heartbeat is time-throttled; only when it actually beats do
// we emit the crack-specific context (tried/lba/extent_idx).
if hb.tick(tried as u64, max_tries as u64) {
tracing::debug!(
target: "freemkv::heartbeat",
phase = "css_crack",
tried,
lba = ext.start_lba + i,
extent_idx,
"scanning"
);
}
let n = (ext.sector_count - i).min(batch);
let want = n as usize * 2048;
match reader.read_sectors(ext.start_lba + i, n as u16, &mut buf[..want], true) {
Ok(_) => {
// A readable batch: the gate is open — reset the locked run.
consecutive_locked = 0;
for s in 0..n as usize {
tried += 1;
let sect = &buf[s * 2048..(s + 1) * 2048];
// Use the HARDENED pack-gated check (Fix 3): a clear stub
// sector with stray bits at 0x14 must NOT count as
// scramble evidence, or a genuinely-unencrypted title
// would falsely report ScrambledUncracked (a false E7023).
if is_scrambled_pack(sect) {
saw_scrambled = true;
if let Some(key) = stevenson::crack_title_key(sect) {
return CrackOutcome::Cracked(CssState {
title_key: key,
crack_span,
});
}
}
if tried >= max_tries {
break 'outer;
}
}
}
// A failed batch still counts toward the budget so a damaged
// region can't loop forever. A CSS-locked failure (`05/6F/03`)
// proves encryption and, in a long enough run, means the read
// gate is shut — track it and early-bail rather than grind.
Err(e) => {
tried += n;
if e.scsi_sense().is_some_and(|s| s.is_css_locked()) {
saw_locked = true;
consecutive_locked += 1;
if consecutive_locked >= CSS_LOCKED_BAIL {
break 'outer;
}
} else {
consecutive_locked = 0;
}
}
}
i += n;
}
}
// Budget exhausted / extents walked / early-bailed with no key recovered.
// The disc is ENCRYPTED-but-uncracked (a hard failure on the initial scan)
// when EITHER a scrambled sector was actually seen, OR — on the initial scan
// only (`fail_on_locked`) — every read was CSS-locked (`05/6F/03`), itself
// proof of scrambling. A re-crack (`fail_on_locked` false) stays soft: a
// lapsed-AGID locked read yields None, not a hard fail, so a crackable title
// in another VTS isn't killed. Only a scan that saw neither a scrambled
// sector nor a CSS-lock is genuinely unencrypted.
if saw_scrambled || (saw_locked && fail_on_locked) {
CrackOutcome::ScrambledUncracked
} else {
CrackOutcome::Unencrypted
}
}
/// Descramble a single CSS-encrypted sector in place.
@@ -67,7 +271,663 @@ pub fn descramble_sector(state: &CssState, sector: &mut [u8]) {
lfsr::descramble_sector(&state.title_key, sector);
}
/// Descramble a whole CSS buffer in place, re-cracking the title key on a VOB
/// region boundary. `title_key` is a CACHE of the last crack, not a fixed disc
/// key: it changes per VTS/VOB region, so it is validated on every scrambled
/// sector and re-cracked on a miss (the standard on-demand per-region rekey).
///
/// This CSS key acquisition is intrinsic to the cipher — CSS has no external key
/// source, the ONLY way to a title key is cracking the data — so it lives with
/// the CSS primitives and runs inside `decrypt::decrypt_sectors` (a public,
/// self-contained CSS decrypt), NOT at the post-decrypt recovery seam that AACS
/// key-fetch and FMTS segment-skip use (those consume external inputs).
///
/// The clear header (`<0x80`) is never scrambled, so its periodic crib predicts
/// the plaintext at `0x80`. Descramble with the cached key; if the crib fails to
/// reappear the key region changed (or the primed key was wrong) — restore the
/// ciphertext, re-crack from this very sector, and descramble again. A crib-less
/// sector (no periodic run) can be neither validated nor cracked, so it rides the
/// cached key — correct, because it lives in the same region as the nearby crib
/// sector that set the cache.
pub fn descramble_region(buf: &mut [u8], title_key: &mut [u8; 5]) {
for chunk in buf.chunks_mut(2048) {
if chunk.len() < 2048 || !is_scrambled(chunk) {
continue;
}
let crib = stevenson::attack_crib(chunk);
// Snapshot the ciphertext (chunk is exactly 2048 here) only when there is
// a crib to validate against, so the common cache-hit path costs no
// per-sector heap allocation.
let mut original = [0u8; 2048];
if crib.is_some() {
original.copy_from_slice(chunk);
}
lfsr::descramble_sector(title_key, chunk);
if let Some(crib) = crib {
if chunk[0x80..0x80 + 10] != crib[..] {
// Cached key is stale for this region — restore the ciphertext and
// crack this sector's own key.
chunk.copy_from_slice(&original);
if let Some(fresh) = stevenson::crack_title_key(chunk) {
*title_key = fresh;
}
lfsr::descramble_sector(title_key, chunk);
}
}
}
}
/// Check if a sector has the CSS scramble flag set.
///
/// This is the RAW flag test — bits 4-5 of the sub-header byte 0x14 — used by
/// the descramble loop (`decrypt::decrypt_sectors`), which has already committed
/// to descrambling a known title's VOB data and only needs to skip the clear
/// NAV packs interleaved in it. For the CRACK SCAN's "did this disc actually
/// contain scrambled content?" decision (which must not false-positive on a
/// clear stub), use [`is_scrambled_pack`] instead.
pub fn is_scrambled(sector: &[u8]) -> bool {
sector.len() >= 2048 && (sector[0x14] >> 4) & 0x03 != 0
}
/// The 4-byte MPEG-2 Program Stream pack-start code (`00 00 01 BA`) every DVD
/// video sector opens with. CSS leaves the clear header (`0x00..0x80`)
/// untouched, so this signature survives scrambling.
pub(crate) const PACK_START: [u8; 4] = [0x00, 0x00, 0x01, 0xBA];
/// Check if a sector is a CSS-scrambled DVD **video pack** — the HARDENED test
/// the crack scan uses to set its `saw_scrambled` evidence flag (Fix 3).
///
/// [`is_scrambled`] keys solely on bits 4-5 of byte 0x14. That single byte is
/// only meaningful inside a real DVD sector — an MPEG-2 Program Stream pack,
/// which ALWAYS begins with the 32-bit pack-start code `00 00 01 BA` at offset
/// 0x00. A tiny clear / nav-only stub (a 0.5 s menu loop, an FBI-warning title)
/// can carry arbitrary bytes that happen to set bits 4-5 of byte 0x14; trusting
/// byte 0x14 alone there would flip the scan's `saw_scrambled` gate and make a
/// genuinely-UNENCRYPTED title report `ScrambledUncracked` — a false E7023.
///
/// Requiring the pack-start signature FIRST means only a sector that is
/// structurally a DVD video pack can be counted as scramble evidence. This does
/// NOT weaken the genuine "encrypted but uncrackable" hard-fail: a real
/// scrambled feature is made of valid PS packs, so its scrambled sectors still
/// pass this check and still drive `ScrambledUncracked` when no key cracks. (The
/// descramble loop keeps the looser [`is_scrambled`]: by the time it runs we
/// already know the title is CSS, and it only needs to skip interleaved clear
/// NAV packs — a wrongly-skipped or wrongly-included sector there is recoverable
/// per-sector, whereas a false scramble verdict in the scan poisons the whole
/// title's outcome.)
pub fn is_scrambled_pack(sector: &[u8]) -> bool {
sector.len() >= 2048 && sector[0x00..0x04] == PACK_START && (sector[0x14] >> 4) & 0x03 != 0
}
#[cfg(test)]
mod tests {
use super::*;
use crate::error::{Error, Result};
// ── is_scrambled ───────────────────────────────────────────────────────
/// is_scrambled returns false for any buffer shorter than one sector,
/// WITHOUT indexing byte 0x14 (which would panic on a tiny buffer). The
/// length guard is short-circuited before the flag read.
///
/// Grounding: `sector.len() >= 2048 && (sector[0x14] >> 4) & 0x03 != 0` —
/// `&&` short-circuits so a 20-byte buffer never reads index 0x14.
/// Mutation: swap the operands so the flag is read first
/// (`(sector[0x14]...) && sector.len() >= 2048`) -> panics indexing a
/// 20-byte slice; this test catches it.
#[test]
fn is_scrambled_short_buffer_is_false_no_panic() {
assert!(!is_scrambled(&[]));
assert!(!is_scrambled(&[0u8; 20])); // shorter than 0x14+1 even
assert!(!is_scrambled(&[0xFFu8; 2047])); // one byte short of a sector
}
/// is_scrambled keys on bits 4-5 of byte 0x14 (the CSS scramble field).
/// A full sector flagged 0x10/0x20/0x30 is scrambled; 0x00 and the
/// high-bit-only values 0x40/0x80 are clear.
///
/// Grounding: `(sector[0x14] >> 4) & 0x03`.
/// Mutation: widen mask to `& 0x0F` -> 0x40 reports scrambled, the 0x40
/// assert fails.
#[test]
fn is_scrambled_uses_bits_4_5_only() {
let mut s = vec![0u8; 2048];
for (flag, expected) in [
(0x00u8, false),
(0x10, true),
(0x20, true),
(0x30, true),
(0x40, false),
(0x80, false),
(0xC0, false),
(0xFF, true), // bits 4-5 set within 0xFF
] {
s[0x14] = flag;
assert_eq!(
is_scrambled(&s),
expected,
"flag byte {flag:#04x} scramble detection"
);
}
}
/// is_scrambled accepts exactly 2048 bytes as the minimum (boundary at the
/// inclusive value 2048).
///
/// Grounding: `sector.len() >= 2048`.
/// Mutation: change `>= 2048` to `> 2048` -> an exact 2048-byte scrambled
/// sector reports false; this fails.
#[test]
fn is_scrambled_exact_sector_length_accepted() {
let mut s = vec![0u8; 2048];
s[0x14] = 0x30;
assert!(is_scrambled(&s), "exactly 2048 bytes must be eligible");
}
/// Fix 3 hardening: `is_scrambled_pack` (the crack-scan evidence gate)
/// requires BOTH the MPEG-PS pack-start code at 0x00 AND the 0x14 scramble
/// bits. A clear / nav-only stub whose bytes happen to set bits 4-5 of 0x14
/// but lacks the pack-start is NOT counted as scramble evidence — without
/// this the scan flips `saw_scrambled` and a genuinely unencrypted title
/// reports `ScrambledUncracked` (the false E7023). The looser `is_scrambled`
/// (descramble gate) still reads the same sector as flagged.
///
/// Grounding: `sector[0x00..0x04] == 00 00 01 BA && (sector[0x14] >> 4)...`.
/// Mutation: drop the pack-start clause -> the 0x14-only sector counts as a
/// scrambled pack; the first assert fails.
#[test]
fn is_scrambled_pack_requires_pack_start_signature() {
let mut s = vec![0u8; 2048];
s[0x14] = 0x30; // scramble bits set, but no pack-start at 0x00
assert!(
!is_scrambled_pack(&s),
"0x14 bits without the MPEG-PS pack-start must NOT count as a scrambled pack"
);
// The looser descramble-gate check still sees the raw flag.
assert!(is_scrambled(&s), "is_scrambled keys on the 0x14 flag alone");
// A near-miss pack-start (wrong final byte) is still rejected.
s[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBB]);
assert!(
!is_scrambled_pack(&s),
"a wrong pack-start byte must not qualify"
);
// The real signature flips it to a scrambled pack.
s[0x00..0x04].copy_from_slice(&PACK_START);
assert!(
is_scrambled_pack(&s),
"valid pack-start + 0x14 bits → scrambled pack"
);
}
// ── crack_key scanning over a mock SectorSource ────────────────────────
/// Records every (lba, count) read; returns a caller-supplied flag byte at
/// 0x14 so we can drive scrambled/clear sectors, or an injected error.
struct MockSource {
reads: std::cell::RefCell<Vec<u32>>,
flag_byte: u8,
fail_all: bool,
/// Every read fails with CSS-locked sense `05/6F/03` (drive refusing
/// scrambled reads because the bus-auth gate isn't open).
lock_all: bool,
/// When set, the sector at `crackable.0` is served as a full
/// Stevenson-crackable scrambled sector (`crackable.1`, 2048 bytes)
/// instead of the uniform `flag_byte` fill. Lets the scan actually
/// reach `CrackOutcome::Cracked` from a synthetic ISO.
crackable: Option<(u32, Vec<u8>)>,
}
impl MockSource {
fn new(flag_byte: u8) -> Self {
Self {
reads: std::cell::RefCell::new(Vec::new()),
flag_byte,
fail_all: false,
lock_all: false,
crackable: None,
}
}
}
/// Build a Stevenson-crackable scrambled sector for `(title_key, seed)`:
/// the cleartext header (0x59..0x80) carries a periodic run that continues
/// across the 0x80 boundary into the encrypted region — the crib
/// `stevenson::crack_title_key` recovers a key from. Mirrors the
/// `synth_periodic_sector` fixture in the stevenson tests but built here
/// from the crate-internal `scramble_sector`.
fn crackable_sector(title_key: &[u8; 5], seed: &[u8; 5], period: usize) -> Vec<u8> {
const RUN_START: usize = 0x59;
const SEED_OFFSET: usize = 0x54;
let mut plaintext = vec![0u8; 2048];
plaintext[0x00..0x04].copy_from_slice(&PACK_START); // valid DVD pack header
plaintext[0x14] = 0x10; // scramble flag
let pat: Vec<u8> = (0..period)
.map(|k| (0xA0u8.wrapping_add(k as u8)) ^ 0x5A)
.collect();
for (i, b) in plaintext.iter_mut().enumerate().skip(RUN_START) {
*b = pat[i % period];
}
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
lfsr::scramble_sector(title_key, &mut plaintext);
plaintext
}
impl SectorSource for MockSource {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
self.reads.borrow_mut().push(lba);
if self.lock_all {
return Err(Error::DiscRead {
sector: lba as u64,
status: Some(2),
sense: Some(crate::scsi::ScsiSense {
sense_key: 0x05,
asc: 0x6F,
ascq: 0x03,
}),
});
}
if self.fail_all {
return Err(Error::DecryptFailed);
}
let n = count as usize * 2048;
let end = n.min(buf.len());
for b in buf[..end].iter_mut() {
*b = 0;
}
// Fill each sector in the batch with the uniform flag byte, EXCEPT a
// designated crackable LBA which gets the full synthetic sector.
for s in 0..count as u32 {
let sect_lba = lba + s;
let base = s as usize * 2048;
if base + 2048 > end {
break;
}
match &self.crackable {
Some((clba, sector)) if *clba == sect_lba => {
buf[base..base + 2048].copy_from_slice(sector);
}
_ => {
// Real DVD video sectors always open with the MPEG-PS
// pack-start code; `is_scrambled` (Fix 3) requires it
// before trusting the 0x14 scramble bits, so the fixture
// must include it for a `flag_byte` of 0x30 to register
// as scrambled.
buf[base..base + 4].copy_from_slice(&PACK_START);
buf[base + 0x14] = self.flag_byte;
}
}
}
Ok(n)
}
}
/// crack_key caps total scanned sectors at 50_000 even when extents are
/// far larger, and counts EVERY scanned sector (clear ones included)
/// toward the budget. With one 200_000-sector extent of clear sectors, it
/// must read exactly 50_000 sectors and return None — never run away.
///
/// Grounding: `let max_tries = 50_000; ... tried += 1` before the read,
/// loop guard `tried < max_tries`.
/// Mutation: change `50_000` to `500_000` -> read count exceeds 50_000;
/// the exact-count assert fails. Removing the `tried += 1` increment ->
/// would read all 200_000; also fails.
#[test]
fn crack_key_caps_total_tries_at_50000() {
let mut src = MockSource::new(0x00); // clear sectors, never a hit
let extents = [Extent {
start_lba: 0,
sector_count: 200_000,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none(), "clear sectors yield no key");
assert_eq!(
src.reads.borrow().len(),
50_000,
"scan must stop at the 50_000-sector budget"
);
}
// ── CrackOutcome: scrambled-but-uncracked vs genuinely unencrypted (Fix 6) ─
/// A scan over CLEAR sectors (scramble flag never set) returns
/// `Unencrypted` — the content is genuinely plaintext, so proceeding
/// without a key is correct.
#[test]
fn crack_outcome_clear_sectors_is_unencrypted() {
let mut src = MockSource::new(0x00); // never scrambled
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
matches!(outcome, CrackOutcome::Unencrypted),
"no scrambled sector seen → Unencrypted, got {outcome:?}"
);
// The Option wrapper collapses Unencrypted → None.
assert!(crack_key(&mut MockSource::new(0x00), &extents, 1).is_none());
}
/// THE Fix 6 regression: a scan that SEES scrambled sectors (flag set) but
/// recovers no key (the mock's zeroed data has no Stevenson crib) must
/// return `ScrambledUncracked` — a HARD failure — NOT `Unencrypted`. The
/// old code conflated this with "unencrypted" and muxed scrambled MPEG as
/// plaintext (garbage at exit 0).
#[test]
fn crack_outcome_scrambled_uncracked_is_hard_failure() {
let mut src = MockSource::new(0x30); // scrambled flag set, no crackable crib
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
outcome.is_scrambled_uncracked(),
"scrambled sectors seen but no key → ScrambledUncracked, got {outcome:?}"
);
// The legacy Option wrapper still collapses this to None (the callers
// that need the distinction now use crack_key_outcome instead).
assert!(crack_key(&mut MockSource::new(0x30), &extents, 1).is_none());
}
/// Even when every read FAILS, a scan that never managed to observe a
/// scrambled sector reports `Unencrypted` (we cannot prove encryption from
/// unreadable data alone — the AACS/keydb paths and the disc-level
/// `css_error` plumbing cover genuinely unreadable encrypted discs).
#[test]
fn crack_outcome_all_reads_fail_is_unencrypted() {
let mut src = MockSource::new(0x30);
src.fail_all = true; // no sector is ever inspected
let extents = [Extent {
start_lba: 0,
sector_count: 10,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
matches!(outcome, CrackOutcome::Unencrypted),
"no readable scrambled sector → Unencrypted, got {outcome:?}"
);
}
/// Fix C (rc.5.1): on the INITIAL scan, a drive that refuses every read with
/// CSS-locked sense (`05/6F/03`) is encrypted-but-locked →
/// `ScrambledUncracked` (a hard failure), NOT `Unencrypted`. This is the
/// rc4.3 bug: every VOB read came back `6F/03`, so the scan saw no scrambled
/// sector and wrongly declared the disc unencrypted → 19 KB garbage.
#[test]
fn crack_outcome_css_locked_initial_is_scrambled_uncracked() {
let mut src = MockSource::new(0x30);
src.lock_all = true; // every read → 05/6F/03
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
let outcome = crack_key_outcome(&mut src, &extents, 1, None);
assert!(
outcome.is_scrambled_uncracked(),
"every read 6F/03 on the initial scan → ScrambledUncracked, got {outcome:?}"
);
}
/// MISSING #1 guard: the re-crack path (the `Option`-returning `crack_key`,
/// `fail_on_locked == false`) must NOT hard-fail on a CSS-locked read — it
/// returns `None`. A lapsed-AGID re-crack of another VTS stays soft so a
/// genuinely crackable title isn't killed by a transient locked read.
#[test]
fn crack_key_recrack_locked_is_none_not_hard_fail() {
let mut src = MockSource::new(0x30);
src.lock_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 100,
}];
assert!(crack_key(&mut src, &extents, 1).is_none());
}
/// Fix F: a fully CSS-locked scan early-bails near `CSS_LOCKED_BAIL`
/// consecutive locked reads instead of grinding the whole 50_000-sector
/// budget (the rc5 "stuck Scanning…" hang on a wedged bridge).
#[test]
fn crack_css_locked_scan_early_bails() {
let mut src = MockSource::new(0x30);
src.lock_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 10_000,
}];
let _ = crack_key_outcome(&mut src, &extents, 1, None);
let n = src.reads.borrow().len();
assert!(
n <= (CSS_LOCKED_BAIL as usize) + 1,
"locked scan early-bails near {CSS_LOCKED_BAIL}, not 10000; read {n}"
);
}
/// The budget spans ALL extents, not per-extent: two extents summing past
/// the cap must still stop at 50_000 total reads.
///
/// Grounding: `tried` is declared outside the `for ext in extents` loop;
/// `if tried >= max_tries { break }` after each extent.
/// Mutation: move `let mut tried = 0` inside the extent loop -> each extent
/// gets its own 50_000 budget; total reads would be 80_000, this fails.
#[test]
fn crack_key_budget_is_shared_across_extents() {
let mut src = MockSource::new(0x00);
let extents = [
Extent {
start_lba: 0,
sector_count: 40_000,
},
Extent {
start_lba: 100_000,
sector_count: 40_000,
},
];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
50_000,
"the 50_000 budget is shared across all extents"
);
}
/// crack_key scans sequentially from each extent's start_lba. The first
/// reads must be at the extent's start_lba, start_lba+1, ... pinning the
/// LBA arithmetic `ext.start_lba + i`.
///
/// Grounding: `reader.read_sectors(ext.start_lba + i, 1, ...)`.
/// Mutation: change `ext.start_lba + i` to just `i` -> the recorded LBAs
/// would start at 0, not 5000; this fails.
#[test]
fn crack_key_scans_from_extent_start_lba() {
let mut src = MockSource::new(0x00);
let extents = [Extent {
start_lba: 5_000,
sector_count: 4,
}];
let _ = crack_key(&mut src, &extents, 1);
let reads = src.reads.borrow();
assert_eq!(
&reads[..],
&[5_000, 5_001, 5_002, 5_003],
"sequential scan from start_lba"
);
}
/// A read error on a sector does NOT abort the scan: crack_key keeps
/// scanning subsequent sectors (the error sector still counts toward the
/// budget). With a small failing extent, every sector is attempted and the
/// function returns None.
///
/// Grounding: `if reader.read_sectors(...).is_ok() && is_scrambled(...)` —
/// an Err simply falls through to `i += 1`.
/// Mutation: change the read-error handling to `reader.read_sectors(...)?`
/// (propagate) -> crack_key would stop after the first error and read only
/// 1 sector; this asserts all 10 were attempted.
#[test]
fn crack_key_continues_past_read_errors() {
let mut src = MockSource::new(0x30);
src.fail_all = true;
let extents = [Extent {
start_lba: 0,
sector_count: 10,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
10,
"read errors must not abort the scan"
);
}
/// Empty extents (no sectors) -> crack_key reads nothing and returns None.
/// A zero-sector extent must not read its start_lba.
///
/// Grounding: `while i < ext.sector_count` with sector_count == 0 never
/// enters.
/// Mutation: change `i < ext.sector_count` to `i <= ext.sector_count` ->
/// one spurious read at start_lba; this asserts zero reads.
#[test]
fn crack_key_empty_extent_reads_nothing() {
let mut src = MockSource::new(0x30);
let extents = [Extent {
start_lba: 42,
sector_count: 0,
}];
let res = crack_key(&mut src, &extents, 1);
assert!(res.is_none());
assert_eq!(
src.reads.borrow().len(),
0,
"zero-sector extent reads nothing"
);
}
/// No extents at all -> immediate None, zero reads.
///
/// Grounding: `for ext in extents` over an empty slice is a no-op.
/// Mutation: any change that reads before the loop would break this.
#[test]
fn crack_key_no_extents_is_none() {
let mut src = MockSource::new(0x30);
let res = crack_key(&mut src, &[], 1);
assert!(res.is_none());
assert_eq!(src.reads.borrow().len(), 0);
}
// ── Scan-level Cracked branch + per-VTS re-crack success (audit §2 / §5 #8) ─
/// SCAN-LEVEL CRACKED (audit gap "MockSource never yields a crackable
/// sector"): drive the full `crack_key_scan` over a synthetic ISO whose
/// scan hits a Stevenson-crackable scrambled sector. The outcome must be
/// `CrackOutcome::Cracked` with a key that round-trips the sector, AND the
/// `crack_span` must be recorded as the half-open extent span (the per-VTS
/// routing key the mux path needs). Previously only the leaf crack and the
/// Uncracked/Unencrypted branches were tested — the Cracked branch and
/// `crack_span` recording were never exercised end-to-end.
#[test]
fn crack_outcome_reaches_cracked_with_span() {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let seed = [0x11, 0x22, 0x33, 0x44, 0x55];
let crackable = crackable_sector(&title_key, &seed, 8);
// The crackable sector sits a few sectors into the extent.
let mut src = MockSource::new(0x00); // surrounding sectors: clear
src.crackable = Some((1003, crackable.clone()));
let extents = [Extent {
start_lba: 1000,
sector_count: 50,
}];
let outcome = crack_key_outcome(&mut src, &extents, 4, None);
let state = match outcome {
CrackOutcome::Cracked(s) => s,
other => panic!("expected Cracked, got {other:?}"),
};
// The recovered key descrambles the crackable sector body.
let mut test = crackable.clone();
descramble_sector(&state, &mut test);
let mut plain = crackable;
lfsr::descramble_sector(&title_key, &mut plain);
assert_eq!(
&test[0x80..],
&plain[0x80..],
"recovered key must round-trip the scrambled sector body"
);
// crack_span = half-open [start, start+count) of the scanned extent.
assert_eq!(
state.crack_span,
Some((1000, 1050)),
"crack_span must record the extent LBA span for per-VTS routing"
);
}
/// CSS_ERROR WIRING (audit §2 / §5 #7): an all-locked synthetic ISO (every
/// VOB read returns CSS-locked sense `05/6F/03` across MULTIPLE extents, as a
/// real encrypted-but-unauthenticated disc image does) must produce the exact
/// outcome the scan converts into `disc.css_error = Some(Error::CssKeyMissing)`
/// — i.e. `CrackOutcome::ScrambledUncracked` / `is_scrambled_uncracked()`,
/// NOT `Unencrypted`. disc/mod.rs's `crack_key_outcome → ScrambledUncracked`
/// arm (where it stamps css_error) is driven by exactly this signal, so this
/// pins the css-layer contract that arm depends on without touching the
/// scan plumbing.
#[test]
fn all_locked_synthetic_iso_yields_css_key_missing_signal() {
let mut src = MockSource::new(0x30);
src.lock_all = true; // every read → 05/6F/03 across the whole "ISO"
let extents = [
Extent {
start_lba: 0,
sector_count: 30,
},
Extent {
start_lba: 5_000,
sector_count: 30,
},
];
let outcome = crack_key_outcome(&mut src, &extents, 16, None);
assert!(
outcome.is_scrambled_uncracked(),
"all-locked ISO → ScrambledUncracked (the css_error=CssKeyMissing \
signal), got {outcome:?}"
);
// The legacy Option wrapper still collapses it to None — callers that
// surface the hard error must use crack_key_outcome, which this proves.
let mut src2 = MockSource::new(0x30);
src2.lock_all = true;
assert!(crack_key(&mut src2, &extents, 16).is_none());
}
/// PER-VTS RE-CRACK SUCCESS (audit gap "success path missing"): the prior
/// re-crack test only covered the locked→None path. Here a re-crack
/// (`crack_key`, `fail_on_locked == false`) over a DIFFERENT VTS's extents
/// finds that VTS's own crackable sector and returns a `CssState` whose
/// `crack_span` matches the new extents — proving a key cracked for one VTS
/// is genuinely re-derived (not reused) for another.
#[test]
fn recrack_succeeds_on_other_vts_extents() {
let title_key = [0xFE, 0xDC, 0xBA, 0x98, 0x76];
let seed = [0x00, 0xFF, 0x80, 0x7F, 0x01];
let crackable = crackable_sector(&title_key, &seed, 5);
let mut src = MockSource::new(0x00);
// The second VTS lives at a disjoint LBA range; its crackable sector is
// the first one in the extent.
src.crackable = Some((9000, crackable));
let other_vts = [Extent {
start_lba: 9000,
sector_count: 20,
}];
let state = crack_key(&mut src, &other_vts, 4).expect("re-crack must recover a key");
assert_eq!(
state.crack_span,
Some((9000, 9020)),
"re-crack span must reflect the OTHER VTS extents, not a reused span"
);
}
}
+574
View File
@@ -0,0 +1,574 @@
//! CSS title-key recovery — Frank A. Stevenson's divide-and-conquer attack
//! (1999), implemented from his published cryptanalysis ("Cryptanalysis of
//! Contents Scrambling System"). It recovers the 5-byte CSS title key from a
//! single scrambled DVD sector with no player keys and no disc-key crack, using
//! only known plaintext. Implemented from that public description; nothing here
//! is copied or translated from any particular CSS software.
//!
//! # The cipher this attacks
//!
//! The content descrambler ([`super::lfsr::descramble_sector`]) seeds its two
//! LFSRs **directly** from `key = title_key XOR sector_seed` (seed =
//! `sector[0x54..0x59]`): LFSR1 from key/seed bytes 0-1, LFSR0 (24-bit) from
//! bytes 2-4 with the pre-conditioning `r0 = r0*2 + 8 - (r0 & 7)`, and each body
//! byte recovered as `plain = TAB1[cipher] ^ (keystream & 0xff)`. There is no
//! title-key mangling on the content path, so the recovery is a single inversion
//! of the sector cipher.
//!
//! # The attack
//!
//! 1. **Known plaintext → keystream.** Because descramble applies TAB1 to the
//! ciphertext, the per-byte keystream is `TAB1[cipher[i]] ^ plain[i]`.
//! 2. **Brute the 16-bit LFSR1 seed.** For each of 2^16 seeds, run LFSR1
//! forward; for the first four steps deduce the LFSR0 output bytes from the
//! keystream (carry-tracked), reconstructing LFSR0's state. For the next six
//! steps clock LFSR0 normally and check it reproduces the keystream — a wrong
//! LFSR1 seed fails fast.
//! 3. **Back-clock LFSR0.** Run four backward steps (each a 256-way search for
//! the byte shifted in) to reach the initial state, then undo the
//! `r0*2 + 8 - (r0 & 7)` pre-conditioning to recover key[2..5].
//! 4. **XOR back the seed.** `key[0..5] ^= sector_seed[0..5]`.
//!
//! Known plaintext for step 1 comes from the longest periodic run in the
//! cleartext `sec[0x00..0x80]`, assumed to continue into the encrypted region at
//! 0x80.
use super::lfsr::descramble_sector;
use super::tables::{TAB1, TAB2, TAB3, TAB4, TAB5};
use crate::consts::SECTOR_BYTES;
const ENCRYPTED_START: usize = 0x80; // byte 128
const SEED_OFFSET: usize = 0x54; // sector seed at bytes 0x54-0x58
const FLAG_BYTE: usize = 0x14;
/// Recover the title key from cipher + known plaintext (the core of Stevenson's
/// attack). `crypted` is the ciphertext starting at sector byte 0x80;
/// `decrypted` is the matching known plaintext; `seed` is `sector[0x54..0x59]`.
/// On success returns the recovered 5-byte title key; `None` if no LFSR seed
/// reproduces the keystream.
///
/// At least 10 bytes of `crypted`/`decrypted` are required (the cipher is
/// iterated 10 times: 4 to reconstruct LFSR0, 6 to validate).
fn recover_title_key_from_plain(
crypted: &[u8],
decrypted: &[u8],
seed: &[u8; 5],
) -> Option<[u8; 5]> {
if crypted.len() < 10 || decrypted.len() < 10 {
return None;
}
// buf[i] = TAB1[cipher[i]] ^ plain[i] — the per-byte content keystream.
let mut buffer = [0u8; 10];
for (i, b) in buffer.iter_mut().enumerate() {
*b = TAB1[crypted[i] as usize] ^ decrypted[i];
}
let mut key = [0u8; 5];
let mut found = false;
for i_try in 0u32..0x1_0000 {
let mut i_t1 = (i_try >> 8) | 0x100;
let mut i_t2 = i_try & 0xff;
let mut i_t3: u32 = 0; // not needed yet
let mut i_t5: u32 = 0;
// Iterate the cipher 4 times to reconstruct LFSR0 (i_t3).
for &b in buffer.iter().take(4) {
let i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
let i_t4 = TAB5[i_t4 as usize] as u32;
// Deduce i_t6 (LFSR0 output, pre-TAB4) and the carry.
let mut i_t6 = b as u32;
if i_t5 != 0 {
i_t6 = (i_t6 + 0xff) & 0xff;
}
if i_t6 < i_t4 {
i_t6 += 0x100;
}
i_t6 -= i_t4;
i_t5 += i_t6 + i_t4;
let i_t6 = TAB4[i_t6 as usize] as u32;
i_t3 = (i_t3 << 8) | i_t6;
i_t5 >>= 8;
}
let i_candidate = i_t3;
// Iterate 6 more times to validate the candidate.
let mut i = 4usize;
while i < 10 {
let i_t4 = (TAB2[i_t2 as usize] ^ TAB3[i_t1 as usize]) as u32;
i_t2 = i_t1 >> 1;
i_t1 = ((i_t1 & 1) << 8) ^ i_t4;
let i_t4 = TAB5[i_t4 as usize] as u32;
let mut i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xff;
i_t3 = (i_t3 << 8) | i_t6;
i_t6 = TAB4[i_t6 as usize] as u32;
i_t5 += i_t6 + i_t4;
if (i_t5 & 0xff) as u8 != buffer[i] {
break;
}
i_t5 >>= 8;
i += 1;
}
if i != 10 {
continue;
}
// Four backward steps of iterating i_t3 to deduce the initial state.
i_t3 = i_candidate;
for _ in 0..4 {
let i_t1_byte = i_t3 & 0xff;
i_t3 >>= 8;
// Brute-force the byte shifted in (top byte of the 24-bit reg).
for j in 0u32..256 {
i_t3 = (i_t3 & 0x1_ffff) | (j << 17);
let i_t6 = (((((((i_t3 >> 3) ^ i_t3) >> 1) ^ i_t3) >> 8) ^ i_t3) >> 5) & 0xff;
if i_t6 == i_t1_byte {
break;
}
}
}
// Undo `i_t3 = i_t3*2 + 8 - (i_t3 & 7)` to recover key[2..5].
let i_t4 = (i_t3 >> 1).wrapping_sub(4);
for i_t5 in 0u32..8 {
let val = i_t4.wrapping_add(i_t5);
if val.wrapping_mul(2).wrapping_add(8).wrapping_sub(val & 7) == i_t3 {
key[0] = (i_try >> 8) as u8;
key[1] = (i_try & 0xff) as u8;
key[2] = (val & 0xff) as u8;
key[3] = ((val >> 8) & 0xff) as u8;
key[4] = ((val >> 16) & 0xff) as u8;
found = true;
break;
}
}
// First fully-validated candidate wins. The 48-bit keystream constraint
// makes a second match cryptographically negligible on real sectors, but
// continuing would let a later spurious match overwrite a correct key.
if found {
break;
}
}
if found {
for (k, &s) in key.iter_mut().zip(seed.iter()) {
*k ^= s;
}
Some(key)
} else {
None
}
}
/// Recover the CSS title key from a scrambled sector using a known plaintext
/// for the encrypted region.
///
/// `plain` is the expected plaintext at byte 0x80 (at least 10 bytes).
/// Returns the recovered key only if it actually descrambles the sector back
/// to `plain` — guarding against the rare spurious LFSR-seed match.
pub fn recover_title_key(sector: &[u8], plain: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_BYTES || plain.len() < 10 {
return None;
}
if sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
let seed: [u8; 5] = [
sector[SEED_OFFSET],
sector[SEED_OFFSET + 1],
sector[SEED_OFFSET + 2],
sector[SEED_OFFSET + 3],
sector[SEED_OFFSET + 4],
];
let crypted = &sector[ENCRYPTED_START..ENCRYPTED_START + 10];
let key = recover_title_key_from_plain(crypted, plain, &seed)?;
if descramble_matches(sector, &key, plain) {
Some(key)
} else {
None
}
}
/// Verify a title key by descrambling a copy of `sector` and checking the
/// known plaintext reappears at byte 0x80.
fn descramble_matches(sector: &[u8], title: &[u8; 5], plain: &[u8]) -> bool {
let mut test = sector.to_vec();
test[FLAG_BYTE] |= 0x10; // ensure scramble flag set for the descrambler
descramble_sector(title, &mut test);
let n = plain.len().min(SECTOR_BYTES - ENCRYPTED_START);
test[ENCRYPTED_START..ENCRYPTED_START + n] == plain[..n]
}
/// Find a repeating pattern just before the encrypted region and assume the
/// plaintext at 0x80 continues it — the known-plaintext step of Stevenson's
/// attack. Scans cleartext `sec[0x00..0x80]` for the longest run that repeats
/// with a cycle length in 2..0x2F. If the run is long enough (`plen > 3` and at
/// least two full cycles), the known plaintext at 0x80 is taken to be the
/// periodic run continuing forward, and [`recover_title_key_from_plain`] is
/// applied.
pub fn crack_title_key(sector: &[u8]) -> Option<[u8; 5]> {
if sector.len() < SECTOR_BYTES {
return None;
}
if sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
// Runaway guard: a single sector's crack is a bounded 2^16 LFSR search and
// should finish in well under a second on any modern CPU. If it ever
// exceeds ~2s wall-clock, something pathological is happening — log it so a
// hang is never silent.
let crack_t0 = std::time::Instant::now();
let result = crack_title_key_inner(sector);
let elapsed = crack_t0.elapsed();
if elapsed.as_secs_f64() > 2.0 {
tracing::warn!(
target: "freemkv::css",
elapsed_ms = elapsed.as_millis() as u64,
found = result.is_some(),
"css crack: single-sector recovery exceeded 2s (runaway guard)"
);
}
result
}
/// Crib: the predicted 10-byte plaintext at byte 0x80.
///
/// Scans the clear header `sec[0x00..0x80]` (never scrambled) for the longest
/// run that repeats with a cycle length in 2..0x2F. If the run is long enough
/// (`plen > 3` and at least two full cycles), the plaintext at 0x80 is taken to
/// be that periodic run continuing forward. Returns `None` for an unscrambled
/// sector or one with no usable run — such a sector can be neither cracked nor
/// key-validated, only descrambled with an externally-cached key.
///
/// The header is untouched by `descramble_sector`, so the crib is identical
/// before and after descramble: the decrypt path uses it as a per-sector
/// "did the cached key descramble correctly?" oracle (the predicted plaintext
/// must reappear at 0x80), and the cracker uses it as its known plaintext.
pub(crate) fn attack_crib(sector: &[u8]) -> Option<[u8; 10]> {
if sector.len() < SECTOR_BYTES || sector[FLAG_BYTE] & 0x30 == 0 {
return None;
}
let mut best_plen: usize = 0;
let mut best_p: usize = 0;
// For all cycle lengths from 2 to 0x2F.
for i in 2usize..0x30 {
// Count bytes that repeat with cycle length i, scanning backward from
// 0x7F. `sec[0x7F - (j % i)] == sec[0x7F - j]`.
let mut j = i + 1;
while j < 0x80 && sector[0x7f - (j % i)] == sector[0x7f - j] {
if j > best_plen {
best_plen = j;
best_p = i;
}
j += 1;
}
}
// Need at least a few repeated bytes and at least one full cycle.
if best_plen > 3 && best_p > 0 && best_plen / best_p >= 2 {
// The known plaintext is the periodic run continuing past 0x80. The
// crib starts at `0x80 - (best_plen/best_p)*best_p` and continues
// through the encrypted region; the bytes at and after 0x80 are the
// predicted plaintext (the pattern repeats with period best_p).
let cycles = best_plen / best_p;
let plain_start = 0x80 - cycles * best_p;
// Each predicted byte is the run sample one or more periods back:
// `sec[plain_start + (i % best_p)]`. For in-run offsets
// (`plain_start + i < 0x80`) the run is exactly periodic, so this
// equals `sec[plain_start + i]`; for offsets at/after 0x80 the raw
// byte is ciphertext, so we MUST wrap within the period rather than
// read it. (Reading `&sec[plain_start..+10]` directly — as before —
// pulled ciphertext into the crib whenever the run covered fewer than
// 10 bytes before 0x80, producing false-negative key recovery.)
let mut plain = [0u8; 10];
for (i, p) in plain.iter_mut().enumerate() {
*p = sector[plain_start + (i % best_p)];
}
Some(plain)
} else {
None
}
}
fn crack_title_key_inner(sector: &[u8]) -> Option<[u8; 5]> {
let plain = attack_crib(sector)?;
let seed: [u8; 5] = [
sector[SEED_OFFSET],
sector[SEED_OFFSET + 1],
sector[SEED_OFFSET + 2],
sector[SEED_OFFSET + 3],
sector[SEED_OFFSET + 4],
];
let crypted = &sector[0x80..0x80 + 10];
if let Some(key) = recover_title_key_from_plain(crypted, &plain, &seed) {
// Verify against the same predicted plaintext.
if descramble_matches(sector, &key, &plain) {
return Some(key);
}
}
None
}
#[cfg(test)]
mod tests {
use super::super::lfsr::scramble_sector;
use super::*;
/// Build a synthetic scrambled sector for a given title key and seed,
/// with `plain` placed as the plaintext at byte 0x80, scrambled with
/// EXACTLY the cipher `descramble_sector` inverts. Returns
/// (scrambled_sector, full_plaintext_body).
fn synth_sector(title_key: &[u8; 5], seed: &[u8; 5], plain: &[u8]) -> (Vec<u8>, Vec<u8>) {
let mut plaintext = vec![0u8; SECTOR_BYTES];
plaintext[0..4].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
plaintext[FLAG_BYTE] = 0x10;
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
plaintext[ENCRYPTED_START..ENCRYPTED_START + plain.len()].copy_from_slice(plain);
let body = plaintext.clone();
// scramble_sector turns the plaintext body into ciphertext and sets
// the scramble flag.
scramble_sector(title_key, &mut plaintext);
(plaintext, body)
}
/// Build a synthetic scrambled sector whose CLEARTEXT (0x00..0x80) ends
/// in a periodic run that continues into the encrypted region — the case
/// `crack_title_key` is designed to crack.
fn synth_periodic_sector(
title_key: &[u8; 5],
seed: &[u8; 5],
period: usize,
) -> (Vec<u8>, Vec<u8>) {
let mut plaintext = vec![0u8; SECTOR_BYTES];
plaintext[FLAG_BYTE] = 0x10;
// A clean periodic run occupying the tail of the cleartext header
// (RUN_START..0x80) and continuing into the encrypted region. This
// mirrors a real VOB: a periodic data run just before the scrambled
// part. The run must NOT overlap the seed bytes (0x54..0x59), or the
// the crib detector would break mid-run. The phase is anchored to
// offset 0 so the run is consistent across the 0x80 boundary.
// Just above the seed (0x54..0x59); gives a 39-byte run (0x59..0x80)
// — enough for >=2 cycles of every tested period (<=19).
const RUN_START: usize = 0x59;
let pat: Vec<u8> = (0..period)
.map(|k| (0xA0u8.wrapping_add(k as u8)) ^ 0x5A)
.collect();
for (i, b) in plaintext.iter_mut().enumerate().skip(RUN_START) {
*b = pat[i % period];
}
// Seed sits below the run, undisturbed.
plaintext[SEED_OFFSET..SEED_OFFSET + 5].copy_from_slice(seed);
let body = plaintext.clone();
scramble_sector(title_key, &mut plaintext);
(plaintext, body)
}
#[test]
fn crack_unscrambled_returns_none() {
let sector = vec![0u8; SECTOR_BYTES];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn crack_too_short_returns_none() {
let sector = vec![0u8; 100];
assert!(crack_title_key(&sector).is_none());
}
#[test]
fn recover_needs_min_plain() {
let sector = vec![0u8; SECTOR_BYTES];
let short_plain = [0u8; 4];
assert!(recover_title_key(&sector, &short_plain).is_none());
}
/// The known plaintext used at byte 0x80 for the direct-recovery tests.
/// A realistic MPEG-2 PES header start.
const PES: [u8; 10] = [0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x80, 0x05, 0x21];
/// MANDATORY round-trip (Task C.1): synthesize a scrambled sector for a
/// known (title_key, seed), then assert recover_title_key returns a key
/// that descrambles the body back to plaintext. CSS title-key recovery is
/// well-defined up to keys that scramble identically; we assert the full
/// body round-trips (the true correctness property), and additionally
/// that the EXACT key is returned for the common case.
#[test]
fn recover_round_trips_known_keys() {
let cases: &[([u8; 5], [u8; 5])] = &[
(
[0x42, 0x13, 0x37, 0xBE, 0xEF],
[0x11, 0x22, 0x33, 0x44, 0x55],
),
(
[0x01, 0x02, 0x03, 0x04, 0x05],
[0xDE, 0xAD, 0xBE, 0xEF, 0x42],
),
(
[0xFE, 0xDC, 0xBA, 0x98, 0x76],
[0x00, 0xFF, 0x80, 0x7F, 0x01],
),
(
[0x9A, 0x78, 0x56, 0x34, 0x12],
[0xA5, 0x5A, 0x0F, 0xF0, 0xCC],
),
(
[0xFF, 0xFF, 0xFF, 0xFF, 0xFF],
[0x01, 0x01, 0x01, 0x01, 0x01],
),
];
for (title_key, seed) in cases {
let (mut sector, body) = synth_sector(title_key, seed, &PES);
let recovered =
recover_title_key(&sector, &PES).expect("recover_title_key returned None");
descramble_sector(&recovered, &mut sector);
assert_eq!(
&sector[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"recovered key did not descramble the full body for \
title={title_key:02x?} seed={seed:02x?}"
);
}
}
/// MANDATORY (Task C.1): the crib-based entry point crack_title_key —
/// no plaintext supplied — recovers a round-tripping key when the
/// cleartext ends in a periodic run that continues into 0x80.
#[test]
fn crack_title_key_recovers_via_attack_pattern() {
for &period in &[2usize, 3, 5, 8, 16] {
let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
let seed = [0x11, 0x22, 0x33, 0x44, 0x55];
let (sector, body) = synth_periodic_sector(&title_key, &seed, period);
let cracked = crack_title_key(&sector)
.unwrap_or_else(|| panic!("crack_title_key returned None for period {period}"));
let mut test = sector.clone();
descramble_sector(&cracked, &mut test);
assert_eq!(
&test[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"crack_title_key key did not round-trip the body (period {period})"
);
}
}
/// recover_title_key_from_plain inverts descramble_sector exactly: scramble
/// a known body, hand back the keystream-derived key, and the recovered
/// key (XOR-back included) reproduces the plaintext.
#[test]
fn recovered_key_descrambles_back_to_plaintext() {
let cases: &[([u8; 5], [u8; 5])] = &[
(
[0x42, 0x13, 0x37, 0xBE, 0xEF],
[0x11, 0x22, 0x33, 0x44, 0x55],
),
(
[0x9A, 0x78, 0x56, 0x34, 0x12],
[0xA5, 0x5A, 0x0F, 0xF0, 0xCC],
),
(
[0xFF, 0xFF, 0xFF, 0xFF, 0xFF],
[0x01, 0x01, 0x01, 0x01, 0x01],
),
];
for (title_key, seed) in cases {
let (mut sector, body) = synth_sector(title_key, seed, &PES);
let recovered =
recover_title_key(&sector, &PES).expect("recover_title_key returned None");
descramble_sector(&recovered, &mut sector);
assert_eq!(
&sector[ENCRYPTED_START..SECTOR_BYTES],
&body[ENCRYPTED_START..SECTOR_BYTES],
"descramble with recovered key did not reproduce the body \
for title={title_key:02x?} seed={seed:02x?}"
);
}
}
// ── early-return guards ────────────────────────────────────────────────
#[test]
fn recover_rejects_sector_one_byte_short() {
let mut sector = vec![0u8; SECTOR_BYTES - 1];
sector[FLAG_BYTE] = 0x30;
assert!(recover_title_key(&sector, &PES).is_none());
}
#[test]
fn recover_rejects_unscrambled_sector() {
let sector = vec![0x00u8; SECTOR_BYTES];
assert!(recover_title_key(&sector, &PES).is_none());
}
#[test]
fn recover_high_flag_bits_are_not_scramble() {
for &flag in &[0x40u8, 0x80, 0xC0] {
let mut sector = vec![0x11u8; SECTOR_BYTES];
sector[FLAG_BYTE] = flag;
assert!(
recover_title_key(&sector, &PES).is_none(),
"flag {flag:#04x} has scramble bits clear; recover must return None"
);
}
}
#[test]
fn crack_high_flag_bits_are_not_scramble() {
for &flag in &[0x40u8, 0x80, 0xC0] {
let mut sector = vec![0x11u8; SECTOR_BYTES];
sector[FLAG_BYTE] = flag;
assert!(
crack_title_key(&sector).is_none(),
"flag {flag:#04x} clear scramble bits -> crack must return None"
);
}
}
#[test]
fn crack_rejects_sector_one_byte_short() {
let mut sector = vec![0u8; SECTOR_BYTES - 1];
if sector.len() > FLAG_BYTE {
sector[FLAG_BYTE] = 0x30;
}
assert!(crack_title_key(&sector).is_none());
}
/// crack_title_key must never panic on a fully scrambled sector with
/// arbitrary (non-periodic) content — it just returns None.
#[test]
fn crack_full_path_never_panics() {
for seed in 0u32..3 {
let mut sector = vec![0u8; SECTOR_BYTES];
sector[FLAG_BYTE] = 0x30;
let mut x = seed.wrapping_mul(2_654_435_761).wrapping_add(7);
for b in sector.iter_mut().skip(0x80) {
x = x.wrapping_mul(1_103_515_245).wrapping_add(12_345);
*b = (x >> 16) as u8;
}
for (i, b) in sector[SEED_OFFSET..SEED_OFFSET + 5].iter_mut().enumerate() {
*b = (seed.wrapping_add(i as u32) ^ 0xA5) as u8;
}
let _ = crack_title_key(&sector);
}
}
}
+191 -29
View File
@@ -24,7 +24,8 @@ pub const TAB1: [u8; 256] = [
0xb7, 0xf7, 0xbf, 0xa2, 0xe7, 0xa7, 0xef, 0xf2, 0xba, 0xfa, 0xb2, 0xaf, 0xea, 0xaa, 0xe2, 0xff,
];
/// Table 2: LFSR1 high-byte feedback permutation.
/// Table 2: LFSR1 high-byte feedback permutation — a fixed constant of the CSS
/// cipher (per the published algorithm).
pub const TAB2: [u8; 256] = [
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x09, 0x08, 0x0b, 0x0a, 0x0d, 0x0c, 0x0f, 0x0e,
0x12, 0x13, 0x10, 0x11, 0x16, 0x17, 0x14, 0x15, 0x1b, 0x1a, 0x19, 0x18, 0x1f, 0x1e, 0x1d, 0x1c,
@@ -40,11 +41,18 @@ pub const TAB2: [u8; 256] = [
0xa4, 0xa5, 0xa6, 0xa7, 0xa0, 0xa1, 0xa2, 0xa3, 0xad, 0xac, 0xaf, 0xae, 0xa9, 0xa8, 0xab, 0xaa,
0xdb, 0xda, 0xd9, 0xd8, 0xdf, 0xde, 0xdd, 0xdc, 0xd2, 0xd3, 0xd0, 0xd1, 0xd6, 0xd7, 0xd4, 0xd5,
0xc9, 0xc8, 0xcb, 0xca, 0xcd, 0xcc, 0xcf, 0xce, 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7,
0xed, 0xec, 0xef, 0xee, 0xe9, 0xe8, 0xeb, 0xea, 0xe4, 0xe5, 0xe6, 0xe7, 0xe0, 0xe1, 0xe2, 0xe3,
0xff, 0xfe, 0xfd, 0xfc, 0xfb, 0xfa, 0xf9, 0xf8, 0xf6, 0xf7, 0xf4, 0xf5, 0xf2, 0xf3, 0xf0, 0xf1,
0xed, 0xec, 0xef, 0xee, 0xe9, 0xe8, 0xeb, 0xea, 0xe4, 0xe5, 0xe6, 0xe7, 0xe0, 0xe1, 0xe2, 0xe3,
];
/// Table 3: LFSR1 low-byte feedback permutation.
/// Table 3: LFSR1 9-bit low-word feedback table (512 entries) — a fixed constant
/// of the CSS cipher (per the published algorithm).
///
/// It is the 8-value block `BASE[i & 7]` repeated 64 times. The CSS LFSR1 step
/// indexes this table with the 9-bit low register (0x100..=0x1FF), but only the
/// low 3 bits select the output — the high bits are ignored, hence the constant
/// blocks. The 512-entry width simply lets the 9-bit index be used without
/// masking.
pub const TAB3: [u8; 512] = [
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
@@ -54,30 +62,30 @@ pub const TAB3: [u8; 512] = [
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe, 0x01, 0x25, 0x48, 0x6c, 0x93, 0xb7, 0xda, 0xfe,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd, 0x02, 0x26, 0x4b, 0x6f, 0x90, 0xb4, 0xd9, 0xfd,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc, 0x03, 0x27, 0x4a, 0x6e, 0x91, 0xb5, 0xd8, 0xfc,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff, 0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff,
];
/// Table 4: LFSR0 byte permutation (used in initialization and output).
@@ -100,8 +108,10 @@ pub const TAB4: [u8; 256] = [
0x0f, 0x8f, 0x4f, 0xcf, 0x2f, 0xaf, 0x6f, 0xef, 0x1f, 0x9f, 0x5f, 0xdf, 0x3f, 0xbf, 0x7f, 0xff,
];
/// Table 5: LFSR1 output permutation for the Stevenson attack.
/// This is the inverse byte-reversal of TAB4.
/// Table 5: LFSR1 output permutation used in the keystream combiner.
/// `TAB5[i] == TAB4[i] ^ 0xFF` (bitwise complement of the TAB4 bit-reversal
/// table). Applied on the normal descramble/recrypt path (lfsr.rs) as well as
/// in the key-recovery fallback (crack.rs).
pub const TAB5: [u8; 256] = [
0xff, 0x7f, 0xbf, 0x3f, 0xdf, 0x5f, 0x9f, 0x1f, 0xef, 0x6f, 0xaf, 0x2f, 0xcf, 0x4f, 0x8f, 0x0f,
0xf7, 0x77, 0xb7, 0x37, 0xd7, 0x57, 0x97, 0x17, 0xe7, 0x67, 0xa7, 0x27, 0xc7, 0x47, 0x87, 0x07,
@@ -120,3 +130,155 @@ pub const TAB5: [u8; 256] = [
0xf8, 0x78, 0xb8, 0x38, 0xd8, 0x58, 0x98, 0x18, 0xe8, 0x68, 0xa8, 0x28, 0xc8, 0x48, 0x88, 0x08,
0xf0, 0x70, 0xb0, 0x30, 0xd0, 0x50, 0x90, 0x10, 0xe0, 0x60, 0xa0, 0x20, 0xc0, 0x40, 0x80, 0x00,
];
#[cfg(test)]
mod tests {
use super::*;
/// Pins the documented relationship `TAB5[i] == TAB4[i] ^ 0xFF` so the
/// table doc cannot drift from the data.
#[test]
fn tab5_is_complement_of_tab4() {
for i in 0..256 {
assert_eq!(
TAB5[i],
TAB4[i] ^ 0xFF,
"TAB5[{i:#04x}] != TAB4[{i:#04x}] ^ 0xFF"
);
}
}
/// TAB1 is a bijection on 0..256. CSS uses it as an invertible output
/// permutation in css_DecryptKey's chained-XOR rounds; if two inputs
/// collided, the key mangling would not be invertible.
///
/// Mutation: duplicate any value (e.g. set TAB1[1] = TAB1[0]) -> the
/// "maps two inputs" assert fires.
#[test]
fn tab1_is_a_permutation() {
let mut seen = [false; 256];
for (i, &v) in TAB1.iter().enumerate() {
assert!(
!seen[v as usize],
"TAB1 maps two inputs to {v:#04x} (collision at index {i:#04x})"
);
seen[v as usize] = true;
}
}
/// TAB1's fixed structural anchors from the CSS spec table:
/// TAB1[0x00] == 0x33 and the inverse TAB1[0x33] == 0x00. These two
/// entries are the canonical first-row / inverse-lookup landmarks of the
/// published CSS TAB1 and pin the table's orientation.
///
/// Grounding: CSS specification TAB1, row 0 col 0 = 0x33; index 0x33
/// (row 3 col 3) = 0x00.
/// Mutation: change the first literal `0x33` in TAB1 -> first assert fails.
#[test]
fn tab1_known_spec_anchors() {
assert_eq!(TAB1[0x00], 0x33, "TAB1[0] is the published 0x33");
assert_eq!(TAB1[0x33], 0x00, "TAB1[0x33] is the published 0x00");
}
/// TAB2 is a permutation of 0..256 (it is the LFSR1 high-byte feedback
/// substitution). A non-bijective TAB2 would bias the LFSR1 keystream.
///
/// Mutation: set TAB2[8] = 0x00 (collides with TAB2[0]) -> assert fires.
#[test]
fn tab2_is_a_permutation() {
let mut seen = [false; 256];
for (i, &v) in TAB2.iter().enumerate() {
assert!(
!seen[v as usize],
"TAB2 maps two inputs to {v:#04x} (collision at index {i:#04x})"
);
seen[v as usize] = true;
}
}
/// TAB3 is the CSS LFSR1 low-word table: the 8-value feedback block
/// BASE = [0x00,0x24,0x49,0x6d,0x92,0xb6,0xdb,0xff]
/// repeated 64 times — `TAB3[i] == BASE[i & 7]`. The high bits of the
/// 9-bit index do not affect the output (the LFSR1 step indexes with the
/// full 9-bit low register but only `& 7` matters). This pins all 512
/// entries to the published cipher's table.
///
/// Mutation: flip any single byte in the TAB3 literal -> the formula
/// check fails at that index.
#[test]
fn tab3_matches_lfsr1_generating_formula() {
const BASE: [u8; 8] = [0x00, 0x24, 0x49, 0x6d, 0x92, 0xb6, 0xdb, 0xff];
for i in 0..512usize {
let expected = BASE[i & 7];
assert_eq!(
TAB3[i], expected,
"TAB3[{i:#05x}] = {:#04x}, formula BASE[i&7] = {expected:#04x}",
TAB3[i]
);
}
}
/// TAB4 is the exact bit-reversal of each byte (CSS uses it to permute
/// LFSR0 bytes on seed and output). TAB4[b] reverses b's 8 bits MSB<->LSB.
/// Therefore it is also an involution: TAB4[TAB4[b]] == b.
///
/// Grounding: TAB4[0x01]=0x80, TAB4[0x80]=0x01, TAB4[0x00]=0x00,
/// TAB4[0xFF]=0xFF.
/// Mutation: set TAB4[1] = 0x40 (not the reversal 0x80) -> bit-reversal
/// check fails at index 1.
#[test]
fn tab4_is_exact_bit_reversal_and_involution() {
for b in 0u16..256 {
let rev = (0..8).fold(0u8, |acc, k| acc | (((b as u8 >> k) & 1) << (7 - k)));
assert_eq!(
TAB4[b as usize], rev,
"TAB4[{b:#04x}] is not the bit-reversal {rev:#04x}"
);
}
for b in 0..256usize {
assert_eq!(
TAB4[TAB4[b] as usize], b as u8,
"TAB4 not an involution at {b:#04x}"
);
}
// Spec landmark entries.
assert_eq!(TAB4[0x01], 0x80);
assert_eq!(TAB4[0x80], 0x01);
assert_eq!(TAB4[0x00], 0x00);
assert_eq!(TAB4[0xFF], 0xFF);
}
/// TAB4 is a permutation (bit-reversal is bijective). Distinct from the
/// reversal test: a table that is "reversal except two swapped entries"
/// would still be a permutation, and a table that is "reversal except one
/// duplicated entry" would fail this but might pass a sampled reversal
/// check — the two tests pin different failure modes.
///
/// Mutation: set TAB4[2] = TAB4[1] -> permutation assert fires.
#[test]
fn tab4_is_a_permutation() {
let mut seen = [false; 256];
for &v in TAB4.iter() {
assert!(!seen[v as usize], "TAB4 maps two inputs to {v:#04x}");
seen[v as usize] = true;
}
}
/// TAB5 is also a permutation (complement of a bijection is a bijection)
/// and its own self-consistency landmark: TAB5[0x00] == 0xFF (TAB4[0]^0xFF)
/// and TAB5[0xFF] == 0x00 (TAB4[0xFF]^0xFF). Pins orientation independent
/// of the complement-loop test.
///
/// Mutation: change the first TAB5 literal 0xff -> 0xfe -> the landmark
/// and permutation checks both catch it.
#[test]
fn tab5_is_permutation_with_anchors() {
let mut seen = [false; 256];
for &v in TAB5.iter() {
assert!(!seen[v as usize], "TAB5 maps two inputs to {v:#04x}");
seen[v as usize] = true;
}
assert_eq!(TAB5[0x00], 0xFF, "TAB5[0] = TAB4[0]^0xFF = 0xFF");
assert_eq!(TAB5[0xFF], 0x00, "TAB5[0xFF] = TAB4[0xFF]^0xFF = 0x00");
}
}
+1628 -60
View File
File diff suppressed because it is too large Load Diff
+753
View File
@@ -0,0 +1,753 @@
//! Structured scan diagnostics — the `--log-level 3` self-diagnosing dump.
//!
//! A bug report log must be self-diagnosing: everything needed to explain
//! *why* freemkv made the choices it did at scan must be in the log, in a
//! compact, machine-parseable form. This module emits one terse line per row
//! (title, cell, stream, decision) under the `tracing` target
//! `freemkv::diag`, which the CLI routes to `log.txt` when `--log-level 3`
//! (debug) is set.
//!
//! Format conventions (stable, greppable):
//! - Every line is prefixed by a `tag=` so a log scraper can filter
//! (`disc`, `title`, `dvd.cell`, `dvd.vattr`, `dvd.aattr`, `bd.clip`,
//! `bd.mark`, `aacs`, `stream`, `decision`).
//! - Raw bytes are shown as `0xNN` next to their decode so a wrong decode
//! is obvious against the raw value.
//! - This module only READS already-parsed scan state — it never re-reads
//! the disc and never mutates anything.
//!
//! The DVD per-cell table (with the raw cell-category byte) is emitted from
//! the IFO scan itself ([`dump_dvd_cells`]), because the per-cell
//! `ifo::DvdCell` detail is lowered away before the `Disc` is built. The
//! `Disc`-level dump ([`dump_disc`]) covers everything that survives
//! lowering: titles, streams, the picked main feature, and AACS state.
use crate::disc::{
AudioChannels, ColorSpace, Disc, DiscTitle, FrameRate, HdrFormat, Resolution, SampleRate,
Stream,
};
use crate::ifo::{CellCategory, DvdTitle};
const DIAG: &str = "freemkv::diag";
// ── small format helpers (pure, unit-testable) ──────────────────────────────
/// Compact name for a [`Resolution`] with the interlace marker preserved.
pub fn res_str(r: Resolution) -> &'static str {
match r {
Resolution::R480i => "480i",
Resolution::R480p => "480p",
Resolution::R576i => "576i",
Resolution::R576p => "576p",
Resolution::R720p => "720p",
Resolution::R1080i => "1080i",
Resolution::R1080p => "1080p",
Resolution::R2160p => "2160p",
Resolution::R4320p => "4320p",
Resolution::Unknown => "res?",
}
}
/// Frames-per-second string for a [`FrameRate`].
pub fn fps_str(f: FrameRate) -> &'static str {
match f {
FrameRate::F23_976 => "23.976",
FrameRate::F24 => "24",
FrameRate::F25 => "25",
FrameRate::F29_97 => "29.97",
FrameRate::F30 => "30",
FrameRate::F50 => "50",
FrameRate::F59_94 => "59.94",
FrameRate::F60 => "60",
FrameRate::Unknown => "fps?",
}
}
/// PAL/NTSC field-rate family inferred from the frame rate (DVD has no
/// explicit field, so this is the colour/standard the muxer stamps).
pub fn tv_system_str(f: FrameRate) -> &'static str {
match f {
FrameRate::F25 | FrameRate::F50 => "PAL",
FrameRate::F23_976 | FrameRate::F29_97 | FrameRate::F59_94 => "NTSC",
_ => "—",
}
}
/// CICP-ish short name for a [`ColorSpace`].
pub fn color_str(c: ColorSpace) -> &'static str {
match c {
ColorSpace::Bt709 => "BT.709",
ColorSpace::Bt2020 => "BT.2020",
ColorSpace::Bt470bg => "BT.470BG",
ColorSpace::Smpte170m => "SMPTE-170M",
ColorSpace::Unknown => "color?",
}
}
/// HDR format short name.
pub fn hdr_str(h: HdrFormat) -> &'static str {
match h {
HdrFormat::Sdr => "SDR",
HdrFormat::Hdr10 => "HDR10",
HdrFormat::Hdr10Plus => "HDR10+",
HdrFormat::DolbyVision => "DoVi",
HdrFormat::Hlg => "HLG",
}
}
/// Channel count from an [`AudioChannels`] layout (what lands in the MKV
/// `Channels` element).
pub fn channel_count(ch: AudioChannels) -> u8 {
match ch {
AudioChannels::Mono => 1,
AudioChannels::Stereo => 2,
AudioChannels::Stereo21 => 3,
AudioChannels::Quad => 4,
AudioChannels::Surround50 => 5,
AudioChannels::Surround51 => 6,
AudioChannels::Surround61 => 7,
AudioChannels::Surround71 => 8,
AudioChannels::Unknown => 0,
}
}
/// Sample-rate in Hz for a [`SampleRate`].
pub fn sample_rate_hz(s: SampleRate) -> u32 {
match s {
SampleRate::S44_1 => 44100,
SampleRate::S48 => 48000,
SampleRate::S88_2 => 88200,
SampleRate::S96 => 96000,
SampleRate::S176_4 => 176400,
SampleRate::S192 => 192000,
SampleRate::S48_96 => 96000,
SampleRate::S48_192 => 192000,
SampleRate::Unknown => 0,
}
}
// ── DVD cell-category dump (from the IFO scan, pre-lowering) ─────────────────
/// One formatted cell row for the DVD per-PGC cell table. Returned as a
/// string so it can be unit-tested without a logger.
///
/// Columns: `idx`, raw category (`cat=0xNN`) + decoded fields, first/last
/// sector, duration, and the keep/drop verdict from the bug-4 leading-cell
/// filter.
pub fn dvd_cell_row(idx: usize, cell: &crate::ifo::DvdCell, dropped: bool) -> String {
let c = CellCategory::decode(cell.category);
// Per-cell keep/skip REASON (self-sufficient bug log): a dropped cell is a
// leading secondary angle/interleave block piece; a kept cell is either the
// first feature cell or genuine feature content. This makes the
// leading-cell-filter decision auditable from the log without the disc.
let verdict = if dropped {
"DROP(leading-secondary-block-piece)"
} else if c.is_secondary_block_piece() {
// Kept despite being a secondary piece — only happens past the leading
// run (the filter stops at the first plain feature cell).
"keep(feature-body)"
} else {
"keep(plain-feature)"
};
format!(
"tag=dvd.cell idx={idx} cat=0x{:02X} block_mode={} block_type={} \
seamless={} ilv={} stc={} angle={} plain={} first={} last={} dur={:.1}s {}",
cell.category,
c.block_mode,
c.block_type,
c.seamless_play as u8,
c.interleaved as u8,
c.stc_discontinuity as u8,
c.seamless_angle as u8,
c.is_plain_feature() as u8,
cell.first_sector,
cell.last_sector,
cell.duration_secs,
verdict,
)
}
/// Emit the per-PGC cell table for one DVD title during the IFO scan.
///
/// `vts`/`title` identify the row group; `title` is the `DvdTitle` whose
/// cells (and bug-4 leading-cell verdict) are dumped. Called from
/// `scan_dvd_titles` while the `DvdTitle` is still in scope (the per-cell
/// category byte is lowered away before the `Disc` exists).
pub fn dump_dvd_cells(vts: u8, title_num: u16, title: &DvdTitle) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
let feature_start = title.feature_start_cell();
tracing::debug!(
target: DIAG,
"tag=dvd.pgc vts={vts} title={title_num} cells={} chapters={} \
dur={:.1}s feature_start_cell={feature_start}",
title.cells.len(),
title.chapters,
title.duration_secs,
);
for (i, cell) in title.cells.iter().enumerate() {
tracing::debug!(target: DIAG, "{}", dvd_cell_row(i, cell, i < feature_start));
}
// Chapter/PTT map (program → cumulative start time).
for (i, &t) in title.chapter_times.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.chap vts={vts} title={title_num} ch={} time={:.1}s",
i + 1,
t,
);
}
}
/// Emit the IFO `video_attr` / `audio_attr` decode for one DVD title set,
/// showing the raw bytes next to their decoded meaning. Called from the IFO
/// scan with the still-parsed `ifo::DvdTitleSet` view.
pub fn dump_dvd_attrs(ts: &crate::ifo::DvdTitleSet) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
tracing::debug!(
target: DIAG,
"tag=dvd.vobs vts={} vob_start_sector={}",
ts.vts_number,
ts.vob_start_sector,
);
let v = &ts.video;
tracing::debug!(
target: DIAG,
"tag=dvd.vattr vts={} codec={:?} res={} aspect={:?} std={:?}",
ts.vts_number,
v.codec,
res_str(v.resolution),
v.aspect,
v.standard,
);
for (i, a) in ts.audio_streams.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.aattr vts={} idx={i} codec={:?} ch={} sr={}Hz lang={:?} sub_id={:?}",
ts.vts_number,
a.codec,
a.channels,
a.sample_rate,
a.language,
a.sub_stream_id.map(|x| format!("0x{x:02X}")),
);
}
for (i, s) in ts.subtitle_streams.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=dvd.sattr vts={} idx={i} lang={:?}",
ts.vts_number,
s.language,
);
}
}
/// Emit the ACTUAL per-physical-sub-stream AC-3 channel counts read off the VOB
/// during the mux-time sub-stream probe (the Silence-of-the-Lambs wrong-stream
/// fix). This is the ground truth the IFO nibble is compared against: each row
/// is `sub_id=0x8x channels=N` for a physical `private_stream_1` AC-3 sub-stream
/// whose first frame was decoded. An empty probe (scrambled / unreadable / short
/// VOB) logs a single `probed=0` line so the absence is explicit in a bug log.
///
/// Self-sufficiency: with `tag=dvd.aattr` (the IFO's declared sub_id + claimed
/// channels) and these `tag=dvd.substream` rows (the physical reality), a bug
/// log alone shows whether the ordinal `0x80` actually carries the declared
/// channel layout — no disc needed to diagnose a wrong-substream rip.
pub fn dump_dvd_substream_probe(title_id: u16, probed: &std::collections::BTreeMap<u8, u8>) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
if probed.is_empty() {
tracing::debug!(
target: DIAG,
"tag=dvd.substream title={title_id} probed=0 (no AC-3 sync in feature head — scrambled/unreadable/none)",
);
return;
}
for (sub, ch) in probed {
tracing::debug!(
target: DIAG,
"tag=dvd.substream title={title_id} sub_id=0x{sub:02X} channels={ch} (physical acmod read from VOB)",
);
}
}
// ── MKV TrackEntry dump (the ACTUAL container elements written) ──────────────
/// `true` when the `--log-level 3` diagnostic target is enabled. Hot-path
/// callers (the opening-frame capture) check this once and skip all work when
/// off, so a normal run pays nothing.
pub fn diag_enabled() -> bool {
tracing::enabled!(target: DIAG, tracing::Level::DEBUG)
}
/// Cap on the number of codecPrivate bytes rendered to hex in a `tag=mkv.track`
/// line. The sequence header / avcC / hvcC prefix that matters for diagnosis
/// (resolution, frame rate, profile) is at the front; a multi-KB blob past this
/// is summarised as `..(+NB)` rather than flooding the log.
const CODEC_PRIVATE_HEX_CAP: usize = 64;
/// Render a track's codecPrivate as an uppercase-hex string for the diagnostic
/// line, capped at [`CODEC_PRIVATE_HEX_CAP`] bytes (`..(+NB)` suffix beyond).
/// `None` / empty → `"none"`. Pure (no logging) so it is directly unit-testable.
fn codec_private_hex(cp: Option<&[u8]>) -> String {
match cp {
Some(b) if !b.is_empty() => {
use std::fmt::Write;
let shown = b.len().min(CODEC_PRIVATE_HEX_CAP);
let mut s = String::with_capacity(shown * 2 + 8);
for byte in &b[..shown] {
let _ = write!(s, "{byte:02X}");
}
if b.len() > CODEC_PRIVATE_HEX_CAP {
let _ = write!(s, "..(+{}B)", b.len() - CODEC_PRIVATE_HEX_CAP);
}
s
}
_ => "none".to_string(),
}
}
/// Frame the raw bytes of one captured opening frame for the `.opening.bin` side
/// file: `[track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw bytes]`. Pure
/// (no I/O) so the record layout is directly unit-testable; `record` appends the
/// returned bytes to the side file.
fn frame_record(track_idx: usize, pts_ns: i64, keyframe: bool, data: &[u8]) -> Vec<u8> {
let mut rec = Vec::with_capacity(14 + data.len());
rec.push(track_idx as u8);
rec.push(keyframe as u8);
rec.extend_from_slice(&pts_ns.to_le_bytes());
rec.extend_from_slice(&(data.len() as u32).to_le_bytes());
rec.extend_from_slice(data);
rec
}
/// Emit the MKV `TrackEntry` elements the muxer is about to WRITE for one
/// track — the Windows-fps-class metadata (FlagInterlaced, FieldOrder,
/// DefaultDuration, DefaultDecodedFieldDuration, Display dims) plus the
/// codecPrivate as hex. With this row a bug log alone is enough to verify why
/// Windows Explorer reports a given frame rate for an interlaced SD track: the
/// container values that drive its fps derivation are all present, no disc and
/// no MediaInfo needed.
///
/// `track_number` is the 1-based MKV track number; `track` is the built
/// [`crate::mux::mkv::MkvTrack`] whose fields map one-to-one onto the emitted
/// elements (see `MkvMuxer::new`). No-op unless the diag target is on.
pub fn dump_mkv_track(track_number: u64, track: &crate::mux::mkv::MkvTrack) {
if !diag_enabled() {
return;
}
// codecPrivate as hex (capped so a multi-KB hvcC doesn't flood the log; the
// sequence header / avcC prefix that matters for diagnosis is at the front).
let cp = codec_private_hex(track.codec_private.as_deref());
let field_order = match track.field_order {
crate::mux::ebml::FIELD_ORDER_TFF => "TFF",
crate::mux::ebml::FIELD_ORDER_BFF => "BFF",
_ => "—",
};
// FlagInterlaced is only written for video tracks (1=interlaced/2=progressive);
// report what the muxer will emit, or "—" for non-video tracks where the
// element is omitted entirely.
let interlaced = if track.track_type == crate::mux::ebml::TRACK_TYPE_VIDEO {
if track.interlaced {
"1(interlaced)"
} else {
"2(progressive)"
}
} else {
"—"
};
tracing::debug!(
target: DIAG,
"tag=mkv.track num={track_number} type={} codec={} flag_interlaced={interlaced} \
field_order={field_order} default_duration_ns={} field_duration_ns={} \
pixel={}x{} display={}x{} cp_len={} cp_hex={cp}",
track.track_type,
track.codec_id,
track.default_duration_ns,
track.field_duration_ns,
track.pixel_width,
track.pixel_height,
track.display_width,
track.display_height,
track.codec_private.as_ref().map_or(0, |b| b.len()),
);
}
// ── Opening-frame capture (first ~N coded frames per track → side file) ──────
/// Number of coded frames captured PER TRACK before the capture goes dormant.
/// ~100 frames covers a DVD's first few seconds of every track (the
/// opening-GOP / still-frame / menu window where mid-GOP open or PTS-floor bugs
/// show up) while bounding the side file to a few MB even for HD I-frames.
const OPENING_FRAMES_PER_TRACK: usize = 100;
/// Captures the first [`OPENING_FRAMES_PER_TRACK`] coded frames of EACH track to
/// a side file (`<output>.opening.bin`) and logs a per-frame summary line, so an
/// opening-GOP / menu / mid-GOP-open issue is diagnosable from a future log +
/// side file WITHOUT the disc. Gated to `--log-level 3`: constructed only when
/// the diag target is on, so a normal run never opens the file or records a byte.
///
/// Side-file record framing (so a reader can split it back into frames):
/// `[track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw frame bytes]`.
pub struct OpeningCapture {
file: std::fs::File,
/// Frames captured so far, per track index. Capture for a track stops once
/// its counter reaches [`OPENING_FRAMES_PER_TRACK`].
counts: Vec<usize>,
}
impl OpeningCapture {
/// Open `<output>.opening.bin` next to the MKV output. Returns `None` (no
/// capture) when the diag target is off OR the side file can't be created —
/// a diagnostic must never fail the rip. `track_count` sizes the per-track
/// counters.
pub fn new(output_path: &std::path::Path, track_count: usize) -> Option<Self> {
if !diag_enabled() {
return None;
}
let mut name = output_path.as_os_str().to_os_string();
name.push(".opening.bin");
match std::fs::File::create(&name) {
Ok(file) => {
tracing::debug!(
target: DIAG,
"tag=mkv.opening.open path={:?} per_track_cap={OPENING_FRAMES_PER_TRACK}",
std::path::Path::new(&name),
);
Some(Self {
file,
counts: vec![0; track_count],
})
}
Err(e) => {
tracing::debug!(
target: DIAG,
"tag=mkv.opening.open path={:?} failed={e} (capture disabled, rip unaffected)",
std::path::Path::new(&name),
);
None
}
}
}
/// Record one coded frame for `track_idx` if that track is still under its
/// per-track cap. Writes the framed raw bytes to the side file and logs a
/// one-line summary. A write error disables further capture for the track
/// (counter pinned to the cap) but never propagates — the rip is unaffected.
pub fn record(&mut self, track_idx: usize, pts_ns: i64, keyframe: bool, data: &[u8]) {
let Some(count) = self.counts.get_mut(track_idx) else {
return;
};
if *count >= OPENING_FRAMES_PER_TRACK {
return;
}
use std::io::Write;
let rec = frame_record(track_idx, pts_ns, keyframe, data);
if let Err(e) = self.file.write_all(&rec) {
// Stop trying on this track; a broken side file must not stall mux.
*count = OPENING_FRAMES_PER_TRACK;
tracing::debug!(
target: DIAG,
"tag=mkv.opening.frame track={track_idx} write_failed={e} (capture stopped for track)",
);
return;
}
*count += 1;
tracing::debug!(
target: DIAG,
"tag=mkv.opening.frame track={track_idx} n={count} type={} size={} pts_ns={pts_ns}",
if keyframe { "key" } else { "delta" },
data.len(),
);
}
}
// ── Disc-level dump (post-lowering: titles, streams, decisions, AACS) ────────
/// Emit the full scan diagnostic block for a built [`Disc`]. Terse, one line
/// per row, under target `freemkv::diag` at DEBUG. No-op unless that target
/// is enabled, so it costs nothing when `--log-level 3` is off.
pub fn dump_disc(disc: &Disc) {
if !tracing::enabled!(target: DIAG, tracing::Level::DEBUG) {
return;
}
tracing::debug!(
target: DIAG,
"tag=disc vol={:?} format={:?} content={:?} cap_sectors={} layers={} titles={} encrypted={}",
disc.volume_id,
disc.format,
disc.content_format,
disc.capacity_sectors,
disc.layers,
disc.titles.len(),
disc.encrypted,
);
dump_aacs(disc);
for (ti, title) in disc.titles.iter().enumerate() {
dump_title(ti, title);
}
// freemkv's top-level DECISION: which title is the main feature.
if let Some(main) = disc.titles.first() {
tracing::debug!(
target: DIAG,
"tag=decision pick=main_feature title_idx=0 playlist={:?} dur={:.1}s \
size={}B clips={} reason=canonical_title_order(fits-disc, fewest-clips, longest, richest-audio)",
main.playlist,
main.duration_secs,
main.size_bytes,
main.clips.len(),
);
}
}
fn dump_aacs(disc: &Disc) {
let Some(a) = disc.aacs.as_ref() else {
if disc.css.is_some() {
tracing::debug!(target: DIAG, "tag=aacs none crypto=CSS(DVD)");
} else if disc.encrypted {
tracing::debug!(target: DIAG, "tag=aacs none crypto=encrypted-no-keys");
} else {
tracing::debug!(target: DIAG, "tag=aacs none crypto=clear");
}
return;
};
// CPS-unit / unit-key counts: at scan `unit_keys` is empty (keys are
// resolved later); the unit-key count is the BE16 in the raw
// Unit_Key_RO.inf if captured. Report both: resolved count and raw len.
tracing::debug!(
target: DIAG,
"tag=aacs version={} bus_enc={} mkb_version={:?} disc_hash={} key_source={:?} \
vuk={} unit_keys_resolved={} uk_ro_bytes={} mkb_bytes={}",
a.version,
a.bus_encryption,
a.mkb_version,
a.disc_hash,
a.key_source.name(),
a.vuk.is_some(),
a.unit_keys.len(),
a.uk_ro.len(),
a.mkb.len(),
);
}
fn dump_title(ti: usize, title: &DiscTitle) {
let (mut nv, mut na, mut ns) = (0u32, 0u32, 0u32);
for s in &title.streams {
match s {
Stream::Video(_) => nv += 1,
Stream::Audio(_) => na += 1,
Stream::Subtitle(_) => ns += 1,
}
}
tracing::debug!(
target: DIAG,
"tag=title idx={ti} playlist={:?} id={} dur={:.1}s size={}B clips={} \
extents={} chapters={} v={nv} a={na} s={ns} fmt={:?}",
title.playlist,
title.playlist_id,
title.duration_secs,
title.size_bytes,
title.clips.len(),
title.extents.len(),
title.chapters.len(),
title.content_format,
);
// Per-clip rows (BD: PlayItem/CLPI; DVD has none).
for (ci, c) in title.clips.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=clip title={ti} idx={ci} id={:?} in={} out={} dur={:.1}s src_packets={}",
c.clip_id,
c.in_time,
c.out_time,
c.duration_secs,
c.source_packets,
);
}
// Per-extent rows (the sectors freemkv will actually rip — the bug-4
// decision is visible here: leading non-feature cells are already gone).
for (ei, e) in title.extents.iter().enumerate() {
tracing::debug!(
target: DIAG,
"tag=extent title={ti} idx={ei} start_lba={} sectors={}",
e.start_lba,
e.sector_count,
);
}
// freemkv's per-stream DECISIONS (what the muxer will write).
for (si, s) in title.streams.iter().enumerate() {
match s {
Stream::Video(v) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=video pid=0x{:04X} codec={:?} \
res={} interlaced={} fps={} std={} color={} hdr={} aspect={:?} secondary={}",
v.pid,
v.codec,
res_str(v.resolution),
v.resolution.is_interlaced(),
fps_str(v.frame_rate),
tv_system_str(v.frame_rate),
color_str(v.color_space),
hdr_str(v.hdr),
v.display_aspect,
v.secondary,
),
Stream::Audio(a) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=audio pid=0x{:04X} codec={:?} \
channels={}({}) sr={}Hz lang={:?} secondary={}",
a.pid,
a.codec,
a.channels,
channel_count(a.channels),
sample_rate_hz(a.sample_rate),
a.language,
a.secondary,
),
Stream::Subtitle(sub) => tracing::debug!(
target: DIAG,
"tag=stream title={ti} idx={si} kind=subtitle pid=0x{:04X} codec={:?} \
lang={:?} forced={}",
sub.pid,
sub.codec,
sub.language,
sub.forced,
),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn res_str_keeps_interlace_marker() {
assert_eq!(res_str(Resolution::R576i), "576i");
assert_eq!(res_str(Resolution::R480i), "480i");
assert_eq!(res_str(Resolution::R2160p), "2160p");
}
#[test]
fn fps_and_tv_system() {
assert_eq!(fps_str(FrameRate::F25), "25");
assert_eq!(tv_system_str(FrameRate::F25), "PAL");
assert_eq!(fps_str(FrameRate::F29_97), "29.97");
assert_eq!(tv_system_str(FrameRate::F29_97), "NTSC");
}
#[test]
fn color_and_hdr() {
assert_eq!(color_str(ColorSpace::Bt470bg), "BT.470BG");
assert_eq!(color_str(ColorSpace::Bt2020), "BT.2020");
assert_eq!(hdr_str(HdrFormat::Hdr10), "HDR10");
assert_eq!(hdr_str(HdrFormat::DolbyVision), "DoVi");
assert_eq!(hdr_str(HdrFormat::Sdr), "SDR");
}
#[test]
fn channel_count_matches_layout() {
assert_eq!(channel_count(AudioChannels::Mono), 1);
assert_eq!(channel_count(AudioChannels::Stereo), 2);
assert_eq!(channel_count(AudioChannels::Surround51), 6);
assert_eq!(channel_count(AudioChannels::Surround71), 8);
}
#[test]
fn sample_rate_hz_values() {
assert_eq!(sample_rate_hz(SampleRate::S48), 48000);
assert_eq!(sample_rate_hz(SampleRate::S96), 96000);
}
#[test]
fn codec_private_hex_renders_caps_and_handles_empty() {
// None / empty → "none" (no hex). The Windows-fps diagnosis only needs
// the seq-header prefix, so render it but cap long blobs.
assert_eq!(codec_private_hex(None), "none");
assert_eq!(codec_private_hex(Some(&[])), "none");
// Short blob: full uppercase hex, no suffix. An MPEG-2 seq header starts
// 00 00 01 B3 — exactly what a reader greps for in a bug log.
assert_eq!(
codec_private_hex(Some(&[0x00, 0x00, 0x01, 0xB3])),
"000001B3"
);
// Over the cap: first CODEC_PRIVATE_HEX_CAP bytes + a "..(+NB)" summary.
let big = vec![0xABu8; CODEC_PRIVATE_HEX_CAP + 5];
let s = codec_private_hex(Some(&big));
assert!(s.starts_with(&"AB".repeat(CODEC_PRIVATE_HEX_CAP)), "{s}");
assert!(s.ends_with("..(+5B)"), "{s}");
}
#[test]
fn frame_record_layout_is_parseable() {
// The .opening.bin record framing must round-trip so a future tool can
// split the side file back into frames without the disc:
// [track:u8][keyframe:u8][pts_ns:i64 LE][len:u32 LE][raw bytes].
let data = [0xDEu8, 0xAD, 0xBE, 0xEF];
let rec = frame_record(2, -40_000_000, true, &data);
assert_eq!(rec.len(), 14 + data.len());
assert_eq!(rec[0], 2, "track index");
assert_eq!(rec[1], 1, "keyframe flag");
assert_eq!(
i64::from_le_bytes(rec[2..10].try_into().unwrap()),
-40_000_000,
"pts_ns survives (signed — opening back-anchor can be negative)"
);
assert_eq!(
u32::from_le_bytes(rec[10..14].try_into().unwrap()),
4,
"len"
);
assert_eq!(&rec[14..], &data, "raw frame bytes follow");
// A non-keyframe records the flag as 0.
let delta = frame_record(0, 0, false, &[]);
assert_eq!(delta[1], 0);
assert_eq!(u32::from_le_bytes(delta[10..14].try_into().unwrap()), 0);
}
/// The cell row shows the raw category byte (0xNN) beside the decode, and
/// the keep/drop verdict. A plain feature cell (0x00) is "keep"; a leading
/// secondary-block cell flagged dropped reads "DROP".
#[test]
fn cell_row_shows_raw_byte_and_verdict() {
let plain = crate::ifo::DvdCell {
first_sector: 100,
last_sector: 199,
category: 0x00,
duration_secs: 12.5,
};
let row = dvd_cell_row(0, &plain, false);
assert!(row.contains("cat=0x00"), "{row}");
assert!(row.contains("block_mode=0"), "{row}");
assert!(row.contains("first=100"), "{row}");
assert!(row.contains("last=199"), "{row}");
assert!(row.contains("dur=12.5s"), "{row}");
assert!(row.contains("keep(plain-feature)"), "{row}");
assert!(!row.contains("DROP"), "{row}");
// 0x90 = in-block cell of an angle block (block_mode=2, block_type=1),
// shown dropped as a leading secondary piece.
let sec = crate::ifo::DvdCell {
first_sector: 0,
last_sector: 9,
category: 0x90,
duration_secs: 1.0,
};
let row = dvd_cell_row(0, &sec, true);
assert!(row.contains("cat=0x90"), "{row}");
assert!(row.contains("block_mode=2"), "{row}");
assert!(row.contains("block_type=1"), "{row}");
assert!(row.contains("DROP(leading-secondary-block-piece)"), "{row}");
}
}
+1265 -32
View File
File diff suppressed because it is too large Load Diff
+1207 -17
View File
File diff suppressed because it is too large Load Diff
+466
View File
@@ -0,0 +1,466 @@
//! Physical AC-3 sub-stream probing for DVD audio routing.
//!
//! ## Why this exists (Silence-of-the-Lambs wrong-substream bug)
//!
//! A DVD VTS IFO declares its audio streams in a fixed table, and freemkv's
//! scan assigns each declared stream a `private_stream_1` sub-stream id purely
//! by per-codec ordinal — the first AC-3 stream becomes `0x80`, the second
//! `0x81`, and so on (`ifo::assign_audio_sub_stream_ids`). That assumes the
//! physical sub-stream order on the wire matches the IFO declaration order.
//!
//! On some discs it does NOT. The R2 PAL "The Silence of the Lambs" feature
//! declares ONE AC-3 audio stream the IFO nibble marks as 5.1 (6 channels), but
//! the physical VOB carries the 5.1 main mix and a 2.0 down-mix on DIFFERENT
//! `0x8x` sub-stream ids, and the 2.0 is the one that happens to land at the
//! ordinal `0x80` slot. Routing the declared 5.1 stream to `0x80` by ordinal
//! therefore muxes the 2.0 down-mix while labelling it 5.1 — the wrong physical
//! track.
//!
//! The robust fix is data-driven and codec/disc agnostic: read each physical
//! AC-3 sub-stream's REAL channel count from the VOB (the `acmod`/`lfeon` of its
//! first frame after the `0x0B77` sync) and route each IFO-declared AC-3 stream
//! to the physical sub-stream whose actual channel count matches the IFO's
//! declared count — instead of trusting the ordinal. This never re-reads the
//! disc beyond a bounded head-of-feature probe and degrades to the original
//! ordinal mapping when the probe yields nothing (unreadable/short VOB).
use crate::disc::Stream;
use crate::mux::codec::ac3;
use crate::mux::ps::PsDemuxer;
use crate::sector::SectorSource;
use std::collections::BTreeMap;
/// How many 2048-byte sectors of the first feature extent to probe. The head of
/// a DVD feature opens with logos/warnings whose audio is frequently a thin 2.0
/// bed on the FIRST sub-stream only — the other physical `0x8x` sub-streams and
/// the main 5.1 mix do not appear until a sector or two further in. 512 sectors
/// (1 MiB) was too short: on Greenland it saw ONLY `0x80`, and only its opening
/// 2.0 frames. 1024 sectors (2 MiB) reliably contains at least one frame of
/// every physical AC-3 sub-stream AND enough of `0x80` to reach its 5.1 frames.
/// Still bounded so a live drive is never hammered (see the project "don't
/// hammer the live drive" rule).
const PROBE_SECTORS: u16 = 1024;
/// Decode the real per-sub-stream AC-3 channel count from a buffer of decrypted
/// MPEG-PS (DVD VOB) bytes.
///
/// Demuxes `private_stream_1` (0xBD), and for each AC-3 sub-stream id
/// (`0x80..=0x87`) records the MAXIMUM channel count seen across EVERY decodable
/// frame in the probe window (`acmod` + `lfeon` at each `0x0B77` sync). Pure and
/// unit-testable — takes the already-read bytes, never touches the disc.
///
/// ## Why the maximum, not the first frame
///
/// The first frame of a sub-stream at the head of a feature is NOT
/// representative. A DVD opens with logos/warnings, and the main `0x80`
/// sub-stream there frequently carries a thin 2.0 bed before transitioning to
/// its real 5.1 main mix a fraction of a second later (observed on Greenland:
/// `0x80`'s first frames are acmod=2 → 2 channels, then it becomes acmod=7+lfe →
/// 6 channels within the same 2 MiB window). Recording only the FIRST frame read
/// `0x80=2` and missed the 5.1 entirely, defeating the channel-match routing.
/// The 5.1 capability of a sub-stream is the *maximum* channel count any of its
/// frames carries, so we scan them all and keep the max.
///
/// Returns a map `sub_id -> max channels`. Sub-streams whose frames are all too
/// short to carry the BSI bits, or that never appear in the buffer, are absent
/// from the map.
pub fn probe_ac3_substream_channels(ps_bytes: &[u8]) -> BTreeMap<u8, u8> {
let mut found: BTreeMap<u8, u8> = BTreeMap::new();
let mut demux = PsDemuxer::new();
let mut packets = demux.feed(ps_bytes);
packets.extend(demux.flush());
for p in packets {
// Only private_stream_1 AC-3 sub-streams (0x80..=0x87).
let Some(sub) = p.sub_stream_id else { continue };
if !(0x80..=0x87).contains(&sub) {
continue;
}
// The PS demux strips the 4-byte AC-3 sub-header but does not align to a
// frame. Walk EVERY 0x0B77 sync in this sub-stream's payload, decode
// each frame's channel count, and keep the largest — the sub-stream's
// real (main-mix) channel capability. See the doc comment above for why
// the first frame alone is unreliable.
if let Some(ch) = max_substream_channels(&p.data) {
let slot = found.entry(sub).or_insert(0);
*slot = (*slot).max(ch);
}
}
found
}
/// Largest AC-3 channel count over every decodable frame in a single
/// sub-stream's payload. Returns `None` when no frame carries enough BSI bits.
///
/// Each frame is advanced by its real `ac3_frame_size` so a frame's compressed
/// body (which can contain stray `0x0B77` byte pairs) cannot be mistaken for a
/// new frame; only when a size is unmappable do we fall back to a +2 byte
/// rescan to re-lock the next genuine sync.
fn max_substream_channels(data: &[u8]) -> Option<u8> {
let mut best: Option<u8> = None;
let mut pos = 0;
while pos < data.len() {
let Some(rel) = ac3::find_ac3_sync(&data[pos..]) else {
break;
};
let start = pos + rel;
let frame = &data[start..];
if let Some(ch) = ac3::acmod_channels(frame) {
if ch > 0 {
best = Some(best.map_or(ch, |b| b.max(ch)));
}
}
// Advance past this frame by its declared size when that is mappable;
// otherwise step 2 bytes past the sync and re-scan for the next one.
let size = ac3::ac3_frame_size(frame);
pos = if (6..=8192).contains(&size) {
start + size
} else {
start + 2
};
}
best
}
/// Re-route the title's declared AC-3 audio streams onto the physical
/// sub-stream ids whose REAL channel counts match, using a probed
/// `sub_id -> channels` map.
///
/// For each declared AC-3 audio stream (in IFO order), it picks the physical
/// `0x8x` sub-stream whose probed channel count equals the stream's declared
/// channel count, never re-using a sub-stream already claimed by an earlier
/// stream. The chosen sub-stream's PID (`0xBD00 | sub_id`) is written back onto
/// the `Stream::Audio` so BOTH mux demux paths (`DiscStream` and the file-backed
/// highway) route by it.
///
/// Conservative — it only ever REASSIGNS among the physical sub-streams the
/// probe actually saw, and only when a better (exact-channel) match exists than
/// the stream's current assignment. A stream whose current sub-stream already
/// matches is left alone; a stream with no matching physical sub-stream keeps
/// its ordinal assignment. So a normal disc (physical order == IFO order) is a
/// no-op.
///
/// Returns the number of streams whose PID was changed (for diagnostics).
pub fn remap_audio_pids(streams: &mut [Stream], probed: &BTreeMap<u8, u8>) -> usize {
if probed.is_empty() {
return 0;
}
// Sub-streams already claimed by a remapped (or matching) earlier stream,
// so two declared streams never collide on one physical sub-stream.
let mut claimed: Vec<u8> = Vec::new();
let mut changed = 0usize;
for s in streams.iter_mut() {
let Stream::Audio(a) = s else { continue };
if a.codec != crate::disc::Codec::Ac3 {
continue;
}
let declared = a.channels.count();
// The sub-id this stream currently routes by (low byte of its PID).
let current_sub = (a.pid & 0x00FF) as u8;
// If the stream's current physical sub-stream already matches its
// declared channel count, keep it and claim it.
if probed.get(&current_sub) == Some(&declared) {
claimed.push(current_sub);
continue;
}
// Otherwise find an unclaimed physical sub-stream whose REAL channel
// count equals the declared count.
let pick = probed
.iter()
.find(|(sub, ch)| **ch == declared && !claimed.contains(*sub))
.map(|(sub, _)| *sub);
if let Some(sub) = pick {
let new_pid = 0xBD00 | sub as u16;
if new_pid != a.pid {
tracing::debug!(
target: "freemkv::scan",
old_pid = a.pid,
new_pid,
declared_channels = declared,
"dvd: re-routed AC-3 audio to physical sub-stream matching channel count"
);
a.pid = new_pid;
changed += 1;
}
claimed.push(sub);
} else {
// No physical match — leave the ordinal assignment, but claim its
// current sub so later streams don't steal a slot it may still use.
claimed.push(current_sub);
}
}
changed
}
/// Probe the first feature extent of a DVD title through a (decrypted) sector
/// source and re-route its AC-3 audio PIDs to the physically-correct
/// sub-streams. A bounded, best-effort scan: any read error or empty probe
/// leaves the ordinal assignment untouched.
///
/// `reader` MUST yield PLAINTEXT VOB bytes (i.e. a `DecryptingSectorSource` on a
/// CSS disc) — probing scrambled sectors yields no AC-3 syncs and is a safe
/// no-op. Returns the number of audio streams whose PID changed.
pub fn probe_and_remap<S: SectorSource + ?Sized>(
reader: &mut S,
title: &mut crate::disc::DiscTitle,
) {
// Only DVD (MPEG-PS) titles carry private_stream_1 AC-3 sub-streams.
if title.content_format != crate::disc::ContentFormat::MpegPs {
return;
}
// Nothing to disambiguate unless there is at least one AC-3 audio stream.
let has_ac3 = title
.streams
.iter()
.any(|s| matches!(s, Stream::Audio(a) if a.codec == crate::disc::Codec::Ac3));
if !has_ac3 {
return;
}
let Some(ext) = title.extents.first() else {
return;
};
let count: u16 = ext.sector_count.min(PROBE_SECTORS as u32) as u16;
if count == 0 {
return;
}
let mut buf = vec![0u8; count as usize * 2048];
// `recovery=false`: a single best-effort attempt — the probe must never
// stall the mux or hammer a marginal drive. On any error, bail to ordinal.
let n = match reader.read_sectors(ext.start_lba, count, &mut buf, false) {
Ok(n) => n,
Err(_) => return,
};
buf.truncate(n);
let probed = probe_ac3_substream_channels(&buf);
crate::diag::dump_dvd_substream_probe(title.playlist_id, &probed);
remap_audio_pids(&mut title.streams, &probed);
}
#[cfg(test)]
mod tests {
use super::*;
use crate::disc::{AudioChannels, AudioStream, Codec, LabelPurpose, SampleRate};
/// Build a single, correctly-SIZED AC-3 frame whose `acmod`/`lfeon` encode a
/// known channel count. `byte4` is `fscod=0 | frmsizecod=0`, so
/// `ac3_frame_size` reports 128 bytes and the frame is zero-padded to exactly
/// that — this lets `max_substream_channels` advance frame-by-frame over a
/// multi-frame payload exactly as it does on real VOB data. The BSI bits are
/// laid down with a writer so the test never hand-miscomputes the lfeon
/// offset, matching `acmod_channels`' reader.
fn ac3_frame(acmod: u8, lfeon: bool) -> Vec<u8> {
let mut bits: Vec<u8> = Vec::new();
let push = |val: u32, n: usize, bits: &mut Vec<u8>| {
for i in (0..n).rev() {
bits.push(((val >> i) & 1) as u8);
}
};
push(acmod as u32, 3, &mut bits);
if (acmod & 0x1) != 0 && acmod != 0x1 {
push(0, 2, &mut bits); // cmixlev
}
if (acmod & 0x4) != 0 {
push(0, 2, &mut bits); // surmixlev
}
if acmod == 0x2 {
push(0, 2, &mut bits); // dsurmod
}
push(lfeon as u32, 1, &mut bits);
// Pack the bit vector MSB-first into bytes (byte6 onward).
let mut tail = Vec::new();
let mut cur = 0u8;
for (i, b) in bits.iter().enumerate() {
cur = (cur << 1) | b;
if i % 8 == 7 {
tail.push(cur);
cur = 0;
}
}
let rem = bits.len() % 8;
if rem != 0 {
cur <<= 8 - rem;
tail.push(cur);
}
// AC-3 frame: 0x0B 0x77 crc(2) byte4(fscod=0,frmsizecod=0) bsid<<3 then BSI.
let mut frame = vec![0x0B, 0x77, 0x00, 0x00, 0x00, 8u8 << 3];
frame.extend_from_slice(&tail);
// frmsizecod=0 @ 48kHz → 64 words = 128 bytes. Pad to the real size so
// the frame-stepping in max_substream_channels lands on the next sync.
frame.resize(128, 0);
frame
}
/// Build a minimal `private_stream_1` PES carrying `frames` for `sub_id`,
/// each preceded only by the 4-byte AC-3 sub-header at the PES head. Mirrors
/// the on-disc layout the PS demux expects: PES start `0x000001BD`, length,
/// PES header (no PTS), sub-header `[sub_id, frame_count, ptr_hi, ptr_lo]`,
/// then the concatenated AC-3 frames.
fn ps_ac3_frames(sub_id: u8, frames: &[Vec<u8>]) -> Vec<u8> {
// PES sub-header for AC-3: sub_id + frame_count + 2-byte access ptr.
let mut payload = vec![sub_id, frames.len() as u8, 0x00, 0x04];
for f in frames {
payload.extend_from_slice(f);
}
// PES packet: start code 00 00 01 BD, length(2), flags(2), hdr_len(0).
let pes_payload_len = 3 + payload.len(); // flags(2)+hdrlen(1)+payload
let mut pkt = vec![0x00, 0x00, 0x01, 0xBD];
pkt.extend_from_slice(&(pes_payload_len as u16).to_be_bytes());
pkt.extend_from_slice(&[0x80, 0x00, 0x00]); // no PTS, header_data_len=0
pkt.extend_from_slice(&payload);
pkt
}
/// Single-frame `private_stream_1` PES — the common case in existing tests.
fn ps_ac3(sub_id: u8, acmod: u8, lfeon: bool) -> Vec<u8> {
ps_ac3_frames(sub_id, &[ac3_frame(acmod, lfeon)])
}
fn ac3_stream(pid: u16, channels: AudioChannels) -> Stream {
Stream::Audio(AudioStream {
pid,
codec: Codec::Ac3,
channels,
language: "en".into(),
sample_rate: SampleRate::S48,
secondary: false,
purpose: LabelPurpose::Normal,
label: String::new(),
})
}
/// The probe decodes the real channel count of each physical sub-stream.
/// 0x80 carries a 2.0 frame (acmod=2,no lfe → 2ch); 0x81 carries 5.1
/// (acmod=7 + lfe → 6ch).
#[test]
fn probe_decodes_per_substream_channels() {
let mut bytes = ps_ac3(0x80, 2, false);
bytes.extend(ps_ac3(0x81, 7, true));
let probed = probe_ac3_substream_channels(&bytes);
assert_eq!(probed.get(&0x80), Some(&2), "0x80 is the 2.0 down-mix");
assert_eq!(probed.get(&0x81), Some(&6), "0x81 is the 5.1 main mix");
}
/// GREENLAND regression — the probe must read each sub-stream's TRUE
/// (max-mix) channel count, not be poisoned by an unrepresentative head
/// frame, and must NOT cross-contaminate between sub-streams.
///
/// Mirrors the real on-disc layout that caused the mis-read: the feature
/// head carries `0x80` opening with a 2.0 frame and THEN a 5.1 frame (its
/// real main mix), interleaved with `0x81` carrying only 2.0. The old
/// first-frame probe read `0x80=2` (the logo bed) and missed the 5.1; the
/// max-over-frames probe must report `0x80=6` and `0x81=2`.
#[test]
fn probe_reads_max_channels_no_cross_contamination() {
let mut bytes = Vec::new();
// 0x80 opens with a 2.0 frame (the logo bed)...
bytes.extend(ps_ac3_frames(0x80, &[ac3_frame(2, false)]));
// ...0x81 interleaves a pure-2.0 PES (must NOT bleed 6 into 0x80)...
bytes.extend(ps_ac3_frames(
0x81,
&[ac3_frame(2, false), ac3_frame(2, false)],
));
// ...then 0x80 reaches its real 5.1 main mix (acmod=7 + lfe → 6 ch),
// with a trailing 2.0 frame in the SAME PES to prove we take the max,
// not the last frame.
bytes.extend(ps_ac3_frames(
0x80,
&[ac3_frame(7, true), ac3_frame(2, false)],
));
let probed = probe_ac3_substream_channels(&bytes);
assert_eq!(
probed.get(&0x80),
Some(&6),
"0x80's real 5.1 mix must win over its 2.0 head/tail frames"
);
assert_eq!(
probed.get(&0x81),
Some(&2),
"0x81 is a pure 2.0 stream — must not absorb 0x80's 6-channel frame"
);
}
/// SILENCE-OF-THE-LAMBS regression: the IFO declares ONE 5.1 AC-3 stream and
/// the ordinal mapping put it at 0x80, but physically 0x80 is the 2.0
/// down-mix and the 5.1 lives at 0x81. After probe+remap the declared 5.1
/// stream must route to 0x81 (PID 0xBD81), NOT the ordinal 0x80.
#[test]
fn remap_routes_declared_51_to_physical_51_substream() {
// Physical layout: 0x80 = 2.0, 0x81 = 5.1 (reversed vs ordinal).
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 2u8);
probed.insert(0x81u8, 6u8);
// Declared: one 5.1 stream, ordinally assigned 0x80 (PID 0xBD80).
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 1, "the one 5.1 stream must be re-routed");
let Stream::Audio(a) = &streams[0] else {
panic!("audio")
};
assert_eq!(
a.pid, 0xBD81,
"declared 5.1 must route to physical 0x81 (the real 5.1), not ordinal 0x80"
);
}
/// Conservative no-op: when the physical order already matches the IFO
/// order (0x80 = 5.1 as declared), remap changes nothing.
#[test]
fn remap_noop_when_physical_matches_ordinal() {
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 6u8); // 0x80 really is the 5.1
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 0, "matching physical order is a no-op");
let Stream::Audio(a) = &streams[0] else {
panic!()
};
assert_eq!(a.pid, 0xBD80);
}
/// Two declared streams (5.1 + 2.0) where the physical order is reversed:
/// 0x80=2.0, 0x81=5.1. The 5.1 declaration must claim 0x81 and the 2.0
/// declaration must claim 0x80 — no collision, both correct.
#[test]
fn remap_two_streams_no_collision() {
let mut probed = BTreeMap::new();
probed.insert(0x80u8, 2u8);
probed.insert(0x81u8, 6u8);
// Declared order: 5.1 first (ordinal 0x80), 2.0 second (ordinal 0x81).
let mut streams = vec![
ac3_stream(0xBD80, AudioChannels::Surround51),
ac3_stream(0xBD81, AudioChannels::Stereo),
];
remap_audio_pids(&mut streams, &probed);
let pids: Vec<u16> = streams
.iter()
.filter_map(|s| match s {
Stream::Audio(a) => Some(a.pid),
_ => None,
})
.collect();
assert_eq!(
pids,
vec![0xBD81, 0xBD80],
"5.1→0x81, 2.0→0x80, no collision"
);
}
/// Empty probe (unreadable / scrambled VOB) is a no-op — the ordinal
/// assignment survives so behaviour never regresses below today's.
#[test]
fn remap_empty_probe_is_noop() {
let probed = BTreeMap::new();
let mut streams = vec![ac3_stream(0xBD80, AudioChannels::Surround51)];
let changed = remap_audio_pids(&mut streams, &probed);
assert_eq!(changed, 0);
let Stream::Audio(a) = &streams[0] else {
panic!()
};
assert_eq!(a.pid, 0xBD80, "no probe data → keep ordinal");
}
}
+1076 -192
View File
File diff suppressed because it is too large Load Diff
+1646
View File
File diff suppressed because it is too large Load Diff
+1395
View File
File diff suppressed because it is too large Load Diff
+1080 -20
View File
File diff suppressed because it is too large Load Diff
+4383 -1557
View File
File diff suppressed because it is too large Load Diff
+1420 -25
View File
File diff suppressed because it is too large Load Diff
+413 -38
View File
@@ -1,11 +1,10 @@
//! Single source of truth for what to do when a sector read fails.
//!
//! Both Pass 1 (`Disc::sweep`) and Pass 2-N (`Disc::patch`) call into
//! `handle_read_error` after every failed `read_sectors`. The handler
//! classifies the error, updates the in-flight context (counters,
//! damage window, retry budgets), and returns a `ReadAction` the caller
//! dispatches on. Every read goes through the same gate — no path can
//! silently skip pause/skip/jump/abort logic.
//! Pass 1 (`Disc::sweep`) calls into `handle_read_error` after every failed
//! `read_sectors`. The handler classifies the error, updates the in-flight
//! context (counters, damage window, retry budgets), and returns a
//! `ReadAction` the caller dispatches on. Pass N patch has its own
//! `handle_read_failure` in `disc/patch.rs` that does not route here.
//!
//! Adding a new error class = add one arm in `handle_read_error`.
//! Adding new logging on errors = one place.
@@ -34,14 +33,20 @@ pub struct ReadCtx {
/// Sliding window of recent read outcomes (true=ok, false=fail).
/// Capped at `damage_window_max`. Drives damage-jump decisions.
pub damage_window: Vec<bool>,
/// Maximum number of outcome entries kept in `damage_window`; the
/// oldest is evicted once this is exceeded. A whole count (e.g. 16).
pub damage_window_max: usize,
/// Fraction of `damage_window` entries that must be failures before
/// the window-based damage-jump fires, as a whole-number percentage
/// (e.g. `12` = 12%).
pub damage_threshold_pct: usize,
/// Trigger a damage-jump after this many consecutive outer-batch
/// failures, even when the damage_window isn't full yet. Pass 1
/// uses a small value (4) so we don't spend ~40 minutes grinding
/// to fill a 16-block window before the first jump on a damage
/// zone we entered cleanly. Pass N uses a larger value (or
/// disables this — see `bisect_on_marginal`) because Pass N's
/// uses a small value (1 — jump on the first outer failure; see
/// the 2026-05-11 rewrite in `for_sweep`) so we don't spend ~40
/// minutes grinding to fill a 16-block window before the first jump
/// on a damage zone we entered cleanly. Pass N uses a larger value
/// (or disables this — see `bisect_on_marginal`) because Pass N's
/// whole job IS to grind on the bad ranges.
pub fast_jump_threshold: u64,
/// Multiplier applied to damage-jump distance. Doubles each jump,
@@ -151,8 +156,12 @@ impl ReadCtx {
/// outer-batch failure — the user's wedge-prevention principle
/// (2026-05-11): once the drive returns ANY recoverable error,
/// retrying the same LBA quickly is what triggers the firmware
/// fast-fail transition. Jump immediately, never retry in Pass 1.
/// Pass N owns retries — it gets per-sector timeouts that don't
/// fast-fail transition. On the damage-jump and marginal paths Pass 1
/// jumps immediately rather than grinding the same LBA. Transient errors
/// (NOT_READY, bridge degradation) are still retried a small bounded
/// number of times (`NOT_READY_MAX_RETRIES` / `BRIDGE_DEGRADATION_MAX_RETRIES`)
/// in both passes before falling through to the skip path.
/// Pass N owns the heavy retries — it gets per-sector timeouts that don't
/// hammer the firmware the same way.
pub fn for_sweep(batch: u16) -> Self {
Self {
@@ -186,6 +195,11 @@ impl ReadCtx {
/// marginal media is part of the job, and the fast-jump
/// threshold is loose so we don't bail too early on a range that
/// has scattered good sectors mixed in.
///
/// `damage_threshold_pct = 6` is looser than Pass 1 (12%): Pass N triggers
/// the damage-skip at half Pass 1 density because the patch loop exists to chip
/// away at bad ranges, so being more eager to skip clustered bad sectors
/// converges faster on the recoverable good sectors inside a range.
pub fn for_patch(batch: u16) -> Self {
Self {
batch,
@@ -194,7 +208,7 @@ impl ReadCtx {
consecutive_outer_failures: 0,
damage_window: Vec::with_capacity(16),
damage_window_max: 16,
damage_threshold_pct: 12,
damage_threshold_pct: PATCH_DAMAGE_THRESHOLD_PCT,
// Pass N is allowed to grind: window-based jump only,
// matching the historical behaviour for patch passes.
fast_jump_threshold: u64::MAX,
@@ -224,6 +238,13 @@ impl ReadCtx {
// drive recovered, so further wedges should reset the skip
// budget instead of accumulating toward a real abort.
self.wedge_count = 0;
// A successful read also means the bridge recovered, so the
// 15s-cooldown retry budget should be available again for the
// next bridge-degradation event. Without this reset the budget
// saturates permanently after 5 cumulative events across the
// whole pass and later degradations skip the cooldown retry,
// needlessly losing data.
self.bridge_degradation_count = 0;
// Outer-success only: a good single-sector read inside a
// bisect doesn't mean we've left the damaged batch. Only an
// outer-batch success resets the outer-failure counter.
@@ -243,6 +264,13 @@ impl ReadCtx {
if self.in_damage_zone && self.consecutive_good >= self.damage_window_max as u64 {
self.in_damage_zone = false;
self.last_error_family = None;
// Reset the damage-jump multiplier so the NEXT zone starts
// from the base jump distance. Without this the multiplier
// stays at whatever the prior zone inflated it to (up to
// MAX_JUMP_MULTIPLIER=64), so the next zone's first jump is
// 64x oversized and skips recoverable data. The field doc
// promises this reset.
self.jump_multiplier = 1;
}
}
@@ -300,7 +328,8 @@ pub enum ReadAction {
// bridge wedges 524 ms after a 5.4-second internal ECC retry. The
// post-failure pauses give the drive — and the bridge — time to settle.
/// Pause between a failed read and the next read attempt — applied
/// uniformly to Pass 1 sweep and Pass N patch.
/// by Pass 1 sweep via `handle_read_error`. Pass N patch uses its own
/// `POST_FAILURE_PAUSE_SECS` (see `disc/patch.rs`).
///
/// 2026-05-11 reframe: a failed read is a failed read, regardless of
/// which pass is running. The prior split (1s for Pass N, 5s for Pass
@@ -318,7 +347,7 @@ const FAIL_PAUSE_SECS: u64 = 5;
/// FIRST read failure after a clean run, before the drive has had a
/// chance to cycle in retries that push it toward fast-fail).
///
/// Empirical: 2026-05-11 Dune Pt 2 wedge incident showed 7 medium
/// Empirical: a 2026-05-11 wedge incident showed 7 medium
/// errors in 6.5 seconds (~1s per attempt + ~1s pause) push the
/// BU40N's firmware into IllegalRequest fast-fail mode permanently.
/// Once there, only physical eject + reload clears it. Giving the
@@ -329,7 +358,7 @@ const FAIL_PAUSE_SECS: u64 = 5;
/// Cost on clean discs: zero (first-error path doesn't trigger).
/// Cost on damaged discs: ~30s × N damage zones; on a 5-zone disc
/// that's 2.5 min extra. Trade for never wedging the drive.
const ZONE_ENTRY_COOLDOWN_SECS: u64 = 30;
pub(crate) const ZONE_ENTRY_COOLDOWN_SECS: u64 = 30;
/// Cooldown when a long streak of failures suggests the drive is
/// stuck in a damage zone and needs MORE breathing room than the
/// standard inter-error pause. Same value as `FAIL_PAUSE_SECS`
@@ -359,9 +388,9 @@ const JUMP_BASE_SECTORS: u64 = 1024;
// When the BU40N (or similar drives) hits a physical-damage cluster,
// its firmware can transition into a "wedge" state where it returns
// HARDWARE_ERROR or ILLEGAL_REQUEST for every subsequent read —
// often for many LBAs after the actual bad sector. Per project docs
// "Bad-sector handling" rule #2: "Recovery requires eject+reload OR
// significant cool-down."
// often for many LBAs after the actual bad sector. Once wedged,
// recovery requires either a physical eject + reload or a significant
// cool-down period; hammering the same LBA only deepens the state.
//
// Pass 1's pre-fix behavior was to immediately AbortPass on the
// first HARDWARE_ERROR / ILLEGAL_REQUEST, killing the rip at
@@ -380,10 +409,10 @@ const JUMP_BASE_SECTORS: u64 = 1024;
/// One-gigabyte jump (1024 MiB) on each wedge. Big enough to clear
/// almost any single-cluster damage zone we've seen.
const WEDGE_JUMP_SECTORS: u64 = 524_288;
/// Cooldown pause after each wedge. Per project docs the drive needs
/// "significant cool-down"; 30 s strikes a balance between giving
/// the drive a chance to recover and not stalling the rip if the
/// drive is permanently stuck.
/// Cooldown pause after each wedge. A wedged drive needs a
/// significant cool-down to leave fast-fail; 30 s strikes a balance
/// between giving the drive a chance to recover and not stalling the
/// rip if the drive is permanently stuck.
const WEDGE_PAUSE_SECS: u64 = 30;
/// Bail after this many consecutive wedges with no good read in
/// between. At 1 GB jumps this lets us scan ~16 GB worth of fully
@@ -401,6 +430,18 @@ const WEDGE_ABORT_THRESHOLD: u64 = 16;
/// next range.
const WEDGE_PASS_N_SKIP_SECTORS: u64 = 64;
/// Single source of truth for the Pass-N damage-window threshold.
/// [`ReadCtx::for_patch`] reads this constant for the Pass-N damage-skip
/// threshold.
///
/// 6% means: with a 16-entry sliding window, the damage-skip fires
/// once 1 out of 16 recent reads has failed. Pass 1 uses a 12%
/// threshold via `damage_threshold_pct` on `for_sweep`; Pass N is
/// twice as eager because patch's whole job is to converge on the
/// bad sub-zones inside a NonTrimmed range — a faster trigger
/// produces tighter convergence in fewer iterations.
pub const PATCH_DAMAGE_THRESHOLD_PCT: usize = 6;
/// THE single error-handling entry point. Updates `ctx`, returns the
/// action the caller must apply.
///
@@ -435,8 +476,13 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
.unwrap_or(SenseFamily::Other);
// Zone-entry tracking: this is the first error after a clean run
// (or the first error of the sweep).
if !ctx.in_damage_zone && !ctx.bisecting {
// (or the first error of the sweep). Capture the genuine
// clean->damaged transition here, BEFORE mutating in_damage_zone,
// so the 30s zone-entry cooldown below keys off the real
// transition rather than re-deriving it from a counter that the
// fast-jump path resets after every jump.
let is_zone_entry_transition = !ctx.in_damage_zone && !ctx.bisecting;
if is_zone_entry_transition {
ctx.in_damage_zone = true;
ctx.zones_entered += 1;
}
@@ -473,13 +519,20 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
);
if is_wedge_transition {
// NOTE: this is the FIRST escalation into the hardware/illegal-request
// sense family — NOT a confirmed wedge. Drives frequently recover and keep
// reading after one such error (a single bad spot), so calling it a "wedge"
// here over-claims (it sent past investigations chasing a drive ghost). A
// genuine wedge is PERSISTENT — see the `wedge_skip` / WEDGE_ABORT_THRESHOLD
// path below, which only fires after repeated fast-fails with no recovery.
tracing::warn!(
target: "freemkv::disc",
phase = "wedge_transition",
phase = "fastfail_escalation",
errors_in_zone = ctx.total_errors,
ms_since_last_success,
new_family = ?current_family,
"drive entered wedge / fast-fail family (was returning recoverable medium errors before this)"
"drive escalated into the fast-fail sense family (was returning recoverable medium \
errors before this) — often transient; only a PERSISTENT run is a real wedge"
);
}
@@ -491,11 +544,17 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
return ReadAction::AbortPass;
}
// 2. Bridge degradation: NOT_READY with the well-known signature
// (sense_key=2, ASC=0x04, ASCQ=0x3E). Drive's bridge is in a
// semi-stuck state but typically recovers after a long cooldown.
// If we've exhausted our retry budget, fall through to the
// marginal/skip path below.
// 2. Bridge degradation: the SCSI status byte is non-standard —
// neither GOOD (0x00), CHECK CONDITION (0x02), nor TRANSPORT
// FAILURE (0xFF). The USB bridge firmware returns these bogus
// status bytes (e.g. 0x04, 0x05) with empty sense data when it
// enters a semi-stuck state preceding a crash. This is keyed on
// the status byte alone, NOT on sense_key/ASC/ASCQ — a real
// NOT_READY 04/3E bad-sector error arrives as CHECK CONDITION
// (0x02) and is handled by the generic NOT_READY branch below.
// The bridge typically recovers after a long cooldown; if we've
// exhausted our retry budget, fall through to the marginal/skip
// path below.
if err.is_bridge_degradation() && ctx.bridge_degradation_count < BRIDGE_DEGRADATION_MAX_RETRIES
{
ctx.bridge_degradation_count += 1;
@@ -541,9 +600,13 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
// AbortPass after N consecutive wedges with no successful
// read in between.
if sense_key == scsi::SENSE_KEY_HARDWARE_ERROR || sense_key == scsi::SENSE_KEY_ILLEGAL_REQUEST {
if !ctx.bisecting {
ctx.wedge_count += 1;
}
// Count every wedge, including bisect-inner ones. A wedge is a
// firmware fast-fail state regardless of whether we're inside a
// bisect; if we did NOT count bisect-inner wedges, a drive that
// wedges mid-bisect would burn a 30s WEDGE_PAUSE cooldown per
// inner sector and never reach WEDGE_ABORT_THRESHOLD from inside
// the bisect — ~16 min of cooldown sleeping on a batch=32 bisect.
ctx.wedge_count += 1;
if ctx.wedge_count >= WEDGE_ABORT_THRESHOLD {
tracing::warn!(
target: "freemkv::disc",
@@ -637,8 +700,7 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
// branch for future tuning. Pass N (bisect_on_marginal=true)
// uses the standard pauses — it's running single-sector retries
// on already-known-bad LBAs by design.
let is_zone_entry =
ctx.consecutive_outer_failures == 1 && !ctx.bisecting && !ctx.bisect_on_marginal;
let is_zone_entry = is_zone_entry_transition && !ctx.bisecting && !ctx.bisect_on_marginal;
let pause_secs = if is_zone_entry {
ZONE_ENTRY_COOLDOWN_SECS
} else if ctx.consecutive_failures >= CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD {
@@ -664,7 +726,7 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
// Two triggers, evaluated in order:
//
// a. **Fast-entry** — `consecutive_outer_failures >= fast_jump_threshold`.
// Fires on Pass 1 (threshold=4) so we don't spend ~40 min
// Fires on Pass 1 (threshold=1) so we don't spend ~40 min
// grinding to fill a 16-block damage window before the
// first jump on a damage zone we entered cleanly. Doesn't
// fire on Pass N (threshold=u64::MAX).
@@ -1001,6 +1063,43 @@ mod tests {
}
}
#[test]
fn pass_1_subsequent_in_zone_errors_skip_long_cooldown() {
// Regression: the fast-jump path resets consecutive_outer_failures
// to 0 after each jump, so the next in-zone error re-increments it
// to 1. Zone-entry must key off the genuine clean->damaged
// transition (in_damage_zone), not the counter, otherwise every
// error in a damaged region pays the 30 s cooldown.
let mut ctx = ReadCtx::for_sweep(32);
// First error: genuine zone entry, gets the long cooldown.
let first = handle_read_error(&medium_err(), &mut ctx);
match first {
ReadAction::JumpAhead { pause_secs, .. } => assert_eq!(
pause_secs,
ZONE_ENTRY_COOLDOWN_SECS + POST_JUMP_EXTRA_PAUSE_SECS
),
other => panic!("expected JumpAhead on first error, got {other:?}"),
}
// We are now still in the damage zone; the jump reset the outer
// counter. A second error must NOT re-arm the 30 s cooldown.
assert!(ctx.in_damage_zone);
let second = handle_read_error(&medium_err(), &mut ctx);
let pause = match second {
ReadAction::JumpAhead { pause_secs, .. } => pause_secs,
ReadAction::SkipBlock { pause_secs } => pause_secs,
other => panic!("expected pausing action, got {other:?}"),
};
assert_ne!(
pause,
ZONE_ENTRY_COOLDOWN_SECS + POST_JUMP_EXTRA_PAUSE_SECS,
"subsequent in-zone error must not pay the 30 s zone-entry cooldown"
);
assert!(
pause <= FAIL_PAUSE_SECS + POST_JUMP_EXTRA_PAUSE_SECS,
"subsequent in-zone pause should be the standard fail pause, got {pause}"
);
}
#[test]
fn pass_n_pauses_uniformly_on_failed_read() {
// Pass N (bisect_on_marginal=true) is exempt from the
@@ -1039,6 +1138,66 @@ mod tests {
);
}
#[test]
fn jump_multiplier_resets_after_damage_zone_exit() {
// A zone that doubles the multiplier must not carry the inflated
// value into the next zone — otherwise the next zone's first
// jump is up to 64x oversized and skips recoverable data.
let mut ctx = ReadCtx::for_sweep(32);
// First zone: a few errors push jumps and double the multiplier.
for _ in 0..4 {
handle_read_error(&medium_err(), &mut ctx);
}
assert!(
ctx.jump_multiplier > 1,
"expected the multiplier to inflate inside a damage zone"
);
// Exit the zone: damage_window_max consecutive good reads.
ctx.bisecting = false;
for _ in 0..ctx.damage_window_max {
ctx.on_success();
}
assert!(!ctx.in_damage_zone, "zone should have exited");
assert_eq!(
ctx.jump_multiplier, 1,
"jump_multiplier must reset to 1 on zone exit"
);
}
#[test]
fn bridge_degradation_count_resets_on_success() {
// After a good read the bridge recovered; the 15s-cooldown retry
// budget must be available again instead of staying saturated
// for the whole pass.
let mut ctx = ReadCtx::for_patch(1);
ctx.bridge_degradation_count = BRIDGE_DEGRADATION_MAX_RETRIES;
ctx.on_success();
assert_eq!(ctx.bridge_degradation_count, 0);
}
#[test]
fn wedge_abort_reachable_during_bisect() {
// A drive that wedges mid-bisect must still reach the abort
// threshold rather than burning a WEDGE_PAUSE cooldown per inner
// sector forever.
let mut ctx = ReadCtx::for_patch(32);
ctx.bisecting = true;
let mut aborted = false;
for _ in 0..WEDGE_ABORT_THRESHOLD {
if matches!(
handle_read_error(&hardware_err(), &mut ctx),
ReadAction::AbortPass
) {
aborted = true;
break;
}
}
assert!(
aborted,
"wedge abort threshold must be reachable from inside a bisect"
);
}
#[test]
fn on_success_resets_failure_counters_and_pushes_window() {
let mut ctx = ReadCtx::for_sweep(32);
@@ -1052,4 +1211,220 @@ mod tests {
assert_eq!(ctx.consecutive_failures, 0);
assert!(*ctx.damage_window.last().unwrap());
}
// ----------------------------------------------------------------
// Additional hardening: retry-budget boundaries, transport-abort
// precedence, and the bounded-jump invariant. These guard against
// off-by-one in the retry caps (which would either hammer a wedging
// drive or give up a recovery one attempt early) and against an
// unbounded jump multiplier skipping the rest of the disc.
// ----------------------------------------------------------------
/// NOT_READY check-condition (status 0x02 so it is NOT classified as
/// bridge degradation, which keys off non-standard status bytes).
/// sense_key=2 with a generic ASC routes to the NOT_READY retry path.
fn not_ready_err() -> Error {
Error::DiscRead {
sector: 100,
status: Some(crate::scsi::SCSI_STATUS_CHECK_CONDITION),
sense: Some(ScsiSense {
sense_key: scsi::SENSE_KEY_NOT_READY,
asc: 0x04,
ascq: 0x00,
}),
}
}
/// Transport failure: SCSI status 0xFF (bridge crash). CLAUDE.md
/// "Bad-sector handling": this aborts the copy.
fn transport_failure_err() -> Error {
Error::DiscRead {
sector: 100,
status: Some(crate::scsi::SCSI_STATUS_TRANSPORT_FAILURE),
sense: None,
}
}
/// Bridge degradation: a non-standard status byte (0x04 - neither
/// GOOD/CHECK/TRANSPORT) with empty sense, per `Error::is_bridge_degradation`.
fn bridge_degradation_err() -> Error {
Error::DiscRead {
sector: 100,
status: Some(0x04),
sense: None,
}
}
#[test]
fn not_ready_retries_capped_at_three_then_falls_through() {
// CLAUDE.md "Bad-sector handling" mode 1: NOT READY -> "Pause 3s,
// retry up to 3x, then mark NonTrimmed." NOT_READY_MAX_RETRIES=3.
// The 1st-3rd NOT_READY must Retry; the 4th must NOT Retry (it
// falls through to skip). Pass N (batch=1) so the marginal-bisect
// branch is irrelevant.
// Mutation that makes this RED: change `ctx.not_ready_retries <
// NOT_READY_MAX_RETRIES` to `<=` (retries 4 times) or to `>`
// (never retries).
let mut ctx = ReadCtx::for_patch(1);
for i in 0..NOT_READY_MAX_RETRIES {
let a = handle_read_error(&not_ready_err(), &mut ctx);
assert!(
matches!(a, ReadAction::Retry { .. }),
"NOT_READY attempt {i} should Retry, got {a:?}"
);
}
// Budget exhausted: the next NOT_READY must not Retry.
let a = handle_read_error(&not_ready_err(), &mut ctx);
assert!(
!matches!(a, ReadAction::Retry { .. }),
"NOT_READY past the retry cap must fall through, got {a:?}"
);
}
#[test]
fn transport_failure_aborts_even_mid_bisect() {
// CLAUDE.md "Bad-sector handling" mode 2: a transport failure
// (bridge crash, status 0xFF) aborts the pass so the outer loop
// can re-enumerate the bridge. This must hold even while
// bisecting and even on Pass N - the wedge-skip/jump paths must
// NOT swallow a real transport crash into a JumpAhead.
// Mutation that makes this RED: move the transport-failure check
// below the HARDWARE/ILLEGAL wedge arm, so a transport failure
// that also carried a wedge-family sense would JumpAhead instead.
let mut ctx = ReadCtx::for_patch(32);
ctx.bisecting = true;
assert_eq!(
handle_read_error(&transport_failure_err(), &mut ctx),
ReadAction::AbortPass
);
// And on a fresh Pass 1 context, still AbortPass.
let mut ctx1 = ReadCtx::for_sweep(32);
assert_eq!(
handle_read_error(&transport_failure_err(), &mut ctx1),
ReadAction::AbortPass
);
}
#[test]
fn bridge_degradation_retries_to_budget_then_falls_through() {
// The bridge-degradation cooldown retry is bounded by
// BRIDGE_DEGRADATION_MAX_RETRIES (=5). The first 5 degradation
// errors must Retry with the long bridge cooldown; the 6th must
// fall through to skip/jump rather than retrying forever and
// stalling the pass.
// Mutation that makes this RED: change the budget comparison
// `ctx.bridge_degradation_count < BRIDGE_DEGRADATION_MAX_RETRIES`
// to `<=` (retries 6 times).
let mut ctx = ReadCtx::for_patch(1);
for i in 0..BRIDGE_DEGRADATION_MAX_RETRIES {
let a = handle_read_error(&bridge_degradation_err(), &mut ctx);
match a {
ReadAction::Retry { pause_secs } => {
assert_eq!(
pause_secs, BRIDGE_DEGRADATION_PAUSE_SECS,
"bridge retry {i} should use the bridge cooldown"
);
}
other => panic!("bridge degradation attempt {i} should Retry, got {other:?}"),
}
}
let a = handle_read_error(&bridge_degradation_err(), &mut ctx);
assert!(
!matches!(a, ReadAction::Retry { .. }),
"bridge degradation past the retry budget must fall through, got {a:?}"
);
}
/// The documented BU40N bad-sector signature: NOT_READY
/// (sense_key=2, ASC=0x04, ASCQ=0x3E) delivered as a CHECK CONDITION
/// (status 0x02). This is the case the old comment on the bridge
/// branch wrongly claimed `is_bridge_degradation` matched.
fn not_ready_04_3e_err() -> Error {
Error::DiscRead {
sector: 100,
status: Some(crate::scsi::SCSI_STATUS_CHECK_CONDITION),
sense: Some(ScsiSense {
sense_key: scsi::SENSE_KEY_NOT_READY,
asc: 0x04,
ascq: 0x3E,
}),
}
}
#[test]
fn not_ready_04_3e_does_not_take_bridge_branch() {
// Regression guard for the misleading-comment fix: the bridge
// branch keys on the *status byte* (non-standard, i.e. not
// GOOD/CHECK/TRANSPORT), NOT on the NOT_READY 04/3E sense. A real
// 04/3E bad-sector error arrives as CHECK CONDITION (0x02), so
// `is_bridge_degradation()` must be false for it, and it must
// route to the generic NOT_READY retry (3 s pause) rather than
// the bridge cooldown (15 s pause).
let err = not_ready_04_3e_err();
assert!(
!err.is_bridge_degradation(),
"04/3E arrives as CHECK CONDITION (0x02); it is not bridge degradation"
);
let mut ctx = ReadCtx::for_patch(1);
match handle_read_error(&err, &mut ctx) {
ReadAction::Retry { pause_secs } => {
assert_eq!(
pause_secs, NOT_READY_PAUSE_SECS,
"04/3E must use the generic NOT_READY pause, not the bridge cooldown"
);
assert_ne!(
pause_secs, BRIDGE_DEGRADATION_PAUSE_SECS,
"04/3E must not take the bridge-degradation branch"
);
// Confirm it really went through the NOT_READY path.
assert_eq!(ctx.not_ready_retries, 1);
assert_eq!(ctx.bridge_degradation_count, 0);
}
other => panic!("04/3E should Retry via the NOT_READY path, got {other:?}"),
}
}
#[test]
fn jump_multiplier_caps_and_jump_distance_stays_bounded() {
// CLAUDE.md damage-jump: multiplier doubles per jump but is
// capped at MAX_JUMP_MULTIPLIER=64 (the "4 GiB cap"); a single
// jump must never be allowed to grow without bound and skip the
// rest of the disc. Drive a long single-sector failure streak on
// a sweep ctx with a tiny window so window-trigger jumps fire
// repeatedly, and verify the multiplier saturates at 64 and the
// emitted jump distance equals JUMP_BASE_SECTORS * batch * 64.
// Mutation that makes this RED: remove the
// `.min(MAX_JUMP_MULTIPLIER)` on the multiplier doubling, or use
// wrapping/non-saturating mul -> distance overshoots or panics.
const MAX_JUMP_MULTIPLIER: u64 = 64;
let batch: u16 = 32;
let mut ctx = ReadCtx::for_sweep(batch);
// Small window + 0% threshold so every failure can window-trigger
// a jump and keep doubling the multiplier toward the cap.
ctx.damage_window_max = 2;
ctx.damage_threshold_pct = 0;
let mut last_jump_sectors = 0u64;
for _ in 0..40 {
// Reset bisecting flag defensively; these are outer failures.
ctx.bisecting = false;
if let ReadAction::JumpAhead { sectors, .. } =
handle_read_error(&medium_err(), &mut ctx)
{
last_jump_sectors = sectors;
}
assert!(
ctx.jump_multiplier <= MAX_JUMP_MULTIPLIER,
"jump_multiplier {} exceeded the cap {}",
ctx.jump_multiplier,
MAX_JUMP_MULTIPLIER
);
}
// After saturation, the jump distance is exactly base*batch*cap.
let expected = JUMP_BASE_SECTORS * batch as u64 * MAX_JUMP_MULTIPLIER;
assert_eq!(
last_jump_sectors, expected,
"saturated jump distance must equal base*batch*64"
);
}
}
File diff suppressed because it is too large Load Diff
+30 -51
View File
@@ -8,26 +8,23 @@
//! during the post-read work; throughput tops out at the *sum* of
//! both costs.
//!
//! 0.17.11 introduced a bespoke producer/consumer split (the now-
//! removed `disc/sweep_pipeline.rs`) to overlap the two stages. 0.18
//! collapses that split — together with the analogous splits patch
//! and mux need — onto the generic [`crate::io::Pipeline`] +
//! [`crate::io::Sink`] primitive. This module is the sweep-specific
//! `Sink` impl; the producer-side state machine (read_error context,
//! decrypt, set_speed, halt) stays in `Disc::sweep` in `disc/mod.rs`.
//! A producer/consumer split overlaps the two stages on the generic
//! [`crate::io::Pipeline`] + [`crate::io::Sink`] primitive. This module
//! is the sweep-specific `Sink` impl; the producer-side state machine
//! (read_error context, decrypt, set_speed, halt) stays in
//! `Disc::sweep` in `disc/mod.rs`.
//!
//! Correctness invariants preserved (same as 0.17.11):
//! Correctness invariants preserved:
//! - Mapfile is single-writer (consumer-only). No locking.
//! - All `read_error::ReadCtx` state stays on the producer thread.
//! - `set_speed` calls happen on the producer thread (same thread that
//! owns the `SectorReader`). No new SCSI concurrency.
//! owns the `SectorSource`). No new SCSI concurrency.
//! - Per-iteration ordering of file-write → mapfile-record is kept
//! intact in the consumer (write before record), so the on-disk
//! invariant "mapfile only marks Finished what the file has
//! received" survives a crash mid-pass.
//! - The BU40N+Initio bridge wedge concern is unchanged: only one
//! SCSI command in flight at a time, error-path timing identical,
//! no new retry logic.
//! - Only one SCSI command is in flight at a time; error-path timing
//! is identical and no new retry logic is introduced.
use std::io::{Seek, SeekFrom, Write};
use std::sync::mpsc::{Receiver, SyncSender, sync_channel};
@@ -40,7 +37,7 @@ use super::mapfile::{MapStats, Mapfile, SectorStatus};
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. 64 KB
/// matches the existing zero_gap chunk size used by the pre-split
/// sweep loop.
const ZERO_CHUNK: usize = 65 * 1024;
const ZERO_CHUNK: usize = 64 * 1024;
/// Producer → Consumer messages. The consumer applies these in FIFO
/// order; ordering of file writes and mapfile records across items is
@@ -151,63 +148,45 @@ impl Sink<WorkItem> for SweepSink {
WorkItem::Good { pos, buf } => {
// Decrypt is on the producer; consumer assumes plaintext.
let len = buf.len() as u64;
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf)
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, len, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&buf)?;
self.map.record(pos, len, SectorStatus::Finished)?;
}
WorkItem::BisectGood { pos, buf } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf[..])
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, 2048, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&buf[..])?;
self.map.record(pos, 2048, SectorStatus::Finished)?;
}
WorkItem::BisectBad { pos } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&self.zero[..2048])
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, 2048, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
self.file.seek(SeekFrom::Start(pos))?;
self.file.write_all(&self.zero[..2048])?;
self.map.record(pos, 2048, SectorStatus::NonTrimmed)?;
}
WorkItem::SkipFill { pos, len } | WorkItem::GapFill { pos, len } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file.seek(SeekFrom::Start(pos))?;
// Subsequent writes are sequential; `WritebackFile`'s
// seek-elision keeps them on the writeback pipeline path.
let mut filled = 0u64;
while filled < len {
let chunk = (len - filled).min(self.zero.len() as u64) as usize;
self.file
.write_all(&self.zero[..chunk])
.map_err(|e| Error::IoError { source: e })?;
self.file.write_all(&self.zero[..chunk])?;
filled += chunk as u64;
}
self.map
.record(pos, len, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
self.map.record(pos, len, SectorStatus::NonTrimmed)?;
}
WorkItem::StatsRequest => {
let stats = self.map.stats();
// DAMAGE only — NOT NonTried. NonTried is the unread remainder
// ahead of the sweep head, not damage; including it made the live
// located drilldown (at-risk movie time + range count) treat the
// whole unread disc as confirmed damage, so at sweep start it
// showed ~full-movie at-risk and melted to 0 as the sweep
// progressed. Matches the one-shot progress path, which already
// excludes NonTried.
let bad_ranges = self.map.ranges_with(&[
SectorStatus::NonTrimmed,
SectorStatus::Unreadable,
SectorStatus::NonScraped,
SectorStatus::NonTried,
]);
// Best-effort: drop on backpressure; producer's cache
// stays current enough.
@@ -230,7 +209,7 @@ impl Sink<WorkItem> for SweepSink {
// Non-regular outputs (/dev/null, pipes) always fail
// sync_all; that's not a real error.
}
self.map.flush().map_err(|e| Error::IoError { source: e })?;
self.map.flush()?;
Ok(ConsumerSummary {
stats: self.map.stats(),
+80
View File
@@ -25,8 +25,14 @@ pub struct DriveCapture {
/// A single GET CONFIGURATION feature response from the drive.
#[derive(Debug, Clone)]
pub struct CapturedFeature {
/// MMC-6 GET CONFIGURATION feature code (e.g. `0x010D` = AACS).
pub code: u16,
/// Static human-readable label from the internal `FEATURES` table —
/// not a device-reported string.
pub name: &'static str,
/// Raw feature-descriptor payload bytes, with the 8-byte GET
/// CONFIGURATION header stripped (i.e. `buf[8..]`). Unlike
/// [`DriveCapture::gc_010c`], which retains the full header.
pub data: Vec<u8>,
}
@@ -114,3 +120,77 @@ pub fn mask_bytes(data: &[u8]) -> Vec<u8> {
})
.collect()
}
#[cfg(test)]
mod tests {
//! Privacy-masking + capture-orchestration tests.
//!
//! `mask_string` / `mask_bytes` redact identifying characters before
//! a drive capture leaves the machine: every ASCII letter → 'A',
//! every ASCII digit → '0', everything else (punctuation, spaces,
//! control bytes, non-ASCII) is preserved verbatim so structural
//! framing (offsets, separators) survives for diffing.
use super::*;
#[test]
fn mask_string_letters_become_a_digits_become_zero() {
// Mixed case letters all collapse to 'A'; digits to '0'.
assert_eq!(mask_string("HL-DT-ST"), "AA-AA-AA");
assert_eq!(mask_string("BU40N"), "AA00A");
}
#[test]
fn mask_string_preserves_non_alnum_punctuation_and_space() {
// Separators and spaces must be preserved so the masked output
// keeps the same shape as the original (the whole point of a
// structure-preserving redaction).
assert_eq!(mask_string("1.04"), "0.00");
assert_eq!(mask_string("a b-c.d_e"), "A A-A.A_A");
}
#[test]
fn mask_string_preserves_non_ascii_chars() {
// is_ascii_alphabetic/is_ascii_digit are false for non-ASCII, so
// multibyte chars pass through unchanged (no mojibake, no panic).
// 'c','a','f' are ASCII letters → 'A'; 'é' is non-ASCII →
// preserved; '9' → '0'.
assert_eq!(mask_string("café9"), "AAAé0");
}
#[test]
fn mask_bytes_matches_string_masking_for_ascii() {
// mask_bytes is the byte-wise analogue: letters→b'A', digits→b'0'.
assert_eq!(mask_bytes(b"HL-DT-ST"), b"AA-AA-AA".to_vec());
assert_eq!(mask_bytes(b"1.04"), b"0.00".to_vec());
}
#[test]
fn mask_bytes_preserves_non_alnum_and_high_bytes() {
// Control bytes (0x00), high bytes (0xFF), and punctuation are
// not ASCII alnum and must survive verbatim — INQUIRY payloads
// are space-padded binary and the framing must be diffable.
let input = [0x00u8, b'A', 0x20, b'7', 0xFF, b'-'];
assert_eq!(mask_bytes(&input), vec![0x00, b'A', 0x20, b'0', 0xFF, b'-']);
}
#[test]
fn feature_table_has_no_duplicate_codes() {
// capture_drive_data iterates FEATURES once per code; a duplicate
// code would silently capture the same feature twice (and bloat
// the report). Each MMC-6 feature code must be unique.
let mut seen = std::collections::HashSet::new();
for &(code, _name) in FEATURES {
assert!(seen.insert(code), "duplicate feature code {code:#06x}");
}
}
#[test]
fn feature_table_includes_aacs_010d() {
// AACS (0x010D) is the feature that gates UHD decryption capture;
// it must be in the table or AACS drives capture incompletely.
assert!(
FEATURES.iter().any(|&(c, _)| c == 0x010D),
"AACS feature 0x010D must be captured"
);
}
}
+68 -16
View File
@@ -1,18 +1,33 @@
//! Linux drive discovery and device resolution.
use crate::drive::DeviceResolution;
use crate::error::{Error, Result};
use crate::identity::DriveId;
/// SCSI peripheral device type 5 = MMC / optical (CD/DVD/BD), held in the
/// low 5 bits of INQUIRY byte 0 (the high 3 bits are the peripheral
/// qualifier, masked off here).
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives by enumerating `/dev/sg*` SCSI-generic nodes,
/// opening each, running INQUIRY, and keeping only devices whose
/// peripheral device type is optical (MMC, type 0x05).
///
/// Devices where `scsi::open` or `DriveId::from_drive` fail are silently
/// skipped — that is intentional for enumeration (a busy or wedged node
/// shouldn't abort discovery of the others).
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
for i in 0..16 {
let path = format!("/dev/sg{i}");
for name in enumerate_sg_names() {
let path = format!("/dev/{name}");
if !std::path::Path::new(&path).exists() {
continue;
}
if let Ok(mut transport) = crate::scsi::open(std::path::Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty() && (id.raw_inquiry[0] & 0x1F) == 0x05 {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((path, id));
}
}
@@ -21,41 +36,78 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
drives
}
/// Enumerate `sg*` device names. Linux assigns `/dev/sgN` sequentially
/// across *all* SCSI-generic devices (disks, tape, HBAs, optical), so a
/// fixed `sg0..15` range can miss an optical drive on a host with many
/// targets. Prefer the exact present-device list from
/// `/sys/class/scsi_generic/`; fall back to a bounded `sg0..15` probe
/// only when sysfs is unreadable (minimal containers).
fn enumerate_sg_names() -> Vec<String> {
let mut names = Vec::new();
if let Ok(entries) = std::fs::read_dir("/sys/class/scsi_generic") {
for entry in entries.flatten() {
let name = entry.file_name().to_string_lossy().to_string();
if name.starts_with("sg") {
names.push(name);
}
}
} else {
for i in 0..16 {
let name = format!("sg{i}");
if std::path::Path::new(&format!("/dev/{name}")).exists() {
names.push(name);
}
}
}
names.sort();
names
}
/// Resolve a device path to its raw `/dev/sg*` SCSI-generic node.
///
/// - `/dev/sg*` paths pass through unchanged ([`DeviceResolution::Direct`]).
/// - `/dev/sr*` block paths are matched (by vendor/product/serial) to the
/// corresponding `/dev/sg*` node ([`DeviceResolution::SrToSg`]); if no
/// match is found the original path is returned with
/// [`DeviceResolution::SrNoSgMatch`].
/// - Any other existing path passes through as [`DeviceResolution::Direct`].
#[allow(dead_code)]
pub fn resolve_device(path: &str) -> Result<(String, Option<String>)> {
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
if path.contains("/sg") {
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
return Ok((path.to_string(), None));
return Ok((path.to_string(), DeviceResolution::Direct));
}
if path.contains("/sr") {
let mut sr_transport = crate::scsi::open(std::path::Path::new(path))?;
let sr_id = DriveId::from_drive(sr_transport.as_mut())?;
drop(sr_transport);
for (sg_path, sg_id) in find_drives() {
if sg_id.vendor_id == sr_id.vendor_id
// Require a non-empty serial before treating vendor/product/
// serial as a unique match. serial_number falls back to an
// empty string when GET CONFIGURATION 0108h is unavailable
// (common on OEM drives); two same-model drives would then
// both compare equal and the first in enumeration order would
// win silently, resolving sr1 to sr0's sg node. An empty
// serial can't disambiguate, so fall through to the no-match
// path instead.
if !sr_id.serial_number.is_empty()
&& sg_id.vendor_id == sr_id.vendor_id
&& sg_id.product_id == sr_id.product_id
&& sg_id.serial_number == sr_id.serial_number
{
let warning =
format!("{path} is a block device (sr) — using {sg_path} (sg) for raw access");
return Ok((sg_path, Some(warning)));
return Ok((sg_path, DeviceResolution::SrToSg));
}
}
return Ok((
path.to_string(),
Some(format!(
"{path} is a block device (sr) — no matching sg device found"
)),
));
return Ok((path.to_string(), DeviceResolution::SrNoSgMatch));
}
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
Ok((path.to_string(), None))
Ok((path.to_string(), DeviceResolution::Direct))
}
+22 -12
View File
@@ -4,9 +4,20 @@
//! to discover optical drives without exclusive access or unmounts. Only
//! the returned paths are then opened for INQUIRY to build full `DriveId`.
use crate::drive::DeviceResolution;
use crate::error::{Error, Result};
use crate::identity::DriveId;
/// SCSI peripheral device type 5 = MMC / optical, in the low 5 bits of
/// INQUIRY byte 0.
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives via the IOKit registry (`scsi::list_drives`),
/// then open each candidate for INQUIRY to build a full `DriveId`.
///
/// Any drive where `scsi::open` or `DriveId::from_drive` fails, or whose
/// peripheral device type is not optical (MMC, type 0x05), is silently
/// skipped — the same MMC filter the Linux and Windows backends apply.
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
let discovered = crate::scsi::list_drives();
@@ -15,7 +26,11 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
match crate::scsi::open(path) {
Ok(mut transport) => {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
drives.push((info.path.clone(), id));
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((info.path.clone(), id));
}
}
}
Err(_) => {
@@ -26,20 +41,15 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
drives
}
pub fn resolve_device(path: &str) -> Result<(String, Option<String>)> {
// Accept /dev/diskN or /dev/rdiskN paths as-is
if path.contains("/disk") || path.contains("/rdisk") {
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
return Ok((path.to_string(), None));
}
/// Resolve a device path on macOS. There is no `sr`→`sg` style
/// substitution here (that is a Linux concern), so any existing path is
/// returned unchanged as [`DeviceResolution::Direct`]; the
/// [`DeviceResolution`] return exists for cross-platform signature parity.
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
if !std::path::Path::new(path).exists() {
return Err(Error::DeviceNotFound {
path: path.to_string(),
});
}
Ok((path.to_string(), None))
Ok((path.to_string(), DeviceResolution::Direct))
}
+1262 -106
View File
File diff suppressed because it is too large Load Diff
+23 -8
View File
@@ -1,9 +1,18 @@
//! Windows drive discovery and device resolution.
use crate::drive::DeviceResolution;
use crate::error::Result;
use crate::identity::DriveId;
use std::path::Path;
/// SCSI peripheral device type 5 = MMC / optical, in the low 5 bits of
/// INQUIRY byte 0.
const SCSI_PERIPHERAL_TYPE_OPTICAL: u8 = 0x05;
/// Discover optical drives. Probes `\\.\CdRom0..15` first; only if none
/// are found does it fall back to scanning drive letters `D..Z`. Each
/// candidate is opened, INQUIRY'd, and kept only if its peripheral device
/// type is optical (MMC, type 0x05). Returns normalized `\\.\` paths.
pub fn find_drives() -> Vec<(String, DriveId)> {
let mut drives = Vec::new();
@@ -12,7 +21,9 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
let path = format!("\\\\.\\CdRom{}", i);
if let Ok(mut transport) = crate::scsi::open(Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty() && (id.raw_inquiry[0] & 0x1F) == 0x05 {
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
drives.push((path, id));
}
}
@@ -25,8 +36,12 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
let path = format!("{}:", letter as char);
if let Ok(mut transport) = crate::scsi::open(Path::new(&path)) {
if let Ok(id) = DriveId::from_drive(transport.as_mut()) {
if !id.raw_inquiry.is_empty() && (id.raw_inquiry[0] & 0x1F) == 0x05 {
drives.push((path, id));
if !id.raw_inquiry.is_empty()
&& (id.raw_inquiry[0] & 0x1F) == SCSI_PERIPHERAL_TYPE_OPTICAL
{
// Normalize so returned paths are consistently in
// \\.\ form regardless of which loop matched.
drives.push((normalize_path(&path), id));
}
}
}
@@ -36,8 +51,11 @@ pub fn find_drives() -> Vec<(String, DriveId)> {
drives
}
pub fn resolve_device(path: &str) -> Result<(String, Option<String>)> {
Ok((normalize_path(path), None))
/// Resolve a device path to its normalized Windows `\\.\` form. Windows
/// has no `sr`→`sg` symlink-target indirection, so resolution is purely a
/// path normalization and always reports [`DeviceResolution::Direct`].
pub fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
Ok((normalize_path(path), DeviceResolution::Direct))
}
/// Normalize a device path to Windows \\.\X: format.
@@ -55,9 +73,6 @@ fn normalize_path(path: &str) -> String {
if trimmed.len() == 2 && trimmed.as_bytes()[1] == b':' {
return format!("\\\\.\\{}", trimmed);
}
if path.to_lowercase().starts_with("cdrom") {
return format!("\\\\.\\{}", path);
}
format!("\\\\.\\{}", path)
}
+49
View File
@@ -0,0 +1,49 @@
//! DVD-Video navigation — read-only resolver for the **main-feature start
//! point** (issue #40). Mirrors what a DVD player's nav VM resolves: First-Play
//! → menu "Play" → title dispatch → the first cell of the feature, so the rip
//! starts at the movie rather than at raw cell 0 (e.g. skipping a leading
//! logo/warning segment when the disc's own navigation does).
//!
//! Byte layout follows the DVD-Video specification (VMGI/VTSI headers,
//! PGC/cell tables, PCI/HLI button packets); the VM command decoder is
//! verified against real discs.
//!
//! Current contents: [`vmcmd`] — the VM command decoder (proven against the
//! SOTL/Greenland test discs). The IFO/PCI parsing and the navigation executor
//! that resolves the start cell build on top of this.
pub mod vmcmd;
use crate::sector::SectorSource;
/// Resolve the feature title's **true start cell** (0-based index into the
/// title PGC's cell list) by following the disc's own navigation — First-Play →
/// menu "Play" → title dispatch — the way a player reaches the movie. This is
/// what lets the rip begin at the feature instead of at raw cell 0 when the
/// disc's nav enters the title past a leading logo/warning segment (e.g. a
/// disc whose "Play" resolves to a later cell than cell 0).
///
/// Returns `None` when navigation cannot be resolved, so the caller falls back
/// to the structural leading-cell filter (today's behaviour, ≈ cell 0 / 0:00).
///
/// TODO(#40): the IFO/PCI parsing + nav executor (built on [`vmcmd`]) land
/// incrementally. Until the executor is complete this returns `None`, so wiring
/// it in is behaviour-neutral; improvements to the resolver take effect here
/// without touching the call site.
pub fn resolve_feature_start(
reader: &mut dyn SectorSource,
udf: &crate::udf::UdfFs,
vtsn: u16,
vts_ttn: u16,
) -> Option<usize> {
// `reader`/`udf` are the seam inputs the nav executor will consume to read
// VIDEO_TS.IFO + the VTS IFOs/menu VOBs. Reserved until that lands.
let _ = (reader, udf);
tracing::trace!(
target: "freemkv::dvdnav",
vtsn,
vts_ttn,
"nav start-cell resolver: unresolved — caller falls back to leading-cell filter"
);
None
}
+408
View File
@@ -0,0 +1,408 @@
//! DVD-Video VM command decoder.
//!
//! An 8-byte navigation command as found in PGC command tables (pre/post/cell)
//! and PCI button info. Decoded per the DVD-Video VM instruction set and
//! verified against real discs.
//!
//! Bit model: the 8 bytes are a big-endian 64-bit word. `byte0` bits 7-5 are the
//! command **type**; for type 1, `byte0` bit 4 selects Link (0) vs Jump (1), and
//! `byte1` bits 3-0 are the sub-command. Compare predicates live in `byte1`
//! bits 6-4 with the operands in bytes 2-5.
//!
//! This module is pure decode + a register model — no I/O, no English (numeric
//! semantics only), matching libfreemkv conventions. The navigation *executor*
//! and IFO/PCI parsing build on top of this.
/// A decoded navigation instruction. Only the variants freemkv's start-point
/// resolver needs are modelled explicitly; everything else is [`Instr::Other`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Instr {
Nop,
/// Stop executing the current command list (resume cell playback).
Break,
/// Goto command line within the same list (1-based).
Goto {
line: u8,
},
/// Leave the current domain.
Exit,
/// Jump to a VMG title (1-based TT_SRPT index).
JumpTt {
ttn: u8,
},
/// Jump to a title within the current VTS (1-based VTS title index).
JumpVtsTt {
ttn: u8,
},
/// Jump to a part-of-title (chapter) within a VTS title.
JumpVtsPtt {
ttn: u8,
pttn: u16,
},
/// Jump to the First-Play PGC.
JumpSsFp,
/// Jump to a Video-Manager menu (`menu` = menu id).
JumpSsVmgm {
menu: u8,
},
/// Jump to a Video-Title-Set menu.
JumpSsVtsm {
vts: u8,
ttn: u8,
menu: u8,
},
/// Jump to a specific VMGM menu PGC.
JumpSsVmgmPgc {
pgcn: u16,
},
/// Call a sub-domain (raw retained; resume handled by the executor).
CallSs {
sub: u8,
},
/// Link to a PGC number within the current domain.
LinkPgcn {
pgcn: u16,
},
/// Link to a part-of-title within the current PGC's title.
LinkPttn {
pttn: u16,
},
/// Link to a program number within the current PGC (1-based).
LinkPgn {
pgn: u8,
},
/// Link to a cell number within the current PGC (1-based).
LinkCn {
cn: u8,
},
/// A link "subset" op (LinkTopCell/NextPG/RSM/…); `sub` is the raw code.
LinkSub {
sub: u8,
},
/// Set a GPRM. `op` is the set-op code (1=mov, 3=add, …); value is immediate
/// (`imm`) when `immediate`, else the contents of register `src`.
SetGprm {
reg: u8,
op: u8,
immediate: bool,
imm: u16,
src: u8,
},
/// Set a system parameter / unmodelled set — executor may ignore.
SetSystem,
/// Anything not individually modelled (kept as raw bytes).
Other([u8; 8]),
}
/// A compare predicate carried by a command (`byte1` bits 6-4). `None` = always.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Compare {
/// Compare op: 1=&,2===,3=!=,4=>=,5=>,6=<=,7=<.
pub op: u8,
/// Left register index (GPRM 0-15, SPRM 128+).
pub lhs_reg: u8,
/// Right side: immediate when `immediate`, else register `rhs_reg`.
pub immediate: bool,
pub imm: u16,
pub rhs_reg: u8,
}
/// A fully decoded command: its predicate (if any) and the instruction.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Command {
pub compare: Option<Compare>,
pub instr: Instr,
}
// Command types — `byte0` bits 7-5.
const TYPE_SPECIAL: u8 = 0;
const TYPE_LINK_JUMP: u8 = 1;
const TYPE_SET_SYSTEM: u8 = 2;
const TYPE_SET_GPRM: u8 = 3;
// Special (type 0) sub-commands — `byte1` bits 3-0.
const SP_GOTO: u8 = 1;
const SP_BREAK: u8 = 2;
// Jump/Call (type 1, direct=1) sub-commands.
const JP_EXIT: u8 = 1;
const JP_JUMP_TT: u8 = 2;
const JP_JUMP_VTS_TT: u8 = 3;
const JP_JUMP_VTS_PTT: u8 = 5;
const JP_JUMP_SS: u8 = 6;
const JP_CALL_SS: u8 = 8;
// Link (type 1, direct=0) sub-commands. NOTE: sub-op 0 is NOP/no-link and 1 is
// the LinkSub form (the DVD-Video VM link instruction).
const LK_SUB: u8 = 1;
const LK_PGCN: u8 = 4;
const LK_PTTN: u8 = 5;
const LK_PGN: u8 = 6;
const LK_CN: u8 = 7;
// JumpSS sub-domain selector — `byte5` bits 7-6.
const SS_FP: u8 = 0;
const SS_VMGM_MENU: u8 = 1;
const SS_VTSM: u8 = 2;
// Operand field widths (spec-defined bit counts).
const MASK_TTN: u8 = 0x7F; // 7-bit title number
const MASK_PGN: u8 = 0x7F; // 7-bit program number
const MASK_LINKOP: u8 = 0x1F; // 5-bit link sub-op
const MASK_REG: u8 = 0x0F; // 4-bit GPRM index
const MASK_MENU: u8 = 0x0F; // 4-bit menu id
const MASK_PTTN: u16 = 0x03FF; // 10-bit part-of-title
const MASK_PGCN: u16 = 0x7FFF; // 15-bit PGC number
#[inline]
fn be16(b: &[u8; 8], o: usize) -> u16 {
((b[o] as u16) << 8) | b[o + 1] as u16
}
// Compare-operand layouts ("if_version"s) per the DVD-Video VM. The op
// nibble is always `byte1` bits 6-4; the immediate flag is `byte1` bit 7. The
// operand *offsets* differ by command family.
//
// v1 (special + link): lhs reg = b[3]; rhs imm = bytes4-5 / rhs reg = b[4].
// v2 (jump + system-set): lhs reg = b[6]; rhs reg = b[7] (registers only).
// v3 (set-GPRM): lhs reg = b[2]; rhs imm = bytes6-7 / rhs reg = b[6].
fn if_v1(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[3],
immediate: b[1] >> 7 != 0,
imm: be16(b, 4),
rhs_reg: b[4],
})
}
fn if_v2(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[6],
immediate: false,
imm: 0,
rhs_reg: b[7],
})
}
fn if_v3(b: &[u8; 8]) -> Option<Compare> {
let op = (b[1] >> 4) & 7;
(op != 0).then(|| Compare {
op,
lhs_reg: b[2],
immediate: b[1] >> 7 != 0,
imm: be16(b, 6),
rhs_reg: b[6],
})
}
/// Decode an 8-byte VM command.
pub fn decode(b: &[u8; 8]) -> Command {
let typ = b[0] >> 5;
let direct = (b[0] >> 4) & 1;
let setop = b[0] & 0x0F;
let cmd = b[1] & 0x0F;
// Compare predicate, with the operand layout for this command family
// (the DVD-Video VM command type dispatch).
let compare = match (typ, direct) {
(TYPE_SPECIAL, _) => if_v1(b),
(TYPE_LINK_JUMP, 1) => if_v2(b), // jump
(TYPE_LINK_JUMP, 0) => if_v1(b), // link
(TYPE_SET_SYSTEM, _) => if_v2(b),
(TYPE_SET_GPRM, _) => if_v3(b),
_ => None, // 4/5/6 compound — not needed by the resolver
};
// JumpSS sub-domain selector lives in byte5 bits 7-6.
let ss_sel = b[5] >> 6;
let instr = match typ {
TYPE_LINK_JUMP if direct == 1 => match cmd {
JP_EXIT => Instr::Exit,
JP_JUMP_TT => Instr::JumpTt {
ttn: b[5] & MASK_TTN,
},
JP_JUMP_VTS_TT => Instr::JumpVtsTt {
ttn: b[5] & MASK_TTN,
},
JP_JUMP_VTS_PTT => Instr::JumpVtsPtt {
ttn: b[5] & MASK_TTN,
pttn: be16(b, 2) & MASK_PTTN,
},
JP_JUMP_SS => match ss_sel {
SS_FP => Instr::JumpSsFp,
SS_VMGM_MENU => Instr::JumpSsVmgm {
menu: b[5] & MASK_MENU,
},
SS_VTSM => Instr::JumpSsVtsm {
vts: b[4],
ttn: b[3],
menu: b[5] & MASK_MENU,
},
_ => Instr::JumpSsVmgmPgc {
pgcn: be16(b, 2) & MASK_PGCN,
},
},
JP_CALL_SS => Instr::CallSs { sub: ss_sel },
_ => Instr::Nop,
},
TYPE_LINK_JUMP => match cmd {
// direct == 0 (link). sub-op 0 = NOP/no-link.
LK_SUB => Instr::LinkSub {
sub: b[7] & MASK_LINKOP,
},
LK_PGCN => Instr::LinkPgcn {
pgcn: be16(b, 6) & MASK_PGCN,
},
LK_PTTN => Instr::LinkPttn {
pttn: be16(b, 6) & MASK_PTTN,
},
LK_PGN => Instr::LinkPgn {
pgn: b[7] & MASK_PGN,
},
LK_CN => Instr::LinkCn { cn: b[7] },
_ => Instr::Nop,
},
TYPE_SPECIAL => match cmd {
SP_GOTO => Instr::Goto { line: b[7] },
SP_BREAK => Instr::Break,
_ => Instr::Nop,
},
TYPE_SET_GPRM => Instr::SetGprm {
reg: b[3] & MASK_REG,
op: setop,
immediate: direct != 0,
imm: be16(b, 4),
src: b[5],
},
TYPE_SET_SYSTEM => Instr::SetSystem,
_ => Instr::Other(*b),
};
Command { compare, instr }
}
#[cfg(test)]
mod tests {
use super::*;
fn h(s: &str) -> [u8; 8] {
let v: Vec<u8> = (0..8)
.map(|i| u8::from_str_radix(&s[i * 2..i * 2 + 2], 16).unwrap())
.collect();
v.try_into().unwrap()
}
// KATs taken from the real SOTL / Greenland discs (decoded in the PoC).
#[test]
fn greenland_first_play_is_jumptt_1() {
let c = decode(&h("3002000000010000"));
assert_eq!(c.instr, Instr::JumpTt { ttn: 1 });
assert!(c.compare.is_none());
}
#[test]
fn sotl_first_play_is_jumpss_vtsm_root() {
// 30 06 ... byte5=0x83 -> sub 2 (VTSM), vts=byte4=1, menu=byte5&0xF=3 (root)
let c = decode(&h("3006000101830000"));
assert_eq!(
c.instr,
Instr::JumpSsVtsm {
vts: 1,
ttn: 1,
menu: 3
}
);
}
#[test]
fn sotl_title_dispatch_is_conditional_linkpgn_2() {
// 20 a6 ... CmpLink: if GPRM0 == 2 -> LinkPGN 2 (cell 2 = the 5:02 start)
let c = decode(&h("20a6000000020002"));
assert_eq!(c.instr, Instr::LinkPgn { pgn: 2 });
let cmp = c.compare.expect("conditional");
assert_eq!(cmp.op, 2); // ==
assert_eq!(cmp.lhs_reg, 0); // GPRM0
assert!(cmp.immediate);
assert_eq!(cmp.imm, 2);
}
#[test]
fn sotl_root_button_is_linkpgcn_37() {
assert_eq!(
decode(&h("2004000000000025")).instr,
Instr::LinkPgcn { pgcn: 37 }
);
}
#[test]
fn greenland_scene_button_is_linkpgn() {
assert_eq!(
decode(&h("2006000000001401")).instr,
Instr::LinkPgn { pgn: 1 }
);
}
#[test]
fn jumpvts_ptt_decodes_ttn_and_pttn() {
// synthetic: 30 05 | ptt(bytes2-3)=0x0002 | ttn(byte5)=1
let c = decode(&h("3005000200010000"));
assert_eq!(c.instr, Instr::JumpVtsPtt { ttn: 1, pttn: 2 });
}
#[test]
fn setgprm_immediate_mov() {
// SOTL First-Play pre[0]: 71 00 | reg=byte3=6 | imm(bytes4-5)=0x03e8 -> g6 = 1000
match decode(&h("7100000603e80000")).instr {
Instr::SetGprm {
reg,
op,
immediate,
imm,
..
} => {
assert_eq!(reg, 6);
assert_eq!(op, 1); // mov
assert!(immediate);
assert_eq!(imm, 1000);
}
other => panic!("expected SetGprm, got {other:?}"),
}
}
// Regression for the link sub-op decode: 0 = NOP, 1 = LinkSub.
#[test]
fn link_subop_zero_is_nop_one_is_linksub() {
assert_eq!(decode(&h("2000000000000000")).instr, Instr::Nop);
assert_eq!(
decode(&h("2001000000000010")).instr,
Instr::LinkSub { sub: 0x10 }
);
}
// if_version_1 register compare: rhs register is byte4 (not byte5).
#[test]
fn link_register_compare_rhs_is_byte4() {
// 20 26: link, cmp=EQ(2), dircmp=0(register) ; cmd=6 LinkPGN
let c = decode(&h("2026000304000002"));
assert_eq!(c.instr, Instr::LinkPgn { pgn: 2 });
let cmp = c.compare.expect("conditional");
assert!(!cmp.immediate);
assert_eq!(cmp.lhs_reg, 3);
assert_eq!(cmp.rhs_reg, 4);
}
// if_version_2 jump compare: both operands are registers in byte6 / byte7.
#[test]
fn jump_compare_uses_bytes6_and_7() {
// 30 22: jump, cmp=EQ(2) ; cmd=2 JumpTT ttn=byte5=5
let c = decode(&h("3022000000050607"));
assert_eq!(c.instr, Instr::JumpTt { ttn: 5 });
let cmp = c.compare.expect("conditional");
assert!(!cmp.immediate);
assert_eq!(cmp.lhs_reg, 6);
assert_eq!(cmp.rhs_reg, 7);
}
}
+986 -9
View File
File diff suppressed because it is too large Load Diff
+33 -1
View File
@@ -8,11 +8,17 @@
//! disc.rip(&mut session, 0, output, |event| {
//! match event.kind {
//! EventKind::BytesRead { bytes, total } => update_progress(bytes, total),
//! EventKind::ReadError { sector, .. } => log_error(sector),
//! EventKind::SectorSkipped { sector } => log_skip(sector),
//! EventKind::BatchSizeChanged { new_size, .. } => note_recovery(new_size),
//! _ => {}
//! }
//! });
//! ```
//!
//! Note: the library currently emits only `BytesRead`, `SectorSkipped`,
//! and `BatchSizeChanged`. The other [`EventKind`] variants are part of
//! the stable event vocabulary for consumers (and future emit sites) but
//! are not produced by the library today.
use crate::error::Error;
@@ -116,3 +122,29 @@ pub enum BatchSizeReason {
/// A no-op event handler. Ignores all events.
pub fn ignore(_event: Event) {}
#[cfg(test)]
mod tests {
use super::*;
/// BatchSizeReason::Shrunk != BatchSizeReason::Probed.
/// These two variants carry distinct meanings (error vs. recovery); they
/// must not compare as equal.
/// Mutation: deriving PartialEq without proper variant discrimination
/// could make two distinct variants equal.
#[test]
fn batch_size_reason_variants_are_not_equal() {
assert_ne!(BatchSizeReason::Shrunk, BatchSizeReason::Probed);
}
/// BatchSizeReason is Clone + Copy: cloning does not move the original.
/// This is required because EventKind::BatchSizeChanged embeds it by value.
/// Mutation: removing Copy would require the caller to clone explicitly;
/// code that passes reason by value would fail to compile.
#[test]
fn batch_size_reason_is_copy() {
let r = BatchSizeReason::Shrunk;
let _r2 = r; // copy, not move
let _r3 = r; // r still usable after copy
}
}
+34 -9
View File
@@ -34,11 +34,9 @@ impl Halt {
Self(Arc::new(AtomicBool::new(false)))
}
/// Wrap an existing `Arc<AtomicBool>` as a `Halt`. Useful as a
/// bridge during the 0.18 deprecation window: callers that already
/// hold an `Arc<AtomicBool>` (e.g. `Drive::halt_flag()`, the
/// deprecated `DiscStream::set_halt`) can adopt the new token API
/// without changing the underlying flag.
/// Wrap an existing `Arc<AtomicBool>` as a `Halt`. A bridge for
/// callers that already hold an `Arc<AtomicBool>` cancellation flag
/// and want to adopt the token API without allocating a new flag.
///
/// Cancelling either side flips the same bit — the wrapping `Halt`
/// and the original `Arc` are two views over one shared flag.
@@ -46,10 +44,9 @@ impl Halt {
Self(flag)
}
/// Borrow the underlying `Arc<AtomicBool>`. Used at boundaries with
/// pre-`Halt` APIs that still take an `Arc<AtomicBool>` directly
/// (`CopyOptions::halt`, the deprecated `DiscStream::set_halt`).
/// Round 3 deletes those boundaries and this accessor with them.
/// Borrow the underlying `Arc<AtomicBool>`. The inverse of
/// [`from_arc`](Self::from_arc): hand the shared flag to an API that
/// still takes a raw `Arc<AtomicBool>` rather than a `Halt`.
pub fn as_arc(&self) -> &Arc<AtomicBool> {
&self.0
}
@@ -71,6 +68,20 @@ impl Default for Halt {
}
}
/// Shared poll interval for halt-aware loops.
///
/// `bounded_syscall` checks the cancellation flag and the deadline
/// every [`POLL_INTERVAL`] while blocked on a worker; the same
/// cadence governs `Pipeline::send_with_halt`'s `try_send` retry.
/// 250 ms is the sweet spot between responsiveness (operator presses
/// Stop, sees it take effect within ~quarter-second) and waste
/// (atomic load + clock read is cheap but not free at thousands of
/// hertz).
///
/// Centralised here so the half-dozen halt-polling loops across `io`
/// can't drift apart silently.
pub const POLL_INTERVAL: std::time::Duration = std::time::Duration::from_millis(250);
#[cfg(test)]
mod tests {
use super::*;
@@ -163,4 +174,18 @@ mod tests {
arc.store(true, Ordering::Relaxed);
assert!(halt.is_cancelled());
}
// ── New comprehensive tests ────────────────────────────────────────────────
/// POLL_INTERVAL is 250ms — a specific value that the multi-thread halt
/// loops depend on for responsiveness guarantees.
/// Mutation: setting POLL_INTERVAL to 5s makes stop requests take 5s to notice.
#[test]
fn poll_interval_is_250ms() {
assert_eq!(
POLL_INTERVAL,
std::time::Duration::from_millis(250),
"POLL_INTERVAL must be 250ms for the guaranteed ~quarter-second cancel latency"
);
}
}
+106
View File
@@ -0,0 +1,106 @@
//! The single hex → bytes parser for the whole workspace.
//!
//! Key material arrives as hex from three third-party sources — the keydb, an
//! online key service, and the mapfile's `# freemkv-vid:` comment — and each
//! used to parse it slightly differently (one stripped `0x`/`0X`, one stripped
//! nothing, one stripped `0x` only). A key written with a prefix one parser
//! didn't expect was silently dropped → "can't decrypt" with no error. This is
//! the one parser they all call, so the prefix/case/validation rules live in
//! exactly one place.
//!
//! Operates on BYTES, not `&str` char indices: the inputs are untrusted, so a
//! multi-byte UTF-8 scalar must reject as malformed, never panic on a
//! mid-codepoint slice.
/// Parse a hex string into bytes. Accepts an optional `0x`/`0X` prefix
/// (case-insensitive), then requires an even run of ASCII hex digits. Any
/// non-hex byte, or an odd length, yields `None`.
pub fn parse_hex_bytes(s: &str) -> Option<Vec<u8>> {
let body = strip_prefix(s.trim());
let bytes = body.as_bytes();
// Empty → empty Vec (a legitimately-empty variable-length field); odd length
// is malformed. (`parse_hex_fixed` enforces a concrete length separately.)
if bytes.len() % 2 != 0 {
return None;
}
let mut out = Vec::with_capacity(bytes.len() / 2);
for pair in bytes.chunks_exact(2) {
out.push(byte(pair[0], pair[1])?);
}
Some(out)
}
/// Parse a hex string into a fixed `[u8; N]`. Accepts an optional `0x`/`0X`
/// prefix; requires EXACTLY `2*N` ASCII hex digits after it. `None` on any
/// non-hex byte or a length mismatch.
pub fn parse_hex_fixed<const N: usize>(s: &str) -> Option<[u8; N]> {
let body = strip_prefix(s.trim());
let bytes = body.as_bytes();
if bytes.len() != 2 * N {
return None;
}
let mut out = [0u8; N];
for (i, slot) in out.iter_mut().enumerate() {
*slot = byte(bytes[2 * i], bytes[2 * i + 1])?;
}
Some(out)
}
/// Strip a single leading `0x` / `0X` if present (case-insensitive).
fn strip_prefix(s: &str) -> &str {
s.strip_prefix("0x")
.or_else(|| s.strip_prefix("0X"))
.unwrap_or(s)
}
/// Combine two ASCII hex-digit bytes into one byte. `as char` is intentional:
/// for a non-ASCII byte it produces a Latin-1 scalar that `to_digit(16)` then
/// rejects — so non-hex (incl. `+`/`-` sign chars) and multi-byte input fail
/// cleanly rather than slipping through `from_str_radix`'s sign handling.
fn byte(hi: u8, lo: u8) -> Option<u8> {
let hi = (hi as char).to_digit(16)?;
let lo = (lo as char).to_digit(16)?;
Some((hi * 16 + lo) as u8)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn fixed_accepts_0x_0x_and_bare_same_result() {
let want = [0x00, 0x11, 0xab, 0xCD, 0xef, 0x42, 0x99, 0x00];
let bare = "0011abcdef429900";
assert_eq!(parse_hex_fixed::<8>(bare), Some(want));
assert_eq!(parse_hex_fixed::<8>(&format!("0x{bare}")), Some(want));
// The case that used to be dropped by one parser but not another.
assert_eq!(parse_hex_fixed::<8>(&format!("0X{bare}")), Some(want));
assert_eq!(parse_hex_fixed::<8>(&format!(" 0X{bare} ")), Some(want));
}
#[test]
fn fixed_rejects_wrong_length_and_non_hex_and_signs() {
assert_eq!(parse_hex_fixed::<16>("00"), None); // too short
assert_eq!(parse_hex_fixed::<2>("00112233"), None); // too long
assert_eq!(parse_hex_fixed::<2>("zz11"), None); // non-hex
assert_eq!(parse_hex_fixed::<2>("+5-A"), None); // sign chars
}
#[test]
fn does_not_panic_on_multibyte_of_exact_byte_length() {
// "中" is 3 bytes; + 29 'a' = 32 bytes → would mis-slice a &str-indexed
// parser. Must reject, not panic.
let s = "中".to_string() + &"a".repeat(29);
assert_eq!(s.len(), 32);
assert_eq!(parse_hex_fixed::<16>(&s), None);
}
#[test]
fn bytes_variable_length_and_odd_rejected() {
assert_eq!(parse_hex_bytes("0xAABBCC"), Some(vec![0xAA, 0xBB, 0xCC]));
assert_eq!(parse_hex_bytes("AABBC"), None); // odd
// Empty (or prefix-only) → empty Vec: a legitimately-empty field.
assert_eq!(parse_hex_bytes(""), Some(vec![]));
assert_eq!(parse_hex_bytes("0x"), Some(vec![]));
}
}
+209 -14
View File
@@ -57,27 +57,48 @@ impl DriveId {
let cdb_inq = [0x12, 0x00, 0x00, 0x00, 0x60, 0x00];
transport.execute(&cdb_inq, DataDirection::FromDevice, &mut inquiry, 5000)?;
// GET CONFIGURATION Feature 010Ch — MMC-6 §6.6
// GET CONFIGURATION Feature 010Ch — MMC-6 §6.6.
// Best-effort: 010Ch (Firmware Information) is an optional feature.
// A drive that lacks it may CHECK CONDITION rather than return an
// empty descriptor, so a failure here is treated as feature-absent
// (empty firmware date + empty raw bytes) instead of aborting the
// whole identity probe.
let mut gc = vec![0u8; 256];
let cdb_gc = [0x46, 0x02, 0x01, 0x0C, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00];
let result = transport.execute(&cdb_gc, DataDirection::FromDevice, &mut gc, 5000)?;
// `bytes_transferred` is device-reported and untrusted; clamp every
// slice end to the actual buffer length before indexing.
let (firmware_date, raw_gc_010c) =
match transport.execute(&cdb_gc, DataDirection::FromDevice, &mut gc, 5000) {
Ok(result) => {
let end = result.bytes_transferred.min(gc.len());
let date = if end > 12 {
String::from_utf8_lossy(&gc[12..24.min(end)])
.trim()
.to_string()
} else {
String::new()
};
(date, gc[..end].to_vec())
}
Err(_) => (String::new(), Vec::new()),
};
let firmware_date = if result.bytes_transferred > 12 {
String::from_utf8_lossy(&gc[12..24.min(result.bytes_transferred)])
.trim()
.to_string()
} else {
String::new()
};
// GET CONFIGURATION Feature 0108h — Serial Number
// GET CONFIGURATION Feature 0108h — Serial Number.
// Best-effort, like 010Ch above: the serial-number feature is
// optional, so a drive that lacks it (CHECK CONDITION) or reports
// too few bytes deliberately yields an empty serial rather than
// failing the identity probe.
let mut gc_serial = vec![0u8; 256];
let cdb_serial = [0x46, 0x02, 0x01, 0x08, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00];
let serial_number = if let Ok(r) =
transport.execute(&cdb_serial, DataDirection::FromDevice, &mut gc_serial, 5000)
{
if r.bytes_transferred > 12 {
String::from_utf8_lossy(&gc_serial[12..r.bytes_transferred])
// `bytes_transferred` is device-reported and untrusted; clamp
// the slice end to the buffer length to avoid an out-of-range
// panic on an oversized reported count.
let end = r.bytes_transferred.min(gc_serial.len());
String::from_utf8_lossy(&gc_serial[12..end])
.trim()
.to_string()
} else {
@@ -94,8 +115,8 @@ impl DriveId {
vendor_specific: ascii_field(&inquiry, 36, 43),
firmware_date,
serial_number,
raw_inquiry: inquiry.to_vec(),
raw_gc_010c: gc[..result.bytes_transferred].to_vec(),
raw_inquiry: inquiry,
raw_gc_010c,
})
}
@@ -155,6 +176,49 @@ fn ascii_field(data: &[u8], start: usize, end: usize) -> String {
#[cfg(test)]
mod tests {
use super::*;
use crate::scsi::{ScsiResult, ScsiTransport};
/// Transport that returns the requested data length but reports a
/// bytes_transferred larger than the caller's buffer — models a drive
/// that lies about its transfer count. The old slicing code panicked
/// on this; the clamps must keep it from indexing out of range.
struct OversizedCountTransport;
impl ScsiTransport for OversizedCountTransport {
fn execute(
&mut self,
cdb: &[u8],
_dir: DataDirection,
buf: &mut [u8],
_timeout_ms: u32,
) -> Result<ScsiResult> {
// Fill plausible ASCII so the from_utf8_lossy paths run.
for b in buf.iter_mut() {
*b = b'A';
}
// INQUIRY (0x12): honest count. GET CONFIGURATION (0x46): lie.
let bytes_transferred = if cdb.first() == Some(&0x12) {
buf.len()
} else {
buf.len() + 4096
};
Ok(ScsiResult {
status: 0,
bytes_transferred,
sense: [0u8; 32],
})
}
}
#[test]
fn from_drive_clamps_oversized_bytes_transferred() {
// Must not panic despite the transport reporting a transfer count
// far beyond the 256-byte GET CONFIGURATION buffers.
let mut t = OversizedCountTransport;
let id = DriveId::from_drive(&mut t).expect("from_drive must not error");
// raw_gc_010c is clamped to the 256-byte buffer, never the lie.
assert_eq!(id.raw_gc_010c.len(), 256);
}
#[test]
fn test_bu40n_identity() {
@@ -190,4 +254,135 @@ mod tests {
assert_eq!(id.vendor_specific.trim(), "16/04/");
assert_eq!(id.firmware_date, "201604250000");
}
// ── New comprehensive tests ────────────────────────────────────────────────
/// ascii_field with a buffer shorter than `start` returns empty string
/// rather than panicking.
/// Spec: SPC-4 §6.4.2 — bytes[8:16] are vendor ID; a truncated buffer
/// (e.g. a device that reports fewer than 8 bytes) must not panic.
/// Mutation: removing the `data.len() > start` guard makes it panic on short inputs.
#[test]
fn ascii_field_short_buffer_returns_empty() {
// Buffer of length 5: start=8 is beyond the end → empty string.
let buf = vec![0u8; 5];
let result = ascii_field(&buf, 8, 16); // SPC-4 vendor ID range
assert!(result.is_empty(), "short buffer must yield empty string");
}
/// ascii_field with a buffer that covers start but not end is clamped.
/// Spec: `ascii_field` documents "clamps to data.len()".
/// Mutation: using `end` directly without `min(data.len())` panics here.
#[test]
fn ascii_field_partial_buffer_is_clamped_not_panicked() {
// Buffer of length 12: vendor_id range is [8..16], but only [8..12] present.
let mut buf = vec![0u8; 12];
buf[8..12].copy_from_slice(b"SONY");
let result = ascii_field(&buf, 8, 16);
// Must not panic; the returned string holds what we wrote.
assert_eq!(result, "SONY");
}
/// from_inquiry extracts the product_id field from INQUIRY bytes [16:32].
/// Spec: SPC-4 §6.4.2 — PRODUCT IDENTIFICATION at offset 16, length 16.
/// Mutation: shifting the product_id slice to [8:24] makes this fail.
#[test]
fn from_inquiry_extracts_product_id_at_offset_16() {
let mut inquiry = vec![0u8; 96];
// Leave vendor_id (8..16) as zeros, write product_id at 16..32.
inquiry[16..32].copy_from_slice(b"BD-RW BDR-209M");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.product_id, "BD-RW BDR-209M",
"product_id must come from INQUIRY bytes 16..32 (SPC-4 §6.4.2)"
);
}
/// from_inquiry extracts product_revision from INQUIRY bytes [32:36].
/// Spec: SPC-4 §6.4.2 — PRODUCT REVISION LEVEL at offset 32, length 4.
/// Mutation: reading revision from [36:40] produces the wrong value.
#[test]
fn from_inquiry_extracts_revision_at_offset_32() {
let mut inquiry = vec![0u8; 96];
inquiry[32..36].copy_from_slice(b"1.53");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.product_revision, "1.53",
"product_revision must come from INQUIRY bytes 32..36 (SPC-4 §6.4.2)"
);
}
/// from_inquiry extracts vendor_specific from INQUIRY bytes [36:43].
/// Spec: SPC-4 §6.4.2 — VENDOR SPECIFIC at offset 36, length 8.
/// Mutation: reading vendor_specific from [32:39] returns the revision instead.
#[test]
fn from_inquiry_extracts_vendor_specific_at_offset_36() {
let mut inquiry = vec![0u8; 96];
inquiry[36..43].copy_from_slice(b"MM01234");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.vendor_specific, "MM01234",
"vendor_specific must come from INQUIRY bytes 36..43 (SPC-4 §6.4.2)"
);
}
/// from_inquiry stores the raw inquiry bytes in raw_inquiry unchanged.
/// Mutation: copying only a slice of inquiry into raw_inquiry truncates it.
#[test]
fn from_inquiry_stores_raw_inquiry() {
let mut inquiry = vec![0u8; 96];
inquiry[8..16].copy_from_slice(b"TESTDRVR");
let id = DriveId::from_inquiry(&inquiry, "");
assert_eq!(
id.raw_inquiry, inquiry,
"raw_inquiry must preserve the full 96-byte buffer"
);
}
/// GET CONFIGURATION failure (transport error) must not abort the
/// identity probe — firmware_date is empty, raw_gc_010c is empty.
/// Mutation: propagating the GET_CONFIGURATION error with `?` aborts from_drive.
#[test]
fn from_drive_gc_failure_yields_empty_firmware_date() {
struct GcFailTransport;
impl ScsiTransport for GcFailTransport {
fn execute(
&mut self,
cdb: &[u8],
_dir: DataDirection,
buf: &mut [u8],
_timeout_ms: u32,
) -> Result<ScsiResult> {
if cdb.first() == Some(&0x12) {
// INQUIRY succeeds with a plausible response.
buf[8..16].copy_from_slice(b"TESTDRV ");
buf[16..32].copy_from_slice(b"FAKE DRIVE MODEL");
buf[32..36].copy_from_slice(b"0001");
buf[36..43].copy_from_slice(b"X000001");
Ok(ScsiResult {
status: 0,
bytes_transferred: buf.len(),
sense: [0u8; 32],
})
} else {
// GET CONFIGURATION fails.
Err(crate::error::Error::ScsiError {
opcode: cdb[0],
status: crate::scsi::SCSI_STATUS_CHECK_CONDITION,
sense: None,
})
}
}
}
let mut t = GcFailTransport;
let id = DriveId::from_drive(&mut t).expect("from_drive must succeed despite GC failure");
assert!(
id.firmware_date.is_empty(),
"firmware_date must be empty when GC fails"
);
assert!(
id.raw_gc_010c.is_empty(),
"raw_gc_010c must be empty when GC fails"
);
}
}
+991 -55
View File
File diff suppressed because it is too large Load Diff
+295
View File
@@ -0,0 +1,295 @@
//! Bounded-syscall primitive: run a (potentially-blocking) operation
//! on a worker thread, with a hard wall-clock deadline and an optional
//! cooperative [`Halt`] poll. The calling thread is never trapped
//! inside a kernel call.
//!
//! ## Why this exists
//!
//! [`crate::halt::Halt`] is cooperative: callers poll
//! `is_cancelled()`. It cannot reach inside a syscall the kernel
//! currently owns the thread for — `libc::sync_file_range`,
//! `libc::fsync`, `File::write` on NFS, and so on. `/api/stop` from
//! autorip therefore can't unstick a thread sitting in such a syscall.
//!
//! [`bounded_syscall`] is the escape hatch: it runs `op` on a fresh
//! worker thread, then `recv_timeout`s on a rendezvous channel for the
//! result. The wait is broken into ~250 ms slices so the calling
//! thread can poll the supplied [`Halt`] in between. If the deadline
//! elapses or the halt fires, the worker is intentionally leaked — the
//! syscall will unwind whenever the kernel decides, or at process
//! exit, but the caller is free to fall back to a degraded code path
//! (skip the sync, log loudly, etc.).
//!
//! ## Trade-offs
//!
//! - **Thread per call.** Cheap (`std::thread::spawn` is < 100 µs on
//! Linux/macOS), but not free. Use on coarse-grained finalisation
//! syscalls (`sync_all`, `sync_file_range(WAIT_AFTER)`), not on hot
//! inner-loop writes.
//! - **Leak on timeout.** A wedged syscall keeps a kernel slot and a
//! user-space thread around for the rest of the process's life.
//! Bounded by the number of independent rip/mux sessions, which is
//! one per disc. The alternative — trapping the caller forever —
//! defeats the entire purpose of `/api/stop`.
//! - **Halt granularity ~250 ms.** Halt observation is not instant;
//! it's the worst-case latency of the `recv_timeout` slice. Good
//! enough for human-driven stop requests; not suitable for hard
//! real-time deadlines.
//!
//! ## Single source of truth
//!
//! Do NOT inline this pattern. Every blocking-syscall wrapper in the
//! rip + mux pipeline calls this helper, so changes (e.g. swapping the
//! channel impl, adjusting the poll slice, adding metrics) land in one
//! place.
//!
//! ## Platform
//!
//! Pure `std::thread` + `std::sync::mpsc`. No `cfg(target_os)` needed
//! here — the helper itself is platform-agnostic. Callers that wrap
//! Linux-only syscalls (`sync_file_range`) still need their own
//! `#[cfg(target_os = "linux")]` gates; this helper does not.
use std::sync::mpsc::{RecvTimeoutError, sync_channel};
use std::thread;
use std::time::{Duration, Instant};
use crate::halt::{Halt, POLL_INTERVAL};
/// Failure outcome from a bounded syscall wrapper.
#[derive(Debug)]
pub(crate) enum BoundedError {
/// The user-visible halt token fired during the wait. The worker
/// thread is intentionally leaked — the caller should fall back to
/// a degraded code path rather than waiting on the syscall to
/// return.
Halted,
/// The deadline elapsed before the syscall returned. Same leak
/// semantics as `Halted`.
Timeout,
/// The worker thread panicked, the OS rejected the thread spawn,
/// or its sender disconnected before sending a result. Treat as a
/// benign no-op (callers usually log and continue) rather than a
/// hard error — by definition no syscall observably ran to
/// completion in this case. In the spawn-failure case no thread is
/// leaked.
WorkerLost,
}
/// Run a (potentially-blocking) operation on a worker thread with a
/// deadline and an optional cooperative halt-token poll. Returns the
/// operation's result if it completes within `timeout`; otherwise one
/// of [`BoundedError::Halted`] / [`BoundedError::Timeout`] /
/// [`BoundedError::WorkerLost`].
///
/// On `Halted` / `Timeout` the worker thread is intentionally leaked:
/// the syscall will unwind whenever the kernel decides, or when the
/// process exits. The calling thread is never trapped inside a kernel
/// call.
///
/// `halt` is polled at [`POLL_INTERVAL`] granularity. Pass `None` for
/// callers that don't (yet) have a halt token plumbed through —
/// behaviour degrades to deadline-only, matching the 0.20.5
/// `wait_after_with_timeout` shape this helper generalises.
///
/// `op` returns `R: Send + 'static`. The closure must own everything
/// it touches because it may outlive this call (timeout / halt cases).
pub(crate) fn bounded_syscall<F, R>(
halt: Option<&Halt>,
timeout: Duration,
op: F,
) -> Result<R, BoundedError>
where
F: FnOnce() -> R + Send + 'static,
R: Send + 'static,
{
// If the caller already requested halt, don't spawn (and leak) a
// worker that would run `op` to completion in the background.
if halt.is_some_and(|h| h.is_cancelled()) {
return Err(BoundedError::Halted);
}
// Rendezvous channel: the worker sends exactly one value (the
// op's return) and then exits. Capacity-0 means the send blocks
// until we receive — fine on the happy path; on the timeout /
// halt path the receiver is dropped and the worker's send
// returns Err, which the worker ignores.
let (tx, rx) = sync_channel::<R>(0);
let _ = thread::Builder::new()
.name("freemkv-bounded-syscall".into())
.spawn(move || {
// Ignore the send error: if we time out (or get halted)
// before the worker finishes, the receiver is dropped
// and `tx.send` returns Err. Either way, the worker has
// nothing more to do.
let _ = tx.send(op());
});
let deadline = Instant::now() + timeout;
loop {
let now = Instant::now();
let remaining = deadline.saturating_duration_since(now);
let slice = remaining.min(POLL_INTERVAL);
match rx.recv_timeout(slice) {
Ok(v) => return Ok(v),
Err(RecvTimeoutError::Timeout) => {
if let Some(h) = halt {
if h.is_cancelled() {
return Err(BoundedError::Halted);
}
}
if Instant::now() >= deadline {
return Err(BoundedError::Timeout);
}
// Otherwise: another slice.
}
Err(RecvTimeoutError::Disconnected) => {
// Worker thread spawn failed, or it panicked before
// sending. Caller treats this as "no syscall ran" —
// typically a no-op + log.
return Err(BoundedError::WorkerLost);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
#[test]
fn op_completes_quickly() {
let r = bounded_syscall(None, Duration::from_secs(2), || 42u32);
assert!(matches!(r, Ok(42)));
}
#[test]
fn op_exceeds_timeout() {
// Op sleeps longer than the deadline → Timeout.
let r = bounded_syscall(None, Duration::from_millis(300), || {
thread::sleep(Duration::from_secs(2));
0u32
});
assert!(matches!(r, Err(BoundedError::Timeout)));
}
#[test]
fn halt_fires_during_wait() {
let halt = Halt::new();
let halt2 = halt.clone();
// Flip the halt from a side thread after ~300 ms — long
// enough that the receive loop has rolled at least one
// 250 ms slice and is sitting in `recv_timeout` again when
// the bit flips.
thread::spawn(move || {
thread::sleep(Duration::from_millis(300));
halt2.cancel();
});
let r = bounded_syscall(Some(&halt), Duration::from_secs(5), || {
thread::sleep(Duration::from_secs(5));
0u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
}
#[test]
fn worker_panics() {
// Worker panics → sender drops without sending → recv sees
// Disconnected → WorkerLost. We use an explicit panic in the
// op closure rather than `panic!()` from inside the channel
// machinery; the spawned thread's panic is contained (no
// process abort) because we don't `.join()` it.
let r = bounded_syscall(None, Duration::from_secs(2), || -> u32 {
panic!("intentional test panic");
});
assert!(matches!(r, Err(BoundedError::WorkerLost)));
}
#[test]
fn halt_already_set_before_call_still_returns_halted() {
// Halt observed on the very first poll slice. The op blocks
// forever; we must not wait the full timeout to notice the
// halt is already set.
let halt = Halt::new();
halt.cancel();
let started = Instant::now();
let r = bounded_syscall(Some(&halt), Duration::from_secs(10), || {
thread::sleep(Duration::from_secs(10));
0u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
// Should bail out within ~1 s; allow 2 s of slack for slow
// CI hosts.
assert!(
started.elapsed() < Duration::from_secs(2),
"halt-already-set took {:?}",
started.elapsed()
);
}
#[test]
fn ok_path_takes_no_halt_token() {
// Sanity: the `None` halt path is the documented zero-config
// form (matches the 0.20.5 `wait_after_with_timeout`
// behaviour). Op returns immediately; we must observe Ok.
let flag = Arc::new(AtomicBool::new(false));
let f2 = flag.clone();
let r = bounded_syscall(None, Duration::from_secs(2), move || {
f2.store(true, Ordering::Relaxed);
"ok"
});
assert!(matches!(r, Ok("ok")));
assert!(flag.load(Ordering::Relaxed));
}
// ── Added hardening tests ───────────────────────────────────────
/// Doc contract (lines 106-110): "If the caller already requested
/// halt, don't spawn (and leak) a worker that would run `op`."
/// When halt is pre-cancelled the op closure must NEVER run — the
/// short-circuit returns Halted before spawning the worker. We
/// prove the op did not execute by checking a side-effect flag.
#[test]
fn pre_cancelled_halt_never_runs_op() {
let halt = Halt::new();
halt.cancel();
let ran = Arc::new(AtomicBool::new(false));
let r2 = ran.clone();
let r = bounded_syscall(Some(&halt), Duration::from_secs(2), move || {
r2.store(true, Ordering::SeqCst);
7u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
// The op closure must not have been scheduled at all.
assert!(
!ran.load(Ordering::SeqCst),
"op ran despite pre-cancelled halt — short-circuit at line 108 broken"
);
}
/// Timeout boundary: with a tiny deadline and an op that sleeps
/// much longer, the helper must return Timeout and must do so
/// roughly at the deadline — NOT wait for the op to finish (that
/// is the whole point of the bounded wrapper; the worker is
/// leaked). Grounds the `Instant::now() >= deadline` arm (line 141)
/// and the leak contract (doc lines 84-88).
#[test]
fn timeout_returns_near_deadline_not_after_op() {
let started = Instant::now();
let r = bounded_syscall(None, Duration::from_millis(100), || {
thread::sleep(Duration::from_secs(3));
0u32
});
let elapsed = started.elapsed();
assert!(matches!(r, Err(BoundedError::Timeout)));
// Must bail near the 100ms deadline (one POLL_INTERVAL slack at
// most), not after the 3s op. Allow generous CI slack but stay
// well under the op's 3s sleep.
assert!(
elapsed < Duration::from_millis(1500),
"timeout did not return near deadline: {elapsed:?} (op should be leaked, not awaited)"
);
}
}
+559
View File
@@ -0,0 +1,559 @@
//! `BytePrefetcher` — `std::io::Read` analogue of
//! [`crate::sector::PrefetchedSectorSource`].
//!
//! Spawns a producer thread that fills a bounded pool of `Vec<u8>`
//! chunks from the underlying reader and ships them through a
//! channel; the consumer pulls filled chunks, uses them, and sends
//! the empty `Vec<u8>` back through a recycle channel so the
//! producer can re-fill in place. Result: zero allocations and zero
//! cross-thread frees in the steady-state hot loop.
//!
//! This is the byte-stream half of the freemkv mux highway —
//! `BytePrefetcher` feeds [`crate::mux::demux_thread::DemuxThread`]
//! for `m2ts://` (the only in-tree caller today, via
//! [`crate::mux::resolve`]), and works for any stream whose source is
//! an `io::Read` rather than a `SectorSource`.
use crate::halt::{Halt, POLL_INTERVAL};
use crossbeam_channel::{Receiver, RecvTimeoutError, SendTimeoutError, Sender, bounded};
use std::io::Read;
use std::thread::JoinHandle;
/// Items flowing through the forward channel.
pub type Batch = std::io::Result<Vec<u8>>;
/// Forward channel depth — how many filled buffers the producer can
/// stay ahead by. Two is enough to absorb a moderate consumer stall
/// without piling up bytes.
const FORWARD_DEPTH: usize = 2;
/// Recycle channel depth = forward + 1 so the producer always has at
/// least one buffer to fill while the consumer holds one.
const RECYCLE_DEPTH: usize = FORWARD_DEPTH + 1;
/// Default chunk size — 16 MiB matches the ISO-mux sector batch and
/// is large enough that per-chunk overhead is amortised; small
/// enough that the in-flight memory footprint stays bounded.
pub const DEFAULT_CHUNK_BYTES: usize = 16 * 1024 * 1024;
/// Returned from [`BytePrefetcher::into_channels`]. Owns the
/// producer-thread join handle so dropping the shell joins the
/// producer.
///
/// Drop blocks the calling thread until the producer exits. To
/// guarantee a prompt exit, drop the forward receiver and the recycle
/// sender first so the producer observes channel disconnection (or
/// cancel the [`Halt`] passed to [`BytePrefetcher::new`], which the
/// producer polls at [`POLL_INTERVAL`] granularity even while parked
/// on a channel op).
pub struct PrefetchShell {
producer: Option<JoinHandle<()>>,
}
impl Drop for PrefetchShell {
fn drop(&mut self) {
if let Some(h) = self.producer.take() {
let _ = h.join();
}
}
}
/// Spawned byte prefetcher. Drop joins the producer thread.
pub struct BytePrefetcher {
rx: Option<Receiver<Batch>>,
recycle_tx: Option<Sender<Vec<u8>>>,
producer: Option<JoinHandle<()>>,
}
impl BytePrefetcher {
/// Spawn the producer thread. `reader` must be `Send` because it
/// moves into the thread. `chunk_bytes` is the size of each
/// recycled buffer; pick the natural batch size of the
/// downstream demuxer (16 MiB for the BD-TS mux pipeline).
pub fn new<R: Read + Send + 'static>(
mut reader: R,
chunk_bytes: usize,
halt: Option<Halt>,
) -> std::io::Result<Self> {
// A zero-length chunk makes every recycled buffer an empty
// slice; `reader.read(&mut [])` returns Ok(0), which the loop
// below treats as EOF — the consumer would see a clean,
// silent zero-byte stream. Callers pass the downstream
// demuxer's batch size, which is always > 0.
debug_assert!(chunk_bytes > 0, "BytePrefetcher chunk_bytes must be > 0");
let (tx, rx) = bounded::<Batch>(FORWARD_DEPTH);
let (recycle_tx, recycle_rx) = bounded::<Vec<u8>>(RECYCLE_DEPTH);
// Seed the recycle pool. Without these the first
// `recycle_rx.recv()` would block forever (no consumer has
// returned a buffer yet).
for _ in 0..RECYCLE_DEPTH {
let _ = recycle_tx.send(vec![0u8; chunk_bytes]);
}
let producer = std::thread::Builder::new()
.name("freemkv-byte-prefetch".into())
.spawn(move || {
// Wrap the feed loop in catch_unwind so a panic in the inner
// `reader.read` (e.g. a decrypt-on-read slice/arith bug) is NOT
// indistinguishable from a clean finish at the demux boundary. A
// clean exit (EOF, halt, consumer disconnect) returns and drops
// `tx` → the demux loop reads RecvError as EOF (correct). A PANIC
// sends an explicit error sentinel first so the demux loop's
// `Ok(Err(_))` arm fires and propagates a typed error instead of
// converting the dropped channel into a clean `DemuxBatch::Eof`
// that would finalize a TRUNCATED mux while reporting success.
let body = std::panic::AssertUnwindSafe(|| {
let cancelled = || halt.as_ref().map(|h| h.is_cancelled()).unwrap_or(false);
// Liveness heartbeat: the producer blocks on the recycle and
// forward channels; a stalled consumer or a wedged reader shows
// up as the beat going silent. Total is unknown, so `pos` is
// cumulative bytes read.
let mut hb = crate::progress::Heartbeat::new("byte_prefetch");
let mut produced_bytes: u64 = 0;
loop {
hb.tick(produced_bytes, 0);
if cancelled() {
return;
}
// Park on the recycle channel, but re-poll halt
// every POLL_INTERVAL: a pure-AtomicBool Halt does
// not disconnect the channel, so a blocking recv()
// would never re-reach the cancel check.
let mut buf = loop {
match recycle_rx.recv_timeout(POLL_INTERVAL) {
Ok(b) => break b,
Err(RecvTimeoutError::Timeout) => {
if cancelled() {
return;
}
}
// Consumer dropped both channels.
Err(RecvTimeoutError::Disconnected) => return,
}
};
// Re-expose the full extent. After a short read the
// prior iteration truncated to n < chunk_bytes, so
// this regrows the length back to chunk_bytes
// without reallocating (capacity was fixed at
// construction and never shrinks).
if buf.len() < chunk_bytes {
buf.resize(chunk_bytes, 0);
} else {
// SAFETY: capacity is at least chunk_bytes
// after construction.
unsafe { buf.set_len(chunk_bytes) };
}
// Read up to one full chunk. Short reads are
// valid and common — pipe `truncate` so the
// consumer sees only the bytes that arrived.
let n = match reader.read(&mut buf[..]) {
Ok(0) => return, // EOF — drop tx, consumer sees RecvError
Ok(n) => n,
Err(e) => {
let _ = tx.send(Err(e));
return;
}
};
produced_bytes += n as u64;
buf.truncate(n);
// Hand off the filled buffer, re-polling halt on
// each timeout slice so a cancel can interrupt a
// producer parked on a saturated forward channel.
let mut pending = Ok(buf);
loop {
match tx.send_timeout(pending, POLL_INTERVAL) {
Ok(()) => break,
Err(SendTimeoutError::Timeout(returned)) => {
if cancelled() {
return;
}
pending = returned;
}
// Consumer dropped.
Err(SendTimeoutError::Disconnected(_)) => return,
}
}
}
});
if std::panic::catch_unwind(body).is_err() {
// Producer panicked mid-stream — surface a typed terminal
// error so the demux thread does NOT read the dropped channel
// as a clean EOF and truncate output.
let _ = tx.send(Err(crate::error::Error::DemuxThreadPanicked.into()));
}
})?;
Ok(Self {
rx: Some(rx),
recycle_tx: Some(recycle_tx),
producer: Some(producer),
})
}
/// Peel off the channels for zero-copy pipeline consumption. The
/// caller (typically [`crate::mux::demux_thread::DemuxThread`])
/// drains `rx`, runs the demuxer in place on each filled buffer,
/// and recycles back through `recycle_tx`.
pub fn into_channels(self) -> (Receiver<Batch>, Sender<Vec<u8>>, PrefetchShell) {
// MOVE the three fields out cleanly — never clone. Each of
// `rx` and `recycle_tx` ends up with exactly ONE live copy:
// the one in the returned tuple. The pre-1.0.0 implementation
// cloned both and then `mem::forget`-ed `self`, leaking the
// originals so an extra live receiver + sender survived
// forever. That defeated the channel-disconnection shutdown:
// when the demux consumer exited early (halt, or a `tx.send`
// error in `demux_thread`), the producer's `recycle_rx.recv()`
// and `tx.send()` never saw all-peers-dropped, so the producer
// never returned and `PrefetchShell::drop`'s `join()` hung.
//
// `ManuallyDrop` + `ptr::read` reads each field out by value
// and suppresses `self`'s own `Drop` (which would otherwise
// double-`join`), leaving NO extra live endpoint behind. This
// is the panic-free equivalent of the `Option::take` approach
// and mirrors `sector::prefetched::into_channels`.
let me = std::mem::ManuallyDrop::new(self);
// SAFETY: `me` is `ManuallyDrop`, so none of these fields will
// be dropped by `me`. Each `ptr::read` performs exactly one
// bitwise move out; every field is read exactly once and never
// touched again, so there are no double-frees and no aliasing.
let producer = unsafe { std::ptr::read(&me.producer) };
// SAFETY: `rx` and `recycle_tx` are always `Some` here —
// `into_channels` is the only way to consume a live
// `BytePrefetcher`; `Drop::drop` is suppressed by `ManuallyDrop`.
let rx = unsafe { std::ptr::read(&me.rx) }.expect("rx always Some before drop");
let recycle =
unsafe { std::ptr::read(&me.recycle_tx) }.expect("recycle_tx always Some before drop");
(rx, recycle, PrefetchShell { producer })
}
}
impl Drop for BytePrefetcher {
fn drop(&mut self) {
// Drop channel endpoints BEFORE joining the producer so the
// producer observes SendTimeoutError::Disconnected (forward tx)
// or RecvTimeoutError::Disconnected (recycle rx) and exits
// promptly. Without this, a non-EOF source fills the depth-2
// forward channel and then spins in send_timeout(POLL_INTERVAL)
// forever because rx is never drained, causing join() to
// deadlock.
drop(self.rx.take());
drop(self.recycle_tx.take());
if let Some(h) = self.producer.take() {
let _ = h.join();
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Endless reader: every `read` fills the whole buffer and never
/// hits EOF, so the producer keeps trying to push batches forward
/// until the forward channel disconnects. Exactly the shape that
/// wedged the pre-1.0.0 `clone + mem::forget` `into_channels`.
struct EndlessReader;
impl Read for EndlessReader {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
buf.fill(0);
Ok(buf.len())
}
}
/// Run `f` on a helper thread and fail if it does not finish within
/// `secs`. Turns a join-deadlock into a test failure instead of a
/// hung CI run.
fn within<F: FnOnce() + Send + 'static>(secs: u64, f: F) {
let (done_tx, done_rx) = bounded::<()>(1);
std::thread::spawn(move || {
f();
let _ = done_tx.send(());
});
assert!(
done_rx
.recv_timeout(std::time::Duration::from_secs(secs))
.is_ok(),
"operation did not complete within {secs}s (deadlock)"
);
}
/// The CRITICAL regression: after `into_channels`, dropping the
/// returned forward receiver + recycle sender must let the producer
/// observe disconnection and exit, so dropping the `PrefetchShell`
/// (which joins the producer) returns promptly. With the old
/// clone+forget the leaked endpoints kept the producer blocked and
/// this join hung forever.
#[test]
fn into_channels_drop_releases_producer() {
within(10, || {
// Small chunk so the producer cycles quickly and fills the
// forward channel without allocating much.
let pf = BytePrefetcher::new(EndlessReader, 4096, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
// Consumer goes away early (halt / abort analogue): drop
// both channel endpoints without draining to EOF.
drop(rx);
drop(recycle_tx);
// Joining the producer must not hang.
drop(shell);
});
}
/// Same property via the halt path: cancel the token, then the
/// producer must exit and the shell join must complete.
#[test]
fn halt_releases_producer() {
within(10, || {
let halt = Halt::new();
let pf = BytePrefetcher::new(EndlessReader, 4096, Some(halt.clone())).expect("spawn");
let (_rx, _recycle_tx, shell) = pf.into_channels();
halt.cancel();
drop(shell);
});
}
// ── Added hardening tests ───────────────────────────────────────
use std::io::Cursor;
/// Drain the forward channel, recycling every buffer, and
/// reassemble the bytes. Returns the concatenation of every
/// delivered chunk. Stops on RecvError (producer dropped tx == EOF)
/// or on the first Err batch (which it returns separately).
fn drain_to_vec(pf: BytePrefetcher) -> (Vec<u8>, Option<std::io::Error>) {
let (rx, recycle_tx, shell) = pf.into_channels();
let mut out = Vec::new();
let mut err = None;
while let Ok(batch) = rx.recv() {
match batch {
Ok(buf) => {
out.extend_from_slice(&buf);
// Recycle so the producer can refill. Ignore send
// error (producer may have already exited at EOF).
let _ = recycle_tx.send(buf);
}
Err(e) => {
err = Some(e);
break;
}
}
}
drop(rx);
drop(recycle_tx);
drop(shell);
(out, err)
}
/// CORE CONTRACT: the prefetcher must deliver every source byte,
/// in order, exactly once — never silently truncate or duplicate.
/// Source is 5000 bytes; chunk size 1024 forces multiple chunks
/// (4 full + 1 short of 904). The reassembled stream must equal the
/// source. Mutation: replacing `buf.truncate(n)` (line 141) with a
/// no-op would over-report bytes on the final short read and this
/// fails.
#[test]
fn delivers_all_bytes_in_order_across_chunks() {
within(10, || {
let src: Vec<u8> = (0..5000u32).map(|i| (i & 0xff) as u8).collect();
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 1024, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none(), "unexpected error batch: {err:?}");
assert_eq!(got, src, "prefetcher truncated or reordered bytes");
});
}
/// Short-read truncation: a reader that returns fewer bytes than
/// requested per call must NOT leave stale tail bytes in the
/// delivered chunk. Cursor over 10 bytes with a 4096 chunk yields a
/// single 10-byte chunk; the consumer must see exactly 10 bytes,
/// not 4096. Grounds `buf.truncate(n)` at line 141. Mutation:
/// delete the truncate and the chunk would carry 4086 zero bytes of
/// padding, failing the length assert.
#[test]
fn short_read_truncates_to_actual_length() {
within(10, || {
let src = vec![0xAB; 10];
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 4096, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none());
assert_eq!(got.len(), 10, "delivered chunk padded past actual read");
assert_eq!(got, src);
});
}
/// EOF semantics: an empty source (Cursor over `[]`) yields
/// `read() == Ok(0)` on the first call, which the producer treats
/// as EOF and returns, dropping tx. The consumer sees RecvError
/// (zero batches), NOT an Err batch and NOT a zero-length Ok batch.
/// Grounds the `Ok(0) => return` arm at line 134. Mutation:
/// changing `Ok(0) => return` to `Ok(0) => continue` would spin
/// forever (within() would time out).
#[test]
fn empty_source_yields_clean_eof_no_batches() {
within(10, || {
let pf = BytePrefetcher::new(Cursor::new(Vec::<u8>::new()), 4096, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
// No Ok batch should ever arrive; first recv must be Err
// (producer dropped tx at EOF).
let first = rx.recv();
assert!(
first.is_err(),
"empty source produced a batch instead of clean EOF: {first:?}"
);
drop(rx);
drop(recycle_tx);
drop(shell);
});
}
/// Error propagation: a reader that fails mid-stream must surface
/// the io::Error as an `Err` batch on the forward channel (line
/// 137), not swallow it. We deliver one good chunk then an error.
/// The consumer must see the good bytes followed by the error.
/// Mutation: changing `let _ = tx.send(Err(e)); return;` to a plain
/// `return` would drop the error silently and this fails.
#[test]
fn read_error_is_propagated_as_err_batch() {
within(10, || {
struct OneThenError {
served: bool,
}
impl Read for OneThenError {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if !self.served {
self.served = true;
let n = buf.len().min(8);
buf[..n].fill(0x11);
Ok(n)
} else {
Err(std::io::Error::other("synthetic mid-stream read failure"))
}
}
}
let pf = BytePrefetcher::new(OneThenError { served: false }, 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert_eq!(got, vec![0x11; 8], "good chunk lost");
let err = err.expect("read error must surface as an Err batch");
assert_eq!(err.kind(), std::io::ErrorKind::Other);
});
}
/// PANIC propagation: a reader that PANICS mid-stream must NOT be read as a
/// clean EOF at the demux boundary. The producer's catch_unwind sends an
/// explicit `Err` sentinel before the thread unwinds, so the consumer sees
/// the good bytes followed by an error batch — never a silent truncation.
/// Without the catch_unwind the panic would just drop `tx`, the consumer
/// would see RecvError (== clean EOF) and the partial output would be
/// finalized as if complete.
#[test]
fn read_panic_surfaces_as_err_batch_not_clean_eof() {
within(10, || {
struct OneThenPanic {
served: bool,
}
impl Read for OneThenPanic {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if !self.served {
self.served = true;
let n = buf.len().min(8);
buf[..n].fill(0x22);
Ok(n)
} else {
panic!("synthetic mid-stream reader panic");
}
}
}
let pf = BytePrefetcher::new(OneThenPanic { served: false }, 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert_eq!(got, vec![0x22; 8], "good chunk lost before the panic");
assert!(
err.is_some(),
"a mid-stream producer PANIC must surface as an Err batch, \
not a clean EOF (which would silently truncate the mux)"
);
});
}
/// Recycle-buffer reuse must NOT leak stale bytes between chunks of
/// different lengths. After a full chunk, a short read reuses the
/// same recycled buffer; lines 123-129 regrow it to chunk_bytes
/// before reading, then line 141 truncates to the short count. We
/// verify the short chunk carries only fresh bytes by reassembling
/// the full stream. Source: 8 bytes of 0xAA + 3 bytes of 0xBB, with
/// chunk_bytes=8 → chunk0 = 8×0xAA, chunk1 = 3×0xBB.
#[test]
fn recycled_buffer_carries_no_stale_tail() {
within(10, || {
let mut src = vec![0xAA; 8];
src.extend_from_slice(&[0xBB; 3]);
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 8, None).expect("spawn");
let (got, err) = drain_to_vec(pf);
assert!(err.is_none());
assert_eq!(
got, src,
"stale bytes from recycled buffer leaked into short chunk"
);
});
}
/// Exact-multiple boundary: when the source length is an exact
/// multiple of chunk_bytes, the final non-empty chunk is followed
/// by an `Ok(0)` EOF read, NOT a spurious empty Ok batch. 12 bytes
/// with chunk_bytes=4 → three 4-byte chunks then clean EOF. Total
/// bytes must equal 12 and no zero-length batch may appear.
#[test]
fn exact_multiple_length_no_trailing_empty_batch() {
within(10, || {
let src = vec![0x42u8; 12];
let pf = BytePrefetcher::new(Cursor::new(src.clone()), 4, None).expect("spawn");
let (rx, recycle_tx, shell) = pf.into_channels();
let mut total = 0usize;
let mut batch_count = 0usize;
while let Ok(Ok(buf)) = rx.recv() {
assert!(!buf.is_empty(), "producer emitted a zero-length batch");
total += buf.len();
batch_count += 1;
let _ = recycle_tx.send(buf);
}
assert_eq!(total, 12);
assert_eq!(batch_count, 3, "expected exactly 3 full chunks");
drop(rx);
drop(recycle_tx);
drop(shell);
});
}
/// Dropping the BytePrefetcher directly (without into_channels)
/// must join the producer cleanly when the source is finite. The
/// producer reaches EOF, drops tx, and exits; Drop's join returns.
/// Grounds the BytePrefetcher Drop impl (lines 202-208). Mutation:
/// removing the `Ok(0) => return` EOF exit would hang this join.
#[test]
fn drop_finite_prefetcher_joins_cleanly() {
within(10, || {
let pf = BytePrefetcher::new(Cursor::new(vec![1u8; 100]), 4096, None).expect("spawn");
// Drop without consuming — producer fills the forward
// channel (capacity 2), reaches EOF on the third read since
// 100 < 4096 (single chunk + EOF), drops tx, exits.
drop(pf);
});
}
/// Regression: dropping a BytePrefetcher directly (without
/// into_channels) with an ENDLESS source must not deadlock. Before
/// the fix, Drop joined the producer while rx/recycle_tx were still
/// alive (sibling field drop order), so the producer filled the
/// depth-2 forward channel and then spun in send_timeout forever
/// (rx never drained, halt=None). The fix drops rx+recycle_tx
/// BEFORE the join so the producer sees SendTimeoutError::Disconnected
/// and exits.
#[test]
fn drop_endless_prefetcher_joins_cleanly() {
within(10, || {
let pf = BytePrefetcher::new(EndlessReader, 4096, None).expect("spawn");
// Drop without consuming — the old Drop deadlocked here.
drop(pf);
});
}
}
+65
View File
@@ -0,0 +1,65 @@
//! Linux read-side platform hooks: sequential-access hint at open +
//! periodic page-cache eviction during streaming reads.
//!
//! ## Why both
//!
//! `POSIX_FADV_SEQUENTIAL` at open widens the kernel's readahead window
//! so each pread aggregates into fewer NFS round-trips. `DONTNEED` on
//! the consumed window (called periodically by the caller) drops the
//! already-read pages from the page cache so an 85 GB streaming ISO
//! read doesn't fill memory and starve concurrent writes (the MKV
//! output during mux). Together they mirror the write-side
//! WritebackPipeline's policy.
//!
//! ## History
//!
//! Pre-Phase-1 (0.20.7 baseline) had both. Phase 1's introduction of
//! `FileSectorSource` silently dropped the read-side DONTNEED, and
//! 0.21.2's revert of `SEQUENTIAL` (mistakenly attributing a regression
//! to it) removed the hint. Net effect: 85 GB of ISO reads pinned in
//! the page cache + no readahead widening → mux throughput collapse
//! from 18 MB/s historical to 2.7-8 MB/s on 0.21.x. Restored in 0.21.6.
use std::fs::File;
use std::os::unix::io::AsRawFd;
pub(super) fn hint_sequential(file: &File, _len_bytes: u64) {
// Best-effort: return value ignored. A fadvise failure has no
// user-observable consequence.
unsafe {
libc::posix_fadvise(file.as_raw_fd(), 0, 0, libc::POSIX_FADV_SEQUENTIAL);
}
}
/// Drop pages in the half-open byte range `[start, start+len)` from
/// the page cache. Called periodically by `read_sectors` to bound the
/// read-side page cache pressure.
pub(super) fn drop_window(file: &File, start: u64, len: u64) {
unsafe {
libc::posix_fadvise(
file.as_raw_fd(),
start as i64,
len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
}
/// Async-prefetch `len` bytes at `offset` into the page cache. The
/// kernel `readahead(2)` syscall queues the I/O and returns
/// immediately — it does NOT wait for completion. Called right after
/// each consumed read so the next batch's I/O overlaps with the
/// caller's processing of the current batch (decrypt + demux + mux).
///
/// Without this hint, with a synchronous demux consumer running at
/// ~50 MB/s and a single-spindle disk capable of ~150 MB/s, the disk
/// sits idle ~70% of each iteration because kernel readahead alone
/// (capped at `/sys/block/<dev>/queue/read_ahead_kb`, default 128 KB)
/// can only pre-stage a tiny slice of the next batch. An explicit
/// `readahead()` of the same size as the current batch tells the
/// kernel to queue the full next-batch read now.
pub(super) fn prefetch(file: &File, offset: u64, len: u64) {
unsafe {
libc::readahead(file.as_raw_fd(), offset as i64, len as usize);
}
}
+53
View File
@@ -0,0 +1,53 @@
//! macOS: hint the kernel to prefetch a generous chunk. macOS has no
//! direct `POSIX_FADV_SEQUENTIAL` equivalent; the idiomatic hint is
//! `fcntl(F_RDADVISE, &radvisory)` describing the byte range you
//! intend to read soon. We point it at the whole file (clamped to a
//! ceiling so a multi-TB ISO doesn't ask the kernel to prefetch
//! everything at once).
use std::fs::File;
use std::os::unix::io::AsRawFd;
/// Cap on the byte length we pass to `F_RDADVISE`. Asking for a
/// multi-GB readahead window is counterproductive — the OS doesn't
/// have that much cache to throw at one fd. 64 MiB is generous for
/// our use case (sweep, mux) so the kernel's prefetch ≥ our app-level
/// pipeline depth.
const RDADVISE_MAX_BYTES: i64 = 64 * 1024 * 1024;
pub(super) fn hint_sequential(file: &File, len_bytes: u64) {
let bytes = (len_bytes as i64).min(RDADVISE_MAX_BYTES);
let mut ra = libc::radvisory {
ra_offset: 0,
ra_count: bytes as libc::c_int,
};
// Best-effort.
unsafe {
libc::fcntl(file.as_raw_fd(), libc::F_RDADVISE, &mut ra);
}
}
/// macOS has no direct `POSIX_FADV_DONTNEED` equivalent for a byte
/// range. `fcntl(F_NOCACHE)` would disable caching globally on the fd
/// (too coarse — we want the unread region to still benefit). Best
/// approximation: no-op. macOS's unified buffer cache is generally
/// less prone to the pin-everything pathology that triggers the
/// regression on Linux NFS clients.
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
/// Async-prefetch the byte range `[offset, offset+len)`. macOS uses
/// the same `fcntl(F_RDADVISE, &radvisory)` primitive as the open-
/// time sequential hint, just targeted at a moving window instead of
/// the whole file. The kernel queues I/O for the requested range and
/// returns immediately.
pub(super) fn prefetch(file: &File, offset: u64, len: u64) {
let bytes = (len as i64).min(RDADVISE_MAX_BYTES);
let mut ra = libc::radvisory {
ra_offset: offset as libc::off_t,
ra_count: bytes as libc::c_int,
};
// Best-effort — kernel hint only.
unsafe {
libc::fcntl(file.as_raw_fd(), libc::F_RDADVISE, &mut ra);
}
}
+521
View File
@@ -0,0 +1,521 @@
//! [`FileSectorSource`] — read 2048-byte sectors from an ISO file on
//! disk via direct `seek + read_exact` (`pread`-equivalent) calls,
//! letting the kernel's own readahead policy manage prefetch.
//!
//! ## Why no app-level buffer
//!
//! Pre-0.21.3 this source held a 32 MiB (later 4 MiB) read-ahead
//! buffer to amortise per-sector NFS round-trips. Empirically that
//! buffer hurt: 32 MiB refills bursted the NFS TCP connection hard
//! enough to starve the concurrent writer, and even a 4 MiB window
//! gave the kernel less freedom to pipeline reads with writes. Direct
//! pread per call lets Linux's readahead widen as it detects the
//! sequential pattern, and naturally interleaves with writeback.
//!
//! ## DONTNEED on the consumed window
//!
//! Without page-cache eviction an 85 GB streaming ISO read pins the
//! entire file in memory, starves the concurrent writer, and collapses
//! mux throughput (observed: 2.7 MB/s mux on 0.21.5 vs. 70 MB/s
//! isolated NFS reads). Every [`READ_DROP_CHUNK_BYTES_DEFAULT`] of
//! consumed bytes we call `posix_fadvise(DONTNEED)` over that window,
//! mirroring the write-side [`crate::io::writeback::WritebackPipeline`]
//! policy.
//!
//! The drop window is accounted by a monotonic forward byte counter,
//! which matches the sequential streaming pattern the mux highway
//! drives. Under random or backward access the dropped range no longer
//! lines up with the bytes actually read — but `DONTNEED` is purely an
//! advisory cache hint with no correctness impact, so this degrades to
//! a slightly imprecise hint rather than a bug.
//!
//! ## Platform open hint
//!
//! On `open()` each platform issues its "sequential access expected"
//! hint so OS-level readahead widens. The hint and the DONTNEED call
//! live in per-OS sibling modules ([`linux::hint_sequential`] et al.)
//! — no inline `#[cfg]` in this file.
//!
//! ## Read-ahead prefetch
//!
//! After every consumed read we issue an OS-level prefetch hint for
//! the next equivalent-sized window (`platform::prefetch`). The
//! kernel queues that I/O asynchronously and returns immediately, so
//! the next batch's read overlaps with the caller's processing of
//! the current batch (decrypt + demux + mux). Without this the disk
//! sits idle ~70% of each iteration because kernel SEQUENTIAL
//! readahead alone (capped at `read_ahead_kb`, default 128 KB) is
//! far smaller than our 16 MiB app-level batch.
#[cfg(target_os = "linux")]
mod linux;
#[cfg(target_os = "macos")]
mod macos;
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
mod other;
#[cfg(target_os = "windows")]
mod windows;
#[cfg(target_os = "linux")]
use linux as platform;
#[cfg(target_os = "macos")]
use macos as platform;
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
use other as platform;
#[cfg(target_os = "windows")]
use windows as platform;
use std::fs::File;
use std::io::{Read, Seek, SeekFrom};
use std::path::Path;
use crate::error::{Error, Result};
use crate::sector::SectorSource;
use crate::consts::{SECTOR_BYTES, SECTOR_BYTES_U64};
/// Bytes-read threshold per `posix_fadvise(DONTNEED)` drop on the
/// read side. Mirrors `WRITEBACK_CHUNK_BYTES` so the read-side page
/// cache stays bounded the same way the write side does.
///
/// 32 MiB is the empirically tuned value on a 7200rpm HDD via SATA:
/// smaller windows (8 / 16 MiB) shorten the
/// kernel-readahead overlap and slow the producer; larger windows
/// (64 / 128 MiB) let the page cache pin enough of the ISO to
/// pressure concurrent writes. Override via `FREEMKV_READ_DROP_CHUNK_MIB`.
const READ_DROP_CHUNK_BYTES_DEFAULT: u64 = 32 * 1024 * 1024;
fn read_drop_chunk_bytes() -> u64 {
std::env::var("FREEMKV_READ_DROP_CHUNK_MIB")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|&n| n > 0)
.map(|n| n * 1024 * 1024)
.unwrap_or(READ_DROP_CHUNK_BYTES_DEFAULT)
}
/// SectorSource backed by a file (ISO image). Every `read_sectors`
/// call is a direct `seek + read_exact` against the underlying file
/// — kernel readahead handles prefetch, and every
/// [`READ_DROP_CHUNK_BYTES_DEFAULT`] bytes of consumed data the
/// platform's `DONTNEED` hook drops the consumed window from the
/// page cache to bound memory pressure.
pub struct FileSectorSource {
file: File,
/// Total file size in sectors. Constant after construction;
/// surfaced via [`SectorSource::capacity_sectors`].
capacity: u32,
/// Bytes read since the last DONTNEED drop. Drives the per-
/// [`read_drop_chunk_bytes`] page-cache eviction in read_sectors.
bytes_read_since_drop: u64,
/// File offset at which the current drop window starts. The next
/// DONTNEED drops from `drop_window_start` for
/// `bytes_read_since_drop` bytes. This advances monotonically with
/// the byte count, so it tracks the actual reads only under the
/// forward-sequential access the mux highway uses; under random
/// access it degrades to a harmless, imprecise advisory hint.
drop_window_start: u64,
/// Cached drop chunk size (resolved from env once at open).
drop_chunk_bytes: u64,
}
impl FileSectorSource {
/// Open an existing ISO file for reading. Capacity is derived
/// from `metadata().len() / 2048`. Returns
/// [`Error::IsoTooLarge`] if the file would exceed the 32-bit
/// LBA address space (~8 TB).
///
/// Issues the platform's "sequential access expected" hint on the
/// fd (Linux `posix_fadvise(SEQUENTIAL)`, macOS `fcntl(F_RDADVISE)`,
/// Windows no-op) so the kernel's readahead widens.
pub fn open(path: &Path) -> Result<Self> {
let file = File::open(path).map_err(|e| Error::IoError { source: e })?;
let len = file
.metadata()
.map_err(|e| Error::IoError { source: e })?
.len();
let sectors = len / SECTOR_BYTES_U64;
if sectors > u32::MAX as u64 {
return Err(Error::IsoTooLarge {
path: path.to_string_lossy().into_owned(),
});
}
let capacity = sectors as u32;
// Best-effort sequential hint. Ignored on platforms without
// an equivalent primitive (or where the API exists but the
// FS doesn't honour it).
platform::hint_sequential(&file, len);
Ok(Self {
file,
capacity,
bytes_read_since_drop: 0,
drop_window_start: 0,
drop_chunk_bytes: read_drop_chunk_bytes(),
})
}
}
impl SectorSource for FileSectorSource {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
out: &mut [u8],
_recovery: bool,
) -> Result<usize> {
let count = count as u32;
let bytes = count as usize * SECTOR_BYTES;
debug_assert!(
out.len() >= bytes,
"FileSectorSource::read_sectors: out len {} < requested {}",
out.len(),
bytes
);
if count == 0 {
return Ok(0);
}
let offset = lba as u64 * SECTOR_BYTES_U64;
self.file
.seek(SeekFrom::Start(offset))
.map_err(|e| Error::IoError { source: e })?;
self.file
.read_exact(&mut out[..bytes])
.map_err(|e| Error::IoError { source: e })?;
// Queue the next batch's read with the kernel before the
// caller starts processing what we just returned. readahead()
// is non-blocking — it queues I/O and returns, so the kernel
// pulls those pages into cache while the consumer (decrypt +
// demux + mux) runs. Next read_sectors call hits a warm cache.
platform::prefetch(&self.file, offset + bytes as u64, bytes as u64);
// Periodic page-cache eviction on the read side. Without
// this, an 85 GB streaming ISO read pins the entire file in
// the kernel page cache, which starves concurrent writes and
// collapses mux throughput. Mirrors the write-side
// WritebackPipeline's DONTNEED policy.
self.bytes_read_since_drop += bytes as u64;
if self.bytes_read_since_drop >= self.drop_chunk_bytes {
let drop_start = self.drop_window_start;
let drop_len = self.bytes_read_since_drop;
platform::drop_window(&self.file, drop_start, drop_len);
self.drop_window_start = drop_start + drop_len;
self.bytes_read_since_drop = 0;
}
Ok(bytes)
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write;
use tempfile::tempdir;
/// Build a deterministic ISO of `sectors` sectors where sector `n`
/// is filled with the byte pattern `((n & 0xff) as u8)`. Lets us
/// verify any sector by content alone.
fn make_iso(path: &std::path::Path, sectors: u32) {
let mut f = std::fs::File::create(path).unwrap();
let mut chunk = vec![0u8; SECTOR_BYTES];
for n in 0..sectors {
let b = (n & 0xff) as u8;
chunk.iter_mut().for_each(|c| *c = b);
f.write_all(&chunk).unwrap();
}
f.flush().unwrap();
}
/// Sectors used by spanning-boundary tests. Pick something that
/// exercises multi-megabyte reads without making test ISOs huge.
/// 8192 sectors = 16 MiB — large enough to cross any readahead
/// chunk size we set the kernel hint to.
const TEST_SPAN_SECTORS: u32 = 8192;
#[test]
fn sequential_reads_match_file() {
let total = TEST_SPAN_SECTORS * 2 + 17;
let dir = tempdir().unwrap();
let path = dir.path().join("seq.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), total);
let mut got = vec![0u8; SECTOR_BYTES];
for lba in 0..total {
src.read_sectors(lba, 1, &mut got, false).unwrap();
let expected = (lba & 0xff) as u8;
assert!(
got.iter().all(|b| *b == expected),
"sector {lba} content mismatch: expected 0x{expected:02x}"
);
}
}
#[test]
fn multi_sector_read_across_chunk_boundary() {
let total = TEST_SPAN_SECTORS * 2;
let dir = tempdir().unwrap();
let path = dir.path().join("span.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
let span_lba = TEST_SPAN_SECTORS - 2;
let mut buf4 = vec![0u8; SECTOR_BYTES * 4];
src.read_sectors(span_lba, 4, &mut buf4, false).unwrap();
for i in 0..4 {
let lba = span_lba + i as u32;
let expected = (lba & 0xff) as u8;
for b in &buf4[i * SECTOR_BYTES..(i + 1) * SECTOR_BYTES] {
assert_eq!(*b, expected, "byte mismatch at sub-sector {i}");
}
}
}
#[test]
fn backward_seek_reads_correct_bytes() {
// Read forward then jump back: the SectorSource contract is
// byte-correctness regardless of access pattern.
let total = TEST_SPAN_SECTORS * 2 + 5;
let dir = tempdir().unwrap();
let path = dir.path().join("back.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
let mut got = vec![0u8; SECTOR_BYTES];
src.read_sectors(TEST_SPAN_SECTORS + 1, 1, &mut got, false)
.unwrap();
src.read_sectors(0, 1, &mut got, false).unwrap();
assert!(got.iter().all(|b| *b == 0));
}
#[test]
fn read_at_eof_returns_correct_bytes() {
// File smaller than the readahead chunk — reads near EOF must
// still return correct bytes.
let total: u32 = 100;
let dir = tempdir().unwrap();
let path = dir.path().join("small.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), total);
let mut got = vec![0u8; SECTOR_BYTES];
src.read_sectors(0, 1, &mut got, false).unwrap();
src.read_sectors(total - 1, 1, &mut got, false).unwrap();
let expected = ((total - 1) & 0xff) as u8;
assert!(got.iter().all(|b| *b == expected));
}
#[test]
fn large_single_read() {
// A multi-MB single read must work — the implementation has
// no app-level chunking, so this just exercises the direct
// pread path on a larger request.
let total = TEST_SPAN_SECTORS + 100;
let dir = tempdir().unwrap();
let path = dir.path().join("big.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
let req = (TEST_SPAN_SECTORS + 1) as u16;
let req_bytes = req as usize * SECTOR_BYTES;
let mut big = vec![0u8; req_bytes];
src.read_sectors(0, req, &mut big, false).unwrap();
assert!(big[..SECTOR_BYTES].iter().all(|b| *b == 0));
let last_lba = req as u32 - 1;
let exp = (last_lba & 0xff) as u8;
let last_off = (req as usize - 1) * SECTOR_BYTES;
assert!(
big[last_off..last_off + SECTOR_BYTES]
.iter()
.all(|b| *b == exp)
);
}
#[test]
fn drop_chunk_size_env_override() {
// Explicit 8 MiB via env var.
// SAFETY: tests in this crate are single-threaded per the
// default cargo test harness, but std::env::set_var is
// declared `unsafe` since Rust 2024 (it can race with other
// threads / TLS). For a test that runs in-process before any
// FileSectorSource construction this is safe in practice.
unsafe {
std::env::set_var("FREEMKV_READ_DROP_CHUNK_MIB", "8");
}
assert_eq!(read_drop_chunk_bytes(), 8 * 1024 * 1024);
unsafe {
std::env::remove_var("FREEMKV_READ_DROP_CHUNK_MIB");
}
assert_eq!(read_drop_chunk_bytes(), READ_DROP_CHUNK_BYTES_DEFAULT);
// Garbage env value falls back to default.
unsafe {
std::env::set_var("FREEMKV_READ_DROP_CHUNK_MIB", "not-a-number");
}
assert_eq!(read_drop_chunk_bytes(), READ_DROP_CHUNK_BYTES_DEFAULT);
unsafe {
std::env::remove_var("FREEMKV_READ_DROP_CHUNK_MIB");
}
}
// ---------------------------------------------------------------
// Additional coverage.
// ---------------------------------------------------------------
/// `count == 0` must short-circuit to Ok(0) WITHOUT seeking or
/// reading, even at an out-of-range LBA — the early-return guard
/// runs before any I/O. Grounding: `if count == 0 { return Ok(0) }`.
#[test]
fn zero_count_returns_zero_no_io() {
let dir = tempdir().unwrap();
let path = dir.path().join("zc.iso");
make_iso(&path, 4);
let mut src = FileSectorSource::open(&path).unwrap();
// LBA far past EOF — must not matter because count==0 returns early.
let mut buf = [0u8; 1];
let n = src.read_sectors(1_000_000, 0, &mut buf, false).unwrap();
assert_eq!(n, 0);
}
/// Reading past EOF must ERROR (read_exact's UnexpectedEof), never
/// return a partial/short count. This is the core "never silently
/// truncate / never return fewer bytes than declared" property of
/// the SectorSource contract. Grounding: `self.file.read_exact(...)`
/// — read_exact fails if the file can't supply the full span.
#[test]
fn read_past_eof_errors_not_truncates() {
let dir = tempdir().unwrap();
let path = dir.path().join("eof.iso");
make_iso(&path, 4); // 4 sectors only
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 4);
// Request 2 sectors starting at LBA 3 → sector 4 doesn't exist.
let mut buf = vec![0u8; 2 * SECTOR_BYTES];
let r = src.read_sectors(3, 2, &mut buf, false);
let err = r.expect_err("reading past EOF must error, not short-read");
let io: std::io::Error = err.into();
assert_eq!(
io.kind(),
std::io::ErrorKind::UnexpectedEof,
"partial read at EOF must surface read_exact's UnexpectedEof"
);
}
/// On a successful full read the returned count MUST equal
/// `count * 2048` exactly — the declared byte count. Grounding:
/// `Ok(bytes)` where `bytes = count * SECTOR_BYTES`.
#[test]
fn full_read_returns_exact_declared_bytes() {
let dir = tempdir().unwrap();
let path = dir.path().join("exact.iso");
make_iso(&path, 16);
let mut src = FileSectorSource::open(&path).unwrap();
let mut buf = vec![0u8; 5 * SECTOR_BYTES];
let n = src.read_sectors(2, 5, &mut buf, false).unwrap();
assert_eq!(n, 5 * SECTOR_BYTES, "must return exactly count*2048 bytes");
}
/// Capacity is `file_len / 2048` (floor); trailing bytes that don't
/// complete a sector are NOT counted. A file of 4 sectors + 100
/// extra bytes reports capacity 4. Grounding: `len / SECTOR_BYTES`
/// integer division in `open`.
#[test]
fn capacity_floors_partial_trailing_sector() {
let dir = tempdir().unwrap();
let path = dir.path().join("partial.iso");
make_iso(&path, 4);
// Append 100 stray bytes (a torn final sector).
{
let mut f = std::fs::OpenOptions::new()
.append(true)
.open(&path)
.unwrap();
f.write_all(&[0xee; 100]).unwrap();
f.flush().unwrap();
}
let src = FileSectorSource::open(&path).unwrap();
assert_eq!(
src.capacity_sectors(),
4,
"partial trailing bytes must not inflate the sector capacity"
);
}
/// An empty file opens cleanly with capacity 0. Grounding:
/// `0 / 2048 == 0`, and the IsoTooLarge guard only fires for
/// oversize files.
#[test]
fn empty_file_capacity_zero() {
let dir = tempdir().unwrap();
let path = dir.path().join("empty.iso");
std::fs::File::create(&path).unwrap();
let src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 0);
}
/// Opening a nonexistent path returns an IoError (NotFound), not a
/// panic. Grounding: `File::open(path).map_err(...)`.
#[test]
fn open_missing_file_errors() {
let dir = tempdir().unwrap();
let path = dir.path().join("does-not-exist.iso");
let err = match FileSectorSource::open(&path) {
Ok(_) => panic!("missing file must error"),
Err(e) => e,
};
let io: std::io::Error = err.into();
assert_eq!(io.kind(), std::io::ErrorKind::NotFound);
}
/// A DONTNEED drop crossing the chunk threshold must not corrupt or
/// short subsequent reads — the eviction is a pure page-cache hint.
/// We read past the DEFAULT 32 MiB drop chunk (16384 sectors) so the
/// eviction block fires at least once, asserting every sector still
/// reads correctly. (Avoids mutating FREEMKV_READ_DROP_CHUNK_MIB to
/// sidestep a parallel-test env race with `drop_chunk_size_env_override`.)
/// Grounding: the `bytes_read_since_drop >= drop_chunk_bytes`
/// eviction block calls only `platform::drop_window` (advisory) and
/// resets counters — no data effect.
#[test]
fn dontneed_eviction_does_not_affect_data() {
// 32 MiB default chunk = 16384 sectors; read a bit past it.
let total = (READ_DROP_CHUNK_BYTES_DEFAULT / SECTOR_BYTES_U64) as u32 + 64;
let dir = tempdir().unwrap();
let path = dir.path().join("drop.iso");
make_iso(&path, total);
let mut src = FileSectorSource::open(&path).unwrap();
// Read in 16-sector batches to keep the loop fast while still
// crossing the drop boundary by byte count.
let batch = 16u16;
let mut got = vec![0u8; batch as usize * SECTOR_BYTES];
let mut lba = 0u32;
while lba + batch as u32 <= total {
src.read_sectors(lba, batch, &mut got, false).unwrap();
for i in 0..batch as u32 {
let expected = ((lba + i) & 0xff) as u8;
let off = i as usize * SECTOR_BYTES;
assert!(
got[off..off + SECTOR_BYTES].iter().all(|x| *x == expected),
"DONTNEED eviction corrupted sector {}",
lba + i
);
}
lba += batch as u32;
}
}
}
+11
View File
@@ -0,0 +1,11 @@
//! Fallback for targets without a known sequential-readahead hint
//! (BSDs, illumos, etc.). No-op — reads still work, they just don't
//! get the OS-level prefetch widening.
use std::fs::File;
pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {}
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
pub(super) fn prefetch(_file: &File, _offset: u64, _len: u64) {}
+28
View File
@@ -0,0 +1,28 @@
//! Windows: the canonical sequential-access hint is
//! `FILE_FLAG_SEQUENTIAL_SCAN`, which must be passed to `CreateFile`
//! at open time and cannot be set afterward via
//! `SetFileInformationByHandle`. Since `FileSectorSource::open` uses a
//! plain `File::open`, the hints in this module are no-op stubs.
use std::fs::File;
/// No-op stub. `FILE_FLAG_SEQUENTIAL_SCAN` can only be set at
/// `CreateFile` open time, which the plain `File::open` path does not
/// do, so there is no post-open hint to issue here.
pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {
tracing::debug!(
target: "mux",
"FileSectorSource hint_sequential: windows no-op stub"
);
}
/// Windows page-cache eviction is not exposed via a posix_fadvise
/// equivalent. The kernel does its own working-set management. No-op
/// for now.
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
/// Windows async-prefetch hint. With FILE_FLAG_SEQUENTIAL_SCAN at
/// open the kernel already prefetches aggressively, so there's no
/// per-range hint we'd add on top. No-op stub for parity with the
/// posix platforms.
pub(super) fn prefetch(_file: &File, _offset: u64, _len: u64) {}
+92
View File
@@ -0,0 +1,92 @@
//! Platform-aware crash-durability primitives.
//!
//! Two flush operations need OS-specific handling to make a write survive a
//! crash / power loss:
//!
//! - [`dir`] — fsync a directory so a prior `rename(2)` into it is durable.
//! After a crash a renamed file's dirent can otherwise be lost even though
//! the rename returned, because it is still page-cache-only. This is a POSIX
//! concept: on Windows std cannot even open a directory as a `File` (it does
//! not set `FILE_FLAG_BACKUP_SEMANTICS`), and NTFS/ReFS commit the rename's
//! dirent without an explicit directory flush — so it is a no-op there
//! rather than a failed open that logs on every marker write.
//!
//! - [`file_durable`] — fsync a file's contents + metadata. Opens the file
//! **read+write**: on Windows `File::sync_all` maps to `FlushFileBuffers`,
//! which requires a handle with write access and returns
//! `ERROR_ACCESS_DENIED` (os error 5) on a read-only handle. (A read-only
//! `File::open` + `sync_all` is legal on POSIX, which is why that bug only
//! bit Windows.) The open mode is platform-uniform, so this lives here with
//! no dispatch.
//!
//! Per the crate convention (see [`crate::io::writeback_file`]), platform
//! dispatch happens once here via cfg-gated `mod` decls — callers carry no
//! inline `#[cfg(...)]`.
use std::io;
use std::path::Path;
#[cfg(not(windows))]
mod posix;
#[cfg(windows)]
mod windows;
#[cfg(not(windows))]
use posix as platform;
#[cfg(windows)]
use windows as platform;
/// fsync a directory so a prior `rename(2)` into it is durable. Best-effort:
/// failures are logged and swallowed, never propagated — the renamed file's
/// bytes are already synced and the caller's write itself succeeded. No-op on
/// Windows (see module docs).
pub fn dir(path: &Path) {
platform::fsync_dir(path)
}
/// Durably flush an existing file's contents + metadata to stable storage.
///
/// Opens the file read+write (not read-only) so the flush succeeds on every
/// platform — see the module docs for the Windows `FlushFileBuffers` rationale.
/// The file must already exist; its bytes are left intact (no create/truncate).
pub fn file_durable(path: &Path) -> io::Result<()> {
let f = std::fs::OpenOptions::new()
.read(true)
.write(true)
.open(path)?;
f.sync_all()
}
#[cfg(test)]
mod tests {
use super::*;
/// `file_durable` opens read+write (so the flush works on Windows) and
/// syncs an existing file; a missing path surfaces as `Err` so the caller
/// treats it as "not durably synced". Platform-uniform — same on
/// unix/windows.
#[test]
fn file_durable_ok_for_existing_err_for_missing() {
let td = tempfile::tempdir().unwrap();
let f = td.path().join("data.bin");
std::fs::write(&f, b"durable").unwrap();
assert!(
file_durable(&f).is_ok(),
"an existing file must open read+write and fsync cleanly"
);
assert!(
file_durable(&td.path().join("absent.bin")).is_err(),
"a missing file must surface the open failure as Err"
);
}
/// `dir` is best-effort: it must return normally for a real directory
/// (POSIX fsyncs it, Windows no-ops) and must swallow — never panic on —
/// a missing directory.
#[test]
fn dir_is_best_effort_never_panics() {
let td = tempfile::tempdir().unwrap();
dir(td.path());
dir(&td.path().join("does-not-exist"));
}
}
+18
View File
@@ -0,0 +1,18 @@
//! POSIX directory-fsync. Active on unix and any non-Windows fallback target
//! (BSD, illumos, …) — all share the same `File::open(dir).sync_all()`
//! semantics. The Windows no-op lives in the sibling `windows` module.
use std::path::Path;
pub(super) fn fsync_dir(dir: &Path) {
match std::fs::File::open(dir) {
Ok(f) => {
if let Err(e) = f.sync_all() {
tracing::warn!(path = %dir.display(), error = %e, "failed to fsync directory");
}
}
Err(e) => {
tracing::warn!(path = %dir.display(), error = %e, "could not open directory to fsync");
}
}
}
+13
View File
@@ -0,0 +1,13 @@
//! Windows directory-fsync: a no-op.
//!
//! Directory fsync is a POSIX concept. std cannot open a directory as a `File`
//! on Windows (it does not set `FILE_FLAG_BACKUP_SEMANTICS`), so the POSIX impl
//! could only ever fail the open and log a spurious warning on every marker /
//! mapfile write. NTFS/ReFS commit a rename's directory entry without an
//! explicit directory flush, so skipping it here is correct — not a durability
//! regression. (File-content durability is handled platform-uniformly by
//! [`super::file_durable`].)
use std::path::Path;
pub(super) fn fsync_dir(_dir: &Path) {}
+26 -9
View File
@@ -10,22 +10,39 @@
//! no-op pipeline — their default cache policies have not been shown
//! to exhibit the same pathology for this access pattern.
//!
//! `Pipeline` + `Sink` (0.18) is the generic producer/consumer primitive
//! `FileSectorSource` is the read-side dual — it implements
//! [`crate::sector::SectorSource`] for an ISO file using direct
//! `pread`-equivalent calls so the kernel's own readahead policy runs
//! (which interleaves naturally with the concurrent writeback). It
//! pairs that with periodic `posix_fadvise(DONTNEED)` drops on the
//! consumed window so an 85 GB streaming ISO read doesn't fill the
//! page cache and starve the concurrent MKV write.
//!
//! `Pipeline` + `Sink` is the generic producer/consumer primitive
//! used by sweep, patch, and mux to overlap reads with writes via a
//! bounded channel + dedicated consumer thread.
//!
//! `byte_prefetcher` is the read-ahead producer feeding the mux
//! pipeline for `io::Read`-backed sources: a worker thread fills a
//! recycled pool of buffers and ships them through a channel, exposing
//! `BytePrefetcher` / `PrefetchShell`.
pub(crate) mod bounded;
pub mod byte_prefetcher;
pub mod file_sector_source;
pub mod fsync;
pub mod sink;
mod writeback;
mod writeback_file;
#[cfg(target_os = "macos")]
pub(crate) mod platform_macos;
pub mod pipeline;
pub(crate) use writeback_file::WritebackFile;
// Re-exports for the 0.18 redesign. Sweep + patch are both wired up
// (disc/sweep.rs, disc/patch.rs); mux migrates separately in autorip.
// `WRITE_THROUGH_DEPTH` is patch-specific and has no other in-tree
// caller — the targeted `#[allow]` keeps the re-export visible without
// dragging the rest of the module under `dead_code`.
#[allow(unused_imports)]
pub use pipeline::WRITE_THROUGH_DEPTH;
pub use pipeline::{DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, Sink};
pub use pipeline::{
DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, READ_PIPELINE_DEPTH, Sink, WRITE_PIPELINE_DEPTH,
WRITE_THROUGH_DEPTH,
};
+1184 -80
View File
File diff suppressed because it is too large Load Diff
+38
View File
@@ -0,0 +1,38 @@
//! Shared macOS `fcntl(F_PREALLOCATE)` definitions.
//!
//! The `libc` crate doesn't expose these symbols across all macOS SDK
//! versions, so we define them locally with values from
//! `/usr/include/sys/fcntl.h`. Two call sites (
//! [`crate::io::writeback_file`] and [`crate::io::sink::preallocate`])
//! need the same constants and `fstore_t` layout — keeping a single
//! source of truth here prevents the two copies from drifting.
//!
//! Module-level cfg gate lives in the parent (`io/mod.rs`); this file
//! is only compiled on macOS, so no inner `#![cfg]` is needed.
/// `fcntl(F_PREALLOCATE)` command number from `sys/fcntl.h`.
pub(crate) const F_PREALLOCATE: libc::c_int = 42;
/// Anchor preallocation at the current physical EOF.
pub(crate) const F_PEOFPOSMODE: libc::c_int = 3;
/// Prefer a contiguous allocation. Try this first; on `EINVAL` (no
/// contiguous run of that size), fall back to `F_ALLOCATEALL`.
pub(crate) const F_ALLOCATECONTIG: libc::c_uint = 0x0000_0002;
/// Allow non-contiguous allocation. Stronger guarantee than just
/// asking for `F_ALLOCATECONTIG` because the kernel will piece
/// together fragments rather than failing.
pub(crate) const F_ALLOCATEALL: libc::c_uint = 0x0000_0004;
/// `fstore_t` from `sys/fcntl.h`. `repr(C)` because we hand it to
/// `fcntl(F_PREALLOCATE)` which writes through the pointer.
#[repr(C)]
#[derive(Clone, Copy)]
pub(crate) struct Fstore {
pub fst_flags: libc::c_uint,
pub fst_posmode: libc::c_int,
pub fst_offset: libc::off_t,
pub fst_length: libc::off_t,
pub fst_bytesalloc: libc::off_t,
}
+233
View File
@@ -0,0 +1,233 @@
//! `LocalFileSink` — `BufWriter<File>` for the common local-disk case.
//!
//! Buffering: 4 MiB internal `BufWriter`. Sized to coalesce the small
//! per-PES writes that come out of the muxer into kernel-page-aligned
//! flushes without making the buffer big enough to matter for memory
//! pressure on a single concurrent rip.
//!
//! `Seek` flushes the underlying `BufWriter` first; otherwise a seek
//! could leapfrog buffered data and silently corrupt the file. This is
//! the same shape `BufWriter` itself uses when it impls `Seek` in
//! stdlib, and is necessary for MKV's seek-back operations (cluster
//! size patch, Cues index, segment header backpatch) to land on the
//! right offset.
//!
//! [`SequentialSink`](super::SequentialSink) is implemented explicitly
//! (not via a blanket impl) so its `finish()` flushes the `BufWriter`
//! and `fsync`s the file even when called through a `dyn` trait object;
//! [`RandomAccessSink`](super::RandomAccessSink) is implemented over the
//! `Seek` impl below.
use std::fs::{File, OpenOptions};
use std::io::{self, BufWriter, Seek, SeekFrom, Write};
use std::path::Path;
use super::preallocate;
use super::{RandomAccessSink, SequentialSink};
const BUFFER_BYTES: usize = 4 * 1024 * 1024;
/// Random-access write sink for local disks.
///
/// Wraps a `BufWriter<File>` with a 4 MiB internal buffer and forwards
/// `Write`/`Seek` so any call site that previously held a `File` or
/// `WritebackFile` can drop this in. `finish()` flushes the buffer and
/// runs `sync_all` on the underlying file so the caller can drop it
/// without losing data.
///
/// Construction always opens the file `create + truncate + read +
/// write`. `read` is enabled so the same handle can be reused for a
/// verification re-read after the mux (the existing
/// `FileSectorSink::create` pattern). On Linux, [`with_size_hint`]
/// additionally calls `fallocate(FALLOC_FL_KEEP_SIZE)` to pre-reserve
/// extents.
///
/// [`with_size_hint`]: Self::with_size_hint
pub struct LocalFileSink {
inner: BufWriter<File>,
}
impl LocalFileSink {
/// Open `path` for writing, truncating any existing contents.
pub fn create(path: &Path) -> io::Result<Self> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(path)?;
Ok(Self {
inner: BufWriter::with_capacity(BUFFER_BYTES, file),
})
}
/// Like [`Self::create`] but additionally calls the per-OS
/// preallocate path with `size_bytes`. On Linux this is
/// `fallocate(FALLOC_FL_KEEP_SIZE)` so the on-disk extents are
/// reserved up front (reducing fragmentation for big sequential
/// muxer output); on other OSes it is a no-op today. Failures
/// from the preallocate call are non-fatal — the file is still
/// returned, just without the size reservation.
pub fn with_size_hint(path: &Path, size_bytes: u64) -> io::Result<Self> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(path)?;
preallocate::preallocate(&file, size_bytes);
Ok(Self {
inner: BufWriter::with_capacity(BUFFER_BYTES, file),
})
}
/// Drain the internal buffer and `fsync` the underlying file.
/// Idempotent with `Drop` (the `BufWriter` also flushes on drop;
/// this call additionally surfaces fsync errors to the caller).
/// [`SequentialSink::finish`](super::SequentialSink::finish)
/// delegates here so the durable flush happens through a trait
/// object too.
pub fn sync_all(&mut self) -> io::Result<()> {
self.inner.flush()?;
self.inner.get_ref().sync_all()
}
}
impl SequentialSink for LocalFileSink {
/// Flush the 4 MiB `BufWriter` and `fsync` the file. Overriding the
/// trait default is what makes a `dyn SequentialSink` / `dyn
/// RandomAccessSink` `finish()` actually durable instead of a no-op.
fn finish(&mut self) -> io::Result<()> {
self.sync_all()
}
}
impl RandomAccessSink for LocalFileSink {}
impl Write for LocalFileSink {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.inner.write(buf)
}
fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {
self.inner.write_all(buf)
}
fn flush(&mut self) -> io::Result<()> {
self.inner.flush()
}
}
impl Seek for LocalFileSink {
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
// Flush before seeking so buffered bytes land at the offset
// they were written for, not the new one.
self.inner.flush()?;
self.inner.get_mut().seek(from)
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Read;
#[test]
fn write_seek_roundtrip() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("rt.bin");
let mut s = LocalFileSink::create(&p).unwrap();
s.write_all(b"AAAA").unwrap();
s.write_all(b"BBBB").unwrap();
// Seek back over the second word and overwrite.
s.seek(SeekFrom::Start(4)).unwrap();
s.write_all(b"CCCC").unwrap();
s.sync_all().unwrap();
drop(s);
let mut f = File::open(&p).unwrap();
let mut got = Vec::new();
f.read_to_end(&mut got).unwrap();
assert_eq!(&got[..], b"AAAACCCC");
}
#[test]
fn drop_flushes() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("drop.bin");
{
let mut s = LocalFileSink::create(&p).unwrap();
s.write_all(b"buffered").unwrap();
// No explicit flush / sync_all — BufWriter drop runs the
// flush and the file should land on disk.
}
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..], b"buffered");
}
#[test]
fn with_size_hint_creates_writable_file() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("sz.bin");
let mut s = LocalFileSink::with_size_hint(&p, 64 * 1024).unwrap();
s.write_all(b"hint-ok").unwrap();
s.sync_all().unwrap();
drop(s);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..], b"hint-ok");
}
// ── Added hardening tests ───────────────────────────────────────
/// `create` must TRUNCATE an existing file (OpenOptions
/// `.truncate(true)`, lines 52-58). Pre-seed a long file, recreate
/// it via the sink, write a shorter payload — the old tail must be
/// gone. Mutation: dropping `.truncate(true)` would leave the stale
/// tail and the length assert fails.
#[test]
fn create_truncates_existing_file() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("trunc.bin");
std::fs::write(&p, vec![0xFFu8; 4096]).unwrap();
let mut s = LocalFileSink::create(&p).unwrap();
s.write_all(b"short").unwrap();
s.sync_all().unwrap();
drop(s);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(
bytes.len(),
5,
"create must truncate the pre-existing 4096 bytes"
);
assert_eq!(&bytes, b"short");
}
/// Seek must flush the BufWriter FIRST so buffered bytes land at
/// their intended offset, not the post-seek one (lines 121-128, and
/// the module doc's silent-corruption warning). We write into the
/// buffer (no explicit flush), seek backward, write again, and
/// confirm the first write stayed at offset 0. Mutation: removing
/// the `self.inner.flush()?` in `seek` would flush the first 4
/// bytes at the seeked offset, corrupting the file.
#[test]
fn seek_flushes_buffer_before_moving() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("seek-flush.bin");
let mut s = LocalFileSink::create(&p).unwrap();
// These bytes sit in the 4 MiB BufWriter, unflushed.
s.write_all(b"HEAD").unwrap();
// Seek forward to offset 10; the buffered HEAD must be flushed
// to offset 0 BEFORE the position moves.
s.seek(SeekFrom::Start(10)).unwrap();
s.write_all(b"TAIL").unwrap();
s.sync_all().unwrap();
drop(s);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(
&bytes[0..4],
b"HEAD",
"buffered head landed at the wrong offset"
);
assert_eq!(&bytes[10..14], b"TAIL");
}
}
+239
View File
@@ -0,0 +1,239 @@
//! Output-sink trait split for the buffering architecture.
//!
//! Two traits, one for each capability axis of an output destination:
//!
//! - [`SequentialSink`] — anything you can `Write` to in order. Sockets,
//! pipes, append-only stores, plain files. Containers that don't need
//! seek (M2TS, fMP4, HEVC elementary) target this.
//! - [`RandomAccessSink`] — everything `SequentialSink` plus a working
//! `Seek`. Local files, NFS files, anything with random-write
//! semantics. Containers that need backpatch (MKV cluster sizes, Cues
//! index, MP4 moov-at-end) target this.
//!
//! `RandomAccessSink: SequentialSink` — every random-access sink is
//! also a valid sequential sink. The muxer is generic over which it
//! requires (`MkvMux<S: RandomAccessSink>`, `M2tsMux<S: SequentialSink>`)
//! so an attempt to mux MKV to a network socket is a compile error.
//!
//! Buffering policy belongs to the concrete sink, not to a wrapper at
//! the call site. `LocalFileSink` wraps a `BufWriter<File>` with a
//! 4 MiB buffer for the common local-disk case; `WritebackFile`
//! (separate module) wraps a `File` with the adaptive-chunk
//! `sync_file_range` machinery for the Linux+NFS case.
use std::io::{Seek, Write};
mod local_file;
mod preallocate;
mod socket;
pub use local_file::LocalFileSink;
pub use socket::{SocketSink, UdpSocketSink};
/// Sequential-only write destination. Sockets, pipes, append-only
/// stores. No seek. Implementations own their write buffering — the
/// trait does not impose or hide any buffering of its own.
///
/// `finish` drains any internal buffering and signals end-of-stream to
/// the underlying transport (close-write on a socket, flush + fsync on
/// a buffered file, etc.). The default impl flushes via [`Write::flush`]
/// — correct for an unbuffered destination — but every concrete sink in
/// this module overrides it to drain its own buffer and run its
/// transport-specific finalisation (socket `shutdown(Write)`, file
/// `fsync`). There is deliberately NO blanket `impl SequentialSink for
/// T`: a blanket impl would force the no-op-style default on every
/// concrete sink (a blanket impl cannot be overridden per-type without a
/// coherence conflict), so a `Box<dyn SequentialSink>` / `&mut dyn
/// SequentialSink` `finish()` call would silently skip the flush and
/// transport shutdown. With explicit per-type impls the vtable dispatches
/// `finish` to the real implementation, so flush + durable-finish
/// actually happen through a trait object.
pub trait SequentialSink: Write + Send {
fn finish(&mut self) -> std::io::Result<()> {
self.flush()
}
}
/// Random-access write destination. Local files, NFS files, anything
/// with a working `Seek`. Inherits the `SequentialSink` contract — a
/// random-access sink is always usable as a sequential sink.
pub trait RandomAccessSink: SequentialSink + Seek {}
/// Pick the right `RandomAccessSink` impl for `dest` based on its
/// filesystem type.
///
/// - Linux + NFS path → `WritebackFile` with its adaptive-chunk
/// sync_file_range machinery and (when supported) `fallocate` size
/// hint.
/// - everything else → [`LocalFileSink`] over `BufWriter<File>`. On
/// non-Linux there is no `WritebackFile` machinery to opt into, and
/// on local Linux the kernel's default writeback policy is already
/// fine.
///
/// `size_hint`, when present, is forwarded to the per-OS preallocate
/// path (`fallocate(KEEP_SIZE)` on Linux, `F_PREALLOCATE` on macOS when
/// implemented, no-op elsewhere).
///
/// Returns a boxed trait object so the call site (mux construction)
/// stays agnostic of which concrete sink got picked.
// Not yet wired into mux::resolve (follow-up commit). Kept `pub(crate)` until
// then so an unfinished signature isn't frozen into the public 1.0 API.
#[allow(dead_code)]
pub(crate) fn open_for_mkv(
dest: &std::path::Path,
size_hint: Option<u64>,
) -> std::io::Result<Box<dyn RandomAccessSink>> {
#[cfg(target_os = "linux")]
{
use crate::platform::fs_type::{FsType, detect};
if detect(dest) == FsType::Nfs {
let wf = match size_hint {
Some(n) => crate::io::WritebackFile::create_with_size_hint(dest, n)?,
None => crate::io::WritebackFile::create(dest)?,
};
return Ok(Box::new(wf));
}
}
// Only Linux differentiates the sink by filesystem type (NFS gets
// the WritebackFile machinery); every other OS always uses
// `LocalFileSink`. Reference `detect` as a value (no call, no
// `statfs` syscall) so it isn't flagged dead on non-Linux while
// still avoiding the wasted probe whose result we'd discard.
#[cfg(not(target_os = "linux"))]
let _ = crate::platform::fs_type::detect;
let sink = match size_hint {
Some(n) => LocalFileSink::with_size_hint(dest, n)?,
None => LocalFileSink::create(dest)?,
};
Ok(Box::new(sink))
}
#[cfg(test)]
mod tests {
use super::*;
// Type-level assertion: the concrete sinks satisfy the trait
// objects. These functions never run; they just have to type-check.
fn _assert_is_sequential(_: &mut dyn SequentialSink) {}
fn _assert_is_random_access(_: &mut dyn RandomAccessSink) {}
#[test]
fn concrete_sinks_satisfy_traits() {
let dir = tempfile::tempdir().unwrap();
// `LocalFileSink` is a random-access (and thus sequential) sink.
let mut s = LocalFileSink::create(&dir.path().join("b.bin")).unwrap();
_assert_is_sequential(&mut s);
_assert_is_random_access(&mut s);
// `WritebackFile` ditto, via its explicit per-type impls.
let mut wf = crate::io::WritebackFile::create(&dir.path().join("c.bin")).unwrap();
_assert_is_sequential(&mut wf);
_assert_is_random_access(&mut wf);
}
#[test]
fn open_for_mkv_returns_a_random_access_sink() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("c.bin");
let mut sink = open_for_mkv(&p, Some(64 * 1024)).unwrap();
use std::io::{Seek, SeekFrom, Write};
sink.write_all(b"hello").unwrap();
sink.seek(SeekFrom::Start(0)).unwrap();
sink.finish().unwrap();
drop(sink);
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..5], b"hello");
}
/// finish() through a `dyn SequentialSink` trait object must
/// dispatch to the concrete sink's override (flush + fsync), not a
/// no-op default. This is the regression test for the silent-no-op
/// finish() bug.
#[test]
fn finish_through_trait_object_flushes_local_file() {
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("trait-finish.bin");
let sink = LocalFileSink::create(&p).unwrap();
// Box as the trait object the production path uses.
let mut boxed: Box<dyn SequentialSink> = Box::new(sink);
boxed.write_all(b"buffered-tail").unwrap();
// finish() through the vtable must drain the 4 MiB BufWriter and
// fsync; the bytes must be visible to a separate reader BEFORE
// we drop the sink (drop-flush must not be what saves us).
boxed.finish().unwrap();
let bytes = std::fs::read(&p).unwrap();
assert_eq!(&bytes[..], b"buffered-tail");
}
// ── Added hardening tests ───────────────────────────────────────
use std::io::{self, Write};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
/// A minimal `SequentialSink` that does NOT override `finish`, so it
/// exercises the trait's DEFAULT impl (lines 51-55), which must call
/// `Write::flush`. We record whether flush ran. This pins the
/// documented contract that the default `finish` is "correct for an
/// unbuffered destination" by flushing. Mutation: changing the
/// default `finish` body from `self.flush()` to `Ok(())` would set
/// `flushed=false` and fail.
struct FlushTracker {
flushed: Arc<AtomicBool>,
bytes: Arc<AtomicUsize>,
}
impl Write for FlushTracker {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.bytes.fetch_add(buf.len(), Ordering::SeqCst);
Ok(buf.len())
}
fn flush(&mut self) -> io::Result<()> {
self.flushed.store(true, Ordering::SeqCst);
Ok(())
}
}
// Uses the DEFAULT finish() — deliberately no override.
impl SequentialSink for FlushTracker {}
#[test]
fn default_finish_flushes() {
let flushed = Arc::new(AtomicBool::new(false));
let bytes = Arc::new(AtomicUsize::new(0));
let mut sink = FlushTracker {
flushed: flushed.clone(),
bytes: bytes.clone(),
};
sink.write_all(b"abc").unwrap();
assert!(
!flushed.load(Ordering::SeqCst),
"flush should not run before finish"
);
sink.finish().unwrap();
assert!(
flushed.load(Ordering::SeqCst),
"default SequentialSink::finish must call Write::flush"
);
assert_eq!(bytes.load(Ordering::SeqCst), 3);
}
/// `open_for_mkv` with `None` size hint must still produce a working
/// random-access sink (the `match size_hint { None => ... }` arm,
/// lines 103-106). Round-trip a seek-back patch through it to prove
/// both Write and Seek dispatch. Mutation: if the None arm returned
/// a sequential-only sink the seek would not compile / would fail.
#[test]
fn open_for_mkv_without_size_hint_is_random_access() {
use std::io::{Seek, SeekFrom};
let dir = tempfile::tempdir().unwrap();
let p = dir.path().join("nohint.bin");
let mut sink = open_for_mkv(&p, None).unwrap();
sink.write_all(b"AAAABBBB").unwrap();
sink.seek(SeekFrom::Start(4)).unwrap();
sink.write_all(b"CCCC").unwrap();
sink.finish().unwrap();
drop(sink);
assert_eq!(std::fs::read(&p).unwrap(), b"AAAACCCC");
}
}
+23
View File
@@ -0,0 +1,23 @@
//! Linux `fallocate(FALLOC_FL_KEEP_SIZE)` preallocation.
//!
//! `KEEP_SIZE` reserves extents without changing the apparent file
//! length, which matches the muxer's expectation that writes still grow
//! the file naturally.
use std::fs::File;
use std::os::unix::io::AsRawFd;
pub(super) fn preallocate_impl(file: &File, size_bytes: u64) {
let fd = file.as_raw_fd();
// Clamp to the signed `off_t` range fallocate expects; an unchecked
// `as i64` cast would wrap a >= 2^63 size to a negative length that
// fallocate rejects with EINVAL (silent no-op).
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX);
// FALLOC_FL_KEEP_SIZE = 0x01.
let rc = unsafe { libc::fallocate(fd, libc::FALLOC_FL_KEEP_SIZE, 0, len) };
tracing::debug!(
target: "mux",
"LocalFileSink fallocate size_hint={size_bytes} rc={rc} ok={}",
rc == 0
);
}
+43
View File
@@ -0,0 +1,43 @@
//! macOS `F_PREALLOCATE` extent reservation.
//!
//! `fcntl(F_PREALLOCATE)` with `F_ALLOCATECONTIG | F_ALLOCATEALL` first
//! (prefer a contiguous run but accept scattered extents to satisfy the
//! full length) and fall back to `F_ALLOCATEALL` alone on failure.
//! Reported file size is unchanged — the muxer's writes still grow it.
use std::fs::File;
use std::os::unix::io::AsRawFd;
use crate::io::platform_macos::{
F_ALLOCATEALL, F_ALLOCATECONTIG, F_PEOFPOSMODE, F_PREALLOCATE, Fstore,
};
pub(super) fn preallocate_impl(file: &File, size_bytes: u64) {
let fd = file.as_raw_fd();
// Clamp to the signed `off_t` range; an unchecked `as off_t` cast
// would wrap a >= 2^63 size to a negative length.
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX) as libc::off_t;
let mut store = Fstore {
// Prefer a contiguous run but accept scattered extents to
// satisfy the full length. Without F_ALLOCATEALL the first
// attempt is best-effort and can return rc=0 with a partial
// allocation, so the fallback below would never fire. Matches
// writeback_file/macos.rs.
fst_flags: F_ALLOCATECONTIG | F_ALLOCATEALL,
fst_posmode: F_PEOFPOSMODE,
fst_offset: 0,
fst_length: len,
fst_bytesalloc: 0,
};
let mut rc = unsafe { libc::fcntl(fd, F_PREALLOCATE, &mut store as *mut Fstore) };
if rc == -1 {
// Fall back to non-contiguous only.
store.fst_flags = F_ALLOCATEALL;
rc = unsafe { libc::fcntl(fd, F_PREALLOCATE, &mut store as *mut Fstore) };
}
tracing::debug!(
target: "mux",
"LocalFileSink F_PREALLOCATE size_hint={size_bytes} rc={rc} bytesalloc={}",
store.fst_bytesalloc
);
}
+27
View File
@@ -0,0 +1,27 @@
//! Per-OS extent preallocation. Best-effort; failures are logged at
//! debug and otherwise swallowed because the file is still usable
//! without the size reservation — only large-file fragmentation gets
//! marginally worse.
use std::fs::File;
#[cfg(target_os = "linux")]
mod linux;
#[cfg(target_os = "macos")]
mod macos;
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
mod other;
#[cfg(target_os = "linux")]
use linux::preallocate_impl;
#[cfg(target_os = "macos")]
use macos::preallocate_impl;
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
use other::preallocate_impl;
/// Reserve `size_bytes` of disk space for `file`'s on-disk extents.
/// Reported file size is unchanged — writes still grow the file
/// naturally; only the allocator's extent map is primed.
pub(super) fn preallocate(file: &File, size_bytes: u64) {
preallocate_impl(file, size_bytes);
}
+10
View File
@@ -0,0 +1,10 @@
//! Fallback preallocate impl. No-op.
use std::fs::File;
pub(super) fn preallocate_impl(_file: &File, size_bytes: u64) {
tracing::debug!(
target: "mux",
"LocalFileSink preallocate size_hint={size_bytes} skipped (no platform impl)"
);
}
+376
View File
@@ -0,0 +1,376 @@
//! TCP / UDP socket sinks (sequential-only).
//!
//! [`SocketSink`] wraps a `TcpStream` in a 1 MiB `BufWriter`. Constructor
//! tunes `SO_SNDBUF` to a caller hint when provided. `finish()` flushes
//! the buffer then `shutdown(Write)`s the socket so the peer sees clean
//! end-of-stream.
//!
//! [`UdpSocketSink`] wraps a connected `UdpSocket`. Each `write` call
//! emits exactly one datagram — the caller is responsible for packetizing
//! to a reasonable MTU (188 × 7 = 1316 bytes for MPEG-TS-over-UDP is the
//! conventional choice). `finish()` is a no-op; UDP has no end-of-stream
//! marker.
//!
//! Both types implement [`SequentialSink`] explicitly so their
//! `finish()` dispatches correctly through a `dyn SequentialSink` trait
//! object (the `SocketSink` override drains the buffer and
//! `shutdown(Write)`s; the `UdpSocketSink` override flushes only).
//! Neither implements `Seek`, so neither satisfies [`RandomAccessSink`]
//! — using one with `MkvMux` is a compile error, which is the design
//! intent.
//!
//! [`SequentialSink`]: super::SequentialSink
//! [`RandomAccessSink`]: super::RandomAccessSink
use std::io::{self, BufWriter, Write};
use std::net::{Shutdown, SocketAddr, TcpStream, ToSocketAddrs, UdpSocket};
use super::SequentialSink;
/// `BufWriter` capacity for [`SocketSink`]. 1 MiB matches the typical
/// kernel send-buffer ceiling and keeps small-write amplification from
/// containers (TS = 188-byte packets, fMP4 fragment headers = ~100 bytes)
/// from translating into syscall storms.
const TCP_BUF_CAPACITY: usize = 1024 * 1024;
/// Sequential-only sink over a TCP connection.
///
/// Wraps a `BufWriter<TcpStream>`; the inner `TcpStream` is kept as a
/// clone so [`finish`](Self::finish) can call `shutdown(Write)` after
/// flushing the buffer (the buffered writer doesn't expose the socket
/// directly).
pub struct SocketSink {
/// Buffered write half. All payload bytes go through this.
buf: BufWriter<TcpStream>,
/// Shutdown handle — clone of the socket inside `buf`. Used only by
/// `finish()` for `shutdown(Write)`; never read or written through.
shutdown_handle: TcpStream,
}
impl SocketSink {
/// Open a TCP connection to `addr` and wrap it for sequential
/// writing. `sndbuf_bytes`, when present, is forwarded to
/// `setsockopt(SO_SNDBUF)` as a kernel hint — the OS may clamp it.
///
/// `addr` accepts anything `ToSocketAddrs` does: `"192.0.2.1:1234"`,
/// `("host", 1234)`, a `SocketAddr`, etc.
pub fn connect<A: ToSocketAddrs>(addr: A, sndbuf_bytes: Option<usize>) -> io::Result<Self> {
let stream = TcpStream::connect(addr)?;
// `set_nodelay(true)` keeps small writes (TS packet trains, fMP4
// moof headers) from sitting in Nagle's algorithm until the buffer
// fills. The BufWriter already absorbs syscall overhead; Nagle
// would just add latency without coalescing more. It is a latency
// hint, not a correctness requirement, so a platform that rejects
// TCP_NODELAY must not fail the connect — demote the error.
let _ = stream.set_nodelay(true);
if let Some(n) = sndbuf_bytes {
set_send_buffer(&stream, n)?;
}
let shutdown_handle = stream.try_clone()?;
Ok(Self {
buf: BufWriter::with_capacity(TCP_BUF_CAPACITY, stream),
shutdown_handle,
})
}
}
impl Write for SocketSink {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.buf.write(buf)
}
fn flush(&mut self) -> io::Result<()> {
self.buf.flush()
}
}
impl SequentialSink for SocketSink {
/// Drain the BufWriter and `shutdown(Write)` the underlying socket
/// so the peer sees a clean EOF. Overriding the trait default is
/// what makes a `dyn SequentialSink` `finish()` send the buffered
/// tail and the EOF instead of silently dropping them.
fn finish(&mut self) -> io::Result<()> {
self.buf.flush()?;
// `shutdown(Write)` signals clean EOF to the peer. Errors here
// are non-fatal — the connection may have already been torn down
// by the peer — but we surface them so callers can log.
self.shutdown_handle.shutdown(Shutdown::Write)
}
}
/// Sequential-only sink over a connected UDP socket.
///
/// Each [`write`](Write::write) call sends exactly one datagram. The
/// caller is responsible for splitting payload at packet boundaries —
/// for MPEG-TS this means 7 × 188 = 1316 bytes per datagram, the
/// industry standard for MPEG-TS-over-UDP. No buffering happens here;
/// adding it would silently merge datagrams.
///
/// `finish()` is a no-op: UDP has no end-of-stream marker. Closing the
/// socket happens on drop.
pub struct UdpSocketSink {
socket: UdpSocket,
}
impl UdpSocketSink {
/// Bind a local UDP socket to an ephemeral port and `connect` it to
/// `peer`. `connect` doesn't open a connection — it just fixes the
/// peer address so subsequent `send` calls don't need to repeat it,
/// and so receive-side filtering rejects packets from other sources.
///
/// `sndbuf_bytes`, when present, is a hint to `SO_SNDBUF`.
pub fn connect<A: ToSocketAddrs>(peer: A, sndbuf_bytes: Option<usize>) -> io::Result<Self> {
// Resolve the peer first so the local bind matches its address
// family. Binding `0.0.0.0:0` (IPv4) and then connecting to an
// IPv6 peer fails with EAFNOSUPPORT, so pick the wildcard that
// matches the resolved family.
let peer_addr = peer
.to_socket_addrs()?
.next()
.ok_or_else(|| io::Error::from(io::ErrorKind::AddrNotAvailable))?;
let bind_addr = match peer_addr {
SocketAddr::V4(_) => "0.0.0.0:0",
SocketAddr::V6(_) => "[::]:0",
};
// Bind to the matching wildcard / any port. The kernel picks an
// ephemeral source port and the source IP at first send.
let socket = UdpSocket::bind(bind_addr)?;
socket.connect(peer_addr)?;
if let Some(n) = sndbuf_bytes {
set_udp_send_buffer(&socket, n)?;
}
Ok(Self { socket })
}
}
impl Write for UdpSocketSink {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
// `send` writes the entire datagram or fails — no partial sends
// for UDP. Match `Write::write`'s contract by reporting bytes
// accepted.
self.socket.send(buf)
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
impl SequentialSink for UdpSocketSink {
/// UDP has no end-of-stream marker, so there is nothing to shut
/// down; `write` already sent each datagram unbuffered. Flush is a
/// no-op but kept explicit so the trait-object `finish()` matches
/// the concrete behaviour.
fn finish(&mut self) -> io::Result<()> {
Ok(())
}
}
// ── Platform `SO_SNDBUF` tuning ────────────────────────────────────────────
//
// std's `TcpStream` / `UdpSocket` don't expose `SO_SNDBUF`. We drop to
// libc on Linux + macOS (the libc-dep targets in Cargo.toml). On other
// targets the hint is silently ignored — the socket still works, the
// kernel just picks its own send-buffer size.
#[cfg(any(target_os = "linux", target_os = "macos"))]
fn set_send_buffer(stream: &TcpStream, bytes: usize) -> io::Result<()> {
use std::os::unix::io::AsRawFd;
setsockopt_sndbuf(stream.as_raw_fd(), bytes)
}
#[cfg(any(target_os = "linux", target_os = "macos"))]
fn set_udp_send_buffer(socket: &UdpSocket, bytes: usize) -> io::Result<()> {
use std::os::unix::io::AsRawFd;
setsockopt_sndbuf(socket.as_raw_fd(), bytes)
}
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
fn set_send_buffer(_stream: &TcpStream, _bytes: usize) -> io::Result<()> {
// Non-Linux-non-macOS targets aren't in Cargo.toml's libc dep list;
// silently ignore the hint rather than failing the connect. Callers
// can detect via the lack of an explicit "sndbuf applied" signal
// (not provided, intentionally — this is a hint, not a guarantee).
Ok(())
}
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
fn set_udp_send_buffer(_socket: &UdpSocket, _bytes: usize) -> io::Result<()> {
Ok(())
}
#[cfg(any(target_os = "linux", target_os = "macos"))]
fn setsockopt_sndbuf(fd: std::os::unix::io::RawFd, bytes: usize) -> io::Result<()> {
// Clamp into c_int range; SO_SNDBUF takes an `int` argument.
let want: libc::c_int = bytes.try_into().unwrap_or(libc::c_int::MAX);
let ret = unsafe {
libc::setsockopt(
fd,
libc::SOL_SOCKET,
libc::SO_SNDBUF,
&want as *const _ as *const libc::c_void,
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
)
};
if ret != 0 {
return Err(io::Error::last_os_error());
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Read;
use std::net::{TcpListener, UdpSocket};
use std::thread;
/// Bind a listener, accept on a thread, return (listener_addr,
/// accepted-bytes future via JoinHandle).
#[test]
fn socket_sink_round_trips_bytes() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let accept = thread::spawn(move || {
let (mut sock, _) = listener.accept().unwrap();
let mut buf = Vec::new();
sock.read_to_end(&mut buf).unwrap();
buf
});
let mut sink = SocketSink::connect(addr, Some(256 * 1024)).unwrap();
// Write enough to overflow the BufWriter at least once, then a
// tail that lives in the buffer until `finish` flushes.
let big: Vec<u8> = (0..(2 * TCP_BUF_CAPACITY))
.map(|i| (i & 0xff) as u8)
.collect();
sink.write_all(&big).unwrap();
sink.write_all(b"tail\n").unwrap();
sink.finish().unwrap();
drop(sink);
let received = accept.join().unwrap();
assert_eq!(received.len(), big.len() + 5);
assert_eq!(&received[..big.len()], &big[..]);
assert_eq!(&received[big.len()..], b"tail\n");
}
#[test]
fn socket_sink_is_sequential_only() {
// Compile-time assertion via dyn — if this ever started
// satisfying `RandomAccessSink`, the trait split would be broken.
fn _assert_seq(_: &mut dyn super::super::SequentialSink) {}
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let _accept = thread::spawn(move || {
let _ = listener.accept();
});
let mut sink = SocketSink::connect(addr, None).unwrap();
_assert_seq(&mut sink);
// The negative is harder to assert directly (no `is_not<T>`),
// but `SocketSink` does not impl `Seek`, so it can't unify with
// `RandomAccessSink`'s super-bound. The Phase 2 blanket impl
// `impl<T: SequentialSink + Seek> RandomAccessSink for T {}` thus
// excludes it by construction.
}
#[test]
fn udp_socket_sink_delivers_datagrams() {
let receiver = UdpSocket::bind("127.0.0.1:0").unwrap();
receiver
.set_read_timeout(Some(std::time::Duration::from_secs(2)))
.unwrap();
let addr = receiver.local_addr().unwrap();
let mut sink = UdpSocketSink::connect(addr, Some(128 * 1024)).unwrap();
sink.write_all(&[1, 2, 3, 4, 5]).unwrap();
sink.write_all(&[9, 9, 9]).unwrap();
sink.finish().unwrap();
let mut buf = [0u8; 64];
let n1 = receiver.recv(&mut buf).unwrap();
assert_eq!(&buf[..n1], &[1, 2, 3, 4, 5]);
let n2 = receiver.recv(&mut buf).unwrap();
assert_eq!(&buf[..n2], &[9, 9, 9]);
}
// ── Added hardening tests ───────────────────────────────────────
/// `SocketSink::finish` must signal a clean EOF to the peer via
/// `shutdown(Write)` (lines 91-97). The receiving side's
/// `read_to_end` only returns when it observes that EOF — if
/// `finish` merely flushed without the shutdown, `read_to_end`
/// would block forever (the socket stays half-open). We assert the
/// receiver completes promptly AND sees the buffered tail.
/// Mutation: replacing the `shutdown(Write)` line with `Ok(())`
/// makes the accept thread hang and the join times out.
#[test]
fn finish_signals_eof_to_peer() {
use std::sync::mpsc;
use std::time::Duration;
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let (tx, rx) = mpsc::channel();
thread::spawn(move || {
let (mut sock, _) = listener.accept().unwrap();
let mut buf = Vec::new();
// Returns only when the peer half-closes (shutdown Write).
sock.read_to_end(&mut buf).unwrap();
let _ = tx.send(buf);
});
let mut sink = SocketSink::connect(addr, None).unwrap();
sink.write_all(b"unflushed-tail").unwrap();
sink.finish().unwrap();
// read_to_end must complete because finish() shut down writes.
let received = rx
.recv_timeout(Duration::from_secs(3))
.expect("peer never saw EOF — finish() did not shutdown(Write)");
assert_eq!(received, b"unflushed-tail");
}
/// UDP `write` must emit ONE datagram per call carrying exactly the
/// bytes passed — no buffering, no coalescing (doc lines 100-108).
/// Two writes of different lengths must arrive as two separate
/// datagrams of those exact lengths, in order. Mutation: adding a
/// BufWriter to UdpSocketSink (the doc explicitly forbids it) would
/// merge these into one datagram and the second `recv` would time
/// out.
#[test]
fn udp_write_is_one_datagram_per_call() {
let receiver = UdpSocket::bind("127.0.0.1:0").unwrap();
receiver
.set_read_timeout(Some(std::time::Duration::from_secs(2)))
.unwrap();
let addr = receiver.local_addr().unwrap();
let mut sink = UdpSocketSink::connect(addr, None).unwrap();
// Distinct lengths so a merge would be detectable.
let n_a = sink.write(&[0xAA; 10]).unwrap();
let n_b = sink.write(&[0xBB; 20]).unwrap();
assert_eq!(n_a, 10);
assert_eq!(n_b, 20);
let mut buf = [0u8; 256];
let first = receiver.recv(&mut buf).unwrap();
assert_eq!(first, 10, "first datagram must be exactly 10 bytes");
assert!(buf[..first].iter().all(|&b| b == 0xAA));
let second = receiver.recv(&mut buf).unwrap();
assert_eq!(second, 20, "second datagram must be exactly 20 bytes");
assert!(buf[..second].iter().all(|&b| b == 0xBB));
}
/// UDP `finish` is a documented no-op (lines 157-165): there is no
/// EOF marker for UDP. Calling it must not error and must not
/// affect prior datagrams. Mutation: if `finish` tried to
/// `shutdown` the UDP socket it could error or close it
/// prematurely; here it must just return Ok.
#[test]
fn udp_finish_is_noop_ok() {
let receiver = UdpSocket::bind("127.0.0.1:0").unwrap();
let addr = receiver.local_addr().unwrap();
let mut sink = UdpSocketSink::connect(addr, None).unwrap();
assert!(sink.finish().is_ok());
// A second finish is equally harmless.
assert!(sink.finish().is_ok());
}
}
+555 -33
View File
@@ -14,9 +14,62 @@
//! a full chunk's worth of work to flush — the wait is near-instant.
//! Dirty cache stays bounded at ~2 × `chunk_bytes` and writes drain
//! continuously instead of in bursts.
//!
//! The chunk size is adaptive: we measure the elapsed time of the
//! `WAIT_AFTER` call over a rolling window of the last 16 chunks and
//! resize the chunk based on the p95. Slow storage (NFS, network
//! shares, HDD) sees larger chunks to amortise per-chunk overhead;
//! fast storage (NVMe) sees smaller chunks to keep cache pressure
//! tight. Bounds: [4 MiB, 256 MiB].
//!
//! ## NFS escape hatch
//!
//! `sync_file_range(WAIT_AFTER)` on an NFS-mounted file can block
//! indefinitely waiting for the server's commit ack. If the server
//! never acks (network partition, server-side hang, slow commit), the
//! syscall never returns and the consumer thread is stuck inside the
//! kernel — `/api/stop` can't reach it because halt is cooperative.
//!
//! When `fstatfs` reports the file lives on an NFS mount
//! (`f_type == NFS_SUPER_MAGIC`), the pipeline skips the WAIT_AFTER +
//! `posix_fadvise(DONTNEED)` dance entirely. NFS clients have their
//! own buffering and commit semantics that handle dirty-page bounds
//! without us forcing the issue. The async `SYNC_FILE_RANGE_WRITE`
//! kickoff still runs (non-blocking by spec) so writeback still gets
//! a nudge.
//!
//! ## Defence in depth: WAIT_AFTER timeout
//!
//! Even on local storage, a degraded disk or odd filesystem driver
//! could in principle wedge inside WAIT_AFTER. Each WAIT_AFTER call
//! runs on a worker thread with a 30s recv_timeout on its result
//! channel. On timeout we log a loud error, set a `degraded` flag,
//! and from then on skip WAIT_AFTER + DONTNEED for the rest of the
//! pipeline's life (same shape as the NFS path). The worker thread
//! is intentionally leaked — it unwinds whenever the syscall
//! eventually returns or the process exits. The mux continues; the
//! original dirty-burst pathology re-emerges but the rip can still
//! finish instead of freezing.
use std::collections::VecDeque;
use std::fs::File;
use std::os::unix::io::{AsRawFd, RawFd};
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
const ADAPTIVE_WINDOW: usize = 16;
const CHUNK_BYTES_MIN: u64 = 4 * 1024 * 1024;
const CHUNK_BYTES_MAX: u64 = 256 * 1024 * 1024;
const ADAPTIVE_GROW_MS: u64 = 200;
const ADAPTIVE_SHRINK_MS: u64 = 20;
/// Every N chunks, emit a `debug!` snapshot of the current chunk
/// size so operators tailing the log can see where the autoscaler
/// settled.
const SIZE_LOG_INTERVAL: u64 = 32;
/// Hard upper bound on a single `sync_file_range(WAIT_AFTER)` call.
/// Beyond this we declare the pipeline degraded and stop calling
/// WAIT_AFTER for the rest of its life.
const WAIT_AFTER_TIMEOUT: Duration = Duration::from_secs(30);
pub(crate) struct WritebackPipeline {
/// Aliases the wrapping `WritebackFile::file`. Only valid for the
@@ -25,9 +78,35 @@ pub(crate) struct WritebackPipeline {
/// `WritebackFile` and never exposed outside that wrapper, which
/// is what keeps the alias sound.
fd: RawFd,
/// An owned clone of the file descriptor, held so that any
/// leaked WAIT_AFTER worker thread retains a valid reference to
/// the underlying file description for the duration of its
/// syscall — even if the original `WritebackFile` is closed first
/// and the OS reuses its fd number. `None` only when `try_clone`
/// failed at construction (rare); the pipeline falls back to the
/// pre-clone `fd` integer in that case, which carries the original
/// fd-reuse risk but is no worse than the previous behaviour.
wait_file: Option<File>,
chunk_bytes: u64,
last_flush_pos: u64,
pending: Option<(u64, u64)>,
/// Rolling window of recent `WAIT_AFTER` elapsed_ms measurements.
wait_after_window: VecDeque<u64>,
/// Count of chunks emitted (used to space out periodic
/// `debug!` size snapshots).
chunk_count: u64,
/// True when the underlying file is on an NFS mount. NFS makes
/// WAIT_AFTER unsafe (can block forever on missing server ack), so
/// we skip it entirely and let the NFS client handle commit on
/// close.
is_nfs: bool,
/// Set the first time WAIT_AFTER exceeds [`WAIT_AFTER_TIMEOUT`].
/// Once set, behaviour matches the NFS path for the rest of the
/// pipeline's life. A plain `AtomicBool`: the flag is only ever
/// touched on the owning thread (the spawned WAIT_AFTER worker never
/// reads or writes it). `AtomicBool` over `bool` only because the
/// load/store sites read cleanly; no sharing is needed today.
degraded: AtomicBool,
}
impl WritebackPipeline {
@@ -36,14 +115,63 @@ impl WritebackPipeline {
/// itself, or kept inside the same struct that owns `file` — the
/// alias is unchecked.
pub(crate) fn new(file: &File, start_pos: u64, chunk_bytes: u64) -> Self {
let fd = file.as_raw_fd();
let is_nfs = detect_nfs(fd);
// Clone the fd so any leaked WAIT_AFTER worker thread keeps the
// file description alive. Log but continue on clone failure.
let wait_file = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
tracing::warn!(
target: "mux",
"WritebackPipeline fd={fd}: try_clone failed ({e}), WAIT_AFTER workers \
will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
tracing::info!(
target: "mux",
"WritebackPipeline fd={fd} is_nfs={is_nfs} chunk_bytes={chunk_bytes} strategy={}",
if is_nfs { "nfs-skip-wait" } else { "wait+dontneed" }
);
Self {
fd: file.as_raw_fd(),
fd,
wait_file,
chunk_bytes,
last_flush_pos: start_pos,
pending: None,
wait_after_window: VecDeque::with_capacity(ADAPTIVE_WINDOW),
chunk_count: 0,
is_nfs,
degraded: AtomicBool::new(false),
}
}
/// True if we should bypass the WAIT_AFTER + DONTNEED finalisation
/// step. NFS always bypasses; local storage bypasses once the
/// pipeline has flipped to degraded after a WAIT_AFTER timeout.
#[inline]
fn skip_wait(&self) -> bool {
self.is_nfs || self.degraded.load(Ordering::Relaxed)
}
/// Produce a fresh per-call `File` clone for the WAIT_AFTER worker.
///
/// Each call to `wait_after_with_timeout` needs its own owned clone
/// so the worker thread keeps the file description alive for the
/// duration of the syscall. We clone from `self.wait_file` (itself a
/// clone taken at construction) rather than from the original file.
///
/// Returns `None` only if `wait_file` is `None` (construction
/// try_clone failed) or if the second-level try_clone fails — both
/// rare; the fallback raw-fd path in `wait_after_with_timeout`
/// handles that case.
#[inline]
fn clone_for_worker(&self) -> Option<File> {
self.wait_file.as_ref().and_then(|f| f.try_clone().ok())
}
/// Caller advanced the file position to `pos`. If a chunk boundary
/// was crossed, kick async writeback for the just-completed chunk
/// and finalise the previous one.
@@ -51,27 +179,146 @@ impl WritebackPipeline {
if pos < self.last_flush_pos.saturating_add(self.chunk_bytes) {
return;
}
unsafe {
let chunk_off = self.last_flush_pos as i64;
let chunk_len = (pos - self.last_flush_pos) as i64;
libc::sync_file_range(self.fd, chunk_off, chunk_len, libc::SYNC_FILE_RANGE_WRITE);
if let Some((prev_off, prev_len)) = self.pending.take() {
libc::sync_file_range(
self.fd,
prev_off as i64,
prev_len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
self.pending = Some((chunk_off as u64, chunk_len as u64));
// Byte offsets are unsigned throughout; the signed cast happens
// only at the libc call boundary where the kernel ABI requires
// `i64`. `saturating_sub` documents and hardens the line-above
// guard that `pos >= last_flush_pos`.
let chunk_off: u64 = self.last_flush_pos;
let chunk_len: u64 = pos.saturating_sub(self.last_flush_pos);
let mut wait_ms: u64 = 0;
let mut fadvise_ms: u64 = 0;
// Async kickoff for the just-completed chunk runs on every
// path (NFS, degraded, normal) — it's nominally non-blocking
// by spec and gives the kernel an early hint that this range
// is ready to flush.
let kickoff_rc = unsafe {
libc::sync_file_range(
self.fd,
chunk_off as i64,
chunk_len as i64,
libc::SYNC_FILE_RANGE_WRITE,
)
};
if kickoff_rc != 0 {
// Non-fatal: the async write-out hint failed, but the data is
// still in the page cache and will be flushed by later fsync /
// kernel writeback. Surface it for diagnosability.
tracing::warn!(
target: "freemkv::io",
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
"sync_file_range(WRITE) kickoff failed"
);
}
if let Some((prev_off, prev_len)) = self.pending.take() {
if self.skip_wait() {
// NFS branch (or degraded fallback after a prior
// timeout): the WAIT_AFTER + DONTNEED dance is what
// hangs on NFS — skip it. We still advance `pending`
// so the next call has a stable cycle.
} else {
// Normal local-storage branch with belt-and-braces
// timeout. If WAIT_AFTER hangs > WAIT_AFTER_TIMEOUT
// we mark the pipeline degraded, log a loud error,
// and fall through to the skip path on subsequent
// calls.
match wait_after_with_timeout(self.clone_for_worker(), self.fd, prev_off, prev_len)
{
Some(ms) => {
wait_ms = ms;
let t_fadv = Instant::now();
unsafe {
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
fadvise_ms = t_fadv.elapsed().as_millis() as u64;
self.record_wait(wait_ms);
}
None => {
// Timeout branch: switch to NFS-style skip
// for the rest of the pipeline's life. Do
// NOT call DONTNEED — if WAIT_AFTER hasn't
// returned, the pages aren't safely flushed.
self.degraded.store(true, Ordering::Relaxed);
// Once degraded we skip DONTNEED, so every subsequent
// chunk's pages stay resident until close — the same
// page-cache exposure profile as NFS. Shrink to the
// floor so that exposure window is as small as the NFS
// path keeps it, instead of whatever the adaptive sizing
// had grown chunk_bytes to (up to 256 MiB).
self.chunk_bytes = CHUNK_BYTES_MIN;
tracing::error!(
target: "mux",
"WritebackPipeline WAIT_AFTER timed out after {}s on chunk off={} len={}, marking writeback degraded (subsequent chunks will skip WAIT_AFTER + DONTNEED, chunk_bytes lowered to floor)",
WAIT_AFTER_TIMEOUT.as_secs(),
prev_off,
prev_len
);
}
}
}
}
self.pending = Some((chunk_off, chunk_len));
self.last_flush_pos = pos;
self.chunk_count += 1;
tracing::trace!(
target: "mux",
"WritebackPipeline chunk off={} len={} wait_after_ms={wait_ms} fadvise_ms={fadvise_ms} chunk_bytes={} skip_wait={}",
chunk_off,
chunk_len,
self.chunk_bytes,
self.skip_wait(),
);
if self.chunk_count % SIZE_LOG_INTERVAL == 0 {
tracing::debug!(
target: "mux",
"WritebackPipeline chunk_bytes={} after {} chunks is_nfs={} degraded={}",
self.chunk_bytes,
self.chunk_count,
self.is_nfs,
self.degraded.load(Ordering::Relaxed),
);
}
}
/// Push a new `WAIT_AFTER` measurement into the rolling window
/// and, if the window is full, adapt `chunk_bytes` based on p95.
fn record_wait(&mut self, wait_ms: u64) {
if self.wait_after_window.len() == ADAPTIVE_WINDOW {
self.wait_after_window.pop_front();
}
self.wait_after_window.push_back(wait_ms);
if self.wait_after_window.len() < ADAPTIVE_WINDOW {
return;
}
// p95 index, derived from the window size so it stays valid if
// ADAPTIVE_WINDOW changes (a hard-coded `[14]` would panic OOB
// for a window <= 14). For the default 16 this is index 15
// (ceil(16 * 95 / 100) - 1 = 15), i.e. the top sample.
let mut sorted: Vec<u64> = self.wait_after_window.iter().copied().collect();
sorted.sort_unstable();
let p95_idx = (ADAPTIVE_WINDOW * 95).div_ceil(100).min(ADAPTIVE_WINDOW) - 1;
let p95 = sorted[p95_idx];
let old = self.chunk_bytes;
let new = if p95 > ADAPTIVE_GROW_MS && self.chunk_bytes < CHUNK_BYTES_MAX {
(self.chunk_bytes * 2).min(CHUNK_BYTES_MAX)
} else if p95 < ADAPTIVE_SHRINK_MS && self.chunk_bytes > CHUNK_BYTES_MIN {
(self.chunk_bytes / 2).max(CHUNK_BYTES_MIN)
} else {
self.chunk_bytes
};
if new != old {
self.chunk_bytes = new;
tracing::info!(
target: "mux",
"WritebackPipeline adaptive chunk_bytes {} -> {} p95_ms={p95}",
old,
new
);
}
}
/// Caller is about to seek away from the current write region.
@@ -85,20 +332,295 @@ impl WritebackPipeline {
/// or when discarding the pipeline.
pub(crate) fn finalize(&mut self) {
if let Some((prev_off, prev_len)) = self.pending.take() {
unsafe {
libc::sync_file_range(
self.fd,
prev_off as i64,
prev_len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
tracing::debug!(
target: "mux",
"WritebackPipeline finalize chunk off={prev_off} len={prev_len} skip_wait={} is_nfs={} degraded={}",
self.skip_wait(),
self.is_nfs,
self.degraded.load(Ordering::Relaxed),
);
if self.skip_wait() {
// NFS / degraded: skip WAIT_AFTER + DONTNEED. close()
// / sync_all() handle commit through their normal
// paths.
return;
}
match wait_after_with_timeout(self.clone_for_worker(), self.fd, prev_off, prev_len) {
Some(_ms) => unsafe {
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
},
None => {
self.degraded.store(true, Ordering::Relaxed);
tracing::error!(
target: "mux",
"WritebackPipeline finalize WAIT_AFTER timed out after {}s on chunk off={prev_off} len={prev_len}, marking writeback degraded",
WAIT_AFTER_TIMEOUT.as_secs(),
);
}
}
}
}
}
/// Probe whether `fd` lives on an NFS mount. Thin wrapper around
/// [`crate::platform::fs_type::detect_fd`] so writeback policy and
/// general-purpose fs-type classification stay in sync (same magic
/// numbers, same musl-vs-glibc cast handling).
///
/// Fails open: any classification other than NFS counts as "not NFS"
/// (including `Unknown` on `fstatfs` error) — better to run the
/// normal local-storage path on a misdetected NFS mount and surface
/// the freeze loudly via [`WAIT_AFTER_TIMEOUT`] than to needlessly
/// disable writeback bounding on every local file because of a
/// transient stat error.
fn detect_nfs(fd: RawFd) -> bool {
matches!(
crate::platform::fs_type::detect_fd(fd),
crate::platform::fs_type::FsType::Nfs
)
}
/// Run `sync_file_range(WAIT_AFTER)` on a worker thread and wait up
/// to [`WAIT_AFTER_TIMEOUT`] for it to return. `Some(elapsed_ms)` on
/// success; `None` on timeout. On timeout the worker thread is
/// intentionally leaked — it unwinds whenever the syscall eventually
/// returns or the process exits.
///
/// This delegates to [`crate::io::bounded::bounded_syscall`], the
/// generic worker-thread + `recv_timeout` primitive, and just adapts it
/// to the WAIT_AFTER call shape: it returns `elapsed_ms` instead of the
/// syscall's `()`, and treats `WorkerLost` as a benign no-op to match
/// the original semantics.
///
/// ## fd lifetime / fd-reuse safety
///
/// `worker_file` is an *owned* `File` (produced by `File::try_clone` at
/// pipeline construction). It is moved into the worker closure so the
/// file description stays alive for exactly as long as the worker thread
/// lives — even if the original `WritebackFile` is closed and the OS
/// reuses its fd number before the worker's syscall returns.
///
/// `fallback_fd` is used only when `worker_file` is `None` (i.e. the
/// `try_clone` at construction failed). In that case the worker captures
/// the raw fd integer, which carries the original fd-reuse risk but is
/// no worse than the pre-fix behaviour.
fn wait_after_with_timeout(
worker_file: Option<File>,
fallback_fd: RawFd,
off: u64,
len: u64,
) -> Option<u64> {
let started = Instant::now();
let result = if let Some(owned) = worker_file {
// Happy path: the closure owns a cloned File that keeps the
// file description alive until the worker drops it.
crate::io::bounded::bounded_syscall(None, WAIT_AFTER_TIMEOUT, move || unsafe {
let fd = owned.as_raw_fd();
libc::sync_file_range(fd, off as i64, len as i64, libc::SYNC_FILE_RANGE_WAIT_AFTER);
// `owned` drops here, closing the cloned fd.
})
} else {
// Fallback: try_clone failed at construction; use the raw fd.
// This carries the pre-fix fd-reuse risk on timeout, but is no
// regression from the original behaviour.
crate::io::bounded::bounded_syscall(None, WAIT_AFTER_TIMEOUT, move || unsafe {
libc::sync_file_range(
fallback_fd,
off as i64,
len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
})
};
match result {
Ok(()) => Some(started.elapsed().as_millis() as u64),
Err(crate::io::bounded::BoundedError::Timeout)
| Err(crate::io::bounded::BoundedError::Halted) => None,
Err(crate::io::bounded::BoundedError::WorkerLost) => {
// Worker thread spawn failed or panicked before sending.
// Treat as a benign success (no syscall ran) rather than
// a degrade trigger — falling through with elapsed_ms=0
// matches the no-op behaviour.
Some(0)
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::NamedTempFile;
/// Helper: build a `WritebackPipeline` over a local tempfile. On
/// every test rig (linux dev box, CI) the tempfile lives on a
/// local FS, so `is_nfs=false` and `skip_wait` returns false until
/// we explicitly mark the pipeline degraded.
fn local_pipeline(chunk_bytes: u64) -> (NamedTempFile, WritebackPipeline) {
let f = NamedTempFile::new().expect("tempfile create");
let pipeline = WritebackPipeline::new(f.as_file(), 0, chunk_bytes);
(f, pipeline)
}
#[test]
fn new_pipeline_starts_active() {
let (_f, p) = local_pipeline(32 * 1024 * 1024);
assert!(!p.is_nfs, "local tempfile must not classify as NFS");
assert!(!p.degraded.load(Ordering::Relaxed));
assert!(!p.skip_wait(), "fresh local pipeline must not skip wait");
}
#[test]
fn degraded_flag_short_circuits_wait() {
let (_f, p) = local_pipeline(32 * 1024 * 1024);
assert!(!p.skip_wait());
p.degraded.store(true, Ordering::Relaxed);
assert!(
p.skip_wait(),
"degraded flag must force the wait+dontneed bypass"
);
}
#[test]
fn record_wait_grows_chunk_on_high_p95() {
let (_f, mut p) = local_pipeline(16 * 1024 * 1024);
// Fill the window with samples above the grow threshold.
for _ in 0..ADAPTIVE_WINDOW {
p.record_wait(ADAPTIVE_GROW_MS + 50);
}
assert!(
p.chunk_bytes > 16 * 1024 * 1024,
"chunk should have grown; got {}",
p.chunk_bytes
);
assert!(p.chunk_bytes <= CHUNK_BYTES_MAX);
}
#[test]
fn record_wait_shrinks_chunk_on_low_p95() {
let (_f, mut p) = local_pipeline(64 * 1024 * 1024);
for _ in 0..ADAPTIVE_WINDOW {
p.record_wait(1); // well under ADAPTIVE_SHRINK_MS
}
assert!(
p.chunk_bytes < 64 * 1024 * 1024,
"chunk should have shrunk; got {}",
p.chunk_bytes
);
assert!(p.chunk_bytes >= CHUNK_BYTES_MIN);
}
#[test]
fn record_wait_no_op_below_window_fill() {
let (_f, mut p) = local_pipeline(16 * 1024 * 1024);
let initial = p.chunk_bytes;
// Only push a few samples; window not full → no adaptation.
for _ in 0..(ADAPTIVE_WINDOW - 1) {
p.record_wait(ADAPTIVE_GROW_MS + 100);
}
assert_eq!(
p.chunk_bytes, initial,
"chunk must not change before window is full"
);
}
#[test]
fn record_wait_clamps_to_chunk_bounds() {
// Grow past the max.
let (_f, mut p) = local_pipeline(CHUNK_BYTES_MAX);
for _ in 0..ADAPTIVE_WINDOW {
p.record_wait(ADAPTIVE_GROW_MS + 1000);
}
assert_eq!(p.chunk_bytes, CHUNK_BYTES_MAX, "must clamp to MAX");
// Shrink past the min.
let (_f, mut p) = local_pipeline(CHUNK_BYTES_MIN);
for _ in 0..ADAPTIVE_WINDOW {
p.record_wait(0);
}
assert_eq!(p.chunk_bytes, CHUNK_BYTES_MIN, "must clamp to MIN");
}
#[test]
fn detect_nfs_local_file_is_false() {
// Local tempfile must not classify as NFS. This locks in the
// consolidation through `crate::platform::fs_type::detect_fd`.
let f = NamedTempFile::new().expect("tempfile create");
use std::os::unix::io::AsRawFd;
assert!(!detect_nfs(f.as_file().as_raw_fd()));
}
#[test]
fn note_progress_below_chunk_is_noop() {
let (_f, mut p) = local_pipeline(32 * 1024 * 1024);
// No-op return before crossing the first chunk boundary.
let before = p.chunk_count;
p.note_progress(1024); // < 32 MiB
assert_eq!(p.chunk_count, before);
assert!(p.pending.is_none());
}
// ── Bug-fix regression tests ────────────────────────────────────────
/// Regression for the fd-reuse / use-after-close fix. Verifies that
/// `WritebackPipeline::new` successfully clones the fd into
/// `wait_file` (i.e. `try_clone` doesn't fail for a normal
/// tempfile) and that `clone_for_worker` returns `Some` — meaning
/// the WAIT_AFTER worker will capture an owned `File` rather than a
/// raw fd integer.
///
/// A deterministic test for the actual fd-reuse race is not clean to
/// write (it would require simultaneously closing the original File
/// and re-opening a new one to steal the fd number while the worker
/// is mid-syscall, which is inherently racy). This test instead pins
/// the structural invariant: on a normal local file, the pipeline
/// holds a valid clone and will give the worker an owned File.
#[test]
fn wait_file_clone_is_present_for_local_tempfile() {
let (_f, p) = local_pipeline(32 * 1024 * 1024);
assert!(
p.wait_file.is_some(),
"wait_file must be Some for a normal local tempfile (try_clone should not fail)"
);
// clone_for_worker must return Some — the worker will get an
// owned File, not fall through to the raw-fd fallback.
let worker_clone = p.clone_for_worker();
assert!(
worker_clone.is_some(),
"clone_for_worker must return Some when wait_file is Some"
);
}
/// Structural: the worker `File` clone returned by `clone_for_worker`
/// is a distinct file descriptor (different fd number) that refers to
/// the same underlying file. Closing the original tempfile must not
/// affect the clone's validity — the OS keeps the file description
/// alive until all file descriptors referring to it are closed.
///
/// We verify "distinct fd number" and "still usable as a raw fd"
/// without actually racing a syscall.
#[test]
fn worker_clone_has_distinct_fd_from_original() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
let pipeline = WritebackPipeline::new(f.as_file(), 0, 32 * 1024 * 1024);
let clone = pipeline
.clone_for_worker()
.expect("clone_for_worker returned None");
let clone_fd = clone.as_raw_fd();
// The clone must have a different fd number — it is a separate
// open file description (dup'd by try_clone).
assert_ne!(
clone_fd, original_fd,
"worker clone must have a distinct fd number from the original"
);
// The clone fd must be valid (non-negative on Unix).
assert!(clone_fd >= 0, "clone fd must be non-negative");
}
}
-129
View File
@@ -1,129 +0,0 @@
//! `WritebackFile` — a `File` wrapper whose reason for existing is the
//! bounded-cache writeback pipeline.
//!
//! Why: large sequential writes (sweep, patch, mux on UHD-scale output)
//! left to the kernel's default writeback policy accumulate hundreds of
//! megabytes of dirty pages and then burst-flush, stalling subsequent
//! writes for seconds at a time. `WritebackFile` drives a continuous
//! [`super::writeback::WritebackPipeline`] that on Linux issues
//! incremental `sync_file_range` + `posix_fadvise(DONTNEED)` calls at
//! 32 MB granularity so dirty pages drain at the same rate they're
//! produced. macOS and Windows fall through to a no-op pipeline — their
//! default cache policies have not been shown to exhibit the same
//! pathology for this access pattern.
//!
//! It implements `Write` and `Seek` so any call site that wrote to a
//! plain `File` through those traits (sweep, patch, mux) can swap in
//! `WritebackFile` without touching the body of the loop. The wrapper
//! also tracks the current file position to feed the pipeline with
//! progress + seek boundaries.
//!
//! See `super::writeback::linux` for the underlying pathology and the
//! strategy.
use std::fs::{File, OpenOptions};
use std::io::{self, Seek, SeekFrom, Write};
use std::path::Path;
use super::writeback::WritebackPipeline;
const WRITEBACK_CHUNK_BYTES: u64 = 32 * 1024 * 1024;
pub(crate) struct WritebackFile {
file: File,
pipeline: WritebackPipeline,
pos: u64,
}
impl WritebackFile {
/// Wrap an open `File`. The current OS file position is queried
/// once so the pipeline starts tracking from wherever the file
/// already is (typically 0 for fresh files; non-zero for resumed
/// or appended files).
pub(crate) fn new(mut file: File) -> io::Result<Self> {
let pos = file.stream_position()?;
let pipeline = WritebackPipeline::new(&file, pos, WRITEBACK_CHUNK_BYTES);
Ok(Self {
file,
pipeline,
pos,
})
}
/// Create a new file at `path` (truncating any existing contents)
/// and wrap it. Convenience for the common
/// `File::create(path)` + `WritebackFile::new(file)` pair so callers
/// don't have to assemble a `File` first.
pub(crate) fn create(path: &Path) -> io::Result<Self> {
let file = File::create(path)?;
Self::new(file)
}
/// Open an existing file at `path` for writing (no truncation) and
/// wrap it. Mirrors `File::open` semantics for the writable case
/// — used by patch / resume paths that mutate an existing ISO in
/// place.
pub(crate) fn open(path: &Path) -> io::Result<Self> {
let file = OpenOptions::new().write(true).open(path)?;
Self::new(file)
}
/// Drain in-flight writeback then issue a full fsync. Use this in
/// place of `File::sync_all`.
pub(crate) fn sync_all(&mut self) -> io::Result<()> {
self.pipeline.finalize();
self.file.sync_all()
}
}
impl Write for WritebackFile {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let n = self.file.write(buf)?;
self.pos += n as u64;
self.pipeline.note_progress(self.pos);
Ok(n)
}
fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {
self.file.write_all(buf)?;
self.pos += buf.len() as u64;
self.pipeline.note_progress(self.pos);
Ok(())
}
fn flush(&mut self) -> io::Result<()> {
self.file.flush()
}
}
impl Seek for WritebackFile {
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
let p = self.file.seek(from)?;
// Only treat seeks that actually move the position as
// boundaries — sweep does a redundant `seek(Current(pos))`
// before every write, and we don't want that to drain the
// pipeline on every iteration.
if p != self.pos {
self.pipeline.handle_seek(p);
self.pos = p;
}
Ok(p)
}
}
impl Drop for WritebackFile {
fn drop(&mut self) {
// Run the pipeline's tail finalize so the last in-flight chunk
// gets its `WAIT_AFTER` + `posix_fadvise(DONTNEED)`. Without
// this, callers that drop a `WritebackFile` without calling
// `sync_all` (panic, early-return, idiomatic `let _ = w;`)
// leave the trailing chunk in cache; the kernel still flushes
// on close, but the bounded-cache invariant fails at the tail.
// We deliberately do *not* call `self.file.sync_all()` here —
// close already triggers a flush, and an `fsync` from `Drop`
// would silently swallow its `io::Error` anyway. `finalize` is
// idempotent so an explicit `sync_all` followed by drop is
// still safe.
self.pipeline.finalize();
}
}
+145
View File
@@ -0,0 +1,145 @@
//! Linux platform impl for [`super::WritebackFile`].
//!
//! - `preallocate`: `fallocate(FALLOC_FL_KEEP_SIZE)` — reserve extents
//! without growing the reported file size. Reduces extent
//! fragmentation on large sequential writes (mux output on NFS in
//! particular).
//! - `durable_sync`: `fsync` wrapped in
//! [`crate::io::bounded::bounded_syscall`] with a 60 s deadline so a
//! wedged NFS server can't trap the calling thread indefinitely.
use std::fs::File;
use std::io;
use std::os::unix::io::AsRawFd;
use std::time::Duration;
/// Pre-reserve extents for `size_bytes` of upcoming sequential writes.
/// Best-effort: a non-zero rc is logged but not propagated, since the
/// caller would just continue with the unreserved file anyway.
pub(super) fn preallocate(file: &File, size_bytes: u64) {
// FALLOC_FL_KEEP_SIZE = 0x01 — keep the reported file size at 0
// (writes grow it normally) while still pre-reserving the extents.
// Clamp to the signed `off_t` range; an unchecked `as i64` cast
// would wrap a >= 2^63 size to a negative length (EINVAL no-op).
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX);
let rc = unsafe { libc::fallocate(file.as_raw_fd(), libc::FALLOC_FL_KEEP_SIZE, 0, len) };
tracing::debug!(
target: "mux",
"WritebackFile fallocate size_hint={size_bytes} rc={rc} ok={}",
rc == 0
);
}
/// Run `fsync` on `file` with a 60 s deadline. On timeout — and
/// likewise on halt or a lost worker — we log and return `Ok(())`: the
/// kernel will still flush on close, so the data is best-effort durable.
/// The alternative (trap the thread for the rest of the rip, or return
/// an error that aborts an otherwise-complete mux) is worse, so all
/// three fallbacks return `Ok(())`. `Ok(())` from these paths is NOT a
/// durability barrier — the durable flush did not complete; only the
/// hang is bounded.
///
/// ## fd-reuse safety
///
/// The `fsync` runs on a bounded worker thread that may be leaked on
/// timeout. To avoid the leaked worker's syscall hitting a recycled fd
/// number after the original `File` is closed, we `try_clone` an owned
/// `File` and move it into the closure. The clone keeps the underlying
/// file description alive for as long as the worker thread lives.
/// On `try_clone` failure (rare) we fall back to the raw fd integer —
/// no worse than the previous behaviour.
pub(super) fn durable_sync(file: &File) -> io::Result<()> {
// Clone so a leaked worker thread retains a valid fd even after the
// original File is closed and its fd number is reused.
let owned = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
let fd = file.as_raw_fd();
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fd={fd}: try_clone failed ({e}), fsync worker will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
let fallback_fd = file.as_raw_fd();
match crate::io::bounded::bounded_syscall(
None,
Duration::from_secs(60),
move || -> io::Result<()> {
let fd = owned.as_ref().map(|f| f.as_raw_fd()).unwrap_or(fallback_fd);
let rc = unsafe { libc::fsync(fd) };
// `owned` (if Some) drops here, releasing the cloned fd.
if rc == 0 {
Ok(())
} else {
Err(io::Error::last_os_error())
}
},
) {
Ok(inner) => inner,
Err(crate::io::bounded::BoundedError::Timeout) => {
tracing::error!(
target: "mux",
"WritebackFile::sync_all fsync timed out after 60s; kernel will flush on close (best-effort)"
);
Ok(())
}
Err(crate::io::bounded::BoundedError::Halted) => {
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fsync skipped (halt requested); data not durably flushed, kernel will flush on close"
);
Ok(())
}
Err(crate::io::bounded::BoundedError::WorkerLost) => {
tracing::error!(
target: "mux",
"WritebackFile::sync_all fsync worker lost before completion; data not durably flushed, kernel will flush on close"
);
Ok(())
}
}
}
#[cfg(test)]
#[cfg(target_os = "linux")]
mod tests {
use super::*;
use tempfile::NamedTempFile;
/// Regression for the fd-reuse / use-after-close fix in `durable_sync`.
///
/// Verifies the structural invariant: `try_clone` succeeds for a normal
/// local tempfile, and the cloned `File` has a distinct fd number from
/// the original. This pins the property that a leaked fsync worker thread
/// captures an owned `File` (and thus keeps the file description alive)
/// rather than a bare fd integer that can be reused after the original
/// `File` closes.
///
/// The actual fd-reuse race is non-deterministic and not cleanly
/// testable without coordinating a simultaneous close + re-open on
/// another thread. A structural test is the accepted substitute.
#[test]
fn durable_sync_worker_uses_owned_clone_with_distinct_fd() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
// try_clone must succeed for a normal local file.
let owned = f
.as_file()
.try_clone()
.expect("try_clone must succeed for a local tempfile");
let clone_fd = owned.as_raw_fd();
// The clone must be a distinct fd (dup'd, not aliased).
assert_ne!(
clone_fd, original_fd,
"owned clone must have a distinct fd number — not an alias of the original"
);
assert!(clone_fd >= 0, "clone fd must be a valid non-negative fd");
// durable_sync must complete without error on the local tempfile.
durable_sync(f.as_file()).expect("durable_sync must return Ok on a local tempfile");
}
}
+159
View File
@@ -0,0 +1,159 @@
//! macOS platform impl for [`super::WritebackFile`].
//!
//! - `preallocate`: `fcntl(F_PREALLOCATE)` — macOS's fallocate-equiv.
//! First attempt requests `F_ALLOCATECONTIG | F_ALLOCATEALL` (prefer a
//! contiguous run but accept scattered extents to satisfy the full
//! length), falling back to `F_ALLOCATEALL` alone on failure.
//! `F_PREALLOCATE` never advances EOF regardless of the flags — only
//! `ftruncate`/writes grow the file — so the reported file size is
//! unchanged; `F_ALLOCATEALL` governs the contiguity fallback, not size.
//! - `durable_sync`: `fcntl(F_FULLFSYNC)` wrapped in
//! [`crate::io::bounded::bounded_syscall`] with a 60 s deadline.
//! F_FULLFSYNC is HFS+/APFS's true-fsync (flushes the disk's own
//! write cache) — what `fsync` should have been on macOS. Falls back
//! to plain `fsync` if F_FULLFSYNC returns ENOTSUP.
use std::fs::File;
use std::io;
use std::os::unix::io::AsRawFd;
use std::time::Duration;
use crate::io::platform_macos::{
F_ALLOCATEALL, F_ALLOCATECONTIG, F_PEOFPOSMODE, F_PREALLOCATE, Fstore,
};
/// `fcntl(F_FULLFSYNC)` opcode. Documented in `man 2 fcntl` on macOS;
/// not in the `libc` crate as a named constant.
const F_FULLFSYNC: libc::c_int = 51;
pub(super) fn preallocate(file: &File, size_bytes: u64) {
// Clamp to the signed `off_t` range; an unchecked `as off_t` cast
// would wrap a >= 2^63 size to a negative length.
let len = i64::try_from(size_bytes).unwrap_or(i64::MAX) as libc::off_t;
let mut fst = Fstore {
fst_flags: F_ALLOCATECONTIG | F_ALLOCATEALL,
fst_posmode: F_PEOFPOSMODE,
fst_offset: 0,
fst_length: len,
fst_bytesalloc: 0,
};
// First attempt: contiguous.
let mut rc = unsafe { libc::fcntl(file.as_raw_fd(), F_PREALLOCATE, &mut fst) };
if rc == -1 {
// Fall back: drop the contiguous hint, allow scattered extents.
fst.fst_flags = F_ALLOCATEALL;
rc = unsafe { libc::fcntl(file.as_raw_fd(), F_PREALLOCATE, &mut fst) };
}
tracing::debug!(
target: "mux",
"WritebackFile F_PREALLOCATE size_hint={size_bytes} rc={rc} bytes_allocated={} ok={}",
fst.fst_bytesalloc,
rc != -1
);
}
/// ## fd-reuse safety
///
/// The F_FULLFSYNC / fsync runs on a bounded worker thread that may be
/// leaked on timeout. To avoid the leaked worker's syscall hitting a
/// recycled fd number after the original `File` is closed, we
/// `try_clone` an owned `File` and move it into the closure. The clone
/// keeps the underlying file description alive for as long as the worker
/// thread lives. On `try_clone` failure (rare) we fall back to the raw
/// fd integer — no worse than the previous behaviour.
pub(super) fn durable_sync(file: &File) -> io::Result<()> {
// Clone so a leaked worker thread retains a valid fd even after the
// original File is closed and its fd number is reused.
let owned = match file.try_clone() {
Ok(f) => Some(f),
Err(e) => {
let fd = file.as_raw_fd();
tracing::warn!(
target: "mux",
"WritebackFile::sync_all fd={fd}: try_clone failed ({e}), F_FULLFSYNC worker will use raw fd (fd-reuse risk on timeout)"
);
None
}
};
let fallback_fd = file.as_raw_fd();
match crate::io::bounded::bounded_syscall(
None,
Duration::from_secs(60),
move || -> io::Result<()> {
let fd = owned.as_ref().map(|f| f.as_raw_fd()).unwrap_or(fallback_fd);
// Try F_FULLFSYNC first. If it isn't supported on this
// filesystem (older HFS, some network mounts) fall back to
// plain fsync — better than nothing.
let rc = unsafe { libc::fcntl(fd, F_FULLFSYNC, 0) };
if rc == 0 {
// `owned` (if Some) drops here, releasing the cloned fd.
return Ok(());
}
let err = io::Error::last_os_error();
if err.raw_os_error() == Some(libc::ENOTSUP) {
let rc = unsafe { libc::fsync(fd) };
// `owned` drops here.
if rc == 0 {
Ok(())
} else {
Err(io::Error::last_os_error())
}
} else {
// `owned` drops here.
Err(err)
}
},
) {
Ok(inner) => inner,
Err(crate::io::bounded::BoundedError::Timeout) => {
tracing::error!(
target: "mux",
"WritebackFile::sync_all F_FULLFSYNC timed out after 60s; kernel will flush on close (best-effort)"
);
Ok(())
}
Err(crate::io::bounded::BoundedError::Halted) => Ok(()),
Err(crate::io::bounded::BoundedError::WorkerLost) => Ok(()),
}
}
#[cfg(test)]
#[cfg(target_os = "macos")]
mod tests {
use super::*;
use tempfile::NamedTempFile;
/// Regression for the fd-reuse / use-after-close fix in `durable_sync`.
///
/// Verifies the structural invariant: `try_clone` succeeds for a normal
/// local tempfile, and the cloned `File` has a distinct fd number from
/// the original. This pins the property that a leaked F_FULLFSYNC/fsync
/// worker thread captures an owned `File` (keeping the file description
/// alive) rather than a bare fd integer that can be reused after the
/// original `File` closes.
///
/// The actual fd-reuse race is non-deterministic; a structural test is
/// the accepted substitute.
#[test]
fn durable_sync_worker_uses_owned_clone_with_distinct_fd() {
let f = NamedTempFile::new().expect("tempfile create");
let original_fd = f.as_file().as_raw_fd();
// try_clone must succeed for a normal local file.
let owned = f
.as_file()
.try_clone()
.expect("try_clone must succeed for a local tempfile");
let clone_fd = owned.as_raw_fd();
// The clone must be a distinct fd (dup'd, not aliased).
assert_ne!(
clone_fd, original_fd,
"owned clone must have a distinct fd number — not an alias of the original"
);
assert!(clone_fd >= 0, "clone fd must be a valid non-negative fd");
// durable_sync must complete without error on the local tempfile.
durable_sync(f.as_file()).expect("durable_sync must return Ok on a local tempfile");
}
}

Some files were not shown because too many files have changed in this diff Show More