Every other repo's CI now runs clippy with --all-targets. libfreemkv,
the crate the other seven build against and the one held up as the
reference workflow, was the last one still linting the library only — so
its ~3,000 tests, by far the largest body of test code in the project,
had never been linted at all. Turning the flag on surfaced 74 findings.
Most were mechanical and applied with clippy --fix. The rest, by hand:
- Four discarded Results in decrypt.rs. css::descramble_region returns a
Result and four CSS tests threw it away, so a descramble that FAILED
would have surfaced as a confusing buffer-comparison mismatch instead
of the actual error. They expect() now.
- A dead `kp` field on the PlantedWalk fixture. The test deliberately
asserts Kp as the explicit AES-G3(dk, 1) relation from [C] §3.2.4
rather than against a stored value — its doc comment says so — which
makes the field not just unused but a trap: the obvious "fix" of
asserting against it would quietly weaken the test to comparing the
fixture with itself. Removed.
- Two hand-rolled ICB counters in the HD-DVD fixtures, a needless mut,
three vec!s that only ever needed arrays, a filter_map whose every arm
was Some, and a Vec::new()+push chain.
- Doc list indentation in mkv.rs and mp4/read.rs, which was mis-rendering
in the generated docs.
- A five-[u8; 16]-tuple return type named FourLevelParts.
Three lints are allowed at the specific sites, with reasons, because
they are wrong for this domain: the underscores in the bitstream-header
literals mark BITFIELD boundaries, not digit groups, so regrouping them
uniformly would satisfy the lint by destroying the only thing they
encode; and in three table-validation loops the loop variable is the
domain value under test (a DTS SFREQ code, an AMODE value, a palette
entry number), which is what the assertion messages name.
I broke DVD ripping earlier today and the real-media acceptance gate
caught it on its first full run. Greenland.iso failed with E7013
"Decryption failed"; reverting only this change made it rip clean in 8
seconds. That is a regression I introduced, not a pre-existing defect.
WHAT I GOT WRONG.
Round 9's crypto lens reported that descramble_region "descrambles with
a key it just proved wrong" when the crib check rejects the cached key
and the re-crack also fails. I agreed, and made it Error::DecryptFailed
to match the AACS path, on the reasoning that CSS has no external key
source so a failed crack on a readable sector should never happen.
The premise was wrong. `attack_crib` is a HEURISTIC, not a proof: it
finds a periodic run in the unscrambled header and predicts the run
continues past 0x80. When that prediction does not hold, the crib
reports a mismatch even for a CORRECT key — and the re-crack then fails
BECAUSE the crib was never valid. So crib mismatch plus crack failure is
the signature of a crib false positive, not of a stale key. The cached
key is not proven wrong; it remains the best available evidence, and on
a real DVD it is very probably right. Real discs hit this constantly.
The deeper error was treating "no key" as one thing across schemes. An
AACS unit key either opens a unit or it does not — the Verify-Media-Key
relation decides it, and a wrong key is provable. A CSS title key is
recovered from the data itself by an attack whose success varies sector
by sector, so "the crack failed here" says something about THIS SECTOR's
plaintext, not about the key. Unifying the policy was right for the
schemes that can prove a key wrong. CSS cannot, and I folded it in
anyway.
decrypt_span keeps its shape and the cross-scheme test keeps its two
AACS arms, with CSS now explicitly excluded and the reason stated.
Three tests asserted the wrong behaviour and are corrected, including
one I rewrote earlier today to pin exactly this. Every one of them
passed the whole time the code was broken — because none of them had
ever seen a real disc.
The lesson is the one I kept stating and then did not act on: 3,013 unit
tests, ~400 mutants killed and nine audit rounds did not catch this, and
one acceptance run did. Synthetic media cannot reproduce what a real
disc does.
There were TWO top-level decrypt paths: decrypt_sectors_impl for CSS and
clear media, whose AACS arm was a bare `return Err` stub, and a wholly
separate decrypt_sectors_mapped for AACS. Each scheme therefore decided
its own answer to "there is no key for these bytes", and nothing held
them to the same one.
They drifted, in opposite directions, within a single release:
css::descramble_region descrambled with a key the sector's own crib had
just proven stale — garbage behind an intact clear header, reported Ok.
the mapped path returned early for any LBA outside every range, before
ever asking whether those bytes were ciphertext, so an unkeyable
encrypted unit passed through and extract counted it as good.
Both were fixed individually earlier today. This removes the shape that
allowed them.
decrypt_span is now the single orchestrator: it owns the loop, the
refusal, and the loss count, and each scheme supplies only what
genuinely differs. apply_aacs_map is a scheme step that reports what it
could not open; it no longer decides what that means. The public
wrappers (decrypt_sectors, _in_content, _mapped) are unchanged in
signature and all funnel through it.
Adding a scheme now means adding an arm here, which means answering the
refusal question. That is the point.
The new test asserts ONE verdict across all three schemes — AACS with no
map, AACS with an encrypted unit outside every range, and CSS whose
re-crack failed — plus that clear media is NOT a refusal. A per-scheme
test cannot hold this: each would keep passing while the two disagreed.
Flipping the AACS arm back to pass-through reds it.
Also removes the last of the tracing-capture scaffolding. Serialising
those captures crate-wide did not fix the 1-in-10 flake, and asserting
the predicates directly made the helper, both capture subscribers and
an unrelated dead OrderSink unused. Deleted rather than left behind.
decrypt_sectors_mapped returned early for any LBA no map range covers,
on the reasoning that unmapped means clear filesystem or nav. "The map
has no key here" and "there is nothing to decrypt here" are different
statements, and only the second makes passing the unit through correct.
On a multi-CPS disc an orphan clip — referenced by no playlist, so in no
title extent and therefore in no range — hits the first and was treated
as the second. The early return fired BEFORE the aacs_unit_encrypted
gate below it, so nothing ever asked whether those bytes were
ciphertext. extract_tree then counted them as bytes_good, dropped the
.partial suffix, set complete = true and exited 0: a scrambled file on
disk with a clean bill of health.
FileResult's own doc already stated the intended contract — "unreadable
sectors AND undecryptable units both land here (extract fails a bad
decrypt loud)". The code did not implement it on this path.
The fix is one line of symmetry. The split-unit branch immediately above
already makes exactly this distinction, checking aacs_unit_seed_encrypted
before refusing. This branch did not, so the same question got two
answers eight lines apart — the duplication shape this release keeps
finding.
Deliberately NOT changed: the decrypt decision on an orphan is still to
refuse rather than guess. Blind trial-decrypt is what the keymap-only
model exists to remove, and extract has no CPS/forensic fetch source.
The defect was never that we declined to key it; it was that declining
looked like success.
The test asserts BOTH directions — a clear out-of-range unit must still
pass through byte-identical, because that is the ordinary whole-disc
read and breaking it would trade one silent defect for a loud one.
descramble_region descrambled with the key a sector's own crib had just
proven stale, whenever the re-crack from that sector also failed. The
clear header is not scrambled, so it survives intact: the sector still
opens with a valid pack start and passes every structural check the PS
demuxer applies. Only the payload is corrupted — exactly where nothing
looks. Ok(0) dropped, exit 0.
CSS has no external key source. The title key comes only from cracking
the data, so on a READABLE sector "no key" is not a missing input, it is
recovery failing on bytes we can see. That should never happen, and when
it does the answer is not to emit something.
Now Error::DecryptFailed — the same verdict the AACS path already gives
for a unit no held key opens. Both alternatives to failing are bad data
reported as success: descrambled with a rejected key it is garbage
behind a valid header, and passed through untouched it is ciphertext
where plaintext is meant to be.
WHY IT WAS POSSIBLE, which matters more than the fix:
There is no single place that owns "what do we do when there is no key".
decrypt_sectors_impl looks like the central dispatch, but its AACS arm
is a `return Err` stub — AACS decrypts entirely through
decrypt_sectors_mapped, a separate top-level path. So CSS decided its
own policy inside css/, AACS decided in decrypt.rs and mux/resolve.rs,
and nothing held them to the same answer. The asymmetry was not an
oversight; it was structurally permitted.
How a disc decrypts is one process — resolve a key for this data, apply
it, refuse if it cannot be proven. Only the resolve-and-apply step is
scheme-specific. Filed as a task: the policy belongs in one orchestrator
with the schemes supplying only what genuinely differs.
Two tests changed rather than added, both of which pinned the old
behaviour: the unit test asserted the sector was descrambled, and the
integration test asserted the scramble flag was cleared, which is what
descrambling-with-any-key does. Neither established that the result was
CORRECT — the fourth bad-test shape.
Five surviving mutants in AacsKeyMap::read_plan, all in the arithmetic
that decides which half of a forensic segment this disc's key opens.
The existing coverage used a forensic range starting exactly on the
extent's first unit. Under that shape several wrong formulas agree with
the right one by arithmetic accident: (lba + range_start) / us and
(lba - range_start) * us both produce the correct kept set.
Real ranges are not shaped like that. A range start comes from a source
packet number — start_spn * 192 through clip_byte_to_lba in
mux/resolve.rs — and 192-byte packets bear no relation to the 3-sector
aligned unit, so range_start % 3 is whatever the disc says.
Getting the parity wrong does not crash. It reads and decrypts the
ALTERNATE variant's half: the units this key does not open decrypt to
garbage, the units it does open are skipped. AACS 2.1 forensic marking
is precisely what makes the two halves differ, so the failure is silent
— a full-length rip carrying the wrong variant.
Two fixtures are needed because no single one kills both: an unaligned
range start with the extent beginning on it inverts the halves under
the + form, and an extent offset one unit-remainder from the range
start inverts them under the * form.
Also pinned the short-tail guard. is for a remnant
SMALLER than a unit — bytes with no following unit to desync. Widened
to <=, the last whole unit of every extent bypasses the phase gate, so
a forensic segment ending at an extent boundary contributes one
alternate-variant unit to the rip.
Fifteen surviving mutants killed, from the highest-risk class: functions a
mutant could replace wholesale with a constant while all 2,555 tests
passed. None of the code was wrong. In every case a test was absent, which
is why eight rounds of reading never found any of them.
The one that generalises is in sector/mod.rs. Its existing test READS as
covering `read_sectors` on the `&mut dyn SectorSource` forwarding impl —
it takes a `&mut dyn`, calls the method, checks the spy. But the receiver
auto-derefs and dispatches through the vtable straight to the spy, so the
forwarding body is never entered. An earlier round hit this exact trap on
`set_unit_base` and fixed it with a generic helper; the read path kept the
test that looked right. Verified by stubbing the forwarding impl to Ok(0):
the new test fails, the old one passes. That makes a tenth distinct shape
of bad test in this audit, and the mutation list is how to find the rest —
any forwarding-impl method in it has the same problem.
decrypt.rs's two existing gate tests assert only `dropped == 0`, which is
precisely what the `Ok(0)` mutant returns; one asserts nothing else at all.
A wrapper that decrypts nothing therefore looked correct while the caller
muxed scrambled MPEG. Now pinned by descrambling a real CSS sector and
comparing against the plaintext it was built from — not against a
re-derived descramble, which would only assert the code agrees with
itself.
css/mod.rs's `is_scrambled_uncracked` turns out to have no production
callers at all; the enum is matched directly. Its three tests all assert
only the true direction, which is exactly why the `-> true` mutant
survived. It is public API, so a consumer routing on it would, under that
mutant, refuse to rip every clear DVD.
aacs/inf.rs's MKB drive read had no test whatsoever. Now pinned
byte-for-byte across multi-pack concatenation, the single-pack case, a
genuinely empty response, and error propagation — an unreadable MKB must
surface as an error, not as an empty one.
aacs/derive.rs's nine mutants are killed with planted MKBs built by
inverting the AACS relations, so no real key material is involved. The
assertions land on the derived Media Key rather than the intermediate
positions: a recovered position that does not actually walk to the planted
key is no better than None. A fixture-guard test asserts the planted MKB
parses, since an unparseable one would make every `-> None` body look
right.
2570 lib tests, debug and release.
Twenty-six confirmed findings from the fourth audit round, landed as one
cluster because they were found by agents working over disjoint file sets.
The one worth calling out is a pair of AACS tests that could not fail.
Both asserted CBC behaviour against a hand-rolled expectation that
happened to be IV-independent, so replacing AACS_IV with sixteen zero
bytes left them passing — they were pinning the code's own arithmetic,
not the published constant. Replaced with a literal witness of the
published IV plus the NIST SP 800-38A F.2.2 CBC-AES128 vector, and
verified the other way round: zeroing AACS_IV now fails three tests.
The rest are allocation and correctness work on hot paths: the Annex-B
writer in demux_sink allocated and freed a whole-frame Vec per frame,
which for a UHD title is ~200,000 allocations over the mmap threshold
plus the page faults to first-touch each one; it now reuses a buffer on
the writer, and still takes the NAL prefix width from the configuration
record rather than assuming four.
Six findings whose real fix lives in a consumer crate are recorded for
re-filing rather than patched here.
The Windows UI needs current winsafe, whose real minimum is 1.89 (its manifest
under-declares 1.87 while it uses NonNull::from_ref). Rather than stop at the
minimum, this goes to current stable and fixes what that costs.
The counter-intuitive result: 1.97 is CHEAPER than 1.89. libfreemkv had 54
clippy errors at 1.89 and 6 at 1.97, because clippy tightened the noisy
collapsible_if lint in between. Stopping at the minimum would have been the most
expensive choice available.
Roughly 47 lints across the eight repos, the large majority auto-fixed:
libfreemkv 6, freemkv-engine 14, bdemu 8, freemkv-keysources 7, autorip 6,
freemkv-unlock 3, freemkv-i18n 3. The hand-fixed ones are a descending sort to
sort_by_key(Reverse), four manual checked-division sites, a loop counter replaced
by enumerate, and a loop whose first let-else became a while-let.
Worth recording for whoever bumps next: clippy is MSRV-AWARE. Those 54 lints only
appear once the crate DECLARES 1.89 or later, because let-chains become
available. A bare `cargo +1.89 clippy` against a manifest still pinned at 1.87
reports clean and is meaningless — gate with the real precommit script, which is
also the only thing that covers build scripts.
The pin still sits below the Mac default, so it keeps doing its job: catching
lint drift locally before CI sees it.
ddts CoreSize wrapped to zero on a maximum-size core. core_size is FSIZE + 1 and
FSIZE is itself 14 bits, so the maximum is 16384 — one past what the 14-bit
CoreSize field holds — and push()'s mask turned that into 0, declaring an empty
core frame. Clamped to 16383 instead: one byte short beats telling a decoder
there is no core. Proven red first (the field read back as 0).
ddts avg/max bitrate under-declared every non-integral frame rate. It computed
sample_rate / frame_samples first, so a 512-sample core at 48 kHz truncated
93.75 frames/s to 93. Multiplying before dividing, with round-to-nearest, keeps
the precision.
set_decrypt_threads skipped the pool swap on a poisoned lock while
DECRYPT_THREADS had already been updated, so the new thread count was reported
as taking effect while the stale pool kept serving. decrypt_pool() deliberately
recovers from poisoning for exactly this reason; the setter now does the same.
The pool Arc is immutable once stored, so a prior panic cannot have left it
half-written.
The CoreSize test decodes the value back out of the emitted box rather than
restating the clamp — a first draft asserted the clamp arithmetic against
itself, which would have passed against the unfixed writer.
Two defects in the encrypt_unit promoted to public API last round, both found by
round 2 auditing that new code.
It returned silently without encrypting when the slice was shorter than
ALIGNED_UNIT_LEN. Its own contract requires the caller to set the container's
encrypted flag BEFORE calling — the header is the key seed — so a silent no-op
leaves a unit advertised as encrypted while still carrying plaintext, with
nothing for an authoring caller to check. It now returns bool and is
#[must_use], so ignoring the refusal is a compile-time warning; every call site
was updated to assert on it.
bool rather than Result deliberately: a wrong buffer length is a programming
error at a library boundary, not a disc condition, and a new Error variant would
mean a new numeric code plus its rendering in another repo.
It also drove CBC from the single-block aes_ecb_encrypt, rebuilding the AES key
schedule for each of the 383 blocks in a unit — an order of magnitude slower
than its inverse, which expands the key once via aes_cbc_decrypt. The missing
counterpart aes_cbc_encrypt now exists alongside it, and encrypt_unit calls it,
so the two directions are symmetric in structure as well as in result. For an
authoring caller encrypting a 90 GB image that removes ~5.6 billion redundant
key expansions.
New test pins the boundary: ALIGNED_UNIT_LEN - 1 returns false and leaves the
buffer byte-identical, ALIGNED_UNIT_LEN succeeds. The existing round-trip and
padding-asymmetry tests still pass, so the CBC rewrite is provably the same
transform.
encrypt_unit becomes public library API rather than a #[cfg(test)] helper.
Authoring an encrypted disc image is a legitimate use of this crate, and
the capability was already written four times over: a pub(crate) test-only
copy in aacs/content.rs plus three hand-rolled duplicates in decrypt.rs,
sector/decrypting.rs and disc/extract.rs. All four now call one function,
removing ~110 lines of duplicated cipher code that could drift from
decrypt_unit independently.
It mirrors decrypt_unit's purity contract: crypto only, no encrypted-flag
handling, because where that flag lives is container-specific (CPI bits in
byte 0 for BD-TS, elsewhere for HD-DVD-PS). Callers set the flag BEFORE
encrypting — bytes 0..16 are the key seed left in plaintext, so touching a
header byte afterwards changes the key a decryptor derives. That footgun is
documented at the function and at every call site.
Two tests pin it: an exact round trip through both directions, and the one
place the pair is deliberately asymmetric — decrypt_unit restores
all-zero-on-disc packets to zero, and the test proves an all-zero plaintext
packet enciphers to non-zero bytes so it is never mistaken for padding.
That asymmetry was previously only prose.
Two decrypt tests were also weaker than their own names:
* aacs_clear_trailing_partial_passes_through asserted only is_ok(), so a
mutant corrupting the clear partial while returning Ok passed. It now
snapshots the buffer and asserts byte equality, matching the
none_keys_is_noop pattern already in the file.
* aacs_decorator_decrypts_encrypted_unit_via_map checked only that 0x47
reappeared at the 192-byte stride, leaving corruption in the other 6112
bytes undetected. The plaintext is fully known, so it now asserts
byte-exact recovery against it.
Both were verified red first by mutating the production path.
The sweep/patch recovery strategy, mapfile, retry-decision state machine,
section-recover, and damage classification move out of libfreemkv into the
new freemkv-engine crate. libfreemkv keeps the raw single-shot read and
SCSI-fact translation (SenseFamily stays in scsi).
- Delete disc/{sweep,patch,mapfile,read_error,section_recover}.rs, the
Disc::copy/sweep/patch methods, the Copy/Sweep/Patch option+result types,
classify_damage/DamageSeverity, progress_snapshot_from_mapfile, and the
three recovery integration tests.
- Trim public surface the recovery deletion orphaned: delete the dead
READ_PIPELINE_DEPTH const, the write-side SectorSink/FileSectorSink (no
consumer), and the DriveSpeed enum (its one live use — set max drive
speed — becomes Drive::SPEED_MAX_KBPS). Make mapfile_path_for,
decrypt_sectors_mapped pub(crate); gate NoopEvents to test.
- Version 1.6.0.
Multi-round audit of the decrypt/AACS/mux-codec refactor. Fixes, in
descending severity:
- mux/mp4/read.rs: bound untrusted-input allocations. `sample_budget`
now also capped by file_len (a fixed-size stsz claiming count=u32::MAX
can't inflate the Vec<SampleRef> past the file's own size); trak scan
capped at MAX_TRACKS matches; find_box() takes only the first match
(cap=1) instead of materializing every match. Removes dead find_boxes
wrapper.
- disc/mod.rs: merge_content_key_ranges now UNIONS same-key overlapping
ranges (coverage-preserving) instead of dropping the non-overlapping
tail, which silently left encrypted LBAs uncovered -> ciphertext
passthrough in the whole-disc sweep/patch map. Different-key overlap
(malformed) still dropped to keep the set disjoint.
- sector/decrypting.rs: remove dead unit_key_idx field + with_unit_key_idx
setter (vestigial from the pre-keymap trial-decrypt design; AACS is
map-only now). Fix stale docs.
- decrypt.rs / resolve.rs / error.rs / extract.rs: doc/comment drift from
the refactor (AacsKeyMap positive-map semantics, resolve_mux_key_map doc
reattachment, decrypt_sectors_in_content legacy-alias, E_MP4_INVALID
meaning, multi-CPS orphan by-design note).
Test coverage (all mutation-verified real):
- DTS NeedMore force-flush buffer bound; FLAC/MPEG-audio PTS carry-forward;
mp4 mdhd timescale=0 divide-by-zero guard, MAX_TRACKS cap, sample-count
file_len bound, MAX_ALLOC_BYTES cap under inflated file_len.
- resolve_fmts_key_map: extracted filter_addressable_segments,
resolve_tie_phase, fill_base_key_gaps as pure behavior-preserving
helpers, each unit-tested (segment filter, phase-tie arms, gap-fill
gaplessness over every extent).
- AacsKeyMap now derives its distinct key-index set once at construction
(from_ranges_phased) instead of re-allocating/sorting it on every
decrypt batch; key_indices() returns the cached slice.
- Extract the whole-disc content-map range merge out of
resolve_content_key_map into merge_content_key_ranges and cover it:
sort/disjoint, shared-clip dedup, overlap drop, adjacent-kept.
- Repoint three tautological AACS tests (which passed only via the
unconditional-Err AACS arm of decrypt_sectors) at the real shipping
path decrypt_sectors_mapped: out-of-range mapped key index, empty pool
vs a non-empty map, and a scrambled encrypted trailing partial (the new
guard) — plus a clear-trailing-partial pass-through and an explicit
fail-loud safety-net test for AACS reaching the unmapped wrapper.
- Add the missing end-to-end AACS round-trip through the decorator +
key map (aacs_decorator_decrypts_encrypted_unit_via_map) and a
mapless-AACS-fails-loud decorator test — the exact class of bug the
TrueHD probe shipped. Fills the // TODO: AACS round-trip test gap.
- Delete the now-dead recovery test fixtures (encrypt_aacs_unit_bad,
AnyLbaUnit) left after the reactive key-fetch path was removed; keep
encrypt_aacs_unit + FixedUnit, now used by the round-trip test.
- resolve_fmts_key_map: filter segments to those addressable within THIS
title's extents; a title with no forensic content (menu/extras playlist,
or a different clip) returns Ok(None) and takes the base Unit-Key/CPS
path instead of hard-failing FmtsKeyMissing. Previously the first
non-forensic title aborted the entire whole-disc sweep
(resolve_content_key_map iterates every title) and blocked muxing any
non-main title.
- FMTS phase probe: an even/odd is_clean tie now only fails loud when
BOTH halves are 0 (no clean decrypt). A both-clean tie is source-zero
padding (is_clean is true for any key on all-zero content) — the key is
valid, default Even, never abort the rip on a padding-heavy sample.
- extract_tree: multi-CPS discs now build the exact per-CPS content map
(resolve_content_key_map) instead of a blanket key-0 map that silently
mis-decrypted every secondary-CPS file into garbage. Single-CPS keeps
the blanket key-0 map (one key opens every unit, incl. orphan clips).
- decrypt_sectors_mapped: a trailing partial unit that is inside a mapped
range AND flagged encrypted in its clear seed now fails loud (a CBC
fragment split across a boundary can't be decrypted) instead of being
emitted as clear. New aacs_unit_seed_encrypted reads the flag on a
partial.
- Correct the stale decrypt_sectors doc (AACS arm now always errors;
AACS decrypts only via decrypt_sectors_mapped).
Every AACS decrypt now goes through the resolved key map (decrypt_sectors_
mapped): the map keys each content unit up front and a missing key fails at
resolve time. The old trial-decrypt path — try each held key per unit, keep
the first-tried plaintext on a miss — is gone; decrypt_sectors_impl's AACS
arm now fails loud (reaching it means a reader was built without its map,
which would silently apply a wrong key). CSS (self-descramble) and the clear
no-op path are unchanged.
Disc::sweep and Disc::patch resolve a whole-disc key map up front for a
decrypting pass (the fetch secures any missing CPS-unit key, fail-loud) and
decrypt via the map — clear nav/filesystem sectors are in no range and pass
through, so the separate content-range gate and the reactive per-unit
key-fetch recovery are no longer needed. extract_tree keys every unit with
the base Unit Key through the map (its encrypted-flag gate skips clear
files). Multipass sweeps stay --raw.
Removes the obsolete non-mapped-AACS trial/gate/recovery tests (the mapped
path and resolve fail-loud are tested directly).
The key map is now purely "these sectors use this key": entry_for returns
Option and an LBA in no range is left untouched (no default-decrypt-
everything fallback). resolve_mux_key_map builds explicit content ranges
for every case — single-CPS keys each content extent, multi-CPS keys each
extent with the key that opens it, and FMTS fills the non-segment content
with the base Unit Key so a whole-disc read decrypts content and passes
nav/filesystem through. Combined with the fail-loud resolve, a map can
never silently apply a wrong key, and clear sectors are never scrambled.
decrypt_sectors_mapped skips a unit with no map entry; read_plan keeps an
unmapped unit (pass-through content) and drops only alternate-phase
forensic units. Tests updated to the Option semantics.
mp4 demuxer (untrusted input): bound every allocation sized from a box
field (stsz/stco/stsc counts, stts/ctts run-lengths, per-sample and moov
sizes, plus an absolute cap so a sparse file can't inflate file_len);
guard the parse_stsd slice and a zero mdhd timescale; cap track count so
the per-track PID can't overflow; rewrite read_moov to handle size==0 /
size<8 / 64-bit largesize; parse esds/AudioSpecificConfig for AAC; write
tkhd duration in the movie timescale.
decrypt: resolve_mux_key_map now fails loud on an extent no key can
classify instead of inheriting the previous extent's key, so a keymap
never silently carries a wrong key; the sweep/patch key-fetch recovery
fails loud when a unit is still unresolved after the retry.
AACS: reject inverted forensic segments in both range builders; compare
the forensic index in u16 space so an out-of-range value can't truncate
onto a valid u8 index. RECOVERED_ERROR no longer latches the damage zone,
preserving the 30s wedge cooldown for a following hard error.
audio: AAC/MP2/MP3/FLAC carry the last PTS across a PES with no timestamp;
the DTS-HD extension-sync search is bounded to after the core; the MP4
16.16 sample-rate field saturates. demux_sink records the video reference
before the kind filter so audio:// / sub:// keep multi-clip PTS continuity
and the DELAY tag.
Remove a dead error variant and the AACS-unsupported-video code; codec
comments cite the primary format specs; assorted doc/naming fixes and
regression tests throughout.
An FMTS forensic segment interleaves our device group's variant with a
foreign group's at the aligned-unit level. The mux was decrypting only
our phase but leaving the alternate (foreign) units in the buffer as
ciphertext, trusting the demux to 'drop untouched ciphertext cleanly'.
It doesn't: random 0x47 bytes at the 192-byte stride hit tracked PIDs,
mis-parse, and trip the demux's concealed-gap keyframe-resync — which
drops GOOD frames of ours around every segment (349 resyncs / ~6391
packets on Stand by Me, visible as playback flaws).
The map already knows which unit each LBA is and, for a forensic
segment, which phase is ours. AacsKeyMap::read_plan turns that into the
title's read plan: every default/CPS unit, plus inside a segment ONLY
our-phase units. The alternate units are never fetched, decrypted, or
handed to the demux — the demux sees one gapless our-variant stream.
- read_plan is general (single-CPS, multi-CPS, FMTS): a map with no
Even/Odd range returns the extents unchanged, so DVD/CSS, single-CPS
UHD and multi-CPS Blu-ray read byte-for-byte as before.
- Wired into build_iso_pipeline (the file-backed highway that muxes
resumed ISOs). Producer re-anchors unit_base per extent, so per-unit
segment reads stay unit-aligned and decrypt correctly.
- Extent gains PartialEq/Eq for the read_plan tests.
Tests: read_plan non-forensic unchanged; forensic omits exactly the
alternate units, kept units match the decrypt gate unit-for-unit. All
2314 lib tests pass; precommit green on 1.86.
Pending: end-to-end ISO re-mux validation (concealed gaps 349 -> ~0).
FMTS (AACS 2.1) now decodes per (LBA, phase): Phase enum + AacsKeyMap::
from_ranges_phased, decrypt only the variant's parity half. resolve_fmts_key_map
does a 2-phase index-1 anchor then per-index phase probe, and sizes the forensic
set to whatever the source returns (no hardcoded 32). KeyFetch is now two explicit
operations (unit_keys / fmts_indexes) and KeySource splits get_uk into
get_unit_keys + get_fmts_indexes. BYPASS_FMTS_KEY gate removed (first-class format).
Teed up for 1.4.5. Local WIP baseline.
The keyserver protocol now returns all 32 index keys as an array for a
forensic content sample (and a single-element array for plain content).
resolve_fmts_key_map sends one forensic batch and maps array element i to
segment index i+1, replacing the per-index blind-probe collection loop
that repeatedly hit the key service. Segment/index parsing and the
aligned-unit content classification are reworked to support this:
- rename variant_select -> index_select (per-index, not per-variant)
- content classification moves to is_clean(buf, ContentFormat) so the
unit selector emits only units the key service accepts
- segment.rs: parse IndividualSegment.tbl index tags + SPN ranges,
build contiguous LBA key ranges from the resolved 32-key array
- decrypt/decorator plumbing for the resolved per-index keys
Fail loud (FmtsKeyMissing) when the forensic query returns < 32 keys or
any segment index stays unresolved.
decrypt:
- decrypt_sectors is now a pure decrypt (apply key, leave plaintext, report
unverified bytes); TS-structure is a separate primitive (is_clean_ts/ps) used
only for key selection and read-verify. The mux passes decrypted bytes through
(the demuxer drops non-conforming packets), ending the NULL-TS conceal loop and
the per-unit key-server refetch storm. Key-proof floor replaces the 75%
supermajority.
recovery:
- Removed the post-read decrypt-verify gate (verify.rs) that mis-aligned the
disc-absolute unit grid against clip-anchored AACS units and false-failed good
clips (e.g. Dunkirk's orphan-CPS clip). Bad sectors are marked by physical read
result; decryptability is proven at scan + mux time.
HD DVD (first-class AACS):
- Role-based candidate-list file sourcing so an HD DVD's /ANY!/ files
(MKBROM.AACS, VTKF000.AACS, CONTENT_CERT.AACS) are found with no disc-type
branch. parse_vtkf parses VTKF000.AACS into the same UnitKeyFile as a BD
Unit_Key_RO.inf, so the shared VUK unwrap applies unchanged. set_unit_base
clip-anchoring. Two decrypt-axis assumptions remain UNVERIFIED-HDDVD-DECRYPT
(no encrypted disc to test).
mux:
- MVC (Blu-ray 3D) track signals unified into one MVCDecoderConfigurationRecord;
release-safe track_vint (3-byte VINT) and pid_index (i32) guards.
hardening:
- Container-aware is_clean / encryption detection; bytes_bad_in_title fail-safe
on a corrupt mapfile; CSS crack gated on DiscFormat::Dvd (HD DVD excluded);
non-vacuous CSS tests; patch NOT_READY/HARDWARE/ILLEGAL_REQUEST/ABORTED
sense-path tests.
decrypt_sectors is now a pure decrypt — apply the CPS unit key, leave the
plaintext, report how many bytes did not reach clean TS ("unverified"). It
never restores ciphertext, nulls, or re-fetches. "Did a key produce clean TS?"
is a key-selection / read-verify signal, not the verdict "did we decrypt?": a
correct key can decrypt a bad-encoded region, and broken TS is a muxer concern
(the demuxer drops the packet and resyncs).
Callers own the policy:
- mux (read > decrypt > mux): pass the decrypted bytes to the muxer, whatever
they are; fail loud only on a genuine can't-decrypt (no key / misaligned).
- sweep/patch (reading from a disc): an unverified unit is a bad read — recover
a fresh key and retry, or fail loud so disc-recovery re-reads it.
Removes three duplicated decisions — the decrypt-time ciphertext restore, the
mux NULL-TS conceal loop, and the per-unit key-server refetch — plus the dead
aacs_unit_still_ciphertext predicate. Key-fetch recovery now samples the on-disc
ciphertext explicitly (a pure decrypt leaves the buffer plaintext) and lives
only on the rip/verify path, never the mux.
Fixes the 30-90s/region mux stalls and key-server storm on bad-encoded UHD runs
that 1.4.1 left behind (it relaxed the gate but not the surrounding machinery).
Replace the AACS-specific inline key-fetch in the decrypt decorator with
a scheme-neutral recovery seam: the input stream (L3) installs a Recover
closure (none / AACS key-fetch) and the decorator (L2) runs it at the
single decrypt-miss point. FMTS (AACS 2.1) forensic-segment units that no
key opens are just undecryptable units, concealed and counted as ordinary
decrypt loss with no FMTS-specific branch ("a loss is a loss"), so the
separate bytes_undecryptable bucket collapses into one loss count.
- sector/recovery.rs: the seam (MissOutcome, none/key_fetch factories),
naming no encryption scheme in its type.
- FMTS: segment routing primitives + BYPASS_FMTS_KEY, and an upfront
ensure_forensic_segments_decryptable gate (Error::FmtsKeyMissing) in
the mux input path, parallel to the unit-key gate.
- CSS descramble/rekey moves from decrypt_sectors into
css::descramble_region: CSS self-recovers from the data itself, so it
stays OFF the seam (which is only for external inputs).
- disc/mod.rs also: main-title selection aligned to largest physical
size; is_regular read from the open file handle, not metadata(path),
fixing a swallowed sync_all on a fresh-rip ISO. decrypt_threads()
resolved once via OnceLock off the per-buffer hot path.
Relocate the shared low-level primitives into a single crypto module:
aes_ecb_encrypt/decrypt, aes_cbc_decrypt, aes_g (from content/variant) and
aesg3 + AESG3_SEED (from keys), plus AACS_IV. Fixes the scatter where AES-G
lived in the 2.1 file and AES-G3 in keys. Relocation only — no rename, no
logic change (logic-hash identical to baseline; 277 items; 2210 tests green).
Pure file+module-path rename. 'content' names the AACS unit-decrypt layer
(distinct from the top-level sector-decrypt driver crate::decrypt), and
'variant' (singular, spec term 'Media Key Variant') names the 2.1 chain.
Logic-hash identical to baseline; 277 items intact; tests green.
Post-read verify gate (new src/disc/verify.rs): UnitVerifier buffers/aligns the disc-absolute read stream into clip-file 6144-byte units, then makes one decryptability() decision per unit (CPI gate -> held keys -> key_fetch -> strict TS). POST_READ_VERIFY const kill-switch; fail-safe contract (only ever downgrades units it is confident are undecryptable; every doubt skips). Hooked into Disc::sweep (producer observes ciphertext -> WorkItem::MarkBad after the Good, FIFO-ordered) and Disc::patch (post-loop reverify_iso reads recovered units whole from the patched ISO). extract::clip_layouts enumerates AACS clips for the gate.
Standards-correct AACS verify: aacs::unit_is_clean_ts is a strict port of libaacs _verify_ts (all 32 TS syncs, not a majority vote); decrypt_unit accepts a key only on it; the majority verify_ts is removed. Deleted the Disc::verify_clips post-pass bolt-on (its primitive is absorbed by the read-path gate).
libaacs/DVD audit fixes: content-cert bus_encryption flag now read from bit 7 (was bit 0 - defeated the bus-key fail-loud gate); cc_id read from offset 14; title_cps_unit range-validated + 1->0 index-converted per libaacs. Corrected attack_crib ("functionally-equivalent" not "exact" port) and read_disc_key (READ DVD STRUCTURE 0xAD, not REPORT KEY) doc comments.
Also includes accumulated uncommitted work: key-fetch seam and TrueHD/DTS audio fix.
When a scrambled AACS unit fails to decrypt under every available key
(a missing/wrong CPS sub-key, or a marginal unit that fails the TS-sync
verify), decrypt_sectors restored the original encrypted bytes and
returned Ok with no signal. Those still-encrypted bytes flowed to the TS
assembler, which silently dropped the non-syncing packets with no loss
counter. The only loss accounting was DiscStream's read-error zero-fill
path, so mux reported lost_video_secs=0 for decrypt-dropped content and
the abort gate accepted the rip even under abort_on_lost_secs=0. A rip
missing real video/audio segments was published as a perfect success.
decrypt_sectors now returns the number of bytes in scrambled units that
no key could decrypt. DecryptingSectorSource accumulates that into a
shared counter exposed via decrypt_loss(); both mux pipelines fold it
into lost_bytes() — the inline DiscStream path directly, and the
file-backed highway via PipelinedPesStream sharing the producer's
counter. Restore-to-original is unchanged, so clear nav-files are never
corrupted; metadata-probe callers that don't read the counter are
unaffected. Adds regression tests at the decrypt and decorator layers.
Test-hardening release, no runtime changes. Adds spec-grounded unit tests
across the silent-corruption surfaces — UDF/MPLS/CLPI/IFO parsing, BD/DVD
title + extent assembly, AACS/CSS key handling, TS/PS demux + codec parsers,
MKV/EBML container output, the mux pipeline, sector prefetch + decrypt
decorator, drive/SCSI sense decoding, label extraction, and core I/O. Each
test is grounded in the format spec or real on-disc behavior and verified to
fail under a targeted source mutation. No behavior changed.
Library-wide review-and-fix pass: tightened AACS keydb/handshake/variant
handling and trailing-partial-unit policy, corrected MPLS mark offset and
added UDF allocation bounds, hardened the mux/codec framing and M2TS paths,
guarded SCSI READ CAPACITY short transfers and unified error mapping, added
overflow guards on untrusted disc input, and made prefetch shutdown
deterministic. Release profile now builds with thin LTO + single codegen unit.
Rename is_unit_encrypted -> is_aacs_scrambled and decide encryption from the unit's MPEG-TS sync bytes (destroyed by the encrypted body) instead of the TP_extra copy-control (byte 0) or TS scrambling-control (byte 7) flags, which discs do not set reliably. One shared predicate now backs the decrypt gate and out-of-band key validation, so callers agree on what 'encrypted' means. Decryption restores the syncs, so a decrypted unit reads as clear and there is no flag to clear.
Introduces the freemkv mux throughput highway: a three-stage thread
pipeline that replaces the inline single-thread read path for any
file-backed source (ISO and m2ts file URLs both route through it).
Thread A: read + decrypt (PrefetchedSectorSource / BytePrefetcher)
Thread B: M2TS demux (DemuxThread)
Thread C: codec parse (PipelinedPesStream, on caller thread)
Each handoff uses a bounded crossbeam channel with a recycled buffer
pool — no allocations or memcpys in the steady-state hot loop.
Component map:
* io/byte_prefetcher.rs (new) — std::io::Read producer thread with
recycled Vec<u8> pool. Pairs with PrefetchedSectorSource (sector
side) so demux_thread::spawn_zero_copy can wire either upstream.
* sector/prefetched.rs — recycled buffer pool added; into_channels()
peels off the rx/recycle_tx/shell triple for zero-copy demux.
* mux/demux_thread.rs (new) — owns the TsDemuxer/PsDemuxer, runs
feed() on its thread, ships Vec<PesPacket> batches.
* mux/pipelined_stream.rs (new) — the read-side Stream impl. Pulls
packets from the demux thread and runs codec parse on the caller.
* mux/resolve.rs — build_iso_pipeline (public) / build_m2ts_pipeline
(private) assemble the three stages; iso:// and m2ts:// both
return PipelinedPesStream.
* mux/m2ts.rs — collapsed to a write-only sink (Mode::Read deleted;
the read direction lives on the highway now).
* mux/codec/h264.rs — find_start_code uses memchr SIMD memmem::find.
* mux/codec/hevc.rs — tightened frame_data initial capacity.
* mux/ts.rs — boundary-packet handling avoids the per-batch 16 MiB
remainder copy; PesAssembler starts at 16 KiB to dodge the 64-page
first-touch fault tax that the previous 256 KiB pre-alloc paid on
every PES boundary.
* mux/disc.rs — gains DiscStream::new_pipeline + read_pipeline as
the legacy autorip ingress (drive + multipass paths still need
on_event / skip_errors before they migrate to the highway).
* io/file_sector_source/* — per-OS prefetch() syscall hook
(Linux readahead, macOS F_RDADVISE, Windows/other no-op).
* decrypt.rs — FREEMKV_DECRYPT_THREADS renamed to FREEMKV_THREADS;
pool sized to all cores by default.
Measured on rip1 testbed (Civil War UHD, 62 GiB ISO → null://):
60 → 322 MB/s warm cache (old new_pipeline path)
60 → 660 MB/s warm cache (highway path, this commit)
60 → 126 MB/s sustained disk-bound
The IsoSectorReader baseline reader was deleted in favour of
FileSectorSource so the freemkv CLI and autorip exercise the same
read path.
decrypt::decrypt_sectors now restores chunks when decrypt_unit_full's
TS-sync verification fails, preventing 0.18.1's silent corruption of
MPLS/CLPI navigation files when DecryptingSectorSource decorates the
sweep reader. Fixes E6009 NoStreams on info iso:// for AACS-encrypted
UHDs ripped without --raw.
Disc::sweep progress takes max(snapshot.bytes_good, bytes_done) so
the user-visible counter never regresses below what the producer has
already sent.
Architecture:
- One stream per format, bidirectional PES (read/write on same type)
- IsoStream merged into DiscStream (one type, any SectorReader)
- Disc::copy() for disc→ISO raw sector dump
- IOStream trait deleted, all byte-level Read/Write removed
- ContentReader/OpenDisc/open_title/open_input/open_output deleted
- CountingStream wrapper for progress tracking
Error codes:
- All io::Error English strings replaced with Error enum variants
- From<Error> for io::Error conversion
- Unused variants removed, new stream/mux variants added
Deleted: mkvout.rs, pesout.rs, isowriter.rs, mkv-muxer-plan.md
Updated: all docs, README stream table, CHANGELOG
238 tests, 0 clippy warnings.
Previously used all-zeros AES key when unit_key_idx was out of range,
producing silently corrupted output. Now returns DecryptFailed error.
Also wired --raw flag through InputOptions → set_raw() on streams.
Matthew Jackson