341e079e1e18c0737a486a6af9e858d63c452fae
934
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matthew Jackson
|
341e079e1e |
Fix two tests that flaked under the parallel harness
testlog: capture through a single global subscriber that routes each event to a thread-local sink and returns Interest::sometimes, so tracing's global per-callsite interest cache can never short-circuit a live capture when another thread rebuilds it. The old scoped with_default lost events at random in the full suite (an empty capture failed the log-accounting assertions ~1 run in 15). labels: capture the escaping test through testlog instead of installing its own process-wide subscriber, which hard-cached every foreign callsite "off" and poisoned the captures above. dirimage: name scratch directories from a process-monotonic counter rather than SystemTime, which resolves to only a microsecond — two of the parallel tests sharing a tag drew the same value, collided on one directory, and the first to finish removed it out from under the other's reads. |
||
|
Matthew Jackson
|
25893f1be4 |
CSS: fix the decrypted-HD-DVD false E7023 at the detection layer, not the public API
Commit
|
||
|
Matthew Jackson
|
efb69e3ba5 |
Close the Ok-but-empty clip hole on HD-DVD; keep it open, and say why, on BD
The same hole, two disc families, two answers — and the asymmetry is now a
decision written into both files instead of an oversight in one.
`file_extents` can return `Ok` and still yield no usable extent: an empty
allocation-descriptor list, or one every entry of which the `sectors > 0 &&
lba > 0` filter discards. An ordinary zero-byte file reaches it; no crafted
disc is needed.
On HD-DVD that was the flagship failure shape. The clip entered neither
`clip_extents` nor `unusable`, and nothing was logged, so the composer's
`any(|n| unusable.contains(..))` guard missed it while the
`filter(|n| clip_extents.contains_key(..))` beside it quietly deleted the
part: a `FEATURE_2.EVO` of size 0 next to a healthy `FEATURE_1.EVO` composed
a FEATURE title out of part one alone, still advertising the whole runtime,
at rc=0, in silence. Half a movie presented as a whole one. Round 1 accounted
for every `Err` from the resolver and left this route open. It now marks the
clip unusable and logs it under its own new code, E6019
(`E_UDF_NO_USABLE_EXTENT`) — deliberately not the neighbouring E6017, which
would file a zero-length file as an authoring hole and send whoever triages
it at the wrong population.
On Blu-ray the identical hole stays open, as previously decided, and the
reasons are now recorded on both sides. BD has no `unusable` set, so closing
it there means inventing a post-loop "every clip_id must appear in `spans`"
invariant that DROPS the title, and it is not settled that an empty-but-Ok
resolve is always a defect; dropping healthy titles is worse than the gap.
The consequence is milder too: on BD the clip is one PlayItem of an otherwise
whole title, on HD-DVD the feature is COMPOSED from parts. Same hole,
different price.
Also in this change:
* bluray: a non-absence SSIF failure that the `.m2ts` fallback papers over is
logged. `unresolved` had exactly one reader, `if let (None, Some(code))`, so
when `/BDMV/STREAM/SSIF/<clip>.ssif` failed with DiscRead /
UdfAdChainTooLong / UdfEmbeddedData and the base view then resolved, the
code was recorded and thrown away: the title shipped base-view 2D off a 3D
disc at rc=0 with no log at all. The site's own doctrine is "ABSENCE is the
only benign failure". Logged, not refused — the base view is a real rip.
* drive: `wait_ready` polled TEST UNIT READY through a bare `execute` and its
60 x 500 ms loop never read `self.halt`, so a Stop during spin-up did
nothing for ~30 s while every other drive path returns Halted at the next
command boundary. `spin_cycle` issued both START STOP UNIT commands outside
`checked_exec` and slept `SPIN_DOWN_IDLE_SECS` + `SPIN_UP_SETTLE_SECS`
blind — ~15 s deaf to Stop, from the recovery path, exactly when the
operator is most likely to press it. Both now use `checked_exec` and
`sleep_until_halted`, which already lived in this file with four tests and
was `#[cfg(test)]`, called from nowhere. It is production code again.
* drive: a READ(10) that returns GOOD status with a residual underrun was
correctly refused and logged NOWHERE, while the sibling `Err` arm warns with
lba/count/status. A residual-underrunning drive was indistinguishable from a
scratched disc — two populations with opposite remedies. It now warns with
transferred vs expected, which is the whole signal.
* error: `all_error_code_constants_are_unique` was a hand-maintained `vec![]`
naming 109 of the 127 declared codes while its doc claimed to pin them all,
and an earlier audit trusted that claim while assigning new ones. The list
is now derived from the declarations by parsing `include_str!("error.rs")`,
so a new constant is covered the moment it is written. A parser self-test
cross-checks the count and three known name/value pairs, so it cannot pass
vacuously.
* testlog: a test-only `tracing` capture (~120 lines, no new dependency) so
the logging contract is enforced rather than commented. Three sites carry
long comments insisting they log the error's OWN code; putting a literal
back broke nothing. They are pinned now, along with the two new log lines.
Captures are serialised process-wide: `tracing`'s interest cache is global
while `with_default` is thread-local, and the rebuild on the exiting
capture can land after the entering one's, leaving the cache at "never"
while a capture is live. That produced a real empty-event flake.
* disc: `scan_with`'s halt wiring for the BD and DVD enumerators had no test —
every BD/DVD cancellation test calls the scanners directly, so passing
`None` on either branch left the suite green while Stop did nothing.
* mux::network: `accept_from_rejects_stream_without_fmkv_header` half-closes
instead of `Shutdown::Both`, which raced an RST against the server's read
and returned ConnectionReset instead of InvalidInput under load. The port
was already ephemeral; that was never the cause.
Gate: fmt, clippy --all-targets -D warnings, and 3439 tests green on 1.97;
precommit.sh libfreemkv clean.
|
||
|
Matthew Jackson
|
4cd9b7baa1 |
CSS: key the DVD crack on the disc, not on the container
`resolve_dvd_title_key` decided whether to run the DVD CSS crack by asking what CONTAINER it was looking at — `ContentFormat::MpegPs`. MPEG program stream is what DVD and HD-DVD have in common, so every HD-DVD title was run through a crack scan for a copy-protection scheme HD-DVD does not use and cannot carry: AACS is its family, and CSS appears nowhere on the disc. Both outcomes of that scan were wrong. The cheap one wasted up to 50,000 sector reads per title. The expensive one returned `CssKeyMissing` — E7023 — refusing a perfectly good HD-DVD with a CSS error, which is what a real CI run produced on the HD-DVD fixture. Reading E7023 there sends whoever triages it looking for a missing DVD key on a disc that never had one, which is how a routing bug spends a day disguised as a key problem. The right axis was already in the codebase and already used: `mux/resolve` asks `disc.format == DiscFormat::Dvd`. This threads the disc format down to the decision and adds `DiscFormat::may_have_css` to name the question. The asymmetry decides the default, so it is worth stating. Running CSS on an HD-DVD costs a wasted scan or a false refusal — visible, recoverable, annoying. NOT running it on a real DVD muxes scrambled sectors as plaintext and exits 0, which is the failure-that-looks-like-success class this project has shipped once already. So `may_have_css` is false ONLY for the families proven CSS-free, and `DiscFormat::Unknown` — a bare reader with no scan behind it — still cracks. An `== Dvd` allow-list would have read as tighter while silently stranding every caller that cannot name its disc. Both directions are pinned: an HD-DVD title must never enter the crack, and an unknown-format title must still enter it. Removing the disc-format clause fails the first and leaves the second and every DVD test green. |
||
|
Matthew Jackson
|
02e7bc605d |
Propagate cancellation out of every title enumerator; account the .clpi read
A cancelled scan was indistinguishable from a disc that simply holds
fewer titles, in all three tree enumerators. Once the operator presses
Stop, `Drive::checked_exec` fails EVERY subsequent SCSI command with
`Error::Halted` and `Drive::read` deliberately preserves the variant —
so a cancel is not one failed read, it is every remaining read failing.
* `scan_bluray_titles` returned a bare `Vec<DiscTitle>` and
`parse_playlist` an `Option`, so neither had a channel to report a
halt. During a cancel each remaining `.mpls` read merely failed and
was skipped by an `if let Ok(..)`, and the scan returned a TRUNCATED
title list at rc=0. Both now return `Result`; `Ok(None)` from
`parse_playlist` keeps its two benign meanings (unparseable MPLS,
playlist under 30 s) and `Err` means the scan is over.
* `scan_dvd_titles` was worse: `Err(_) => return Vec::new()` turned a
cancel into ZERO titles at rc=0 — a DVD reported as carrying no
video at all. `ifo::parse_vmg` compounded it, treating a halted
title set as one of the placeholder TT_SRPT entries it is designed
to tolerate and continuing to the next; both now propagate.
* HD-DVD was already correct and is the model this follows, including
its reasoning: a cancelled enumeration that returned `Ok` would be
cached, displayed and ripped from as if it were the disc.
The cancel is propagated at ALL FOUR Blu-ray read sites, not just the
obvious one — the playlist read, the `.clpi` read, and both
`file_extents` calls in the clip resolver — and the halt is polled
between playlists and again after the loop, so the final iteration
cannot slip a truncated list through. The resolver's `note` closure
previously EXEMPTED `Halted` from its disc-defect classification, which
was right (a cancel is not an authoring hole) but left it swallowed:
with no exemption and no propagation a cancel landing on `file_extents`
returned a title claiming its full runtime with that clip's bytes
silently missing. Measured with the fix reverted: `Ok(Some((768000,
[])))`.
Second defect, same function: the `.clpi` read and `clpi::parse` sat
inside `if let Ok(..) && let Ok(..)`, with extent resolution nested
inside. A scratched CLIPINF sector, a missing `.clpi` or a malformed one
skipped the whole block — no extents, nothing added to `total_size` —
but `duration_ticks` had ALREADY been summed from the play items, so the
title still advertised the movie's full runtime. The `if let` discarded
the error, so nothing was logged either: a failure that looks like
success, which is this crate's flagship defect class. It is now a
`match` that drops the title and warns with the error's OWN code, the
same classification the extent resolver below it already uses. Logging a
fixed code would account a scratched disc as an authoring hole.
Deliberately NOT added: a post-loop "every unique clip_id must appear in
`spans`" invariant. `file_extents` can also return `Ok(vec![])`, or a
vector whose every entry the `sectors > 0 && lba > 0` filter discards,
and it is not settled that an empty-but-Ok resolve is always a defect
rather than a legitimate healthy-disc state (zero-length placeholder
ADs). Dropping healthy titles is a worse failure than that residual gap,
so the route stays open, with a comment at the site recording the
decision so the next audit finds it instead of re-deriving it.
Tests: `parse_playlist_missing_clpi_yields_no_extent_no_size` asserted a
title WAS returned with `size_bytes == 0` and empty extents, and its
docstring quoted the buggy control flow as the specification — it
blessed the defect. Inverted and renamed. `parse_playlist_keeps_exactly_
30_seconds` encodes correct behaviour but used a fixture wiring no
STREAM/CLIPINF, so the `.clpi` fix would have dropped its title for a
reason unrelated to the boundary it pins; repointed at the fully wired
fixture rather than weakening the assertion. Four new halt-propagation
tests (BD playlist read, BD `.clpi` read, BD extent resolve, DVD title
set + VMG), each proven red before green.
|
||
|
Matthew Jackson
|
0563b58f2e |
Audit round 7: account for every clip that cannot be resolved
Ten lenses over v1.6.4..HEAD, every claim read against the code before it was believed. Seven confirmed; six are here, one is recorded for the next round. All of these are the same family — a failure wearing the shape of success — which is the family that once shipped 9 MB of ciphertext inside a main-movie m2ts at rc=0. A clip whose extents cannot be resolved is now accounted for, in both disc readers. Only `UdfUnrecordedExtent` used to count: every other way `file_extents` can fail — a scratched sector under the clip's ICB (DiscRead), an allocation-descriptor chain that never terminated, a file whose data is embedded rather than extent-mapped — fell through to the ordinary "file absent" path. On Blu-ray that yielded a title advertising its full runtime with a clip's bytes silently missing, because the size and the play-item timing had already counted it. On HD-DVD it was worse: the clip was never added to `unusable`, so a split feature still composed from FEATURE_1 alone and offered half a movie as the whole thing. Neither emitted a single log line. Absence is still benign — a 2D disc has no .ssif and the extension fallback exists for exactly that. `Halted` is excluded deliberately, and that exclusion is the whole reason the first version of this fix was wrong. Cancellation makes EVERY drive command return `Halted`; classifying it as a disc defect would have dropped each remaining playlist in turn and handed back a truncated title list at success — the same defect, wearing a cancel. `parse_playlist` returns Option and has no channel to propagate a halt, so the existing behaviour is preserved rather than made worse. Propagating it properly is next round's work. Both log sites now emit the error's OWN code instead of a hardcoded 6017. Accounting a scratched disc (E6000) as an authoring hole would send anyone triaging it looking for the wrong thing entirely. AD type 3 is embedded data, not a descriptor list (ECMA-167 4/14.6.8). `read_icb_extents` lumped it in with the reserved values and decoded the file's own CONTENT as (length, LBA) pairs, manufacturing extents out of arbitrary bytes and pointing the reader at unrelated sectors. This same release already taught `read_directory` to honour type 3; this is the file half of that decision. It is an error rather than an empty list, because an empty list reaches the caller as a clip that contributed nothing while its declared duration still counts it — the silent loss pointed the other way. A legally zero-length embedded file still returns an empty list. New code E6018: reusing DiscRead would have mislabelled a deterministic structural property as transient I/O and fed the retry and NonTrimmed machinery a byte that will never change. `file_extents_addressing`, `extents_abs_at` and `AbsExtent` drop to `pub(crate)`. The first hands back unrecorded extents UNFLAGGED, in a shape identical to the safe call's return; its doc says callers must use `file_extents` instead, but a doc comment is not a guard. No dependent crate references any of the three. Three tests close gaps the audit found, each proven red before green: a held AC-3 access unit must not resume as a normal frame after its track poisons; the PS resume cursor must survive a drain that rebases it (three separate mutants caught); and AD type 3 must be refused rather than decoded. The first attempt at the HD-DVD test passed with the fix reverted, which made it worthless — it needed a VTI fixture before the composition path ran at all. Also: four error codes were missing from the uniqueness test that claims to cover every published code, so a new variant reusing 6014, 6016 or 6017 would have passed it. |
||
|
Matthew Jackson
|
68a1a55958 |
Audit round 4-6: disc parsing, extents, codecs and drive faults
Squashed from 12 commits. Every fix was proven red-before-green and killed by a mutation; the reasoning for each is in the private audit record. UDF and extents Honour ICB types rather than assuming a Short AD, so an AD-type-3 directory is no longer decoded from FID bytes into a silently empty listing. Carry the ECMA-167 recorded flag through to the resolvers: an allocated-but-never- written extent used to reach the read plan as ordinary content and splice undefined sectors into the rip. file_extents now refuses such a file, and only when the hole actually occupies byte space — a zero-length one displaces nothing, and refusing on it dropped whole titles off discs that ripped correctly. Type-2 sparse extents are kept alongside type-1; they were falling into a catch-all that exited the descriptor loop and returned a truncated list as complete. merge_ranges no longer claims a sector neither input covered. A short skip or an over-long AD chain errors instead of truncating. HD-DVD and Blu-ray scanning Bound the XPL nesting depth, title count, clips and chapters per title, and memoize the clip-name fallback probe — four separate amplification axes, each of which alone left the worst case unbounded. The clip and title caps are 512, ~10x any retail disc, and a test pins the product of cap and probe budget. The scan is cancellable: it returned Ok with titles carrying no streams when halted, presenting a cancelled scan as a successful one. A clip dropped for an unrecorded extent now says so. Codecs and muxing Resume a held E-AC-3 access unit rather than rescanning from its first frame, and drop it on a discontinuity — a stale hold indexed past the end of the new buffer. Map every ISO 639-1 code instead of collapsing fifteen languages to und. Correct the DVD palette order. Detect a skip past EOF. Drive and I/O Classify dead-bus faults so the wedged-drive path can see them; a catch-all arm had been flattening the variants before the classifier ran. A prefetch producer that dies now reports SourceTerminated instead of Ok(0), which the reader legitimately read as a short read and zero-filled — a whole title could be fabricated and the pass reported complete. Also: charge Ok(0) reads to the CSS crack budget, drop the unreachable soft re-crack, and send disc-derived strings to logs through the debug formatter so a crafted label cannot paint an operator's terminal. |
||
|
Matthew Jackson
|
b64a96042e | Trim single-clip titles to their playlist marks (1.6.4) | ||
|
Matthew Jackson
|
074b1ee829 |
Stop an uncrackable VTS borrowing another VTS's title key
`resolve_vts_key` called the `Option`-returning `css::crack_key`, which collapses "no scrambled sector was seen" with "scrambled sectors were seen and no key came out", and then fell back to the disc-wide key. A multi-VTS CSS DVD whose second title set resists the Stevenson scan was descrambled under the FIRST set's key: corrupt PES behind an intact header, written out with `complete = true` at exit 0. `CrackOutcome` exists precisely to keep those two apart, and its doc says callers must surface the second as a hard error. Every sibling path in the crate already does — `Disc::decrypt_keys_for_title` and the mux path both map `ScrambledUncracked` to `CssKeyMissing`. This was the one path that did not. The ordering comment 15 lines above describes this exact outcome as the bug it exists to prevent; ordering makes the crack far more likely to succeed, but it cannot make a failed crack safe. Also, three things nothing could catch: - The demux output filename took the stream's language raw while the base beside it was sanitised. A language code is three raw STN bytes through `from_utf8_lossy`, and `00 00 00` is the ordinary "undefined" encoding on real discs — a NUL in a path fails `File::create` with InvalidInput, taking the whole export down before one track file opened. `sanitize` now maps control characters too; it did not. - `css::crack_key_scan`'s short-read handling was dead code under test: every source in the module returned the full request, so reverting `advance` to the requested count, or dropping the `.max(1)`, left the suite green. The `.max(1)` is load-bearing — without it a source returning `Ok(0)` never moves the cursor and never increments the budget, so the scan spins forever. That mutation now HANGS the test rather than failing it, which is the honest demonstration. - `MAX_SUBDIRS`'s const-assert carried `#[cfg(not(test))]` inside a `#[cfg(test)] mod tests`, so it compiled in no configuration and could never fire — the dead gate the test above it was written to replace. Moved to module scope, and verified it now rejects a wrong constant at compile time. |
||
|
Matthew Jackson
|
8f9bde9b9a |
Bound the third hostile CSV, and stop two docs overclaiming
`forced_sub` was the last unbounded attacker-controlled list in paramount.rs. `MAX_COM_INDICES` capped the two `*_com1_idx` attributes; this one had nothing capping it at all, and unlike them it has no value to filter — a cell is a classification of the position it sits at, so its bound has to be positional. Extracted as `forced_subs` for the same reason `com_indices` was extracted: through `labels_from_feature` the bound is unobservable, because the subtitle loop cannot reach those cells either, so the assertion could not fail. The `MAX_COM_INDICES` doc claimed the entry-allocation argument for the whole constant. It is the VALUE filter that caps the set (values below the bound, so at most that many distinct entries, however long the attribute); the `take` caps the WORK. Both are real and they are not the same bound; the doc now says which is which. `jar_inventory_dedup_is_not_quadratic` called itself proof by deadline. It is a hang guard — a return to the linear scan runs for minutes and would wedge CI rather than fail it — and no assertion in it can tell a BTreeSet from any other sub-quadratic dedup. Renamed and documented for what it does. Its margin was measured before keeping it: 0.14s debug / 0.07s release against 10s, ~70x, unlike the 6x that made paramount.rs's wall-clock test flake. |
||
|
Matthew Jackson
|
2d1563c63a |
Round 2: stop two of round 1's tests claiming more than they prove
Both are mine, and both pass with the bound they "cover" deleted. `a_hostile_commentary_index_list_is_bounded_not_merely_fast` asserted only label purposes. The out-of-range filler it feeds is unobservable at the label level and a HashSet collapses the repeats, so removing MAX_COM_INDICES entirely leaves it green. What it DOES catch is a bound set too low — verified at 2, where the real indices stop resolving. Named and documented for that, and it no longer implies it guards enforcement. `an_index_that_cannot_address_any_cell_is_not_retained` asserted through the labels, where retention is by definition unobservable: the loop never queries a cell that high. It now reads the set through `com_indices`, where the claim is checkable. Enforcement was and remains proven by `distinct_unaddressable_indices_are_refused_not_stored`, which was red at 50,000 retained entries and green at zero. Two tests, two properties; neither pretends to the other's job now. |
||
|
Matthew Jackson
|
cbb127a175 |
Bound the commentary-index parse, and stop proving it with a clock
The gate caught `commentary_index_lookup_is_not_quadratic` failing, then passing on a re-run. Measured both ways: 1.62s alone, OVER 10s against its own 10s deadline while the suite's other 3,347 tests ran concurrently. A 6x margin against a shared CPU is not a margin, and a test that gets re-run until it passes is not a test. It was also measuring the wrong thing. Replacing the linear scan with a HashSet bounded the LOOKUP; the set was still built from every entry the disc declared. `playlists.xml` is attacker-controlled and has no length of its own, so a hostile disc could still force an unbounded allocation before any lookup happened — the parse, not the query, was the exposure. The code's own comment said "unbounded parsed input" and only fixed half of it. `MAX_COM_INDICES` bounds both halves, at the one value that cannot change behaviour: an index at or beyond `u16::MAX` can never match a cell, because the labelling loops break at `u16::try_from(i + 1)`. Real authoring is nowhere near it — the BD STN table admits 32 streams. The parse moved into a `com_indices` helper so the bound is OBSERVABLE. Through `labels_from_feature` it is not: a HashSet collapses repeated values, and an out-of-range index changes no label, so the obvious test passes with or without the cap — an assertion that cannot fail, which is what the first draft of this fix shipped. The test now hands in 50,000 DISTINCT unaddressable indices and asserts they are refused. Proven red with the bound removed (50,000 kept), green with it. Applies to both `sub_com1_idx` and `aud_com1_idx`. NOT done here: `labels/mod.rs:1421`'s `jar_inventory_dedup_is_not_quadratic` is the same wall-clock shape and has the same flakiness. Named so it is not lost. |
||
|
Matthew Jackson
|
9c6b7baf83 |
Cover the dir:// input door, and correct the CSS scramble-gate docs
Three round-1 audit findings on v1.6.0..HEAD.
Tests: `mux::resolve::input("dir://…")` — the door the CLI rips a folder
through — was never driven with a real folder. The only dir:// input test
uses a missing path, which fails in DirImage::open long before
`image_input`, so dropping the `is_folder` argument (and with it
`session::apply_folder_encryption_verdict`) passed the whole suite. Two
tests now close it, stated as a DIFFERENTIAL between the two doors,
because that is the invariant the shared function exists to hold: a
clear folder that kept its AACS/ directory must scan the same and select
the same extents through `scan_dir` and through `dir://`, and a
scrambled one must be refused through both with E9063. Proven red
against a mutated guard (both fail with E7022, the tree-shape verdict
the probe overrides) and green with it restored.
Docs: css/mod.rs documented `is_scrambled`, a function this line renamed
to `has_scramble_flag_bits`. Eleven stale sites, two of them broken
rustdoc links. Two were not stale names but false statements — they said
the descramble loop keeps the looser raw-flag test, when
`descramble_sector` and `descramble_region` both gate on
`is_scrambled_pack`; that parenthetical is rewritten to say what the
code does and why (the measured VIDEO_TS.IFO case: 38 titles became 10,
silently, at exit 0). The one mention that must stay is the sentence
explaining why the name was rejected.
Constants: io::image_writer and dirimage::encode each re-declared the
2048-byte sector that consts::SECTOR_BYTES already exports; both now
alias it. Note `BATCH_SECTORS = 2048` in image_writer is a different
quantity (sectors per batch, not bytes) and is deliberately left alone.
Bare 2048 literals elsewhere in the crate are out of scope here.
|
||
|
Matthew Jackson
|
f434b9cf2c |
Drop six unused crates, and align the rest with the workspace
Two problems, both invisible until the whole graph is looked at together. DEAD: num-bigint, sha2, num-traits, num-integer, cmac and cbc are declared here and referenced nowhere -- not in src, tests or benches. They were being compiled, audited and offered version bumps forever for no reason. Removing beats bumping. cbc nearly survived the sweep: a substring search for "cbc" matches 44 occurrences of ycbcr_to_rgb in the DVD subtitle decoder, so it looked used. Only a word-boundary search exposed it. SKEW: this crate was the outlier on every shared dependency -- aes 0.8, rand 0.8, base64 0.22.1 and zip 2 against 0.9 / 0.10 / 0.23 / 8 elsewhere. Cargo cannot unify across a major version, so it compiled BOTH: 32 duplicated crates in the freemkv binary's graph, including two complete AES implementations (aes 0.8 + 0.9, cipher 0.4 + 0.5), two digest stacks and two getrandom. Two crypto stacks in one product is worth removing on its own. The aes bump is an API rename -- BlockCipher-prefixed traits, Array for GenericArray -- and the obvious translation uses Array::from_slice, which the new version deprecates and clippy's -D warnings would reject. These use the From<[T; N]> conversion the crate points at instead. 3441 tests pass in debug and release. The AACS crypto here is covered by known-answer tests, so a byte-order or sizing mistake in that rename could not have passed. |
||
|
Matthew Jackson
|
17622a1b59 |
Stop trusting a byte offset when the feed is not what was measured
A clip's span is measured over the title's full extents when the disc is scanned, and a frame is placed by the offset it was read from. Those only agree while the mux reads every byte the scan counted. On a disc carrying a forensic segment it does not: the read plan omits the units belonging to another device group, so fewer bytes are fed than the spans describe, and the shortfall grows through the title. Every frame past the first segment then looks earlier than it is — placed in a clip it did not come from, or dropped at a join for failing marks it was never inside. The spans still tile one another perfectly, so the check that asks whether they can be trusted cannot see any of it. The plan is now compared against the extents it was built from. When they differ the spans are dropped and placement falls back to timestamps, which is what that path is for and what the surrounding comment already promised would happen when an offset stops meaning anything. Ordinary discs are untouched: with no forensic segment the plan IS the extents. |
||
|
Matthew Jackson
|
32824fba5b |
Make four guards testable, and one guard see both spellings
The provenance guard knew one way to lose a frame's source offset — writing it as absent. Omitting the field entirely does the same thing, because the frame type fills it in by default, and the guard read straight past that. A parser rewritten into the second spelling would have kept its green light while its track silently went back to being placed by guesswork. It now reports which spelling it found and where. The title-count clamp was asserted over a fixture too small to hold more titles than the cap allows, so the walk stopped when the buffer ran out and the clamp was never what bounded it — the assertion held with the clamp deleted. The fixture now carries more entries than the cap. The subdirectory limit was checked by restating the constant's own definition; the guard itself had never run, and deleting it changed nothing. The limit is lowered under test so a real folder can exceed it, and the test now walks one and requires the refusal. One test also carried two unrelated grounding notes while the test they described had none, so reading the note above a test told you about a different one. |
||
|
Matthew Jackson
|
109afcdcf7 |
Keep both halves of a clip file that two play items share
A playlist may point at the same .m2ts from several play items — a seamless split, a looped segment, multiple angles. The file has one set of bytes, so it has one span, and the byte a frame was read from therefore identifies the FILE, not which play item's range it falls in. Placement took the first of them and judged every frame against its marks, so everything past that range's end was treated as material the playlist excludes and dropped: half the clip missing from the rip, with the timeline still charged for its full duration. The offset narrows a frame to the run; only its timestamp can finish the job, and each play item carries its own marks. A frame is now matched against the marks of the entry it actually falls in, falling back to the first when it falls in none, which leaves genuinely-excluded material dropped as before. The test for this construction asserted only that such a playlist is trusted, and never placed a frame from the second range — so the loss it described in prose was invisible to it. It now places one from every range and requires them all to survive. The deferred-mux replay of buffered frames wrote them without their source offset, quietly sending the head of every such title down the timestamp heuristic the rest of this work exists to retire. It passes the offset through now, and the frame writer that omits it is compiled out of the library entirely: no production path can discard provenance any more. Also: the AC-3 buffer-reuse test could not see the regression it named. Feeding equal-sized packets, a fresh allocation per call yields the same capacity as a reused one. It now feeds a large packet then a small one, where only a reused buffer keeps the larger capacity. |
||
|
Matthew Jackson
|
dd749132d5 |
Audit round: an ADTS frame that cannot hold its own CRC, and three drifts
An ADTS header that declares a CRC follows must be at least nine bytes — seven of header plus the two the CRC occupies — because the declared frame length counts them. The structural gate compared against a flat seven and never read the bit that says whether a CRC is there at all, so a frame whose own header describes something impossible was accepted and handed to the muxer as decodable. The pipeline's spawn doc named Sweep, and the thread Sweep would have created, as callers to look for. Neither has been in this crate since the recovery passes moved out. The same paragraph already records fixing this once, for a different departed caller — it simply drifted again a sentence later, so it now says to name callers that live here or name none. Whether a disc is structurally AACS-encrypted was spelled out by hand in both the fast identify and the full scan. They agreed today; nothing made them agree tomorrow, and disagreeing would mean the same disc reported encrypted by one path and clear by the other. There is one definition now, and the comment that pointed at it by line number points at its name. The AC-3 parser built a fresh buffer on every packet — of the order of a hundred thousand times per title — to work around a borrow it cannot avoid. The copy stays; the allocation does not. The buffer is now lent out and handed back, and a test pins that, because reverting it would be invisible in behaviour. Left alone deliberately: send/send_with_halt and finish/finish_with_halt look like one action under two names, and are not. After the consumer fails, one must still accept items and the other must refuse them; that difference is what stops a producer reading an entire disc for a write that died on its first frame. Collapsing them was tried here and the existing test caught it. Both now say so where the choice is made. |
||
|
Matthew Jackson
|
3ff2abbff5 |
Add an error for an image shorter than its recovery data
A recovery resumed against an image that has been truncated since the previous pass — a full disk, an interrupted transfer, a remount — cannot repair it: the pass only revisits the ranges the recovery data calls bad, so everything past the cut stays a hole while the counts still describe a whole disc. There was no way to say that. Reusing the recovery-data error would have been wrong, because the recovery data is intact; the image is not. Registered in the uniqueness list, the code table and the range check, so a future variant cannot silently reuse 6015 or map to the wrong code. |
||
|
Matthew Jackson
|
835e97ce71 |
Give a frame that outruns its epoch's video a provisional offset
The demuxer interleaves, so at a boundary the streams do not reset on the same frame: audio for the next segment can reach the muxer before the video frame that opens the epoch it belongs to. Only the primary video track may open one, so those frames rode the just-ended offset and landed a whole segment in the past. Downstream the strictly-monotonic block nudge then crushed the entire run onto one instant a tick apart, which is audible. Such a frame is recognised on its OWN raw PTS, which within an epoch only advances, so a large backward step is unambiguous — a different signal from the shared frontier, which is what the old false-positive ratchet keyed on. It then rides a provisional offset private to its track, computed the same way the video path computes a real one, and drops it the moment the video retires an epoch, so the run rejoins with no seam. The offset is deliberately private: it never writes offset_ns, never advances the frontier and never retires an epoch, so it cannot move the video timeline. Letting a passive track open a REAL epoch was tried first and inflated a 476.776 s title to 656.216 s, because every track meets a boundary at its own pace and the video path rebased again on top. Measured on a real DVD title with 8 cell boundaries: audio frames stamped inside one cadence 120 -> 0, outsized gaps 7 -> 0, subtitle 1 -> 0, and the video span byte-identical at 476.484. On the HD-DVD title the remaining boundary gap on its first audio track falls from 0.999 s to 0.275 s. Acceptance: 76 pass, 0 fail, 0 skip. |
||
|
Matthew Jackson
|
70e1807e61 |
Place a straggler by its own epoch's end, not the current frontier
A non-video frame can reach the muxer after the video that opened the next epoch has already been processed. It carries an old-epoch PTS, so adding the new offset flings it forward by a whole clip. The remap that exists to catch this compared the frame against the CURRENT frontier and only accepted it within one backstep, which is the wrong yardstick: by the time a straggler is seen the frontier has moved on into a new epoch, and how far below it the frame lands says nothing about where it belongs. Retiring an epoch now records the frontier it closed at alongside its offset, and a straggler is judged against the end of the epoch its PTS came from. Keeping the whole history rather than one previous offset also fixes the case a single `prev_offset_ns` cannot express at all: a frame from two or more epochs back. Measured on a real HD-DVD title whose second audio track put its final packet at 12834.587 s in a 6434.100 s file. That frame sits 23 s below the current frontier — outside the old window, so it was refused — but 0.15 s below the end of the clip it actually came from. It now lands at 6417.216, one 32 ms cadence step after its neighbour, and the track spans 6416.160 with no outsized gaps. The other two streams are unchanged. A title that never rebases retires no epoch, so the lookup finds nothing and the mapping is bit-identical to before. |
||
|
Matthew Jackson
|
418abfe79e |
test: cover the 1.6.1 provenance work where it was assumed, not asserted
Four parsers (dvdsub, flac, lpcm, mpegaudio) stamp a source offset on every frame but had no test that read one back, so a regression to `source: None` would have been caught only by the brace-balanced audit in codec/mod.rs — a lint, not a behavioural check. Each now asserts the emitted frame carries the offset of the packet that supplied its first byte. The Blu-ray feed spans had no direct test at all. Add one that walks a multi-item playlist and requires the spans to tile the feed with no gap or overlap; it catches a one-sector-per-clip drift, which is exactly the error class that would misattribute frames near a seam. `no_provenance_still_places_by_marks` asserted only that placement returned something, which passes for a frame placed in the wrong clip. Its probe timestamp lands in an overlap between two clips in the real mark table, so pinning one clip would assert a coin-flip; instead require the offset to be one that a clip actually containing that timestamp would produce. |
||
|
Matthew Jackson
|
42c62d46b6 |
Test the DVD title table's sector-offset arithmetic
The TT_SRPT pointer at 0xC4 is a SECTOR offset from the start of VIDEO_TS.IFO. Reading it as a byte offset lands 1/2048th of the way in and does not error — it silently returns a different title list, which is how a disc enumerating 38 titles and an image enumerating 10 looked like a scan difference rather than a failure. The table already had tests for its entry layout, dedup and the 99-title clamp. What it did not have was anything pinning the offset unit itself, or the grouping of titles by title set, or a truncated table keeping the rows it can read rather than failing a whole disc. |
||
|
Matthew Jackson
|
282651186c |
Guard the invariant that provenance is universal
Every emitted frame must carry the source byte offset of the packet it came from. That held only for video for as long as it existed, and nothing asserted it, so nothing caught it: ten parsers built frames with `source: None` and a multi-clip title could not place audio or subtitles by byte at all. Finding them took a brace-balanced scan of the tree by hand — a regex cannot do it, because a Frame literal contains nested braces and a non-greedy match stops at the first `}`, which is how five sites survived the first pass. This is that scan, as a test. A second test fails if a codec module is added and not listed, since an unchecked parser is exactly how the gap persists. Modules that emit no Frame are named explicitly rather than skipped silently. The guard tripped on itself twice while being written — first on its own doc comment, then on its own string literals — so it strips line comments and excludes its own module. A check that matches prose about the defect rather than the defect is the same mistake in a different place. |
||
|
Matthew Jackson
|
8ec71834dd |
Stamp the last frames that were still leaving without provenance
A real rip caught these. The warning added for a track placed from timestamps under a seam plan fired once, on one subtitle track of a 23-clip title, eighteen minutes in — a display set that arrives as a lone non-PCS segment is emitted straight through rather than accumulated, and that path still built its frame with source: None. Auditing the rest the same way found four more: flac, lpcm, mpegaudio and the passthrough parser. All are one-PES-one-frame, so the packet's own facts are the unit's facts. Every Frame construction in every codec parser now carries a source, checked by walking balanced braces rather than by eye — the earlier count was taken with a regex that cannot see a block containing nested braces, which is how these survived the first pass. |
||
|
Matthew Jackson
|
991977f297 |
Place a non-advancing mark table by provenance
Marks that do not advance across a title are normal, not a defect: each clip file carries its own STC, so one clip's IN has no ordering relationship to the previous clip's. Refusing those tables dropped exactly the branched titles this type exists for onto the inference path that cannot read them, and that is the overrun. Lifting it was tried once before every track carried a source offset, and nine audio and subtitle tracks with nothing to place them by pinned themselves to clip 0 and dropped most of the title. Every parser stamps provenance now, so the clip comes from the byte offset and the marks keep one job: whether a frame lies inside its own clip's [in, out]. Still gated on spans_trusted. Without usable spans there is no offset to place by and inference is all that is left, so a table inference cannot read is still refused rather than silently truncating a title. A frame reaching the timestamp path under a plan now says so once per track. It is not expected any more, and it is how a mostly-dropped track reached a user without a single line in the log. |
||
|
Matthew Jackson
|
d4a32d3a26 |
Every parser now carries provenance, on the one shared buffer
dts, ac3 and truehd assembled access units across PES packets with three private implementations of the same bookkeeping, and none of them carried the source byte offset. They now hold a PesBuf, so a unit takes the timestamp AND the source of the packet covering its first byte, from the same mark, and no codec can answer that question its own way again. dts is the reference case: its pts_marks already implemented the rule correctly, and all 61 of its existing tests -- including the PTS attribution ones -- pass unchanged on the shared type. That is the evidence the type preserves the behaviour dts had right. ac3 kept a single carry-over timestamp and one anchor offset, so it could only attribute the first unit in a call; it now resolves each unit at its own offset. truehd had no attribution at all beyond a running clock. New tests cover the case that motivated this: a unit whose first bytes arrive in one packet and whose remainder arrives in the next keeps the FIRST packet's offset. At a clip seam those two packets belong to different clips, and taking the later one places the audio in the wrong one. Clippy on the pinned toolchain caught an empty `if` block left where dts used to clear stale marks -- restored as an explicit clear, with why it is still needed once drain keeps the covering mark. |
||
|
Matthew Jackson
|
813edd0965 |
One rule for which packet a unit's facts come from
A parser that assembles an access unit across PES packets has to answer one question for every unit it emits: which packet carried this unit's FIRST byte? Its timestamp comes from that packet, and so does the source byte offset that says which clip of a multi-clip title it belongs to. The packets that complete the unit carry later values that must not override it. That question was being answered three ways. DTS kept a deque of (offset, pts) markers and took the one covering offset 0. AC-3 kept a single carry-over timestamp. TrueHD kept its own. PGS and DVD subtitles each held a pending unit with just a start time. None of them carried the source offset at all, which is why provenance existed only for video and why nine audio and subtitle tracks on a branched title had nothing to place them by. So it lives in one place now. PesBuf owns the bytes AND the marks; PesFacts returns a packet's timestamp, source and discontinuity together, so a parser cannot take one from one packet and another from the next, because it does not assemble them itself. The type answers WHICH packet. How a codec reads a timestamp out of that packet stayed the codec's business at first, and that turned out to be the same drift one level down: dvdsub read pts alone and returned 0 for a packet carrying only dts, while everything else took pts.or(dts). Now there is one derivation. It is not a choice between two fields — for audio and subtitles there is no reordering, so dts IS the presentation time and reading it is reading the same value from whichever field the packet used. Reordering video never calls it; that path reconstructs display order instead. Migrated: adts, pgs, dvdsub. dts, ac3 and truehd follow. |
||
|
Matthew Jackson
|
b16eacd6b4 |
Restore the refusal: only the video parsers carry provenance
Accepting a table whose marks do not advance was wrong, and a real 22-clip title showed exactly how. The rip failed the drop-volume gate at 84 percent with more frames dropped than kept. The cause is a fact the code never stated: provenance is VIDEO-ONLY. Every audio and subtitle parser -- dts, ac3, adts, truehd, pgs -- builds its frames with `source: None`, while hevc, h264 and vc1 propagate it. So on that title the video track placed correctly by byte offset and nine audio and subtitle tracks arrived with nothing to place them by. The branch added to keep an unprovenanced frame from stranding did the stranding itself. It held such a frame on its track's current clip, reasoning that the cursor only ever advances under provenance and so could not be wrong -- which is true only for a track that eventually receives some. A track that never receives any stays pinned to clip 0 for the whole title, and every frame past clip 0's OUT mark is dropped. That is what the disc showed: nine tracks pinned to clip 0, the first drop nine milliseconds past its OUT. So the refusal goes back, now with the real reason recorded, and the test asserts it rather than asserting the behaviour that failed. Lifting it needs the audio and subtitle parsers to carry provenance first, stamped from the PES that STARTED each access unit -- the same rule au_assembly already applies to video. The diagnostics that found this are kept. |
||
|
Matthew Jackson
|
776a4fd6eb |
Say which frame and which marks caused a provenance drop
The heuristic path has always logged its first drop per track with the frame's timestamp and the clip marks it was judged against. The provenance path did not log at all, so a title that dropped MOST of its frames there failed the volume gate without a single line saying which frame, which clip, or which marks — a full-length rip produced zero drop events and an error, which is not a diagnosable failure. Also pins the all-identical-spans hazard as a test. Every PlayItem referencing one clip file gives every clip the same span, which passes the tiling check and is therefore "trusted" while carrying no information at all about which PlayItem a byte belongs to. That is not what the discs on hand do — theirs have one distinct span per clip — but the check conflates "the spans tile" with "the spans distinguish", and only the second justifies placing a frame by its byte offset. |
||
|
Matthew Jackson
|
a0f76a4f18 |
Say how many DISTINCT feed spans a title's clips share
A seamlessly branched title re-references one clip file from several PlayItems with different mark ranges, and those references share a single feed span because the bytes are read once. A byte offset alone cannot then tell them apart. Whether that is happening on a given title is the fact that decides whether provenance can identify a clip on its own, and it was not observable. distinct_spans < clips says it directly. |
||
|
Matthew Jackson
|
afa0a213d6 |
Say which placement strategy a title got
Whether a title was placed by the seam plan or fell back to inference is the single most useful fact about a branched rip, and it was invisible. The two looked identical in the output, so telling them apart meant rebuilding and re-ripping — which is exactly what it cost to find that most branched discs were silently on the inference path. Log both flags once at plan construction, and log each refusal with the reason that caused it. |
||
|
Matthew Jackson
|
e5f92e591a |
Place a restarting-clock playlist by provenance instead of refusing it
A playlist whose clips each restart their own STC has marks that all cover the same low values, so they are not points on one title-wide clock. from_clips refused those tables outright and fell back to PTS-jump inference. That refusal predates provenance, and it is only about inference. With the feed spans tiling the title the clip comes from the frame's byte offset and the marks are never read across clips: each clip's offset_ns maps its own private clock onto the output timeline, which is exactly the right operation for a restarting STC. Refusing unconditionally therefore turned the seam plan off on most of the branched discs on hand — the very titles it was written for. They ran on inference with a timeline minutes past the title's real length: one 2h29m title came out with its audio and video both spanning 3h15m. So the refusal now applies only when the spans cannot be trusted, where it is still load-bearing: without a usable clock or a usable byte offset there is nothing to place with. Accepting these tables opens one hole, and it is the same stranding the refusal guarded against. A frame arriving with no byte offset would fall through to the mark heuristics, which on this kind of table are meaningless and can strand a track on a clip it has already left, dropping the rest of the title. Such a frame now stays on its track's current clip; the cursor only ever advances under provenance, which is never wrong about which clip a byte came from. |
||
|
Matthew Jackson
|
79dbb2cf84 |
Resolve a frame clip by binary search, not a linear scan
The provenance lookup runs once per frame per track. A linear scan over the clip list is fine for the 11-PlayItem fixture and is not fine for the real hoard: two discs there carry 900 PlayItems in one title, and several carry 250-450, so a scan would be hundreds of comparisons on every one of millions of frames. spans_trusted already guarantees the spans tile the feed contiguously and in order, which makes this a partition point. A repeated clip reuses its first reference span, so the search walks back to the FIRST entry sharing it — the bytes are read once, so the material is emitted once, at that entry offset, and the answer is stable regardless of which duplicate the partition lands on. Also adds a test at the real 900-clip scale: every clip places, output stays monotonic across all 900, an arbitrary byte resolves to the right clip out of ascending order, and a byte past the end belongs to no clip. |
||
|
Matthew Jackson
|
b9ca75f471 |
Identify a frame clip by provenance, not by inferring it from timestamps
Four audit rounds each fixed one rule in SeamPlan::place and broke another, because the question the rules were answering has no answer. Inside a seamless-branching overlap clip k OUT comes AFTER clip k+1 IN — 57.8s of overlap on the real fixture table — so a single timestamp is legitimately inside two clips, and a clip file is not trimmed to its marks, so it also carries material from before its own IN. No rule over timestamps can say which clip a frame came from, and each attempt was right for one disc layout and silently wrong for another: 65s of rewind, 17 minutes stranded, 28 minutes dropped, 55s refused. Frames already carry the byte offset they were read from (PesFrame::source, stamped by the TS demuxer). Clip now carries the byte span its stream occupies in the title feed, recorded while the extents are gathered. So the clip is a LOOKUP: the offset falls in exactly one span. There is no decision to get wrong. Every track of a clip lives in the same stream file and therefore shares one span, so video, audio and subtitles agree by construction. Divergence between them — each track guessing separately under its own tolerance — is how audio and video ended up on different clips and drifted apart in the first place. spans_trusted gates the whole path: unless the spans tile the feed contiguously from zero, an offset means nothing and provenance is ignored in favour of the mark heuristics, which is the 1.6.0 behaviour. A broken map degrades instead of confidently selecting a wrong clip for every frame. A clip referenced twice reuses its first span (the bytes are read once) and is still trusted. Sources that stamp no provenance — a mkv:// remux, the deserialize hop — take the heuristics, which is what they have always used and where they have always been right, because they have no overlapping clips to be ambiguous about. 36 timeline tests, six of them new and covering: the overlap case marks cannot see, all tracks agreeing, out-of-marks material dropped AND counted, a holed span map, a discontiguous one, a repeated clip, and no provenance at all. |
||
|
Matthew Jackson
|
adc8ee37be |
Say when an entry is omitted for not being a plain file
A symlink to a directory — a normal way to keep tens of gigabytes of streams off the system disk — was dropped from the plan silently. `entry.file_type()` reports the LINK, so such a subtree never enters `dirs`, and `metadata()` then follows it and reports a directory, so it fails the is_file() test and was skipped with nothing said. The result is the worst class this release exists to close: PLAYLIST and CLIPINF still synthesize, so the folder scans and enumerates titles, every clip resolves to no extents, the mux takes its clean-EOF path, and a near-empty MKV is written at exit 0. It is still skipped rather than followed — following link targets invites cycles and escapes from the folder — but at the same volume as the unrepresentable-name skip eight lines above. |
||
|
Matthew Jackson
|
c3e82f28b5 |
Enforce the no-rewind invariant instead of guessing at clip crossings
Round 7 enumerated the state space of SeamPlan::place and found three more holes in the mark heuristics — the third consecutive round to find a defect here, each time in the guard the previous round added: - a clip file opening ON its own IN mark, after a tail frame past the previous clip OUT had already advanced the cursor, read as another crossing and jumped a clip too far: 28 minutes of clip 6 dropped; - the last clip was unguarded entirely (`clip + 1 < len` skipped the check), so the 9->10 seam rewound 75.6s with dropped_total() at 0 — the exact round-5 signature, reachable without any corrupt input; - a glitch mid-clip satisfies "inside the current clip", so the round-6 stranding survived for the rest of that clip. Three patches to the same heuristic, three wrong in a new way. So stop patching it. Everything above the placement is still heuristics over marks, but the property they exist to serve is now checked DIRECTLY: a track output must not run backwards. A placement that would rewind by more than DISCONTINUITY_BACKSTEP_NS is refused; the scan then looks for a LATER clip containing the frame that does not rewind, which is exactly what a genuine overlap crossing looks like; and if none exists the frame is dropped and counted, because emitting it anywhere moves the track backwards. The tolerance is the backstep constant because B-frame reorder legitimately emits out of order by a fraction of a second, while a rewind that matters is orders of magnitude larger. The cursor moves ONLY when a later clip actually accepts the frame. My first version of this advanced on a failed search, which strands a track exactly as before — caught immediately by the round-6 regression test. All 29 previous timeline tests still pass, including continuity_preserves_forward_gap. The three round-7 scenarios now have a test each, with the real 00801.mpls numbers. |
||
|
Matthew Jackson
|
6b67c52249 |
Round 6: a large backstep only crosses when it would be misplaced
The round-5 fix advanced the clip cursor on ANY backward step over 3s. That also fires for a corrupt PTS, and for a legitimate STC discontinuity inside one clip — and nothing moves the cursor back, because a forward step matches neither past_out nor stepped_back. Every later frame then sits below the new clip IN and is dropped: on the fixture table that is ~17 minutes of one track gone, and the only volume gate compares total drops against ALL tracks frames, so it exits 0. The branch now also requires the frame to be INSIDE the current clip marks, which is the only case that would otherwise be silently placed at the old offset — the rewind. A frame outside them needs no help: the containment check drops and counts it, the cursor stays put, and the next good frame is placed normally. Both behaviours have a test, each confirmed to fail without the guard. Separately, the dir:// PES input path scanned the folder and never applied the encryption verdict scan_dir exists to produce, so the same folder ripped through one door and failed through the other asking for a key it does not need. That logic now lives in one function, session::apply_folder_encryption_verdict, called by both. |
||
|
Matthew Jackson
|
dd9e92ed52 |
Retry a short read instead of skipping past it
The previous commit bounded the crack scan inspection by the bytes actually read, which stopped it examining stale buffer bytes from an earlier batch. But the cursor still advanced by the REQUESTED count, so those sectors were skipped outright — trading "scans the wrong data" for "silently scans less than it thinks", on exactly the damaged media where a title key is hardest to find. The cursor now advances by what was read, so the next iteration resumes where the read stopped. Floored at one sector so a source returning Ok(0) cannot spin. Found by reading the fix again rather than by the next audit round. |
||
|
Matthew Jackson
|
65ccbcbd92 |
Scan only the sectors the crack actually read
read_sectors returns the number of bytes written and a source may return Ok with fewer than asked — a recovery read over a damaged region does exactly that. The crack loop discarded the count and inspected all n sectors of a buffer that is REUSED across batches, so the tail still held the previous batch sectors. Cracking a key from those means cracking from data belonging to a different extent, possibly a different VTS, while crack_span records the CURRENT one. A key that opens nothing in this region is then installed, and its wrong descrambles are only partly caught by the per-sector crib. |
||
|
Matthew Jackson
|
aa149099ec |
Make the public per-sector descramble refuse a non-pack sector
descramble_region was fixed to require the MPEG-2 pack start code, but descramble_sector — the PUBLIC per-sector entry point, and the one the module-level example tells callers to use — still keyed on the byte 0x14 flag bits alone. The crate own documented guidance therefore led straight back into the defect this release exists to fix: a VIDEO_TS.IFO sector holding 0x15 at 0x14 while starting 00 26 00 00 loses 1912 of its 2048 bytes, and because that sector carries TT_SRPT the disc enumerates 38 titles while an image decrypted from it enumerates 10, at exit 0. It has no callers inside the crate, which is exactly why it survived three rounds: nothing exercised it. Putting the guard inside the function rather than in each caller is what keeps the safe path the easy one. The integration test that covers it built its sector from the flag byte alone, which no real scrambled sector looks like, so it stopped representing the path it names — the same fixture-realism gap already fixed in four other places this release. |
||
|
Matthew Jackson
|
7deacf1761 |
A clip join must never rewind the output timeline
A hole in the seam fix itself, found in audit round 5 and confirmed by measurement against the real 00801.mpls marks already in the test file. Nothing upstream trims a clip stream to its marks, so a clip file can open with material from BEFORE its IN. With the cursor on clip 5 (7708.99..7910.79) a frame at 7845.00 — clip 6 pre-mark lead-in, 8s below clip 6 IN of 7853.00 — sits further from that mark than the 250ms tolerance, so neither crossing rule fires. The cursor stays on clip 5, and 7845.00 IS inside clip 5 range, so the frame is PLACED with clip 5 offset. Measured: output went from 3714.51s to 3649.51s. Backwards 65 seconds, with dropped_total() still 0 — no counter, no gate, nothing noticed — while the whole 57.8s overlap band was emitted a second time over clip 5 written tail. That is the collide-and-flatten symptom this type exists to remove. A track own PTS only ever runs forward inside a clip, so a backward step larger than DISCONTINUITY_BACKSTEP_NS is always a clip change. Such a frame now advances to the first clip that could contain it — which terminates on its own, rather than running to the end of the list, which is what the next_in bound exists to prevent — and a frame still below that clip IN is material the playlist excludes, so the containment check drops and COUNTS it. All 27 pre-existing timeline tests still pass, including continuity_preserves_forward_gap, which is the one that catches an over-eager heuristic here. The new test asserts the output never moves backwards at a join and was confirmed to FAIL before the fix. |
||
|
Matthew Jackson
|
47789f71bc |
Test the demux seam gates too
The same gap as the MKV muxer: DemuxSink builds its timeline with TimelineContinuity::with_clips, but no test ever handed it a title carrying usable PlayItem marks, so frames_mapped and dropped_total() were always zero and both gates in finish() were unreachable. Deleting either left the whole suite green while a demux:// export of a seamless-branching title wrote a directory of zero-byte track files beside a populated chapters document and reported success at exit 0. The test gives the title two clips with marks that exclude every frame, and was confirmed to FAIL with the gate removed. |
||
|
Matthew Jackson
|
c31d9fc88e |
Test the seam gate that stops a drift-broken title being skipped
Audit finding: both seam gates in MkvMuxer::finish were dead code under test. set_clips was never called anywhere in the suite, so continuity.dropped_total() was always zero and neither branch could be reached. The ordering of the two zero-frame checks is load-bearing and the comment says so: a title the seam plan emptied also has a zero frame count, and MkvInvalid is classified by is_skippable_title_stub as an empty nav/menu stub — so reporting it that way makes an all-titles rip drop a real feature and finish the rest at exit 0. Swapping the checks left the entire suite green. The new test builds a two-clip plan (from_clips needs at least two, strictly increasing) whose marks exclude every frame, and asserts both that the error is SinkWroteNothing and that it is NOT classified as skippable. Confirmed to FAIL with the two checks swapped. |
||
|
Matthew Jackson
|
0e8c31a9c3 |
Round 3: fix three defects introduced by the round-2 fixes
Auditing my own fixes found all three. None were in the original code. The VTS crack sort was byte-wise case-SENSITIVE while the filters that select those files (vts_group_of / is_title_vob) are case-insensitive. On a case-sensitive volume a set holding vts_01_1.vob beside VTS_01_2.VOB sorted part 2 first, because V (0x56) precedes v (0x76) — reintroducing exactly the budget-exhaustion the ordering exists to prevent. Now sorted on the same uppercase normalisation the filters apply. Refusing a name that round-trips to empty aborted the WHOLE plan. A sidecar folder named with a single emoji made a backup un-rippable that 1.6.0 handled fine, and reported it as a collision with a file that does not exist. It is now skipped with a warning: the entry is unaddressable either way, but one irrelevant file should not cost the user their rip. The mtime check now applies only to files whose CONTENT the plan read — the IFOs, whose bytes 0xC0/0xC4 place every VOB. Everything else is planned from size alone, which is already checked, so comparing mtime there bought nothing and risked a real false positive: disc backups commonly live on exFAT/FAT32, which stores local time, so a long rip spanning a DST transition would see a whole-hour shift on an untouched multi-gigabyte VOB and abort hours in. |
||
|
Matthew Jackson
|
03d088abfc |
Catch names the reader cannot tell apart, and in-place content changes
Two dirimage findings, both silent-wrong-output. The per-directory uniqueness check compared raw host names, but the reader does not see raw host names: parse_udf_name trims leading and trailing whitespace and drops any code unit char::from_u32 rejects — which is every half of the surrogate pairs the encoder emits for non-BMP characters. So " 00000.m2ts" and "00000.m2ts", or "A<astral>.m2ts" and "A.m2ts", were two entries at plan time and ONE name at read time. find/read_file take the first match, so a title resolved to the wrong file extents and muxed the wrong bytes at exit 0 — the exact shadowing DirNameCollision exists to prevent. The key is now derived by round-tripping the name through the very encoder and parser that will be used, so it cannot drift from them. A name that survives that round trip as empty is refused outright: it would exist in the image and be addressable by nothing. The new test builds a real folder, calls plan, and was confirmed to FAIL with the fix reverted. Separately, the plan-vs-read revalidation compared file LENGTH only, while the plan depends on CONTENT: a DVD VOB placement comes from bytes 0xC0/0xC4 of its IFO, and IFOs occupy a whole number of sectors so an in-place rewrite keeps the length. A re-authoring tool touching the folder mid-rip would pass the size check while every title extent pointed at stale sectors. mtime is now compared alongside size, and only when both sides report one, so a filesystem without timestamps falls back to the old behaviour rather than failing every read. |
||
|
Matthew Jackson
|
84e0ba9fa8 |
Anchor the folder encryption probe on a unit boundary
probe_folder_encryption sampled AACS units starting from the largest extent anywhere in the disc. AACS units are 3 sectors and a unit boundary is only guaranteed at the START of a clip, so that anchor is only correct when the largest extent happens to be a clip first extent. For any clip over ~2 GiB it is not. The planner caps an allocation descriptor at MAX_AD_BYTES = 524287 sectors, so every full piece of a split file ties on sector_count and max_by_key returns the LAST tie — an extent beginning (k-1)*524287 sectors into the file. 524287 % 3 == 1, so that start misses the unit boundary for two file sizes in three. The 6144-byte sample windows then begin mid-source-packet and the byte read as the CPI flag is content. Both verdicts fail in a costly direction: a decrypted folder is rejected as encrypted (DirImageEncrypted on something perfectly rippable — the exact case scan_dir was added to rescue), or real ciphertext reads as clear and the mux writes it out as video at exit 0. is_unit_aligned cannot catch it: it measures against the same wrong base. Now the largest TITLE first extent, which is unit-aligned by construction and is also the more meaningful sample — the main feature rather than whichever fragment happened to be biggest. The existing tests could not reach this: their fixture m2ts is 786,432 bytes, a single extent, which is always its own first. |
||
|
Matthew Jackson
|
bc5e3453b4 |
Order a VTS crack by filename, not by directory order
Round 1 removed a largest-first sort from resolve_vts_key because largest-first is the 1.5.1 garbage bug. Auditing that fix showed it was only half right: `planned` comes from walking the UDF directory, which yields File Identifier Descriptors in on-disc authoring order with nothing sorting them. So deleting the sort did not restore playback order, it left the order undefined — whatever the disc happened to list first. DVD-Video numbers a title sets VOBs in playback order by spec (VTS_xx_1.VOB .. VTS_xx_9.VOB, single digit), so ascending filename IS playback order and is deterministic regardless of how the directory is laid out. Order decides correctness here: crack_key shares one sector budget across the whole extent list, a CSS DVDs biggest cell opens with a long clear run, and CSS recovers the title key from scrambled data itself. Starting in the wrong place can exhaust the budget without ever meeting scrambled data, whereupon the caller falls back to the disc-wide key and the whole VTS is descrambled wrongly — corrupt PES behind an intact header, written out at exit 0. |
||
|
Matthew Jackson
|
6d9791affc |
Audit round 1: playback order, silent title drops, image durability
Four fixes from the first audit round. Every finding was verified against a pinned tree and read directly before being accepted. resolve_vts_key sorted a VTS title-VOB extents largest-first. That is the 1.5.1 garbage bug, and it grew back in a new code path: the comment claimed it "matched the scan heuristic", but that heuristic WAS the bug and had already been fixed in decrypt_keys_for_title, which documents the rule (PLAYBACK ORDER, never largest-cell-first) and pins it with a regression test. A CSS DVDs biggest cell opens with a long clear run and crack_key shares one sector budget across the extent list, so starting there can exhaust it without ever MEETING scrambled data — and CSS recovers the key from scrambled data itself. The crack then returns None, the caller falls back to the disc-wide key, and every VOB in that VTS is descrambled wrongly: corrupt PES behind an intact header, written out as a complete extract at exit 0. parse_pgcit dropped titles silently in THREE places — an unparseable PGC, an out-of-range PGC index, and a truncated entry table. The finder caught one; the other two turned up on reading the function. parse_vmg already counts and warns per skipped title SET for exactly this reason, and this was the last place a disc could quietly report fewer titles than it has. write_image called flush() and returned Ok. flush() only pushes bytes into the page cache and promises nothing about durability, so a 6-90 GB image could be reported complete while still unwritten — a crash or an unmounted volume then leaves a truncated file the caller was told was finished. Now into_inner (so a buffered-write error surfaces instead of being dropped by BufWriter::drop) followed by sync_all. timeline used abs() on a saturating_sub result. Every other comparison in that module is saturating because the timestamps come off a disc and are not trusted; abs() panics on i64::MIN, which saturating_sub can produce. |
||
|
Matthew Jackson
|
d1551eb588 |
Rename is_scrambled to has_scramble_flag_bits
The 1.6.1 DVD fix removed the USE of the loose predicate but left the predicate sitting there with the better name. Anyone asking "is this sector scrambled?" finds is_scrambled before is_scrambled_pack, and reintroduces the defect that destroyed 1912 bytes of a real VIDEO_TS.IFO — the sector carrying TT_SRPT — so the disc enumerated 38 titles and an image decrypted from it enumerated 10, silently, at exit 0. Byte 0x14 only means "scrambling control" inside an MPEG-2 pack. In an IFO, UDF or ISO 9660 sector it is whatever that format stores there. The new name says what the function actually tests and nothing more, so the honest question has the obvious name and the dangerous one has to be asked for deliberately. Its doc comment also claimed a caller, decrypt::decrypt_sectors, that does not exist — so the name was an invitation and the documentation was an argument for accepting it. It has no production callers at all; it stays public because an integration test asserts the flag extraction directly. |