testlog: capture through a single global subscriber that routes each
event to a thread-local sink and returns Interest::sometimes, so
tracing's global per-callsite interest cache can never short-circuit a
live capture when another thread rebuilds it. The old scoped
with_default lost events at random in the full suite (an empty capture
failed the log-accounting assertions ~1 run in 15).
labels: capture the escaping test through testlog instead of installing
its own process-wide subscriber, which hard-cached every foreign
callsite "off" and poisoned the captures above.
dirimage: name scratch directories from a process-monotonic counter
rather than SystemTime, which resolves to only a microsecond — two of
the parallel tests sharing a tag drew the same value, collided on one
directory, and the first to finish removed it out from under the
other's reads.
Commit 4cd9b7b ("key the DVD crack on the disc, not on the container") fixed a
real bug — a decrypted HD-DVD hit E7023 (CssKeyMissing) because the per-title
CSS crack keyed on the MPEG-PS container, which DVD and HD-DVD share — but did
it by adding a required `disc_format: DiscFormat` parameter to the PUBLIC
`DiscStream::new` and `build_iso_pipeline`, a `disc_format` field to
`MuxInput::Iso`/`Live`, and `DiscFormat::may_have_css`, threading the axis down
through mux/driver and mux/resolve. That changed the public API and broke every
downstream caller's compilation (freemkv-engine's integration test now needed 8
args, autorip's MuxInput arms a new field). 1.6.4 shipped and worked with these
exact signatures; a bug fix must not reshape them, and needing a whole
disc-format plumb for HD-DVD was a code smell.
Revert all of that plumbing (public signatures restored to their pre-4cd9b7b
form; no `disc_format` parameter or field, no `may_have_css`, anywhere), and fix
the ACTUAL bug where it lives: the scramble-detection heuristic.
Root cause: `is_scrambled_pack` counted a sector as CSS scramble evidence on
pack-start (00 00 01 BA) + bits 4-5 of byte 0x14. Offset 0x14 is only the PES
scrambling-control field when the sector is a genuine elementary-stream pack. An
HD-DVD `.evo` RDI navigation pack is private_stream_2 (stream_id 0xBF), an
MPEG-PS pack exactly like a DVD VOB, whose byte 0x14 is raw nav payload that
routinely has bits 4-5 set. On a decrypted HD-DVD (None keys, MPEG-PS, so it
reaches the crack) those nav packs flipped the scan's `saw_scrambled` flag; the
crack then found no key — there is no CSS on an HD-DVD — and the scan returned
ScrambledUncracked, hard-failing a good disc with E7023.
Fix: exclude the MPEG-PS structural stream_ids CSS never scrambles — system
header (0xBB), padding (0xBE), private_stream_2 (0xBF) — by the stream_id at
offset 0x11. This is the refinement the DVD design notes already called for
("matches CrackTitleKey"). It needs no format plumbing because byte 0x11 lives
in the CSS-clear header (0x00-0x7F, untouched by scrambling), so it is the true
stream_id even on ciphertext. A decrypted HD-DVD now scans to Unencrypted and
muxes cleanly.
The DVD CSS crack is preserved and proven: a genuinely CSS-scrambled DVD sector
is always video (0xE0-0xEF) or private_stream_1 (0xBD), never an excluded id, so
its scrambled packs still set saw_scrambled and still hard-fail an uncrackable
disc — the "ciphertext muxed as plaintext at rc=0" catastrophe cannot slip
through. Red-before-green both directions: dropping the 0x11 exclusion turns the
decrypted-HD-DVD case back into E7023; inverting it (only nav ids count) turns a
real uncrackable DVD into Unencrypted and strands a crackable one. Both mutations
are caught by tests.
Gate (cargo +1.97): fmt, clippy --all-targets -D warnings, 3539 tests green;
freemkv-engine and autorip both compile against this tree again; precommit.sh
libfreemkv clean.
The same hole, two disc families, two answers — and the asymmetry is now a
decision written into both files instead of an oversight in one.
`file_extents` can return `Ok` and still yield no usable extent: an empty
allocation-descriptor list, or one every entry of which the `sectors > 0 &&
lba > 0` filter discards. An ordinary zero-byte file reaches it; no crafted
disc is needed.
On HD-DVD that was the flagship failure shape. The clip entered neither
`clip_extents` nor `unusable`, and nothing was logged, so the composer's
`any(|n| unusable.contains(..))` guard missed it while the
`filter(|n| clip_extents.contains_key(..))` beside it quietly deleted the
part: a `FEATURE_2.EVO` of size 0 next to a healthy `FEATURE_1.EVO` composed
a FEATURE title out of part one alone, still advertising the whole runtime,
at rc=0, in silence. Half a movie presented as a whole one. Round 1 accounted
for every `Err` from the resolver and left this route open. It now marks the
clip unusable and logs it under its own new code, E6019
(`E_UDF_NO_USABLE_EXTENT`) — deliberately not the neighbouring E6017, which
would file a zero-length file as an authoring hole and send whoever triages
it at the wrong population.
On Blu-ray the identical hole stays open, as previously decided, and the
reasons are now recorded on both sides. BD has no `unusable` set, so closing
it there means inventing a post-loop "every clip_id must appear in `spans`"
invariant that DROPS the title, and it is not settled that an empty-but-Ok
resolve is always a defect; dropping healthy titles is worse than the gap.
The consequence is milder too: on BD the clip is one PlayItem of an otherwise
whole title, on HD-DVD the feature is COMPOSED from parts. Same hole,
different price.
Also in this change:
* bluray: a non-absence SSIF failure that the `.m2ts` fallback papers over is
logged. `unresolved` had exactly one reader, `if let (None, Some(code))`, so
when `/BDMV/STREAM/SSIF/<clip>.ssif` failed with DiscRead /
UdfAdChainTooLong / UdfEmbeddedData and the base view then resolved, the
code was recorded and thrown away: the title shipped base-view 2D off a 3D
disc at rc=0 with no log at all. The site's own doctrine is "ABSENCE is the
only benign failure". Logged, not refused — the base view is a real rip.
* drive: `wait_ready` polled TEST UNIT READY through a bare `execute` and its
60 x 500 ms loop never read `self.halt`, so a Stop during spin-up did
nothing for ~30 s while every other drive path returns Halted at the next
command boundary. `spin_cycle` issued both START STOP UNIT commands outside
`checked_exec` and slept `SPIN_DOWN_IDLE_SECS` + `SPIN_UP_SETTLE_SECS`
blind — ~15 s deaf to Stop, from the recovery path, exactly when the
operator is most likely to press it. Both now use `checked_exec` and
`sleep_until_halted`, which already lived in this file with four tests and
was `#[cfg(test)]`, called from nowhere. It is production code again.
* drive: a READ(10) that returns GOOD status with a residual underrun was
correctly refused and logged NOWHERE, while the sibling `Err` arm warns with
lba/count/status. A residual-underrunning drive was indistinguishable from a
scratched disc — two populations with opposite remedies. It now warns with
transferred vs expected, which is the whole signal.
* error: `all_error_code_constants_are_unique` was a hand-maintained `vec![]`
naming 109 of the 127 declared codes while its doc claimed to pin them all,
and an earlier audit trusted that claim while assigning new ones. The list
is now derived from the declarations by parsing `include_str!("error.rs")`,
so a new constant is covered the moment it is written. A parser self-test
cross-checks the count and three known name/value pairs, so it cannot pass
vacuously.
* testlog: a test-only `tracing` capture (~120 lines, no new dependency) so
the logging contract is enforced rather than commented. Three sites carry
long comments insisting they log the error's OWN code; putting a literal
back broke nothing. They are pinned now, along with the two new log lines.
Captures are serialised process-wide: `tracing`'s interest cache is global
while `with_default` is thread-local, and the rebuild on the exiting
capture can land after the entering one's, leaving the cache at "never"
while a capture is live. That produced a real empty-event flake.
* disc: `scan_with`'s halt wiring for the BD and DVD enumerators had no test —
every BD/DVD cancellation test calls the scanners directly, so passing
`None` on either branch left the suite green while Stop did nothing.
* mux::network: `accept_from_rejects_stream_without_fmkv_header` half-closes
instead of `Shutdown::Both`, which raced an RST against the server's read
and returned ConnectionReset instead of InvalidInput under load. The port
was already ephemeral; that was never the cause.
Gate: fmt, clippy --all-targets -D warnings, and 3439 tests green on 1.97;
precommit.sh libfreemkv clean.
`resolve_dvd_title_key` decided whether to run the DVD CSS crack by asking
what CONTAINER it was looking at — `ContentFormat::MpegPs`. MPEG program
stream is what DVD and HD-DVD have in common, so every HD-DVD title was
run through a crack scan for a copy-protection scheme HD-DVD does not
use and cannot carry: AACS is its family, and CSS appears nowhere on the
disc.
Both outcomes of that scan were wrong. The cheap one wasted up to 50,000
sector reads per title. The expensive one returned `CssKeyMissing` —
E7023 — refusing a perfectly good HD-DVD with a CSS error, which is what
a real CI run produced on the HD-DVD fixture. Reading E7023 there sends
whoever triages it looking for a missing DVD key on a disc that never had
one, which is how a routing bug spends a day disguised as a key problem.
The right axis was already in the codebase and already used: `mux/resolve`
asks `disc.format == DiscFormat::Dvd`. This threads the disc format down
to the decision and adds `DiscFormat::may_have_css` to name the question.
The asymmetry decides the default, so it is worth stating. Running CSS on
an HD-DVD costs a wasted scan or a false refusal — visible, recoverable,
annoying. NOT running it on a real DVD muxes scrambled sectors as
plaintext and exits 0, which is the failure-that-looks-like-success class
this project has shipped once already. So `may_have_css` is false ONLY
for the families proven CSS-free, and `DiscFormat::Unknown` — a bare
reader with no scan behind it — still cracks. An `== Dvd` allow-list
would have read as tighter while silently stranding every caller that
cannot name its disc.
Both directions are pinned: an HD-DVD title must never enter the crack,
and an unknown-format title must still enter it. Removing the disc-format
clause fails the first and leaves the second and every DVD test green.
A cancelled scan was indistinguishable from a disc that simply holds
fewer titles, in all three tree enumerators. Once the operator presses
Stop, `Drive::checked_exec` fails EVERY subsequent SCSI command with
`Error::Halted` and `Drive::read` deliberately preserves the variant —
so a cancel is not one failed read, it is every remaining read failing.
* `scan_bluray_titles` returned a bare `Vec<DiscTitle>` and
`parse_playlist` an `Option`, so neither had a channel to report a
halt. During a cancel each remaining `.mpls` read merely failed and
was skipped by an `if let Ok(..)`, and the scan returned a TRUNCATED
title list at rc=0. Both now return `Result`; `Ok(None)` from
`parse_playlist` keeps its two benign meanings (unparseable MPLS,
playlist under 30 s) and `Err` means the scan is over.
* `scan_dvd_titles` was worse: `Err(_) => return Vec::new()` turned a
cancel into ZERO titles at rc=0 — a DVD reported as carrying no
video at all. `ifo::parse_vmg` compounded it, treating a halted
title set as one of the placeholder TT_SRPT entries it is designed
to tolerate and continuing to the next; both now propagate.
* HD-DVD was already correct and is the model this follows, including
its reasoning: a cancelled enumeration that returned `Ok` would be
cached, displayed and ripped from as if it were the disc.
The cancel is propagated at ALL FOUR Blu-ray read sites, not just the
obvious one — the playlist read, the `.clpi` read, and both
`file_extents` calls in the clip resolver — and the halt is polled
between playlists and again after the loop, so the final iteration
cannot slip a truncated list through. The resolver's `note` closure
previously EXEMPTED `Halted` from its disc-defect classification, which
was right (a cancel is not an authoring hole) but left it swallowed:
with no exemption and no propagation a cancel landing on `file_extents`
returned a title claiming its full runtime with that clip's bytes
silently missing. Measured with the fix reverted: `Ok(Some((768000,
[])))`.
Second defect, same function: the `.clpi` read and `clpi::parse` sat
inside `if let Ok(..) && let Ok(..)`, with extent resolution nested
inside. A scratched CLIPINF sector, a missing `.clpi` or a malformed one
skipped the whole block — no extents, nothing added to `total_size` —
but `duration_ticks` had ALREADY been summed from the play items, so the
title still advertised the movie's full runtime. The `if let` discarded
the error, so nothing was logged either: a failure that looks like
success, which is this crate's flagship defect class. It is now a
`match` that drops the title and warns with the error's OWN code, the
same classification the extent resolver below it already uses. Logging a
fixed code would account a scratched disc as an authoring hole.
Deliberately NOT added: a post-loop "every unique clip_id must appear in
`spans`" invariant. `file_extents` can also return `Ok(vec![])`, or a
vector whose every entry the `sectors > 0 && lba > 0` filter discards,
and it is not settled that an empty-but-Ok resolve is always a defect
rather than a legitimate healthy-disc state (zero-length placeholder
ADs). Dropping healthy titles is a worse failure than that residual gap,
so the route stays open, with a comment at the site recording the
decision so the next audit finds it instead of re-deriving it.
Tests: `parse_playlist_missing_clpi_yields_no_extent_no_size` asserted a
title WAS returned with `size_bytes == 0` and empty extents, and its
docstring quoted the buggy control flow as the specification — it
blessed the defect. Inverted and renamed. `parse_playlist_keeps_exactly_
30_seconds` encodes correct behaviour but used a fixture wiring no
STREAM/CLIPINF, so the `.clpi` fix would have dropped its title for a
reason unrelated to the boundary it pins; repointed at the fully wired
fixture rather than weakening the assertion. Four new halt-propagation
tests (BD playlist read, BD `.clpi` read, BD extent resolve, DVD title
set + VMG), each proven red before green.
Ten lenses over v1.6.4..HEAD, every claim read against the code before
it was believed. Seven confirmed; six are here, one is recorded for the
next round. All of these are the same family — a failure wearing the
shape of success — which is the family that once shipped 9 MB of
ciphertext inside a main-movie m2ts at rc=0.
A clip whose extents cannot be resolved is now accounted for, in both
disc readers. Only `UdfUnrecordedExtent` used to count: every other way
`file_extents` can fail — a scratched sector under the clip's ICB
(DiscRead), an allocation-descriptor chain that never terminated, a file
whose data is embedded rather than extent-mapped — fell through to the
ordinary "file absent" path. On Blu-ray that yielded a title advertising
its full runtime with a clip's bytes silently missing, because the size
and the play-item timing had already counted it. On HD-DVD it was worse:
the clip was never added to `unusable`, so a split feature still composed
from FEATURE_1 alone and offered half a movie as the whole thing. Neither
emitted a single log line. Absence is still benign — a 2D disc has no
.ssif and the extension fallback exists for exactly that.
`Halted` is excluded deliberately, and that exclusion is the whole reason
the first version of this fix was wrong. Cancellation makes EVERY drive
command return `Halted`; classifying it as a disc defect would have
dropped each remaining playlist in turn and handed back a truncated title
list at success — the same defect, wearing a cancel. `parse_playlist`
returns Option and has no channel to propagate a halt, so the existing
behaviour is preserved rather than made worse. Propagating it properly is
next round's work.
Both log sites now emit the error's OWN code instead of a hardcoded 6017.
Accounting a scratched disc (E6000) as an authoring hole would send
anyone triaging it looking for the wrong thing entirely.
AD type 3 is embedded data, not a descriptor list (ECMA-167 4/14.6.8).
`read_icb_extents` lumped it in with the reserved values and decoded the
file's own CONTENT as (length, LBA) pairs, manufacturing extents out of
arbitrary bytes and pointing the reader at unrelated sectors. This same
release already taught `read_directory` to honour type 3; this is the
file half of that decision. It is an error rather than an empty list,
because an empty list reaches the caller as a clip that contributed
nothing while its declared duration still counts it — the silent loss
pointed the other way. A legally zero-length embedded file still returns
an empty list. New code E6018: reusing DiscRead would have mislabelled a
deterministic structural property as transient I/O and fed the retry and
NonTrimmed machinery a byte that will never change.
`file_extents_addressing`, `extents_abs_at` and `AbsExtent` drop to
`pub(crate)`. The first hands back unrecorded extents UNFLAGGED, in a
shape identical to the safe call's return; its doc says callers must use
`file_extents` instead, but a doc comment is not a guard. No dependent
crate references any of the three.
Three tests close gaps the audit found, each proven red before green:
a held AC-3 access unit must not resume as a normal frame after its track
poisons; the PS resume cursor must survive a drain that rebases it (three
separate mutants caught); and AD type 3 must be refused rather than
decoded. The first attempt at the HD-DVD test passed with the fix
reverted, which made it worthless — it needed a VTI fixture before the
composition path ran at all.
Also: four error codes were missing from the uniqueness test that claims
to cover every published code, so a new variant reusing 6014, 6016 or
6017 would have passed it.
`dtolnay/rust-toolchain` is not a dependency in the sense Dependabot
means: it is versioned by the Rust release it installs, and the tag we
pin is the toolchain CI is pinned to on purpose — precommit.sh runs the
same one locally, so a lint that passes on a developer's newer default
cannot pass CI by accident.
Read as semver, those tags produced a 1.97.0 -> 1.100.0 proposal, a Rust
version that does not exist. Every such PR 404s on toolchain download
across all eight repos and regenerates weekly: eight permanently-red PRs
that promote.yml then has to special-case when it decides whether dev is
green.
freemkv already carries this ignore; this is the same block in the other
seven, so the fleet stays uniform. Bumping the toolchain stays a
deliberate, all-eight-repos change made by hand together with
precommit.sh.
Squashed from 12 commits. Every fix was proven red-before-green and killed by a
mutation; the reasoning for each is in the private audit record.
UDF and extents
Honour ICB types rather than assuming a Short AD, so an AD-type-3 directory
is no longer decoded from FID bytes into a silently empty listing. Carry the
ECMA-167 recorded flag through to the resolvers: an allocated-but-never-
written extent used to reach the read plan as ordinary content and splice
undefined sectors into the rip. file_extents now refuses such a file, and
only when the hole actually occupies byte space — a zero-length one displaces
nothing, and refusing on it dropped whole titles off discs that ripped
correctly. Type-2 sparse extents are kept alongside type-1; they were falling
into a catch-all that exited the descriptor loop and returned a truncated
list as complete. merge_ranges no longer claims a sector neither input
covered. A short skip or an over-long AD chain errors instead of truncating.
HD-DVD and Blu-ray scanning
Bound the XPL nesting depth, title count, clips and chapters per title, and
memoize the clip-name fallback probe — four separate amplification axes, each
of which alone left the worst case unbounded. The clip and title caps are 512,
~10x any retail disc, and a test pins the product of cap and probe budget.
The scan is cancellable: it returned Ok with titles carrying no streams when
halted, presenting a cancelled scan as a successful one. A clip dropped for an
unrecorded extent now says so.
Codecs and muxing
Resume a held E-AC-3 access unit rather than rescanning from its first frame,
and drop it on a discontinuity — a stale hold indexed past the end of the new
buffer. Map every ISO 639-1 code instead of collapsing fifteen languages to
und. Correct the DVD palette order. Detect a skip past EOF.
Drive and I/O
Classify dead-bus faults so the wedged-drive path can see them; a catch-all
arm had been flattening the variants before the classifier ran. A prefetch
producer that dies now reports SourceTerminated instead of Ok(0), which the
reader legitimately read as a short read and zero-filled — a whole title
could be fabricated and the pass reported complete.
Also: charge Ok(0) reads to the CSS crack budget, drop the unreachable soft
re-crack, and send disc-derived strings to logs through the debug formatter so
a crafted label cannot paint an operator's terminal.
`resolve_vts_key` called the `Option`-returning `css::crack_key`, which
collapses "no scrambled sector was seen" with "scrambled sectors were
seen and no key came out", and then fell back to the disc-wide key. A
multi-VTS CSS DVD whose second title set resists the Stevenson scan was
descrambled under the FIRST set's key: corrupt PES behind an intact
header, written out with `complete = true` at exit 0.
`CrackOutcome` exists precisely to keep those two apart, and its doc says
callers must surface the second as a hard error. Every sibling path in
the crate already does — `Disc::decrypt_keys_for_title` and the mux path
both map `ScrambledUncracked` to `CssKeyMissing`. This was the one path
that did not. The ordering comment 15 lines above describes this exact
outcome as the bug it exists to prevent; ordering makes the crack far
more likely to succeed, but it cannot make a failed crack safe.
Also, three things nothing could catch:
- The demux output filename took the stream's language raw while the
base beside it was sanitised. A language code is three raw STN bytes
through `from_utf8_lossy`, and `00 00 00` is the ordinary "undefined"
encoding on real discs — a NUL in a path fails `File::create` with
InvalidInput, taking the whole export down before one track file
opened. `sanitize` now maps control characters too; it did not.
- `css::crack_key_scan`'s short-read handling was dead code under test:
every source in the module returned the full request, so reverting
`advance` to the requested count, or dropping the `.max(1)`, left the
suite green. The `.max(1)` is load-bearing — without it a source
returning `Ok(0)` never moves the cursor and never increments the
budget, so the scan spins forever. That mutation now HANGS the test
rather than failing it, which is the honest demonstration.
- `MAX_SUBDIRS`'s const-assert carried `#[cfg(not(test))]` inside a
`#[cfg(test)] mod tests`, so it compiled in no configuration and could
never fire — the dead gate the test above it was written to replace.
Moved to module scope, and verified it now rejects a wrong constant at
compile time.
`forced_sub` was the last unbounded attacker-controlled list in
paramount.rs. `MAX_COM_INDICES` capped the two `*_com1_idx` attributes;
this one had nothing capping it at all, and unlike them it has no value
to filter — a cell is a classification of the position it sits at, so
its bound has to be positional. Extracted as `forced_subs` for the same
reason `com_indices` was extracted: through `labels_from_feature` the
bound is unobservable, because the subtitle loop cannot reach those
cells either, so the assertion could not fail.
The `MAX_COM_INDICES` doc claimed the entry-allocation argument for the
whole constant. It is the VALUE filter that caps the set (values below
the bound, so at most that many distinct entries, however long the
attribute); the `take` caps the WORK. Both are real and they are not
the same bound; the doc now says which is which.
`jar_inventory_dedup_is_not_quadratic` called itself proof by deadline.
It is a hang guard — a return to the linear scan runs for minutes and
would wedge CI rather than fail it — and no assertion in it can tell a
BTreeSet from any other sub-quadratic dedup. Renamed and documented for
what it does. Its margin was measured before keeping it: 0.14s debug /
0.07s release against 10s, ~70x, unlike the 6x that made paramount.rs's
wall-clock test flake.
Both are mine, and both pass with the bound they "cover" deleted.
`a_hostile_commentary_index_list_is_bounded_not_merely_fast` asserted
only label purposes. The out-of-range filler it feeds is unobservable at
the label level and a HashSet collapses the repeats, so removing
MAX_COM_INDICES entirely leaves it green. What it DOES catch is a bound
set too low — verified at 2, where the real indices stop resolving. Named
and documented for that, and it no longer implies it guards enforcement.
`an_index_that_cannot_address_any_cell_is_not_retained` asserted through
the labels, where retention is by definition unobservable: the loop never
queries a cell that high. It now reads the set through `com_indices`,
where the claim is checkable.
Enforcement was and remains proven by
`distinct_unaddressable_indices_are_refused_not_stored`, which was red at
50,000 retained entries and green at zero. Two tests, two properties;
neither pretends to the other's job now.
The gate caught `commentary_index_lookup_is_not_quadratic` failing, then
passing on a re-run. Measured both ways: 1.62s alone, OVER 10s against
its own 10s deadline while the suite's other 3,347 tests ran
concurrently. A 6x margin against a shared CPU is not a margin, and a
test that gets re-run until it passes is not a test.
It was also measuring the wrong thing. Replacing the linear scan with a
HashSet bounded the LOOKUP; the set was still built from every entry the
disc declared. `playlists.xml` is attacker-controlled and has no length
of its own, so a hostile disc could still force an unbounded allocation
before any lookup happened — the parse, not the query, was the exposure.
The code's own comment said "unbounded parsed input" and only fixed half
of it.
`MAX_COM_INDICES` bounds both halves, at the one value that cannot
change behaviour: an index at or beyond `u16::MAX` can never match a
cell, because the labelling loops break at `u16::try_from(i + 1)`. Real
authoring is nowhere near it — the BD STN table admits 32 streams.
The parse moved into a `com_indices` helper so the bound is OBSERVABLE.
Through `labels_from_feature` it is not: a HashSet collapses repeated
values, and an out-of-range index changes no label, so the obvious test
passes with or without the cap — an assertion that cannot fail, which is
what the first draft of this fix shipped. The test now hands in 50,000
DISTINCT unaddressable indices and asserts they are refused. Proven red
with the bound removed (50,000 kept), green with it.
Applies to both `sub_com1_idx` and `aud_com1_idx`.
NOT done here: `labels/mod.rs:1421`'s `jar_inventory_dedup_is_not_quadratic`
is the same wall-clock shape and has the same flakiness. Named so it is
not lost.
Three round-1 audit findings on v1.6.0..HEAD.
Tests: `mux::resolve::input("dir://…")` — the door the CLI rips a folder
through — was never driven with a real folder. The only dir:// input test
uses a missing path, which fails in DirImage::open long before
`image_input`, so dropping the `is_folder` argument (and with it
`session::apply_folder_encryption_verdict`) passed the whole suite. Two
tests now close it, stated as a DIFFERENTIAL between the two doors,
because that is the invariant the shared function exists to hold: a
clear folder that kept its AACS/ directory must scan the same and select
the same extents through `scan_dir` and through `dir://`, and a
scrambled one must be refused through both with E9063. Proven red
against a mutated guard (both fail with E7022, the tree-shape verdict
the probe overrides) and green with it restored.
Docs: css/mod.rs documented `is_scrambled`, a function this line renamed
to `has_scramble_flag_bits`. Eleven stale sites, two of them broken
rustdoc links. Two were not stale names but false statements — they said
the descramble loop keeps the looser raw-flag test, when
`descramble_sector` and `descramble_region` both gate on
`is_scrambled_pack`; that parenthetical is rewritten to say what the
code does and why (the measured VIDEO_TS.IFO case: 38 titles became 10,
silently, at exit 0). The one mention that must stay is the sentence
explaining why the name was rejected.
Constants: io::image_writer and dirimage::encode each re-declared the
2048-byte sector that consts::SECTOR_BYTES already exports; both now
alias it. Note `BATCH_SECTORS = 2048` in image_writer is a different
quantity (sectors per batch, not bytes) and is deliberately left alone.
Bare 2048 literals elsewhere in the crate are out of scope here.
These are eight public repos that ship one product on one version, and
they had drifted: freemkv-engine carried no LICENSE at all (GitHub
reported its licence as none), two crates had no code of conduct, four
had no contributing guide, and none had a security policy — so there was
no private route to report a vulnerability in a disc-decryption tool.
SECURITY.md names GitHub Security Advisories on each repo rather than an
address, so there is nothing to keep in sync and no inbox to go stale.
dev is where work lands and is meant to be pushed to often, so what runs
there should be the cheap answer to "did I break it": lint, tests and the
Linux build. The macOS and Windows jobs now run on qa and main instead of
on every push to dev.
Nothing is deleted and no platform goes unchecked before a release. qa.yml
already covers macOS and Windows independently, and the jobs that live only
here -- the Intel macOS build, the Windows release build -- still run, on
the branches where a cross-platform break is worth blocking on.
They are SKIPPED on dev via `if`, not left unscheduled. A queued job would
be worse than a slow one: release.sh's CI gate refuses while any run for
the commit is still in progress, so a job that never gets a runner blocks
releases silently, with no error anywhere. That is the failure the
real-media note in qa.yml describes, and it is why this is an `if` on the
job rather than a narrower set of trigger branches.
Version bump and changelog for the 1.6.3 sync. All eight crates ship the
same version, so the crates with no functional change this cycle say so
rather than carrying an empty section.
Two problems, both invisible until the whole graph is looked at together.
DEAD: num-bigint, sha2, num-traits, num-integer, cmac and cbc are
declared here and referenced nowhere -- not in src, tests or benches.
They were being compiled, audited and offered version bumps forever for
no reason. Removing beats bumping.
cbc nearly survived the sweep: a substring search for "cbc" matches 44
occurrences of ycbcr_to_rgb in the DVD subtitle decoder, so it looked
used. Only a word-boundary search exposed it.
SKEW: this crate was the outlier on every shared dependency -- aes 0.8,
rand 0.8, base64 0.22.1 and zip 2 against 0.9 / 0.10 / 0.23 / 8
elsewhere. Cargo cannot unify across a major version, so it compiled
BOTH: 32 duplicated crates in the freemkv binary's graph, including two
complete AES implementations (aes 0.8 + 0.9, cipher 0.4 + 0.5), two
digest stacks and two getrandom. Two crypto stacks in one product is
worth removing on its own.
The aes bump is an API rename -- BlockCipher-prefixed traits, Array for
GenericArray -- and the obvious translation uses Array::from_slice,
which the new version deprecates and clippy's -D warnings would reject.
These use the From<[T; N]> conversion the crate points at instead.
3441 tests pass in debug and release. The AACS crypto here is covered by
known-answer tests, so a byte-order or sizing mistake in that rename
could not have passed.
Version updates were never configured here -- only security alerts, which
report but never open a pull request. So the absence of Dependabot PRs was
not "nothing to update", it was "nothing was checking".
Updates target `dev`, never `main`. main is a release pointer that
release.sh moves to each tag, so a bot commit on it would put work there
that no tag contains -- the exact state that aborted the 1.6.2 cascade at
its last step, and pointing a scheduled bot at main would recreate it
weekly.
Minor and patch bumps are grouped into one PR rather than one per crate:
eight repos times a handful of dependencies is a volume nobody reads, and
an unread PR queue is indistinguishable from no updates at all.
The freemkv crates are excluded. They depend on each other by git tag,
re-pinned by release.sh inside the release commit, and Dependabot cannot
see that cascade -- a PR bumping one could pin a version whose tag does
not exist yet.
It fired on `release: published` and committed to main, the branch the
release points at its tag, so it could only ever leave main ahead of the
tag. release.sh now makes the same edit inside the release commit.
In this repo it was a no-op regardless: the pattern it rewrote does not
appear in this README, so it has been running and changing nothing.
A clip's span is measured over the title's full extents when the disc is
scanned, and a frame is placed by the offset it was read from. Those only
agree while the mux reads every byte the scan counted.
On a disc carrying a forensic segment it does not: the read plan omits the
units belonging to another device group, so fewer bytes are fed than the
spans describe, and the shortfall grows through the title. Every frame past
the first segment then looks earlier than it is — placed in a clip it did
not come from, or dropped at a join for failing marks it was never inside.
The spans still tile one another perfectly, so the check that asks whether
they can be trusted cannot see any of it.
The plan is now compared against the extents it was built from. When they
differ the spans are dropped and placement falls back to timestamps, which
is what that path is for and what the surrounding comment already promised
would happen when an offset stops meaning anything. Ordinary discs are
untouched: with no forensic segment the plan IS the extents.
The provenance guard knew one way to lose a frame's source offset —
writing it as absent. Omitting the field entirely does the same thing,
because the frame type fills it in by default, and the guard read straight
past that. A parser rewritten into the second spelling would have kept its
green light while its track silently went back to being placed by guesswork.
It now reports which spelling it found and where.
The title-count clamp was asserted over a fixture too small to hold more
titles than the cap allows, so the walk stopped when the buffer ran out and
the clamp was never what bounded it — the assertion held with the clamp
deleted. The fixture now carries more entries than the cap.
The subdirectory limit was checked by restating the constant's own
definition; the guard itself had never run, and deleting it changed
nothing. The limit is lowered under test so a real folder can exceed it,
and the test now walks one and requires the refusal.
One test also carried two unrelated grounding notes while the test they
described had none, so reading the note above a test told you about a
different one.
A playlist may point at the same .m2ts from several play items — a
seamless split, a looped segment, multiple angles. The file has one set of
bytes, so it has one span, and the byte a frame was read from therefore
identifies the FILE, not which play item's range it falls in. Placement
took the first of them and judged every frame against its marks, so
everything past that range's end was treated as material the playlist
excludes and dropped: half the clip missing from the rip, with the
timeline still charged for its full duration.
The offset narrows a frame to the run; only its timestamp can finish the
job, and each play item carries its own marks. A frame is now matched
against the marks of the entry it actually falls in, falling back to the
first when it falls in none, which leaves genuinely-excluded material
dropped as before.
The test for this construction asserted only that such a playlist is
trusted, and never placed a frame from the second range — so the loss it
described in prose was invisible to it. It now places one from every range
and requires them all to survive.
The deferred-mux replay of buffered frames wrote them without their source
offset, quietly sending the head of every such title down the timestamp
heuristic the rest of this work exists to retire. It passes the offset
through now, and the frame writer that omits it is compiled out of the
library entirely: no production path can discard provenance any more.
Also: the AC-3 buffer-reuse test could not see the regression it named.
Feeding equal-sized packets, a fresh allocation per call yields the same
capacity as a reused one. It now feeds a large packet then a small one,
where only a reused buffer keeps the larger capacity.
An ADTS header that declares a CRC follows must be at least nine bytes —
seven of header plus the two the CRC occupies — because the declared frame
length counts them. The structural gate compared against a flat seven and
never read the bit that says whether a CRC is there at all, so a frame
whose own header describes something impossible was accepted and handed to
the muxer as decodable.
The pipeline's spawn doc named Sweep, and the thread Sweep would have
created, as callers to look for. Neither has been in this crate since the
recovery passes moved out. The same paragraph already records fixing this
once, for a different departed caller — it simply drifted again a sentence
later, so it now says to name callers that live here or name none.
Whether a disc is structurally AACS-encrypted was spelled out by hand in
both the fast identify and the full scan. They agreed today; nothing made
them agree tomorrow, and disagreeing would mean the same disc reported
encrypted by one path and clear by the other. There is one definition now,
and the comment that pointed at it by line number points at its name.
The AC-3 parser built a fresh buffer on every packet — of the order of a
hundred thousand times per title — to work around a borrow it cannot
avoid. The copy stays; the allocation does not. The buffer is now lent out
and handed back, and a test pins that, because reverting it would be
invisible in behaviour.
Left alone deliberately: send/send_with_halt and finish/finish_with_halt
look like one action under two names, and are not. After the consumer
fails, one must still accept items and the other must refuse them; that
difference is what stops a producer reading an entire disc for a write
that died on its first frame. Collapsing them was tried here and the
existing test caught it. Both now say so where the choice is made.
A recovery resumed against an image that has been truncated since the
previous pass — a full disk, an interrupted transfer, a remount — cannot
repair it: the pass only revisits the ranges the recovery data calls bad,
so everything past the cut stays a hole while the counts still describe a
whole disc. There was no way to say that. Reusing the recovery-data error
would have been wrong, because the recovery data is intact; the image is
not.
Registered in the uniqueness list, the code table and the range check, so
a future variant cannot silently reuse 6015 or map to the wrong code.
The `consumers` job checked out all five dependents against this commit and
ran `cargo check --all-targets`. That proves they still COMPILE, which
catches a changed signature and nothing else. The failures worth catching
here keep every signature intact and change behaviour: the library still
builds, the dependent still builds, and the dependent's tests are what go
red. Those never ran.
Run their suites instead. Same checkouts, same patched paths — only the
verb changes.
The demuxer interleaves, so at a boundary the streams do not reset on the
same frame: audio for the next segment can reach the muxer before the video
frame that opens the epoch it belongs to. Only the primary video track may
open one, so those frames rode the just-ended offset and landed a whole
segment in the past. Downstream the strictly-monotonic block nudge then
crushed the entire run onto one instant a tick apart, which is audible.
Such a frame is recognised on its OWN raw PTS, which within an epoch only
advances, so a large backward step is unambiguous — a different signal from
the shared frontier, which is what the old false-positive ratchet keyed on.
It then rides a provisional offset private to its track, computed the same
way the video path computes a real one, and drops it the moment the video
retires an epoch, so the run rejoins with no seam.
The offset is deliberately private: it never writes offset_ns, never
advances the frontier and never retires an epoch, so it cannot move the
video timeline. Letting a passive track open a REAL epoch was tried first
and inflated a 476.776 s title to 656.216 s, because every track meets a
boundary at its own pace and the video path rebased again on top.
Measured on a real DVD title with 8 cell boundaries: audio frames stamped
inside one cadence 120 -> 0, outsized gaps 7 -> 0, subtitle 1 -> 0, and the
video span byte-identical at 476.484. On the HD-DVD title the remaining
boundary gap on its first audio track falls from 0.999 s to 0.275 s.
Acceptance: 76 pass, 0 fail, 0 skip.
A non-video frame can reach the muxer after the video that opened the next
epoch has already been processed. It carries an old-epoch PTS, so adding
the new offset flings it forward by a whole clip. The remap that exists to
catch this compared the frame against the CURRENT frontier and only
accepted it within one backstep, which is the wrong yardstick: by the time
a straggler is seen the frontier has moved on into a new epoch, and how far
below it the frame lands says nothing about where it belongs.
Retiring an epoch now records the frontier it closed at alongside its
offset, and a straggler is judged against the end of the epoch its PTS came
from. Keeping the whole history rather than one previous offset also fixes
the case a single `prev_offset_ns` cannot express at all: a frame from two
or more epochs back.
Measured on a real HD-DVD title whose second audio track put its final
packet at 12834.587 s in a 6434.100 s file. That frame sits 23 s below the
current frontier — outside the old window, so it was refused — but 0.15 s
below the end of the clip it actually came from. It now lands at 6417.216,
one 32 ms cadence step after its neighbour, and the track spans 6416.160
with no outsized gaps. The other two streams are unchanged.
A title that never rebases retires no epoch, so the lookup finds nothing
and the mapping is bit-identical to before.
Four parsers (dvdsub, flac, lpcm, mpegaudio) stamp a source offset on
every frame but had no test that read one back, so a regression to
`source: None` would have been caught only by the brace-balanced audit
in codec/mod.rs — a lint, not a behavioural check. Each now asserts the
emitted frame carries the offset of the packet that supplied its first
byte.
The Blu-ray feed spans had no direct test at all. Add one that walks a
multi-item playlist and requires the spans to tile the feed with no gap
or overlap; it catches a one-sector-per-clip drift, which is exactly the
error class that would misattribute frames near a seam.
`no_provenance_still_places_by_marks` asserted only that placement
returned something, which passes for a frame placed in the wrong clip.
Its probe timestamp lands in an overlap between two clips in the real
mark table, so pinning one clip would assert a coin-flip; instead
require the offset to be one that a clip actually containing that
timestamp would produce.
The TT_SRPT pointer at 0xC4 is a SECTOR offset from the start of
VIDEO_TS.IFO. Reading it as a byte offset lands 1/2048th of the way in and
does not error — it silently returns a different title list, which is how
a disc enumerating 38 titles and an image enumerating 10 looked like a
scan difference rather than a failure.
The table already had tests for its entry layout, dedup and the 99-title
clamp. What it did not have was anything pinning the offset unit itself,
or the grouping of titles by title set, or a truncated table keeping the
rows it can read rather than failing a whole disc.
release.sh dates the heading when it cuts the tag, and refuses a bare
'## [1.6.1]' because it cannot tell an undated heading from one it has
already stamped. Shipping a tag whose public changelog says UNRELEASED is
permanent and cannot be corrected inside that tag, so the check is right
to stop rather than guess.
Every emitted frame must carry the source byte offset of the packet it
came from. That held only for video for as long as it existed, and
nothing asserted it, so nothing caught it: ten parsers built frames with
`source: None` and a multi-clip title could not place audio or subtitles
by byte at all.
Finding them took a brace-balanced scan of the tree by hand — a regex
cannot do it, because a Frame literal contains nested braces and a
non-greedy match stops at the first `}`, which is how five sites survived
the first pass. This is that scan, as a test.
A second test fails if a codec module is added and not listed, since an
unchecked parser is exactly how the gap persists. Modules that emit no
Frame are named explicitly rather than skipped silently.
The guard tripped on itself twice while being written — first on its own
doc comment, then on its own string literals — so it strips line comments
and excludes its own module. A check that matches prose about the defect
rather than the defect is the same mistake in a different place.
The entries had grown into narrative paragraphs; each is now the outcome
first and the mechanism in a sentence or two. The website changelog page
mirrors libfreemkv's, so it inherits this directly.
Two things that shipped in this cycle were not in it at all:
dir:// as a source. It was pulled from 1.6.1 on 2026-08-05 because the
folder reader produced a wrong title list on a real DVD. That defect was
the CSS descramble bug below, fixed since — the acceptance suite now
shows a folder reporting the same 38 titles as its ISO, with matching
streams, languages and runtime, and the CLI has been wired for it all
along. It ships, so it is listed.
The CSS title-table corruption itself: a disc enumerating 38 titles
produced a decrypted image enumerating 10, silently, at exit 0. That is
a data-integrity fix a user needs to know about.
A real rip caught these. The warning added for a track placed from
timestamps under a seam plan fired once, on one subtitle track of a
23-clip title, eighteen minutes in — a display set that arrives as a lone
non-PCS segment is emitted straight through rather than accumulated, and
that path still built its frame with source: None.
Auditing the rest the same way found four more: flac, lpcm, mpegaudio and
the passthrough parser. All are one-PES-one-frame, so the packet's own
facts are the unit's facts.
Every Frame construction in every codec parser now carries a source,
checked by walking balanced braces rather than by eye — the earlier
count was taken with a regex that cannot see a block containing nested
braces, which is how these survived the first pass.
Marks that do not advance across a title are normal, not a defect: each
clip file carries its own STC, so one clip's IN has no ordering
relationship to the previous clip's. Refusing those tables dropped
exactly the branched titles this type exists for onto the inference path
that cannot read them, and that is the overrun.
Lifting it was tried once before every track carried a source offset, and
nine audio and subtitle tracks with nothing to place them by pinned
themselves to clip 0 and dropped most of the title. Every parser stamps
provenance now, so the clip comes from the byte offset and the marks keep
one job: whether a frame lies inside its own clip's [in, out].
Still gated on spans_trusted. Without usable spans there is no offset to
place by and inference is all that is left, so a table inference cannot
read is still refused rather than silently truncating a title.
A frame reaching the timestamp path under a plan now says so once per
track. It is not expected any more, and it is how a mostly-dropped track
reached a user without a single line in the log.
dts, ac3 and truehd assembled access units across PES packets with three
private implementations of the same bookkeeping, and none of them carried
the source byte offset. They now hold a PesBuf, so a unit takes the
timestamp AND the source of the packet covering its first byte, from the
same mark, and no codec can answer that question its own way again.
dts is the reference case: its pts_marks already implemented the rule
correctly, and all 61 of its existing tests -- including the PTS
attribution ones -- pass unchanged on the shared type. That is the
evidence the type preserves the behaviour dts had right.
ac3 kept a single carry-over timestamp and one anchor offset, so it could
only attribute the first unit in a call; it now resolves each unit at its
own offset. truehd had no attribution at all beyond a running clock.
New tests cover the case that motivated this: a unit whose first bytes
arrive in one packet and whose remainder arrives in the next keeps the
FIRST packet's offset. At a clip seam those two packets belong to
different clips, and taking the later one places the audio in the wrong
one.
Clippy on the pinned toolchain caught an empty `if` block left where dts
used to clear stale marks -- restored as an explicit clear, with why it
is still needed once drain keeps the covering mark.
A parser that assembles an access unit across PES packets has to answer
one question for every unit it emits: which packet carried this unit's
FIRST byte? Its timestamp comes from that packet, and so does the source
byte offset that says which clip of a multi-clip title it belongs to.
The packets that complete the unit carry later values that must not
override it.
That question was being answered three ways. DTS kept a deque of
(offset, pts) markers and took the one covering offset 0. AC-3 kept a
single carry-over timestamp. TrueHD kept its own. PGS and DVD subtitles
each held a pending unit with just a start time. None of them carried
the source offset at all, which is why provenance existed only for video
and why nine audio and subtitle tracks on a branched title had nothing
to place them by.
So it lives in one place now. PesBuf owns the bytes AND the marks;
PesFacts returns a packet's timestamp, source and discontinuity
together, so a parser cannot take one from one packet and another from
the next, because it does not assemble them itself.
The type answers WHICH packet. How a codec reads a timestamp out of that
packet stayed the codec's business at first, and that turned out to be
the same drift one level down: dvdsub read pts alone and returned 0 for
a packet carrying only dts, while everything else took pts.or(dts). Now
there is one derivation. It is not a choice between two fields — for
audio and subtitles there is no reordering, so dts IS the presentation
time and reading it is reading the same value from whichever field the
packet used. Reordering video never calls it; that path reconstructs
display order instead.
Migrated: adts, pgs, dvdsub. dts, ac3 and truehd follow.
Accepting a table whose marks do not advance was wrong, and a real
22-clip title showed exactly how. The rip failed the drop-volume gate at
84 percent with more frames dropped than kept.
The cause is a fact the code never stated: provenance is VIDEO-ONLY.
Every audio and subtitle parser -- dts, ac3, adts, truehd, pgs -- builds
its frames with `source: None`, while hevc, h264 and vc1 propagate it.
So on that title the video track placed correctly by byte offset and
nine audio and subtitle tracks arrived with nothing to place them by.
The branch added to keep an unprovenanced frame from stranding did the
stranding itself. It held such a frame on its track's current clip,
reasoning that the cursor only ever advances under provenance and so
could not be wrong -- which is true only for a track that eventually
receives some. A track that never receives any stays pinned to clip 0
for the whole title, and every frame past clip 0's OUT mark is dropped.
That is what the disc showed: nine tracks pinned to clip 0, the first
drop nine milliseconds past its OUT.
So the refusal goes back, now with the real reason recorded, and the
test asserts it rather than asserting the behaviour that failed.
Lifting it needs the audio and subtitle parsers to carry provenance
first, stamped from the PES that STARTED each access unit -- the same
rule au_assembly already applies to video. The diagnostics that found
this are kept.
The heuristic path has always logged its first drop per track with the
frame's timestamp and the clip marks it was judged against. The
provenance path did not log at all, so a title that dropped MOST of its
frames there failed the volume gate without a single line saying which
frame, which clip, or which marks — a full-length rip produced zero drop
events and an error, which is not a diagnosable failure.
Also pins the all-identical-spans hazard as a test. Every PlayItem
referencing one clip file gives every clip the same span, which passes
the tiling check and is therefore "trusted" while carrying no
information at all about which PlayItem a byte belongs to. That is not
what the discs on hand do — theirs have one distinct span per clip — but
the check conflates "the spans tile" with "the spans distinguish", and
only the second justifies placing a frame by its byte offset.
A seamlessly branched title re-references one clip file from several
PlayItems with different mark ranges, and those references share a single
feed span because the bytes are read once. A byte offset alone cannot
then tell them apart.
Whether that is happening on a given title is the fact that decides
whether provenance can identify a clip on its own, and it was not
observable. distinct_spans < clips says it directly.
Whether a title was placed by the seam plan or fell back to inference is
the single most useful fact about a branched rip, and it was invisible.
The two looked identical in the output, so telling them apart meant
rebuilding and re-ripping — which is exactly what it cost to find that
most branched discs were silently on the inference path.
Log both flags once at plan construction, and log each refusal with the
reason that caused it.
A playlist whose clips each restart their own STC has marks that all
cover the same low values, so they are not points on one title-wide
clock. from_clips refused those tables outright and fell back to PTS-jump
inference.
That refusal predates provenance, and it is only about inference. With
the feed spans tiling the title the clip comes from the frame's byte
offset and the marks are never read across clips: each clip's offset_ns
maps its own private clock onto the output timeline, which is exactly
the right operation for a restarting STC.
Refusing unconditionally therefore turned the seam plan off on most of
the branched discs on hand — the very titles it was written for. They
ran on inference with a timeline minutes past the title's real length:
one 2h29m title came out with its audio and video both spanning 3h15m.
So the refusal now applies only when the spans cannot be trusted, where
it is still load-bearing: without a usable clock or a usable byte offset
there is nothing to place with.
Accepting these tables opens one hole, and it is the same stranding the
refusal guarded against. A frame arriving with no byte offset would fall
through to the mark heuristics, which on this kind of table are
meaningless and can strand a track on a clip it has already left,
dropping the rest of the title. Such a frame now stays on its track's
current clip; the cursor only ever advances under provenance, which is
never wrong about which clip a byte came from.
The provenance lookup runs once per frame per track. A linear scan over the
clip list is fine for the 11-PlayItem fixture and is not fine for the real
hoard: two discs there carry 900 PlayItems in one title, and several carry
250-450, so a scan would be hundreds of comparisons on every one of millions of
frames.
spans_trusted already guarantees the spans tile the feed contiguously and in
order, which makes this a partition point. A repeated clip reuses its first
reference span, so the search walks back to the FIRST entry sharing it — the
bytes are read once, so the material is emitted once, at that entry offset, and
the answer is stable regardless of which duplicate the partition lands on.
Also adds a test at the real 900-clip scale: every clip places, output stays
monotonic across all 900, an arbitrary byte resolves to the right clip out of
ascending order, and a byte past the end belongs to no clip.
Four audit rounds each fixed one rule in SeamPlan::place and broke another,
because the question the rules were answering has no answer. Inside a
seamless-branching overlap clip k OUT comes AFTER clip k+1 IN — 57.8s of
overlap on the real fixture table — so a single timestamp is legitimately
inside two clips, and a clip file is not trimmed to its marks, so it also
carries material from before its own IN. No rule over timestamps can say which
clip a frame came from, and each attempt was right for one disc layout and
silently wrong for another: 65s of rewind, 17 minutes stranded, 28 minutes
dropped, 55s refused.
Frames already carry the byte offset they were read from (PesFrame::source,
stamped by the TS demuxer). Clip now carries the byte span its stream occupies
in the title feed, recorded while the extents are gathered. So the clip is a
LOOKUP: the offset falls in exactly one span. There is no decision to get wrong.
Every track of a clip lives in the same stream file and therefore shares one
span, so video, audio and subtitles agree by construction. Divergence between
them — each track guessing separately under its own tolerance — is how audio
and video ended up on different clips and drifted apart in the first place.
spans_trusted gates the whole path: unless the spans tile the feed contiguously
from zero, an offset means nothing and provenance is ignored in favour of the
mark heuristics, which is the 1.6.0 behaviour. A broken map degrades instead of
confidently selecting a wrong clip for every frame. A clip referenced twice
reuses its first span (the bytes are read once) and is still trusted.
Sources that stamp no provenance — a mkv:// remux, the deserialize hop — take
the heuristics, which is what they have always used and where they have always
been right, because they have no overlapping clips to be ambiguous about.
36 timeline tests, six of them new and covering: the overlap case marks cannot
see, all tracks agreeing, out-of-marks material dropped AND counted, a holed
span map, a discontiguous one, a repeated clip, and no provenance at all.
A symlink to a directory — a normal way to keep tens of gigabytes of streams
off the system disk — was dropped from the plan silently. `entry.file_type()`
reports the LINK, so such a subtree never enters `dirs`, and `metadata()` then
follows it and reports a directory, so it fails the is_file() test and was
skipped with nothing said.
The result is the worst class this release exists to close: PLAYLIST and
CLIPINF still synthesize, so the folder scans and enumerates titles, every clip
resolves to no extents, the mux takes its clean-EOF path, and a near-empty MKV
is written at exit 0.
It is still skipped rather than followed — following link targets invites
cycles and escapes from the folder — but at the same volume as the
unrepresentable-name skip eight lines above.
Round 7 enumerated the state space of SeamPlan::place and found three more
holes in the mark heuristics — the third consecutive round to find a defect
here, each time in the guard the previous round added:
- a clip file opening ON its own IN mark, after a tail frame past the previous
clip OUT had already advanced the cursor, read as another crossing and jumped
a clip too far: 28 minutes of clip 6 dropped;
- the last clip was unguarded entirely (`clip + 1 < len` skipped the check), so
the 9->10 seam rewound 75.6s with dropped_total() at 0 — the exact round-5
signature, reachable without any corrupt input;
- a glitch mid-clip satisfies "inside the current clip", so the round-6
stranding survived for the rest of that clip.
Three patches to the same heuristic, three wrong in a new way. So stop patching
it. Everything above the placement is still heuristics over marks, but the
property they exist to serve is now checked DIRECTLY: a track output must not
run backwards. A placement that would rewind by more than
DISCONTINUITY_BACKSTEP_NS is refused; the scan then looks for a LATER clip
containing the frame that does not rewind, which is exactly what a genuine
overlap crossing looks like; and if none exists the frame is dropped and
counted, because emitting it anywhere moves the track backwards.
The tolerance is the backstep constant because B-frame reorder legitimately
emits out of order by a fraction of a second, while a rewind that matters is
orders of magnitude larger.
The cursor moves ONLY when a later clip actually accepts the frame. My first
version of this advanced on a failed search, which strands a track exactly as
before — caught immediately by the round-6 regression test.
All 29 previous timeline tests still pass, including
continuity_preserves_forward_gap. The three round-7 scenarios now have a test
each, with the real 00801.mpls numbers.
The round-5 fix advanced the clip cursor on ANY backward step over 3s. That
also fires for a corrupt PTS, and for a legitimate STC discontinuity inside one
clip — and nothing moves the cursor back, because a forward step matches
neither past_out nor stepped_back. Every later frame then sits below the new
clip IN and is dropped: on the fixture table that is ~17 minutes of one track
gone, and the only volume gate compares total drops against ALL tracks frames,
so it exits 0.
The branch now also requires the frame to be INSIDE the current clip marks,
which is the only case that would otherwise be silently placed at the old
offset — the rewind. A frame outside them needs no help: the containment check
drops and counts it, the cursor stays put, and the next good frame is placed
normally. Both behaviours have a test, each confirmed to fail without the guard.
Separately, the dir:// PES input path scanned the folder and never applied the
encryption verdict scan_dir exists to produce, so the same folder ripped
through one door and failed through the other asking for a key it does not
need. That logic now lives in one function, session::apply_folder_encryption_verdict,
called by both.
The previous commit bounded the crack scan inspection by the bytes actually
read, which stopped it examining stale buffer bytes from an earlier batch. But
the cursor still advanced by the REQUESTED count, so those sectors were skipped
outright — trading "scans the wrong data" for "silently scans less than it
thinks", on exactly the damaged media where a title key is hardest to find.
The cursor now advances by what was read, so the next iteration resumes where
the read stopped. Floored at one sector so a source returning Ok(0) cannot spin.
Found by reading the fix again rather than by the next audit round.
read_sectors returns the number of bytes written and a source may return Ok
with fewer than asked — a recovery read over a damaged region does exactly
that. The crack loop discarded the count and inspected all n sectors of a
buffer that is REUSED across batches, so the tail still held the previous
batch sectors.
Cracking a key from those means cracking from data belonging to a different
extent, possibly a different VTS, while crack_span records the CURRENT one. A
key that opens nothing in this region is then installed, and its wrong
descrambles are only partly caught by the per-sector crib.
descramble_region was fixed to require the MPEG-2 pack start code, but
descramble_sector — the PUBLIC per-sector entry point, and the one the
module-level example tells callers to use — still keyed on the byte 0x14 flag
bits alone. The crate own documented guidance therefore led straight back into
the defect this release exists to fix: a VIDEO_TS.IFO sector holding 0x15 at
0x14 while starting 00 26 00 00 loses 1912 of its 2048 bytes, and because that
sector carries TT_SRPT the disc enumerates 38 titles while an image decrypted
from it enumerates 10, at exit 0.
It has no callers inside the crate, which is exactly why it survived three
rounds: nothing exercised it. Putting the guard inside the function rather than
in each caller is what keeps the safe path the easy one.
The integration test that covers it built its sector from the flag byte alone,
which no real scrambled sector looks like, so it stopped representing the path
it names — the same fixture-realism gap already fixed in four other places this
release.
A hole in the seam fix itself, found in audit round 5 and confirmed by
measurement against the real 00801.mpls marks already in the test file.
Nothing upstream trims a clip stream to its marks, so a clip file can open with
material from BEFORE its IN. With the cursor on clip 5 (7708.99..7910.79) a
frame at 7845.00 — clip 6 pre-mark lead-in, 8s below clip 6 IN of 7853.00 —
sits further from that mark than the 250ms tolerance, so neither crossing rule
fires. The cursor stays on clip 5, and 7845.00 IS inside clip 5 range, so the
frame is PLACED with clip 5 offset.
Measured: output went from 3714.51s to 3649.51s. Backwards 65 seconds, with
dropped_total() still 0 — no counter, no gate, nothing noticed — while the
whole 57.8s overlap band was emitted a second time over clip 5 written tail.
That is the collide-and-flatten symptom this type exists to remove.
A track own PTS only ever runs forward inside a clip, so a backward step larger
than DISCONTINUITY_BACKSTEP_NS is always a clip change. Such a frame now
advances to the first clip that could contain it — which terminates on its own,
rather than running to the end of the list, which is what the next_in bound
exists to prevent — and a frame still below that clip IN is material the
playlist excludes, so the containment check drops and COUNTS it.
All 27 pre-existing timeline tests still pass, including
continuity_preserves_forward_gap, which is the one that catches an over-eager
heuristic here. The new test asserts the output never moves backwards at a
join and was confirmed to FAIL before the fix.
The same gap as the MKV muxer: DemuxSink builds its timeline with
TimelineContinuity::with_clips, but no test ever handed it a title carrying
usable PlayItem marks, so frames_mapped and dropped_total() were always zero
and both gates in finish() were unreachable.
Deleting either left the whole suite green while a demux:// export of a
seamless-branching title wrote a directory of zero-byte track files beside a
populated chapters document and reported success at exit 0.
The test gives the title two clips with marks that exclude every frame, and was
confirmed to FAIL with the gate removed.
Audit finding: both seam gates in MkvMuxer::finish were dead code under test.
set_clips was never called anywhere in the suite, so continuity.dropped_total()
was always zero and neither branch could be reached.
The ordering of the two zero-frame checks is load-bearing and the comment says
so: a title the seam plan emptied also has a zero frame count, and MkvInvalid
is classified by is_skippable_title_stub as an empty nav/menu stub — so
reporting it that way makes an all-titles rip drop a real feature and finish the
rest at exit 0. Swapping the checks left the entire suite green.
The new test builds a two-clip plan (from_clips needs at least two, strictly
increasing) whose marks exclude every frame, and asserts both that the error is
SinkWroteNothing and that it is NOT classified as skippable. Confirmed to FAIL
with the two checks swapped.
Auditing my own fixes found all three. None were in the original code.
The VTS crack sort was byte-wise case-SENSITIVE while the filters that select
those files (vts_group_of / is_title_vob) are case-insensitive. On a
case-sensitive volume a set holding vts_01_1.vob beside VTS_01_2.VOB sorted
part 2 first, because V (0x56) precedes v (0x76) — reintroducing exactly the
budget-exhaustion the ordering exists to prevent. Now sorted on the same
uppercase normalisation the filters apply.
Refusing a name that round-trips to empty aborted the WHOLE plan. A sidecar
folder named with a single emoji made a backup un-rippable that 1.6.0 handled
fine, and reported it as a collision with a file that does not exist. It is now
skipped with a warning: the entry is unaddressable either way, but one
irrelevant file should not cost the user their rip.
The mtime check now applies only to files whose CONTENT the plan read — the
IFOs, whose bytes 0xC0/0xC4 place every VOB. Everything else is planned from
size alone, which is already checked, so comparing mtime there bought nothing
and risked a real false positive: disc backups commonly live on exFAT/FAT32,
which stores local time, so a long rip spanning a DST transition would see a
whole-hour shift on an untouched multi-gigabyte VOB and abort hours in.
Two dirimage findings, both silent-wrong-output.
The per-directory uniqueness check compared raw host names, but the reader does
not see raw host names: parse_udf_name trims leading and trailing whitespace
and drops any code unit char::from_u32 rejects — which is every half of the
surrogate pairs the encoder emits for non-BMP characters. So " 00000.m2ts" and
"00000.m2ts", or "A<astral>.m2ts" and "A.m2ts", were two entries at plan time
and ONE name at read time. find/read_file take the first match, so a title
resolved to the wrong file extents and muxed the wrong bytes at exit 0 — the
exact shadowing DirNameCollision exists to prevent.
The key is now derived by round-tripping the name through the very encoder and
parser that will be used, so it cannot drift from them. A name that survives
that round trip as empty is refused outright: it would exist in the image and
be addressable by nothing. The new test builds a real folder, calls plan, and
was confirmed to FAIL with the fix reverted.
Separately, the plan-vs-read revalidation compared file LENGTH only, while the
plan depends on CONTENT: a DVD VOB placement comes from bytes 0xC0/0xC4 of its
IFO, and IFOs occupy a whole number of sectors so an in-place rewrite keeps the
length. A re-authoring tool touching the folder mid-rip would pass the size
check while every title extent pointed at stale sectors. mtime is now compared
alongside size, and only when both sides report one, so a filesystem without
timestamps falls back to the old behaviour rather than failing every read.
probe_folder_encryption sampled AACS units starting from the largest extent
anywhere in the disc. AACS units are 3 sectors and a unit boundary is only
guaranteed at the START of a clip, so that anchor is only correct when the
largest extent happens to be a clip first extent.
For any clip over ~2 GiB it is not. The planner caps an allocation descriptor
at MAX_AD_BYTES = 524287 sectors, so every full piece of a split file ties on
sector_count and max_by_key returns the LAST tie — an extent beginning
(k-1)*524287 sectors into the file. 524287 % 3 == 1, so that start misses the
unit boundary for two file sizes in three.
The 6144-byte sample windows then begin mid-source-packet and the byte read as
the CPI flag is content. Both verdicts fail in a costly direction: a decrypted
folder is rejected as encrypted (DirImageEncrypted on something perfectly
rippable — the exact case scan_dir was added to rescue), or real ciphertext
reads as clear and the mux writes it out as video at exit 0. is_unit_aligned
cannot catch it: it measures against the same wrong base.
Now the largest TITLE first extent, which is unit-aligned by construction and
is also the more meaningful sample — the main feature rather than whichever
fragment happened to be biggest.
The existing tests could not reach this: their fixture m2ts is 786,432 bytes,
a single extent, which is always its own first.
Round 1 removed a largest-first sort from resolve_vts_key because
largest-first is the 1.5.1 garbage bug. Auditing that fix showed it was only
half right: `planned` comes from walking the UDF directory, which yields File
Identifier Descriptors in on-disc authoring order with nothing sorting them. So
deleting the sort did not restore playback order, it left the order undefined —
whatever the disc happened to list first.
DVD-Video numbers a title sets VOBs in playback order by spec (VTS_xx_1.VOB ..
VTS_xx_9.VOB, single digit), so ascending filename IS playback order and is
deterministic regardless of how the directory is laid out.
Order decides correctness here: crack_key shares one sector budget across the
whole extent list, a CSS DVDs biggest cell opens with a long clear run, and CSS
recovers the title key from scrambled data itself. Starting in the wrong place
can exhaust the budget without ever meeting scrambled data, whereupon the
caller falls back to the disc-wide key and the whole VTS is descrambled wrongly
— corrupt PES behind an intact header, written out at exit 0.
Four fixes from the first audit round. Every finding was verified against a
pinned tree and read directly before being accepted.
resolve_vts_key sorted a VTS title-VOB extents largest-first. That is the 1.5.1
garbage bug, and it grew back in a new code path: the comment claimed it
"matched the scan heuristic", but that heuristic WAS the bug and had already
been fixed in decrypt_keys_for_title, which documents the rule (PLAYBACK ORDER,
never largest-cell-first) and pins it with a regression test. A CSS DVDs biggest
cell opens with a long clear run and crack_key shares one sector budget across
the extent list, so starting there can exhaust it without ever MEETING
scrambled data — and CSS recovers the key from scrambled data itself. The crack
then returns None, the caller falls back to the disc-wide key, and every VOB in
that VTS is descrambled wrongly: corrupt PES behind an intact header, written
out as a complete extract at exit 0.
parse_pgcit dropped titles silently in THREE places — an unparseable PGC, an
out-of-range PGC index, and a truncated entry table. The finder caught one; the
other two turned up on reading the function. parse_vmg already counts and warns
per skipped title SET for exactly this reason, and this was the last place a
disc could quietly report fewer titles than it has.
write_image called flush() and returned Ok. flush() only pushes bytes into the
page cache and promises nothing about durability, so a 6-90 GB image could be
reported complete while still unwritten — a crash or an unmounted volume then
leaves a truncated file the caller was told was finished. Now into_inner (so a
buffered-write error surfaces instead of being dropped by BufWriter::drop)
followed by sync_all.
timeline used abs() on a saturating_sub result. Every other comparison in that
module is saturating because the timestamps come off a disc and are not
trusted; abs() panics on i64::MIN, which saturating_sub can produce.
The 1.6.1 DVD fix removed the USE of the loose predicate but left the
predicate sitting there with the better name. Anyone asking "is this sector
scrambled?" finds is_scrambled before is_scrambled_pack, and reintroduces the
defect that destroyed 1912 bytes of a real VIDEO_TS.IFO — the sector carrying
TT_SRPT — so the disc enumerated 38 titles and an image decrypted from it
enumerated 10, silently, at exit 0.
Byte 0x14 only means "scrambling control" inside an MPEG-2 pack. In an IFO,
UDF or ISO 9660 sector it is whatever that format stores there. The new name
says what the function actually tests and nothing more, so the honest question
has the obvious name and the dangerous one has to be asked for deliberately.
Its doc comment also claimed a caller, decrypt::decrypt_sectors, that does not
exist — so the name was an invitation and the documentation was an argument
for accepting it. It has no production callers at all; it stays public because
an integration test asserts the flag extraction directly.
The first cut ran release-profile tests on all three platforms everywhere,
which invented coverage no crate had ever claimed. bdemu is Linux-only — its
ci.yml has no macOS or Windows job at all, and its tests call
ExitStatus::signal(), which does not exist on Windows, so they do not compile
there let alone run. The rest follow the policy ci.yml already set: tests
execute on Linux and macOS, Windows compiles them so the cfg(windows) halves
still get codegen before release time.
That policy is now explicit in qa.yml as a windows-build job rather than
implied by which jobs happen to exist.
Recorded, not papered over: running the suites on Windows DOES find real
defects — autorip has five Windows path-handling failures in mover.rs and
freemkv-engine has three tests hardcoding /dev/null. freemkv ships a Windows
GUI, so they matter. They are pre-existing rather than 1.6.1 regressions, and
fixing them is its own piece of work.
Also build the binary before cli-parity.sh. cli-integration.sh builds its own;
cli-parity.sh only checks for one and exits 2 with "build it first", so both
legs died before running a single case.
Three release-profile failures on macOS and Windows, all found by the qa gate
on its first run. Release-profile tests on those platforms had never run
before it existed, so none of these were regressions — they had simply never
been visible.
available_space returned None on every non-unix target. That did not merely
skip a test, it skipped the GATE: a Windows user extracting a disc to a full
volume got a confusing failure part-way through instead of a clear refusal up
front, and Windows is where the GUI ships. GetDiskFreeSpaceExW is declared
directly against kernel32, matching how scsi::windows already reaches Win32
rather than pulling in a binding crate for one call. It asks for
FreeBytesAvailableToCaller, which accounts for per-user quotas — the same
question f_bavail answers on unix.
The other two asserted on wall-clock timing with no margin:
- sleep_until_halted_wakes_mid_sleep bounded the wait at 350 ms and measured
377 ms on a loaded runner. That bound measures the scheduler, not the wake.
What the test is for is distinguishing "woke because the flag flipped" from
"woke because the 10 s timeout expired", and 2 s does that just as well.
- abandon_loses_to_a_close_already_committed released at 600 ms against two
300 ms grace windows plus a 250 ms poll cadence, so the windows could expire
first and the caller abandoned — a race, not a defect. The intervals are
scaled up so jitter is small relative to them; the ordering under test is
unchanged, only the margin.
Every push to qa now tags v<version>-rc<N>, N incrementing, before the gates
run. That answers "which build is on qa, and is it the one I tested?" without
anyone having to remember it.
The tag lands whether the run goes green or red, deliberately. A red candidate
needs a name more than a green one does: "rc3 failed release-tests on windows"
is a sentence you can act on, "qa is red" is not. Red on qa is the gate doing
its job — the branch saying this is not production worth yet.
release.yml now excludes v*-rc*. Its trigger was v*, which matches the
candidate tags, so without this every push to qa would have built and PUBLISHED
a GitHub release — including for the candidates that failed.
dev -> qa -> main, across every public repo.
dev is where work lands and is meant to be pushed to often, so ci.yml stays
the fast answer: fmt, clippy, unit tests, leak-guard. qa is the release
candidate, and qa.yml is the claim that a commit is production worth —
release-profile tests on all three platforms, and the Linux cross-target
clippy that the local precommit gate has always run but CI never did.
Release profile matters as its own gate: overflow checks are off, debug_assert
is compiled out, and inlining changes what the optimiser can prove. A test
that only passes in debug never guarded the binary anyone ships.
qa.yml checks siblings out at qa rather than dev. This is not a monorepo, and
a qa run resolving its dependencies from unreleased dev tips would go green on
a combination that is not the one shipping — the exact mismatch the branch
model exists to prevent. ci.yml now tracks whichever branch triggered it for
the same reason, since it also fires on qa.
The consequence is that repos move to qa together, in dependency order. qa is
backfilled from main so an unchanged crate still presents working code to the
crates built on it.
css::is_scrambled reads bits 4-5 of byte 0x14 and nothing else. That is a
sound test once a caller has committed to a title's VOB data, where every
sector is an MPEG-2 PS pack and byte 0x14 always means what it says.
descramble_region is not such a caller: it is handed arbitrary regions of a
disc, so it also sees IFO, UDF and ISO 9660 sectors — raw structures where
byte 0x14 is whatever that format happens to store there.
Measured on a real disc: the second sector of VIDEO_TS.IFO holds 0x15 at
offset 0x14 while starting 00 26 00 00, which is not a pack. The flag test
read it as scrambled, descrambled it, and destroyed 1912 of its 2048 bytes.
That sector carries TT_SRPT, so the title table went with it — the disc
enumerated 38 titles and an image decrypted from it enumerated 10, silently,
at exit 0.
is_scrambled_pack already existed with the right predicate. Use it here. It
costs nothing: a genuinely scrambled VOB sector always carries the pack start
code, and no IFO sector does.
Verified end to end — the decrypted image's `info` output is now identical to
the source disc's, 38 titles both, differing only in the CSS: Encrypted line.
The fixtures moved with it. Four of them built a sector by setting byte 0x14
alone, which no real scrambled sector looks like; they now build packs.
The diagnostic hashed IFO contents but not where they came from, so a
content difference could not be told apart from a path resolving to a
different place. It now prints file_start_lba and file_extents next to the
hash.
That measurement is what inverted this investigation: both a CSS disc and
its decrypted copy resolve VIDEO_TS.IFO to the same extent, and only one
sector of the fourteen differs — by exactly a descrambled payload with the
scrambling-control bits cleared. So the decrypted copy holds the correct
bytes, and an iso:// scan of a CSS disc is parsing a still-scrambled IFO
sector, because scan_iso opens a plain FileSectorSource.
Two opt-in dumps, driven by FMKV_IMAGE, that narrow where a decrypted DVD
image loses titles:
dump_title_sets_for_an_image reports what survives parse_vmg. On one disc
the CSS image yields 13 title sets and 38 titles; its decrypted copy
yields 8 and 10. Sets 8, 9, 10, 12 and 13 are dropped outright, set 11
parses but returns no titles at all, and set 7 returns 3 of 5.
dump_vts_ifo_reads_for_an_image reads every VTS IFO and hashes the
CONTENT. All thirteen are byte-identical across the two images, so
parse_vts is handed the same bytes and the same TT_SRPT info and still
fails on one of them — the divergence is in what it reads from the READER
afterwards, which is file_start_lba and the PGCIT.
An earlier version of the second dump compared only length and magic and
so wrongly reported the images as identical; it hashes the bytes now.
parse_vmg skipped any title set whose parse failed, with no log and no
counter. A real disc enumerated 38 titles from one image and 10 from
another, and the 28 discarded failures were invisible — the symptom read
as a scan difference rather than as dropped reads, which is most of why
it took so long to localise.
Behaviour is unchanged: a disc may legitimately carry placeholder TT_SRPT
entries, so one failure is still not fatal. It now warns per skip with the
title set and the error, and once at the end with kept-versus-declared.
Also adds an opt-in diagnostic that reads every VTS IFO from an image, to
separate a read failure from a parse failure. It reports all 13 sets
reading identically from both a CSS image and its decrypted copy, which is
what proves the 38-to-10 loss is downstream of these reads.
dvd_placement_invariant_on_a_real_folder checks, per title set, the sum
ifo.rs relies on: file_start_lba(VTS_nn_0.IFO) + vtstt_vobs must land on
VTS_nn_1.VOB. It reports all 13 sets correct on a real DVD folder, which
is what excluded placement as the cause.
dump_titles_for_an_image prints every title a scan produces with the
numbers canonical_title_order sorts on. It is what showed the real shape:
the same disc scans to 38 titles as a CSS image and 10 once decrypted,
with identical capacity and a byte-complete image.
Both are #[ignore]d and read their target from the environment, so they
cost the gate nothing and are there for whoever picks the defect up.
The zero-frame check ran before the seam gate, and its error is
classified as a skippable nav stub — so a title the plan dropped
ENTIRELY was reported as an empty stub and an all-titles rip would omit
a real feature and finish the rest at exit 0. The seam case is decided
first now, with a code that is not skippable.
The demux sink read a frame's track kind out of the FILTERED slot, which
is empty for a class the export drops. On an audio:// or sub:// export
the video track was therefore called non-video and handed the permissive
crossing rule — the same defect round 4 fixed for a Dolby Vision layer,
reintroduced one file over. Video tracks are now recorded before the kind
filter, beside the primary-video reference that exists for this reason.
Its frame counter counted frames PLACED, not written, while its name and
doc claimed otherwise. Renamed and documented for what it is, including
that it cannot see a single lost track among many.
Placement: files in a subdirectory of VIDEO_TS were never given data.
They were declared at full size with no extents, so they appeared in the
tree and read as nothing. The same folder under BDMV was always placed
correctly. And the duplicate title-set guard keyed on the constraint maps,
so an IFO declaring no offsets inserted nothing and a colliding second IFO
went undetected — it keys on the groups seen now.
Display for SeamPlanDroppedMost and ShortImageRead discarded their
payloads, and four new variants were missing from the code-uniqueness
test.
The cap was added without a test. Its companion — a modest oversize is
honoured as a gap — already existed, so this pins the other side: an
offset past the ceiling is refused rather than grown into, which is what
keeps a rewritten IFO from planning a multi-terabyte image.
Round 3 counted frames the clip marks excluded and reported them at
finish. Counting is not bounding: the only other gate was a global
zero-frame check, and its error is additionally classified as a skippable
nav stub, so a title whose marks do not line up with its PES clock could
discard almost all of itself and still exit 0 — a two-hour feature
emitting seconds, which is the defect this change set already shipped
once. Dropping more than was kept is never a real join, so it now fails.
The demux sink had no zero-output guard at all, so a fully-dropped title
finished cleanly: a directory of zero-byte track files beside a populated
chapters document. It now refuses, keyed on frames having been OFFERED —
a chapters-only export, or a track class the title does not carry,
legitimately writes none, and two existing tests correctly said so.
A title set's placement group is the parsed number, so VTS_01_0.IFO and
VTS_1_0.IFO land on one key and the second silently overwrote the first's
constraint, placing a VOB where the IFO the reader uses does not point.
Refused rather than resolved by arrival order.
A Dolby Vision enhancement layer is a second video track: it does not
drive epochs, but it does carry B-frame reorder. The crossing rule was
keyed on driving epochs, so the EL took the branch whose premise is that
the track has no reorder — and its ordinary reorder dip near the end of a
clip, which during an overlap also lands inside the next clip's range,
was read as a join. The EL was then placed on the next clip's offset, out
of step with the base-layer frame it must be co-timed with by the width
of the overlap: the same desync the per-track cursor was added to remove,
reintroduced for one track.
The property the rule actually depends on is whether a backward step can
be reorder, so it is now keyed on that. Both sinks derive it from the
track kind rather than from the epoch driver.
Two findings from the same escalation, both silent-wrong-output.
The plan was built for every multi-clip title. Only a Blu-ray PlayItem's
IN/OUT are positions in the clock the PES PTS runs on. HD-DVD fills the
same fields from the XPL's title-relative times and a DVD's come from
cell tables, so a plan built from them is an identity map with a drop
filter: it suppresses the layer-break rebase inference performs, and
drops whatever falls outside marks the PTS was never measured against. An
earlier reading of this called HD-DVD safe because its marks are
contiguous and every computed offset was zero — true, and irrelevant,
because they were zero in the wrong clock. Gated on the content format,
with a test using an HD-DVD-shaped table that the clock check alone
accepts.
The crossing test was also one-shot. A table whose clips restart their
own bases could miss it, and a missed crossing STRANDS the track: every
later frame falls outside the stranded clip's marks and is dropped for
the rest of the title. Counting drops, which is all the previous round
added, does not bound them. A table that is not one advancing clock is
now refused outright and falls back to inference, which is the documented
safe path for those titles.
Also from the same round: read_sectors added an unchecked lba + i, where
callers deliberately saturate their LBAs — a wrap folds the read back to
a low sector and hands the muxer another file's bytes. classify added 1
to two numbers parsed verbatim out of a filename. read_head used a single
read() where a short read on a network mount silently records no
placement constraint at all. And the page-cache eviction added last round
released only the read that crossed its threshold rather than everything
accumulated, so seven eighths of what was read stayed pinned.
The round-1 tests asserted arithmetic about MAX_CS0_NAME_BYTES and never
called plan(), so both would have passed with the guard deleted — which
is the failure mode this audit exists to catch, committed by the audit's
own fix. They now build a real folder containing a 255-byte name and
require the planner to refuse it, plus a companion proving a name at the
cap is still accepted so the guard is not merely refusing everything.
The subdirectory cap keeps its arithmetic-only test — creating 65,535
directories is not reasonable in a unit test — but now says so instead of
implying coverage it does not have.
The folder encryption probe returned "not encrypted" when it had sampled
nothing at all — a title shorter than one aligned unit skipped the loop
entirely. That verdict CLEARS the structural one an AACS directory
raised, so a genuinely encrypted folder would have been ripped as clear
and written ciphertext as video at exit 0. With no evidence it now keeps
the structural verdict, and its bounds arithmetic no longer trusts
disc-derived values not to wrap.
Reading an IFO header swallowed every I/O error and returned an empty
buffer, which sent each placement offset through unwrap_or(0) and
recorded no constraint at all — a permission error on one file produced a
silently misplaced VOB. The directory walk swallowed the same class while
claiming to skip only vanished files. Both now propagate; only NotFound
is skipped.
Four things the round-1 changes left inconsistent: two new error codes had
no doc comments, were absent from the io::Error mapping, printed no path
in Display, and were missing from the test that proves codes are distinct.
The demux sink dropped frames silently while the MKV muxer reported them.
And set_clips had been inserted INTO write_frame's doc comment, leaving
write_frame undocumented and its paragraphs describing the wrong function.
uid/gid used 0 as "not specified"; UDF's sentinel is 0xFFFFFFFF, and 0 is
root.
A DVD title set records where its VOBS begins as an offset inside its own
IFO, and the planner honours that offset because honouring it is what
makes a real backup readable. Nothing bounded it: a regenerated .BUP or a
hand-assembled folder naming an offset far past the content grew the
image to wherever it pointed — a u32 sector count reaches ~8.8 TB, and
writing that to an iso:// destination fills a disk with zeros before
anything notices. Capped at 128 GiB, which clears BD-100 with room.
Metadata is materialized up front and held for the life of the image, at
a 2 KiB File Entry per node, so the 100,000-entry cap alone permitted
~205 MB of it for content of no size at all — and the mux holds two
images at once while probing. The module claimed a budget of a few MiB;
that budget is now enforced rather than asserted.
Host reads had no page-cache eviction. The ISO source documents what that
costs, measured: an 85 GB read pins the whole file, starves the writer,
and collapses the mux to 2.7 MB/s against 70 MB/s isolated. A folder
source reads host files the same way, so it now uses the same eviction —
the hints move from private-to-that-module to crate-internal rather than
being reimplemented.
The placement code skips that field for the Video Manager, which is
right, but said it was skipping it because the field is the menu cell
address table. It is the title search pointer table, and it is an offset
inside the IFO rather than a pointer to another file — which is the
actual reason it constrains nothing. ifo.rs reads the same offset under
the correct name, so the two would have drifted.
A sparse passive track — a subtitle with no event near a clip's mark —
was held to the dense-video crossing window, so it stayed on the previous
clip's offset until its PTS passed that clip's OUT and every event in
between was mistimed by the overlap. Video keeps the tight window,
because its backward steps are also B-frame reorder; passive tracks have
no reorder, so any backward step into the next clip's range is a join.
Frames the marks exclude were dropped without a trace. Dropping is right
at a join, but this codebase has shipped complete-looking wrong output
before, so the count is kept per track and reported when the mux
finishes, alongside the pre-cluster counter that exists for the same
reason.
A File Identifier Descriptor records its name length in one byte, and the
length was narrowed with a cast: a 255-byte name — POSIX NAME_MAX,
entirely ordinary — encodes to 256 and wrote zero, which would read every
later entry in that directory from the wrong offset. A directory's link
count is 16 bits and was computed as 1 + subdirectory count, which the
global entry cap alone permits overflowing. Both are refused while
planning, where the tree can still be rejected cleanly.
The module and struct docs described inference as the whole algorithm;
they now say which path decides what.
The folder reader produces a wrong title list on a real DVD — the title
table is read at a sector offset relative to VIDEO_TS.IFO's start, and a
synthesized layout does not put that file where the offset resolves the
same way. Output is silently wrong at exit 0, so the feature is not
wired to the CLI in this release. The library module stays, unreachable,
and ships once both disc families are verified against real content.
The first cut placed every track with the cursor the video had moved. At
an overlap join the previous clip's audio is still arriving after video
has crossed, and those tail frames sit inside both clips' mark ranges —
so they took the new clip's offset, jumped forward by the overlap, and
collided with the new clip's own audio, which the muxer's monotonic nudge
then flattened. A remux confirmed it: the timeline length was already
correct and the original symptom was still there, 169 audio packets on
the tick floor.
A track's PTS only runs forward inside a clip, so its own backward step
to the next clip's IN is its crossing. That is per track, so the cursor
is too.
A seamless-branching title's PlayItems do not chain contiguously: one
clip's OUT can sit after the next clip's IN, where the disc stores the
join twice, or before it, where the playlist skips material. The mux
never saw those marks — its own header said so — and inferred seams from
PTS jumps instead.
Inference cannot recover this. A forward jump is ambiguous: it means the
playlist skipped, or it means frames were lost to damaged media, and
compressing the latter would falsify timing on exactly the rips that most
need it faithful. An overlap smaller than the B-frame reorder threshold
is invisible to inference entirely, and its duplicate content then
collided in the muxer, where the monotonic nudge flattened a run of audio
onto the tick floor and put sound ahead of picture for the rest of the
film.
Measured on one 11-PlayItem title: the file declared 7893.385 s, which is
what the playlist says the title is, and carried packets to 8029.298 s.
Both numbers came from the same program on the same disc. Four skips
totalling 135.9 s became dead timeline, and a 1.79 s overlap put audio
1.8 s ahead at the half-hour mark. Five of forty-seven titles were
affected; every single-clip title was exact.
So the marks are read. Each clip contributes exactly out - in, laid end to
end, so the output runs as long as the playlist says and a join never
rewinds. Titles without usable marks — DVD, HD-DVD, file sources — keep
the inference path unchanged, and clips that already chain contiguously
produce a constant offset, which is pinned by a test.
Users keep discs as extracted folders — a DVD VIDEO_TS or a Blu-ray BDMV,
usually a backup that is already decrypted. dir:// could only ever be a
destination, so those folders could be produced and never read back.
Everything above the sector layer wants a UdfFs over a SectorSource, and
every UdfFs read re-reads the ICB off that source at call time, so a
folder has to present itself as sectors. It does: dirimage plans a block
layout over the real files, encodes a UDF 1.02 filesystem for the
metadata, and serves data straight from disk. read_filesystem then parses
it exactly as it parses a disc, so nothing above changes — and the
iso:// arm of input() is now shared rather than duplicated, so dir://
inherits its decrypt gates, title selection and stream pruning.
The encoder is validated by more than its own reader: macOS mounts the
synthesized image and the mounted files compare byte-identical to the
originals. A round-trip through our own parser could not have shown that
— the tag CRC seeds at zero, and a wrong seed would satisfy us and no
real driver.
DVD placement is not free packing: a VTS IFO records where its title
VOBS begins relative to itself, so the VOB has to land exactly there.
Unsatisfiable marks fail loudly rather than misplace the file. 3D folders
are refused for now: the scanner detects SSIF and the planner cannot
alias its extents yet, so accepting them would produce quiet nonsense.
Left for later: metadata capture, HD-DVD, FMTS, encrypted folders.
Writing an iso:// meant a raw sector copy off a drive, so an existing
image could only ever be a source. Decrypting one you already had meant
finding the disc again.
The engine's copy is the recovery path — mapfile, multipass, damage-jump,
auto-resume — and all of it exists because optical media returns read
errors. A file does not, so a non-drive source gets write_image instead:
sectors in, bytes out, once, no recovery machinery. Keeping them apart is
not just tidiness. The mapfile identity check compares AACS unit keys and
the VID, both empty for an already-decrypted source, so identity passes
for any such source — a second run with different input to the same
output path would resume over the previous image and report success.
A short read is an error rather than a zero-fill: padding a truncated
source yields an image that looks complete and is not, which is the worst
outcome for a copy someone means to keep.
Entries had grown into investigative narratives — root causes, disc
counts, PIDs, percentages, function names — which is the right level for
a post-mortem and the wrong one for a public changelog. Each entry is now
one to three sentences leading with what a user observes.
Halves the file (1538 -> 735 lines) without dropping a version. API
signature changes stay explicit, since a consumer migrates against those.
Matthew Jackson