Compare commits

...
114 Commits
Author SHA1 Message Date
Matthew Jackson 87bc07dad8 v1.6.8: bump version (freemkv-unlock git-pinned for the tag) 2026-08-21 12:54:48 -07:00
Matthew Jackson 743c5830b6 docs: 1.6.8 changelog (unified release) 2026-08-21 12:47:34 -07:00
Matthew Jackson 80c8e4ca24 ci: require a CHANGELOG entry for the crate version on main
Adds a 'changelog' job that fails when Cargo.toml's version has no matching
'## [X.Y.Z]' heading in CHANGELOG.md, hard on main and PRs into main, warning
on dev/qa — so the unified release can't ship a crate with a blank changelog.
2026-08-21 11:23:45 -07:00
Matthew Jackson f49e79c198 restore freemkv-unlock path dep for local dev (post-v1.6.7) 2026-08-21 09:31:40 -07:00
Matthew Jackson 231cbf3d76 v1.6.7: bump version (freemkv-unlock git-pinned for the tag) 2026-08-21 09:31:31 -07:00
Matthew Jackson 21c7b12e2b docs: 1.6.7 changelog (unified release, autorip-driven) 2026-08-21 09:23:55 -07:00
Matthew Jackson 799039df21 restore freemkv-unlock path dep for local dev (post-v1.6.6) 2026-08-20 20:03:48 -07:00
Matthew Jackson 497b7726dc v1.6.6: bump version (freemkv-unlock git-pinned for the tag) 2026-08-20 20:03:44 -07:00
Matthew Jackson 94b549a9d8 docs: 1.6.6 changelog 2026-08-20 19:56:40 -07:00
Matthew Jackson fc2d35a19c v1.6.5: bump version (freemkv-unlock git-pinned for the tag) 2026-08-20 06:35:01 +00:00
Matthew Jackson 3c51190c32 docs: 1.6.5 changelog 2026-08-19 21:20:42 -07:00
Matthew Jackson dfe0390445 mux/h264: treat open-GOP intra access unit as a resync keyframe
The B1 resync gate (mux/resync.rs) arms on a concealed-gap / clip-seam
discontinuity and drops video frames until it sees a keyframe. The H.264
parser flagged a keyframe only on an IDR slice (NAL type 5), so on a BD
open-GOP tail whose only random-access point is a non-IDR I-frame (the
recovery point) the gate never disarmed and dropped every remaining frame
to EOF — truncating the video by up to a GOP-plus tail and failing the
cli-acceptance timeline_sound strict span check for BD.

Promote an intra-coded access unit (measured coding_type == I) to a
keyframe, alongside the existing IDR-by-NAL-type flag. This is the H.264
equivalent of HEVC's open-GOP IRAP (CRA) handling, which the HEVC parser
already flags by NAL type — H.264 has no such NAL tag, so the measured
intra coding type is the signal. Keying off coding type rather than a
recovery_point SEI covers every signalling: an SEI-tagged recovery point
(which is intra by construction) and a disc that marks the recovery point
by GOP structure alone. Consistent with videomap.rs's model that a
keyframe IS the intra decode-restart point; the open-GOP anchor now also
gets the self-contained SPS/PPS re-assert, mirroring HEVC at CRA. IDR
stays flagged by NAL type, so a slice whose header fails to parse
(coding_type == None) is unaffected. HEVC, DVD/MPEG-2 unchanged.

Tests: open_gop_intra_access_unit_is_a_keyframe (red before this change),
inter_coded_access_unit_is_not_a_keyframe (precision guard).
2026-08-19 13:32:30 -07:00
Matthew Jackson 8678a2bf94 drive: select the platform module once, not per function
find_drives and resolve_device each carried a three-arm cfg(target_os) block in
their bodies. Alias the current platform's module once (`use linux as platform`
under a single cfg) and let both dispatch through `platform::…`, so the function
bodies are cfg-free and a new entry point cannot forget an arm.
2026-08-19 09:10:26 -07:00
Matthew Jackson d953b46023 Stop the HD-DVD VTI read and the extract progress channel from hiding loss
Two audit fixes plus two consistency cleanups, all in the changed 1.6.5 surface:

- hddvd: the VTI clip-order read used `.ok()`, flattening an unreadable
  authored order (a scratched sector under the .vti — the name came from the
  directory, so it is never "absent") into "no order" with no diagnostic. The
  sibling clip-extent arms log every read failure with its code; this one now
  does too, then falls back to the per-clip heuristic exactly as an unauthored
  disc would. Behaviour is otherwise unchanged; the loud line is the point.

- extract: the progress sink hardcoded bytes_unreadable_total: 0 and counted
  every zero-filled hole as good, so a progress-only consumer saw a holed
  extraction climb to a clean 100%. Thread the running unreadable total through
  and report the real good/unreadable split. The authoritative ExtractResult was
  already truthful; only the live channel lied.

- labels: correct a stale comment that described testlog's old lock-based
  capture; it now installs one global subscriber and routes to a thread-local
  sink.

- ps: import SYSTEM_HEADER from consts instead of re-declaring 0xBB, matching
  its sibling stream-id constants and the file's single-source rule.
2026-08-19 09:10:26 -07:00
Matthew Jackson 443ac6bcb1 Fix two tests that flaked under the parallel harness
testlog: capture through a single global subscriber that routes each
event to a thread-local sink and returns Interest::sometimes, so
tracing's global per-callsite interest cache can never short-circuit a
live capture when another thread rebuilds it. The old scoped
with_default lost events at random in the full suite (an empty capture
failed the log-accounting assertions ~1 run in 15).

labels: capture the escaping test through testlog instead of installing
its own process-wide subscriber, which hard-cached every foreign
callsite "off" and poisoned the captures above.

dirimage: name scratch directories from a process-monotonic counter
rather than SystemTime, which resolves to only a microsecond — two of
the parallel tests sharing a tag drew the same value, collided on one
directory, and the first to finish removed it out from under the
other's reads.
2026-08-19 01:18:38 -07:00
Matthew Jackson 794393f24e CSS: fix the decrypted-HD-DVD false E7023 at the detection layer, not the public API
Commit b02c402 ("key the DVD crack on the disc, not on the container") fixed a
real bug — a decrypted HD-DVD hit E7023 (CssKeyMissing) because the per-title
CSS crack keyed on the MPEG-PS container, which DVD and HD-DVD share — but did
it by adding a required `disc_format: DiscFormat` parameter to the PUBLIC
`DiscStream::new` and `build_iso_pipeline`, a `disc_format` field to
`MuxInput::Iso`/`Live`, and `DiscFormat::may_have_css`, threading the axis down
through mux/driver and mux/resolve. That changed the public API and broke every
downstream caller's compilation (freemkv-engine's integration test now needed 8
args, autorip's MuxInput arms a new field). 1.6.4 shipped and worked with these
exact signatures; a bug fix must not reshape them, and needing a whole
disc-format plumb for HD-DVD was a code smell.

Revert all of that plumbing (public signatures restored to their pre-b02c402
form; no `disc_format` parameter or field, no `may_have_css`, anywhere), and fix
the ACTUAL bug where it lives: the scramble-detection heuristic.

Root cause: `is_scrambled_pack` counted a sector as CSS scramble evidence on
pack-start (00 00 01 BA) + bits 4-5 of byte 0x14. Offset 0x14 is only the PES
scrambling-control field when the sector is a genuine elementary-stream pack. An
HD-DVD `.evo` RDI navigation pack is private_stream_2 (stream_id 0xBF), an
MPEG-PS pack exactly like a DVD VOB, whose byte 0x14 is raw nav payload that
routinely has bits 4-5 set. On a decrypted HD-DVD (None keys, MPEG-PS, so it
reaches the crack) those nav packs flipped the scan's `saw_scrambled` flag; the
crack then found no key — there is no CSS on an HD-DVD — and the scan returned
ScrambledUncracked, hard-failing a good disc with E7023.

Fix: exclude the MPEG-PS structural stream_ids CSS never scrambles — system
header (0xBB), padding (0xBE), private_stream_2 (0xBF) — by the stream_id at
offset 0x11. This is the refinement the DVD design notes already called for
("matches CrackTitleKey"). It needs no format plumbing because byte 0x11 lives
in the CSS-clear header (0x00-0x7F, untouched by scrambling), so it is the true
stream_id even on ciphertext. A decrypted HD-DVD now scans to Unencrypted and
muxes cleanly.

The DVD CSS crack is preserved and proven: a genuinely CSS-scrambled DVD sector
is always video (0xE0-0xEF) or private_stream_1 (0xBD), never an excluded id, so
its scrambled packs still set saw_scrambled and still hard-fail an uncrackable
disc — the "ciphertext muxed as plaintext at rc=0" catastrophe cannot slip
through. Red-before-green both directions: dropping the 0x11 exclusion turns the
decrypted-HD-DVD case back into E7023; inverting it (only nav ids count) turns a
real uncrackable DVD into Unencrypted and strands a crackable one. Both mutations
are caught by tests.

Gate (cargo +1.97): fmt, clippy --all-targets -D warnings, 3539 tests green;
freemkv-engine and autorip both compile against this tree again; precommit.sh
libfreemkv clean.
2026-08-18 23:26:22 -07:00
Matthew Jackson c6e2614c4c Close the Ok-but-empty clip hole on HD-DVD; keep it open, and say why, on BD
The same hole, two disc families, two answers — and the asymmetry is now a
decision written into both files instead of an oversight in one.

`file_extents` can return `Ok` and still yield no usable extent: an empty
allocation-descriptor list, or one every entry of which the `sectors > 0 &&
lba > 0` filter discards. An ordinary zero-byte file reaches it; no crafted
disc is needed.

On HD-DVD that was the flagship failure shape. The clip entered neither
`clip_extents` nor `unusable`, and nothing was logged, so the composer's
`any(|n| unusable.contains(..))` guard missed it while the
`filter(|n| clip_extents.contains_key(..))` beside it quietly deleted the
part: a `FEATURE_2.EVO` of size 0 next to a healthy `FEATURE_1.EVO` composed
a FEATURE title out of part one alone, still advertising the whole runtime,
at rc=0, in silence. Half a movie presented as a whole one. Round 1 accounted
for every `Err` from the resolver and left this route open. It now marks the
clip unusable and logs it under its own new code, E6019
(`E_UDF_NO_USABLE_EXTENT`) — deliberately not the neighbouring E6017, which
would file a zero-length file as an authoring hole and send whoever triages
it at the wrong population.

On Blu-ray the identical hole stays open, as previously decided, and the
reasons are now recorded on both sides. BD has no `unusable` set, so closing
it there means inventing a post-loop "every clip_id must appear in `spans`"
invariant that DROPS the title, and it is not settled that an empty-but-Ok
resolve is always a defect; dropping healthy titles is worse than the gap.
The consequence is milder too: on BD the clip is one PlayItem of an otherwise
whole title, on HD-DVD the feature is COMPOSED from parts. Same hole,
different price.

Also in this change:

* bluray: a non-absence SSIF failure that the `.m2ts` fallback papers over is
  logged. `unresolved` had exactly one reader, `if let (None, Some(code))`, so
  when `/BDMV/STREAM/SSIF/<clip>.ssif` failed with DiscRead /
  UdfAdChainTooLong / UdfEmbeddedData and the base view then resolved, the
  code was recorded and thrown away: the title shipped base-view 2D off a 3D
  disc at rc=0 with no log at all. The site's own doctrine is "ABSENCE is the
  only benign failure". Logged, not refused — the base view is a real rip.

* drive: `wait_ready` polled TEST UNIT READY through a bare `execute` and its
  60 x 500 ms loop never read `self.halt`, so a Stop during spin-up did
  nothing for ~30 s while every other drive path returns Halted at the next
  command boundary. `spin_cycle` issued both START STOP UNIT commands outside
  `checked_exec` and slept `SPIN_DOWN_IDLE_SECS` + `SPIN_UP_SETTLE_SECS`
  blind — ~15 s deaf to Stop, from the recovery path, exactly when the
  operator is most likely to press it. Both now use `checked_exec` and
  `sleep_until_halted`, which already lived in this file with four tests and
  was `#[cfg(test)]`, called from nowhere. It is production code again.

* drive: a READ(10) that returns GOOD status with a residual underrun was
  correctly refused and logged NOWHERE, while the sibling `Err` arm warns with
  lba/count/status. A residual-underrunning drive was indistinguishable from a
  scratched disc — two populations with opposite remedies. It now warns with
  transferred vs expected, which is the whole signal.

* error: `all_error_code_constants_are_unique` was a hand-maintained `vec![]`
  naming 109 of the 127 declared codes while its doc claimed to pin them all,
  and an earlier audit trusted that claim while assigning new ones. The list
  is now derived from the declarations by parsing `include_str!("error.rs")`,
  so a new constant is covered the moment it is written. A parser self-test
  cross-checks the count and three known name/value pairs, so it cannot pass
  vacuously.

* testlog: a test-only `tracing` capture (~120 lines, no new dependency) so
  the logging contract is enforced rather than commented. Three sites carry
  long comments insisting they log the error's OWN code; putting a literal
  back broke nothing. They are pinned now, along with the two new log lines.
  Captures are serialised process-wide: `tracing`'s interest cache is global
  while `with_default` is thread-local, and the rebuild on the exiting
  capture can land after the entering one's, leaving the cache at "never"
  while a capture is live. That produced a real empty-event flake.

* disc: `scan_with`'s halt wiring for the BD and DVD enumerators had no test —
  every BD/DVD cancellation test calls the scanners directly, so passing
  `None` on either branch left the suite green while Stop did nothing.

* mux::network: `accept_from_rejects_stream_without_fmkv_header` half-closes
  instead of `Shutdown::Both`, which raced an RST against the server's read
  and returned ConnectionReset instead of InvalidInput under load. The port
  was already ephemeral; that was never the cause.

Gate: fmt, clippy --all-targets -D warnings, and 3439 tests green on 1.97;
precommit.sh libfreemkv clean.
2026-08-18 21:51:55 -07:00
Matthew Jackson b02c402dae CSS: key the DVD crack on the disc, not on the container
`resolve_dvd_title_key` decided whether to run the DVD CSS crack by asking
what CONTAINER it was looking at — `ContentFormat::MpegPs`. MPEG program
stream is what DVD and HD-DVD have in common, so every HD-DVD title was
run through a crack scan for a copy-protection scheme HD-DVD does not
use and cannot carry: AACS is its family, and CSS appears nowhere on the
disc.

Both outcomes of that scan were wrong. The cheap one wasted up to 50,000
sector reads per title. The expensive one returned `CssKeyMissing` —
E7023 — refusing a perfectly good HD-DVD with a CSS error, which is what
a real CI run produced on the HD-DVD fixture. Reading E7023 there sends
whoever triages it looking for a missing DVD key on a disc that never had
one, which is how a routing bug spends a day disguised as a key problem.

The right axis was already in the codebase and already used: `mux/resolve`
asks `disc.format == DiscFormat::Dvd`. This threads the disc format down
to the decision and adds `DiscFormat::may_have_css` to name the question.

The asymmetry decides the default, so it is worth stating. Running CSS on
an HD-DVD costs a wasted scan or a false refusal — visible, recoverable,
annoying. NOT running it on a real DVD muxes scrambled sectors as
plaintext and exits 0, which is the failure-that-looks-like-success class
this project has shipped once already. So `may_have_css` is false ONLY
for the families proven CSS-free, and `DiscFormat::Unknown` — a bare
reader with no scan behind it — still cracks. An `== Dvd` allow-list
would have read as tighter while silently stranding every caller that
cannot name its disc.

Both directions are pinned: an HD-DVD title must never enter the crack,
and an unknown-format title must still enter it. Removing the disc-format
clause fails the first and leaves the second and every DVD test green.
2026-08-18 19:11:20 -07:00
Matthew Jackson 128e3bb120 Propagate cancellation out of every title enumerator; account the .clpi read
A cancelled scan was indistinguishable from a disc that simply holds
fewer titles, in all three tree enumerators. Once the operator presses
Stop, `Drive::checked_exec` fails EVERY subsequent SCSI command with
`Error::Halted` and `Drive::read` deliberately preserves the variant —
so a cancel is not one failed read, it is every remaining read failing.

  * `scan_bluray_titles` returned a bare `Vec<DiscTitle>` and
    `parse_playlist` an `Option`, so neither had a channel to report a
    halt. During a cancel each remaining `.mpls` read merely failed and
    was skipped by an `if let Ok(..)`, and the scan returned a TRUNCATED
    title list at rc=0. Both now return `Result`; `Ok(None)` from
    `parse_playlist` keeps its two benign meanings (unparseable MPLS,
    playlist under 30 s) and `Err` means the scan is over.
  * `scan_dvd_titles` was worse: `Err(_) => return Vec::new()` turned a
    cancel into ZERO titles at rc=0 — a DVD reported as carrying no
    video at all. `ifo::parse_vmg` compounded it, treating a halted
    title set as one of the placeholder TT_SRPT entries it is designed
    to tolerate and continuing to the next; both now propagate.
  * HD-DVD was already correct and is the model this follows, including
    its reasoning: a cancelled enumeration that returned `Ok` would be
    cached, displayed and ripped from as if it were the disc.

The cancel is propagated at ALL FOUR Blu-ray read sites, not just the
obvious one — the playlist read, the `.clpi` read, and both
`file_extents` calls in the clip resolver — and the halt is polled
between playlists and again after the loop, so the final iteration
cannot slip a truncated list through. The resolver's `note` closure
previously EXEMPTED `Halted` from its disc-defect classification, which
was right (a cancel is not an authoring hole) but left it swallowed:
with no exemption and no propagation a cancel landing on `file_extents`
returned a title claiming its full runtime with that clip's bytes
silently missing. Measured with the fix reverted: `Ok(Some((768000,
[])))`.

Second defect, same function: the `.clpi` read and `clpi::parse` sat
inside `if let Ok(..) && let Ok(..)`, with extent resolution nested
inside. A scratched CLIPINF sector, a missing `.clpi` or a malformed one
skipped the whole block — no extents, nothing added to `total_size` —
but `duration_ticks` had ALREADY been summed from the play items, so the
title still advertised the movie's full runtime. The `if let` discarded
the error, so nothing was logged either: a failure that looks like
success, which is this crate's flagship defect class. It is now a
`match` that drops the title and warns with the error's OWN code, the
same classification the extent resolver below it already uses. Logging a
fixed code would account a scratched disc as an authoring hole.

Deliberately NOT added: a post-loop "every unique clip_id must appear in
`spans`" invariant. `file_extents` can also return `Ok(vec![])`, or a
vector whose every entry the `sectors > 0 && lba > 0` filter discards,
and it is not settled that an empty-but-Ok resolve is always a defect
rather than a legitimate healthy-disc state (zero-length placeholder
ADs). Dropping healthy titles is a worse failure than that residual gap,
so the route stays open, with a comment at the site recording the
decision so the next audit finds it instead of re-deriving it.

Tests: `parse_playlist_missing_clpi_yields_no_extent_no_size` asserted a
title WAS returned with `size_bytes == 0` and empty extents, and its
docstring quoted the buggy control flow as the specification — it
blessed the defect. Inverted and renamed. `parse_playlist_keeps_exactly_
30_seconds` encodes correct behaviour but used a fixture wiring no
STREAM/CLIPINF, so the `.clpi` fix would have dropped its title for a
reason unrelated to the boundary it pins; repointed at the fully wired
fixture rather than weakening the assertion. Four new halt-propagation
tests (BD playlist read, BD `.clpi` read, BD extent resolve, DVD title
set + VMG), each proven red before green.
2026-08-18 14:27:10 -07:00
Matthew Jackson dc100f1c80 Audit round 7: account for every clip that cannot be resolved
Ten lenses over v1.6.4..HEAD, every claim read against the code before
it was believed. Seven confirmed; six are here, one is recorded for the
next round. All of these are the same family — a failure wearing the
shape of success — which is the family that once shipped 9 MB of
ciphertext inside a main-movie m2ts at rc=0.

A clip whose extents cannot be resolved is now accounted for, in both
disc readers. Only `UdfUnrecordedExtent` used to count: every other way
`file_extents` can fail — a scratched sector under the clip's ICB
(DiscRead), an allocation-descriptor chain that never terminated, a file
whose data is embedded rather than extent-mapped — fell through to the
ordinary "file absent" path. On Blu-ray that yielded a title advertising
its full runtime with a clip's bytes silently missing, because the size
and the play-item timing had already counted it. On HD-DVD it was worse:
the clip was never added to `unusable`, so a split feature still composed
from FEATURE_1 alone and offered half a movie as the whole thing. Neither
emitted a single log line. Absence is still benign — a 2D disc has no
.ssif and the extension fallback exists for exactly that.

`Halted` is excluded deliberately, and that exclusion is the whole reason
the first version of this fix was wrong. Cancellation makes EVERY drive
command return `Halted`; classifying it as a disc defect would have
dropped each remaining playlist in turn and handed back a truncated title
list at success — the same defect, wearing a cancel. `parse_playlist`
returns Option and has no channel to propagate a halt, so the existing
behaviour is preserved rather than made worse. Propagating it properly is
next round's work.

Both log sites now emit the error's OWN code instead of a hardcoded 6017.
Accounting a scratched disc (E6000) as an authoring hole would send
anyone triaging it looking for the wrong thing entirely.

AD type 3 is embedded data, not a descriptor list (ECMA-167 4/14.6.8).
`read_icb_extents` lumped it in with the reserved values and decoded the
file's own CONTENT as (length, LBA) pairs, manufacturing extents out of
arbitrary bytes and pointing the reader at unrelated sectors. This same
release already taught `read_directory` to honour type 3; this is the
file half of that decision. It is an error rather than an empty list,
because an empty list reaches the caller as a clip that contributed
nothing while its declared duration still counts it — the silent loss
pointed the other way. A legally zero-length embedded file still returns
an empty list. New code E6018: reusing DiscRead would have mislabelled a
deterministic structural property as transient I/O and fed the retry and
NonTrimmed machinery a byte that will never change.

`file_extents_addressing`, `extents_abs_at` and `AbsExtent` drop to
`pub(crate)`. The first hands back unrecorded extents UNFLAGGED, in a
shape identical to the safe call's return; its doc says callers must use
`file_extents` instead, but a doc comment is not a guard. No dependent
crate references any of the three.

Three tests close gaps the audit found, each proven red before green:
a held AC-3 access unit must not resume as a normal frame after its track
poisons; the PS resume cursor must survive a drain that rebases it (three
separate mutants caught); and AD type 3 must be refused rather than
decoded. The first attempt at the HD-DVD test passed with the fix
reverted, which made it worthless — it needed a VTI fixture before the
composition path ran at all.

Also: four error codes were missing from the uniqueness test that claims
to cover every published code, so a new variant reusing 6014, 6016 or
6017 would have passed it.
2026-08-18 13:46:03 -07:00
Matthew Jackson 286b5d4527 ci: stop Dependabot proposing Rust versions that do not exist
`dtolnay/rust-toolchain` is not a dependency in the sense Dependabot
means: it is versioned by the Rust release it installs, and the tag we
pin is the toolchain CI is pinned to on purpose — precommit.sh runs the
same one locally, so a lint that passes on a developer's newer default
cannot pass CI by accident.

Read as semver, those tags produced a 1.97.0 -> 1.100.0 proposal, a Rust
version that does not exist. Every such PR 404s on toolchain download
across all eight repos and regenerates weekly: eight permanently-red PRs
that promote.yml then has to special-case when it decides whether dev is
green.

freemkv already carries this ignore; this is the same block in the other
seven, so the fleet stays uniform. Bumping the toolchain stays a
deliberate, all-eight-repos change made by hand together with
precommit.sh.
2026-08-18 12:14:58 -07:00
Matthew Jackson d9f3f82c65 Audit round 4-6: disc parsing, extents, codecs and drive faults
Squashed from 12 commits. Every fix was proven red-before-green and killed by a
mutation; the reasoning for each is in the private audit record.

UDF and extents
  Honour ICB types rather than assuming a Short AD, so an AD-type-3 directory
  is no longer decoded from FID bytes into a silently empty listing. Carry the
  ECMA-167 recorded flag through to the resolvers: an allocated-but-never-
  written extent used to reach the read plan as ordinary content and splice
  undefined sectors into the rip. file_extents now refuses such a file, and
  only when the hole actually occupies byte space — a zero-length one displaces
  nothing, and refusing on it dropped whole titles off discs that ripped
  correctly. Type-2 sparse extents are kept alongside type-1; they were falling
  into a catch-all that exited the descriptor loop and returned a truncated
  list as complete. merge_ranges no longer claims a sector neither input
  covered. A short skip or an over-long AD chain errors instead of truncating.

HD-DVD and Blu-ray scanning
  Bound the XPL nesting depth, title count, clips and chapters per title, and
  memoize the clip-name fallback probe — four separate amplification axes, each
  of which alone left the worst case unbounded. The clip and title caps are 512,
  ~10x any retail disc, and a test pins the product of cap and probe budget.
  The scan is cancellable: it returned Ok with titles carrying no streams when
  halted, presenting a cancelled scan as a successful one. A clip dropped for an
  unrecorded extent now says so.

Codecs and muxing
  Resume a held E-AC-3 access unit rather than rescanning from its first frame,
  and drop it on a discontinuity — a stale hold indexed past the end of the new
  buffer. Map every ISO 639-1 code instead of collapsing fifteen languages to
  und. Correct the DVD palette order. Detect a skip past EOF.

Drive and I/O
  Classify dead-bus faults so the wedged-drive path can see them; a catch-all
  arm had been flattening the variants before the classifier ran. A prefetch
  producer that dies now reports SourceTerminated instead of Ok(0), which the
  reader legitimately read as a short read and zero-filled — a whole title
  could be fabricated and the pass reported complete.

Also: charge Ok(0) reads to the CSS crack budget, drop the unreachable soft
re-crack, and send disc-derived strings to logs through the debug formatter so
a crafted label cannot paint an operator's terminal.
2026-08-16 13:22:24 -07:00
Matthew Jackson e30209091e restore freemkv-unlock path dep for local dev (post-v1.6.4) 2026-08-15 09:13:30 -07:00
Matthew Jackson ac46e2382f v1.6.4: bump version (freemkv-unlock git-pinned for the tag) 2026-08-15 09:13:26 -07:00
Matthew Jackson 862c1460c3 Trim single-clip titles to their playlist marks (1.6.4) 2026-08-15 08:14:31 -07:00
Matthew Jackson fea64d07e2 Stop an uncrackable VTS borrowing another VTS's title key
`resolve_vts_key` called the `Option`-returning `css::crack_key`, which
collapses "no scrambled sector was seen" with "scrambled sectors were
seen and no key came out", and then fell back to the disc-wide key. A
multi-VTS CSS DVD whose second title set resists the Stevenson scan was
descrambled under the FIRST set's key: corrupt PES behind an intact
header, written out with `complete = true` at exit 0.

`CrackOutcome` exists precisely to keep those two apart, and its doc says
callers must surface the second as a hard error. Every sibling path in
the crate already does — `Disc::decrypt_keys_for_title` and the mux path
both map `ScrambledUncracked` to `CssKeyMissing`. This was the one path
that did not. The ordering comment 15 lines above describes this exact
outcome as the bug it exists to prevent; ordering makes the crack far
more likely to succeed, but it cannot make a failed crack safe.

Also, three things nothing could catch:

- The demux output filename took the stream's language raw while the
  base beside it was sanitised. A language code is three raw STN bytes
  through `from_utf8_lossy`, and `00 00 00` is the ordinary "undefined"
  encoding on real discs — a NUL in a path fails `File::create` with
  InvalidInput, taking the whole export down before one track file
  opened. `sanitize` now maps control characters too; it did not.

- `css::crack_key_scan`'s short-read handling was dead code under test:
  every source in the module returned the full request, so reverting
  `advance` to the requested count, or dropping the `.max(1)`, left the
  suite green. The `.max(1)` is load-bearing — without it a source
  returning `Ok(0)` never moves the cursor and never increments the
  budget, so the scan spins forever. That mutation now HANGS the test
  rather than failing it, which is the honest demonstration.

- `MAX_SUBDIRS`'s const-assert carried `#[cfg(not(test))]` inside a
  `#[cfg(test)] mod tests`, so it compiled in no configuration and could
  never fire — the dead gate the test above it was written to replace.
  Moved to module scope, and verified it now rejects a wrong constant at
  compile time.
2026-08-11 17:47:56 -07:00
Matthew Jackson 6e8da15c34 Bound the third hostile CSV, and stop two docs overclaiming
`forced_sub` was the last unbounded attacker-controlled list in
paramount.rs. `MAX_COM_INDICES` capped the two `*_com1_idx` attributes;
this one had nothing capping it at all, and unlike them it has no value
to filter — a cell is a classification of the position it sits at, so
its bound has to be positional. Extracted as `forced_subs` for the same
reason `com_indices` was extracted: through `labels_from_feature` the
bound is unobservable, because the subtitle loop cannot reach those
cells either, so the assertion could not fail.

The `MAX_COM_INDICES` doc claimed the entry-allocation argument for the
whole constant. It is the VALUE filter that caps the set (values below
the bound, so at most that many distinct entries, however long the
attribute); the `take` caps the WORK. Both are real and they are not
the same bound; the doc now says which is which.

`jar_inventory_dedup_is_not_quadratic` called itself proof by deadline.
It is a hang guard — a return to the linear scan runs for minutes and
would wedge CI rather than fail it — and no assertion in it can tell a
BTreeSet from any other sub-quadratic dedup. Renamed and documented for
what it does. Its margin was measured before keeping it: 0.14s debug /
0.07s release against 10s, ~70x, unlike the 6x that made paramount.rs's
wall-clock test flake.
2026-08-11 16:59:33 -07:00
Matthew Jackson 8898351cb1 Round 2: stop two of round 1's tests claiming more than they prove
Both are mine, and both pass with the bound they "cover" deleted.

`a_hostile_commentary_index_list_is_bounded_not_merely_fast` asserted
only label purposes. The out-of-range filler it feeds is unobservable at
the label level and a HashSet collapses the repeats, so removing
MAX_COM_INDICES entirely leaves it green. What it DOES catch is a bound
set too low — verified at 2, where the real indices stop resolving. Named
and documented for that, and it no longer implies it guards enforcement.

`an_index_that_cannot_address_any_cell_is_not_retained` asserted through
the labels, where retention is by definition unobservable: the loop never
queries a cell that high. It now reads the set through `com_indices`,
where the claim is checkable.

Enforcement was and remains proven by
`distinct_unaddressable_indices_are_refused_not_stored`, which was red at
50,000 retained entries and green at zero. Two tests, two properties;
neither pretends to the other's job now.
2026-08-11 11:54:28 -07:00
Matthew Jackson 7a03be5496 Bound the commentary-index parse, and stop proving it with a clock
The gate caught `commentary_index_lookup_is_not_quadratic` failing, then
passing on a re-run. Measured both ways: 1.62s alone, OVER 10s against
its own 10s deadline while the suite's other 3,347 tests ran
concurrently. A 6x margin against a shared CPU is not a margin, and a
test that gets re-run until it passes is not a test.

It was also measuring the wrong thing. Replacing the linear scan with a
HashSet bounded the LOOKUP; the set was still built from every entry the
disc declared. `playlists.xml` is attacker-controlled and has no length
of its own, so a hostile disc could still force an unbounded allocation
before any lookup happened — the parse, not the query, was the exposure.
The code's own comment said "unbounded parsed input" and only fixed half
of it.

`MAX_COM_INDICES` bounds both halves, at the one value that cannot
change behaviour: an index at or beyond `u16::MAX` can never match a
cell, because the labelling loops break at `u16::try_from(i + 1)`. Real
authoring is nowhere near it — the BD STN table admits 32 streams.

The parse moved into a `com_indices` helper so the bound is OBSERVABLE.
Through `labels_from_feature` it is not: a HashSet collapses repeated
values, and an out-of-range index changes no label, so the obvious test
passes with or without the cap — an assertion that cannot fail, which is
what the first draft of this fix shipped. The test now hands in 50,000
DISTINCT unaddressable indices and asserts they are refused. Proven red
with the bound removed (50,000 kept), green with it.

Applies to both `sub_com1_idx` and `aud_com1_idx`.

NOT done here: `labels/mod.rs:1421`'s `jar_inventory_dedup_is_not_quadratic`
is the same wall-clock shape and has the same flakiness. Named so it is
not lost.
2026-08-11 11:30:53 -07:00
Matthew Jackson de94e12864 Cover the dir:// input door, and correct the CSS scramble-gate docs
Three round-1 audit findings on v1.6.0..HEAD.

Tests: `mux::resolve::input("dir://…")` — the door the CLI rips a folder
through — was never driven with a real folder. The only dir:// input test
uses a missing path, which fails in DirImage::open long before
`image_input`, so dropping the `is_folder` argument (and with it
`session::apply_folder_encryption_verdict`) passed the whole suite. Two
tests now close it, stated as a DIFFERENTIAL between the two doors,
because that is the invariant the shared function exists to hold: a
clear folder that kept its AACS/ directory must scan the same and select
the same extents through `scan_dir` and through `dir://`, and a
scrambled one must be refused through both with E9063. Proven red
against a mutated guard (both fail with E7022, the tree-shape verdict
the probe overrides) and green with it restored.

Docs: css/mod.rs documented `is_scrambled`, a function this line renamed
to `has_scramble_flag_bits`. Eleven stale sites, two of them broken
rustdoc links. Two were not stale names but false statements — they said
the descramble loop keeps the looser raw-flag test, when
`descramble_sector` and `descramble_region` both gate on
`is_scrambled_pack`; that parenthetical is rewritten to say what the
code does and why (the measured VIDEO_TS.IFO case: 38 titles became 10,
silently, at exit 0). The one mention that must stay is the sentence
explaining why the name was rejected.

Constants: io::image_writer and dirimage::encode each re-declared the
2048-byte sector that consts::SECTOR_BYTES already exports; both now
alias it. Note `BATCH_SECTORS = 2048` in image_writer is a different
quantity (sectors per batch, not bytes) and is deliberately left alone.
Bare 2048 literals elsewhere in the crate are out of scope here.
2026-08-11 10:23:37 -07:00
Matthew Jackson 21ec1d3f80 restore freemkv-unlock path dep for local dev (post-v1.6.3) 2026-08-10 09:04:40 -07:00
Matthew Jackson 6658a80819 v1.6.3: bump version (freemkv-unlock git-pinned for the tag) 2026-08-10 09:04:35 -07:00
Matthew Jackson 3db11a192c Carry the same licence and community files as the other repos
These are eight public repos that ship one product on one version, and
they had drifted: freemkv-engine carried no LICENSE at all (GitHub
reported its licence as none), two crates had no code of conduct, four
had no contributing guide, and none had a security policy — so there was
no private route to report a vulnerability in a disc-decryption tool.

SECURITY.md names GitHub Security Advisories on each repo rather than an
address, so there is nothing to keep in sync and no inbox to go stale.
2026-08-09 22:08:22 -07:00
Matthew Jackson a425a71cd5 Keep the cross-platform jobs for the release candidate
dev is where work lands and is meant to be pushed to often, so what runs
there should be the cheap answer to "did I break it": lint, tests and the
Linux build. The macOS and Windows jobs now run on qa and main instead of
on every push to dev.

Nothing is deleted and no platform goes unchecked before a release. qa.yml
already covers macOS and Windows independently, and the jobs that live only
here -- the Intel macOS build, the Windows release build -- still run, on
the branches where a cross-platform break is worth blocking on.

They are SKIPPED on dev via `if`, not left unscheduled. A queued job would
be worse than a slow one: release.sh's CI gate refuses while any run for
the commit is still in progress, so a job that never gets a runner blocks
releases silently, with no error anywhere. That is the failure the
real-media note in qa.yml describes, and it is why this is an `if` on the
job rather than a narrower set of trigger branches.
2026-08-09 20:24:07 -07:00
Matthew Jackson f79745d815 Release 1.6.3
Version bump and changelog for the 1.6.3 sync. All eight crates ship the
same version, so the crates with no functional change this cycle say so
rather than carrying an empty section.
2026-08-09 20:01:33 -07:00
Matthew Jackson 5d8f77e049 Drop six unused crates, and align the rest with the workspace
Two problems, both invisible until the whole graph is looked at together.

DEAD: num-bigint, sha2, num-traits, num-integer, cmac and cbc are
declared here and referenced nowhere -- not in src, tests or benches.
They were being compiled, audited and offered version bumps forever for
no reason. Removing beats bumping.

cbc nearly survived the sweep: a substring search for "cbc" matches 44
occurrences of ycbcr_to_rgb in the DVD subtitle decoder, so it looked
used. Only a word-boundary search exposed it.

SKEW: this crate was the outlier on every shared dependency -- aes 0.8,
rand 0.8, base64 0.22.1 and zip 2 against 0.9 / 0.10 / 0.23 / 8
elsewhere. Cargo cannot unify across a major version, so it compiled
BOTH: 32 duplicated crates in the freemkv binary's graph, including two
complete AES implementations (aes 0.8 + 0.9, cipher 0.4 + 0.5), two
digest stacks and two getrandom. Two crypto stacks in one product is
worth removing on its own.

The aes bump is an API rename -- BlockCipher-prefixed traits, Array for
GenericArray -- and the obvious translation uses Array::from_slice,
which the new version deprecates and clippy's -D warnings would reject.
These use the From<[T; N]> conversion the crate points at instead.

3441 tests pass in debug and release. The AACS crypto here is covered by
known-answer tests, so a byte-order or sizing mistake in that rename
could not have passed.
2026-08-09 18:37:24 -07:00
dependabot[bot] 79a16a9279 Bump softprops/action-gh-release from 2 to 3 (#4)
Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 2 to 3.
- [Release notes](https://github.com/softprops/action-gh-release/releases)
- [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md)
- [Commits](https://github.com/softprops/action-gh-release/compare/v2...v3)

---
updated-dependencies:
- dependency-name: softprops/action-gh-release
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-09 17:14:30 -07:00
dependabot[bot] 219a77b581 Bump actions/checkout from 5 to 7 (#5)
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-09 16:58:27 -07:00
Matthew Jackson be378cbc34 Have Dependabot watch the dependencies, and land its PRs on dev
Version updates were never configured here -- only security alerts, which
report but never open a pull request. So the absence of Dependabot PRs was
not "nothing to update", it was "nothing was checking".

Updates target `dev`, never `main`. main is a release pointer that
release.sh moves to each tag, so a bot commit on it would put work there
that no tag contains -- the exact state that aborted the 1.6.2 cascade at
its last step, and pointing a scheduled bot at main would recreate it
weekly.

Minor and patch bumps are grouped into one PR rather than one per crate:
eight repos times a handful of dependencies is a volume nobody reads, and
an unread PR queue is indistinguishable from no updates at all.

The freemkv crates are excluded. They depend on each other by git tag,
re-pinned by release.sh inside the release commit, and Dependabot cannot
see that cascade -- a PR bumping one could pin a version whose tag does
not exist yet.
2026-08-09 16:48:31 -07:00
Matthew Jackson 63200a1d1f Retire the README version bot
It fired on `release: published` and committed to main, the branch the
release points at its tag, so it could only ever leave main ahead of the
tag. release.sh now makes the same edit inside the release commit.

In this repo it was a no-op regardless: the pattern it rewrote does not
appear in this README, so it has been running and changing nothing.
2026-08-09 07:20:44 -07:00
Matthew Jackson 526dd3552b restore freemkv-unlock path dep for local dev (post-v1.6.2) 2026-08-08 22:18:33 -07:00
Matthew Jackson 8d17904459 v1.6.2: bump version (freemkv-unlock git-pinned for the tag) 2026-08-08 22:18:28 -07:00
Matthew Jackson 1bdb8add2f Stop trusting a byte offset when the feed is not what was measured
A clip's span is measured over the title's full extents when the disc is
scanned, and a frame is placed by the offset it was read from. Those only
agree while the mux reads every byte the scan counted.

On a disc carrying a forensic segment it does not: the read plan omits the
units belonging to another device group, so fewer bytes are fed than the
spans describe, and the shortfall grows through the title. Every frame past
the first segment then looks earlier than it is — placed in a clip it did
not come from, or dropped at a join for failing marks it was never inside.
The spans still tile one another perfectly, so the check that asks whether
they can be trusted cannot see any of it.

The plan is now compared against the extents it was built from. When they
differ the spans are dropped and placement falls back to timestamps, which
is what that path is for and what the surrounding comment already promised
would happen when an offset stops meaning anything. Ordinary discs are
untouched: with no forensic segment the plan IS the extents.
2026-08-08 19:42:53 -07:00
Matthew Jackson c9562c07a3 Make four guards testable, and one guard see both spellings
The provenance guard knew one way to lose a frame's source offset —
writing it as absent. Omitting the field entirely does the same thing,
because the frame type fills it in by default, and the guard read straight
past that. A parser rewritten into the second spelling would have kept its
green light while its track silently went back to being placed by guesswork.
It now reports which spelling it found and where.

The title-count clamp was asserted over a fixture too small to hold more
titles than the cap allows, so the walk stopped when the buffer ran out and
the clamp was never what bounded it — the assertion held with the clamp
deleted. The fixture now carries more entries than the cap.

The subdirectory limit was checked by restating the constant's own
definition; the guard itself had never run, and deleting it changed
nothing. The limit is lowered under test so a real folder can exceed it,
and the test now walks one and requires the refusal.

One test also carried two unrelated grounding notes while the test they
described had none, so reading the note above a test told you about a
different one.
2026-08-08 19:06:37 -07:00
Matthew Jackson 7f9dcf1870 Keep both halves of a clip file that two play items share
A playlist may point at the same .m2ts from several play items — a
seamless split, a looped segment, multiple angles. The file has one set of
bytes, so it has one span, and the byte a frame was read from therefore
identifies the FILE, not which play item's range it falls in. Placement
took the first of them and judged every frame against its marks, so
everything past that range's end was treated as material the playlist
excludes and dropped: half the clip missing from the rip, with the
timeline still charged for its full duration.

The offset narrows a frame to the run; only its timestamp can finish the
job, and each play item carries its own marks. A frame is now matched
against the marks of the entry it actually falls in, falling back to the
first when it falls in none, which leaves genuinely-excluded material
dropped as before.

The test for this construction asserted only that such a playlist is
trusted, and never placed a frame from the second range — so the loss it
described in prose was invisible to it. It now places one from every range
and requires them all to survive.

The deferred-mux replay of buffered frames wrote them without their source
offset, quietly sending the head of every such title down the timestamp
heuristic the rest of this work exists to retire. It passes the offset
through now, and the frame writer that omits it is compiled out of the
library entirely: no production path can discard provenance any more.

Also: the AC-3 buffer-reuse test could not see the regression it named.
Feeding equal-sized packets, a fresh allocation per call yields the same
capacity as a reused one. It now feeds a large packet then a small one,
where only a reused buffer keeps the larger capacity.
2026-08-08 17:18:35 -07:00
Matthew Jackson 3c5c681399 Audit round: an ADTS frame that cannot hold its own CRC, and three drifts
An ADTS header that declares a CRC follows must be at least nine bytes —
seven of header plus the two the CRC occupies — because the declared frame
length counts them. The structural gate compared against a flat seven and
never read the bit that says whether a CRC is there at all, so a frame
whose own header describes something impossible was accepted and handed to
the muxer as decodable.

The pipeline's spawn doc named Sweep, and the thread Sweep would have
created, as callers to look for. Neither has been in this crate since the
recovery passes moved out. The same paragraph already records fixing this
once, for a different departed caller — it simply drifted again a sentence
later, so it now says to name callers that live here or name none.

Whether a disc is structurally AACS-encrypted was spelled out by hand in
both the fast identify and the full scan. They agreed today; nothing made
them agree tomorrow, and disagreeing would mean the same disc reported
encrypted by one path and clear by the other. There is one definition now,
and the comment that pointed at it by line number points at its name.

The AC-3 parser built a fresh buffer on every packet — of the order of a
hundred thousand times per title — to work around a borrow it cannot
avoid. The copy stays; the allocation does not. The buffer is now lent out
and handed back, and a test pins that, because reverting it would be
invisible in behaviour.

Left alone deliberately: send/send_with_halt and finish/finish_with_halt
look like one action under two names, and are not. After the consumer
fails, one must still accept items and the other must refuse them; that
difference is what stops a producer reading an entire disc for a write
that died on its first frame. Collapsing them was tried here and the
existing test caught it. Both now say so where the choice is made.
2026-08-08 16:55:55 -07:00
Matthew Jackson 43d9fb583a Add an error for an image shorter than its recovery data
A recovery resumed against an image that has been truncated since the
previous pass — a full disk, an interrupted transfer, a remount — cannot
repair it: the pass only revisits the ranges the recovery data calls bad,
so everything past the cut stays a hole while the counts still describe a
whole disc. There was no way to say that. Reusing the recovery-data error
would have been wrong, because the recovery data is intact; the image is
not.

Registered in the uniqueness list, the code table and the range check, so
a future variant cannot silently reuse 6015 or map to the wrong code.
2026-08-08 13:35:44 -07:00
Matthew Jackson 60ed45c047 changelog: 1.6.2 2026-08-08 08:53:13 -07:00
Matthew Jackson 07cc9fa5f9 ci: run the dependents' test suites, not just a type-check
The `consumers` job checked out all five dependents against this commit and
ran `cargo check --all-targets`. That proves they still COMPILE, which
catches a changed signature and nothing else. The failures worth catching
here keep every signature intact and change behaviour: the library still
builds, the dependent still builds, and the dependent's tests are what go
red. Those never ran.

Run their suites instead. Same checkouts, same patched paths — only the
verb changes.
2026-08-08 08:46:17 -07:00
Matthew Jackson 1e6bb8babd Give a frame that outruns its epoch's video a provisional offset
The demuxer interleaves, so at a boundary the streams do not reset on the
same frame: audio for the next segment can reach the muxer before the video
frame that opens the epoch it belongs to. Only the primary video track may
open one, so those frames rode the just-ended offset and landed a whole
segment in the past. Downstream the strictly-monotonic block nudge then
crushed the entire run onto one instant a tick apart, which is audible.

Such a frame is recognised on its OWN raw PTS, which within an epoch only
advances, so a large backward step is unambiguous — a different signal from
the shared frontier, which is what the old false-positive ratchet keyed on.
It then rides a provisional offset private to its track, computed the same
way the video path computes a real one, and drops it the moment the video
retires an epoch, so the run rejoins with no seam.

The offset is deliberately private: it never writes offset_ns, never
advances the frontier and never retires an epoch, so it cannot move the
video timeline. Letting a passive track open a REAL epoch was tried first
and inflated a 476.776 s title to 656.216 s, because every track meets a
boundary at its own pace and the video path rebased again on top.

Measured on a real DVD title with 8 cell boundaries: audio frames stamped
inside one cadence 120 -> 0, outsized gaps 7 -> 0, subtitle 1 -> 0, and the
video span byte-identical at 476.484. On the HD-DVD title the remaining
boundary gap on its first audio track falls from 0.999 s to 0.275 s.
Acceptance: 76 pass, 0 fail, 0 skip.
2026-08-08 00:48:10 -07:00
Matthew Jackson 64f894c9f3 Place a straggler by its own epoch's end, not the current frontier
A non-video frame can reach the muxer after the video that opened the next
epoch has already been processed. It carries an old-epoch PTS, so adding
the new offset flings it forward by a whole clip. The remap that exists to
catch this compared the frame against the CURRENT frontier and only
accepted it within one backstep, which is the wrong yardstick: by the time
a straggler is seen the frontier has moved on into a new epoch, and how far
below it the frame lands says nothing about where it belongs.

Retiring an epoch now records the frontier it closed at alongside its
offset, and a straggler is judged against the end of the epoch its PTS came
from. Keeping the whole history rather than one previous offset also fixes
the case a single `prev_offset_ns` cannot express at all: a frame from two
or more epochs back.

Measured on a real HD-DVD title whose second audio track put its final
packet at 12834.587 s in a 6434.100 s file. That frame sits 23 s below the
current frontier — outside the old window, so it was refused — but 0.15 s
below the end of the clip it actually came from. It now lands at 6417.216,
one 32 ms cadence step after its neighbour, and the track spans 6416.160
with no outsized gaps. The other two streams are unchanged.

A title that never rebases retires no epoch, so the lookup finds nothing
and the mapping is bit-identical to before.
2026-08-08 00:02:57 -07:00
Matthew Jackson 3820348d2f test: cover the 1.6.1 provenance work where it was assumed, not asserted
Four parsers (dvdsub, flac, lpcm, mpegaudio) stamp a source offset on
every frame but had no test that read one back, so a regression to
`source: None` would have been caught only by the brace-balanced audit
in codec/mod.rs — a lint, not a behavioural check. Each now asserts the
emitted frame carries the offset of the packet that supplied its first
byte.

The Blu-ray feed spans had no direct test at all. Add one that walks a
multi-item playlist and requires the spans to tile the feed with no gap
or overlap; it catches a one-sector-per-clip drift, which is exactly the
error class that would misattribute frames near a seam.

`no_provenance_still_places_by_marks` asserted only that placement
returned something, which passes for a frame placed in the wrong clip.
Its probe timestamp lands in an overlap between two clips in the real
mark table, so pinning one clip would assert a coin-flip; instead
require the offset to be one that a clip actually containing that
timestamp would produce.
2026-08-07 22:16:02 -07:00
Matthew Jackson 102ae82933 Test the DVD title table's sector-offset arithmetic
The TT_SRPT pointer at 0xC4 is a SECTOR offset from the start of
VIDEO_TS.IFO. Reading it as a byte offset lands 1/2048th of the way in and
does not error — it silently returns a different title list, which is how
a disc enumerating 38 titles and an image enumerating 10 looked like a
scan difference rather than a failure.

The table already had tests for its entry layout, dedup and the 99-title
clamp. What it did not have was anything pinning the offset unit itself,
or the grouping of titles by title set, or a truncated table keeping the
rows it can read rather than failing a whole disc.
2026-08-07 22:01:37 -07:00
Matthew Jackson 9d05f54ddb restore freemkv-unlock path dep for local dev (post-v1.6.1) 2026-08-07 19:08:46 -07:00
Matthew Jackson 47abd1270b v1.6.1: bump version (freemkv-unlock git-pinned for the tag) 2026-08-07 19:08:42 -07:00
Matthew Jackson 96c23ffb87 changelog: mark 1.6.1 UNRELEASED so release.sh can date it
release.sh dates the heading when it cuts the tag, and refuses a bare
'## [1.6.1]' because it cannot tell an undated heading from one it has
already stamped. Shipping a tag whose public changelog says UNRELEASED is
permanent and cannot be corrected inside that tag, so the check is right
to stop rather than guess.
2026-08-07 17:57:29 -07:00
Matthew Jackson f2b0e00bea Guard the invariant that provenance is universal
Every emitted frame must carry the source byte offset of the packet it
came from. That held only for video for as long as it existed, and
nothing asserted it, so nothing caught it: ten parsers built frames with
`source: None` and a multi-clip title could not place audio or subtitles
by byte at all.

Finding them took a brace-balanced scan of the tree by hand — a regex
cannot do it, because a Frame literal contains nested braces and a
non-greedy match stops at the first `}`, which is how five sites survived
the first pass. This is that scan, as a test.

A second test fails if a codec module is added and not listed, since an
unchecked parser is exactly how the gap persists. Modules that emit no
Frame are named explicitly rather than skipped silently.

The guard tripped on itself twice while being written — first on its own
doc comment, then on its own string literals — so it strips line comments
and excludes its own module. A check that matches prose about the defect
rather than the defect is the same mistake in a different place.
2026-08-07 13:37:27 -07:00
Matthew Jackson bca886a9c3 changelog: tighten 1.6.1, and record two entries it was missing
The entries had grown into narrative paragraphs; each is now the outcome
first and the mechanism in a sentence or two. The website changelog page
mirrors libfreemkv's, so it inherits this directly.

Two things that shipped in this cycle were not in it at all:

dir:// as a source. It was pulled from 1.6.1 on 2026-08-05 because the
folder reader produced a wrong title list on a real DVD. That defect was
the CSS descramble bug below, fixed since — the acceptance suite now
shows a folder reporting the same 38 titles as its ISO, with matching
streams, languages and runtime, and the CLI has been wired for it all
along. It ships, so it is listed.

The CSS title-table corruption itself: a disc enumerating 38 titles
produced a decrypted image enumerating 10, silently, at exit 0. That is
a data-integrity fix a user needs to know about.
2026-08-07 13:15:36 -07:00
Matthew Jackson 7cda914f29 Stamp the last frames that were still leaving without provenance
A real rip caught these. The warning added for a track placed from
timestamps under a seam plan fired once, on one subtitle track of a
23-clip title, eighteen minutes in — a display set that arrives as a lone
non-PCS segment is emitted straight through rather than accumulated, and
that path still built its frame with source: None.

Auditing the rest the same way found four more: flac, lpcm, mpegaudio and
the passthrough parser. All are one-PES-one-frame, so the packet's own
facts are the unit's facts.

Every Frame construction in every codec parser now carries a source,
checked by walking balanced braces rather than by eye — the earlier
count was taken with a regex that cannot see a block containing nested
braces, which is how these survived the first pass.
2026-08-07 10:01:39 -07:00
Matthew Jackson 11538981de Place a non-advancing mark table by provenance
Marks that do not advance across a title are normal, not a defect: each
clip file carries its own STC, so one clip's IN has no ordering
relationship to the previous clip's. Refusing those tables dropped
exactly the branched titles this type exists for onto the inference path
that cannot read them, and that is the overrun.

Lifting it was tried once before every track carried a source offset, and
nine audio and subtitle tracks with nothing to place them by pinned
themselves to clip 0 and dropped most of the title. Every parser stamps
provenance now, so the clip comes from the byte offset and the marks keep
one job: whether a frame lies inside its own clip's [in, out].

Still gated on spans_trusted. Without usable spans there is no offset to
place by and inference is all that is left, so a table inference cannot
read is still refused rather than silently truncating a title.

A frame reaching the timestamp path under a plan now says so once per
track. It is not expected any more, and it is how a mostly-dropped track
reached a user without a single line in the log.
2026-08-07 07:49:53 -07:00
Matthew Jackson 8edb53942c Every parser now carries provenance, on the one shared buffer
dts, ac3 and truehd assembled access units across PES packets with three
private implementations of the same bookkeeping, and none of them carried
the source byte offset. They now hold a PesBuf, so a unit takes the
timestamp AND the source of the packet covering its first byte, from the
same mark, and no codec can answer that question its own way again.

dts is the reference case: its pts_marks already implemented the rule
correctly, and all 61 of its existing tests -- including the PTS
attribution ones -- pass unchanged on the shared type. That is the
evidence the type preserves the behaviour dts had right.

ac3 kept a single carry-over timestamp and one anchor offset, so it could
only attribute the first unit in a call; it now resolves each unit at its
own offset. truehd had no attribution at all beyond a running clock.

New tests cover the case that motivated this: a unit whose first bytes
arrive in one packet and whose remainder arrives in the next keeps the
FIRST packet's offset. At a clip seam those two packets belong to
different clips, and taking the later one places the audio in the wrong
one.

Clippy on the pinned toolchain caught an empty `if` block left where dts
used to clear stale marks -- restored as an explicit clear, with why it
is still needed once drain keeps the covering mark.
2026-08-07 07:45:32 -07:00
Matthew Jackson c718cd9e9e One rule for which packet a unit's facts come from
A parser that assembles an access unit across PES packets has to answer
one question for every unit it emits: which packet carried this unit's
FIRST byte? Its timestamp comes from that packet, and so does the source
byte offset that says which clip of a multi-clip title it belongs to.
The packets that complete the unit carry later values that must not
override it.

That question was being answered three ways. DTS kept a deque of
(offset, pts) markers and took the one covering offset 0. AC-3 kept a
single carry-over timestamp. TrueHD kept its own. PGS and DVD subtitles
each held a pending unit with just a start time. None of them carried
the source offset at all, which is why provenance existed only for video
and why nine audio and subtitle tracks on a branched title had nothing
to place them by.

So it lives in one place now. PesBuf owns the bytes AND the marks;
PesFacts returns a packet's timestamp, source and discontinuity
together, so a parser cannot take one from one packet and another from
the next, because it does not assemble them itself.

The type answers WHICH packet. How a codec reads a timestamp out of that
packet stayed the codec's business at first, and that turned out to be
the same drift one level down: dvdsub read pts alone and returned 0 for
a packet carrying only dts, while everything else took pts.or(dts). Now
there is one derivation. It is not a choice between two fields — for
audio and subtitles there is no reordering, so dts IS the presentation
time and reading it is reading the same value from whichever field the
packet used. Reordering video never calls it; that path reconstructs
display order instead.

Migrated: adts, pgs, dvdsub. dts, ac3 and truehd follow.
2026-08-07 07:19:58 -07:00
Matthew Jackson e32f874387 Restore the refusal: only the video parsers carry provenance
Accepting a table whose marks do not advance was wrong, and a real
22-clip title showed exactly how. The rip failed the drop-volume gate at
84 percent with more frames dropped than kept.

The cause is a fact the code never stated: provenance is VIDEO-ONLY.
Every audio and subtitle parser -- dts, ac3, adts, truehd, pgs -- builds
its frames with `source: None`, while hevc, h264 and vc1 propagate it.
So on that title the video track placed correctly by byte offset and
nine audio and subtitle tracks arrived with nothing to place them by.

The branch added to keep an unprovenanced frame from stranding did the
stranding itself. It held such a frame on its track's current clip,
reasoning that the cursor only ever advances under provenance and so
could not be wrong -- which is true only for a track that eventually
receives some. A track that never receives any stays pinned to clip 0
for the whole title, and every frame past clip 0's OUT mark is dropped.
That is what the disc showed: nine tracks pinned to clip 0, the first
drop nine milliseconds past its OUT.

So the refusal goes back, now with the real reason recorded, and the
test asserts it rather than asserting the behaviour that failed.

Lifting it needs the audio and subtitle parsers to carry provenance
first, stamped from the PES that STARTED each access unit -- the same
rule au_assembly already applies to video. The diagnostics that found
this are kept.
2026-08-07 03:21:21 -07:00
Matthew Jackson b1f9adc82b Say which frame and which marks caused a provenance drop
The heuristic path has always logged its first drop per track with the
frame's timestamp and the clip marks it was judged against. The
provenance path did not log at all, so a title that dropped MOST of its
frames there failed the volume gate without a single line saying which
frame, which clip, or which marks — a full-length rip produced zero drop
events and an error, which is not a diagnosable failure.

Also pins the all-identical-spans hazard as a test. Every PlayItem
referencing one clip file gives every clip the same span, which passes
the tiling check and is therefore "trusted" while carrying no
information at all about which PlayItem a byte belongs to. That is not
what the discs on hand do — theirs have one distinct span per clip — but
the check conflates "the spans tile" with "the spans distinguish", and
only the second justifies placing a frame by its byte offset.
2026-08-07 03:05:32 -07:00
Matthew Jackson 16a894f835 Say how many DISTINCT feed spans a title's clips share
A seamlessly branched title re-references one clip file from several
PlayItems with different mark ranges, and those references share a single
feed span because the bytes are read once. A byte offset alone cannot
then tell them apart.

Whether that is happening on a given title is the fact that decides
whether provenance can identify a clip on its own, and it was not
observable. distinct_spans < clips says it directly.
2026-08-07 02:45:57 -07:00
Matthew Jackson 9bcd752216 Say which placement strategy a title got
Whether a title was placed by the seam plan or fell back to inference is
the single most useful fact about a branched rip, and it was invisible.
The two looked identical in the output, so telling them apart meant
rebuilding and re-ripping — which is exactly what it cost to find that
most branched discs were silently on the inference path.

Log both flags once at plan construction, and log each refusal with the
reason that caused it.
2026-08-07 02:01:47 -07:00
Matthew Jackson c1cd2e4b06 Place a restarting-clock playlist by provenance instead of refusing it
A playlist whose clips each restart their own STC has marks that all
cover the same low values, so they are not points on one title-wide
clock. from_clips refused those tables outright and fell back to PTS-jump
inference.

That refusal predates provenance, and it is only about inference. With
the feed spans tiling the title the clip comes from the frame's byte
offset and the marks are never read across clips: each clip's offset_ns
maps its own private clock onto the output timeline, which is exactly
the right operation for a restarting STC.

Refusing unconditionally therefore turned the seam plan off on most of
the branched discs on hand — the very titles it was written for. They
ran on inference with a timeline minutes past the title's real length:
one 2h29m title came out with its audio and video both spanning 3h15m.

So the refusal now applies only when the spans cannot be trusted, where
it is still load-bearing: without a usable clock or a usable byte offset
there is nothing to place with.

Accepting these tables opens one hole, and it is the same stranding the
refusal guarded against. A frame arriving with no byte offset would fall
through to the mark heuristics, which on this kind of table are
meaningless and can strand a track on a clip it has already left,
dropping the rest of the title. Such a frame now stays on its track's
current clip; the cursor only ever advances under provenance, which is
never wrong about which clip a byte came from.
2026-08-07 01:17:04 -07:00
Matthew Jackson cbafe71769 Resolve a frame clip by binary search, not a linear scan
The provenance lookup runs once per frame per track. A linear scan over the
clip list is fine for the 11-PlayItem fixture and is not fine for the real
hoard: two discs there carry 900 PlayItems in one title, and several carry
250-450, so a scan would be hundreds of comparisons on every one of millions of
frames.

spans_trusted already guarantees the spans tile the feed contiguously and in
order, which makes this a partition point. A repeated clip reuses its first
reference span, so the search walks back to the FIRST entry sharing it — the
bytes are read once, so the material is emitted once, at that entry offset, and
the answer is stable regardless of which duplicate the partition lands on.

Also adds a test at the real 900-clip scale: every clip places, output stays
monotonic across all 900, an arbitrary byte resolves to the right clip out of
ascending order, and a byte past the end belongs to no clip.
2026-08-06 23:22:38 -07:00
Matthew Jackson 3c1375a245 Identify a frame clip by provenance, not by inferring it from timestamps
Four audit rounds each fixed one rule in SeamPlan::place and broke another,
because the question the rules were answering has no answer. Inside a
seamless-branching overlap clip k OUT comes AFTER clip k+1 IN — 57.8s of
overlap on the real fixture table — so a single timestamp is legitimately
inside two clips, and a clip file is not trimmed to its marks, so it also
carries material from before its own IN. No rule over timestamps can say which
clip a frame came from, and each attempt was right for one disc layout and
silently wrong for another: 65s of rewind, 17 minutes stranded, 28 minutes
dropped, 55s refused.

Frames already carry the byte offset they were read from (PesFrame::source,
stamped by the TS demuxer). Clip now carries the byte span its stream occupies
in the title feed, recorded while the extents are gathered. So the clip is a
LOOKUP: the offset falls in exactly one span. There is no decision to get wrong.

Every track of a clip lives in the same stream file and therefore shares one
span, so video, audio and subtitles agree by construction. Divergence between
them — each track guessing separately under its own tolerance — is how audio
and video ended up on different clips and drifted apart in the first place.

spans_trusted gates the whole path: unless the spans tile the feed contiguously
from zero, an offset means nothing and provenance is ignored in favour of the
mark heuristics, which is the 1.6.0 behaviour. A broken map degrades instead of
confidently selecting a wrong clip for every frame. A clip referenced twice
reuses its first span (the bytes are read once) and is still trusted.

Sources that stamp no provenance — a mkv:// remux, the deserialize hop — take
the heuristics, which is what they have always used and where they have always
been right, because they have no overlapping clips to be ambiguous about.

36 timeline tests, six of them new and covering: the overlap case marks cannot
see, all tracks agreeing, out-of-marks material dropped AND counted, a holed
span map, a discontiguous one, a repeated clip, and no provenance at all.
2026-08-06 23:08:13 -07:00
Matthew Jackson 651bfaaa2e Say when an entry is omitted for not being a plain file
A symlink to a directory — a normal way to keep tens of gigabytes of streams
off the system disk — was dropped from the plan silently. `entry.file_type()`
reports the LINK, so such a subtree never enters `dirs`, and `metadata()` then
follows it and reports a directory, so it fails the is_file() test and was
skipped with nothing said.

The result is the worst class this release exists to close: PLAYLIST and
CLIPINF still synthesize, so the folder scans and enumerates titles, every clip
resolves to no extents, the mux takes its clean-EOF path, and a near-empty MKV
is written at exit 0.

It is still skipped rather than followed — following link targets invites
cycles and escapes from the folder — but at the same volume as the
unrepresentable-name skip eight lines above.
2026-08-06 22:48:27 -07:00
Matthew Jackson cce4e5effe Enforce the no-rewind invariant instead of guessing at clip crossings
Round 7 enumerated the state space of SeamPlan::place and found three more
holes in the mark heuristics — the third consecutive round to find a defect
here, each time in the guard the previous round added:

- a clip file opening ON its own IN mark, after a tail frame past the previous
  clip OUT had already advanced the cursor, read as another crossing and jumped
  a clip too far: 28 minutes of clip 6 dropped;
- the last clip was unguarded entirely (`clip + 1 < len` skipped the check), so
  the 9->10 seam rewound 75.6s with dropped_total() at 0 — the exact round-5
  signature, reachable without any corrupt input;
- a glitch mid-clip satisfies "inside the current clip", so the round-6
  stranding survived for the rest of that clip.

Three patches to the same heuristic, three wrong in a new way. So stop patching
it. Everything above the placement is still heuristics over marks, but the
property they exist to serve is now checked DIRECTLY: a track output must not
run backwards. A placement that would rewind by more than
DISCONTINUITY_BACKSTEP_NS is refused; the scan then looks for a LATER clip
containing the frame that does not rewind, which is exactly what a genuine
overlap crossing looks like; and if none exists the frame is dropped and
counted, because emitting it anywhere moves the track backwards.

The tolerance is the backstep constant because B-frame reorder legitimately
emits out of order by a fraction of a second, while a rewind that matters is
orders of magnitude larger.

The cursor moves ONLY when a later clip actually accepts the frame. My first
version of this advanced on a failed search, which strands a track exactly as
before — caught immediately by the round-6 regression test.

All 29 previous timeline tests still pass, including
continuity_preserves_forward_gap. The three round-7 scenarios now have a test
each, with the real 00801.mpls numbers.
2026-08-06 12:31:23 -07:00
Matthew Jackson 8e6f1511ea Round 6: a large backstep only crosses when it would be misplaced
The round-5 fix advanced the clip cursor on ANY backward step over 3s. That
also fires for a corrupt PTS, and for a legitimate STC discontinuity inside one
clip — and nothing moves the cursor back, because a forward step matches
neither past_out nor stepped_back. Every later frame then sits below the new
clip IN and is dropped: on the fixture table that is ~17 minutes of one track
gone, and the only volume gate compares total drops against ALL tracks frames,
so it exits 0.

The branch now also requires the frame to be INSIDE the current clip marks,
which is the only case that would otherwise be silently placed at the old
offset — the rewind. A frame outside them needs no help: the containment check
drops and counts it, the cursor stays put, and the next good frame is placed
normally. Both behaviours have a test, each confirmed to fail without the guard.

Separately, the dir:// PES input path scanned the folder and never applied the
encryption verdict scan_dir exists to produce, so the same folder ripped
through one door and failed through the other asking for a key it does not
need. That logic now lives in one function, session::apply_folder_encryption_verdict,
called by both.
2026-08-06 11:02:48 -07:00
Matthew Jackson d68d73e68d Retry a short read instead of skipping past it
The previous commit bounded the crack scan inspection by the bytes actually
read, which stopped it examining stale buffer bytes from an earlier batch. But
the cursor still advanced by the REQUESTED count, so those sectors were skipped
outright — trading "scans the wrong data" for "silently scans less than it
thinks", on exactly the damaged media where a title key is hardest to find.

The cursor now advances by what was read, so the next iteration resumes where
the read stopped. Floored at one sector so a source returning Ok(0) cannot spin.

Found by reading the fix again rather than by the next audit round.
2026-08-06 10:15:53 -07:00
Matthew Jackson 648bcce5b6 Scan only the sectors the crack actually read
read_sectors returns the number of bytes written and a source may return Ok
with fewer than asked — a recovery read over a damaged region does exactly
that. The crack loop discarded the count and inspected all n sectors of a
buffer that is REUSED across batches, so the tail still held the previous
batch sectors.

Cracking a key from those means cracking from data belonging to a different
extent, possibly a different VTS, while crack_span records the CURRENT one. A
key that opens nothing in this region is then installed, and its wrong
descrambles are only partly caught by the per-sector crib.
2026-08-06 09:31:37 -07:00
Matthew Jackson 45af2152a8 Make the public per-sector descramble refuse a non-pack sector
descramble_region was fixed to require the MPEG-2 pack start code, but
descramble_sector — the PUBLIC per-sector entry point, and the one the
module-level example tells callers to use — still keyed on the byte 0x14 flag
bits alone. The crate own documented guidance therefore led straight back into
the defect this release exists to fix: a VIDEO_TS.IFO sector holding 0x15 at
0x14 while starting 00 26 00 00 loses 1912 of its 2048 bytes, and because that
sector carries TT_SRPT the disc enumerates 38 titles while an image decrypted
from it enumerates 10, at exit 0.

It has no callers inside the crate, which is exactly why it survived three
rounds: nothing exercised it. Putting the guard inside the function rather than
in each caller is what keeps the safe path the easy one.

The integration test that covers it built its sector from the flag byte alone,
which no real scrambled sector looks like, so it stopped representing the path
it names — the same fixture-realism gap already fixed in four other places this
release.
2026-08-06 09:26:15 -07:00
Matthew Jackson abbc725acf A clip join must never rewind the output timeline
A hole in the seam fix itself, found in audit round 5 and confirmed by
measurement against the real 00801.mpls marks already in the test file.

Nothing upstream trims a clip stream to its marks, so a clip file can open with
material from BEFORE its IN. With the cursor on clip 5 (7708.99..7910.79) a
frame at 7845.00 — clip 6 pre-mark lead-in, 8s below clip 6 IN of 7853.00 —
sits further from that mark than the 250ms tolerance, so neither crossing rule
fires. The cursor stays on clip 5, and 7845.00 IS inside clip 5 range, so the
frame is PLACED with clip 5 offset.

Measured: output went from 3714.51s to 3649.51s. Backwards 65 seconds, with
dropped_total() still 0 — no counter, no gate, nothing noticed — while the
whole 57.8s overlap band was emitted a second time over clip 5 written tail.
That is the collide-and-flatten symptom this type exists to remove.

A track own PTS only ever runs forward inside a clip, so a backward step larger
than DISCONTINUITY_BACKSTEP_NS is always a clip change. Such a frame now
advances to the first clip that could contain it — which terminates on its own,
rather than running to the end of the list, which is what the next_in bound
exists to prevent — and a frame still below that clip IN is material the
playlist excludes, so the containment check drops and COUNTS it.

All 27 pre-existing timeline tests still pass, including
continuity_preserves_forward_gap, which is the one that catches an over-eager
heuristic here. The new test asserts the output never moves backwards at a
join and was confirmed to FAIL before the fix.
2026-08-06 09:20:39 -07:00
Matthew Jackson 2e3ba374f3 Test the demux seam gates too
The same gap as the MKV muxer: DemuxSink builds its timeline with
TimelineContinuity::with_clips, but no test ever handed it a title carrying
usable PlayItem marks, so frames_mapped and dropped_total() were always zero
and both gates in finish() were unreachable.

Deleting either left the whole suite green while a demux:// export of a
seamless-branching title wrote a directory of zero-byte track files beside a
populated chapters document and reported success at exit 0.

The test gives the title two clips with marks that exclude every frame, and was
confirmed to FAIL with the gate removed.
2026-08-06 09:05:30 -07:00
Matthew Jackson 84b4a6ddae Test the seam gate that stops a drift-broken title being skipped
Audit finding: both seam gates in MkvMuxer::finish were dead code under test.
set_clips was never called anywhere in the suite, so continuity.dropped_total()
was always zero and neither branch could be reached.

The ordering of the two zero-frame checks is load-bearing and the comment says
so: a title the seam plan emptied also has a zero frame count, and MkvInvalid
is classified by is_skippable_title_stub as an empty nav/menu stub — so
reporting it that way makes an all-titles rip drop a real feature and finish the
rest at exit 0. Swapping the checks left the entire suite green.

The new test builds a two-clip plan (from_clips needs at least two, strictly
increasing) whose marks exclude every frame, and asserts both that the error is
SinkWroteNothing and that it is NOT classified as skippable. Confirmed to FAIL
with the two checks swapped.
2026-08-06 08:45:58 -07:00
Matthew Jackson 3f4bf3f290 Round 3: fix three defects introduced by the round-2 fixes
Auditing my own fixes found all three. None were in the original code.

The VTS crack sort was byte-wise case-SENSITIVE while the filters that select
those files (vts_group_of / is_title_vob) are case-insensitive. On a
case-sensitive volume a set holding vts_01_1.vob beside VTS_01_2.VOB sorted
part 2 first, because V (0x56) precedes v (0x76) — reintroducing exactly the
budget-exhaustion the ordering exists to prevent. Now sorted on the same
uppercase normalisation the filters apply.

Refusing a name that round-trips to empty aborted the WHOLE plan. A sidecar
folder named with a single emoji made a backup un-rippable that 1.6.0 handled
fine, and reported it as a collision with a file that does not exist. It is now
skipped with a warning: the entry is unaddressable either way, but one
irrelevant file should not cost the user their rip.

The mtime check now applies only to files whose CONTENT the plan read — the
IFOs, whose bytes 0xC0/0xC4 place every VOB. Everything else is planned from
size alone, which is already checked, so comparing mtime there bought nothing
and risked a real false positive: disc backups commonly live on exFAT/FAT32,
which stores local time, so a long rip spanning a DST transition would see a
whole-hour shift on an untouched multi-gigabyte VOB and abort hours in.
2026-08-06 08:25:54 -07:00
Matthew Jackson edb6f0035b Catch names the reader cannot tell apart, and in-place content changes
Two dirimage findings, both silent-wrong-output.

The per-directory uniqueness check compared raw host names, but the reader does
not see raw host names: parse_udf_name trims leading and trailing whitespace
and drops any code unit char::from_u32 rejects — which is every half of the
surrogate pairs the encoder emits for non-BMP characters. So " 00000.m2ts" and
"00000.m2ts", or "A<astral>.m2ts" and "A.m2ts", were two entries at plan time
and ONE name at read time. find/read_file take the first match, so a title
resolved to the wrong file extents and muxed the wrong bytes at exit 0 — the
exact shadowing DirNameCollision exists to prevent.

The key is now derived by round-tripping the name through the very encoder and
parser that will be used, so it cannot drift from them. A name that survives
that round trip as empty is refused outright: it would exist in the image and
be addressable by nothing. The new test builds a real folder, calls plan, and
was confirmed to FAIL with the fix reverted.

Separately, the plan-vs-read revalidation compared file LENGTH only, while the
plan depends on CONTENT: a DVD VOB placement comes from bytes 0xC0/0xC4 of its
IFO, and IFOs occupy a whole number of sectors so an in-place rewrite keeps the
length. A re-authoring tool touching the folder mid-rip would pass the size
check while every title extent pointed at stale sectors. mtime is now compared
alongside size, and only when both sides report one, so a filesystem without
timestamps falls back to the old behaviour rather than failing every read.
2026-08-06 08:00:23 -07:00
Matthew Jackson 5fc28056c3 Anchor the folder encryption probe on a unit boundary
probe_folder_encryption sampled AACS units starting from the largest extent
anywhere in the disc. AACS units are 3 sectors and a unit boundary is only
guaranteed at the START of a clip, so that anchor is only correct when the
largest extent happens to be a clip first extent.

For any clip over ~2 GiB it is not. The planner caps an allocation descriptor
at MAX_AD_BYTES = 524287 sectors, so every full piece of a split file ties on
sector_count and max_by_key returns the LAST tie — an extent beginning
(k-1)*524287 sectors into the file. 524287 % 3 == 1, so that start misses the
unit boundary for two file sizes in three.

The 6144-byte sample windows then begin mid-source-packet and the byte read as
the CPI flag is content. Both verdicts fail in a costly direction: a decrypted
folder is rejected as encrypted (DirImageEncrypted on something perfectly
rippable — the exact case scan_dir was added to rescue), or real ciphertext
reads as clear and the mux writes it out as video at exit 0. is_unit_aligned
cannot catch it: it measures against the same wrong base.

Now the largest TITLE first extent, which is unit-aligned by construction and
is also the more meaningful sample — the main feature rather than whichever
fragment happened to be biggest.

The existing tests could not reach this: their fixture m2ts is 786,432 bytes,
a single extent, which is always its own first.
2026-08-06 07:37:59 -07:00
Matthew Jackson 51de027d15 Order a VTS crack by filename, not by directory order
Round 1 removed a largest-first sort from resolve_vts_key because
largest-first is the 1.5.1 garbage bug. Auditing that fix showed it was only
half right: `planned` comes from walking the UDF directory, which yields File
Identifier Descriptors in on-disc authoring order with nothing sorting them. So
deleting the sort did not restore playback order, it left the order undefined —
whatever the disc happened to list first.

DVD-Video numbers a title sets VOBs in playback order by spec (VTS_xx_1.VOB ..
VTS_xx_9.VOB, single digit), so ascending filename IS playback order and is
deterministic regardless of how the directory is laid out.

Order decides correctness here: crack_key shares one sector budget across the
whole extent list, a CSS DVDs biggest cell opens with a long clear run, and CSS
recovers the title key from scrambled data itself. Starting in the wrong place
can exhaust the budget without ever meeting scrambled data, whereupon the
caller falls back to the disc-wide key and the whole VTS is descrambled wrongly
— corrupt PES behind an intact header, written out at exit 0.
2026-08-06 07:31:35 -07:00
Matthew Jackson 96282ecda7 Audit round 1: playback order, silent title drops, image durability
Four fixes from the first audit round. Every finding was verified against a
pinned tree and read directly before being accepted.

resolve_vts_key sorted a VTS title-VOB extents largest-first. That is the 1.5.1
garbage bug, and it grew back in a new code path: the comment claimed it
"matched the scan heuristic", but that heuristic WAS the bug and had already
been fixed in decrypt_keys_for_title, which documents the rule (PLAYBACK ORDER,
never largest-cell-first) and pins it with a regression test. A CSS DVDs biggest
cell opens with a long clear run and crack_key shares one sector budget across
the extent list, so starting there can exhaust it without ever MEETING
scrambled data — and CSS recovers the key from scrambled data itself. The crack
then returns None, the caller falls back to the disc-wide key, and every VOB in
that VTS is descrambled wrongly: corrupt PES behind an intact header, written
out as a complete extract at exit 0.

parse_pgcit dropped titles silently in THREE places — an unparseable PGC, an
out-of-range PGC index, and a truncated entry table. The finder caught one; the
other two turned up on reading the function. parse_vmg already counts and warns
per skipped title SET for exactly this reason, and this was the last place a
disc could quietly report fewer titles than it has.

write_image called flush() and returned Ok. flush() only pushes bytes into the
page cache and promises nothing about durability, so a 6-90 GB image could be
reported complete while still unwritten — a crash or an unmounted volume then
leaves a truncated file the caller was told was finished. Now into_inner (so a
buffered-write error surfaces instead of being dropped by BufWriter::drop)
followed by sync_all.

timeline used abs() on a saturating_sub result. Every other comparison in that
module is saturating because the timestamps come off a disc and are not
trusted; abs() panics on i64::MIN, which saturating_sub can produce.
2026-08-06 07:11:52 -07:00
Matthew Jackson 6e2ee9e002 Rename is_scrambled to has_scramble_flag_bits
The 1.6.1 DVD fix removed the USE of the loose predicate but left the
predicate sitting there with the better name. Anyone asking "is this sector
scrambled?" finds is_scrambled before is_scrambled_pack, and reintroduces the
defect that destroyed 1912 bytes of a real VIDEO_TS.IFO — the sector carrying
TT_SRPT — so the disc enumerated 38 titles and an image decrypted from it
enumerated 10, silently, at exit 0.

Byte 0x14 only means "scrambling control" inside an MPEG-2 pack. In an IFO,
UDF or ISO 9660 sector it is whatever that format stores there. The new name
says what the function actually tests and nothing more, so the honest question
has the obvious name and the dangerous one has to be asked for deliberately.

Its doc comment also claimed a caller, decrypt::decrypt_sectors, that does not
exist — so the name was an invitation and the documentation was an argument
for accepting it. It has no production callers at all; it stays public because
an integration test asserts the flag extraction directly.
2026-08-05 22:27:04 -07:00
Matthew Jackson 0f8035b15a Scope the qa matrix to the platforms each crate actually supports
The first cut ran release-profile tests on all three platforms everywhere,
which invented coverage no crate had ever claimed. bdemu is Linux-only — its
ci.yml has no macOS or Windows job at all, and its tests call
ExitStatus::signal(), which does not exist on Windows, so they do not compile
there let alone run. The rest follow the policy ci.yml already set: tests
execute on Linux and macOS, Windows compiles them so the cfg(windows) halves
still get codegen before release time.

That policy is now explicit in qa.yml as a windows-build job rather than
implied by which jobs happen to exist.

Recorded, not papered over: running the suites on Windows DOES find real
defects — autorip has five Windows path-handling failures in mover.rs and
freemkv-engine has three tests hardcoding /dev/null. freemkv ships a Windows
GUI, so they matter. They are pre-existing rather than 1.6.1 regressions, and
fixing them is its own piece of work.

Also build the binary before cli-parity.sh. cli-integration.sh builds its own;
cli-parity.sh only checks for one and exits 2 with "build it first", so both
legs died before running a single case.
2026-08-05 21:22:30 -07:00
Matthew Jackson f67fd92b60 Give Windows a free-space gate, and stop two tests timing the scheduler
Three release-profile failures on macOS and Windows, all found by the qa gate
on its first run. Release-profile tests on those platforms had never run
before it existed, so none of these were regressions — they had simply never
been visible.

available_space returned None on every non-unix target. That did not merely
skip a test, it skipped the GATE: a Windows user extracting a disc to a full
volume got a confusing failure part-way through instead of a clear refusal up
front, and Windows is where the GUI ships. GetDiskFreeSpaceExW is declared
directly against kernel32, matching how scsi::windows already reaches Win32
rather than pulling in a binding crate for one call. It asks for
FreeBytesAvailableToCaller, which accounts for per-user quotas — the same
question f_bavail answers on unix.

The other two asserted on wall-clock timing with no margin:

- sleep_until_halted_wakes_mid_sleep bounded the wait at 350 ms and measured
  377 ms on a loaded runner. That bound measures the scheduler, not the wake.
  What the test is for is distinguishing "woke because the flag flipped" from
  "woke because the 10 s timeout expired", and 2 s does that just as well.

- abandon_loses_to_a_close_already_committed released at 600 ms against two
  300 ms grace windows plus a 250 ms poll cadence, so the windows could expire
  first and the caller abandoned — a race, not a defect. The intervals are
  scaled up so jitter is small relative to them; the ordering under test is
  unchanged, only the margin.
2026-08-05 21:20:29 -07:00
Matthew Jackson 7ced0d9b61 Stamp every qa push as a release candidate
Every push to qa now tags v<version>-rc<N>, N incrementing, before the gates
run. That answers "which build is on qa, and is it the one I tested?" without
anyone having to remember it.

The tag lands whether the run goes green or red, deliberately. A red candidate
needs a name more than a green one does: "rc3 failed release-tests on windows"
is a sentence you can act on, "qa is red" is not. Red on qa is the gate doing
its job — the branch saying this is not production worth yet.

release.yml now excludes v*-rc*. Its trigger was v*, which matches the
candidate tags, so without this every push to qa would have built and PUBLISHED
a GitHub release — including for the candidates that failed.
2026-08-05 21:03:01 -07:00
Matthew Jackson c2c00a874c Add the qa gate: dev for speed, qa for proof
dev -> qa -> main, across every public repo.

dev is where work lands and is meant to be pushed to often, so ci.yml stays
the fast answer: fmt, clippy, unit tests, leak-guard. qa is the release
candidate, and qa.yml is the claim that a commit is production worth —
release-profile tests on all three platforms, and the Linux cross-target
clippy that the local precommit gate has always run but CI never did.

Release profile matters as its own gate: overflow checks are off, debug_assert
is compiled out, and inlining changes what the optimiser can prove. A test
that only passes in debug never guarded the binary anyone ships.

qa.yml checks siblings out at qa rather than dev. This is not a monorepo, and
a qa run resolving its dependencies from unreleased dev tips would go green on
a combination that is not the one shipping — the exact mismatch the branch
model exists to prevent. ci.yml now tracks whichever branch triggered it for
the same reason, since it also fires on qa.

The consequence is that repos move to qa together, in dependency order. qa is
backfilled from main so an unchanged crate still presents working code to the
crates built on it.
2026-08-05 20:59:34 -07:00
Matthew Jackson 87463ed3ee Require a pack start code before descrambling a sector
css::is_scrambled reads bits 4-5 of byte 0x14 and nothing else. That is a
sound test once a caller has committed to a title's VOB data, where every
sector is an MPEG-2 PS pack and byte 0x14 always means what it says.
descramble_region is not such a caller: it is handed arbitrary regions of a
disc, so it also sees IFO, UDF and ISO 9660 sectors — raw structures where
byte 0x14 is whatever that format happens to store there.

Measured on a real disc: the second sector of VIDEO_TS.IFO holds 0x15 at
offset 0x14 while starting 00 26 00 00, which is not a pack. The flag test
read it as scrambled, descrambled it, and destroyed 1912 of its 2048 bytes.
That sector carries TT_SRPT, so the title table went with it — the disc
enumerated 38 titles and an image decrypted from it enumerated 10, silently,
at exit 0.

is_scrambled_pack already existed with the right predicate. Use it here. It
costs nothing: a genuinely scrambled VOB sector always carries the pack start
code, and no IFO sector does.

Verified end to end — the decrypted image's `info` output is now identical to
the source disc's, 38 titles both, differing only in the CSS: Encrypted line.

The fixtures moved with it. Four of them built a sector by setting byte 0x14
alone, which no real scrambled sector looks like; they now build packs.
2026-08-05 20:59:14 -07:00
Matthew Jackson 0c5e419b07 Report the resolved extent alongside each IFO read
The diagnostic hashed IFO contents but not where they came from, so a
content difference could not be told apart from a path resolving to a
different place. It now prints file_start_lba and file_extents next to the
hash.

That measurement is what inverted this investigation: both a CSS disc and
its decrypted copy resolve VIDEO_TS.IFO to the same extent, and only one
sector of the fourteen differs — by exactly a descrambled payload with the
scrambling-control bits cleared. So the decrypted copy holds the correct
bytes, and an iso:// scan of a CSS disc is parsing a still-scrambled IFO
sector, because scan_iso opens a plain FileSectorSource.
2026-08-05 20:24:20 -07:00
Matthew Jackson 218cd1f6f8 Diagnostics that localise the decrypted-DVD title loss
Two opt-in dumps, driven by FMKV_IMAGE, that narrow where a decrypted DVD
image loses titles:

dump_title_sets_for_an_image reports what survives parse_vmg. On one disc
the CSS image yields 13 title sets and 38 titles; its decrypted copy
yields 8 and 10. Sets 8, 9, 10, 12 and 13 are dropped outright, set 11
parses but returns no titles at all, and set 7 returns 3 of 5.

dump_vts_ifo_reads_for_an_image reads every VTS IFO and hashes the
CONTENT. All thirteen are byte-identical across the two images, so
parse_vts is handed the same bytes and the same TT_SRPT info and still
fails on one of them — the divergence is in what it reads from the READER
afterwards, which is file_start_lba and the PGCIT.

An earlier version of the second dump compared only length and magic and
so wrongly reported the images as identical; it hashes the bytes now.
2026-08-05 20:14:46 -07:00
Matthew Jackson 2c4851e6c1 Say when a title set is dropped from a DVD scan
parse_vmg skipped any title set whose parse failed, with no log and no
counter. A real disc enumerated 38 titles from one image and 10 from
another, and the 28 discarded failures were invisible — the symptom read
as a scan difference rather than as dropped reads, which is most of why
it took so long to localise.

Behaviour is unchanged: a disc may legitimately carry placeholder TT_SRPT
entries, so one failure is still not fatal. It now warns per skip with the
title set and the error, and once at the end with kept-versus-declared.

Also adds an opt-in diagnostic that reads every VTS IFO from an image, to
separate a read failure from a parse failure. It reports all 13 sets
reading identically from both a CSS image and its decrypted copy, which is
what proves the 38-to-10 loss is downstream of these reads.
2026-08-05 19:41:43 -07:00
Matthew Jackson 7511adb64e Two opt-in scan diagnostics for the DVD decrypted-image defect
dvd_placement_invariant_on_a_real_folder checks, per title set, the sum
ifo.rs relies on: file_start_lba(VTS_nn_0.IFO) + vtstt_vobs must land on
VTS_nn_1.VOB. It reports all 13 sets correct on a real DVD folder, which
is what excluded placement as the cause.

dump_titles_for_an_image prints every title a scan produces with the
numbers canonical_title_order sorts on. It is what showed the real shape:
the same disc scans to 38 titles as a CSS image and 10 once decrypted,
with identical capacity and a byte-complete image.

Both are #[ignore]d and read their target from the environment, so they
cost the gate nothing and are there for whoever picks the defect up.
2026-08-05 19:33:58 -07:00
Matthew Jackson 84a6c0d373 Round 5: fix the gates added in round 4, and two placement holes
The zero-frame check ran before the seam gate, and its error is
classified as a skippable nav stub — so a title the plan dropped
ENTIRELY was reported as an empty stub and an all-titles rip would omit
a real feature and finish the rest at exit 0. The seam case is decided
first now, with a code that is not skippable.

The demux sink read a frame's track kind out of the FILTERED slot, which
is empty for a class the export drops. On an audio:// or sub:// export
the video track was therefore called non-video and handed the permissive
crossing rule — the same defect round 4 fixed for a Dolby Vision layer,
reintroduced one file over. Video tracks are now recorded before the kind
filter, beside the primary-video reference that exists for this reason.

Its frame counter counted frames PLACED, not written, while its name and
doc claimed otherwise. Renamed and documented for what it is, including
that it cannot see a single lost track among many.

Placement: files in a subdirectory of VIDEO_TS were never given data.
They were declared at full size with no extents, so they appeared in the
tree and read as nothing. The same folder under BDMV was always placed
correctly. And the duplicate title-set guard keyed on the constraint maps,
so an IFO declaring no offsets inserted nothing and a colliding second IFO
went undetected — it keys on the groups seen now.

Display for SeamPlanDroppedMost and ShortImageRead discarded their
payloads, and four new variants were missing from the code-uniqueness
test.
2026-08-05 18:40:14 -07:00
Matthew Jackson 7cc5b0c93d Test the image-size cap against a real folder
The cap was added without a test. Its companion — a modest oversize is
honoured as a gap — already existed, so this pins the other side: an
offset past the ceiling is refused rather than grown into, which is what
keeps a rewritten IFO from planning a multi-terabyte image.
2026-08-05 18:30:23 -07:00
Matthew Jackson f4fc730d83 Make the drop count gate the verdict, not just the log
Round 3 counted frames the clip marks excluded and reported them at
finish. Counting is not bounding: the only other gate was a global
zero-frame check, and its error is additionally classified as a skippable
nav stub, so a title whose marks do not line up with its PES clock could
discard almost all of itself and still exit 0 — a two-hour feature
emitting seconds, which is the defect this change set already shipped
once. Dropping more than was kept is never a real join, so it now fails.

The demux sink had no zero-output guard at all, so a fully-dropped title
finished cleanly: a directory of zero-byte track files beside a populated
chapters document. It now refuses, keyed on frames having been OFFERED —
a chapters-only export, or a track class the title does not carry,
legitimately writes none, and two existing tests correctly said so.

A title set's placement group is the parsed number, so VTS_01_0.IFO and
VTS_1_0.IFO land on one key and the second silently overwrote the first's
constraint, placing a VOB where the IFO the reader uses does not point.
Refused rather than resolved by arrival order.
2026-08-05 18:25:47 -07:00
Matthew Jackson db0ca3367f Key the seam-crossing rule on reorder, not on driving epochs
A Dolby Vision enhancement layer is a second video track: it does not
drive epochs, but it does carry B-frame reorder. The crossing rule was
keyed on driving epochs, so the EL took the branch whose premise is that
the track has no reorder — and its ordinary reorder dip near the end of a
clip, which during an overlap also lands inside the next clip's range,
was read as a join. The EL was then placed on the next clip's offset, out
of step with the base-layer frame it must be co-timed with by the width
of the overlap: the same desync the per-track cursor was added to remove,
reintroduced for one track.

The property the rule actually depends on is whether a backward step can
be reorder, so it is now keyed on that. Both sinks derive it from the
track kind rather than from the epoch driver.
2026-08-05 17:52:13 -07:00
Matthew Jackson cc870a4614 Keep the seam plan to Blu-ray, and stop a missed crossing truncating a title
Two findings from the same escalation, both silent-wrong-output.

The plan was built for every multi-clip title. Only a Blu-ray PlayItem's
IN/OUT are positions in the clock the PES PTS runs on. HD-DVD fills the
same fields from the XPL's title-relative times and a DVD's come from
cell tables, so a plan built from them is an identity map with a drop
filter: it suppresses the layer-break rebase inference performs, and
drops whatever falls outside marks the PTS was never measured against. An
earlier reading of this called HD-DVD safe because its marks are
contiguous and every computed offset was zero — true, and irrelevant,
because they were zero in the wrong clock. Gated on the content format,
with a test using an HD-DVD-shaped table that the clock check alone
accepts.

The crossing test was also one-shot. A table whose clips restart their
own bases could miss it, and a missed crossing STRANDS the track: every
later frame falls outside the stranded clip's marks and is dropped for
the rest of the title. Counting drops, which is all the previous round
added, does not bound them. A table that is not one advancing clock is
now refused outright and falls back to inference, which is the documented
safe path for those titles.

Also from the same round: read_sectors added an unchecked lba + i, where
callers deliberately saturate their LBAs — a wrap folds the read back to
a low sector and hands the muxer another file's bytes. classify added 1
to two numbers parsed verbatim out of a filename. read_head used a single
read() where a short read on a network mount silently records no
placement constraint at all. And the page-cache eviction added last round
released only the read that crossed its threshold rather than everything
accumulated, so seven eighths of what was read stayed pinned.
2026-08-05 17:31:43 -07:00
Matthew Jackson 211d293aee Make the name-cap tests exercise the planner, not the constants
The round-1 tests asserted arithmetic about MAX_CS0_NAME_BYTES and never
called plan(), so both would have passed with the guard deleted — which
is the failure mode this audit exists to catch, committed by the audit's
own fix. They now build a real folder containing a 255-byte name and
require the planner to refuse it, plus a companion proving a name at the
cap is still accepted so the guard is not merely refusing everything.

The subdirectory cap keeps its arithmetic-only test — creating 65,535
directories is not reasonable in a unit test — but now says so instead of
implying coverage it does not have.
2026-08-05 17:21:43 -07:00
Matthew Jackson abf47137ef Audit round 2 fixes: an unsafe default, four omissions, and two swallowed errors
The folder encryption probe returned "not encrypted" when it had sampled
nothing at all — a title shorter than one aligned unit skipped the loop
entirely. That verdict CLEARS the structural one an AACS directory
raised, so a genuinely encrypted folder would have been ripped as clear
and written ciphertext as video at exit 0. With no evidence it now keeps
the structural verdict, and its bounds arithmetic no longer trusts
disc-derived values not to wrap.

Reading an IFO header swallowed every I/O error and returned an empty
buffer, which sent each placement offset through unwrap_or(0) and
recorded no constraint at all — a permission error on one file produced a
silently misplaced VOB. The directory walk swallowed the same class while
claiming to skip only vanished files. Both now propagate; only NotFound
is skipped.

Four things the round-1 changes left inconsistent: two new error codes had
no doc comments, were absent from the io::Error mapping, printed no path
in Display, and were missing from the test that proves codes are distinct.
The demux sink dropped frames silently while the MKV muxer reported them.
And set_clips had been inserted INTO write_frame's doc comment, leaving
write_frame undocumented and its paragraphs describing the wrong function.

uid/gid used 0 as "not specified"; UDF's sentinel is 0xFFFFFFFF, and 0 is
root.
2026-08-05 17:16:03 -07:00
Matthew Jackson 00332e4dba Bound the synthesized image, and stop it pinning the page cache
A DVD title set records where its VOBS begins as an offset inside its own
IFO, and the planner honours that offset because honouring it is what
makes a real backup readable. Nothing bounded it: a regenerated .BUP or a
hand-assembled folder naming an offset far past the content grew the
image to wherever it pointed — a u32 sector count reaches ~8.8 TB, and
writing that to an iso:// destination fills a disk with zeros before
anything notices. Capped at 128 GiB, which clears BD-100 with room.

Metadata is materialized up front and held for the life of the image, at
a 2 KiB File Entry per node, so the 100,000-entry cap alone permitted
~205 MB of it for content of no size at all — and the mux holds two
images at once while probing. The module claimed a budget of a few MiB;
that budget is now enforced rather than asserted.

Host reads had no page-cache eviction. The ISO source documents what that
costs, measured: an 85 GB read pins the whole file, starves the writer,
and collapses the mux to 2.7 MB/s against 70 MB/s isolated. A folder
source reads host files the same way, so it now uses the same eviction —
the hints move from private-to-that-module to crate-internal rather than
being reimplemented.
2026-08-05 17:10:19 -07:00
Matthew Jackson 6152028271 The VMG's 0xC4 is TT_SRPT, not VMGM_C_ADT
The placement code skips that field for the Video Manager, which is
right, but said it was skipping it because the field is the menu cell
address table. It is the title search pointer table, and it is an offset
inside the IFO rather than a pointer to another file — which is the
actual reason it constrains nothing. ifo.rs reads the same offset under
the correct name, so the two would have drifted.
2026-08-05 17:03:19 -07:00
Matthew Jackson abe464d799 Audit round 1 fixes: sparse-track joins, silent drops, and two encoder wraps
A sparse passive track — a subtitle with no event near a clip's mark —
was held to the dense-video crossing window, so it stayed on the previous
clip's offset until its PTS passed that clip's OUT and every event in
between was mistimed by the overlap. Video keeps the tight window,
because its backward steps are also B-frame reorder; passive tracks have
no reorder, so any backward step into the next clip's range is a join.

Frames the marks exclude were dropped without a trace. Dropping is right
at a join, but this codebase has shipped complete-looking wrong output
before, so the count is kept per track and reported when the mux
finishes, alongside the pre-cluster counter that exists for the same
reason.

A File Identifier Descriptor records its name length in one byte, and the
length was narrowed with a cast: a 255-byte name — POSIX NAME_MAX,
entirely ordinary — encodes to 256 and wrote zero, which would read every
later entry in that directory from the wrong offset. A directory's link
count is 16 bits and was computed as 1 + subdirectory count, which the
global entry cap alone permits overflowing. Both are refused while
planning, where the tree can still be rejected cleanly.

The module and struct docs described inference as the whole algorithm;
they now say which path decides what.
2026-08-05 16:58:49 -07:00
Matthew Jackson 862c4e90ab changelog: dir:// is not in 1.6.1
The folder reader produces a wrong title list on a real DVD — the title
table is read at a sector offset relative to VIDEO_TS.IFO's start, and a
synthesized layout does not put that file where the offset resolves the
same way. Output is silently wrong at exit 0, so the feature is not
wired to the CLI in this release. The library module stays, unreachable,
and ships once both disc families are verified against real content.
2026-08-05 16:44:23 -07:00
Matthew Jackson 7cfeadbcd3 changelog: dir:// as a source, and the multi-clip Blu-ray timing fix 2026-08-05 15:47:47 -07:00
Matthew Jackson 9742fe00fd Cross a clip join per track, not on the video's frame
The first cut placed every track with the cursor the video had moved. At
an overlap join the previous clip's audio is still arriving after video
has crossed, and those tail frames sit inside both clips' mark ranges —
so they took the new clip's offset, jumped forward by the overlap, and
collided with the new clip's own audio, which the muxer's monotonic nudge
then flattened. A remux confirmed it: the timeline length was already
correct and the original symptom was still there, 169 audio packets on
the tick floor.

A track's PTS only runs forward inside a clip, so its own backward step
to the next clip's IN is its crossing. That is per track, so the cursor
is too.
2026-08-05 15:46:40 -07:00
Matthew Jackson e814310a3e rustfmt the seam-plan tests
Line-wrapping only, no behaviour change.
2026-08-05 15:28:40 -07:00
Matthew Jackson 480c873be1 Merge branch 'fix/mpls-seam-timeline' into dev 2026-08-05 15:24:37 -07:00
Matthew Jackson 5043baaebe Place Blu-ray clips by the playlist's marks, not by guessing at PTS jumps
A seamless-branching title's PlayItems do not chain contiguously: one
clip's OUT can sit after the next clip's IN, where the disc stores the
join twice, or before it, where the playlist skips material. The mux
never saw those marks — its own header said so — and inferred seams from
PTS jumps instead.

Inference cannot recover this. A forward jump is ambiguous: it means the
playlist skipped, or it means frames were lost to damaged media, and
compressing the latter would falsify timing on exactly the rips that most
need it faithful. An overlap smaller than the B-frame reorder threshold
is invisible to inference entirely, and its duplicate content then
collided in the muxer, where the monotonic nudge flattened a run of audio
onto the tick floor and put sound ahead of picture for the rest of the
film.

Measured on one 11-PlayItem title: the file declared 7893.385 s, which is
what the playlist says the title is, and carried packets to 8029.298 s.
Both numbers came from the same program on the same disc. Four skips
totalling 135.9 s became dead timeline, and a 1.79 s overlap put audio
1.8 s ahead at the half-hour mark. Five of forty-seven titles were
affected; every single-clip title was exact.

So the marks are read. Each clip contributes exactly out - in, laid end to
end, so the output runs as long as the playlist says and a join never
rewinds. Titles without usable marks — DVD, HD-DVD, file sources — keep
the inference path unchanged, and clips that already chain contiguously
produce a constant offset, which is pinned by a test.
2026-08-05 15:24:31 -07:00
Matthew Jackson 4d63a11d63 Read a disc folder as an input: dir:// becomes a source
Users keep discs as extracted folders — a DVD VIDEO_TS or a Blu-ray BDMV,
usually a backup that is already decrypted. dir:// could only ever be a
destination, so those folders could be produced and never read back.

Everything above the sector layer wants a UdfFs over a SectorSource, and
every UdfFs read re-reads the ICB off that source at call time, so a
folder has to present itself as sectors. It does: dirimage plans a block
layout over the real files, encodes a UDF 1.02 filesystem for the
metadata, and serves data straight from disk. read_filesystem then parses
it exactly as it parses a disc, so nothing above changes — and the
iso:// arm of input() is now shared rather than duplicated, so dir://
inherits its decrypt gates, title selection and stream pruning.

The encoder is validated by more than its own reader: macOS mounts the
synthesized image and the mounted files compare byte-identical to the
originals. A round-trip through our own parser could not have shown that
— the tag CRC seeds at zero, and a wrong seed would satisfy us and no
real driver.

DVD placement is not free packing: a VTS IFO records where its title
VOBS begins relative to itself, so the VOB has to land exactly there.
Unsatisfiable marks fail loudly rather than misplace the file. 3D folders
are refused for now: the scanner detects SSIF and the planner cannot
alias its extents yet, so accepting them would produce quiet nonsense.

Left for later: metadata capture, HD-DVD, FMTS, encrypted folders.
2026-08-05 15:24:18 -07:00
Matthew Jackson 2310e65f44 Make iso:// a destination for any image source, not just a drive
Writing an iso:// meant a raw sector copy off a drive, so an existing
image could only ever be a source. Decrypting one you already had meant
finding the disc again.

The engine's copy is the recovery path — mapfile, multipass, damage-jump,
auto-resume — and all of it exists because optical media returns read
errors. A file does not, so a non-drive source gets write_image instead:
sectors in, bytes out, once, no recovery machinery. Keeping them apart is
not just tidiness. The mapfile identity check compares AACS unit keys and
the VID, both empty for an already-decrypted source, so identity passes
for any such source — a second run with different input to the same
output path would resume over the previous image and report success.

A short read is an error rather than a zero-fill: padding a truncated
source yields an image that looks complete and is not, which is the worst
outcome for a copy someone means to keep.
2026-08-05 14:45:01 -07:00
AnimeFN 4bd1c9210c Fix NTSC chapter/duration drift: dvd_time_t is a timecode, not seconds
`dvd_time_t` stores H:M:S:F as non-drop-frame timecode with a rate flag,
not elapsed wall-clock time. On NTSC the seconds field advances every 30
frames, but a frame lasts 1001/30000 s, so 30 frames occupy 1.001 s of
real time. Reading H:M:S as literal seconds under-reports real time by
exactly 0.1% — 3.6 s per hour, growing with elapsed time, which is what
made chapter marks drift ~4 s by the 67-minute mark.

Convert the whole timecode through an integer frame count and apply the
exact 1001/30000 fraction once, instead of reading H:M:S literally and
dividing only the frame remainder by a decimal 29.97. Chapter marks sum
per-cell frame counts as integers and convert once per chapter, so every
mark lands on an exact frame boundary instead of accumulating f64
rounding across a long title.

PAL is arithmetically unaffected: 25 frames = 1.000 s exactly, so the
old and new paths agree to within one ULP. Sweeping all 900k H:M:S:F
combinations under 10 hours, 12 differ, by at most 8.9e-16 s — the new
path divides one exact integer instead of adding a rounded fraction to a
large one, so where they differ it is the more accurate of the two.

Five existing NTSC fixtures asserted the old literal-seconds values and
were updated to real seconds (each is exactly 1.001x its old figure);
`bcd_secs` now documents that its argument is timecode, not real time.

Fixes #1
Reported-by: AnimeFN <admin@animefn.com>
2026-08-05 13:48:11 -07:00
Matthew Jackson bfce73e73c changelog: write it for users, not for the people who fixed it
Entries had grown into investigative narratives — root causes, disc
counts, PIDs, percentages, function names — which is the right level for
a post-mortem and the wrong one for a public changelog. Each entry is now
one to three sentences leading with what a user observes.

Halves the file (1538 -> 735 lines) without dropping a version. API
signature changes stay explicit, since a consumer migrates against those.
2026-08-03 08:31:04 -07:00
Matthew Jackson ca4e6f5a13 restore freemkv-unlock path dep for local dev (post-v1.6.0) 2026-08-03 05:57:31 -07:00
69 changed files with 17842 additions and 2832 deletions
+63
View File
@@ -0,0 +1,63 @@
version: 2
# Dependency updates land on `dev`, never on `main`.
#
# `main` here is a RELEASE POINTER that release.sh moves to each tag. A bot
# commit on it would put work there that no tag contains, which is exactly the
# state that aborted the 1.6.2 cascade at the last step -- so pointing
# Dependabot at main would recreate that failure on a schedule.
updates:
- package-ecosystem: cargo
directory: /
target-branch: dev
schedule:
interval: weekly
open-pull-requests-limit: 5
# One PR per week for the routine bumps instead of one per crate. Eight
# repos times a handful of crates is a volume nobody reads, and an
# unread PR queue is indistinguishable from no updates at all.
groups:
minor-and-patch:
update-types:
- minor
- patch
ignore:
# The freemkv crates depend on each other by GIT TAG, re-pinned by
# release.sh as part of the release commit. Dependabot cannot see that
# cascade, so a PR bumping one of these would fight the release process
# and could pin a version whose tag does not exist yet.
- dependency-name: freemkv-unlock
- dependency-name: libfreemkv
- dependency-name: freemkv-keysources
- dependency-name: freemkv-i18n
- dependency-name: freemkv-engine
# The workflows are now real infrastructure -- the release cascade, the
# cross-platform hash matrix, the disc gate -- so their actions need the same
# attention as the crates.
- package-ecosystem: github-actions
directory: /
target-branch: dev
schedule:
interval: weekly
open-pull-requests-limit: 5
groups:
actions:
update-types:
- minor
- patch
ignore:
# NOT a dependency: `dtolnay/rust-toolchain` is versioned by the RUST
# release it installs, and the tag we pin is the toolchain CI is pinned
# to on purpose -- precommit.sh runs the same one locally so a lint that
# passes on a developer's newer default cannot pass CI by accident.
#
# Dependabot reads those tags as semver and proposed 1.97.0 -> 1.100.0,
# a Rust version that does not exist. Every such PR 404s on toolchain
# download across all eight repos, and they regenerate weekly -- eight
# permanently-red PRs that promote.yml then has to special-case when it
# decides whether dev is green.
#
# Bumping the toolchain is a deliberate, all-eight-repos change, made by
# hand together with precommit.sh. There is nothing here for a bot.
- dependency-name: dtolnay/rust-toolchain
+94 -36
View File
@@ -2,17 +2,45 @@ name: CI
on:
push:
# `dev` is where work lands and where CI must be green. `main` only ever
# moves at release time, to the tagged commit, so a push to it is the
# release validation run rather than day-to-day feedback.
branches: [main, dev]
# dev -> qa -> main. `dev` is where work lands and is meant to be pushed
# to often: these are the FAST checks, so a mistake surfaces in minutes.
# `qa` is the release candidate — it runs these too, plus the expensive
# suite in qa.yml. `main` only ever moves at release time, to a tagged
# commit that was already green on qa.
branches: [main, dev, qa]
pull_request:
jobs:
lint:
# CI owns the changelog: main may never carry a released version without a
# CHANGELOG.md entry for it. The release orchestrator dates an existing
# `## [X.Y.Z]` heading — it does NOT invent one — so a version bump that
# forgot its changelog entry must fail here rather than ship a blank release.
# Hard-fails on main (and PRs targeting main); warns elsewhere so an
# in-progress dev/qa branch whose entry isn't written yet is not blocked.
changelog:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Require a CHANGELOG entry for the crate version on main
shell: bash
run: |
set -euo pipefail
version=$(grep -m1 '^version = ' Cargo.toml | sed -E 's/.*"([^"]+)".*/\1/')
if grep -qE "^## \[${version}\]" CHANGELOG.md; then
echo "CHANGELOG.md has an entry for ${version}."
exit 0
fi
msg="CHANGELOG.md has no '## [${version}]' entry"
if [ "${GITHUB_REF_NAME:-}" = "main" ] || [ "${GITHUB_BASE_REF:-}" = "main" ]; then
echo "::error::${msg} — main requires a changelog entry for the released version."
exit 1
fi
echo "::warning::${msg} (allowed on ${GITHUB_REF_NAME:-unknown}; required before it can reach main)."
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
path: libfreemkv
# libfreemkv path-deps ../freemkv-unlock on the BRANCH tip (release.sh
@@ -24,10 +52,10 @@ jobs:
# Both repos go into subdirectories because actions/checkout refuses a
# `path:` outside $GITHUB_WORKSPACE, and `../freemkv-unlock` is outside.
# With this layout the path dep resolves exactly as it does locally.
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
repository: freemkv/freemkv-unlock
ref: dev
ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}"
path: freemkv-unlock
- uses: dtolnay/rust-toolchain@1.97.0
with:
@@ -50,7 +78,7 @@ jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
path: libfreemkv
# libfreemkv path-deps ../freemkv-unlock on the BRANCH tip (release.sh
@@ -62,10 +90,10 @@ jobs:
# Both repos go into subdirectories because actions/checkout refuses a
# `path:` outside $GITHUB_WORKSPACE, and `../freemkv-unlock` is outside.
# With this layout the path dep resolves exactly as it does locally.
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
repository: freemkv/freemkv-unlock
ref: dev
ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}"
path: freemkv-unlock
- uses: dtolnay/rust-toolchain@1.97.0
- uses: Swatinem/rust-cache@v2
@@ -75,9 +103,22 @@ jobs:
working-directory: libfreemkv
check-macos:
# dev is the FAST lane: this job still runs, but on the release-candidate
# branches rather than on every push to dev. Nothing is deleted and no
# platform stops being checked before a release -- qa.yml independently
# covers macOS and Windows, and the jobs unique to this file (the Intel
# macOS build, the Windows release build) run here on qa and main. A push
# to dev is meant to be cheap and frequent; waiting on three runner pools
# to agree is what a release candidate is for.
#
# `if` SKIPS the job (it does not queue). A queued job would be far worse
# than a slow one: release.sh's CI gate refuses while any run for the
# commit is still in progress, so a never-scheduled job blocks releases
# silently -- see the note on real-media in qa.yml.
if: github.ref_name == 'qa' || github.ref_name == 'main'
runs-on: macos-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
path: libfreemkv
# libfreemkv path-deps ../freemkv-unlock on the BRANCH tip (release.sh
@@ -89,10 +130,10 @@ jobs:
# Both repos go into subdirectories because actions/checkout refuses a
# `path:` outside $GITHUB_WORKSPACE, and `../freemkv-unlock` is outside.
# With this layout the path dep resolves exactly as it does locally.
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
repository: freemkv/freemkv-unlock
ref: dev
ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}"
path: freemkv-unlock
- uses: dtolnay/rust-toolchain@1.97.0
- uses: Swatinem/rust-cache@v2
@@ -102,9 +143,22 @@ jobs:
working-directory: libfreemkv
check-windows:
# dev is the FAST lane: this job still runs, but on the release-candidate
# branches rather than on every push to dev. Nothing is deleted and no
# platform stops being checked before a release -- qa.yml independently
# covers macOS and Windows, and the jobs unique to this file (the Intel
# macOS build, the Windows release build) run here on qa and main. A push
# to dev is meant to be cheap and frequent; waiting on three runner pools
# to agree is what a release candidate is for.
#
# `if` SKIPS the job (it does not queue). A queued job would be far worse
# than a slow one: release.sh's CI gate refuses while any run for the
# commit is still in progress, so a never-scheduled job blocks releases
# silently -- see the note on real-media in qa.yml.
if: github.ref_name == 'qa' || github.ref_name == 'main'
runs-on: windows-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
path: libfreemkv
# libfreemkv path-deps ../freemkv-unlock on the BRANCH tip (release.sh
@@ -116,10 +170,10 @@ jobs:
# Both repos go into subdirectories because actions/checkout refuses a
# `path:` outside $GITHUB_WORKSPACE, and `../freemkv-unlock` is outside.
# With this layout the path dep resolves exactly as it does locally.
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
repository: freemkv/freemkv-unlock
ref: dev
ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}"
path: freemkv-unlock
- uses: dtolnay/rust-toolchain@1.97.0
- uses: Swatinem/rust-cache@v2
@@ -147,22 +201,22 @@ jobs:
consumers:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with: { path: libfreemkv }
- uses: actions/checkout@v5
with: { repository: freemkv/freemkv-unlock, ref: dev, path: freemkv-unlock }
- uses: actions/checkout@v5
with: { repository: freemkv/freemkv-keysources, ref: dev, path: freemkv-keysources }
- uses: actions/checkout@v5
with: { repository: freemkv/freemkv-engine, ref: dev, path: freemkv-engine }
- uses: actions/checkout@v5
with: { repository: freemkv/freemkv-i18n, ref: dev, path: freemkv-i18n }
- uses: actions/checkout@v5
with: { repository: freemkv/freemkv, ref: dev, path: freemkv }
- uses: actions/checkout@v5
with: { repository: freemkv/autorip, ref: dev, path: autorip }
- uses: actions/checkout@v5
with: { repository: freemkv/bdemu, ref: dev, path: bdemu }
- uses: actions/checkout@v7
with: { repository: freemkv/freemkv-unlock, ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}", path: freemkv-unlock }
- uses: actions/checkout@v7
with: { repository: freemkv/freemkv-keysources, ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}", path: freemkv-keysources }
- uses: actions/checkout@v7
with: { repository: freemkv/freemkv-engine, ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}", path: freemkv-engine }
- uses: actions/checkout@v7
with: { repository: freemkv/freemkv-i18n, ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}", path: freemkv-i18n }
- uses: actions/checkout@v7
with: { repository: freemkv/freemkv, ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}", path: freemkv }
- uses: actions/checkout@v7
with: { repository: freemkv/autorip, ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}", path: autorip }
- uses: actions/checkout@v7
with: { repository: freemkv/bdemu, ref: "${{ github.ref_name == 'qa' && 'qa' || 'dev' }}", path: bdemu }
- name: Point every dependent at THIS libfreemkv commit
shell: bash
run: |
@@ -185,13 +239,17 @@ jobs:
freemkv
autorip
bdemu
- run: cargo check --all-targets
# `cargo check` alone only proves the dependents still COMPILE against
# this commit. It cannot see a behavioural change — the library keeps its
# signatures and a dependent's tests start failing. That is the shape of
# every defect worth catching here, so run their suites too.
- run: cargo test --tests
working-directory: freemkv-keysources
- run: cargo check --all-targets
- run: cargo test --tests
working-directory: freemkv-engine
- run: cargo check --all-targets
- run: cargo test --tests
working-directory: freemkv
- run: cargo check --all-targets
- run: cargo test --tests
working-directory: autorip
- run: cargo check --all-targets
- run: cargo test --tests
working-directory: bdemu
+1 -1
View File
@@ -11,7 +11,7 @@ jobs:
leak-guard:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Compute commit range
+133
View File
@@ -0,0 +1,133 @@
name: qa
# ── The qa gate: "is this production worth?" ────────────────────────────────
#
# dev -> qa -> main.
#
# `dev` is for committing often. ci.yml answers "is it green" in minutes with
# fmt, clippy and the unit suite, so a mistake surfaces while it is still cheap
# to fix. `qa` is the release-candidate branch, and THIS workflow is the claim
# that a commit is production worth: everything expensive that can run without
# physical media. `main` only ever receives a qa that went green here.
#
# Sibling repos are checked out at `qa`, NOT `dev`. A qa run that resolved its
# dependencies from dev tips would be validating a combination that is not the
# one being released, which is the exact failure this branch exists to prevent.
#
# What this gate CANNOT cover: `disc://` and real `iso://` need physical media,
# and no hosted runner has an optical drive or the image hoard. Those run on a
# self-hosted runner (see the media job at the end) and are the one leg that
# stays on hardware.
on:
push:
branches: [qa]
workflow_dispatch:
jobs:
# ── Name the candidate ────────────────────────────────────────────────
#
# Every push to `qa` is a release candidate, so every push gets a tag:
# v<version>-rc<N>, N incrementing. That is the answer to "which build is on
# qa right now, and is it the one I tested?" — a question that otherwise gets
# answered from memory.
#
# This runs FIRST and does not depend on the gates, deliberately. A red
# candidate needs a name more than a green one does: "rc3 failed
# release-tests on windows" is a sentence you can act on; "qa is red" is not.
# Red on qa is a working gate, not an incident — it is the branch saying this
# is not production worth yet. Fix on dev, get dev green, push qa again.
#
# release.yml excludes v*-rc* so a candidate never publishes a release.
rc-tag:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Stamp the next rc
shell: bash
run: |
set -euo pipefail
v=$(sed -n 's/^version = "\(.*\)"/\1/p' Cargo.toml | head -1)
[ -n "$v" ] || { echo "no version in Cargo.toml" >&2; exit 1; }
# Numeric sort on the rc ordinal: -rc10 must beat -rc9, and a plain
# lexical sort gets that backwards from the tenth candidate on.
n=$(git tag -l "v$v-rc*" | sed "s|^v$v-rc||" | sort -n | tail -1)
tag="v$v-rc$(( ${n:-0} + 1 ))"
git tag "$tag"
git push origin "$tag"
echo "### Candidate \`$tag\`" >> "$GITHUB_STEP_SUMMARY"
# The debug suite runs on every dev push. Release is a DIFFERENT build:
# overflow checks are off, debug_assert! is compiled out, and inlining
# changes what the optimiser can prove. A test that only passes in debug is
# a test that never guarded the binary anyone actually ships.
release-tests:
strategy:
fail-fast: false
matrix:
os: ['ubuntu-latest', 'macos-latest']
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v7
with:
path: libfreemkv
- uses: actions/checkout@v7
with:
repository: freemkv/freemkv-unlock
ref: qa
path: freemkv-unlock
- uses: dtolnay/rust-toolchain@1.97.0
- uses: Swatinem/rust-cache@v2
with:
workspaces: libfreemkv
- run: cargo test --release --tests
working-directory: libfreemkv
# clippy's output is target-dependent: cfg-gated code only gets linted on
# the target it compiles for. Linting solely on the dev machine's host
# target is how a lint that CI rejects reaches a push.
cross-lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
path: libfreemkv
- uses: actions/checkout@v7
with:
repository: freemkv/freemkv-unlock
ref: qa
path: freemkv-unlock
- uses: dtolnay/rust-toolchain@1.97.0
with:
components: clippy
- uses: Swatinem/rust-cache@v2
with:
workspaces: libfreemkv
- run: rustup target add x86_64-unknown-linux-gnu
- run: cargo clippy --all-targets --target x86_64-unknown-linux-gnu -- -D warnings
working-directory: libfreemkv
# Windows compiles the tests but does not run them, matching the policy
# ci.yml already set. The value here is codegen: the #[cfg(windows)] halves
# of the SCSI transport and platform layers compile on no other runner, so
# without this they are first built at release time.
windows-build:
runs-on: windows-latest
steps:
- uses: actions/checkout@v7
with:
path: libfreemkv
- uses: actions/checkout@v7
with:
repository: freemkv/freemkv-unlock
ref: qa
path: freemkv-unlock
- uses: dtolnay/rust-toolchain@1.97.0
- uses: Swatinem/rust-cache@v2
with:
workspaces: libfreemkv
- run: cargo build --release --tests
working-directory: libfreemkv
+9 -4
View File
@@ -4,6 +4,11 @@ on:
push:
tags:
- 'v*'
# NOT the release-candidate tags. Every push to `qa` stamps a
# v<version>-rc<N> so a run can be named, and 'v*' matches those too —
# which would have this workflow build and PUBLISH a GitHub release for
# every candidate, including the red ones.
- '!v*-rc*'
permissions:
contents: write
@@ -12,7 +17,7 @@ jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- name: Verify Cargo.toml version matches tag
run: |
CARGO_VER="v$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')"
@@ -32,7 +37,7 @@ jobs:
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@1.97.0
- uses: Swatinem/rust-cache@v2
# libfreemkv is a library — Cargo.lock isn't tracked, so --locked
@@ -51,8 +56,8 @@ jobs:
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
uses: softprops/action-gh-release@v3
with:
generate_release_notes: true
-36
View File
@@ -1,36 +0,0 @@
name: Update README version
on:
release:
types: [published]
permissions:
contents: write
jobs:
update:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
ref: main
token: ${{ secrets.ORG_DISPATCH_TOKEN }}
- name: Update version in README
run: |
VERSION="${{ github.event.release.tag_name }}"
FILE="README.md"
# Update cargo dependency version (e.g. "0.2" -> "0.3")
MAJOR_MINOR="${VERSION#v}"
MAJOR_MINOR="${MAJOR_MINOR%.*}"
sed -i "s|libfreemkv = \"[0-9]*\.[0-9]*\"|libfreemkv = \"${MAJOR_MINOR}\"|" "$FILE"
- name: Commit and push
run: |
VERSION="${{ github.event.release.tag_name }}"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add README.md
git diff --cached --quiet || git commit -m "Update to libfreemkv ${VERSION}"
git push
+743 -1296
View File
File diff suppressed because it is too large Load Diff
+6 -12
View File
@@ -1,6 +1,6 @@
[package]
name = "libfreemkv"
version = "1.6.0"
version = "1.6.8"
edition = "2024"
rust-version = "1.97"
license = "MIT"
@@ -22,20 +22,14 @@ codegen-units = 1
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sha1 = "0.10"
sha2 = "0.10"
aes = "0.8"
cbc = "0.1"
aes = "0.9"
# Interim path dep for local cross-repo dev; the release script re-pins this to
# `{ git = ".../freemkv-unlock", tag = "vX.Y.Z" }` before tagging libfreemkv (so
# the released tag resolves freemkv-unlock from git, not a sibling path).
freemkv-unlock = { git = "https://github.com/freemkv/freemkv-unlock", tag = "v1.6.0" }
num-bigint = "0.4"
num-traits = "0.2"
num-integer = "0.1"
rand = "0.8"
cmac = "0.7"
zip = { version = "2", default-features = false, features = ["deflate"] }
base64 = "0.22.1"
freemkv-unlock = { git = "https://github.com/freemkv/freemkv-unlock", tag = "v1.6.8" }
rand = "0.10"
zip = { version = "8", default-features = false, features = ["deflate"] }
base64 = "0.23"
# Read-only XML DOM parser (pure Rust, forbid(unsafe_code), entity-expansion
# bounded). Parses the HD-DVD Advanced-Content playlist `ADV_OBJ/VPLST000.XPL`
# — untrusted disc bytes — into authoritative titles/clips/chapters. A real
+22
View File
@@ -0,0 +1,22 @@
# Security Policy
## Supported versions
| Version | Supported |
| ------- | --------- |
| 1.6.x | Yes |
| < 1.6 | No |
Only the current 1.6.x line receives security fixes.
## Reporting a vulnerability
Report vulnerabilities privately through GitHub Security Advisories:
https://github.com/freemkv/libfreemkv/security/advisories/new
Do not open a public issue for a security report. Include the affected
version, steps to reproduce, and the impact you believe the issue has.
## Response time
You will get an initial response within 7 days.
-83
View File
@@ -1,83 +0,0 @@
// Minimal ISO dumper — find exact stall point
use libfreemkv::Drive;
use std::io::{BufWriter, Write};
use std::path::Path;
use std::time::Instant;
fn main() {
let args: Vec<String> = std::env::args().collect();
if args.len() < 3 {
eprintln!("Usage: iso_dump <device> <output>");
std::process::exit(1);
}
let mut drive = Drive::open(Path::new(&args[1])).unwrap();
drive.wait_ready().unwrap();
let _ = drive.init();
let _ = drive.probe_disc();
// AACS handshake — required to read past the protected area
eprint!("Scanning disc... ");
let _ = libfreemkv::Disc::scan(&mut drive, &libfreemkv::ScanOptions::default());
eprintln!("OK");
let cap = drive.read_capacity().unwrap();
let batch = libfreemkv::disc::detect_max_batch_sectors(drive.device_path());
eprintln!("Device: {} | {} sectors | batch {}", args[1], cap, batch);
let file = std::fs::File::create(&args[2]).unwrap();
let mut w = BufWriter::with_capacity(4 * 1024 * 1024, file);
let mut buf = vec![0u8; batch as usize * 2048];
let mut lba: u32 = 0;
let start = Instant::now();
let mut last = Instant::now();
let mut bytes: u64 = 0;
let mut last_bytes: u64 = 0;
while lba < cap {
let count = ((cap - lba) as u16).min(batch);
let n = count as usize * 2048;
// Tiny yield between reads — test if pacing prevents firmware throttle
std::thread::yield_now();
let t0 = Instant::now();
let ok = drive.read(lba, count, &mut buf[..n], true).is_ok();
let read_ms = t0.elapsed().as_millis();
// Flag slow reads
if read_ms > 2000 {
eprintln!("\n SLOW READ: LBA {} took {}ms (ok={})", lba, read_ms, ok);
}
if !ok {
buf[..n].fill(0);
}
w.write_all(&buf[..n]).unwrap();
lba += count as u32;
bytes += n as u64;
if last.elapsed().as_millis() >= 1000 {
let delta = bytes - last_bytes;
let speed = delta as f64 / last.elapsed().as_secs_f64() / 1_048_576.0;
let avg = bytes as f64 / start.elapsed().as_secs_f64() / 1_048_576.0;
let pct = bytes as f64 / (cap as f64 * 2048.0) * 100.0;
eprint!(
"\r {:.1}% LBA {} | {:.0} MB/s (avg {:.0}) | {:.1} GB ",
pct,
lba,
speed,
avg,
bytes as f64 / 1e9
);
last_bytes = bytes;
last = Instant::now();
}
}
w.flush().unwrap();
eprintln!(
"\nDone: {:.1} GB in {:.0}s",
bytes as f64 / 1e9,
start.elapsed().as_secs_f64()
);
}
+10 -6
View File
@@ -4,7 +4,7 @@
#[cfg(test)]
use aes::Aes128;
#[cfg(test)]
use aes::cipher::{KeyInit, generic_array::GenericArray};
use aes::cipher::{Array, KeyInit};
use super::crypto::{aes_cbc_decrypt, aes_cbc_encrypt, aes_ecb_encrypt};
// Only this module's test fixtures build CBC ciphertext by hand now — the
@@ -418,7 +418,7 @@ pub(crate) fn decrypt_bus(unit: &mut [u8], read_data_key: &[u8; 16]) {
mod tests {
use super::super::crypto::aes_ecb_decrypt;
use super::*;
use aes::cipher::BlockEncrypt; // test fixtures build ciphertext directly
use aes::cipher::BlockCipherEncrypt; // test fixtures build ciphertext directly
/// [`encrypt_unit`] is the exact inverse of [`decrypt_unit`]: whatever an
/// authoring caller encrypts, the read path must recover byte-for-byte.
@@ -725,7 +725,7 @@ mod tests {
}
// CBC encrypt bytes 16..6143
let cipher = Aes128::new(GenericArray::from_slice(&encrypt_key));
let cipher = Aes128::new(&encrypt_key.into());
let mut prev = AACS_IV;
let num_blocks = (ALIGNED_UNIT_LEN - 16) / 16;
for i in 0..num_blocks {
@@ -733,7 +733,9 @@ mod tests {
for j in 0..16 {
plain[off + j] ^= prev[j];
}
let mut block = GenericArray::clone_from_slice(&plain[off..off + 16]);
let mut chunk = [0u8; 16];
chunk.copy_from_slice(&plain[off..off + 16]);
let mut block: Array<u8, _> = chunk.into();
cipher.encrypt_block(&mut block);
plain[off..off + 16].copy_from_slice(&block);
prev.copy_from_slice(&plain[off..off + 16]);
@@ -1496,7 +1498,7 @@ mod tests {
let plain = unit.clone();
// Forward: CBC-encrypt unit[s+16 .. s+2048] per sector under AACS IV.
let cipher = Aes128::new(GenericArray::from_slice(&rdk));
let cipher = Aes128::new(&rdk.into());
for s in (0..ALIGNED_UNIT_LEN).step_by(SECTOR_BYTES) {
let mut prev = AACS_IV;
let body = s + 16;
@@ -1507,7 +1509,9 @@ mod tests {
for j in 0..16 {
unit[off + j] ^= prev[j];
}
let mut blk = GenericArray::clone_from_slice(&unit[off..off + 16]);
let mut chunk = [0u8; 16];
chunk.copy_from_slice(&unit[off..off + 16]);
let mut blk: Array<u8, _> = chunk.into();
cipher.encrypt_block(&mut blk);
unit[off..off + 16].copy_from_slice(&blk);
prev.copy_from_slice(&unit[off..off + 16]);
+10 -8
View File
@@ -8,7 +8,7 @@
//! content / keys / variant modules.
use aes::Aes128;
use aes::cipher::{BlockDecrypt, BlockEncrypt, KeyInit, generic_array::GenericArray};
use aes::cipher::{Array, BlockCipherDecrypt, BlockCipherEncrypt, KeyInit};
/// Fixed IV used by AACS for all AES-CBC operations. [C] §2.1.2 (default CBC IV, `iv0`).
pub(crate) const AACS_IV: [u8; 16] = [
@@ -39,13 +39,13 @@ pub(crate) fn new_cipher_for(key: &[u8; 16]) -> Aes128 {
fn new_cipher(key: &[u8; 16]) -> Aes128 {
#[cfg(test)]
KEY_EXPANSIONS.with(|c| c.set(c.get() + 1));
Aes128::new(GenericArray::from_slice(key))
Aes128::new(&(*key).into())
}
/// AES-128-ECB encrypt a single 16-byte block. [C] §2.1.1 (`AES-128E`).
pub(crate) fn aes_ecb_encrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut block = GenericArray::clone_from_slice(data);
let cipher = Aes128::new(&(*key).into());
let mut block: Array<u8, _> = (*data).into();
cipher.encrypt_block(&mut block);
let mut out = [0u8; 16];
out.copy_from_slice(&block);
@@ -54,8 +54,8 @@ pub(crate) fn aes_ecb_encrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
/// AES-128-ECB decrypt a single 16-byte block. [C] §2.1.1 (`AES-128D`).
pub(crate) fn aes_ecb_decrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut block = GenericArray::clone_from_slice(data);
let cipher = Aes128::new(&(*key).into());
let mut block: Array<u8, _> = (*data).into();
cipher.decrypt_block(&mut block);
let mut out = [0u8; 16];
out.copy_from_slice(&block);
@@ -88,7 +88,7 @@ pub(crate) fn aes_cbc_encrypt(key: &[u8; 16], data: &mut [u8]) {
for j in 0..16 {
block[j] = data[offset + j] ^ prev[j];
}
let mut ga = GenericArray::clone_from_slice(&block);
let mut ga: Array<u8, _> = block.into();
cipher.encrypt_block(&mut ga);
data[offset..offset + 16].copy_from_slice(&ga);
prev.copy_from_slice(&ga);
@@ -140,7 +140,9 @@ pub(crate) fn cbc_decrypt_blocks(cipher: &Aes128, data: &mut [u8]) {
p.copy_from_slice(&data[(i - 1) * 16..i * 16]);
p
};
let mut block = GenericArray::clone_from_slice(&data[offset..offset + 16]);
let mut chunk = [0u8; 16];
chunk.copy_from_slice(&data[offset..offset + 16]);
let mut block: Array<u8, _> = chunk.into();
cipher.decrypt_block(&mut block);
for j in 0..16 {
data[offset + j] = block[j] ^ prev[j];
+4
View File
@@ -114,6 +114,10 @@ pub mod coding_type {
pub mod pes_stream_id {
/// Video stream (`110x xxxx`; freemkv emits the base id `0xE0`).
pub const VIDEO: u8 = 0xE0;
/// system_header start code — the MPEG-PS `00 00 01 BB` structural header
/// (rate/bound bounds), never an elementary stream. On a DVD NAV pack it
/// follows the pack header, so it lands at sector offset 0x11.
pub const SYSTEM_HEADER: u8 = 0xBB;
/// private_stream_1 — AC-3 / DTS / LPCM / PGS subtitle payloads.
pub const PRIVATE_STREAM_1: u8 = 0xBD;
/// padding_stream — stuffing bytes only, no payload to demux.
+516 -78
View File
@@ -75,7 +75,7 @@ pub fn crack_key(
extents: &[Extent],
batch_sectors: u16,
) -> Option<CssState> {
crack_key_scan(reader, extents, batch_sectors, None, false).into_state()
crack_key_scan(reader, extents, batch_sectors, None).into_state()
}
/// Outcome of a CSS crack scan that distinguishes the THREE cases the bare
@@ -85,7 +85,7 @@ pub fn crack_key(
///
/// - [`CrackOutcome::Cracked`] — a scrambled sector yielded a title key.
/// - [`CrackOutcome::Unencrypted`] — NO scrambled sector was seen across the
/// scanned extents (`is_scrambled` never true): the content is genuinely
/// scanned extents (`is_scrambled_pack` never true): the content is genuinely
/// plaintext, so proceeding without a key is correct.
/// - [`CrackOutcome::ScrambledUncracked`] — scrambled sectors WERE seen but no
/// key could be recovered (the Stevenson attack found no crackable crib, or
@@ -136,7 +136,7 @@ pub fn crack_key_outcome(
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
) -> CrackOutcome {
crack_key_scan(reader, extents, batch_sectors, halt, true)
crack_key_scan(reader, extents, batch_sectors, halt)
}
/// Resolve a DVD title's CSS descramble key from the reader when the caller
@@ -212,10 +212,6 @@ fn crack_key_scan(
extents: &[Extent],
batch_sectors: u16,
halt: Option<&crate::halt::Halt>,
// True only on the INITIAL scan: a fully CSS-locked (`05/6F/03`) result is a
// hard `ScrambledUncracked`. False on the per-VTS re-crack so a lapsed-AGID
// locked read returns None instead of killing a genuinely crackable title.
fail_on_locked: bool,
) -> CrackOutcome {
// Batch the reads: a live optical drive at 1 sector/read is glacial, and the
// crack only needs to FIND one scrambled sector whose 0x80 plaintext matches
@@ -277,11 +273,43 @@ fn crack_key_scan(
}
let n = (ext.sector_count - i).min(batch);
let want = n as usize * 2048;
// How far the cursor advances. Set from the bytes actually READ on
// the Ok path so a short read is RETRIED from where it stopped
// rather than skipped: bounding only the inspection (which is what
// stops stale buffer bytes being scanned) would otherwise leave
// those sectors unexamined, quietly shrinking the crack's coverage
// on exactly the damaged media where a key is hardest to find.
let mut advance = n;
match reader.read_sectors(ext.start_lba + i, n as u16, &mut buf[..want], true) {
Ok(_) => {
Ok(got) => {
// A readable batch: the gate is open — reset the locked run.
consecutive_locked = 0;
for s in 0..n as usize {
// Inspect only what was actually READ. The trait returns the
// byte count and a source may return Ok with fewer bytes
// than asked (a recovery read over a damaged region), while
// `buf` is reused across batches — so the tail still holds
// the PREVIOUS batch's sectors. Scanning those means
// cracking a key from data belonging to a different extent,
// and possibly a different VTS, while `crack_span` records
// the CURRENT one: a key that opens nothing here gets
// installed and its wrong descrambles are only partly caught
// by the per-sector crib.
let usable = (got / 2048).min(n as usize);
// At least one, so a source returning Ok(0) cannot spin here.
advance = (usable as u32).max(1);
if usable == 0 {
// Nothing was inspected, so the per-sector `tried`
// charge below never runs — but the cursor still moves
// one sector (the `.max(1)` above). Charge that sector
// to the budget, or `tried` stays frozen and the loop
// is bounded only by the disc-declared
// `ext.sector_count`: the anti-grind budget stops
// applying to exactly the misbehaving source it exists
// for. Mirrors the `Err` arm's `tried += n`, which
// likewise charges an uninspected advance.
tried += 1;
}
for s in 0..usable {
tried += 1;
let sect = &buf[s * 2048..(s + 1) * 2048];
// Use the HARDENED pack-gated check (Fix 3): a clear stub
@@ -319,19 +347,34 @@ fn crack_key_scan(
}
}
}
i += n;
i += advance;
}
}
// Budget exhausted / extents walked / early-bailed with no key recovered.
// The disc is ENCRYPTED-but-uncracked (a hard failure on the initial scan)
// when EITHER a scrambled sector was actually seen, OR — on the initial scan
// only (`fail_on_locked`) — every read was CSS-locked (`05/6F/03`), itself
// proof of scrambling. A re-crack (`fail_on_locked` false) stays soft: a
// lapsed-AGID locked read yields None, not a hard fail, so a crackable title
// in another VTS isn't killed. Only a scan that saw neither a scrambled
// sector nor a CSS-lock is genuinely unencrypted.
if saw_scrambled || (saw_locked && fail_on_locked) {
// The disc is ENCRYPTED-but-uncracked (a hard failure) when EITHER a
// scrambled sector was actually seen, OR every read was CSS-locked
// (`05/6F/03`), itself proof of scrambling. Only a scan that saw neither a
// scrambled sector nor a CSS-lock is genuinely unencrypted.
//
// A prior revision made this conditional on a caller-supplied
// `fail_on_locked: bool`, documented as "false on the per-VTS re-crack, so
// a lapsed-AGID locked read returns None instead of killing a genuinely
// crackable title." That parameter never had an observable effect: its
// ONLY non-test caller with `false` was [`crack_key`], whose `Option`
// return collapses `ScrambledUncracked` and `Unencrypted` alike to `None`
// via [`CrackOutcome::into_state`] — so the branch this comment describes
// was unreachable from the moment it was introduced (see the crate's audit
// notes for the git-archaeology). The one production caller that DOES
// observe the `Cracked` / `Unencrypted` / `ScrambledUncracked` split for a
// per-VTS re-crack (`Disc::decrypt_keys_for_title`) has always gone
// through [`crack_key_outcome`], which hardcoded this to always-hard-fail.
// Reconnecting the soft variant there would mean a locked read on a live
// drive gets reported as `Unencrypted` (`title_is_clear = true`) and the
// title is muxed with NO key — precisely the silent-garbage failure mode
// `CrackOutcome` exists to prevent. So the parameter is removed rather
// than revived: the scan is unconditionally hard-fail-on-locked.
if saw_scrambled || saw_locked {
CrackOutcome::ScrambledUncracked
} else {
CrackOutcome::Unencrypted
@@ -339,7 +382,25 @@ fn crack_key_scan(
}
/// Descramble a single CSS-encrypted sector in place.
///
/// A no-op unless the sector is a scrambled MPEG-2 PS PACK. The pack start code
/// is checked, not just the byte 0x14 flag bits, for the same reason
/// `descramble_region` checks it: 0x14 only means "scrambling control" inside a
/// pack, and in an IFO it is whatever that format stores there. A real
/// `VIDEO_TS.IFO` sector holds 0x15 there while starting `00 26 00 00`;
/// descrambling it destroyed 1912 of its 2048 bytes, and because that sector
/// carries TT_SRPT the disc enumerated 38 titles while an image decrypted from
/// it enumerated 10, silently, at exit 0.
///
/// This function has no callers inside the crate, but the module-level example
/// above prescribes it — so the crate's own documented guidance led straight
/// into that defect. Making the guard part of the function, rather than
/// something each caller must remember, is what keeps the safe path the easy
/// one.
pub fn descramble_sector(state: &CssState, sector: &mut [u8]) {
if !is_scrambled_pack(sector) {
return;
}
lfsr::descramble_sector(&state.title_key, sector);
}
@@ -364,22 +425,33 @@ pub fn descramble_sector(state: &CssState, sector: &mut [u8]) {
///
/// # Errors
///
/// [`Error::DecryptFailed`] when a sector's own crib proves the cached key stale
/// and the re-crack from that same sector also fails. CSS has no external key
/// source — the title key comes only from cracking the data — so on a readable
/// sector this is not a missing input, it is recovery failing on data we can
/// see. Emitting the sector anyway means one of two bad outcomes: descrambled
/// with the key its crib just rejected, which yields garbage behind an intact
/// clear header (valid pack start, passes every structural check the PS demuxer
/// applies, corruption confined to the PES payload where nothing looks); or
/// passed through still scrambled, which is ciphertext delivered where plaintext
/// is meant to be. Both are bad data reported as success.
/// Never returns `Err` — the signature is `Result` only to match the decrypt
/// seam it is dispatched from, and the `usize` is that seam's legacy
/// always-zero loss count (see [`crate::decrypt::decrypt_sectors`]).
///
/// This matches the AACS sibling, which returns [`Error::DecryptFailed`] rather
/// than apply a neighbouring CPS unit's key.
/// This section used to document an [`Error::DecryptFailed`] for the case where
/// a sector's crib rejects the cached key and the re-crack from that sector also
/// fails. That behaviour was tried and REVERTED, for the reason set out at the
/// `None =>` arm below: crib mismatch plus crack failure is the signature of a
/// crib FALSE POSITIVE, not of a stale key, and failing there made real discs
/// unrippable. The arm descrambles with the cached key and returns `Ok`.
pub fn descramble_region(buf: &mut [u8], title_key: &mut [u8; 5]) -> crate::error::Result<usize> {
for chunk in buf.chunks_mut(2048) {
if chunk.len() < 2048 || !is_scrambled(chunk) {
// `is_scrambled_pack`, NOT the looser `is_scrambled`. The raw flag test
// is only safe once a caller has committed to a title's VOB data, where
// every sector is a pack and byte 0x14 always means what it says. This
// function is handed arbitrary regions of a disc, so it also sees IFO,
// UDF and ISO 9660 sectors — raw structures where byte 0x14 is whatever
// that format stores there.
//
// Measured: `VIDEO_TS.IFO` on one disc holds 0x15 at offset 0x14 of its
// second sector. The raw test read bits 4-5 as "scrambled", descrambled
// it, and destroyed 1912 of 2048 bytes. That sector carries TT_SRPT, so
// the title table went with it: the disc enumerated 38 titles and an
// image decrypted from it enumerated 10, silently, at exit 0. Requiring
// the pack start code first costs nothing — a genuinely scrambled VOB
// sector always has it, and no IFO sector does.
if chunk.len() < 2048 || !is_scrambled_pack(chunk) {
continue;
}
let crib = stevenson::attack_crib(chunk);
@@ -433,15 +505,25 @@ pub fn descramble_region(buf: &mut [u8], title_key: &mut [u8; 5]) -> crate::erro
Ok(0)
}
/// Check if a sector has the CSS scramble flag set.
/// Whether bits 4-5 of the sub-header byte 0x14 are set. NOTHING MORE.
///
/// This is the RAW flag test — bits 4-5 of the sub-header byte 0x14 — used by
/// the descramble loop (`decrypt::decrypt_sectors`), which has already committed
/// to descrambling a known title's VOB data and only needs to skip the clear
/// NAV packs interleaved in it. For the CRACK SCAN's "did this disc actually
/// contain scrambled content?" decision (which must not false-positive on a
/// clear stub), use [`is_scrambled_pack`] instead.
pub fn is_scrambled(sector: &[u8]) -> bool {
/// This is deliberately NOT called `is_scrambled`. Byte 0x14 only means "scrambling control" inside an MPEG-2 Program
/// Stream pack; in an IFO, UDF or ISO 9660 sector it is whatever that format
/// stores there. Treating the flag alone as proof of scrambling is what
/// destroyed 1912 bytes of a real disc's `VIDEO_TS.IFO` — the sector carrying
/// TT_SRPT — so the disc enumerated 38 titles and an image decrypted from it
/// enumerated 10, silently, at exit 0.
///
/// **Callers want [`is_scrambled_pack`].** It asks the same question and also
/// requires the pack start code, which every genuinely scrambled VOB sector
/// carries and no IFO sector does.
///
/// It stays public only because an integration test asserts the flag
/// extraction directly. It has no production callers, and its previous doc
/// comment claimed one (`decrypt::decrypt_sectors`) that did not exist — so
/// the name was an invitation and the documentation was an argument for
/// accepting it.
pub fn has_scramble_flag_bits(sector: &[u8]) -> bool {
sector.len() >= 2048 && (sector[0x14] >> 4) & 0x03 != 0
}
@@ -453,7 +535,7 @@ pub(crate) const PACK_START: [u8; 4] = [0x00, 0x00, 0x01, 0xBA];
/// Check if a sector is a CSS-scrambled DVD **video pack** — the HARDENED test
/// the crack scan uses to set its `saw_scrambled` evidence flag (Fix 3).
///
/// [`is_scrambled`] keys solely on bits 4-5 of byte 0x14. That single byte is
/// [`has_scramble_flag_bits`] keys solely on bits 4-5 of byte 0x14. That byte is
/// only meaningful inside a real DVD sector — an MPEG-2 Program Stream pack,
/// which ALWAYS begins with the 32-bit pack-start code `00 00 01 BA` at offset
/// 0x00. A tiny clear / nav-only stub (a 0.5 s menu loop, an FBI-warning title)
@@ -465,14 +547,53 @@ pub(crate) const PACK_START: [u8; 4] = [0x00, 0x00, 0x01, 0xBA];
/// structurally a DVD video pack can be counted as scramble evidence. This does
/// NOT weaken the genuine "encrypted but uncrackable" hard-fail: a real
/// scrambled feature is made of valid PS packs, so its scrambled sectors still
/// pass this check and still drive `ScrambledUncracked` when no key cracks. (The
/// descramble loop keeps the looser [`is_scrambled`]: by the time it runs we
/// already know the title is CSS, and it only needs to skip interleaved clear
/// NAV packs — a wrongly-skipped or wrongly-included sector there is recoverable
/// per-sector, whereas a false scramble verdict in the scan poisons the whole
/// title's outcome.)
/// pass this check and still drive `ScrambledUncracked` when no key cracks.
///
/// The pack-start check alone is not enough, and the sector's PES `stream_id`
/// (offset 0x11, the byte after the 14-byte pack header's `00 00 01` PES
/// prefix) is the second load-bearing gate. CSS scrambles ONLY elementary
/// streams — video (`0xE0..=0xEF`) and private_stream_1 audio/subpicture
/// (`0xBD`) — and it never touches the clear header, so byte 0x11 is the TRUE
/// stream_id even on a scrambled sector. The MPEG-PS structural packets that
/// are never CSS-scrambled — system_header (`0xBB`), padding (`0xBE`) and
/// private_stream_2 (`0xBF`, DVD PCI/DSI navigation) — must be excluded,
/// because on those the byte at 0x14 is NOT a PES scrambling-control field but
/// raw payload/structure whose bits 4-5 land set by chance.
///
/// This is the DETECTION defect a decrypted HD-DVD tripped: an HD-DVD `.evo` is
/// MPEG-PS exactly like a DVD `.vob`, and its RDI navigation packs are
/// private_stream_2 (`0xBF`) whose payload byte at 0x14 routinely has bits 4-5
/// set. With no `0x11` gate those nav packs flipped the crack scan's
/// `saw_scrambled` flag on a disc that carries no CSS at all; the crack then
/// found no key (there is none) and the scan returned `ScrambledUncracked`,
/// hard-failing a perfectly good HD-DVD with `CssKeyMissing` — E7023. Excluding
/// `0xBB/0xBE/0xBF` at 0x11 makes the evidence gate match what the crack itself
/// can act on (the Stevenson attack only recovers a key from a scrambled ES
/// pack), so a decrypted HD-DVD now scans to `Unencrypted` and muxes cleanly.
///
/// This does NOT weaken the genuine "encrypted but uncrackable" hard-fail on a
/// real DVD: a scrambled DVD feature is made of video (`0xE0..`) and
/// private_stream_1 (`0xBD`) packs, none of which are excluded, so its
/// scrambled sectors still set `saw_scrambled` and still drive
/// `ScrambledUncracked` when no key cracks. Byte 0x11 is in the clear header,
/// so a scrambled DVD pack can never masquerade as `0xBB/0xBE/0xBF`.
///
/// The DESCRAMBLE path gates on this same function — [`descramble_sector`] and
/// [`descramble_region`] both call it, not the raw flag test — because the raw
/// test does not merely mis-skip a sector there: it descrambles one that was
/// never scrambled and destroys it. The measured case is written up on
/// [`descramble_region`]: a `VIDEO_TS.IFO` sector holding 0x15 at offset 0x14
/// lost 1912 of its 2048 bytes, taking TT_SRPT with it, and the disc's 38
/// titles became 10 — silently, at exit 0. One gate, both paths.
pub fn is_scrambled_pack(sector: &[u8]) -> bool {
sector.len() >= 2048 && sector[0x00..0x04] == PACK_START && (sector[0x14] >> 4) & 0x03 != 0
use crate::consts::pes_stream_id::{PADDING_STREAM, PRIVATE_STREAM_2, SYSTEM_HEADER};
sector.len() >= 2048
&& sector[0x00..0x04] == PACK_START
&& !matches!(
sector[0x11],
SYSTEM_HEADER | PADDING_STREAM | PRIVATE_STREAM_2
)
&& (sector[0x14] >> 4) & 0x03 != 0
}
#[cfg(test)]
@@ -510,7 +631,10 @@ mod tests {
for (i, b) in sector.iter_mut().enumerate().skip(0x80) {
*b = ((i * 37 + 11) % 251) as u8;
}
assert!(is_scrambled(&sector), "fixture must be a scrambled sector");
assert!(
has_scramble_flag_bits(&sector),
"fixture must be a scrambled sector"
);
assert!(
stevenson::attack_crib(&sector).is_some(),
"fixture must yield a crib, or the mismatch branch is never entered"
@@ -525,7 +649,22 @@ mod tests {
let out = descramble_region(&mut sector, &mut key)
.expect("a crib false positive must NOT fail the rip");
assert_eq!(out, 0, "CSS reports no loss term of its own");
// The "no loss term" contract belongs to the SEAM, not to this
// function: `decrypt_sectors`' `usize` is a legacy always-zero count
// that the CSS arm feeds from here. Asserting `out == 0` on
// `descramble_region` alone only restates its single `Ok(0)` return —
// a body replaced by `Ok(0)` satisfies it just as well. Assert it one
// level up, where the value is actually assembled and returned, so the
// arm dispatch and the plumbing are exercised too.
assert_eq!(out, 0);
let mut seam_sector = sector;
let mut seam_keys = crate::decrypt::DecryptKeys::Css { title_key: key };
assert_eq!(
crate::decrypt::decrypt_sectors(&mut seam_sector, &mut seam_keys, 0)
.expect("a crib false positive must NOT fail the rip at the seam either"),
0,
"CSS reports no loss term of its own through decrypt_sectors"
);
assert_eq!(
key, key_before,
"a failed re-crack must leave the cached key in place — it is still \
@@ -553,9 +692,9 @@ mod tests {
);
}
// ── is_scrambled ───────────────────────────────────────────────────────
// ── has_scramble_flag_bits ─────────────────────────────────────────────
/// is_scrambled returns false for any buffer shorter than one sector,
/// has_scramble_flag_bits returns false for any buffer shorter than one sector,
/// WITHOUT indexing byte 0x14 (which would panic on a tiny buffer). The
/// length guard is short-circuited before the flag read.
///
@@ -565,13 +704,13 @@ mod tests {
/// (`(sector[0x14]...) && sector.len() >= 2048`) -> panics indexing a
/// 20-byte slice; this test catches it.
#[test]
fn is_scrambled_short_buffer_is_false_no_panic() {
assert!(!is_scrambled(&[]));
assert!(!is_scrambled(&[0u8; 20])); // shorter than 0x14+1 even
assert!(!is_scrambled(&[0xFFu8; 2047])); // one byte short of a sector
fn has_scramble_flag_bits_short_buffer_is_false_no_panic() {
assert!(!has_scramble_flag_bits(&[]));
assert!(!has_scramble_flag_bits(&[0u8; 20])); // shorter than 0x14+1 even
assert!(!has_scramble_flag_bits(&[0xFFu8; 2047])); // one byte short of a sector
}
/// is_scrambled keys on bits 4-5 of byte 0x14 (the CSS scramble field).
/// has_scramble_flag_bits keys on bits 4-5 of byte 0x14 (the CSS scramble field).
/// A full sector flagged 0x10/0x20/0x30 is scrambled; 0x00 and the
/// high-bit-only values 0x40/0x80 are clear.
///
@@ -579,7 +718,7 @@ mod tests {
/// Mutation: widen mask to `& 0x0F` -> 0x40 reports scrambled, the 0x40
/// assert fails.
#[test]
fn is_scrambled_uses_bits_4_5_only() {
fn has_scramble_flag_bits_uses_bits_4_5_only() {
let mut s = vec![0u8; 2048];
for (flag, expected) in [
(0x00u8, false),
@@ -593,24 +732,27 @@ mod tests {
] {
s[0x14] = flag;
assert_eq!(
is_scrambled(&s),
has_scramble_flag_bits(&s),
expected,
"flag byte {flag:#04x} scramble detection"
);
}
}
/// is_scrambled accepts exactly 2048 bytes as the minimum (boundary at the
/// has_scramble_flag_bits accepts exactly 2048 bytes as the minimum (boundary at the
/// inclusive value 2048).
///
/// Grounding: `sector.len() >= 2048`.
/// Mutation: change `>= 2048` to `> 2048` -> an exact 2048-byte scrambled
/// sector reports false; this fails.
#[test]
fn is_scrambled_exact_sector_length_accepted() {
fn has_scramble_flag_bits_exact_sector_length_accepted() {
let mut s = vec![0u8; 2048];
s[0x14] = 0x30;
assert!(is_scrambled(&s), "exactly 2048 bytes must be eligible");
assert!(
has_scramble_flag_bits(&s),
"exactly 2048 bytes must be eligible"
);
}
/// Fix 3 hardening: `is_scrambled_pack` (the crack-scan evidence gate)
@@ -618,8 +760,8 @@ mod tests {
/// bits. A clear / nav-only stub whose bytes happen to set bits 4-5 of 0x14
/// but lacks the pack-start is NOT counted as scramble evidence — without
/// this the scan flips `saw_scrambled` and a genuinely unencrypted title
/// reports `ScrambledUncracked` (the false E7023). The looser `is_scrambled`
/// (descramble gate) still reads the same sector as flagged.
/// reports `ScrambledUncracked` (the false E7023). The raw flag test
/// `has_scramble_flag_bits` still reads the same sector as flagged.
///
/// Grounding: `sector[0x00..0x04] == 00 00 01 BA && (sector[0x14] >> 4)...`.
/// Mutation: drop the pack-start clause -> the 0x14-only sector counts as a
@@ -633,7 +775,10 @@ mod tests {
"0x14 bits without the MPEG-PS pack-start must NOT count as a scrambled pack"
);
// The looser descramble-gate check still sees the raw flag.
assert!(is_scrambled(&s), "is_scrambled keys on the 0x14 flag alone");
assert!(
has_scramble_flag_bits(&s),
"has_scramble_flag_bits keys on the 0x14 flag alone"
);
// A near-miss pack-start (wrong final byte) is still rejected.
s[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBB]);
assert!(
@@ -648,6 +793,52 @@ mod tests {
);
}
/// Detection fix: `is_scrambled_pack` must EXCLUDE the MPEG-PS structural
/// packets that CSS never scrambles — system_header (0xBB), padding (0xBE)
/// and private_stream_2 (0xBF, DVD PCI/DSI and HD-DVD `.evo` RDI nav) — by
/// their `stream_id` at offset 0x11. On those packets byte 0x14 is raw
/// payload/structure, not a PES scrambling-control field, so its bits 4-5
/// land set by chance.
///
/// This is the exact decrypted-HD-DVD defect: an `.evo` RDI pack is
/// private_stream_2 (0xBF) with the pack-start code and bits set at 0x14, so
/// the old gate flipped `saw_scrambled` on a CSS-free disc → the crack found
/// no key → `ScrambledUncracked` → E7023 on a good HD-DVD.
///
/// Grounding: `!matches!(sector[0x11], SYSTEM_HEADER | PADDING_STREAM |
/// PRIVATE_STREAM_2)`. Mutation: drop the 0x11 exclusion → the 0xBF nav pack
/// counts as scrambled evidence and the first assert fails. The video and
/// private_stream_1 packs prove the gate does NOT reject a real scrambled
/// DVD sector (the catastrophic direction).
#[test]
fn is_scrambled_pack_excludes_nav_and_structural_stream_ids() {
use crate::consts::pes_stream_id::{
PADDING_STREAM, PRIVATE_STREAM_1, PRIVATE_STREAM_2, SYSTEM_HEADER, VIDEO, VIDEO_MAX,
};
// A pack-start pack with 0x14 scramble bits set, varying only 0x11.
let mut s = vec![0u8; 2048];
s[0x00..0x04].copy_from_slice(&PACK_START);
s[0x14] = 0x30;
for excluded in [SYSTEM_HEADER, PADDING_STREAM, PRIVATE_STREAM_2] {
s[0x11] = excluded;
assert!(
!is_scrambled_pack(&s),
"stream_id {excluded:#04x} is a structural/nav pack CSS never scrambles — \
it must NOT count as scramble evidence (else a decrypted HD-DVD RDI pack → E7023)"
);
}
// A genuinely scramblable elementary-stream pack must STILL register —
// proving the exclusion did not weaken real-DVD CSS detection.
for scramblable in [VIDEO, VIDEO_MAX, PRIVATE_STREAM_1, 0xE2] {
s[0x11] = scramblable;
assert!(
is_scrambled_pack(&s),
"stream_id {scramblable:#04x} is a scramblable ES pack — a real CSS DVD's \
scrambled sector must still be counted, never passed through as plaintext"
);
}
}
// ── crack_key scanning over a mock SectorSource ────────────────────────
/// Records every (lba, count) read; returns a caller-supplied flag byte at
@@ -664,6 +855,16 @@ mod tests {
/// instead of the uniform `flag_byte` fill. Lets the scan actually
/// reach `CrackOutcome::Cracked` from a synthetic ISO.
crackable: Option<(u32, Vec<u8>)>,
/// Sectors actually filled per batch, however many were asked for —
/// the SHORT READ a `recovery: true` source is allowed to return over
/// a damaged region. `Some(0)` is the degenerate case that must not
/// spin the scan.
short_read: Option<usize>,
/// PES `stream_id` written at offset 0x11 of each uniform-fill sector.
/// `0x00` (the default) is a scramblable-looking pack; set to
/// private_stream_2 (`0xBF`) to model an HD-DVD `.evo` RDI nav pack,
/// which carries the pack-start code and 0x14 bits but no CSS.
stream_id: u8,
}
impl MockSource {
@@ -674,6 +875,8 @@ mod tests {
fail_all: false,
lock_all: false,
crackable: None,
short_read: None,
stream_id: 0x00,
}
}
}
@@ -724,14 +927,19 @@ mod tests {
if self.fail_all {
return Err(Error::DecryptFailed);
}
let n = count as usize * 2048;
// A short read fills, and reports, fewer sectors than asked.
let filled = match self.short_read {
Some(k) => (k as u16).min(count),
None => count,
};
let n = filled as usize * 2048;
let end = n.min(buf.len());
for b in buf[..end].iter_mut() {
*b = 0;
}
// Fill each sector in the batch with the uniform flag byte, EXCEPT a
// designated crackable LBA which gets the full synthetic sector.
for s in 0..count as u32 {
for s in 0..filled as u32 {
let sect_lba = lba + s;
let base = s as usize * 2048;
if base + 2048 > end {
@@ -743,11 +951,14 @@ mod tests {
}
_ => {
// Real DVD video sectors always open with the MPEG-PS
// pack-start code; `is_scrambled` (Fix 3) requires it
// before trusting the 0x14 scramble bits, so the fixture
// must include it for a `flag_byte` of 0x30 to register
// as scrambled.
// pack-start code; `is_scrambled_pack` requires it before
// trusting the 0x14 scramble bits, so the fixture must
// include it for a `flag_byte` of 0x30 to register as
// scrambled. `stream_id` (byte 0x11) defaults to 0x00 (a
// scramblable pack); an HD-DVD RDI nav pack sets it to
// private_stream_2 (0xBF), which the 0x11 exclusion drops.
buf[base..base + 4].copy_from_slice(&PACK_START);
buf[base + 0x11] = self.stream_id;
buf[base + 0x14] = self.flag_byte;
}
}
@@ -756,6 +967,93 @@ mod tests {
}
}
// ── Short reads: the branch nothing exercised ─────────────────────────
//
// `crack_key_scan` passes `recovery = true`, which is precisely the mode
// where a `SectorSource` may return Ok with fewer bytes than asked. Every
// source in this module returned the full request, so `usable`, `advance`
// and the `.max(1)` anti-spin guard were dead code under test: reverting
// `advance` to `n`, or dropping the `.max(1)`, left the whole suite green.
/// A short batch is RE-READ from where it stopped, not skipped. Skipping
/// would quietly shrink the crack's coverage on exactly the damaged media
/// where a key is hardest to find.
#[test]
fn a_short_read_resumes_from_where_it_stopped() {
let mut src = MockSource::new(0x00);
src.short_read = Some(1);
let ext = [crate::disc::Extent {
start_lba: 100,
sector_count: 4,
}];
let _ = crack_key_scan(&mut src, &ext, 4, None);
let reads = src.reads.borrow().clone();
assert_eq!(
reads,
vec![100, 101, 102, 103],
"a source that filled one sector per batch must be asked for the \
next one, not advanced a whole batch past it"
);
}
/// A source that reads NOTHING must terminate. Without the `.max(1)` the
/// cursor never moves and `tried` never increments — the budget cannot end
/// the loop, so the scan spins forever inside a library whose whole job is
/// surviving hostile input.
#[test]
fn a_source_that_returns_zero_sectors_terminates() {
let mut src = MockSource::new(0x00);
src.short_read = Some(0);
let ext = [crate::disc::Extent {
start_lba: 0,
sector_count: 8,
}];
let outcome = crack_key_scan(&mut src, &ext, 4, None);
assert!(
matches!(outcome, CrackOutcome::Unencrypted),
"nothing was read, so nothing scrambled was seen"
);
assert!(
src.reads.borrow().len() <= 8,
"the cursor must advance even on an empty read; got {} reads over \
an 8-sector extent",
src.reads.borrow().len()
);
}
/// The 50_000-sector budget must hold whatever the source returns, not
/// only when the source delivers sectors.
///
/// `tried` is incremented ONLY per inspected sector, inside
/// `for s in 0..usable`. An `Ok(0)` inspects nothing, so that loop never
/// runs — yet `advance` is forced to 1 to stop the scan spinning, so the
/// cursor keeps walking. The budget is then never consulted and the scan
/// runs for the extent's full, disc-declared `sector_count`: a misbehaving
/// or adversarial source (an emulated drive, a bridge answering short)
/// converts the anti-grind bound into no bound at all.
///
/// Mutation: delete the `tried` charge in the `usable == 0` arm and this
/// goes red at 60_000 reads.
#[test]
fn a_source_that_returns_zero_sectors_still_obeys_the_scan_budget() {
const MAX_TRIES: usize = 50_000;
let mut src = MockSource::new(0x00);
src.short_read = Some(0);
// Deliberately LARGER than the budget: if the budget is what stops the
// scan, the extent's own length is never reached.
let ext = [crate::disc::Extent {
start_lba: 0,
sector_count: 60_000,
}];
let _ = crack_key_scan(&mut src, &ext, 4, None);
let reads = src.reads.borrow().len();
assert!(
reads <= MAX_TRIES,
"an Ok(0)-returning source must be stopped by the {MAX_TRIES}-sector \
budget, not by the disc-declared extent length; got {reads} reads"
);
}
/// crack_key caps total scanned sectors at 50_000 even when extents are
/// far larger, and counts EVERY scanned sector (clear ones included)
/// toward the budget. With one 200_000-sector extent of clear sectors, it
@@ -825,6 +1123,103 @@ mod tests {
assert!(crack_key(&mut MockSource::new(0x30), &extents, 1).is_none());
}
/// `descramble_region` is handed arbitrary disc regions, so it also sees
/// IFO/UDF/ISO-9660 sectors — raw structures where byte 0x14 is whatever
/// that format happens to store there, NOT a scrambling-control field.
///
/// Measured on a real disc: the second sector of `VIDEO_TS.IFO` holds 0x15
/// at offset 0x14 (bits 4-5 set) while starting `00 26 00 00`, which is not
/// a pack. Descrambling it destroyed 1912 of its 2048 bytes, and because
/// that sector carries TT_SRPT the whole title table went with it — the
/// disc enumerated 38 titles, an image decrypted from it enumerated 10, at
/// exit 0 with no diagnostic. Guard on the pack start code, not the flag.
/// The PUBLIC per-sector entry point must refuse a non-pack sector too.
///
/// Audit finding: `descramble_region` was fixed to require the pack start
/// code, but `descramble_sector` — which the module-level example tells
/// callers to use — still keyed on the 0x14 flag alone, so the crate's own
/// documented path reached the same defect.
#[test]
fn descramble_sector_refuses_a_non_pack_sector() {
let mut ifo_like = vec![0u8; 2048];
for (i, b) in ifo_like.iter_mut().enumerate() {
*b = (i as u8).wrapping_mul(37).wrapping_add(11);
}
ifo_like[0x00..0x04].copy_from_slice(&[0x00, 0x26, 0x00, 0x00]); // not a pack
ifo_like[0x14] = 0x15; // flag bits set — the trap
assert!(has_scramble_flag_bits(&ifo_like) && !is_scrambled_pack(&ifo_like));
let pristine = ifo_like.clone();
let state = CssState {
title_key: [0x42, 0x13, 0x37, 0xBE, 0xEF],
crack_span: None,
};
descramble_sector(&state, &mut ifo_like);
assert_eq!(
ifo_like, pristine,
"a non-pack sector must survive byte-identical through the public API"
);
}
#[test]
fn descramble_region_leaves_a_non_pack_sector_alone_even_with_the_flag_set() {
let mut ifo_like = vec![0u8; 2048];
for (i, b) in ifo_like.iter_mut().enumerate() {
*b = (i as u8).wrapping_mul(37).wrapping_add(11);
}
ifo_like[0x00..0x04].copy_from_slice(&[0x00, 0x26, 0x00, 0x00]); // not a pack
ifo_like[0x14] = 0x15; // bits 4-5 set: reads as "scrambled" to the raw test
assert!(
has_scramble_flag_bits(&ifo_like) && !is_scrambled_pack(&ifo_like),
"fixture must be exactly the case the two predicates disagree on"
);
let pristine = ifo_like.clone();
let mut key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
descramble_region(&mut ifo_like, &mut key).expect("region descramble");
assert_eq!(
ifo_like, pristine,
"a non-pack sector must survive byte-identical — descrambling it \
corrupts the very structures that enumerate titles"
);
}
/// The other half of the contract: the guard must not cost coverage of
/// sectors that genuinely ARE scrambled VOB data.
#[test]
fn descramble_region_still_descrambles_a_real_scrambled_pack() {
let mut pack = vec![0u8; 2048];
for (i, b) in pack.iter_mut().enumerate() {
*b = (i as u8).wrapping_mul(29).wrapping_add(3);
}
pack[0x00..0x04].copy_from_slice(&PACK_START);
pack[0x14] = 0x30;
assert!(is_scrambled_pack(&pack));
let scrambled = pack.clone();
let mut key = [0x42, 0x13, 0x37, 0xBE, 0xEF];
descramble_region(&mut pack, &mut key).expect("region descramble");
assert_ne!(
pack[0x80..],
scrambled[0x80..],
"the encrypted region of a real pack must actually be transformed"
);
assert_eq!(
(pack[0x14] >> 4) & 0x03,
0,
"the scrambling-control bits must be cleared once descrambled"
);
assert_eq!(
pack[0x15..0x80],
scrambled[0x15..0x80],
"CSS only scrambles from 0x80 on; the rest of the header, byte 0x14 \
aside, must be untouched"
);
assert_eq!(pack[..0x14], scrambled[..0x14]);
}
/// Even when every read FAILS, a scan that never managed to observe a
/// scrambled sector reports `Unencrypted` (we cannot prove encryption from
/// unreadable data alone — the AACS/keydb paths and the disc-level
@@ -864,12 +1259,17 @@ mod tests {
);
}
/// MISSING #1 guard: the re-crack path (the `Option`-returning `crack_key`,
/// `fail_on_locked == false`) must NOT hard-fail on a CSS-locked read — it
/// returns `None`. A lapsed-AGID re-crack of another VTS stays soft so a
/// genuinely crackable title isn't killed by a transient locked read.
/// `crack_key` (the `Option`-returning convenience wrapper) collapses
/// `ScrambledUncracked` and `Unencrypted` alike to `None` via
/// [`CrackOutcome::into_state`], so an all-locked scan still reads `None`
/// here even though the scan itself now treats every CSS-lock as a hard
/// `ScrambledUncracked` (see `crack_key_scan`'s removal of the dead
/// `fail_on_locked` parameter). Callers that need to tell "locked/
/// uncrackable" apart from "genuinely clear" must use `crack_key_outcome`,
/// which the `all_locked_synthetic_iso_yields_css_key_missing_signal` test
/// pins directly.
#[test]
fn crack_key_recrack_locked_is_none_not_hard_fail() {
fn crack_key_all_locked_collapses_to_none() {
let mut src = MockSource::new(0x30);
src.lock_all = true;
let extents = [Extent {
@@ -955,7 +1355,7 @@ mod tests {
/// budget). With a small failing extent, every sector is attempted and the
/// function returns None.
///
/// Grounding: `if reader.read_sectors(...).is_ok() && is_scrambled(...)` —
/// Grounding: `if reader.read_sectors(...).is_ok() && is_scrambled_pack(...)` —
/// an Err simply falls through to `i += 1`.
/// Mutation: change the read-error handling to `reader.read_sectors(...)?`
/// (propagate) -> crack_key would stop after the first error and read only
@@ -1194,6 +1594,44 @@ mod tests {
);
}
/// The decrypted-HD-DVD regression, end to end through `resolve_dvd_title_key`.
/// An `.evo` is MPEG-PS (`ContentFormat::MpegPs`) exactly like a DVD `.vob`
/// and carries `None` keys once decrypted, so it reaches the CSS crack. Its
/// RDI navigation packs are private_stream_2 (0xBF) with the pack-start code
/// and bits set at offset 0x14 — but HD-DVD carries no CSS at all. The 0x11
/// exclusion in `is_scrambled_pack` keeps those nav packs from flipping the
/// scan's `saw_scrambled` gate, so the scan returns `Unencrypted` and the
/// title muxes cleanly instead of hard-failing.
///
/// Catches the mutation of dropping the 0x11 exclusion: without it every 0xBF
/// RDI pack counts as scramble evidence, the crack finds no key (there is
/// none), and the scan returns `ScrambledUncracked` → `CssKeyMissing` (E7023)
/// on a perfectly good HD-DVD — the exact defect a real CI run produced.
#[test]
fn resolve_dvd_title_key_decrypted_hddvd_rdi_packs_scan_clean_no_e7023() {
let mut src = MockSource::new(0x30); // 0x14 bits set…
src.stream_id = crate::consts::pes_stream_id::PRIVATE_STREAM_2; // …but a 0xBF nav pack
let extents = [Extent {
start_lba: 0,
sector_count: 64,
}];
let mut keys = crate::decrypt::DecryptKeys::None;
resolve_dvd_title_key(
&mut src,
&extents,
&mut keys,
8,
crate::disc::ContentFormat::MpegPs,
false,
None,
)
.expect("a decrypted HD-DVD's RDI nav packs are not CSS — the scan must not hard-fail");
assert!(
matches!(keys, crate::decrypt::DecryptKeys::None),
"no CSS key exists on an HD-DVD; keys must stay None and the title mux clean"
);
}
/// `raw` is deliberate ciphertext passthrough: even a scrambled-uncrackable
/// title must return `Ok` and leave `keys` untouched (`None`) — no crack, no
/// hard-fail. This is the `--raw` guarantee.
@@ -1359,8 +1797,8 @@ mod tests {
/// PER-VTS RE-CRACK SUCCESS (audit gap "success path missing"): the prior
/// re-crack test only covered the locked→None path. Here a re-crack
/// (`crack_key`, `fail_on_locked == false`) over a DIFFERENT VTS's extents
/// finds that VTS's own crackable sector and returns a `CssState` whose
/// (`crack_key`) over a DIFFERENT VTS's extents finds that VTS's own
/// crackable sector and returns a `CssState` whose
/// `crack_span` matches the new extents — proving a key cracked for one VTS
/// is genuinely re-derived (not reused) for another.
#[test]
+66 -1
View File
@@ -695,6 +695,49 @@ mod tests {
// ── `decrypt_sectors_in_content` (now a legacy alias of `decrypt_sectors`) ──
/// `DecryptKeys::None` is a no-op even with a content map + scrambled bytes.
/// A forensic map's read plan is NOT the extents it was given — and that is
/// the precondition the mux's provenance guard keys on.
///
/// A clip's feed span is measured over the FULL extents at scan time, while
/// the mux reads this reduced plan, so the byte offsets stamped on frames
/// and the offsets recorded in the spans describe different streams. The
/// deficit accumulates, so every frame after the first segment resolves to
/// an earlier clip than it came from. The spans still tile each other, so
/// the tiling check cannot see it; the mux compares the plan against the
/// full extents instead and stops trusting provenance when they differ.
#[test]
fn a_forensic_read_plan_drops_units_the_full_extents_include() {
let full = vec![crate::disc::Extent {
start_lba: 1000,
sector_count: 60,
}];
// No forensic segment: the plan IS the extents, byte for byte, so
// provenance stays trustworthy on an ordinary disc.
let plain = AacsKeyMap::from_ranges_phased(vec![(1000, 1060, 5, Phase::All)]);
assert_eq!(
plain.read_plan(&full, 3),
full,
"a non-forensic map must return the extents unchanged"
);
// With alternate phases, units are omitted — fewer sectors are read
// than the spans describe.
let phased = AacsKeyMap::from_ranges_phased(vec![(1000, 1060, 5, Phase::Even)]);
let plan = phased.read_plan(&full, 3);
let planned: u32 = plan.iter().map(|e| e.sector_count).sum();
let whole: u32 = full.iter().map(|e| e.sector_count).sum();
assert!(
planned < whole,
"a forensic segment must drop units: planned {planned} of {whole}"
);
assert_ne!(
plan, full,
"the plan differs from the extents, which is exactly what the mux \
detects before deciding whether a byte offset means anything"
);
}
#[test]
fn content_gate_none_keys_is_noop() {
let mut keys = DecryptKeys::None;
@@ -987,6 +1030,12 @@ mod tests {
/// scramble flag.
fn make_css_sector(title_key: &[u8; 5], seed: &[u8; 5], body_fill: u8) -> (Vec<u8>, Vec<u8>) {
let mut sector = vec![body_fill; 2048];
// A real scrambled DVD sector is an MPEG-2 PS pack, so it begins with
// the pack start code. The descrambler requires it before trusting
// byte 0x14 — without it this fixture is a sector shape that cannot
// occur on a disc, and the test would pass while the production gate
// rejected every sector like it.
sector[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
sector[0x14] = 0x30; // scramble flag (bits 4-5)
sector[0x54..0x59].copy_from_slice(seed);
let plaintext = sector.clone();
@@ -1064,6 +1113,9 @@ mod tests {
period: usize,
) -> (Vec<u8>, Vec<u8>) {
let mut plaintext = vec![0u8; 2048];
// Real scrambled DVD sectors are MPEG-2 PS packs; the scramble policy
// requires the pack start code as well as the flag bits.
plaintext[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
plaintext[0x14] = 0x10; // scramble flag
// Periodic run from 0x59 (just above the seed) through 0x80 and on into
// the encrypted region; phase anchored to offset 0 so it is continuous
@@ -1847,11 +1899,24 @@ mod tests {
/// below its own range start, or a zero unit size, are both map bugs — they
/// must return a defined answer rather than panicking on debug overflow or
/// dividing by zero.
///
/// Every case here asserts the DEFINED answer, not merely the absence of a
/// panic. The zero-unit-size case used to be written
/// `assert!(unit_is_our_phase(100, 30, 0, Phase::Even) || true)`, which
/// accepts both answers and so pinned nothing at all: the guards could
/// invert and it would still pass. The answer is knowable —
/// `saturating_sub` gives 70, `max(1)` makes the divisor 1, unit index 70
/// is even — so pin it.
#[test]
fn phase_gate_does_not_panic_on_a_malformed_map() {
use super::{Phase, unit_is_our_phase};
// Unit below its own range start: saturating_sub clamps to 0, and unit
// 0 is even.
assert!(unit_is_our_phase(10, 100, 3, Phase::Even));
assert!(unit_is_our_phase(100, 30, 0, Phase::Even) || true);
// Zero unit size: max(1) makes the divisor 1, so the index is the raw
// offset 70 — even.
assert!(unit_is_our_phase(100, 30, 0, Phase::Even));
// Both malformations at once: offset 0 over divisor 1 is unit 0, even.
assert!(unit_is_our_phase(5, 5, 0, Phase::Even));
}
}
+1
View File
@@ -642,6 +642,7 @@ mod tests {
size_bytes,
clips: (0..n_clips)
.map(|i| Clip {
feed_span: None,
clip_id: format!("{i:05}"),
in_time: 0,
out_time: 0,
+743
View File
@@ -0,0 +1,743 @@
//! ECMA-167 / UDF 1.02 descriptor encoder.
//!
//! Turns a [`Layout`](super::layout::Layout) — a directory tree with every
//! ICB, directory-data and file-data block already assigned — into the set of
//! metadata sectors a real UDF volume would carry. Nothing here touches the
//! filesystem: it is a pure function from layout to sectors, which is what
//! makes it testable against the production parser in `udf.rs`.
//!
//! What is emitted, in volume order:
//!
//! | sector | descriptor |
//! |---|---|
//! | 16, 17, 18 | Volume Recognition Sequence — `BEA01`, `NSR02`, `TEA01` (ECMA-167 2/9.1) |
//! | 32… | Main Volume Descriptor Sequence — PVD, IUVD, PD, LVD, USD, TD |
//! | 48… | Reserve VDS (byte-identical but for the tag locations) |
//! | 64, 65 | Logical Volume Integrity Sequence — LVID, TD |
//! | 256 | Anchor Volume Descriptor Pointer |
//! | `part_start` + 0, +1 | File Set Descriptor, TD |
//! | `part_start` + … | File Entries (ICBs) and directory data (FIDs) |
//! | last sector | Anchor Volume Descriptor Pointer (copy) |
//!
//! UDF revision 1.02 with a single Type-1 partition map is deliberate: it is
//! the DVD-Video profile, it is the shape `read_filesystem` takes when
//! `num_partition_maps < 2`, and it avoids the UDF 2.50 Metadata Partition
//! entirely. That also means a synthetic image never exercises the Metadata
//! Partition path in `udf.rs` (`:946-991`) — see the module docs on `dirimage`.
use super::layout::{DirNode, Layout};
use crate::error::{Error, Result};
use std::collections::BTreeMap;
/// Logical block / sector size. Fixed for every optical profile this crate
/// reads, and the same quantity as [`crate::consts::SECTOR_BYTES`] — aliased
/// rather than re-declared so the two cannot drift apart. The short name is
/// kept because it appears in ~25 extent and offset expressions across
/// `dirimage`, where the longer one would bury the arithmetic.
pub(super) use crate::consts::SECTOR_BYTES as SECTOR;
/// Descriptor version recorded in every tag. 2 = ECMA-167 2nd edition, which
/// is what UDF revisions up to and including 2.00 require.
const DESC_VERSION: u16 = 2;
/// UDF revision recorded in the domain EntityID suffix (1.02, BCD-ish u16).
const UDF_REVISION: u16 = 0x0102;
/// A fixed recording timestamp, so an image synthesized from the same folder
/// twice is byte-identical. Real mtimes would make every test golden-file
/// comparison and every `dir:// -> iso://` re-run differ for no benefit.
const FIXED_TIME: Timestamp = Timestamp {
year: 2000,
month: 1,
day: 1,
};
struct Timestamp {
year: i16,
month: u8,
day: u8,
}
/// The synthesized metadata: absolute LBA → sector contents. Data sectors are
/// NOT here; they are served from the backing files.
pub(super) type MetaSectors = BTreeMap<u32, Box<[u8; SECTOR]>>;
/// The descriptor-tag CRC of ECMA-167 7.2.4: polynomial 0x1021, initial value
/// ZERO, no reflection, no final XOR — the variant catalogued as CRC-16/XMODEM
/// (check value 0x31C3), NOT CCITT-FALSE, which seeds at 0xFFFF and would make
/// every descriptor this crate writes fail a conformant driver's validation.
fn crc16(data: &[u8]) -> u16 {
let mut crc: u16 = 0;
for &b in data {
crc ^= (b as u16) << 8;
for _ in 0..8 {
crc = if crc & 0x8000 != 0 {
(crc << 1) ^ 0x1021
} else {
crc << 1
};
}
}
crc
}
/// Write an ECMA-167 3/7.2 descriptor tag over `buf[0..16]`.
///
/// `tag_loc` is the block number of the sector holding the descriptor —
/// ABSOLUTE for the volume-space descriptors (AVDP, VDS, LVID) and
/// PARTITION-RELATIVE for everything inside the partition (FSD, File Entries).
/// Getting that wrong is the classic reason a hand-built volume mounts nowhere:
/// a driver that validates the tag location rejects the descriptor outright.
///
/// `desc_len` is the descriptor's total length including the tag; the CRC
/// covers `buf[16..desc_len]`.
fn finish_tag(buf: &mut [u8], tag_id: u16, tag_loc: u32, desc_len: usize) {
buf[0..2].copy_from_slice(&tag_id.to_le_bytes());
buf[2..4].copy_from_slice(&DESC_VERSION.to_le_bytes());
buf[4] = 0; // checksum, filled below
buf[5] = 0; // reserved
buf[6..8].copy_from_slice(&0u16.to_le_bytes()); // tag serial number
let crc_len = desc_len - 16;
let crc = crc16(&buf[16..desc_len]);
buf[8..10].copy_from_slice(&crc.to_le_bytes());
buf[10..12].copy_from_slice(&(crc_len as u16).to_le_bytes());
buf[12..16].copy_from_slice(&tag_loc.to_le_bytes());
// ECMA-167 3/7.2.3: sum of bytes 0..16 EXCLUDING byte 4, modulo 256.
let sum: u32 = buf[0..16]
.iter()
.enumerate()
.filter(|(i, _)| *i != 4)
.map(|(_, b)| *b as u32)
.sum();
buf[4] = (sum % 256) as u8;
}
/// ECMA-167 1/7.2.1 charspec: type 0 (CS0) + "OSTA Compressed Unicode".
fn put_charspec(buf: &mut [u8]) {
buf[0] = 0;
let id = b"OSTA Compressed Unicode";
buf[1..1 + id.len()].copy_from_slice(id);
}
/// ECMA-167 1/7.4 EntityID: flags byte, 23 identifier bytes, 8 suffix bytes.
fn put_entity_id(buf: &mut [u8], id: &[u8], suffix: &[u8]) {
buf[0] = 0;
let n = id.len().min(23);
buf[1..1 + n].copy_from_slice(&id[..n]);
let m = suffix.len().min(8);
buf[24..24 + m].copy_from_slice(&suffix[..m]);
}
/// The `*OSTA UDF Compliant` domain EntityID suffix: UDF revision, domain
/// flags (0 = neither hard nor soft write-protected), reserved.
fn domain_suffix() -> [u8; 8] {
let mut s = [0u8; 8];
s[0..2].copy_from_slice(&UDF_REVISION.to_le_bytes());
s
}
/// This crate's implementation EntityID suffix: OS class / OS identifier
/// (0 = undefined, deliberately — the image is not OS-specific) + 6 free bytes.
fn impl_suffix() -> [u8; 8] {
[0u8; 8]
}
fn put_impl_id(buf: &mut [u8]) {
put_entity_id(buf, b"*freemkv", &impl_suffix());
}
fn put_domain_id(buf: &mut [u8]) {
put_entity_id(buf, b"*OSTA UDF Compliant", &domain_suffix());
}
/// OSTA CS0 d-string: a compression-ID byte, the characters, then the used
/// length in the FIELD'S LAST byte (ECMA-167 1/7.2.12 + UDF 2.1.3). An
/// all-zero field is the empty string.
fn put_dstring(buf: &mut [u8], s: &str) {
if s.is_empty() {
return;
}
let encoded = encode_cs0(s);
// Leave room for the trailing length byte.
let room = buf.len() - 1;
let n = encoded.len().min(room);
buf[..n].copy_from_slice(&encoded[..n]);
buf[buf.len() - 1] = n as u8;
}
/// OSTA CS0: compression ID 8 (one byte per character) when every character
/// is ASCII, otherwise compression ID 16 (UTF-16BE).
///
/// ASCII rather than Latin-1 for the 8-bit form on purpose: `parse_udf_name`
/// (`udf.rs:1467`) decodes a compression-8 name with `from_utf8_lossy`, so a
/// 0x80-0xFF byte — legal CS0 — would come back as U+FFFD. Every character
/// above 0x7F therefore takes the 16-bit form, which that parser decodes
/// correctly.
pub(super) fn encode_cs0(s: &str) -> Vec<u8> {
if s.is_ascii() {
let mut v = Vec::with_capacity(1 + s.len());
v.push(8u8);
v.extend_from_slice(s.as_bytes());
v
} else {
let mut v = vec![16u8];
for u in s.encode_utf16() {
v.extend_from_slice(&u.to_be_bytes());
}
v
}
}
/// ECMA-167 1/7.3 timestamp, 12 bytes. Type 1 (local time) with a zero
/// offset, i.e. UTC.
fn put_timestamp(buf: &mut [u8]) {
buf[0..2].copy_from_slice(&0x1000u16.to_le_bytes());
buf[2..4].copy_from_slice(&FIXED_TIME.year.to_le_bytes());
buf[4] = FIXED_TIME.month;
buf[5] = FIXED_TIME.day;
}
/// ECMA-167 3/7.1 extent_ad: length in BYTES, then location.
fn put_extent_ad(buf: &mut [u8], len_bytes: u32, lba: u32) {
buf[0..4].copy_from_slice(&len_bytes.to_le_bytes());
buf[4..8].copy_from_slice(&lba.to_le_bytes());
}
/// ECMA-167 4/14.14.2 long_ad: length+type, then lb_addr (block, partition
/// reference), then 6 implementation-use bytes.
fn put_long_ad(buf: &mut [u8], len_bytes: u32, lba: u32) {
buf[0..4].copy_from_slice(&len_bytes.to_le_bytes());
buf[4..8].copy_from_slice(&lba.to_le_bytes());
buf[8..10].copy_from_slice(&0u16.to_le_bytes()); // partition reference 0
}
/// ECMA-167 4/14.14.1 short_ad. The top two bits of the length word are the
/// extent TYPE (0 = recorded and allocated), which is exactly why `udf.rs`
/// masks with `0x3FFF_FFFF` when it reads one back — the mask is the field
/// boundary, not a truncation bug.
fn put_short_ad(buf: &mut [u8], len_bytes: u32, lba: u32) {
debug_assert!(len_bytes <= 0x3FFF_FFFF, "AD length must fit 30 bits");
buf[0..4].copy_from_slice(&len_bytes.to_le_bytes());
buf[4..8].copy_from_slice(&lba.to_le_bytes());
}
fn blank() -> Box<[u8; SECTOR]> {
Box::new([0u8; SECTOR])
}
// ── Volume-space descriptors ────────────────────────────────────────────────
/// ECMA-167 2/9.1 Volume Structure Descriptor: the three-sector recognition
/// sequence an OS looks for before it will even consider the volume UDF.
fn volume_recognition(id: &[u8; 5]) -> Box<[u8; SECTOR]> {
let mut s = blank();
s[0] = 0; // structure type
s[1..6].copy_from_slice(id);
s[6] = 1; // structure version
s
}
/// ECMA-167 3/10.1 Primary Volume Descriptor.
fn primary_volume(volume_id: &str, lba: u32, seq: u32) -> Box<[u8; SECTOR]> {
let mut s = blank();
s[16..20].copy_from_slice(&seq.to_le_bytes());
s[20..24].copy_from_slice(&0u32.to_le_bytes()); // PVD number
put_dstring(&mut s[24..56], volume_id);
s[56..58].copy_from_slice(&1u16.to_le_bytes()); // volume sequence number
s[58..60].copy_from_slice(&1u16.to_le_bytes()); // max volume sequence number
s[60..62].copy_from_slice(&2u16.to_le_bytes()); // interchange level
s[62..64].copy_from_slice(&2u16.to_le_bytes()); // max interchange level
s[64..68].copy_from_slice(&1u32.to_le_bytes()); // character set list
s[68..72].copy_from_slice(&1u32.to_le_bytes()); // max character set list
// UDF 2.2.2.5: the first 8 characters of the volume set identifier must be
// unique. A fixed hex prefix plus the volume id is sufficient here — the
// image is single-volume and never joins a real volume set.
put_dstring(&mut s[72..200], &format!("46524D4B{volume_id}"));
put_charspec(&mut s[200..264]); // descriptor character set
put_charspec(&mut s[264..328]); // explanatory character set
put_timestamp(&mut s[376..388]);
put_impl_id(&mut s[388..420]);
finish_tag(&mut s[..], 1, lba, 512);
s
}
/// ECMA-167 3/10.4 + UDF 2.2.7 Implementation Use Volume Descriptor
/// (`*UDF LV Info`). Not read by `udf.rs`, required by the spec.
fn impl_use_volume(volume_id: &str, lba: u32, seq: u32) -> Box<[u8; SECTOR]> {
let mut s = blank();
s[16..20].copy_from_slice(&seq.to_le_bytes());
put_entity_id(&mut s[20..52], b"*UDF LV Info", &domain_suffix());
put_charspec(&mut s[52..116]); // LVI charset
put_dstring(&mut s[116..244], volume_id); // logical volume identifier
put_impl_id(&mut s[352..384]);
finish_tag(&mut s[..], 4, lba, 512);
s
}
/// ECMA-167 3/10.5 Partition Descriptor — the descriptor `read_filesystem`
/// takes `partition_start` from (offset 188).
fn partition(part_start: u32, part_sectors: u32, lba: u32, seq: u32) -> Box<[u8; SECTOR]> {
let mut s = blank();
s[16..20].copy_from_slice(&seq.to_le_bytes());
s[20..22].copy_from_slice(&1u16.to_le_bytes()); // partition flags: allocated
s[22..24].copy_from_slice(&0u16.to_le_bytes()); // partition number
put_entity_id(&mut s[24..56], b"+NSR02", &[]);
// s[56..184] partition contents use = Partition Header Descriptor. All
// zero: a read-only partition records no unallocated/freed space tables.
s[184..188].copy_from_slice(&1u32.to_le_bytes()); // access type: read only
s[188..192].copy_from_slice(&part_start.to_le_bytes());
s[192..196].copy_from_slice(&part_sectors.to_le_bytes());
put_impl_id(&mut s[196..228]);
finish_tag(&mut s[..], 5, lba, 512);
s
}
/// ECMA-167 3/10.6 Logical Volume Descriptor. Carries the FSD long_ad and the
/// partition map table; `read_filesystem` reads `num_partition_maps` at 268
/// and takes the single-partition path when it is 1.
fn logical_volume(
volume_id: &str,
fsd_lba: u32,
integrity_lba: u32,
integrity_sectors: u32,
lba: u32,
seq: u32,
) -> Box<[u8; SECTOR]> {
let mut s = blank();
s[16..20].copy_from_slice(&seq.to_le_bytes());
put_charspec(&mut s[20..84]);
put_dstring(&mut s[84..212], volume_id);
s[212..216].copy_from_slice(&(SECTOR as u32).to_le_bytes()); // logical block size
put_domain_id(&mut s[216..248]);
// Logical volume contents use = long_ad of the File Set Descriptor,
// partition-relative. One sector.
put_long_ad(&mut s[248..264], SECTOR as u32, fsd_lba);
s[264..268].copy_from_slice(&6u32.to_le_bytes()); // map table length
s[268..272].copy_from_slice(&1u32.to_le_bytes()); // number of partition maps
put_impl_id(&mut s[272..304]);
put_extent_ad(
&mut s[432..440],
integrity_sectors * SECTOR as u32,
integrity_lba,
);
// ECMA-167 3/10.7.2 Type 1 partition map.
s[440] = 1; // map type
s[441] = 6; // map length
s[442..444].copy_from_slice(&1u16.to_le_bytes()); // volume sequence number
s[444..446].copy_from_slice(&0u16.to_le_bytes()); // partition number
finish_tag(&mut s[..], 6, lba, 446);
s
}
/// ECMA-167 3/10.8 Unallocated Space Descriptor with zero extents — the whole
/// volume is accounted for by the partition.
fn unallocated_space(lba: u32, seq: u32) -> Box<[u8; SECTOR]> {
let mut s = blank();
s[16..20].copy_from_slice(&seq.to_le_bytes());
s[20..24].copy_from_slice(&0u32.to_le_bytes());
finish_tag(&mut s[..], 7, lba, 24);
s
}
/// ECMA-167 3/10.9 Terminating Descriptor.
fn terminating(lba: u32) -> Box<[u8; SECTOR]> {
let mut s = blank();
finish_tag(&mut s[..], 8, lba, 512);
s
}
/// ECMA-167 3/10.10 + UDF 2.2.6 Logical Volume Integrity Descriptor, closed.
fn integrity(
part_sectors: u32,
files: u32,
dirs: u32,
next_uid: u64,
lba: u32,
) -> Box<[u8; SECTOR]> {
let mut s = blank();
put_timestamp(&mut s[16..28]);
s[28..32].copy_from_slice(&1u32.to_le_bytes()); // integrity type: close
// s[32..40] next integrity extent: none.
s[40..48].copy_from_slice(&next_uid.to_le_bytes()); // logical volume contents use: next unique id
s[72..76].copy_from_slice(&1u32.to_le_bytes()); // number of partitions
s[76..80].copy_from_slice(&46u32.to_le_bytes()); // length of implementation use
s[80..84].copy_from_slice(&0u32.to_le_bytes()); // free space: none (read-only)
s[84..88].copy_from_slice(&part_sectors.to_le_bytes()); // size table
put_impl_id(&mut s[88..120]);
s[120..124].copy_from_slice(&files.to_le_bytes());
s[124..128].copy_from_slice(&dirs.to_le_bytes());
s[128..130].copy_from_slice(&UDF_REVISION.to_le_bytes()); // min read revision
s[130..132].copy_from_slice(&UDF_REVISION.to_le_bytes()); // min write revision
s[132..134].copy_from_slice(&UDF_REVISION.to_le_bytes()); // max write revision
finish_tag(&mut s[..], 9, lba, 134);
s
}
/// ECMA-167 3/10.2 Anchor Volume Descriptor Pointer. `read_filesystem` reads
/// the main VDS extent from offsets 16..24 and sweeps it.
fn anchor(main_lba: u32, reserve_lba: u32, vds_sectors: u32, lba: u32) -> Box<[u8; SECTOR]> {
let mut s = blank();
put_extent_ad(&mut s[16..24], vds_sectors * SECTOR as u32, main_lba);
put_extent_ad(&mut s[24..32], vds_sectors * SECTOR as u32, reserve_lba);
finish_tag(&mut s[..], 2, lba, 512);
s
}
/// ECMA-167 4/14.1 File Set Descriptor. `read_filesystem` requires tag 256 at
/// the first block of the (metadata =) partition and reads the root ICB block
/// from offset 404.
fn file_set(volume_id: &str, root_icb: u32, lba: u32) -> Box<[u8; SECTOR]> {
let mut s = blank();
put_timestamp(&mut s[16..28]);
s[28..30].copy_from_slice(&3u16.to_le_bytes()); // interchange level
s[30..32].copy_from_slice(&3u16.to_le_bytes()); // max interchange level
s[32..36].copy_from_slice(&1u32.to_le_bytes()); // character set list
s[36..40].copy_from_slice(&1u32.to_le_bytes()); // max character set list
s[40..44].copy_from_slice(&0u32.to_le_bytes()); // file set number
s[44..48].copy_from_slice(&0u32.to_le_bytes()); // file set descriptor number
put_charspec(&mut s[48..112]);
put_dstring(&mut s[112..240], volume_id);
put_charspec(&mut s[240..304]);
put_dstring(&mut s[304..336], volume_id);
put_long_ad(&mut s[400..416], SECTOR as u32, root_icb);
put_domain_id(&mut s[416..448]);
finish_tag(&mut s[..], 256, lba, 512);
s
}
// ── Partition-space descriptors ─────────────────────────────────────────────
/// UDF permission word: read + execute for owner, group and other. No write
/// bit anywhere — the volume is read-only.
const PERM_R_X: u32 = 0x0000_1000 | 0x0000_0400 | 0x0000_0080 | 0x0000_0020 | 0x4 | 0x1;
/// ECMA-167 4/14.9 File Entry (tag 261).
///
/// Tag 261 rather than the Extended File Entry (266) real BD-ROMs use: an EFE
/// requires UDF 2.00+, and this image declares 1.02. `udf.rs` reads both — the
/// 261 field offsets it uses (l_ea 168, l_ad 172, ADs at 176 + l_ea) are the
/// ones written here.
///
/// `extents` are partition-relative (block, byte-length) pairs, already split
/// so no single one exceeds the 30-bit AD length field.
fn file_entry(
is_dir: bool,
info_len: u64,
extents: &[(u32, u32)],
link_count: u16,
unique_id: u64,
lba: u32,
) -> Result<Box<[u8; SECTOR]>> {
let mut s = blank();
// ICB tag (ECMA-167 4/14.6) at offset 16.
s[16..20].copy_from_slice(&0u32.to_le_bytes()); // prior recorded direct entries
s[20..22].copy_from_slice(&4u16.to_le_bytes()); // strategy type 4
s[24..26].copy_from_slice(&1u16.to_le_bytes()); // max number of entries
s[27] = if is_dir { 4 } else { 5 }; // file type: directory / byte sequence
// s[28..34] parent ICB location: not recorded (permitted).
// s[34..36] ICB flags: 0 => short allocation descriptors. `udf.rs:601`
// reads exactly this word to pick its AD stride.
s[34..36].copy_from_slice(&0u16.to_le_bytes());
// UDF's sentinel for "not specified" is 0xFFFFFFFF, not 0 — 0 is a real
// uid/gid (root). A synthesized image has no meaningful owner, and a driver
// that maps these through would otherwise report every file as root-owned.
s[36..40].copy_from_slice(&u32::MAX.to_le_bytes()); // uid: not specified
s[40..44].copy_from_slice(&u32::MAX.to_le_bytes()); // gid: not specified
s[44..48].copy_from_slice(&PERM_R_X.to_le_bytes());
s[48..50].copy_from_slice(&link_count.to_le_bytes());
s[56..64].copy_from_slice(&info_len.to_le_bytes());
let blocks: u64 = extents
.iter()
.map(|(_, len)| (*len as u64).div_ceil(SECTOR as u64))
.sum();
s[64..72].copy_from_slice(&blocks.to_le_bytes()); // logical blocks recorded
put_timestamp(&mut s[72..84]); // access
put_timestamp(&mut s[84..96]); // modification
put_timestamp(&mut s[96..108]); // attribute
s[108..112].copy_from_slice(&1u32.to_le_bytes()); // checkpoint
put_impl_id(&mut s[128..160]);
s[160..168].copy_from_slice(&unique_id.to_le_bytes());
s[168..172].copy_from_slice(&0u32.to_le_bytes()); // length of EAs
let l_ad = extents.len() * 8;
// A short AD is 8 bytes and the entry has 2048 - 176 = 1872 bytes for
// them, i.e. 234 extents — over 200 GiB at the per-AD ceiling. Beyond
// that an Allocation Extent Descriptor chain would be required; refuse
// rather than write a truncated list.
if 176 + l_ad > SECTOR {
return Err(Error::DirImageTooLarge);
}
s[172..176].copy_from_slice(&(l_ad as u32).to_le_bytes());
for (i, (elba, len)) in extents.iter().enumerate() {
let off = 176 + i * 8;
put_short_ad(&mut s[off..off + 8], *len, *elba);
}
finish_tag(&mut s[..], 261, lba, 176 + l_ad);
Ok(s)
}
/// ECMA-167 4/14.4 File Identifier Descriptor, appended to `buf`.
///
/// FIDs are packed with no inter-descriptor padding beyond the 4-byte
/// alignment the spec mandates, and they are allowed to span logical blocks —
/// which is also what `read_directory` (`udf.rs:1312`) assumes: it walks the
/// directory extent as one flat byte run and STOPS at the first non-257 tag,
/// so any block-alignment gap would truncate the directory.
fn push_fid(buf: &mut Vec<u8>, name: &str, icb_lba: u32, is_dir: bool, is_parent: bool) {
let start = buf.len();
let name_field: Vec<u8> = if is_parent {
Vec::new()
} else {
encode_cs0(name)
};
let l_fi = name_field.len();
let mut fid = vec![0u8; 38];
fid[16..18].copy_from_slice(&1u16.to_le_bytes()); // file version number
let mut chars = 0u8;
if is_dir {
chars |= 0x02;
}
if is_parent {
chars |= 0x08;
}
fid[18] = chars;
// The planner refuses any name whose encoding exceeds what this byte can
// hold (`layout::MAX_CS0_NAME_BYTES`), so this cannot wrap in practice. The
// assert states the invariant where it is relied on rather than trusting a
// check three files away; a wrap here would desynchronise the directory.
debug_assert!(
l_fi <= u8::MAX as usize,
"FID name length must fit one byte"
);
fid[19] = l_fi as u8;
put_long_ad(&mut fid[20..36], SECTOR as u32, icb_lba);
fid[36..38].copy_from_slice(&0u16.to_le_bytes()); // length of implementation use
buf.extend_from_slice(&fid);
buf.extend_from_slice(&name_field);
let unpadded = buf.len() - start;
let padded = unpadded.div_ceil(4) * 4;
buf.resize(start + padded, 0);
// The tag is written last: its CRC covers the descriptor body, which the
// padding is not part of (ECMA-167 4/14.4.9 counts padding outside the
// CRC'd length).
let tag_loc_placeholder = 0;
finish_tag(
&mut buf[start..start + unpadded],
257,
tag_loc_placeholder,
unpadded,
);
}
/// Serialize one directory's FID list (parent entry first, then children).
pub(super) fn dir_fids(dir: &DirNode) -> Vec<u8> {
let mut buf = Vec::new();
push_fid(&mut buf, "", dir.parent_icb_lba, true, true);
for sub in &dir.dirs {
push_fid(&mut buf, &sub.name, sub.icb_lba, true, false);
}
for f in &dir.files {
push_fid(&mut buf, &f.name, f.icb_lba, false, false);
}
buf
}
/// Patch every FID's tag location to the block it actually lands in. ECMA-167
/// 3/7.2.2 makes the tag location the block of the descriptor, and a FID that
/// spans two blocks records the block it STARTS in.
fn fix_fid_tag_locations(buf: &mut [u8], first_block: u32) {
let mut pos = 0usize;
while pos + 38 <= buf.len() {
let l_fi = buf[pos + 19] as usize;
let l_iu = u16::from_le_bytes([buf[pos + 36], buf[pos + 37]]) as usize;
let unpadded = 38 + l_iu + l_fi;
if pos + unpadded > buf.len() {
break;
}
let block = first_block + (pos / SECTOR) as u32;
finish_tag(&mut buf[pos..pos + unpadded], 257, block, unpadded);
pos += unpadded.div_ceil(4) * 4;
}
}
// ── Whole-image assembly ────────────────────────────────────────────────────
/// Volume-space block of the Volume Recognition Sequence.
const VRS_START: u32 = 16;
/// Volume-space block of the Main Volume Descriptor Sequence.
pub(super) const MAIN_VDS_START: u32 = 32;
/// Volume-space block of the Reserve Volume Descriptor Sequence.
pub(super) const RESERVE_VDS_START: u32 = 48;
/// Sectors reserved for each VDS. ECMA-167 3/10.2.1 requires an anchor to
/// record at least 16.
pub(super) const VDS_SECTORS: u32 = 16;
/// Volume-space block of the Logical Volume Integrity Sequence.
pub(super) const LVID_START: u32 = 64;
/// Sectors reserved for the integrity sequence (LVID + TD).
pub(super) const LVID_SECTORS: u32 = 2;
/// The mandatory anchor block (ECMA-167 3/10.2).
pub(super) const ANCHOR_LBA: u32 = 256;
/// First block a partition may start at. Everything above is volume space.
pub(super) const MIN_PART_START: u32 = 320;
/// Emit the six-descriptor Volume Descriptor Sequence at `start`.
fn write_vds(out: &mut MetaSectors, layout: &Layout, start: u32) {
let vid = &layout.volume_id;
out.insert(start, primary_volume(vid, start, 1));
out.insert(start + 1, impl_use_volume(vid, start + 1, 2));
out.insert(
start + 2,
partition(layout.part_start, layout.part_sectors, start + 2, 3),
);
out.insert(
start + 3,
logical_volume(vid, 0, LVID_START, LVID_SECTORS, start + 3, 4),
);
out.insert(start + 4, unallocated_space(start + 4, 5));
out.insert(start + 5, terminating(start + 5));
}
/// Recursively emit one directory's File Entry and FID list, then its
/// children's.
fn write_dir(out: &mut MetaSectors, layout: &Layout, dir: &DirNode) -> Result<()> {
let mut fids = dir_fids(dir);
fix_fid_tag_locations(&mut fids, dir.data_lba);
debug_assert_eq!(fids.len(), dir.data_bytes as usize);
// A directory's link count is 1 (its own FID in the parent) plus one for
// each child directory's parent FID pointing back at it.
// The planner caps subdirectory fan-out (`layout::MAX_SUBDIRS`) so this
// cannot overflow; saturating rather than wrapping keeps a future change to
// that cap from silently producing a wrong count.
let link_count = (dir.dirs.len() as u16).saturating_add(1);
let fe = file_entry(
true,
fids.len() as u64,
&[(dir.data_lba, fids.len() as u32)],
link_count,
dir.unique_id,
dir.icb_lba,
)?;
out.insert(layout.part_start + dir.icb_lba, fe);
for (i, chunk) in fids.chunks(SECTOR).enumerate() {
let mut s = blank();
s[..chunk.len()].copy_from_slice(chunk);
out.insert(layout.part_start + dir.data_lba + i as u32, s);
}
for f in &dir.files {
let extents: Vec<(u32, u32)> = f.extents.iter().map(|e| (e.lba, e.bytes)).collect();
let fe = file_entry(false, f.size, &extents, 1, f.unique_id, f.icb_lba)?;
out.insert(layout.part_start + f.icb_lba, fe);
}
for sub in &dir.dirs {
write_dir(out, layout, sub)?;
}
Ok(())
}
/// Build every metadata sector of the synthesized volume.
pub(super) fn encode(layout: &Layout) -> Result<MetaSectors> {
let mut out = MetaSectors::new();
out.insert(VRS_START, volume_recognition(b"BEA01"));
out.insert(VRS_START + 1, volume_recognition(b"NSR02"));
out.insert(VRS_START + 2, volume_recognition(b"TEA01"));
write_vds(&mut out, layout, MAIN_VDS_START);
write_vds(&mut out, layout, RESERVE_VDS_START);
out.insert(
LVID_START,
integrity(
layout.part_sectors,
layout.file_count,
layout.dir_count,
layout.next_unique_id,
LVID_START,
),
);
out.insert(LVID_START + 1, terminating(LVID_START + 1));
let avdp = anchor(MAIN_VDS_START, RESERVE_VDS_START, VDS_SECTORS, ANCHOR_LBA);
out.insert(ANCHOR_LBA, avdp);
let last = layout.total_sectors - 1;
out.insert(
last,
anchor(MAIN_VDS_START, RESERVE_VDS_START, VDS_SECTORS, last),
);
// Partition block 0 must hold the File Set Descriptor: `read_filesystem`
// reads exactly `metadata_start` (== partition start on a single-partition
// volume) and rejects the volume outright if the tag there is not 256.
out.insert(
layout.part_start,
file_set(&layout.volume_id, layout.root.icb_lba, 0),
);
out.insert(layout.part_start + 1, terminating(1));
write_dir(&mut out, layout, &layout.root)?;
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
/// The reference check value for CRC-16/XMODEM — poly 0x1021 seeded at 0,
/// which is what ECMA-167 7.2.4 specifies: "123456789" → 0x31C3. Seeding
/// at 0xFFFF instead (CCITT-FALSE) yields 0x29B1, and that mutant is
/// invisible to `udf.rs`, which never verifies a tag CRC — it would only
/// show up as a volume no operating system will mount.
#[test]
fn crc16_matches_the_ecma167_check_value() {
assert_eq!(crc16(b"123456789"), 0x31C3);
assert_ne!(crc16(b"123456789"), 0x29B1, "not the 0xFFFF-seeded variant");
}
/// ECMA-167 3/7.2.3: the checksum is the sum of the tag's first 16 bytes
/// EXCLUDING the checksum byte itself, modulo 256.
#[test]
fn tag_checksum_excludes_its_own_byte() {
let mut buf = [0u8; 512];
buf[16..24].copy_from_slice(&[1, 2, 3, 4, 5, 6, 7, 8]);
finish_tag(&mut buf, 261, 0x1234, 512);
let sum: u32 = buf[0..16]
.iter()
.enumerate()
.filter(|(i, _)| *i != 4)
.map(|(_, b)| *b as u32)
.sum();
assert_eq!(buf[4] as u32, sum % 256);
// And the recorded CRC covers the body, not the tag.
let crc = u16::from_le_bytes([buf[8], buf[9]]);
assert_eq!(crc, crc16(&buf[16..512]));
assert_eq!(u16::from_le_bytes([buf[10], buf[11]]), 496);
assert_eq!(
u32::from_le_bytes([buf[12], buf[13], buf[14], buf[15]]),
0x1234
);
}
/// ASCII takes compression ID 8; anything above takes 16 (UTF-16BE),
/// because `parse_udf_name` decodes compression-8 bytes as UTF-8.
#[test]
fn cs0_picks_the_encoding_the_parser_can_decode() {
assert_eq!(encode_cs0("AB"), vec![8, b'A', b'B']);
let e = encode_cs0("Ä");
assert_eq!(e[0], 16);
assert_eq!(&e[1..], &[0x00, 0xC4]);
assert_eq!(crate::udf::parse_udf_name(&e), "Ä");
}
/// A d-string records its used length in the field's LAST byte, and the
/// production parser must read the same string back.
#[test]
fn dstring_round_trips_through_the_production_parser() {
let mut field = [0u8; 32];
put_dstring(&mut field, "FREEMKV");
assert_eq!(field[31], 8, "compid byte + 7 characters");
assert_eq!(crate::udf::parse_dstring_for_test(&field), "FREEMKV");
}
}
File diff suppressed because it is too large Load Diff
+362
View File
@@ -0,0 +1,362 @@
//! `dir://` as an image-level SOURCE: a synthetic UDF volume over a folder.
//!
//! A user's extracted disc — a DVD `VIDEO_TS/` or a Blu-ray `BDMV/`, typically
//! a MakeMKV-style backup — has files but no sectors, and everything above the
//! sector layer in this crate wants sectors: `Disc::scan_image`, `UdfFs`,
//! `ifo.rs`, `mpls.rs`, `clpi.rs` and the mux all read through a
//! [`SectorSource`]. [`DirImage`] supplies one.
//!
//! The trick is that nothing is emulated. A real, minimal, valid UDF 1.02
//! volume is synthesized over the folder:
//!
//! * **Metadata sectors** (anchors, the volume descriptor sequences, the File
//! Set Descriptor, every File Entry, every directory's FID list) are encoded
//! into RAM by [`encode`] — a few MiB even for a large Blu-ray.
//! * **Data sectors** are not materialized at all. Each one maps to a byte
//! range of a real file, read on demand.
//!
//! So `udf::read_filesystem` parses this image by exactly the same code path it
//! parses a real disc with, and every consumer above it is unchanged. The cost
//! is that a single-partition synthetic volume never exercises the UDF 2.50
//! Metadata Partition path (`udf.rs:946-991`) that every real BD-ROM uses —
//! this module's tests do not cover that block and must not be read as if they
//! did.
//!
//! What this module deliberately does NOT do:
//!
//! * **3D / SSIF** — rejected up front ([`Error::DirImageSsifUnsupported`]).
//! An SSIF aliases the same sectors as its base and dependent `.m2ts`; the
//! planner allocates disjoint extents, so a 3D folder would produce silently
//! wrong output.
//! * **HD-DVD `HVDVD_TS/`** — no title enumerator constraint is modelled.
//! * **Encrypted folders** — a folder whose content is still AACS-scrambled is
//! rejected by the caller-side probe, not decrypted here.
mod encode;
mod layout;
use crate::error::{Error, Result};
#[cfg(target_os = "linux")]
use crate::io::file_sector_source::linux::drop_window;
#[cfg(target_os = "macos")]
use crate::io::file_sector_source::macos::drop_window;
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
use crate::io::file_sector_source::other::drop_window;
#[cfg(target_os = "windows")]
use crate::io::file_sector_source::windows::drop_window;
use crate::sector::SectorSource;
use encode::{MetaSectors, SECTOR};
use std::fs::File;
use std::io::{Read, Seek, SeekFrom};
use std::path::{Path, PathBuf};
/// How many host files may be held open at once.
///
/// A Blu-ray `BDMV/` can exceed a thousand files while macOS `RLIMIT_NOFILE`
/// defaults to 256, so "open every file up front" is not available. Reads are
/// overwhelmingly sequential through one large stream file at a time, so a
/// small LRU keeps the hit rate near 1 while bounding descriptors.
const HANDLE_CACHE: usize = 16;
/// One file's bytes at one place in the image.
#[derive(Debug, Clone)]
struct DataRange {
/// Absolute first block.
start_lba: u32,
/// Blocks covered (the last one may be partially used, and is zero-padded).
sectors: u32,
/// Index into [`DirImage::files`].
file: usize,
/// Byte offset within the file at which this range's bytes begin.
offset: u64,
/// Byte length of the range.
bytes: u64,
}
/// A file the image reads through.
#[derive(Debug)]
struct FileRef {
host: PathBuf,
disc_path: String,
size: u64,
/// Host mtime at plan time — see `layout::FileNode::mtime` for why size
/// alone is not enough.
mtime: Option<std::time::SystemTime>,
}
/// A synthesized UDF disc image over a host directory.
///
/// Owns everything it reads through (`PathBuf`s and its own file handles), so
/// it is `Send + 'static` and can be moved into `build_iso_pipeline`, which
/// hands it to `PrefetchedSectorSource`'s producer thread.
pub struct DirImage {
meta: MetaSectors,
/// Sorted by `start_lba`, non-overlapping.
ranges: Vec<DataRange>,
files: Vec<FileRef>,
open: Vec<(usize, File)>,
total_sectors: u32,
volume_id: String,
data_bytes: u64,
}
impl std::fmt::Debug for DirImage {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("DirImage")
.field("volume_id", &self.volume_id)
.field("total_sectors", &self.total_sectors)
.field("files", &self.files.len())
.field("meta_sectors", &self.meta.len())
.finish()
}
}
impl DirImage {
/// Plan and encode an image over `root`.
///
/// Every error is decided here, at plan time, where it can name the file
/// responsible — the read path is deliberately left with nothing to decide
/// except "this file changed underneath me".
pub fn open(root: &Path) -> Result<Self> {
let plan = layout::plan(root)?;
let meta = encode::encode(&plan)?;
let mut nodes = Vec::new();
layout::flatten(&plan.root, &mut nodes);
let mut files = Vec::with_capacity(nodes.len());
let mut ranges = Vec::new();
for (idx, node) in nodes.iter().enumerate() {
// Carry the plan-time mtime ONLY for files whose CONTENT the plan
// read — the DVD IFOs, whose bytes 0xC0/0xC4 decide where every VOB
// is placed (`layout::place_video_ts` -> `read_head`).
//
// For every other file the plan depends on the SIZE alone, and size
// is already checked. Comparing mtime on those buys nothing and
// costs real false positives: disc backups commonly live on
// exFAT/FAT32, which stores local time, so a long rip spanning a
// DST transition sees a whole-hour shift on a file nobody touched
// and would abort hours in, blaming a change that did not happen.
// The multi-gigabyte VOBs are exactly the files a long rip re-opens
// after the handle cache evicts them.
let content_sensitive = node
.disc_path
.rsplit('.')
.next()
.is_some_and(|e| e.eq_ignore_ascii_case("IFO"));
files.push(FileRef {
host: node.host.clone(),
disc_path: node.disc_path.clone(),
size: node.size,
mtime: content_sensitive.then_some(node.mtime).flatten(),
});
let mut offset = 0u64;
for e in &node.extents {
ranges.push(DataRange {
start_lba: plan.part_start + e.lba,
sectors: (e.bytes as u64).div_ceil(SECTOR as u64) as u32,
file: idx,
offset,
bytes: e.bytes as u64,
});
offset += e.bytes as u64;
}
}
ranges.sort_by_key(|r| r.start_lba);
debug_assert!(
ranges
.windows(2)
.all(|w| w[0].start_lba + w[0].sectors <= w[1].start_lba),
"planned data ranges must not overlap"
);
let data_bytes = layout::total_data_bytes(&plan.root);
tracing::info!(
target: "freemkv::dirimage",
volume_id = %plan.volume_id,
files = files.len(),
dirs = plan.dir_count,
meta_blocks = layout::metadata_block_count(&plan.root),
total_sectors = plan.total_sectors,
"synthesized UDF image over directory"
);
Ok(Self {
meta,
ranges,
files,
open: Vec::new(),
total_sectors: plan.total_sectors,
volume_id: plan.volume_id,
data_bytes,
})
}
/// UDF volume identifier the image declares (the folder's own name).
pub fn volume_id(&self) -> &str {
&self.volume_id
}
/// Total bytes of real file content the image carries — the folder's size,
/// not the image's (which also counts metadata and inter-file gaps).
pub fn data_bytes(&self) -> u64 {
self.data_bytes
}
/// The range covering `lba`, if any.
fn range_at(&self, lba: u32) -> Option<&DataRange> {
let i = self.ranges.partition_point(|r| r.start_lba <= lba);
let r = self.ranges.get(i.checked_sub(1)?)?;
(lba < r.start_lba + r.sectors).then_some(r)
}
/// Borrow an open handle for `file`, opening it (and evicting the
/// least-recently-used handle) if necessary.
///
/// Opening is also where the plan is revalidated. A folder is not a disc:
/// a file can be shortened or replaced between planning and reading, and
/// zero-filling the difference would turn "the user deleted something"
/// into corrupt output at exit 0. The size is re-checked here, and a
/// truncation that happens while the handle is already open is caught by
/// the short read in [`Self::fill`].
fn handle(&mut self, file: usize) -> Result<&mut File> {
if let Some(pos) = self.open.iter().position(|(i, _)| *i == file) {
// `open` is ordered most-recently-used first.
let entry = self.open.remove(pos);
self.open.insert(0, entry);
return Ok(&mut self.open[0].1);
}
let f = File::open(&self.files[file].host).map_err(Error::from)?;
let md = f.metadata().map_err(Error::from)?;
// Size AND mtime. Size alone is content-blind, and this plan depends on
// content: a DVD's VOB placement comes from bytes 0xC0/0xC4 of its IFO,
// and an IFO rewritten in place keeps its length because IFOs occupy a
// whole number of sectors. The size check would pass while every title
// extent pointed at the wrong sectors — corrupt video behind an intact
// structure, reported complete at exit 0.
//
// Only compared when both sides have a timestamp; a platform or
// filesystem that reports none simply falls back to the size check
// rather than failing every read.
let changed_size = md.len() != self.files[file].size;
let changed_mtime = match (self.files[file].mtime, md.modified().ok()) {
(Some(planned), Some(live)) => planned != live,
_ => false,
};
if changed_size || changed_mtime {
return Err(Error::DirImageFileChanged {
path: self.files[file].disc_path.clone(),
});
}
if self.open.len() >= HANDLE_CACHE {
self.open.pop();
}
self.open.insert(0, (file, f));
Ok(&mut self.open[0].1)
}
/// Fill `out` (a whole number of sectors) from one data range, starting at
/// `lba`. `out` is already zeroed, so a file's tail sector comes back
/// zero-padded — which is exactly what `file_extents`' `div_ceil(2048)`
/// (`udf.rs:816`) makes every consumer expect.
fn fill(&mut self, r: &DataRange, lba: u32, out: &mut [u8]) -> Result<()> {
let within = (lba - r.start_lba) as u64 * SECTOR as u64;
let want = (r.bytes.saturating_sub(within)).min(out.len() as u64) as usize;
if want == 0 {
return Ok(());
}
let at = r.offset + within;
let file = r.file;
let h = self.handle(file)?;
h.seek(SeekFrom::Start(at)).map_err(Error::from)?;
let res = h.read_exact(&mut out[..want]);
if res.is_ok() {
// Release the window just read, every time.
//
// The ISO source accumulates and drops in chunks because it reads
// one file linearly, so a running start offset always names the
// bytes it has consumed. Reads here jump between files, so there is
// no single cursor to accumulate against — an accumulated byte
// count paired with one read's offset names 1/Nth of what was
// actually consumed and leaves the rest pinned, which is how the
// first version of this got it wrong.
//
// Dropping per read costs one advisory syscall per batch (4-16 MiB),
// which is nothing against the read itself, and it is correct
// regardless of how reads interleave across files.
if let Some((_, fh)) = self.open.iter().find(|(i, _)| *i == file) {
drop_window(fh, at, want as u64);
}
}
match res {
Ok(()) => Ok(()),
// The file shrank while the handle was open. Same verdict as the
// size check in `handle`, reached the other way.
Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
Err(Error::DirImageFileChanged {
path: self.files[file].disc_path.clone(),
})
}
Err(e) => Err(Error::from(e)),
}
}
}
impl SectorSource for DirImage {
fn capacity_sectors(&self) -> u32 {
self.total_sectors
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
let need = count as usize * SECTOR;
if buf.len() < need {
return Err(Error::UdfBufferTooSmall);
}
buf[..need].fill(0);
// Walk the request in RUNS, not sector by sector. A mux batch is 8192
// sectors and almost always lands entirely inside one stream file's
// extent; per-sector seek+read would issue 8192 syscalls for what is
// one 16 MiB sequential read.
let mut i = 0u32;
while i < count as u32 {
// Checked: callers saturate their LBAs (`disc/dvd.rs` builds a cell
// start as `vob_start_sector.saturating_add(cell.first_sector)`, and
// the prefetcher adds an offset the same way), so a crafted IFO can
// present a request at the very top of the address space. Wrapping
// here would fold `at` back to a LOW sector and hand the muxer a
// different file's bytes with nothing reported.
let Some(at) = lba.checked_add(i) else {
break;
};
let off = i as usize * SECTOR;
if let Some(s) = self.meta.get(&at) {
buf[off..off + SECTOR].copy_from_slice(&s[..]);
i += 1;
continue;
}
// Metadata blocks all sit below the data floor, so a data range is
// never interrupted by one.
match self.range_at(at).cloned() {
Some(r) => {
let run = (r.start_lba + r.sectors - at).min(count as u32 - i);
let end = off + run as usize * SECTOR;
self.fill(&r, at, &mut buf[off..end])?;
i += run;
}
// A gap between planned extents. Reads as zeros, exactly as an
// unrecorded sector of a real image does.
None => i += 1,
}
}
Ok(need)
}
}
#[cfg(test)]
mod tests;
File diff suppressed because it is too large Load Diff
+916 -111
View File
File diff suppressed because it is too large Load Diff
+164 -22
View File
@@ -7,13 +7,32 @@ use crate::udf;
impl Disc {
/// Scan DVD titles from IFO files (VIDEO_TS.IFO + VTS_XX_0.IFO).
///
/// Cancellation: `halt` is polled before the IFO tree is read, and an IFO
/// read that fails with [`Error::Halted`] — how a live drive reports a
/// Stop, since `Drive::checked_exec` fails EVERY command once its flag is
/// set — is propagated rather than swallowed. Every other IFO failure
/// keeps its best-effort `Ok(vec![])`.
///
/// It has to be an error and not an empty title list, for the same reason
/// spelled out on [`Disc::scan_hddvd_titles`]: a cancelled enumeration
/// that returned `Ok` would be indistinguishable from a disc that
/// genuinely holds fewer titles. This one was the worst of the three
/// enumerators — a bare `Err(_) => return Vec::new()` turned an operator
/// Stop into ZERO titles at rc=0, a disc reported as carrying no video at
/// all.
pub(super) fn scan_dvd_titles(
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
) -> Vec<DiscTitle> {
halt: Option<&crate::halt::Halt>,
) -> Result<Vec<DiscTitle>> {
if halt.is_some_and(|h| h.is_cancelled()) {
return Err(Error::Halted);
}
let dvd_info = match ifo::parse_vmg(reader, udf_fs) {
Ok(info) => info,
Err(_) => return Vec::new(),
Err(Error::Halted) => return Err(Error::Halted),
Err(_) => return Ok(Vec::new()),
};
let mut titles = Vec::new();
@@ -243,7 +262,14 @@ impl Disc {
}
}
titles
// Polled again AFTER the loop: a cancel raised during the IFO reads
// that `parse_vmg` performs per title set has nothing left to poll,
// so without this a partially enumerated disc could still be handed
// back as success.
if halt.is_some_and(|h| h.is_cancelled()) {
return Err(Error::Halted);
}
Ok(titles)
}
}
@@ -528,6 +554,118 @@ mod tests {
// Tests
// ---------------------------------------------------------------
/// A `SectorSource` that fails every read at or above `halt_at` with
/// [`Error::Halted`] — how a LIVE DRIVE behaves once the operator presses
/// Stop: `Drive::checked_exec` fails every SCSI command with `Halted` from
/// then on, and `Drive::read` deliberately preserves the variant. Reads
/// below the threshold still succeed, so the scan gets far enough to have
/// something to truncate.
struct HaltingReader<'a> {
inner: &'a mut MemDisc,
halt_at: u32,
}
impl SectorSource for HaltingReader<'_> {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> crate::error::Result<usize> {
if lba >= self.halt_at {
return Err(crate::error::Error::Halted);
}
self.inner.read_sectors(lba, count, buf, recovery)
}
}
/// A Stop on a LIVE DRIVE never touches `ScanOptions::halt`: `Drive` has
/// its own flag and `checked_exec` fails every SCSI command with
/// [`Error::Halted`] once it is set. The DVD enumerator must not swallow
/// that into a successful scan.
///
/// This was the worst of the three enumerators. RED BEFORE GREEN, two
/// distinct swallows, both measured with the fix reverted:
/// * `ifo::parse_vmg` treats a failed title set as a placeholder entry
/// and continues, so a cancel landing on VTS_02's IFO returned
/// `Ok([VTS_01_1.VOB])` — one title from a two-title disc.
/// * `scan_dvd_titles`'s `Err(_) => return Vec::new()` turned a cancel
/// landing on VIDEO_TS.IFO itself into ZERO titles at rc=0 — a disc
/// reported as holding no video at all.
///
/// Both are indistinguishable from a real disc, and both are now
/// `Err(Error::Halted)`.
#[test]
fn halted_ifo_read_is_not_reported_as_a_shorter_disc() {
// Two title sets: VTS_01's IFO data at PART_START+6000, VTS_02's at
// PART_START+7000. Both ICBs sit far below, so the filesystem
// metadata resolves and only the second title set's CONTENT is
// cancelled — the truncation case.
let mut disc = MemDisc::new();
let vmg = build_vmg(&[(1, 1, 1), (1, 2, 1)]);
let vts1 = build_vts(100, 0x00, &[], &[], &[(0, 9)], false);
let vts2 = build_vts(200, 0x00, &[], &[], &[(0, 19)], false);
let udf = build_video_ts_fs(
&mut disc,
&[
FileSpec {
name: "VIDEO_TS.IFO".into(),
icb_lba: 60,
data_lba: 5000,
contents: vmg,
},
FileSpec {
name: "VTS_01_0.IFO".into(),
icb_lba: 62,
data_lba: 6000,
contents: vts1,
},
FileSpec {
name: "VTS_02_0.IFO".into(),
icb_lba: 64,
data_lba: 7000,
contents: vts2,
},
],
);
// Sanity: both title sets enumerate when nothing is cancelled, so a
// short list below can only be the cancel.
assert_eq!(
Disc::scan_dvd_titles(&mut disc, &udf, None)
.expect("scan")
.len(),
2,
"fixture must offer two title sets"
);
let mut reader = HaltingReader {
inner: &mut disc,
halt_at: PART_START + 7000, // VTS_02_0.IFO's data extent
};
let res = Disc::scan_dvd_titles(&mut reader, &udf, None);
assert!(
matches!(res, Err(crate::error::Error::Halted)),
"a cancelled title-set read must surface as a cancelled scan, not \
as a disc with fewer titles; got {:?}",
res.map(|ts| ts.iter().map(|t| t.playlist.clone()).collect::<Vec<_>>())
);
// The same cancel one level up: VIDEO_TS.IFO itself. This is the
// `Err(_) => Vec::new()` path — a cancel that used to report a DVD as
// carrying no titles whatsoever.
let mut reader = HaltingReader {
inner: &mut disc,
halt_at: PART_START + 5000,
};
let res = Disc::scan_dvd_titles(&mut reader, &udf, None);
assert!(
matches!(res, Err(crate::error::Error::Halted)),
"a cancelled VMG read must surface as a cancelled scan, not as an \
empty disc; got {:?}",
res.map(|ts| ts.len())
);
}
/// scan_dvd_titles returns empty when VIDEO_TS.IFO can't be parsed
/// (dvd.rs: `parse_vmg(...) Err → return Vec::new()`). Never panics.
#[test]
@@ -535,7 +673,11 @@ mod tests {
let mut disc = MemDisc::new();
// VIDEO_TS exists but VIDEO_TS.IFO is missing.
let udf = build_video_ts_fs(&mut disc, &[]);
assert!(Disc::scan_dvd_titles(&mut disc, &udf).is_empty());
assert!(
Disc::scan_dvd_titles(&mut disc, &udf, None)
.expect("scan")
.is_empty()
);
}
/// Single VTS, single title, one cell. Extent absolute LBA =
@@ -571,7 +713,7 @@ mod tests {
},
],
);
let titles = Disc::scan_dvd_titles(&mut disc, &udf);
let titles = Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan");
assert_eq!(titles.len(), 1);
let t = &titles[0];
assert_eq!(t.extents.len(), 1);
@@ -627,7 +769,7 @@ mod tests {
},
],
);
let titles = Disc::scan_dvd_titles(&mut disc, &udf);
let titles = Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan");
assert_eq!(titles.len(), 1);
let t = &titles[0];
assert_eq!(t.extents.len(), 1);
@@ -685,7 +827,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
assert_eq!(t.extents.len(), 1);
let got = t.extents[0].start_lba;
// The one correct answer: all three terms summed (9000 + 700 + 33).
@@ -743,7 +885,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
assert_eq!(t.extents.len(), 2);
assert_eq!(t.extents[0].start_lba, 9500); // ifo_lba(9000) + 500 + 0
assert_eq!(t.extents[0].sector_count, 100);
@@ -784,7 +926,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
let v = t
.streams
.iter()
@@ -839,7 +981,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
let v = t
.streams
.iter()
@@ -900,7 +1042,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
let audios: Vec<_> = t
.streams
.iter()
@@ -911,7 +1053,7 @@ mod tests {
.collect();
assert_eq!(audios.len(), 2);
assert_eq!(audios[0].codec, Codec::Ac3);
assert_eq!(audios[0].language, "en");
assert_eq!(audios[0].language, "eng");
assert_eq!(audios[1].codec, Codec::Dts);
// Real channel layouts survive the scan (not a 1ch placeholder): the
// AC-3 is 5.1 (6ch), the DTS is 2.0 (2ch).
@@ -970,7 +1112,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
let audios: Vec<_> = t
.streams
.iter()
@@ -1027,7 +1169,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
let subs: Vec<_> = t
.streams
.iter()
@@ -1040,7 +1182,7 @@ mod tests {
// Languages preserved in order.
assert_eq!(
subs.iter().map(|s| s.language.as_str()).collect::<Vec<_>>(),
vec!["en", "fr", "de"]
vec!["eng", "fra", "deu"]
);
// PIDs are 0x20 + ordinal, all distinct.
let pids: Vec<u16> = subs.iter().map(|s| s.pid).collect();
@@ -1087,7 +1229,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
let sub = t
.streams
.iter()
@@ -1097,7 +1239,7 @@ mod tests {
})
.expect("subtitle stream");
assert_eq!(sub.codec, Codec::DvdSub);
assert_eq!(sub.language, "en");
assert_eq!(sub.language, "eng");
assert!(
sub.codec_data.is_some(),
"non-zero palette must yield codec_data"
@@ -1141,7 +1283,7 @@ mod tests {
},
],
);
let titles = Disc::scan_dvd_titles(&mut disc, &udf);
let titles = Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan");
assert_eq!(titles.len(), 2);
// title_number is a running counter across all title sets.
assert_eq!(titles[0].playlist_id, 1);
@@ -1178,7 +1320,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
// One program in the program map → one chapter time (0.0 for the
// first program). Name is the ordinal from chapter_name(0).
assert_eq!(t.chapters.len(), 1);
@@ -1263,7 +1405,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
// The leading 0x90 cell is dropped: 2 feature extents, not 3.
assert_eq!(t.extents.len(), 2, "leading angle sub-block cell dropped");
// First extent starts at the feature cell (vob 1000 + 100), not at 1000+0.
@@ -1315,7 +1457,7 @@ mod tests {
},
],
);
let t = &Disc::scan_dvd_titles(&mut disc, &udf)[0];
let t = &Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan")[0];
// Nothing dropped: both cells become extents, starting at the very head.
assert_eq!(t.extents.len(), 2);
assert_eq!(t.extents[0].start_lba, 9000 + 1000); // ifo_lba + vtstt + 0, head intact
@@ -1358,7 +1500,7 @@ mod tests {
},
],
);
let titles = Disc::scan_dvd_titles(&mut disc, &udf);
let titles = Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan");
let t = &titles[0];
let audio_pids: Vec<u16> = t
.streams
+462 -37
View File
@@ -107,8 +107,10 @@ struct PlannedFile {
size: u64,
/// Inline (ICB-embedded) data, if any. When `Some`, `extents` is empty.
inline: Option<Vec<u8>>,
/// Absolute disc extents `(abs_lba, byte_len)`.
extents: Vec<(u32, u32)>,
/// Absolute disc extents, each carrying whether it was ever RECORDED (an
/// ECMA-167 4/14.14.1.1 type-1 extent is allocated but not recorded: it
/// occupies the file's byte space and its contents are zeros).
extents: Vec<crate::udf::AbsExtent>,
}
impl Disc {
@@ -144,7 +146,7 @@ impl Disc {
let fs = udf::read_filesystem(reader)?;
let mut planned: Vec<PlannedFile> = Vec::new();
let mut dirs: Vec<PathBuf> = Vec::new();
let mut seen_hosts: std::collections::HashMap<PathBuf, String> =
let mut seen_hosts: std::collections::HashMap<String, String> =
std::collections::HashMap::new();
plan_tree(
reader,
@@ -235,6 +237,9 @@ impl Disc {
let mut result = ExtractResult::default();
let total_bytes = required;
let mut done_bytes: u64 = 0;
// Cumulative zero-filled-unreadable bytes, so the live progress channel
// can report the good/unreadable split instead of pinning unreadable at 0.
let mut done_unreadable: u64 = 0;
// CSS per-VTS key cache; only consulted for CSS discs.
let is_css = matches!(base_keys, DecryptKeys::Css { .. });
@@ -254,7 +259,7 @@ impl Disc {
let key = match vts_keys.get(&vts) {
Some(k) => k.clone(),
None => {
let k = self.resolve_vts_key(&vts, &planned, &mut dec, &base_keys);
let k = self.resolve_vts_key(&vts, &planned, &mut dec, &base_keys)?;
vts_keys.insert(vts.clone(), k.clone());
k
}
@@ -269,8 +274,15 @@ impl Disc {
// non-tolerate), so extract_one_file already zero-filled it and
// counted it in bytes_unreadable — one 'lost' bucket covers both
// media damage and decrypt failure.
let (fr, halted) =
extract_one_file(&mut dec, dest, pf, total_bytes, &mut done_bytes, opts)?;
let (fr, halted) = extract_one_file(
&mut dec,
dest,
pf,
total_bytes,
&mut done_bytes,
&mut done_unreadable,
opts,
)?;
result.bytes_good = result.bytes_good.saturating_add(fr.bytes_good);
result.bytes_unreadable = result.bytes_unreadable.saturating_add(fr.bytes_unreadable);
@@ -291,38 +303,115 @@ impl Disc {
/// VOB extents (keyless Stevenson attack). The disc-wide key is reused when
/// it already covers this VTS (single-VTS discs, or this VTS's span). The
/// reader is borrowed from the decrypting decorator (its inner source).
///
/// FALLIBLE, because the three outcomes of a crack are not two.
/// [`crate::css::CrackOutcome`] exists precisely to separate "no scrambled
/// sector was seen, so there is nothing to decrypt" from "scrambled sectors
/// were seen and no key came out", and its doc says callers MUST surface
/// the second as a hard error. This function used the `Option`-returning
/// `crack_key`, which collapses both into `None`, and then fell back to the
/// disc-wide key — descrambling this VTS with ANOTHER VTS's title key. The
/// hazard is described in the ordering note below, which was written about
/// the same fallback; ordering makes the crack far more likely to succeed,
/// but it cannot make a failed crack safe.
fn resolve_vts_key<S: SectorSource>(
&self,
vts: &str,
planned: &[PlannedFile],
dec: &mut DecryptingSectorSource<S>,
base_keys: &DecryptKeys,
) -> DecryptKeys {
) -> Result<DecryptKeys> {
// Gather the title VOB extents for this VTS (VTS_xx_1.VOB .. _9.VOB;
// VTS_xx_0.VOB is the menu and is clear, so excluded from the crack).
// Gather this VTS's title VOBs BY NAME, ascending.
//
// `planned` comes from walking the UDF directory, which yields File
// Identifier Descriptors in on-disc order — authoring order, with no
// guarantee it matches playback order. Relying on it would leave the
// crack starting wherever the disc's directory happened to list first.
//
// DVD-Video numbers a title set's VOBs in playback order by spec
// (VTS_xx_1.VOB .. VTS_xx_9.VOB, single digit), so ascending name IS
// playback order and it is deterministic regardless of directory
// layout. That matters because ORDER decides correctness here: see the
// note below on the shared crack budget.
let mut files: Vec<&PlannedFile> = planned
.iter()
.filter(|pf| {
vts_group_of(&pf.disc_name).as_deref() == Some(vts) && is_title_vob(&pf.disc_name)
})
.collect();
// Case-INSENSITIVE, matching `vts_group_of`/`is_title_vob`, which
// selected these files case-insensitively. A byte-wise sort put
// 'V' (0x56) before 'v' (0x76), so a set holding `vts_01_1.vob`
// alongside `VTS_01_2.VOB` — reachable on a case-sensitive volume —
// started the crack at part 2 and could exhaust the budget in a clear
// run, which is the whole failure this ordering exists to prevent.
files.sort_by_key(|f| f.disc_name.to_ascii_uppercase());
let mut extents: Vec<crate::disc::Extent> = Vec::new();
for pf in planned {
if vts_group_of(&pf.disc_name).as_deref() == Some(vts) && is_title_vob(&pf.disc_name) {
for &(abs_lba, byte_len) in &pf.extents {
extents.push(crate::disc::Extent {
start_lba: abs_lba,
sector_count: (byte_len as u64).div_ceil(SECTOR_BYTES_U64) as u32,
});
}
for pf in files {
// Unrecorded extents hold no bytes the VOB ever wrote, so they can
// carry no scrambled sector for the crack to work from — feeding
// them in spends the shared sector budget on media that is not part
// of the title.
for ext in pf.extents.iter().filter(|e| e.recorded) {
extents.push(crate::disc::Extent {
start_lba: ext.lba,
sector_count: (ext.len as u64).div_ceil(SECTOR_BYTES_U64) as u32,
});
}
}
if extents.is_empty() {
return base_keys.clone();
return Ok(base_keys.clone());
}
// Largest extent first (movie body), matching the scan heuristic.
extents.sort_by_key(|e| std::cmp::Reverse(e.sector_count));
// PLAYBACK ORDER — do NOT sort. This is the 1.5.1 garbage bug, and it
// grew back here in a new code path: the comment this replaced said it
// "matched the scan heuristic", but that heuristic WAS the bug and was
// already fixed in `Disc::decrypt_keys_for_title`, which documents the
// rule ("PLAYBACK ORDER, never largest-cell-first") and pins it with
// `decrypt_keys_for_title_scans_playback_order_not_largest_first`.
//
// Why order decides correctness: a CSS DVD's biggest cell opens with a
// long CLEAR run, and `crack_key`'s sector budget is shared across the
// whole extent list. Starting at the largest cell can exhaust the budget
// without ever MEETING a scrambled sector — and CSS recovers the title
// key from scrambled data itself, so the scan has to reach some. The
// crack then returns None, this function falls back to `base_keys`, and
// every VOB in the VTS is descrambled with the wrong key: corrupt PES
// behind an intact header, written out as a complete extract at exit 0.
//
// The files above are ordered by name and each file's own extents stay
// in file order, which together is playback order for a DVD title set.
// Crack against the raw (still-scrambled) inner reader, NOT the
// decrypting view — `crack_key` runs the descrambler itself.
match crate::css::crack_key(dec.inner_mut(), &extents, 64) {
Some(state) => DecryptKeys::Css {
match crate::css::crack_key_outcome(dec.inner_mut(), &extents, 64, None) {
crate::css::CrackOutcome::Cracked(state) => Ok(DecryptKeys::Css {
title_key: state.title_key,
},
None => base_keys.clone(),
}),
// No scrambled sector anywhere in this VTS: the content is clear,
// and any key descrambles it as a no-op. The disc-wide key is the
// right answer, and this is the ONLY case that ever was.
crate::css::CrackOutcome::Unencrypted => Ok(base_keys.clone()),
// Scrambled sectors WERE seen and no key came out. Reusing the
// disc-wide key here writes corrupt PES behind an intact header and
// reports a complete extract at exit 0.
//
// `CssKeyMissing` is the per-TITLE code rather than the disc-level
// `CssNoDiscKey` because the two are treated differently elsewhere:
// it is what `error::is_skippable_title_stub` matches, and the MUX
// path really does skip a title on it and carry on with the rest of
// the disc.
//
// EXTRACT does not skip. This error is `?`-propagated by the only
// caller, out of the loop over every planned file, so one
// uncrackable VTS aborts the whole extract — including files from
// sibling VTS groups that would have cracked. That is deliberate:
// aborting loudly is the safe failure here, and pinned by
// `a_scrambled_vts_that_cannot_be_cracked_fails_instead_of_borrowing_a_key`.
// Do not read the code choice as a promise that extraction
// continues past this point.
crate::css::CrackOutcome::ScrambledUncracked => Err(Error::CssKeyMissing),
}
}
}
@@ -369,7 +458,7 @@ fn plan_tree(
is_root: bool,
files: &mut Vec<PlannedFile>,
dirs: &mut Vec<PathBuf>,
seen_hosts: &mut std::collections::HashMap<PathBuf, String>,
seen_hosts: &mut std::collections::HashMap<String, String>,
) -> Result<()> {
for entry in &dir.entries {
if entry.name.is_empty() {
@@ -387,13 +476,48 @@ fn plan_tree(
let safe = sanitize_component(&entry.name)?;
let child_rel = host_rel.join(&safe);
let child_disc = format!("{disc_path}/{}", entry.name);
// Collision: two distinct disc paths → same host path.
if let Some(prev) = seen_hosts.insert(child_rel.clone(), child_disc.clone())
&& prev != child_disc
{
return Err(Error::DirNameCollision {
host: child_rel.to_string_lossy().into_owned(),
});
// Collision: two distinct disc paths → same host FILE. The key must
// model the HOST's namespace, not the disc's, and the two differ twice
// over:
//
// * CASE. macOS APFS and Windows NTFS are case-insensitive by default,
// so `Movie` and `movie` are one file there. Keyed by the
// case-preserving path they were two entries, nothing collided, and
// the second extraction overwrote the first with both files still
// reported `complete: true`.
// * The `.partial` SUFFIX. `extract_one_file` streams through
// `<final>.partial` before renaming, so a file's temp path lives in
// the same namespace as every other file's final path: a disc
// holding both `X` and `X.partial` planned two distinct final names,
// and extracting `X` then truncated the real `X.partial`.
//
// Only files get the `.partial` alias — directories are created
// directly and never stream through a temp name — but it is checked
// against every entry's primary key, so a directory `X.partial` beside
// a file `X` is caught too. The alias stores the OWNING file's disc
// path, so the `prev != child_disc` test keeps its meaning: a repeated
// identical disc path is not a collision, two different ones are.
//
// The fold is `to_lowercase`, which closes the ASCII and simple-Unicode
// case classes. It is NOT full case folding, and it does NOT normalize:
// APFS also unifies NFC/NFD, so `é` recorded as U+00E9 and as
// `e`+U+0301 remain two keys here and one file there. That residual is
// a false NEGATIVE (a missed collision, never a spurious one), so this
// is a strict improvement rather than a complete model of the host.
let mut register = |key: PathBuf| -> Result<()> {
let folded = key.to_string_lossy().to_lowercase();
if let Some(prev) = seen_hosts.insert(folded, child_disc.clone())
&& prev != child_disc
{
return Err(Error::DirNameCollision {
host: key.to_string_lossy().into_owned(),
});
}
Ok(())
};
register(child_rel.clone())?;
if !entry.is_dir {
register(with_partial_suffix(&child_rel))?;
}
if entry.is_dir {
dirs.push(child_rel.clone());
@@ -439,6 +563,7 @@ fn extract_one_file<S: SectorSource>(
pf: &PlannedFile,
total_bytes: u64,
done_bytes: &mut u64,
done_unreadable: &mut u64,
opts: &ExtractOptions,
) -> Result<(FileResult, bool)> {
let final_path = dest.join(&pf.host_rel);
@@ -468,16 +593,53 @@ fn extract_one_file<S: SectorSource>(
finalize_file(writer, &partial_path, pf.size, &final_path)?;
fr.complete = true;
*done_bytes = done_bytes.saturating_add(pf.size);
report(opts, *done_bytes, total_bytes);
report(opts, *done_bytes, *done_unreadable, total_bytes);
return Ok((fr, false));
}
let mut written: u64 = 0;
let mut buf = vec![0u8; READ_BATCH_SECTORS as usize * SECTOR_BYTES];
'extents: for &(abs_lba, byte_len) in &pf.extents {
'extents: for &crate::udf::AbsExtent {
lba: abs_lba,
len: byte_len,
recorded,
} in &pf.extents
{
if written >= pf.size {
break;
}
// ECMA-167 4/14.14.1.1 type 1: allocated but NOT recorded. The extent
// is part of the file's byte space and its contents are defined to be
// zeros, so write the zeros WITHOUT reading the media — those sectors
// hold nothing this file ever wrote (on an AACS disc, ciphertext that
// decrypts to noise). This mirrors `UdfFs::read_file_limited`, which
// takes the same decision from the same flag. Skipping the extent
// entirely instead would slide every later extent's bytes down by the
// hole's length.
if !recorded {
let sectors = (byte_len as u64).div_ceil(SECTOR_BYTES_U64);
let hole_bytes = (sectors * SECTOR_BYTES_U64).min(pf.size.saturating_sub(written));
let mut left = hole_bytes;
for b in buf.iter_mut() {
*b = 0;
}
while left > 0 {
let n = left.min(buf.len() as u64) as usize;
write_all(&mut writer, &buf[..n], &partial_path)?;
written = written.saturating_add(n as u64);
*done_bytes = done_bytes.saturating_add(n as u64);
left -= n as u64;
}
fr.bytes_good = fr.bytes_good.saturating_add(hole_bytes);
let cont = report(opts, *done_bytes, *done_unreadable, total_bytes);
if opts.cancelled(cont) {
return Ok((fr, true));
}
if written >= pf.size {
break 'extents;
}
continue;
}
// Anchor AACS unit alignment at THIS extent's start (clip-anchored
// gate, not absolute LBA 0 and NOT the file's first extent). A file
// may span multiple extents (fragmented / Long-AD / continuation ICB
@@ -517,10 +679,11 @@ fn extract_one_file<S: SectorSource>(
}
write_all(&mut writer, &buf[..usable], &partial_path)?;
fr.bytes_unreadable = fr.bytes_unreadable.saturating_add(usable as u64);
*done_unreadable = done_unreadable.saturating_add(usable as u64);
}
written = written.saturating_add(usable as u64);
*done_bytes = done_bytes.saturating_add(usable as u64);
let cont = report(opts, *done_bytes, total_bytes);
let cont = report(opts, *done_bytes, *done_unreadable, total_bytes);
sector_off += batch;
if opts.cancelled(cont) {
// Leave the `.partial`; do NOT rename. The aggregate run
@@ -633,15 +796,20 @@ fn finalize_file(
/// Emit a progress report. Returns `true` to continue, `false` if the sink
/// requested an early stop (or there is no sink — always continue).
fn report(opts: &ExtractOptions, done: u64, total: u64) -> bool {
fn report(opts: &ExtractOptions, done: u64, unreadable: u64, total: u64) -> bool {
match opts.progress {
Some(p) => {
let pp = crate::progress::PassProgress {
kind: crate::progress::PassKind::Mux,
work_done: done,
work_total: total,
bytes_good_total: done,
bytes_unreadable_total: 0,
// `done` counts good AND zero-filled-unreadable bytes together;
// split them so a consumer driven only by the live progress
// channel sees a holed extraction as holed rather than as a clean
// climb to 100%. The final ExtractResult already carries the true
// split — this used to pin unreadable at 0 and call every byte good.
bytes_good_total: done.saturating_sub(unreadable),
bytes_unreadable_total: unreadable,
bytes_pending_total: 0,
bytes_retryable_total: 0,
bytes_total_disc: total,
@@ -684,7 +852,53 @@ fn available_space(dir: &Path) -> Option<u64> {
Some(avail)
}
#[cfg(not(unix))]
/// Windows has no `statvfs`. This returned `None` unconditionally, which does
/// not merely skip a test — it skipped the free-space GATE, so a Windows user
/// extracting a disc to a full volume got a confusing failure part-way through
/// instead of a clear refusal up front. libfreemkv ships a Windows GUI, so that
/// is the platform where the friendly error matters most.
///
/// `GetDiskFreeSpaceExW` is declared directly against kernel32, matching how
/// `scsi::windows` already reaches the Win32 API rather than pulling in a
/// binding crate for one call.
#[cfg(windows)]
fn available_space(dir: &Path) -> Option<u64> {
use std::os::windows::ffi::OsStrExt;
unsafe extern "system" {
fn GetDiskFreeSpaceExW(
lpDirectoryName: *const u16,
lpFreeBytesAvailableToCaller: *mut u64,
lpTotalNumberOfBytes: *mut u64,
lpTotalNumberOfFreeBytes: *mut u64,
) -> i32;
}
// Wide, NUL-terminated. An interior NUL cannot reach the API, so reject it
// rather than silently truncating the path and measuring the wrong volume.
let mut wide: Vec<u16> = dir.as_os_str().encode_wide().collect();
if wide.contains(&0) {
return None;
}
wide.push(0);
let mut avail: u64 = 0;
// FreeBytesAvailableToCaller, not TotalNumberOfFreeBytes: it accounts for
// per-user quotas, which is what "can I actually write this much" means and
// what `statvfs`'s `f_bavail` gives on the unix side.
let ok = unsafe {
GetDiskFreeSpaceExW(
wide.as_ptr(),
&mut avail,
std::ptr::null_mut(),
std::ptr::null_mut(),
)
};
if ok == 0 { None } else { Some(avail) }
}
/// Neither unix nor Windows: no way to ask, so the gate is skipped.
#[cfg(not(any(unix, windows)))]
fn available_space(_dir: &Path) -> Option<u64> {
None
}
@@ -1079,6 +1293,17 @@ mod tests {
s
}
/// Build a file ICB whose FIRST short AD is an ECMA-167 4/14.14.1.1 type-1
/// (allocated, NOT recorded) extent and whose second is ordinary recorded
/// data. Both are `sectors_each` sectors long.
fn build_hole_then_data_icb(sectors_each: u32, hole_lba: u32, data_lba: u32) -> [u8; 2048] {
let mut s = build_two_extent_icb(sectors_each, hole_lba, data_lba);
let len = sectors_each * SECTOR_BYTES as u32;
// Re-stamp AD #0 with extent type 1 in bits 30..31 of the length field.
s[216..220].copy_from_slice(&(0x4000_0000u32 | (len & 0x3FFF_FFFF)).to_le_bytes());
s
}
/// Encrypt the clear unit from `clear_aacs_unit(tag)` under `unit_key` so
/// `aacs::content::decrypt_unit` recovers it cleanly (zero decrypt loss).
/// `tag` distinguishes two units' payloads.
@@ -1288,6 +1513,9 @@ mod tests {
// mirroring css::mod tests' `crackable_sector`.
let seed = [0x11u8, 0x22, 0x33, 0x44, 0x55];
let mut plain = vec![0u8; 2048];
// Pack start code: a real scrambled sector is an MPEG-2 PS pack, and
// the descrambler requires it before trusting byte 0x14.
plain[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
plain[0x14] = 0x10; // scramble flag
let pat: Vec<u8> = (0..8)
.map(|k| (0xA0u8.wrapping_add(k as u8)) ^ 0x5A)
@@ -1489,6 +1717,62 @@ mod tests {
assert!(matches!(err, Error::DirNameCollision { .. }));
}
/// Two disc names differing only by CASE are one host file on macOS APFS
/// and Windows NTFS, both case-insensitive by default. Keyed by the
/// case-preserving path, the collision map sees two entries, raises
/// nothing, and the second file extracted overwrites the first — while both
/// `PlannedFile`s report `complete: true`. Silent data loss reported as a
/// clean extract is the one outcome this crate must never produce, so the
/// host-equivalence key has to model the host's namespace, not the disc's.
#[test]
fn names_differing_only_by_case_are_a_collision() {
let root = DirSpec {
name: String::new(),
icb_lba: 10,
dir_data_lba: 11,
files: vec![
file("Movie", 30, 31, b"a".to_vec(), false),
file("movie", 32, 33, b"b".to_vec(), false),
],
subdirs: vec![],
};
let mut disc = build_disc(root);
let out = TmpDir::new("case_collision");
let err = clear_disc()
.extract_tree(&mut disc, out.path(), &ExtractOptions::default())
.expect_err("two names that fold to one host file must collide");
assert!(matches!(err, Error::DirNameCollision { .. }), "got {err:?}");
}
/// A file's in-flight `.partial` path shares the host namespace with every
/// other planned file's FINAL path. A disc carrying both `X` and
/// `X.partial` plans two distinct final names, so nothing collides — but
/// extracting `X` writes through `X.partial`, the same host path the other
/// file owns. Whichever lands second truncates the other, and both entries
/// are still reported complete.
#[test]
fn a_files_partial_path_colliding_with_another_files_final_name_is_an_error() {
let root = DirSpec {
name: String::new(),
icb_lba: 10,
dir_data_lba: 11,
files: vec![
file("X", 30, 31, b"aaaa".to_vec(), false),
file("X.partial", 32, 33, b"bbbb".to_vec(), false),
],
subdirs: vec![],
};
let mut disc = build_disc(root);
let out = TmpDir::new("partial_collision");
let err = clear_disc()
.extract_tree(&mut disc, out.path(), &ExtractOptions::default())
.expect_err(
"X's temp path IS X.partial's final path — one host file for two \
disc files, so it must be refused up front",
);
assert!(matches!(err, Error::DirNameCollision { .. }), "got {err:?}");
}
/// A non-empty target dir is refused without `--force`, and accepted with.
#[test]
fn non_empty_target_requires_force() {
@@ -1618,6 +1902,63 @@ mod tests {
);
}
/// An ECMA-167 4/14.14.1.1 type-1 extent is ALLOCATED BUT NOT RECORDED:
/// the space belongs to the file and occupies its byte range, but nothing
/// was ever written there, and the standard defines its contents as zeros.
/// `read_icb_extents` keeps the flag (`IcbExtent::recorded`) and
/// `read_file_limited` honours it by emitting zeros WITHOUT touching the
/// media. The tree extractor reads the same ICBs and must agree: reading
/// those sectors returns whatever the media happens to hold there — on an
/// AACS disc, ciphertext that decrypts to noise — and writes it into the
/// extracted file as if the disc had recorded it.
///
/// The hole here is filled with a recognisable non-zero pattern, so
/// "emitted zeros" and "read the media" are told apart by CONTENT.
#[test]
fn extract_tree_zero_fills_an_unrecorded_extent_instead_of_reading_it() {
const SECTORS_EACH: u32 = 1;
const HOLE: u32 = 5000;
const DATA: u32 = 5004;
let hole_bytes = vec![0xEEu8; SECTOR_BYTES];
let data_bytes = vec![0x5Au8; SECTOR_BYTES];
// The file's byte space: the hole's zeros FIRST, then the real data.
let mut expect = vec![0u8; SECTOR_BYTES];
expect.extend_from_slice(&data_bytes);
let mut disc = MemDisc::new();
build_udf_skeleton(&mut disc, 10);
let mut root_fids = Vec::new();
push_fid(&mut root_fids, "", 10, true, true);
push_fid(&mut root_fids, "INDEX.BDMV", 42, false, false);
disc.put(
PART_START + 42,
build_hole_then_data_icb(SECTORS_EACH, HOLE, DATA),
);
disc.put_bytes(PART_START + HOLE, &hole_bytes);
disc.put_bytes(PART_START + DATA, &data_bytes);
disc.put(PART_START + 10, build_dir_icb(11, root_fids.len() as u32));
disc.put_bytes(PART_START + 11, &root_fids);
let out = TmpDir::new("unrecorded_extent");
let res = clear_disc()
.extract_tree(&mut disc, out.path(), &ExtractOptions::default())
.expect("extract");
let got = read_out(out.path(), "INDEX.BDMV").expect("file written");
assert_eq!(
got, expect,
"an unrecorded extent contributes zeros to the file, never the \
bytes that happen to sit on those sectors"
);
assert_eq!(
res.bytes_unreadable, 0,
"a hole is not a read failure — nothing was attempted"
);
assert!(res.complete);
}
/// Focused alignment-computation check underpinning the per-extent fix:
/// when each extent anchors its OWN start as the unit base, the extent's
/// own batch starts are always unit-aligned; anchoring a later extent
@@ -1759,6 +2100,9 @@ mod tests {
// VTS groups regardless of which extents were gathered, masking
// this exact regression.
plain[0x00..0x04].copy_from_slice(&crate::css::PACK_START);
// Pack start code: a real scrambled sector is an MPEG-2 PS pack, and
// the descrambler requires it before trusting byte 0x14.
plain[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
plain[0x14] = 0x10; // scramble flag
let pat: Vec<u8> = (0..8)
.map(|k| (0xA0u8.wrapping_add(k as u8) ^ marker) ^ 0x5A)
@@ -1826,6 +2170,87 @@ mod tests {
assert!(res.complete);
}
/// A VTS that IS scrambled but whose key could not be recovered must
/// FAIL, not borrow another VTS's key.
///
/// `crack_key` returns `Option`, which collapses "no scrambled sector was
/// seen" with "scrambled sectors were seen and no key came out"; the
/// fallback then descrambled this VTS under the disc-wide key and
/// `extract_tree` reported `complete = true` at exit 0, because no read had
/// failed. `CrackOutcome` exists to keep those two apart, and every sibling
/// path in the crate already honours it.
///
/// The fixture: VTS_01 is crackable, VTS_02 is scrambled with NO periodic
/// crib, so its crack genuinely fails.
#[test]
fn a_scrambled_vts_that_cannot_be_cracked_fails_instead_of_borrowing_a_key() {
let key_1 = [0x10u8, 0x20, 0x30, 0x40, 0x50];
let (_plain_1, scrambled_1) = {
let seed = [0x11u8, 0x22, 0x33, 0x44, 0x01];
let mut plain = vec![0u8; 2048];
plain[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
plain[0x14] = 0x10;
let pat: Vec<u8> = (0..8)
.map(|k| (0xA0u8.wrapping_add(k as u8) ^ 0x01) ^ 0x5A)
.collect();
for (i, b) in plain.iter_mut().enumerate().skip(0x59) {
*b = pat[i % 8];
}
plain[0x54..0x59].copy_from_slice(&seed);
let mut scrambled = plain.clone();
lfsr::scramble_sector(&key_1, &mut scrambled);
(plain, scrambled)
};
// Scrambled — the pack header and the 0x14 flag are set, so the scan
// SEES ciphertext — but the cleartext region carries no periodic run,
// so the Stevenson crib never forms and no key is recoverable.
let uncrackable = {
let mut sect = vec![0u8; 2048];
sect[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
sect[0x14] = 0x10;
for (i, b) in sect.iter_mut().enumerate().skip(0x59) {
// Non-repeating, so no run of any period survives to 0x80.
*b = (i as u8).wrapping_mul(37).wrapping_add(11);
}
sect
};
let root = DirSpec {
name: String::new(),
icb_lba: 10,
dir_data_lba: 11,
files: Vec::new(),
subdirs: vec![DirSpec {
name: "VIDEO_TS".to_string(),
icb_lba: 20,
dir_data_lba: 21,
files: vec![
file("VTS_01_1.VOB", 30, 5000, scrambled_1, false),
file("VTS_02_1.VOB", 32, 6000, uncrackable, false),
],
subdirs: vec![],
}],
};
let mut disc = build_disc(root);
let out = TmpDir::new("css_uncrackable_vts");
let mut d = clear_disc();
d.content_format = crate::disc::ContentFormat::MpegPs;
d.css = Some(crate::css::CssState {
title_key: [0xFFu8; 5],
crack_span: None,
});
let err = d
.extract_tree(&mut disc, out.path(), &ExtractOptions::default())
.expect_err(
"a VTS whose key could not be recovered must be a hard error, \
not a silent extract under another VTS's key",
);
assert!(
matches!(err, Error::CssKeyMissing),
"expected CssKeyMissing, got {err:?}"
);
}
/// `Borrowed` is a thin forwarding wrapper the decrypting decorator uses
/// to avoid taking ownership of the caller's reader -- every
/// `SectorSource` method must forward to the wrapped `&mut dyn
+1777 -52
View File
File diff suppressed because it is too large Load Diff
+354 -17
View File
@@ -173,6 +173,17 @@ pub struct Clip {
pub duration_secs: f64,
/// Source packet count (from CLPI, 0 if unavailable)
pub source_packets: u32,
/// Byte range this clip's stream occupies within the TITLE'S FEED — the
/// concatenation of the title's extents, in the order the mux reads them.
/// `None` when it could not be determined.
///
/// This is PROVENANCE, and it is what makes clip assignment a lookup
/// instead of a guess. During an overlap two clips' mark ranges both
/// contain the same timestamp, so no rule over timestamps alone can say
/// which clip a frame came from — four audit rounds each fixed one such
/// rule and broke another. A frame carries its source byte offset
/// (`PesFrame::source`), and that offset falls in exactly one clip's span.
pub feed_span: Option<(u64, u64)>,
}
/// A stream within a title.
@@ -468,6 +479,25 @@ pub struct Extent {
/// Union a set of extents into sorted, merged, disjoint `(start_lba,
/// sector_count)` ranges — the pure, testable core of
/// [`Disc::encrypted_content_ranges`]. Reuses [`crate::udf::merge_ranges`].
/// Is this disc structurally AACS-encrypted — i.e. does it carry an AACS
/// directory?
///
/// THE ONE definition of that question. `Disc::identify` (the fast path, which
/// only reads the filesystem) and `Disc::scan_with` (the full scan) both need
/// it, and both used to spell the same two `find_dir` calls out by hand. They
/// agreed today; nothing made them agree tomorrow. Adding a third AACS
/// location, or excluding an empty placeholder directory, to one copy and not
/// the other would silently desync the fast identify from the full scan — the
/// same disc reported encrypted by one and clear by the other.
///
/// This is STRUCTURAL, not cryptographic: it says the tree looks like an
/// encrypted disc, not that any sector actually is. A folder copied verbatim
/// from a decrypted disc keeps its `AACS/` and answers true here (see
/// `session::scan_dir`, which corrects for exactly that).
pub(crate) fn aacs_dir_present(udf_fs: &crate::udf::UdfFs) -> bool {
udf_fs.find_dir("/AACS").is_some() || udf_fs.find_dir("/BDMV/AACS").is_some()
}
fn merged_extents<'a>(extents: impl Iterator<Item = &'a Extent>) -> Vec<(u32, u32)> {
let mut ranges: Vec<(u32, u32)> = extents.map(|e| (e.start_lba, e.sector_count)).collect();
ranges.sort_by_key(|r| r.0);
@@ -1617,8 +1647,7 @@ impl Disc {
// (no titles needed: BD/UHD/FMTS come from the MKB generation). It no
// longer defaults to BluRay or defers UHD/FMTS to the full scan.
let format = Self::detect_disc_format(&mut buffered, &udf_fs, &[]);
let encrypted =
udf_fs.find_dir("/AACS").is_some() || udf_fs.find_dir("/BDMV/AACS").is_some();
let encrypted = aacs_dir_present(&udf_fs);
let layers = if capacity > 24_000_000 { 2 } else { 1 };
Ok(DiscId {
@@ -1961,6 +1990,21 @@ impl Disc {
Self::read_aacs_inputs_from_reader(&mut reader, &udf_fs)
}
/// Same as [`Disc::read_aacs_inputs`] but over an extracted disc FOLDER.
///
/// `dir://` is an image-level source: `dirimage` synthesizes a real UDF
/// volume over the folder, so the AACS inputs are read by exactly the same
/// reader path an ISO uses. Without this the online key fetch was silently
/// unavailable for folders — the CLI's fetch helper matched only `Iso` and
/// returned None for a folder, so the same disc that fetched its key fine
/// as an ISO failed as an extracted directory. A sink is a sink: any input
/// has to work with any output, and that includes the key path.
pub fn read_aacs_inputs_from_dir(dir: &std::path::Path) -> Result<(Vec<u8>, Vec<u8>, u8)> {
let mut reader = crate::dirimage::DirImage::open(dir)?;
let udf_fs = udf::read_filesystem(&mut reader)?;
Self::read_aacs_inputs_from_reader(&mut reader, &udf_fs)
}
/// Same as [`Disc::read_aacs_inputs`] but reads from a live drive. The
/// out-of-band Unit Key path fetches the disc's key files from the drive,
/// resolves a key from them however it likes, then applies it via
@@ -1989,8 +2033,7 @@ impl Disc {
let scan_with_t0 = std::time::Instant::now();
tracing::info!(target: "freemkv::scan", phase = "scan_with", "begin");
// 2. Resolve encryption (AACS, CSS, or none)
let encrypted =
udf_fs.find_dir("/AACS").is_some() || udf_fs.find_dir("/BDMV/AACS").is_some();
let encrypted = aacs_dir_present(&udf_fs);
let (aacs, aacs_error) = if !encrypted {
(None, None)
@@ -2013,17 +2056,17 @@ impl Disc {
// derived below from the AACS MKB generation, not the tree.
let (mut titles, content_format) = if udf_fs.find_dir("/BDMV").is_some() {
(
Self::scan_bluray_titles(reader, &udf_fs),
Self::scan_bluray_titles(reader, &udf_fs, opts.halt.as_ref())?,
ContentFormat::BdTs,
)
} else if udf_fs.find_dir("/HVDVD_TS").is_some() {
(
Self::scan_hddvd_titles(reader, &udf_fs),
Self::scan_hddvd_titles(reader, &udf_fs, opts.halt.as_ref())?,
ContentFormat::MpegPs,
)
} else if udf_fs.find_dir("/VIDEO_TS").is_some() {
(
Self::scan_dvd_titles(reader, &udf_fs),
Self::scan_dvd_titles(reader, &udf_fs, opts.halt.as_ref())?,
ContentFormat::MpegPs,
)
} else {
@@ -2144,7 +2187,9 @@ impl Disc {
/// **Sort priority (titles[0] = most likely main feature):**
/// 1. Real titles (`size_bytes ≤ capacity_bytes`) before virtual
/// composites. The capacity check is a hard "physically
/// possible data on this disc" gate.
/// possible data on this disc" gate. `capacity_bytes == 0` means
/// the capacity is UNKNOWN (READ CAPACITY failed), so the gate is
/// skipped entirely rather than demoting every real title.
/// 2. Among real titles, LARGEST physical size first — the main
/// feature is the biggest real title on the disc. (This replaced
/// the old clip-count ordering, which mis-ranked chapter-per-clip
@@ -2206,8 +2251,18 @@ impl Disc {
// A title bigger than the whole disc is a "play-all" composite artifact
// (its declared size double-counts clips shared with other playlists) —
// demote it below any real single title.
let a_oversize = a.size_bytes > capacity_bytes;
let b_oversize = b.size_bytes > capacity_bytes;
//
// `capacity_bytes == 0` means the capacity is UNKNOWN, not that the
// disc holds nothing: `read_udf` substitutes 0 when READ CAPACITY
// fails and scans on regardless. Applied literally the gate would
// INVERT there — every real title (`size_bytes > 0`) would be
// "oversize" and demoted, while a CLPI-less `size_bytes == 0` title
// would not, landing at `titles[0]` ahead of the feature. With no
// capacity to compare against, the gate is inert and the size /
// duration / audio keys decide the order on their own.
let capacity_known = capacity_bytes > 0;
let a_oversize = capacity_known && a.size_bytes > capacity_bytes;
let b_oversize = capacity_known && b.size_bytes > capacity_bytes;
a_oversize
.cmp(&b_oversize)
// PRIMARY: largest physical size = the main feature. Robust where
@@ -3532,8 +3587,8 @@ mod tests {
) -> (crate::udf::fixture::MemDisc, udf::UdfFs) {
use crate::udf::fixture::*;
let files = vec![
file("MAIN.EVO", 100, 5_000, main_bytes, true),
file("OTHER.EVO", 101, 50_000, other_bytes, true),
file("MAIN.EVO", 100, 5_000, main_bytes as u64, true),
file("OTHER.EVO", 101, 50_000, other_bytes as u64, true),
];
let root = DirSpec {
name: String::new(),
@@ -3555,6 +3610,237 @@ mod tests {
(disc, udf)
}
/// A cancelled [`crate::halt::Halt`] must stop the HD-DVD title scan.
///
/// The scan is bounded but big — up to `MAX_HDDVD_CLIPS` clips, each
/// costing an ICB resolve plus a 16 MiB `EVO_PROBE_SECTORS` stream probe —
/// so on a live drive an operator Stop that only takes effect after the
/// whole enumerator returns is no Stop at all. `ScanOptions::halt` is
/// already honoured by the CSS crack and the forced-subtitle probe; the
/// title enumerator must honour it too.
///
/// It must also not report a HALF-ENUMERATED disc as a successful scan:
/// a truncated title list is indistinguishable from a disc that genuinely
/// holds fewer titles.
#[test]
fn scan_with_cancelled_halt_stops_the_hddvd_title_scan() {
use crate::udf::fixture::PART_START;
/// Counts reads that land on CLIP DATA (at or past the first clip's
/// data extent) — i.e. the per-clip stream probing, the expensive part
/// of the scan. Everything below that is filesystem metadata.
struct CountingReader<'a> {
inner: &'a mut crate::udf::fixture::MemDisc,
clip_reads: usize,
}
impl SectorSource for CountingReader<'_> {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
if lba >= PART_START + 5_000 {
self.clip_reads += 1;
}
self.inner.read_sectors(lba, count, buf, recovery)
}
}
let (mut disc, udf) = hddvd_two_clip_disc(3_000_000, 5_000_000);
let halt = crate::halt::Halt::new();
halt.cancel();
let opts = ScanOptions {
halt: Some(halt),
..Default::default()
};
let mut reader = CountingReader {
inner: &mut disc,
clip_reads: 0,
};
let res = Disc::scan_with(&mut reader, 3_997_952, None, None, &opts, udf);
let clip_reads = reader.clip_reads;
assert!(
matches!(res, Err(Error::Halted)),
"a cancelled scan must say so, not return a partial title list as \
a completed scan; got {:?}",
res.map(|d| d.titles.len())
);
assert_eq!(
clip_reads, 0,
"cancellation must be observed before the per-clip stream probes, \
not after all of them"
);
}
/// `scan_with` must hand `ScanOptions::halt` to the BLU-RAY enumerator.
///
/// Every BD and DVD cancellation test in this crate calls
/// `scan_bluray_titles` / `scan_dvd_titles` DIRECTLY, so the three-line
/// wiring in `scan_with` that connects the operator's Stop to them was
/// covered by nothing at all: replacing `opts.halt.as_ref()` with `None`
/// on either branch left the whole suite green while a Stop during a BD
/// or DVD scan silently did nothing. Only the HD-DVD branch was pinned.
///
/// The fixture is deliberately the smallest disc that takes the BD branch
/// — a bare `/BDMV` with no PLAYLIST — because the point under test is the
/// ARGUMENT, not the enumerator's own (separately tested) halt polling.
/// Nothing between `scan_with`'s entry and the enumerator reads the flag
/// (the disc is unencrypted, so the AACS path is skipped), so a green here
/// can only mean the flag arrived.
///
/// Mutation: `Self::scan_bluray_titles(reader, &udf_fs, None)?` fails here.
#[test]
fn scan_with_passes_the_halt_flag_to_the_bluray_enumerator() {
use crate::udf::fixture::*;
let mut disc = MemDisc::new();
let root = DirSpec {
name: String::new(),
icb_lba: 10,
dir_data_lba: 11,
files: Vec::new(),
subdirs: vec![DirSpec {
name: "BDMV".into(),
icb_lba: 12,
dir_data_lba: 13,
files: Vec::new(),
subdirs: vec![],
}],
};
build_udf_skeleton(&mut disc, 10);
lay_dir(&mut disc, &root);
let udf = crate::udf::read_filesystem(&mut disc).expect("fs");
// Sanity: the same disc scans clean when nothing is cancelled, so a
// pass below cannot be some unrelated failure wearing Halted.
assert!(
Disc::scan_with(&mut disc, 500_000, None, None, &ScanOptions::default(), udf).is_ok(),
"fixture must scan successfully when not cancelled"
);
let udf = crate::udf::read_filesystem(&mut disc).expect("fs");
let halt = crate::halt::Halt::new();
halt.cancel();
let opts = ScanOptions {
halt: Some(halt),
..Default::default()
};
let res = Disc::scan_with(&mut disc, 500_000, None, None, &opts, udf);
assert!(
matches!(res, Err(Error::Halted)),
"a cancelled BD scan must say so; returning a title list built \
after Stop reports a truncated enumeration as a completed one. \
Got {:?}",
res.map(|d| d.titles.len())
);
}
/// The DVD half of the same wiring, and the same reasoning.
///
/// Mutation: `Self::scan_dvd_titles(reader, &udf_fs, None)?` fails here.
#[test]
fn scan_with_passes_the_halt_flag_to_the_dvd_enumerator() {
use crate::udf::fixture::*;
let mut disc = MemDisc::new();
let root = DirSpec {
name: String::new(),
icb_lba: 10,
dir_data_lba: 11,
files: Vec::new(),
subdirs: vec![DirSpec {
name: "VIDEO_TS".into(),
icb_lba: 50,
dir_data_lba: 51,
files: vec![
file_with("VIDEO_TS.IFO", 60, 5000, dvd_vmg_bytes(), true),
file_with("VTS_01_0.IFO", 62, 6000, dvd_vts_bytes(1000, 10, 10), true),
],
subdirs: vec![],
}],
};
build_udf_skeleton(&mut disc, 10);
lay_dir(&mut disc, &root);
let udf = crate::udf::read_filesystem(&mut disc).expect("fs");
assert!(
Disc::scan_with(&mut disc, 500_000, None, None, &ScanOptions::default(), udf).is_ok(),
"fixture must scan successfully when not cancelled"
);
let udf = crate::udf::read_filesystem(&mut disc).expect("fs");
let halt = crate::halt::Halt::new();
halt.cancel();
let opts = ScanOptions {
halt: Some(halt),
..Default::default()
};
let res = Disc::scan_with(&mut disc, 500_000, None, None, &opts, udf);
assert!(
matches!(res, Err(Error::Halted)),
"a cancelled DVD scan must say so, not hand back whatever it had \
enumerated so far as a finished scan. Got {:?}",
res.map(|d| d.titles.len())
);
}
/// A Stop on a LIVE DRIVE never touches `ScanOptions::halt`: `Drive` has
/// its own flag and `checked_exec` fails every SCSI command with
/// [`Error::Halted`] once it is set. The HD-DVD enumerator must not
/// swallow that into a successful scan.
///
/// Measured before this was fixed: the scan returned `Ok` with both
/// titles present and ZERO streams on each — a cancelled scan wearing the
/// shape of a disc whose clips carry no video or audio. Downstream that is
/// a title list to cache, display and rip from.
#[test]
fn halted_reads_do_not_report_the_hddvd_scan_as_successful() {
use crate::udf::fixture::PART_START;
/// Fails clip-data reads the way a live drive does once Stop is
/// pressed; filesystem metadata below the first clip still resolves,
/// so the scan gets far enough to enumerate titles.
struct HaltingReader<'a> {
inner: &'a mut crate::udf::fixture::MemDisc,
}
impl SectorSource for HaltingReader<'_> {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
if lba >= PART_START + 5_000 {
return Err(Error::Halted);
}
self.inner.read_sectors(lba, count, buf, recovery)
}
}
let (mut disc, udf) = hddvd_two_clip_disc(3_000_000, 5_000_000);
let mut reader = HaltingReader { inner: &mut disc };
let res = Disc::scan_with(
&mut reader,
3_997_952,
None,
None,
&ScanOptions::default(),
udf,
);
assert!(
matches!(res, Err(Error::Halted)),
"reads cancelled by the drive's own halt flag must surface as a \
cancelled scan, not as titles that merely look stream-less; got \
{:?}",
res.map(|d| d
.titles
.iter()
.map(|t| (t.playlist.clone(), t.streams.len()))
.collect::<Vec<_>>())
);
}
/// `scan_with`'s `capacity_bytes = capacity as u64 * 2048` feeds
/// `canonical_title_order`'s "bigger than the whole disc = play-all
/// composite" threshold. Chosen so `capacity * 2048` clears both titles
@@ -3994,6 +4280,7 @@ mod tests {
t.size_bytes = size_bytes;
t.clips = (0..n_clips)
.map(|i| Clip {
feed_span: None,
clip_id: format!("{i:05}"),
in_time: 0,
out_time: 1,
@@ -4394,6 +4681,7 @@ mod tests {
size_bytes,
clips: (0..n_clips)
.map(|i| Clip {
feed_span: None,
clip_id: format!("{i:05}"),
in_time: 0,
out_time: 0,
@@ -5371,7 +5659,7 @@ mod tests {
use crate::aacs::content::ALIGNED_UNIT_LEN;
use crate::aacs::crypto::AACS_IV;
use aes::Aes128;
use aes::cipher::{BlockEncrypt, KeyInit, generic_array::GenericArray};
use aes::cipher::{Array, BlockCipherEncrypt, KeyInit};
let mut unit = clear[..ALIGNED_UNIT_LEN].to_vec();
// Flag the unit encrypted (CPI bits on byte 0) before key derivation so
// the recovered plaintext header matches and `decrypt_unit`'s CPI gate
@@ -5379,14 +5667,14 @@ mod tests {
unit[0] |= 0xC0;
let mut header = [0u8; 16];
header.copy_from_slice(&unit[..16]);
let cipher = Aes128::new(GenericArray::from_slice(uk));
let mut blk = GenericArray::clone_from_slice(&header);
let cipher = Aes128::new(&(*uk).into());
let mut blk: Array<u8, _> = header.into();
cipher.encrypt_block(&mut blk);
let mut dk = [0u8; 16];
for i in 0..16 {
dk[i] = blk[i] ^ header[i];
}
let bc = Aes128::new(GenericArray::from_slice(&dk));
let bc = Aes128::new(&dk.into());
let mut prev = AACS_IV;
let mut i = 16;
while i + 16 <= ALIGNED_UNIT_LEN {
@@ -5394,7 +5682,7 @@ mod tests {
for j in 0..16 {
b[j] = unit[i + j] ^ prev[j];
}
let mut g = GenericArray::clone_from_slice(&b);
let mut g: Array<u8, _> = b.into();
bc.encrypt_block(&mut g);
for j in 0..16 {
unit[i + j] = g[j];
@@ -6772,6 +7060,55 @@ mod tests {
);
}
/// `capacity_bytes == 0` means the disc capacity is UNKNOWN — `read_udf`
/// substitutes 0 when READ CAPACITY fails (transient spin-up, SCSI error)
/// and proceeds with the scan. It does NOT mean "the disc holds nothing".
///
/// With a literal reading of the gate, 0 inverts it: EVERY real title is
/// `size_bytes > 0` and therefore "oversize", while a title with no CLPI
/// (`size_bytes == 0`) is not — so the empty title sorts to `titles[0]`
/// ahead of the feature and `freemkv -t 1` rips garbage. The gate must be
/// INERT when the capacity is unknown.
#[test]
fn canonical_order_unknown_capacity_does_not_demote_every_real_title() {
const UNKNOWN: u64 = 0; // READ CAPACITY failed
let feature = title_with("00800.mpls", 7_320.0, 57_200_000_000, 1);
// A playlist whose CLPI files are missing/unparseable: no declared size.
let sizeless = title_with("00001.mpls", 120.0, 0, 1);
let mut titles = [sizeless, feature];
titles.sort_by(|a, b| Disc::canonical_title_order(a, b, UNKNOWN));
assert_eq!(
titles[0].playlist, "00800.mpls",
"with an UNKNOWN capacity the real feature must still sort first; \
a size-0 title must not be promoted ahead of it"
);
assert_eq!(titles[1].playlist, "00001.mpls");
}
/// Control for [`canonical_order_unknown_capacity_does_not_demote_every_real_title`]:
/// making the gate inert on an UNKNOWN capacity must not make it dead. With
/// a KNOWN capacity a genuinely oversize play-all composite is still demoted
/// below a smaller real title — even though "largest size first" would
/// otherwise rank it first. Asserted on the comparator in both argument
/// orders so an inconsistent comparator cannot pass.
#[test]
fn canonical_order_known_capacity_still_demotes_a_genuinely_oversize_title() {
use std::cmp::Ordering;
const CAP: u64 = 58_500_000_000;
let composite = title_with("00020.mpls", 15_180.0, 92_400_000_000, 253);
let real = title_with("00800.mpls", 7_320.0, 57_200_000_000, 1);
assert_eq!(
Disc::canonical_title_order(&real, &composite, CAP),
Ordering::Less,
"a known capacity must still demote the oversize composite"
);
assert_eq!(
Disc::canonical_title_order(&composite, &real, CAP),
Ordering::Greater,
"…in either argument order"
);
}
// ── audio_richness: the same-size / same-duration tiebreak ─────────────
/// A title carrying the given audio tracks, with size and duration fixed so
+433 -59
View File
@@ -8,6 +8,17 @@ pub fn extract_scsi_context(e: &Error) -> (u8, Option<crate::scsi::ScsiSense>) {
match e {
Error::ScsiError { status, sense, .. } => (*status, *sense),
Error::DiscRead { status, sense, .. } => (status.unwrap_or(0), *sense),
// A failed `ioctl(SG_IO)` and a vanished device never produced a SCSI
// reply at all — they are dead-bus faults, not recoverable bad sectors.
// `Error::is_scsi_transport_failure` (error.rs) already declares both
// variants transport failures so sweep / patch / fill_extents abort the
// pass instead of zero-filling against a wedged device; callers that
// flatten an error through here (Drive::read_one, DiscStream::fill_extents,
// freemkv-engine's recovery sweep/patch) would otherwise collapse them to
// status 0 and silently destroy that classification.
Error::IoError { .. } | Error::DeviceNotFound { .. } => {
(crate::scsi::SCSI_STATUS_TRANSPORT_FAILURE, None)
}
_ => (0, None),
}
}
@@ -24,6 +35,15 @@ pub(crate) mod macos;
#[cfg(windows)]
pub(crate) mod windows;
// Pick the platform module ONCE, here, so the cross-platform entry points below
// dispatch through `platform::…` with no per-function `#[cfg]` in their bodies.
#[cfg(target_os = "linux")]
pub(crate) use linux as platform;
#[cfg(target_os = "macos")]
pub(crate) use macos as platform;
#[cfg(windows)]
pub(crate) use windows as platform;
use crate::error::{Error, Result};
use crate::event::Event;
use crate::identity::DriveId;
@@ -279,22 +299,33 @@ impl Drive {
for attempt in 0..60u64 {
hb.tick(attempt, 60);
let mut buf = [0u8; 0];
if self
.scsi
.as_mut()
.execute(&tur, crate::scsi::DataDirection::None, &mut buf, 5_000)
.is_ok()
{
tracing::info!(
target: "freemkv::drive",
phase = "wait_ready",
attempts = attempt + 1,
elapsed_ms = t0.elapsed().as_millis() as u64,
"end"
);
return Ok(());
// `checked_exec`, not a bare `execute`. This poll was the ONE
// drive path that talked to the transport directly, and it is the
// longest-running one: 60 x 500 ms = ~30 s. It never read
// `self.halt`, so an operator Stop pressed while a cold drive spun
// up was ignored for up to half a minute while every other path in
// this file returned `Halted` at its next command boundary. A Stop
// that does nothing for 30 s is indistinguishable from a hung app.
//
// A TUR that FAILS is the ordinary not-ready-yet answer and must
// keep the loop going — only `Halted` aborts it.
match self.checked_exec(&tur, crate::scsi::DataDirection::None, &mut buf, 5_000) {
Ok(_) => {
tracing::info!(
target: "freemkv::drive",
phase = "wait_ready",
attempts = attempt + 1,
elapsed_ms = t0.elapsed().as_millis() as u64,
"end"
);
return Ok(());
}
Err(Error::Halted) => return Err(Error::Halted),
Err(_) => {}
}
std::thread::sleep(std::time::Duration::from_millis(500));
// Halt-aware backoff: the flag can also flip DURING the 500 ms
// gap, which is where most of the 30 s is actually spent.
sleep_until_halted(&self.halt, std::time::Duration::from_millis(500))?;
}
tracing::warn!(
target: "freemkv::drive",
@@ -926,11 +957,34 @@ impl Drive {
// range NonTrimmed and retries (a loud miss, never a silent commit).
// The sector/file path enforces the same invariant in
// sector/prefetched.rs; this is the live-drive counterpart.
Ok(_) => Err(Error::DiscRead {
sector: lba as u64,
status: None,
sense: None,
}),
//
// SAY SO. The refusal was correct and completely silent: the
// sibling `Err(e)` arm below warns with lba/count/status, this one
// logged nothing at all, so a drive that residual-underruns on
// GOOD status was indistinguishable in the logs from a scratched
// disc — two different populations (replace the drive vs. clean
// the disc) collapsed into one. Absence of a log is itself the
// defect here; the read still fails either way.
//
// `scsi_status`/`sense` are deliberately absent from the line
// rather than faked: the command SUCCEEDED, so there is no sense
// data to report. `transferred` vs `expected` is the whole signal.
Ok(result) => {
tracing::warn!(
target: "freemkv::drive",
lba,
count,
transferred = result.bytes_transferred,
expected = count as usize * 2048,
code = crate::error::E_DISC_READ,
"READ(10) returned GOOD status with a residual underrun; refusing the short transfer"
);
Err(Error::DiscRead {
sector: lba as u64,
status: None,
sense: None,
})
}
Err(Error::Halted) => Err(Error::Halted),
Err(e) => {
let (status, sense) = extract_scsi_context(&e);
@@ -1086,14 +1140,25 @@ impl Drive {
let stop = [SCSI_START_STOP_UNIT, 0, 0, 0, 0x00, 0]; // START=0, LOEJ=0 → spin down
let start = [SCSI_START_STOP_UNIT, 0, 0, 0, 0x01, 0]; // START=1, LOEJ=0 → spin up
let mut buf = [0u8; 0];
self.scsi
.as_mut()
.execute(&stop, crate::scsi::DataDirection::None, &mut buf, 30_000)?;
std::thread::sleep(std::time::Duration::from_secs(SPIN_DOWN_IDLE_SECS));
self.scsi
.as_mut()
.execute(&start, crate::scsi::DataDirection::None, &mut buf, 30_000)?;
std::thread::sleep(std::time::Duration::from_secs(SPIN_UP_SETTLE_SECS));
// `checked_exec` + `sleep_until_halted`, not `execute` + a blind
// `thread::sleep`. This routine is ~15 s of deliberate waiting
// (`SPIN_DOWN_IDLE_SECS` + `SPIN_UP_SETTLE_SECS`) issued from the
// recovery path, i.e. exactly when a run is going badly and the
// operator is most likely to press Stop. Both commands bypassed the
// halt flag and both sleeps were unconditional, so the whole cycle was
// deaf: Stop appeared to hang. Nothing here is un-interruptible — a
// half-finished spin cycle leaves the drive spun down, which the next
// command spins back up.
self.checked_exec(&stop, crate::scsi::DataDirection::None, &mut buf, 30_000)?;
sleep_until_halted(
&self.halt,
std::time::Duration::from_secs(SPIN_DOWN_IDLE_SECS),
)?;
self.checked_exec(&start, crate::scsi::DataDirection::None, &mut buf, 30_000)?;
sleep_until_halted(
&self.halt,
std::time::Duration::from_secs(SPIN_UP_SETTLE_SECS),
)?;
Ok(())
}
@@ -1299,11 +1364,18 @@ pub(crate) fn decode_read_capacity(buf: &[u8; 8], bytes_transferred: usize) -> R
last_lba.checked_add(1).ok_or(Error::DiscCapacityOverflow)
}
/// Halt-aware sleep primitive — wakes within ~100 ms of `halt` flipping
/// to true. Kept for the unit tests that cover the slicing behaviour;
/// production code paths no longer sleep on the recovery hot path
/// (recovery loop removed in 0.13.6).
#[cfg(test)]
/// Halt-aware sleep primitive — wakes within ~100 ms of `halt` flipping to
/// true, returning [`Error::Halted`].
///
/// This was `#[cfg(test)]` for two releases, kept alive only by the four unit
/// tests below, while the two production paths that actually sleep — the
/// `wait_ready` poll backoff (60 x 500 ms) and `spin_cycle`'s spin-down/settle
/// pauses (`SPIN_DOWN_IDLE_SECS` + `SPIN_UP_SETTLE_SECS`) — used a plain
/// `std::thread::sleep` and so were DEAF to the operator's Stop for ~30 s and
/// ~15 s respectively. Every other drive path returns `Halted` at the next
/// `checked_exec` boundary; a Stop pressed during spin-up simply did nothing
/// visible until the poll ran out. The primitive existed; the sleeping code
/// just did not call it. It is production code again.
fn sleep_until_halted(halt: &AtomicBool, total: std::time::Duration) -> Result<()> {
const SLICE: std::time::Duration = std::time::Duration::from_millis(100);
let deadline = std::time::Instant::now() + total;
@@ -1322,18 +1394,7 @@ fn sleep_until_halted(halt: &AtomicBool, total: std::time::Duration) -> Result<(
/// Internal: discover drive paths + IDs without opening full Drive objects.
fn discover_drives() -> Vec<(String, DriveId)> {
#[cfg(target_os = "linux")]
{
linux::find_drives()
}
#[cfg(target_os = "macos")]
{
macos::find_drives()
}
#[cfg(windows)]
{
windows::find_drives()
}
platform::find_drives()
}
/// Structured outcome of [`resolve_device`] — a machine-readable signal
@@ -1353,20 +1414,14 @@ pub enum DeviceResolution {
/// Resolve a device path to its raw SCSI device. Returns the resolved
/// path plus a structured [`DeviceResolution`] signal describing whether
/// any substitution happened; the application layer maps that to UX text.
///
/// Staged, not yet wired: the cross-platform dispatch is kept ready for the
/// caller that will consume it, so the per-platform implementations below it
/// (and their tests) stay live. `allow(dead_code)` marks that deliberately —
/// this is not an accidental orphan.
#[allow(dead_code)]
pub(crate) fn resolve_device(path: &str) -> Result<(String, DeviceResolution)> {
#[cfg(target_os = "linux")]
{
linux::resolve_device(path)
}
#[cfg(target_os = "macos")]
{
macos::resolve_device(path)
}
#[cfg(windows)]
{
windows::resolve_device(path)
}
platform::resolve_device(path)
}
#[cfg(test)]
@@ -1406,8 +1461,18 @@ mod halt_tests {
let r = sleep_until_halted(&flag, Duration::from_secs(10));
assert!(matches!(r, Err(Error::Halted)));
let waited = t0.elapsed();
// Flag flipped at ~150 ms; we wake within one 100 ms slice → <300 ms.
assert!(waited < Duration::from_millis(350), "waited {waited:?}");
// What this test is for: the sleep must end because the flag flipped,
// NOT because the 10 s timeout expired. Anything comfortably under
// that proves it, and the lower bound proves it did not return early
// for some other reason.
//
// The upper bound used to be 350 ms — flag at ~150 ms plus one 100 ms
// poll slice, with a little slack. That measures the SCHEDULER, not
// this function: a loaded CI runner took 377 ms and failed, which says
// nothing about whether the wake worked. Bound it well below the
// timeout instead, so the assertion still distinguishes the two
// outcomes it exists to distinguish.
assert!(waited < Duration::from_secs(2), "waited {waited:?}");
assert!(waited >= Duration::from_millis(140), "waited {waited:?}");
}
@@ -1926,6 +1991,90 @@ mod command_tests {
assert!(err.scsi_sense().is_none());
}
/// `extract_scsi_context` must map the two non-SCSI dead-bus faults
/// (a failed `ioctl(SG_IO)` → `Error::IoError`, a vanished fd →
/// `Error::DeviceNotFound`) to the 0xFF TRANSPORT_FAILURE sentinel, not
/// to 0x00. Everything else keeps the (0, None) catch-all.
/// Spec: `Error::is_scsi_transport_failure` (error.rs) declares both
/// variants transport failures so sweep/patch/fill_extents abort
/// the pass instead of zero-filling against a wedged device.
/// Mutation: returning (0, None) here flattens IoError into
/// `DiscRead { status: Some(0) }`, which is_scsi_transport_failure
/// rejects — a wedged USB bridge zero-fills the whole title.
#[test]
fn extract_scsi_context_maps_dead_bus_faults_to_transport_failure() {
let (status, sense) = extract_scsi_context(&Error::IoError {
source: std::io::Error::from(std::io::ErrorKind::NotConnected),
});
assert_eq!(
status,
crate::scsi::SCSI_STATUS_TRANSPORT_FAILURE,
"a failed ioctl(SG_IO) is a transport-layer fault"
);
assert!(sense.is_none(), "no SCSI reply means no sense data");
let (status, sense) = extract_scsi_context(&Error::DeviceNotFound {
path: "/dev/sg9".into(),
});
assert_eq!(
status,
crate::scsi::SCSI_STATUS_TRANSPORT_FAILURE,
"a vanished device is a transport-layer fault"
);
assert!(sense.is_none());
// Control: the catch-all still yields (0, None) for unrelated errors,
// so the two asserts above are about these variants specifically.
assert_eq!(extract_scsi_context(&Error::Halted), (0, None));
// Control: real SCSI replies still pass their own status through.
let s = crate::scsi::ScsiSense {
sense_key: 3,
asc: 0x11,
ascq: 0x00,
};
assert_eq!(
extract_scsi_context(&Error::ScsiError {
opcode: 0x28,
status: 0x02,
sense: Some(s),
}),
(0x02, Some(s))
);
}
/// End-to-end: an `Error::IoError` raised by the transport must still be
/// classified as a transport failure after `Drive::read` flattens it into
/// `Error::DiscRead`. Without the `extract_scsi_context` mapping the
/// variant is destroyed (status becomes `Some(0)`) and the caller treats a
/// dead bus as a recoverable bad sector.
#[test]
fn read_io_error_surfaces_as_transport_failure_not_a_bad_sector() {
let mut d = Drive::from_transport_for_test(Box::new(AlwaysErr {
err: || Error::IoError {
source: std::io::Error::from(std::io::ErrorKind::NotConnected),
},
}));
let mut buf = vec![0u8; 2048];
let err = d.read(42, 1, &mut buf, false).unwrap_err();
assert!(
err.is_scsi_transport_failure(),
"a wedged bus must abort the pass, not zero-fill: got {err:?}"
);
// The same for a device that vanished mid-read.
let mut d = Drive::from_transport_for_test(Box::new(AlwaysErr {
err: || Error::DeviceNotFound {
path: "/dev/sg9".into(),
},
}));
let err = d.read(42, 1, &mut buf, false).unwrap_err();
assert!(
err.is_scsi_transport_failure(),
"a vanished device must abort the pass: got {err:?}"
);
}
#[test]
fn read_returns_halted_before_dispatch_without_touching_transport() {
// When the halt flag is set, checked_exec returns Halted BEFORE
@@ -2579,6 +2728,231 @@ mod command_tests {
);
}
/// A Stop pressed before the poll starts must be answered at once.
///
/// `wait_ready` was the ONE drive path that called
/// `self.scsi.as_mut().execute(..)` instead of `checked_exec`, and its
/// 60 x 500 ms loop never read `self.halt`. A cancel during spin-up was
/// therefore ignored for ~30 s — the test above measures exactly how long
/// — while every other drive path returns `Halted` at its next command
/// boundary. Thirty seconds of a dead Stop button is indistinguishable
/// from a hung application.
///
/// Mutation: restoring the bare `execute` makes this run the full poll and
/// return `DeviceNotReady`, failing both assertions.
#[test]
fn wait_ready_returns_halted_when_stopped_before_the_poll() {
struct NeverReady;
impl ScsiTransport for NeverReady {
fn execute(
&mut self,
cdb: &[u8],
_dir: DataDirection,
_data: &mut [u8],
_timeout_ms: u32,
) -> Result<ScsiResult> {
Err(Error::ScsiError {
opcode: cdb[0],
status: 2,
sense: None,
})
}
}
let mut d = Drive::from_transport_for_test(Box::new(NeverReady));
d.halt();
let t0 = std::time::Instant::now();
let r = d.wait_ready();
assert!(
matches!(r, Err(Error::Halted)),
"a Stop is the operator, not a drive that failed to spin up: {r:?}"
);
assert!(
t0.elapsed() < std::time::Duration::from_secs(5),
"the poll must abandon immediately, not run its ~30 s course"
);
}
/// ...and a Stop pressed PART WAY THROUGH the poll must be answered at the
/// next command boundary, not at the end of the 30 s.
///
/// The flag is flipped by the transport itself on its third TEST UNIT
/// READY, which is deterministic (no wall-clock race): `checked_exec`
/// re-reads the flag after the command completes, so the loop must exit on
/// attempt three of sixty.
#[test]
fn wait_ready_returns_halted_when_stopped_during_the_poll() {
struct StopsOnThirdPoll {
halt: Arc<AtomicBool>,
seen: usize,
}
impl ScsiTransport for StopsOnThirdPoll {
fn execute(
&mut self,
cdb: &[u8],
_dir: DataDirection,
_data: &mut [u8],
_timeout_ms: u32,
) -> Result<ScsiResult> {
self.seen += 1;
if self.seen == 3 {
self.halt.store(true, Ordering::Relaxed);
}
Err(Error::ScsiError {
opcode: cdb[0],
status: 2,
sense: None,
})
}
}
let mut d = Drive::from_transport_for_test(Box::new(StopsOnThirdPoll {
halt: Arc::new(AtomicBool::new(false)),
seen: 0,
}));
// Hand the transport the drive's OWN flag, so setting it is exactly
// what an operator's Stop does.
let flag = d.halt_flag();
d.scsi = Box::new(StopsOnThirdPoll {
halt: flag,
seen: 0,
});
let t0 = std::time::Instant::now();
let r = d.wait_ready();
assert!(
matches!(r, Err(Error::Halted)),
"a mid-poll Stop must surface as Halted, not as DeviceNotReady \
after the full 30 s: {r:?}"
);
assert!(
t0.elapsed() < std::time::Duration::from_secs(5),
"must exit on the third poll, not the sixtieth"
);
}
/// `spin_cycle` must not be deaf to Stop for its ~15 s of deliberate
/// waiting.
///
/// Both START STOP UNIT commands went out through a bare `execute` and
/// both pauses (`SPIN_DOWN_IDLE_SECS` + `SPIN_UP_SETTLE_SECS`) were plain
/// `std::thread::sleep`, so the whole routine ignored the halt flag. It
/// runs from the RECOVERY path — precisely when a run is going badly and
/// the operator is most likely to press Stop.
///
/// The flag is set BEFORE the call, so this is deterministic: the very
/// first `checked_exec` must refuse.
///
/// Mutation: restoring either bare `execute` lets the first command
/// through and the test then waits out a real 5 s sleep, failing the
/// elapsed bound.
#[test]
fn spin_cycle_returns_halted_when_stopped_before_it_starts() {
let RecordingHarness {
drive: mut d,
cdb,
timeouts: _to,
} = recording(TransportOutcome::Ok(0));
d.halt();
let t0 = std::time::Instant::now();
let r = d.spin_cycle();
assert!(
matches!(r, Err(Error::Halted)),
"a Stop must abort the spin cycle: {r:?}"
);
assert!(
t0.elapsed() < std::time::Duration::from_secs(2),
"no sleep may run after the flag is set"
);
assert!(
cdb.lock().unwrap().is_empty(),
"not one START STOP UNIT may be issued after Stop"
);
}
/// ...and a Stop that lands DURING the spin-down pause must wake it.
///
/// This is the half a `checked_exec` alone cannot fix: the flag flips
/// while the thread is parked inside the 5 s `SPIN_DOWN_IDLE_SECS` sleep,
/// so only a halt-aware sleep can observe it. `sleep_until_halted` — which
/// already lived in this file with four tests, marked `#[cfg(test)]` and
/// called from nowhere — wakes within ~100 ms.
///
/// The margin is deliberately enormous (≈0.3 s against 5 s) so the bound
/// is not a wall-clock race: a plain `thread::sleep` cannot come in under
/// two seconds, and the halt-aware one cannot take that long.
#[test]
fn spin_cycle_wakes_from_its_spin_down_pause_when_stopped() {
let RecordingHarness {
drive: mut d,
cdb: _cdb,
timeouts: _to,
} = recording(TransportOutcome::Ok(0));
let flag = d.halt_flag();
let stopper = std::thread::spawn(move || {
std::thread::sleep(std::time::Duration::from_millis(200));
flag.store(true, Ordering::Relaxed);
});
let t0 = std::time::Instant::now();
let r = d.spin_cycle();
stopper.join().expect("stopper thread");
assert!(
matches!(r, Err(Error::Halted)),
"a Stop during the spin-down pause must surface as Halted: {r:?}"
);
assert!(
t0.elapsed() < std::time::Duration::from_secs(2),
"the pause must be halt-aware, not a blind {SPIN_DOWN_IDLE_SECS}s \
thread::sleep; took {:?}",
t0.elapsed()
);
}
/// A READ(10) that completes with GOOD status but a residual underrun is
/// correctly refused — and must SAY SO.
///
/// The refusal itself already existed and is right: the tail of the buffer
/// still holds stale bytes, so committing them would be silent
/// corruption. What was missing is any trace of it. The sibling `Err(e)`
/// arm warns with lba/count/status; this arm logged NOTHING, so a drive
/// that residual-underruns on GOOD status produced exactly the same
/// journal as a scratched disc — two populations with opposite remedies
/// (replace the drive vs. clean the disc) collapsed into one, with the
/// operator sent after the wrong one.
///
/// Mutation: deleting the `tracing::warn!` leaves no event; logging a
/// different code, or dropping the transferred/expected pair that is the
/// entire signal, fails the field assertions.
#[test]
fn read_logs_a_good_status_short_transfer() {
let RecordingHarness {
drive: mut d,
cdb: _cdb,
timeouts: _to,
} = recording(TransportOutcome::Ok(1024)); // half of one sector
let mut buf = vec![0u8; 2048];
let (r, events) = crate::testlog::capture(|| d.read(7, 1, &mut buf, false));
assert!(
matches!(r, Err(Error::DiscRead { sector: 7, .. })),
"a short transfer stays a failed read: {r:?}"
);
let line = events
.iter()
.find(|e| e.target == "freemkv::drive" && e.field("transferred").is_some())
.unwrap_or_else(|| {
panic!("a silently refused short transfer is the defect; got {events:?}")
});
assert_eq!(line.level, tracing::Level::WARN);
assert_eq!(line.field("lba"), Some("7"));
assert_eq!(line.field("transferred"), Some("1024"));
assert_eq!(
line.field("expected"),
Some("2048"),
"the underrun is only legible as transferred-vs-expected"
);
assert_eq!(
line.field("code"),
Some(crate::error::E_DISC_READ.to_string().as_str())
);
}
// ── report_key / mode_sense / read_buffer empty-vs-some ─────────
#[test]
+422 -119
View File
@@ -45,6 +45,7 @@ pub const E_INVALID_CDB_LENGTH: u16 = 4001;
// I/O (5xxx)
pub const E_IO_ERROR: u16 = 5000;
pub const E_SOURCE_TERMINATED: u16 = 5001;
// Disc format (6xxx)
pub const E_DISC_READ: u16 = 6000;
@@ -62,6 +63,23 @@ pub const E_MAPFILE_INVALID: u16 = 6011;
pub const E_SELECTION_PID_UNKNOWN: u16 = 6014;
pub const E_UDF_BUFFER_TOO_SMALL: u16 = 6012;
pub const E_UDF_NOT_FILESYSTEM: u16 = 6013;
pub const E_IMAGE_TRUNCATED: u16 = 6015;
pub const E_UDF_AD_CHAIN_TOO_LONG: u16 = 6016;
pub const E_UDF_UNRECORDED_EXTENT: u16 = 6017;
pub const E_UDF_EMBEDDED_DATA: u16 = 6018;
/// A file that EXISTS and whose allocation descriptors resolved without error,
/// yet yields not one usable extent: an empty AD list, or a list every entry of
/// which is zero-length or points at LBA 0. Reported by the HD-DVD clip
/// resolver (`disc::hddvd`).
///
/// It has no [`Error`] variant on purpose. `UdfFs::file_extents` returns
/// `Ok(vec![])` here rather than failing — the emptiness is only a defect in
/// the eye of a caller that needs bytes — so the condition is DETECTED by the
/// caller, not returned to it. The code exists so that detection can be
/// accounted in the log with something other than a neighbouring error's code:
/// logging it as E6017 would file a zero-length AD list as an authoring hole
/// and send whoever triages it at the wrong population.
pub const E_UDF_NO_USABLE_EXTENT: u16 = 6019;
// AACS (7xxx)
pub const E_AACS_NO_KEYS: u16 = 7000;
@@ -170,6 +188,41 @@ pub const E_DIR_INSUFFICIENT_SPACE: u16 = 9027;
pub const E_DIR_NAME_COLLISION: u16 = 9028;
/// A `dir://` create_dir_all / file write / rename failed.
pub const E_DIR_WRITE_FAILED: u16 = 9029;
/// A `dir://` SOURCE folder carries `BDMV/STREAM/SSIF/` (Blu-ray 3D). The
/// scanner detects SSIF unconditionally and would rip it as 3D, but the
/// synthetic-image planner has no extent-aliasing support (an SSIF interleaves
/// the same sectors as the base/dependent `.m2ts`), so the output would be
/// silently wrong. Rejected up front instead.
pub const E_DIR_IMAGE_SSIF_UNSUPPORTED: u16 = 9061;
/// A `dir://` SOURCE `VIDEO_TS` folder's IFO-declared VOB offsets cannot be
/// satisfied by any placement: the required start sector of a VOB lies BELOW
/// the end of the file that must precede it. Carries the offending file's disc
/// path. A silently misplaced VOB would rip the wrong sectors.
pub const E_DIR_IMAGE_PLACEMENT: u16 = 9062;
/// A `dir://` SOURCE folder still carries live AACS-encrypted content: it has
/// an `AACS/` directory AND the sampled content units are genuinely scrambled.
/// `dir://` sources are decrypted-folder only.
pub const E_DIR_IMAGE_ENCRYPTED: u16 = 9063;
/// A `dir://` SOURCE folder holds no disc structure the image synthesizer
/// understands (no `BDMV/`, no `VIDEO_TS/`).
pub const E_DIR_IMAGE_UNSUPPORTED_TREE: u16 = 9064;
/// A file inside a `dir://` SOURCE folder changed (shrank / was removed)
/// between planning and reading. Zero-filling the gap would produce corrupt
/// output at exit 0, so the read fails instead. Carries the disc path.
pub const E_DIR_IMAGE_FILE_CHANGED: u16 = 9065;
/// A `dir://` SOURCE folder does not fit a 32-bit sector address space
/// (> 2^32 sectors ≈ 8 TiB), or holds more entries than a UDF tree can carry.
pub const E_DIR_IMAGE_TOO_LARGE: u16 = 9066;
/// A name in the folder is too long to record in a UDF directory entry: the
/// File Identifier Descriptor stores the encoded length in one byte.
pub const E_DIR_NAME_TOO_LONG: u16 = 9067;
/// One directory in the folder holds more subdirectories than a UDF link count
/// can express (it is 16 bits, one per child plus one for its own entry).
pub const E_DIR_IMAGE_FANOUT: u16 = 9068;
/// A title's clip marks excluded more frames than they kept.
pub const E_SEAM_PLAN_DROPPED_MOST: u16 = 9069;
/// A sink finished having written no frames at all.
pub const E_SINK_WROTE_NOTHING: u16 = 9070;
pub const E_M2TS_PACKET_MALFORMED: u16 = 9021;
/// A `network://` output target resolved to no address that is safe to
/// connect to (every resolved IP was loopback / private / link-local /
@@ -243,6 +296,8 @@ pub const E_SYNC_WORKER_LOST: u16 = 9057;
/// READ CAPACITY returned a short or overflowing transfer.
pub const E_DISC_CAPACITY_MALFORMED: u16 = 9047;
pub const E_DRIVE_INQUIRY_SHORT: u16 = 9058;
pub const E_SHORT_IMAGE_READ: u16 = 9059;
pub const E_EMPTY_IMAGE: u16 = 9060;
// ── Error enum ──────────────────────────────────────────────────────────────
@@ -355,6 +410,18 @@ pub enum Error {
UdfNotFound {
path: String,
},
/// The file's ICB allocation list contains an unrecorded (ECMA-167
/// 4/14.14.1.1 type-1/type-2) extent: space allocated to the file at that
/// location but never written, so its true content there is zeros while
/// the media holds something else.
///
/// Raised by [`crate::udf::UdfFs::file_extents`] because a
/// `(lba, sector_count)` read plan cannot express a hole — reading it
/// splices undefined sectors into the rip as content, and dropping it
/// slides every later extent's byte space.
UdfUnrecordedExtent {
path: String,
},
/// The reader was addressable but the bytes are structurally NOT a UDF
/// filesystem — a deterministic tag/format mismatch (e.g. no Anchor Volume
/// Descriptor Pointer at sector 256, no partition descriptor, no File Set
@@ -367,10 +434,52 @@ pub enum Error {
/// 2048-byte sector. A contract violation on the public reader API —
/// returned instead of panicking on the slice.
UdfBufferTooSmall,
/// A file's allocation-descriptor continuation chain did not end within the
/// hop budget the UDF reader allows.
///
/// The budget exists so a crafted or corrupt disc cannot loop the reader
/// forever. Hitting it is NOT the end of the chain: the extents beyond that
/// point are unknown, so the extent list in hand describes only part of the
/// file. Returning that list would let a caller zero-pad the remainder to
/// the declared size and report a mostly-empty file as a complete
/// extraction, so the read fails instead.
UdfAdChainTooLong,
/// A file's ICB declares its data EMBEDDED inline (ECMA-167 4/14.6.8
/// allocation-descriptor type 3), so it has no out-of-line extents at all.
///
/// Returned only when a caller asked for a read plan over such a file.
/// The bytes in the allocation-descriptor field are then the file's own
/// CONTENT, not descriptors, so decoding them as (length, LBA) pairs
/// manufactures extents out of arbitrary data and points the reader at
/// unrelated sectors — a rip that completes at rc=0 carrying whatever
/// happened to be there. `read_directory` already refuses the same shape
/// for directories; this is the file half of that decision.
///
/// A file that legitimately stores its data this way is tiny (an ICB caps
/// it at well under 2 KiB — the AACS `*.inf` key files are the usual
/// case), and the callers that expect one read it via `read_inline_data`
/// long before extents are ever requested. A stream file cannot be one.
UdfEmbeddedData,
DiscTitleRange {
index: usize,
count: usize,
},
/// An image-level write read fewer bytes than the sector count it asked for.
///
/// Distinct from [`Error::IoError`] on purpose: the read SUCCEEDED and simply
/// returned less than a whole sector run, which for a file-backed source means
/// the source is shorter than its declared capacity. Zero-filling the gap
/// would produce an image that looks complete and is not — the worst outcome
/// for a copy someone intends to keep — so it is an error instead.
ShortImageRead {
lba: u32,
expected: u32,
got: u32,
},
/// An image-level write was asked for zero sectors. A zero-byte image is
/// never the intent, and reporting it here names the problem at the point it
/// is knowable rather than leaving an empty file behind.
EmptyImage,
IfoParse,
/// A title produced NO muxable frames: the mux driver's pump ended without
/// any video track's `codec_private` resolving, or the MKV muxer reached
@@ -408,6 +517,14 @@ pub enum Error {
MapfileInvalid {
kind: &'static str,
},
/// The image a mapfile describes is shorter than the mapfile's own total
/// size, so the two no longer agree about the same disc. Resuming against
/// it would treat the absent tail as already recovered. `have` and `want`
/// are byte lengths; a missing file reports `have == 0`.
ImageTruncated {
have: u64,
want: u64,
},
// AACS (7xxx)
AacsNoKeys,
@@ -671,6 +788,19 @@ pub enum Error {
/// it silently leaves encrypted. The producer surfaces this rather
/// than emit still-encrypted bytes.
ExtentNotUnitAligned,
/// A [`crate::sector::SectorSource`] that feeds its reads from a
/// producer thread has terminated for good — the thread exited after
/// an error or before delivering the extents it was given — so it can
/// never return another byte.
///
/// It exists because the alternative answer is a lie: a dead source
/// that reports `Ok(0)` is indistinguishable from end-of-stream, and
/// `DiscStream::fill_extents` legitimately reads a short count as a
/// skippable hole — zero-filling and advancing over every remaining
/// sector of the title and still returning success. Unlike a bad
/// sector, this condition cannot be retried at a smaller size or
/// skipped past, so every consumer must abort the pass on it.
SourceTerminated,
/// An MPEG-TS packet under construction violated the 188-byte fixed
/// size (over-long adaptation field, overflowing payload, or a
/// short/mis-assembled packet). Indicates a muxer invariant break,
@@ -715,6 +845,58 @@ pub enum Error {
DirWriteFailed {
errno: Option<i32>,
},
/// A `dir://` SOURCE folder carries `BDMV/STREAM/SSIF/` (Blu-ray 3D),
/// which the synthetic-image planner cannot represent. See
/// [`E_DIR_IMAGE_SSIF_UNSUPPORTED`].
DirImageSsifUnsupported,
/// A `dir://` SOURCE `VIDEO_TS` placement constraint is unsatisfiable.
/// `path` is the disc path of the file that could not be placed.
DirImagePlacement {
path: String,
},
/// A `dir://` SOURCE folder still carries live AACS-encrypted content.
DirImageEncrypted,
/// A `dir://` SOURCE folder holds no recognized disc structure.
DirImageUnsupportedTree,
/// A file in a `dir://` SOURCE folder changed between plan and read.
DirImageFileChanged {
path: String,
},
/// A `dir://` SOURCE folder exceeds the addressable image size.
DirImageTooLarge,
/// A name is too long to record in a UDF directory entry.
///
/// The File Identifier Descriptor stores the encoded name length in ONE
/// byte, so a name whose OSTA CS0 encoding exceeds 254 bytes cannot be
/// described. Truncating the length field instead would desynchronise the
/// whole directory — every later entry in it would be read from the wrong
/// offset — so an over-long name is refused while the tree is still being
/// planned.
DirNameTooLong {
path: String,
},
/// One directory holds more subdirectories than a UDF link count can express.
///
/// A directory's File Entry records its link count in 16 bits, and that
/// count is one per child directory plus one for its own entry in its
/// parent. Beyond that the count silently wraps, so the tree is refused.
DirImageFanout {
path: String,
},
/// A title's PlayItem marks excluded more frames than they kept.
///
/// Placing clips by their marks drops whatever falls outside them, which is
/// correct at a join — a disc stores the join twice. Discarding the
/// majority of a title is not a join; it means the marks do not describe
/// the clock the frames are on. Refused rather than written, because the
/// result otherwise looks like a complete file containing seconds of a
/// feature.
SeamPlanDroppedMost {
dropped: u64,
written: u64,
},
/// A sink finished having written no frames at all.
SinkWroteNothing,
}
impl Error {
@@ -736,14 +918,20 @@ impl Error {
Error::ScsiError { .. } => E_SCSI_ERROR,
Error::InvalidCdbLength { .. } => E_INVALID_CDB_LENGTH,
Error::IoError { .. } => E_IO_ERROR,
Error::SourceTerminated => E_SOURCE_TERMINATED,
Error::DiscRead { .. } => E_DISC_READ,
Error::Halted => E_HALTED,
Error::MplsParse => E_MPLS_PARSE,
Error::ClpiParse => E_CLPI_PARSE,
Error::UdfNotFound { .. } => E_UDF_NOT_FOUND,
Error::UdfUnrecordedExtent { .. } => E_UDF_UNRECORDED_EXTENT,
Error::UdfNotFilesystem => E_UDF_NOT_FILESYSTEM,
Error::UdfBufferTooSmall => E_UDF_BUFFER_TOO_SMALL,
Error::UdfAdChainTooLong => E_UDF_AD_CHAIN_TOO_LONG,
Error::UdfEmbeddedData => E_UDF_EMBEDDED_DATA,
Error::DiscTitleRange { .. } => E_DISC_TITLE_RANGE,
Error::ShortImageRead { .. } => E_SHORT_IMAGE_READ,
Error::EmptyImage => E_EMPTY_IMAGE,
Error::IfoParse => E_IFO_PARSE,
Error::MkvInvalid => E_MKV_INVALID,
Error::MkvSourceInvalid => E_MKV_SOURCE_INVALID,
@@ -752,6 +940,7 @@ impl Error {
Error::NoStreams => E_NO_STREAMS,
Error::SelectionPidUnknown { .. } => E_SELECTION_PID_UNKNOWN,
Error::MapfileInvalid { .. } => E_MAPFILE_INVALID,
Error::ImageTruncated { .. } => E_IMAGE_TRUNCATED,
Error::AacsNoKeys => E_AACS_NO_KEYS,
Error::AacsCertShort => E_AACS_CERT_SHORT,
Error::AacsAgidAlloc => E_AACS_AGID_ALLOC,
@@ -830,6 +1019,16 @@ impl Error {
Error::DirInsufficientSpace { .. } => E_DIR_INSUFFICIENT_SPACE,
Error::DirNameCollision { .. } => E_DIR_NAME_COLLISION,
Error::DirWriteFailed { .. } => E_DIR_WRITE_FAILED,
Error::DirImageSsifUnsupported => E_DIR_IMAGE_SSIF_UNSUPPORTED,
Error::DirImagePlacement { .. } => E_DIR_IMAGE_PLACEMENT,
Error::DirImageEncrypted => E_DIR_IMAGE_ENCRYPTED,
Error::DirImageUnsupportedTree => E_DIR_IMAGE_UNSUPPORTED_TREE,
Error::DirNameTooLong { .. } => E_DIR_NAME_TOO_LONG,
Error::DirImageFanout { .. } => E_DIR_IMAGE_FANOUT,
Error::SeamPlanDroppedMost { .. } => E_SEAM_PLAN_DROPPED_MOST,
Error::SinkWroteNothing => E_SINK_WROTE_NOTHING,
Error::DirImageFileChanged { .. } => E_DIR_IMAGE_FILE_CHANGED,
Error::DirImageTooLarge => E_DIR_IMAGE_TOO_LARGE,
}
}
}
@@ -936,6 +1135,19 @@ impl std::fmt::Display for Error {
},
Error::Halted => write!(f, "E{}", self.code()),
Error::UdfNotFound { path } => write!(f, "E{}: {}", self.code(), path),
Error::UdfUnrecordedExtent { path } => write!(f, "E{}: {}", self.code(), path),
Error::SeamPlanDroppedMost { dropped, written } => {
write!(f, "E{} {dropped}/{written}", self.code())
}
Error::ShortImageRead { lba, expected, got } => {
write!(f, "E{} {lba} {expected} {got}", self.code())
}
Error::DirImagePlacement { path }
| Error::DirImageFileChanged { path }
| Error::DirNameTooLong { path }
| Error::DirImageFanout { path } => {
write!(f, "E{}: {}", self.code(), path)
}
Error::DiscTitleRange { index, count } => {
write!(f, "E{}: {}/{}", self.code(), index, count)
}
@@ -989,7 +1201,21 @@ impl std::error::Error for Error {
impl From<std::io::Error> for Error {
fn from(e: std::io::Error) -> Self {
Error::IoError { source: e }
// If this `io::Error` is one WE produced (`From<Error> for io::Error`
// carries the typed value in its boxed payload), give the original
// back instead of burying it in `Error::IoError`. That wrapper is not
// neutral: `is_scsi_transport_failure` treats `IoError` as a
// transport-layer fault (dead bus / wedged bridge), so re-wrapping a
// round-tripped `DiscRead` MEDIUM ERROR turned a skippable bad sector
// into a pass-aborting bridge wedge — the exact inverse of what that
// arm exists for. Any error that crosses a thread boundary as an
// `io::Error` (the prefetch channel's `Batch`) keeps its
// classification, its SCSI status, and its sense data.
match e.downcast::<Error>() {
Ok(typed) => typed,
// A genuine OS/`std` error — the `IoError` wrapper is correct here.
Err(io) => Error::IoError { source: io },
}
}
}
@@ -1004,7 +1230,6 @@ impl From<Error> for std::io::Error {
return source;
}
let code = e.code();
let msg = e.to_string();
// Map our error categories to io::ErrorKind
let kind = match code {
// Device access-denied semantics map to PermissionDenied;
@@ -1087,9 +1312,31 @@ impl From<Error> for std::io::Error {
// 9027 insufficient space / 9029 write failed: a filesystem-level
// failure, not bad input.
E_DIR_INSUFFICIENT_SPACE | E_DIR_WRITE_FAILED => std::io::ErrorKind::Other,
// dir:// SOURCE gates (9061–9064, 9066–9068): the folder handed in
// cannot be turned into a disc image — a 3D SSIF tree, an
// unsatisfiable VIDEO_TS placement, still-encrypted content, an
// unrecognized tree, one too large to address, a name too long to
// record, or a directory whose fan-out overflows its link count.
// All are properties of the input, decided before a byte is read.
E_DIR_IMAGE_SSIF_UNSUPPORTED
| E_DIR_IMAGE_PLACEMENT
| E_DIR_IMAGE_ENCRYPTED
| E_DIR_IMAGE_UNSUPPORTED_TREE
| E_DIR_IMAGE_TOO_LARGE
| E_DIR_NAME_TOO_LONG
| E_DIR_IMAGE_FANOUT => std::io::ErrorKind::InvalidInput,
// 9065: the folder changed underneath a running read. Not bad input
// at plan time — a mid-flight mutation of the source.
E_DIR_IMAGE_FILE_CHANGED => std::io::ErrorKind::InvalidData,
_ => std::io::ErrorKind::Other,
};
std::io::Error::new(kind, msg)
// Carry the typed value itself as the payload rather than its
// stringification. `Display` is unchanged (`io::Error` delegates to the
// boxed error, whose `Display` is the same `E<code>[: …]` string), so
// `error_code` and every consumer built on it are unaffected — but the
// typed error now SURVIVES the conversion and `From<io::Error> for
// Error` can hand it back intact.
std::io::Error::new(kind, e)
}
}
@@ -1108,13 +1355,14 @@ pub type Result<T> = std::result::Result<T, Error>;
/// removing comes back.
///
/// [`From<Error> for io::Error`] is the ONLY path from a typed [`Error`] to an
/// `io::Error` in this crate, and it stringifies (`io::Error::new(kind, msg)`
/// where `msg` is the `Error`'s `E<code>[: …]` [`Display`](std::fmt::Display)
/// string) rather than boxing the typed value — no code path constructs an
/// `io::Error` that still holds a `crate::error::Error` via `get_ref`. So the
/// only recognised shape is the round-tripped `E<code>` message prefix.
/// `io::Error` in this crate. It boxes the typed value as the payload
/// (`io::Error::new(kind, e)`), whose [`Display`](std::fmt::Display) is the
/// same `E<code>[: …]` string the stringifying version produced — so this
/// parse is unaffected, and `From<io::Error> for Error` can additionally
/// `downcast` the payload back to the exact typed error. Errors that did NOT
/// come from this crate carry no `E<code>` prefix and yield `None`.
pub fn error_code(e: &std::io::Error) -> Option<u16> {
// Round-tripped: `From<Error> for io::Error` stringifies as "E<code>[: …]".
// Round-tripped: `From<Error> for io::Error` renders as "E<code>[: …]".
let s = e.to_string();
let digits = s.strip_prefix('E')?;
let end = digits
@@ -1243,6 +1491,22 @@ impl Error {
)
}
/// True if the read SOURCE itself is gone, as opposed to one range of
/// media being unreadable. Kept separate from
/// [`is_scsi_transport_failure`](Self::is_scsi_transport_failure) —
/// which is about the bus/bridge and drives "power-cycle the drive"
/// advice — because a terminated producer thread is neither a wedged
/// bridge nor a bad sector, and reporting it as SCSI status 0xFF would
/// be a fabricated status byte.
///
/// What it shares with a transport failure is the only thing the read
/// loops need to know: retrying smaller or skipping ahead cannot
/// recover anything, so the pass must abort rather than fabricate
/// zeros for the rest of the title.
pub fn is_source_terminated(&self) -> bool {
matches!(self, Error::SourceTerminated)
}
/// True if this error indicates bridge degradation — the SCSI status
/// is neither GOOD (0x00), CHECK CONDITION (0x02), nor transport failure
/// (0xFF). Observed on the Initio INIC-1618L USB bridge preceding a full
@@ -1425,6 +1689,27 @@ mod tests {
.code(),
Error::DirNameCollision { host: "x".into() }.code(),
Error::DirWriteFailed { errno: Some(28) }.code(),
Error::DirImageSsifUnsupported.code(),
Error::DirImagePlacement { path: "x".into() }.code(),
Error::DirImageEncrypted.code(),
Error::DirImageUnsupportedTree.code(),
Error::DirImageFileChanged { path: "x".into() }.code(),
Error::DirImageTooLarge.code(),
Error::DirNameTooLong { path: "x".into() }.code(),
Error::DirImageFanout { path: "x".into() }.code(),
Error::SeamPlanDroppedMost {
dropped: 1,
written: 0,
}
.code(),
Error::SinkWroteNothing.code(),
Error::ShortImageRead {
lba: 0,
expected: 1,
got: 0,
}
.code(),
Error::EmptyImage.code(),
];
let mut sorted = codes.to_vec();
sorted.sort();
@@ -1468,6 +1753,13 @@ mod tests {
E_PLATFORM_NOT_IMPLEMENTED,
),
(Error::MapfileInvalid { kind: "hex" }, E_MAPFILE_INVALID),
(
Error::ImageTruncated {
have: 0,
want: 1024,
},
E_IMAGE_TRUNCATED,
),
(Error::DiscUrlNotDirect, E_DISC_URL_NOT_DIRECT),
(Error::ExtentNotUnitAligned, E_EXTENT_NOT_UNIT_ALIGNED),
// Both CSS no-key verdicts: numeric-only Display, no English.
@@ -1621,119 +1913,129 @@ mod tests {
// ── New comprehensive tests ────────────────────────────────────────────────
/// Every `pub const E_*` code declared in this file, as
/// `(name, value)`, PARSED OUT OF THE SOURCE at compile time.
///
/// WHY THIS IS PARSED AND NOT LISTED. The uniqueness test below used to
/// carry a hand-maintained `vec![]` of the constants, with a doc comment
/// claiming it "pins all code assignments". It did not. At the time this
/// was written the file declared **127** `pub const E_*` and the vector
/// named **109** of them: eighteen codes — `E_DIR_IMAGE_FANOUT`,
/// `E_DIR_INSUFFICIENT_SPACE`, `E_DIR_MULTIPASS_REJECTED`,
/// `E_DIR_NAME_COLLISION`, `E_DIR_NAME_TOO_LONG`, `E_DIR_NOT_EMPTY`,
/// `E_DIR_RAW_REJECTED`, `E_DIR_SOURCE_UNSUPPORTED`, `E_DIR_WRITE_FAILED`,
/// `E_DRIVE_INQUIRY_SHORT`, `E_EMPTY_IMAGE`, `E_MP4_UNKNOWN_RESOLUTION`,
/// `E_SEAM_PLAN_DROPPED_MOST`, `E_SHORT_IMAGE_READ`, `E_SINK_WROTE_NOTHING`,
/// `E_SOURCE_TERMINATED`, `E_SYNC_TIMEOUT`, `E_SYNC_WORKER_LOST` — were
/// outside the guarantee entirely, so a new variant colliding with any of
/// them passed green. Worse, an earlier audit READ that doc comment and
/// trusted it while assigning new codes.
///
/// The defect is not the eighteen omissions, it is that the list is
/// hand-maintained at all: adding a constant and forgetting the vector is
/// a silent no-op, which is the definition of a guarantee that decays.
/// Deriving the list from the declarations makes forgetting impossible —
/// the only way to escape the check is to stop declaring the constant.
///
/// `include_str!` of this very file is the cheapest seam that does that:
/// no `build.rs`, no proc macro, no dependency, and — being `#[cfg(test)]`
/// — not one byte of the source embedded in a release build.
fn declared_error_codes() -> Vec<(&'static str, u16)> {
const SRC: &str = include_str!("error.rs");
SRC.lines()
.filter_map(|line| {
// Only real declarations at column 0. A retired code is
// recorded as `// 2001: burned/retired`, which carries no `pub
// const` and is therefore correctly invisible here.
let decl = line.strip_prefix("pub const ")?;
let (name, value) = decl.split_once(": u16 = ")?;
if !name.starts_with("E_") {
return None;
}
let value = value.strip_suffix(';')?;
Some((
name,
value
.parse::<u16>()
.unwrap_or_else(|_| panic!("non-literal error code for `{name}`")),
))
})
.collect()
}
/// The parser above must actually FIND the declarations, and read their
/// values correctly. Without this, a `declared_error_codes` that returned
/// an empty vector would make the uniqueness test below pass vacuously —
/// exactly the failure mode (a guarantee that is really a no-op) this
/// whole change exists to remove.
///
/// Mutation: a `strip_prefix` typo, an off-by-one in the name slice, or a
/// parser that drops the last line fails here.
#[test]
fn declared_error_codes_parses_the_declarations_it_claims_to() {
let declared = declared_error_codes();
// Independent count of the declarations, computed a different way
// from the parser under test.
let expected = include_str!("error.rs")
.lines()
.filter(|l| l.starts_with("pub const E_"))
.count();
assert_eq!(
declared.len(),
expected,
"the parser must see every `pub const E_*` line"
);
assert!(
expected >= 120,
"sanity floor: this file declares well over a hundred codes, got {expected}"
);
// Names and values must match the compiled constants, so a parser that
// mis-slices the name or mis-reads the digits cannot pass.
for (name, value) in [
("E_DEVICE_NOT_FOUND", E_DEVICE_NOT_FOUND),
("E_UDF_UNRECORDED_EXTENT", E_UDF_UNRECORDED_EXTENT),
("E_KEYDB_PARSE", E_KEYDB_PARSE),
] {
assert!(
declared.contains(&(name, value)),
"`{name}` = {value} not parsed out of the source; got {:?}",
declared
.iter()
.filter(|(n, _)| *n == name)
.collect::<Vec<_>>()
);
}
// A comment-only retired code must NOT be picked up as a constant.
assert!(
!declared.iter().any(|(n, _)| n.is_empty()),
"no empty names"
);
}
/// Every published error code constant must be unique.
/// This pins all code assignments: a new variant that accidentally reuses
/// an existing code will make this test fail.
/// Mutation: changing E_KEYDB_PARSE from 8004 to 8000 (duplicating E_KEYDB_CONNECT) fails here.
///
/// The set is derived from the declarations by
/// [`declared_error_codes`], so — unlike the hand-kept vector this
/// replaced — a newly added constant is covered the moment it is
/// written, with no second edit to remember.
///
/// Mutation: changing E_KEYDB_PARSE from 8004 to 8000 (duplicating
/// E_KEYDB_CONNECT) fails here, and so now does the same collision on any
/// of the eighteen codes the old hand-kept list had never heard of.
#[test]
fn all_error_code_constants_are_unique() {
let mut codes = vec![
E_DEVICE_NOT_FOUND,
E_DEVICE_PERMISSION,
E_DEVICE_NOT_READY,
E_DEVICE_RESET_FAILED,
E_SCSI_INTERFACE_UNAVAILABLE,
E_DEVICE_LOCKED,
E_IOKIT_PLUGIN_FAILED,
E_UNSUPPORTED_DRIVE,
E_PROFILE_PARSE,
E_UNSUPPORTED_PLATFORM,
E_PLATFORM_NOT_IMPLEMENTED,
E_UNLOCK_FAILED,
E_SIGNATURE_MISMATCH,
E_SCSI_ERROR,
E_INVALID_CDB_LENGTH,
E_IO_ERROR,
E_DISC_READ,
E_MPLS_PARSE,
E_CLPI_PARSE,
E_UDF_NOT_FOUND,
E_DISC_TITLE_RANGE,
E_IFO_PARSE,
E_MKV_INVALID,
E_NO_STREAMS,
E_HALTED,
E_MAPFILE_INVALID,
E_UDF_BUFFER_TOO_SMALL,
E_UDF_NOT_FILESYSTEM,
E_AACS_NO_KEYS,
E_AACS_CERT_SHORT,
E_AACS_AGID_ALLOC,
E_AACS_CERT_REJECTED,
E_AACS_CERT_READ,
E_AACS_CERT_VERIFY,
E_AACS_KEY_READ,
E_AACS_KEY_REJECTED,
E_AACS_KEY_VERIFY,
E_AACS_VID_READ,
E_AACS_VID_MAC,
E_AACS_DATA_KEY,
E_DECRYPT_FAILED,
E_CSS_AUTH_FAILED,
E_AACS_HOST_CERT_REJECTED,
E_AACS_RAW_READ_UNSUPPORTED,
E_AACS_VID_UNAVAILABLE,
E_AACS_MK_UNAVAILABLE,
E_AACS_VUK_NOT_IN_KEYDB,
E_DRIVE_PROFILE_MISSING,
E_VID_CDB_UNAVAILABLE,
E_NO_DISC_KEY,
E_CSS_KEY_MISSING,
E_CSS_NO_DISC_KEY,
E_KEY_SERVICE_UNAVAILABLE,
E_KEY_SERVICE_UNAUTHORIZED,
E_KEY_SERVICE_RATE_LIMITED,
E_AACS_NO_HOST_CERT,
E_AACS_BUS_KEY_UNAVAILABLE,
E_FMTS_KEY_MISSING,
E_KEYDB_CONNECT,
E_KEYDB_HTTP,
E_KEYDB_INVALID,
E_KEYDB_WRITE,
E_KEYDB_PARSE,
E_KEYDB_LOAD,
E_KEYDB_UNSUPPORTED_SCHEME,
E_KEYDB_TOO_MANY_REDIRECTS,
E_STREAM_READ_ONLY,
E_STREAM_WRITE_ONLY,
E_STREAM_URL_INVALID,
E_STREAM_URL_MISSING_PATH,
E_STREAM_URL_MISSING_PORT,
E_NETWORK_ADDR_BLOCKED,
E_MUX_EMPTY,
E_MUX_HEADER_BUFFER_EXCEEDED,
E_MKV_LACING_INVALID,
E_MKV_SOURCE_INVALID,
E_MKV_UNENCODABLE,
E_MP4_NO_VIDEO_TRACK,
E_MP4_INVALID,
E_MP4_MISSING_CODEC_PRIVATE,
E_PES_FRAME_TOO_LARGE,
E_PES_INVALID_MAGIC,
E_PES_TRACK_TOO_LARGE,
E_ISO_TOO_LARGE,
E_NO_METADATA,
E_DISC_URL_NOT_DIRECT,
E_HEVC_PARAM_PARSE,
E_MUX_TRACK_RANGE,
E_FMP4_UNIMPLEMENTED,
E_DEMUX_THREAD_PANICKED,
E_PIPELINE_JOIN_TIMEOUT,
E_PIPELINE_CONSUMER_PANICKED,
E_SWEEP_CONSUMER_GONE,
E_PIPELINE_CONSUMER_GONE,
E_DISC_CAPACITY_OVERFLOW,
E_M2TS_PACKET_MALFORMED,
E_EXTENT_NOT_UNIT_ALIGNED,
E_DISC_CAPACITY_MALFORMED,
];
let original_len = codes.len();
codes.sort();
codes.dedup();
assert_eq!(
codes.len(),
original_len,
"duplicate error code constants detected — check error.rs"
let mut by_code: std::collections::BTreeMap<u16, Vec<&str>> =
std::collections::BTreeMap::new();
for (name, value) in declared_error_codes() {
by_code.entry(value).or_default().push(name);
}
let dupes: Vec<_> = by_code
.iter()
.filter(|(_, names)| names.len() > 1)
.collect();
assert!(
dupes.is_empty(),
"duplicate error code constants detected — check error.rs: {dupes:?}"
);
}
@@ -1761,6 +2063,7 @@ mod tests {
assert!((6000..7000).contains(&E_DISC_READ));
assert!((6000..7000).contains(&E_HALTED));
assert!((6000..7000).contains(&E_MAPFILE_INVALID));
assert!((6000..7000).contains(&E_IMAGE_TRUNCATED));
// AACS (7xxx)
assert!((7000..8000).contains(&E_AACS_NO_KEYS));
assert!((7000..8000).contains(&E_NO_DISC_KEY));
+605 -106
View File
@@ -49,7 +49,7 @@ pub struct DvdTitle {
pub cells: Vec<DvdCell>,
/// Chapter start times in seconds (derived from program map + cell times)
pub chapter_times: Vec<f64>,
/// Subtitle palette from PGC: 16 entries of [padding, Y, Cb, Cr].
/// Subtitle palette from PGC: 16 entries of [padding, Y, Cr, Cb].
pub palette: Option<Vec<[u8; 4]>>,
}
@@ -272,41 +272,102 @@ fn sub_slice(data: &[u8], offset: usize, len: usize) -> Result<&[u8]> {
// ── BCD time parsing ────────────────────────────────────────────────────────
/// Convert DVD BCD playback time (4 bytes) to seconds.
/// The nominal (timecode) frame rate and the exact real-time frame duration
/// implied by a DVD `dvd_time_t` rate flag.
///
/// Format: `[hours_bcd, minutes_bcd, seconds_bcd, frames_and_rate]`
/// This distinction is the whole point of the type: `nominal_fps` is the rate
/// at which the BCD *seconds* field advances, while `frame_num / frame_den` is
/// how long a frame actually lasts. For PAL the two agree (25 frames = 1.000 s).
/// For NTSC they do not: the seconds field advances every 30 frames, but the
/// video runs at 30000/1001 fps, so 30 frames occupy 1001/1000 real seconds.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct DvdRate {
/// Frames per timecode second: 25 (PAL) or 30 (NTSC — *not* 29.97).
pub nominal_fps: u32,
/// Real seconds per frame as an exact rational, `frame_num / frame_den`.
pub frame_num: u32,
/// Denominator of the exact frame duration.
pub frame_den: u32,
}
impl DvdRate {
/// Decode bits 7-6 of `dvd_time_t.frame_u`.
///
/// `0b01` = 25 fps (PAL), `0b11` = 30000/1001 fps (NTSC).
/// `0b00` and `0b10` are unspecified/reserved and yield `None`.
pub fn from_flag(flag: u8) -> Option<DvdRate> {
match flag {
0x01 => Some(DvdRate {
nominal_fps: 25,
frame_num: 1,
frame_den: 25,
}),
0x03 => Some(DvdRate {
nominal_fps: 30,
frame_num: 1001,
frame_den: 30000,
}),
_ => None,
}
}
/// Exact real-time seconds for a count of timecode frames.
pub fn frames_to_secs(&self, frames: u64) -> f64 {
(frames as f64) * (self.frame_num as f64) / (self.frame_den as f64)
}
}
/// Total non-drop-frame timecode frames in a 4-byte DVD BCD time.
///
/// Format: `[hours_bcd, minutes_bcd, seconds_bcd, rate_and_frames]`
/// - Byte 0: hours in BCD (e.g. 0x01 = 1 hour, 0x12 = 12 hours)
/// - Byte 1: minutes in BCD
/// - Byte 2: seconds in BCD
/// - Byte 3: bits 7-6 = frame rate flag (01=25fps, 11=29.97fps),
/// bits 5-0 = frame count in BCD
///
/// Returns 0.0 for invalid BCD digits rather than erroring,
/// since some authoring tools produce malformed time fields.
pub fn bcd_to_secs(bcd: &[u8]) -> f64 {
/// **`dvd_time_t` is a timecode, not elapsed wall-clock time.** The seconds
/// field advances once every [`DvdRate::nominal_fps`] frames. On NTSC discs
/// that is every 30 frames, but 30 frames of 30000/1001 fps video last
/// 1001/1000 s — so reading H:M:S as literal seconds under-reports real time
/// by exactly 0.1% (3.6 s per hour). Callers that need real seconds must go
/// through the frame count, which is what [`bcd_to_secs`] does.
///
/// Returns `None` when the slice is short or the rate flag is unspecified,
/// since without a nominal rate the frame count cannot be interpreted.
pub fn bcd_to_frames(bcd: &[u8]) -> Option<(u64, DvdRate)> {
if bcd.len() < 4 {
return 0.0;
return None;
}
let rate = DvdRate::from_flag((bcd[3] >> 6) & 0x03)?;
let hours = bcd_byte(bcd[0]) as u64;
let minutes = bcd_byte(bcd[1]) as u64;
let seconds = bcd_byte(bcd[2]) as u64;
let frames = bcd_byte(bcd[3] & 0x3F) as u64;
let tc_secs = hours * 3600 + minutes * 60 + seconds;
Some((tc_secs * rate.nominal_fps as u64 + frames, rate))
}
let hours = bcd_byte(bcd[0]);
let minutes = bcd_byte(bcd[1]);
let seconds = bcd_byte(bcd[2]);
let rate_flag = (bcd[3] >> 6) & 0x03;
let frame_count = bcd_byte(bcd[3] & 0x3F);
let fps: f64 = match rate_flag {
0x01 => 25.0,
0x03 => 29.97,
_ => 0.0, // unknown rate — ignore frame contribution
};
let total = (hours as f64) * 3600.0 + (minutes as f64) * 60.0 + (seconds as f64);
if fps > 0.0 {
total + (frame_count as f64) / fps
} else {
total
/// Convert DVD BCD playback time (4 bytes) to real elapsed seconds.
///
/// See [`bcd_to_frames`] for the layout and for why the timecode must be
/// converted through a frame count rather than read as literal seconds.
///
/// Returns 0.0 for invalid BCD digits rather than erroring, since some
/// authoring tools produce malformed time fields. When the rate flag is
/// unspecified the nominal rate is unknowable, so this falls back to reading
/// H:M:S as literal seconds and ignoring the frame field.
pub fn bcd_to_secs(bcd: &[u8]) -> f64 {
match bcd_to_frames(bcd) {
Some((frames, rate)) => rate.frames_to_secs(frames),
None => {
if bcd.len() < 4 {
return 0.0;
}
(bcd_byte(bcd[0]) as f64) * 3600.0
+ (bcd_byte(bcd[1]) as f64) * 60.0
+ (bcd_byte(bcd[2]) as f64)
}
}
}
@@ -360,16 +421,47 @@ pub fn parse_vmg(reader: &mut dyn SectorSource, udf: &UdfFs) -> Result<DvdInfo>
// Parse each VTS IFO
let mut title_sets = Vec::new();
let mut skipped = 0usize;
for (&vts_number, titles_info) in &title_set_map {
match parse_vts(reader, udf, vts_number, titles_info) {
Ok(ts) => title_sets.push(ts),
Err(_) => {
// Skip unreadable title sets — some DVDs have placeholder entries.
// The operator's Stop is not a placeholder title set. Once the
// drive's halt flag is set EVERY command fails with `Halted`, so
// swallowing it here would skip every REMAINING title set in turn
// and return `Ok` with a truncated list — a cancelled scan that
// is indistinguishable from a disc genuinely holding fewer
// titles. `scan_dvd_titles` propagates it to the caller.
Err(Error::Halted) => return Err(Error::Halted),
Err(e) => {
// Some discs carry placeholder TT_SRPT entries for title sets
// that are not really there, so one failure is not fatal. But
// the failure must not be INVISIBLE: every skipped set is a
// title the user will never see, and swallowing the reason made
// a disc that enumerated 38 titles from one image and 10 from
// another look like a scan difference rather than 28 dropped
// reads.
skipped += 1;
tracing::warn!(
target: "freemkv::scan",
vts = vts_number,
titles = titles_info.len(),
error = %e,
"title set could not be parsed; its titles are omitted"
);
continue;
}
}
}
if skipped > 0 {
tracing::warn!(
target: "freemkv::scan",
skipped,
kept = title_sets.len(),
declared = title_set_map.len(),
"some title sets were omitted from the scan"
);
}
Ok(DvdInfo { title_sets })
}
@@ -606,7 +698,16 @@ fn parse_video_attr(data: &[u8]) -> Result<DvdVideoAttr> {
}
/// Parse one audio stream attribute block (8 bytes at `offset`).
fn parse_audio_attr(data: &[u8], offset: usize) -> Result<DvdAudioAttr> {
/// `pub(crate)` for the CROSS-MODULE tests only. The sole production caller is
/// `parse_vts_attributes` in this file; `src/mux/mkv.rs`'s `#[cfg(test)]` block
/// calls it directly so its language-mapping tests run the real parser over real
/// on-disc IFO bytes end to end, instead of a hand-built `DvdAudioAttr` that
/// could agree with the muxer while both disagree with the disc. Narrowing this
/// would mean either a `#[cfg(test)]`/`#[cfg(not(test))]` pair of signatures
/// that can drift apart, or moving those tests away from the code they exist to
/// pin — both worse than the widened crate-internal visibility, which reaches no
/// public API.
pub(crate) fn parse_audio_attr(data: &[u8], offset: usize) -> Result<DvdAudioAttr> {
let b0 = byte_at(data, offset)?;
let b1 = byte_at(data, offset + 1)?;
@@ -633,25 +734,9 @@ fn parse_audio_attr(data: &[u8], offset: usize) -> Result<DvdAudioAttr> {
let channels = (b1 & 0x07) + 1; // (channels - 1) in low 3 bits of byte 1
// Language code: bytes 2-3 as ISO 639
// Language code: bytes 2-3 as ISO 639-1 (the DVD-Video spec's form).
let lang_bytes = sub_slice(data, offset + 2, 2)?;
let language = if lang_bytes[0] >= b'a'
&& lang_bytes[0] <= b'z'
&& lang_bytes[1] >= b'a'
&& lang_bytes[1] <= b'z'
{
String::from_utf8_lossy(lang_bytes).to_string()
} else if lang_bytes[0] == 0 && lang_bytes[1] == 0 {
String::new()
} else {
// Try to interpret as printable ASCII
let s: String = lang_bytes
.iter()
.filter(|&&b| b.is_ascii_alphanumeric())
.map(|&b| b as char)
.collect();
s
};
let language = dvd_lang_to_iso639_2(&parse_raw_dvd_lang_bytes(lang_bytes));
Ok(DvdAudioAttr {
codec,
@@ -700,9 +785,27 @@ fn assign_audio_sub_stream_ids(streams: &mut [DvdAudioAttr]) {
/// Parse one subtitle stream attribute block (6 bytes at `offset`).
fn parse_subtitle_attr(data: &[u8], offset: usize) -> Result<DvdSubtitleAttr> {
// Language code: bytes 2-3 as ISO 639
// Language code: bytes 2-3 as ISO 639-1 (the DVD-Video spec's form).
let lang_bytes = sub_slice(data, offset + 2, 2)?;
let language = if lang_bytes[0] >= b'a'
let language = dvd_lang_to_iso639_2(&parse_raw_dvd_lang_bytes(lang_bytes));
Ok(DvdSubtitleAttr { language })
}
/// Decode the raw 2-byte on-disc language code shared by the audio and
/// subtitle attribute blocks. A pair of lowercase a-z bytes is taken
/// verbatim (the ISO 639-1 code the DVD-Video spec puts there); an all-zero
/// pair means unspecified (empty string); anything else falls through to an
/// ASCII-alphanumeric salvage — letters (either case) and digits are kept,
/// everything else (control bytes, punctuation, high bytes from a corrupt or
/// hostile disc) is dropped.
///
/// The salvage is deliberately not narrowed to a-z: whatever survives is only
/// ever a lookup key for [`dvd_lang_to_iso639_2`], which degrades anything it
/// does not recognize to `und`, so a stray `X` or `5` costs nothing and cannot
/// reach an output stream as a language code.
fn parse_raw_dvd_lang_bytes(lang_bytes: &[u8]) -> String {
if lang_bytes[0] >= b'a'
&& lang_bytes[0] <= b'z'
&& lang_bytes[1] >= b'a'
&& lang_bytes[1] <= b'z'
@@ -711,15 +814,40 @@ fn parse_subtitle_attr(data: &[u8], offset: usize) -> Result<DvdSubtitleAttr> {
} else if lang_bytes[0] == 0 && lang_bytes[1] == 0 {
String::new()
} else {
let s: String = lang_bytes
lang_bytes
.iter()
.filter(|&&b| b.is_ascii_alphanumeric())
.map(|&b| b as char)
.collect();
s
};
.collect()
}
}
Ok(DvdSubtitleAttr { language })
/// Convert a DVD IFO audio/subtitle attribute's language code — ISO 639-1
/// (2 lowercase letters) per the DVD-Video spec, or empty when unspecified —
/// to the ISO 639-2 form every downstream consumer (`disc::AudioStream` /
/// `disc::SubtitleStream::language`, and in turn Matroska's `Language`
/// element per RFC 9559 §12 and the MP4 sink's `mdhd` language) requires.
///
/// Uses `labels::vocab::iso639_1_to_iso639_2`, which spans the WHOLE of ISO
/// 639-1 (plus the withdrawn spellings `iw`/`in`/`ji` that DVD-Video's
/// frozen-1988 language list still puts on disc). The narrower
/// `vocab::menu_lang` table is deliberately NOT used here: it exists for
/// Blu-ray menu-graphic filename tokens and knows only 25 languages, so a
/// Region-2 disc's Romanian, Bulgarian, Croatian, Serbian, Slovak, Slovenian,
/// Hebrew, Estonian, Latvian, Lithuanian and Icelandic tracks would all fold
/// onto `und` together. DVD streams carry an empty `label`, so the language
/// is the only thing distinguishing one subtitle track from the next — a
/// valid code that is identical for six tracks is worse for the user than the
/// invalid one it replaced. Both tables normalize to ISO 639-2/T, so they
/// agree wherever they overlap.
///
/// An empty or unrecognized code degrades to `"und"` (ISO 639-2 / Matroska's
/// own "undetermined" value) — a valid element value — rather than passing
/// through an invalid 2-letter code or an empty string. Never guesses.
fn dvd_lang_to_iso639_2(raw: &str) -> String {
crate::labels::vocab::iso639_1_to_iso639_2(raw)
.unwrap_or("und")
.to_string()
}
// ── PGC parser ──────────────────────────────────────────────────────────────
@@ -740,16 +868,36 @@ fn parse_pgcit(
let entries_start = pgcit_offset + 8;
let mut titles = Vec::new();
// Every `continue` below drops a title the user will never see. None of
// them may be silent: `parse_vmg` already counts and warns per skipped
// title SET, and this function was the remaining place where a disc could
// quietly report fewer titles than it has.
let mut skipped = 0usize;
for &(chapter_count, vts_title_num) in titles_info {
// VTS title numbers are 1-based; map to PGC index (typically 1:1)
let pgc_index = vts_title_num.saturating_sub(1) as usize;
if pgc_index >= num_pgcs as usize {
skipped += 1;
tracing::warn!(
target: "freemkv::scan",
pgc_index,
num_pgcs,
"title points past the end of the PGC table; its title is omitted"
);
continue;
}
let entry_offset = entries_start + pgc_index * 8;
if entry_offset + 8 > data.len() {
skipped += 1;
tracing::warn!(
target: "freemkv::scan",
pgc_index,
entry_offset,
len = data.len(),
"PGC entry table is truncated; this title is omitted"
);
continue;
}
@@ -761,13 +909,38 @@ fn parse_pgcit(
match parse_pgc(data, pgc_abs, chapter_count) {
Ok(title) => titles.push(title),
// By design: a single unparseable PGC (truncated/corrupt entry,
// authoring-tool quirk) must not lose the whole title list.
// Skip it and keep collecting the titles that do parse.
Err(_) => continue,
Err(e) => {
// By design a single unparseable PGC (truncated/corrupt entry,
// authoring-tool quirk) must not lose the whole title list. But
// "not fatal" is not the same as "not worth saying": every PGC
// skipped here is a title the user will never see, and this was
// the only remaining silent one — `parse_vmg` above already
// counts and warns per skipped title SET for exactly this
// reason. A disc quietly reporting fewer titles than it has is
// the failure this release exists to stop.
skipped += 1;
tracing::warn!(
target: "freemkv::scan",
pgc = pgc_index + 1,
of = num_pgcs,
error = %e,
"PGC could not be parsed; its title is omitted"
);
continue;
}
}
}
if skipped > 0 {
tracing::warn!(
target: "freemkv::scan",
skipped,
kept = titles.len(),
declared = titles_info.len(),
"some titles were omitted from this title set"
);
}
Ok(titles)
}
@@ -838,15 +1011,33 @@ fn parse_pgc(data: &[u8], pgc_offset: usize, chapters: u16) -> Result<DvdTitle>
let mut times = Vec::new();
if pgm_map_offset > 0 && nr_of_programs > 0 && cell_playback_offset > 0 {
let pgm_base = pgc_offset + pgm_map_offset;
// Collect cell durations
let mut cell_durations = Vec::with_capacity(num_cells);
// Collect per-cell durations as exact timecode FRAME counts. Summing
// frames (integers) and converting once per chapter keeps every mark
// on an exact frame boundary — summing f64 seconds would let rounding
// drift accumulate across a two-hour title.
//
// A cell whose rate flag is unspecified has no frame count, so it
// contributes plain seconds instead; that is tracked separately and
// added on top. In practice a PGC never mixes rates.
let mut cell_frames: Vec<(u64, f64)> = Vec::with_capacity(num_cells);
let mut rate: Option<DvdRate> = None;
let cell_base = pgc_offset + cell_playback_offset;
for i in 0..num_cells {
let co = cell_base + i * 24;
if co + 8 <= data.len() {
cell_durations.push(bcd_to_secs(&data[co + 4..co + 8]));
} else {
cell_durations.push(0.0);
if co + 8 > data.len() {
cell_frames.push((0, 0.0));
continue;
}
let t = &data[co + 4..co + 8];
match bcd_to_frames(t) {
Some((f, r)) => {
// First rate wins; a mixed-rate PGC is malformed, and
// reinterpreting earlier cells would be worse than
// staying on the rate the title started in.
rate.get_or_insert(r);
cell_frames.push((f, 0.0));
}
None => cell_frames.push((0, bcd_to_secs(t))),
}
}
// Program map: each byte is the first cell number (1-based) for that program
@@ -856,17 +1047,19 @@ fn parse_pgc(data: &[u8], pgc_offset: usize, chapters: u16) -> Result<DvdTitle>
}
let first_cell = data[pgm_base + p] as usize;
// Chapter time = sum of cell durations before this program's first cell.
// Clamp to cell_durations.len(): a crafted/corrupt IFO can set first_cell
// Clamp to cell_frames.len(): a crafted/corrupt IFO can set first_cell
// beyond the actual cell count, which would panic the slice index.
let end = first_cell.saturating_sub(1).min(cell_durations.len());
let time: f64 = cell_durations[..end].iter().sum();
let end = first_cell.saturating_sub(1).min(cell_frames.len());
let frames: u64 = cell_frames[..end].iter().map(|&(f, _)| f).sum();
let extra: f64 = cell_frames[..end].iter().map(|&(_, s)| s).sum();
let time = rate.map_or(0.0, |r| r.frames_to_secs(frames)) + extra;
times.push(time);
}
}
times
};
// Extract subtitle palette at PGC offset 0xA4: 16 colors × 4 bytes [padding, Y, Cb, Cr]
// Extract subtitle palette at PGC offset 0xA4: 16 colors × 4 bytes [padding, Y, Cr, Cb]
let palette = if pgc_offset + 0xA4 + 64 <= data.len() {
let mut colors = Vec::with_capacity(16);
for i in 0..16 {
@@ -982,12 +1175,60 @@ mod tests {
#[test]
fn bcd_to_secs_with_frames() {
// 0 hours, 1 minute, 30 seconds, 15 frames at 29.97fps
// 0 hours, 1 minute, 30 seconds, 15 frames of NTSC timecode.
let bcd = [0x00, 0x01, 0x30, 0b11_010101];
let secs = bcd_to_secs(&bcd);
// 0b010101 = 0x15, BCD = 15 frames
let expected = 0.0 + 60.0 + 30.0 + 15.0 / 29.97;
assert!((secs - expected).abs() < 0.01, "got {}", secs);
// 0b010101 = 0x15, BCD = 15 frames.
// Timecode 00:01:30:15 is 90*30 + 15 = 2715 frames, and each frame
// lasts 1001/30000 s, so real time is 2715*1001/30000 = 90.5905 s
// (NOT 90.5 s — NTSC timecode runs 0.1% slow against the clock).
let expected = 2715.0 * 1001.0 / 30000.0;
assert!((secs - expected).abs() < 1e-9, "got {}", secs);
assert!((secs - 90.5905).abs() < 1e-9, "got {}", secs);
}
/// The NTSC 0.1% pull-down must be applied to the whole timecode, not just
/// the frame field. Regression test for issue freemkv#25: a title's chapter marks
/// drifted ~3.6 s per hour because H:M:S was read as literal seconds.
#[test]
fn bcd_ntsc_timecode_is_not_literal_seconds() {
// One hour of NTSC timecode = 3603.6 s of real time.
let one_hour = [0x01, 0x00, 0x00, 0b11_000000];
assert!((bcd_to_secs(&one_hour) - 3603.6).abs() < 1e-6);
// 00:01:02:00 -> 1860 frames -> 62.062 s (the issue freemkv#25 chapter 2 value).
let ch2 = [0x00, 0x01, 0x02, 0b11_000000];
assert!((bcd_to_secs(&ch2) - 62.062).abs() < 1e-9);
// A 20-minute NTSC episode cell is 00:19:58:24 = 35964 frames.
let ep = [0x00, 0x19, 0x58, 0b11_100100];
assert_eq!(bcd_to_frames(&ep).unwrap().0, 35964);
assert!((bcd_to_secs(&ep) - 1199.9988).abs() < 1e-6);
}
/// PAL timecode is exact (25 frames = 1.000 s), so the fix must leave every
/// PAL value bit-identical.
#[test]
fn bcd_pal_unaffected_by_ntsc_fix() {
let pal = [0x01, 0x23, 0x45, 0b01_000000];
assert!((bcd_to_secs(&pal) - (3600.0 + 23.0 * 60.0 + 45.0)).abs() < 1e-9);
let pal_frames = [0x00, 0x00, 0x10, 0b01_010010]; // 10 s + 12 frames
assert!((bcd_to_secs(&pal_frames) - 10.48).abs() < 1e-9);
let r = DvdRate::from_flag(0x01).unwrap();
assert_eq!((r.nominal_fps, r.frame_num, r.frame_den), (25, 1, 25));
}
#[test]
fn dvd_rate_from_flag_rejects_reserved() {
assert!(DvdRate::from_flag(0x00).is_none());
assert!(DvdRate::from_flag(0x02).is_none());
let n = DvdRate::from_flag(0x03).unwrap();
assert_eq!((n.nominal_fps, n.frame_num, n.frame_den), (30, 1001, 30000));
}
#[test]
fn bcd_to_frames_none_on_short_or_unknown_rate() {
assert!(bcd_to_frames(&[0x00, 0x00]).is_none());
assert!(bcd_to_frames(&[0x00, 0x00, 0x00, 0b00_000000]).is_none());
assert!(bcd_to_frames(&[0x00, 0x00, 0x00, 0b10_000000]).is_none());
}
#[test]
@@ -1133,12 +1374,16 @@ mod tests {
pgc[co + 23] = 144; // last sector = 400
let title = parse_pgc(&pgc, 0, 5).unwrap();
let expected = 1.0 * 3600.0 + 59.0 * 60.0 + 30.0;
// 01:59:30:00 is NTSC *timecode*, so 7170 timecode seconds = 215100
// frames, and the real running time is 215100 * 1001/30000 = 7177.17 s.
// Reading the BCD as literal seconds (7170.0) was the issue freemkv#25 bug.
let expected = 7170.0 * 30.0 * 1001.0 / 30000.0;
assert!(
(title.duration_secs - expected).abs() < 0.1,
(title.duration_secs - expected).abs() < 1e-6,
"expected ~{expected}s, got {}s",
title.duration_secs
);
assert!((title.duration_secs - 7177.17).abs() < 1e-6);
assert_eq!(title.chapters, 5);
assert_eq!(title.cells.len(), 2);
assert_eq!(title.cells[0].first_sector, 100);
@@ -1239,7 +1484,7 @@ mod tests {
data[0] = 0x00;
// b1: bits 2-0=101 (channels-1=5) => 0x05
data[1] = 0x05;
// language "en"
// on-disc language "en" (ISO 639-1) -> parsed as ISO 639-2 "eng"
data[2] = b'e';
data[3] = b'n';
@@ -1247,7 +1492,7 @@ mod tests {
assert_eq!(attr.codec, Codec::Ac3);
assert_eq!(attr.sample_rate, 48000);
assert_eq!(attr.channels, 6);
assert_eq!(attr.language, "en");
assert_eq!(attr.language, "eng");
}
#[test]
@@ -1377,7 +1622,7 @@ mod tests {
assert_eq!(attr.codec, Codec::Dts);
assert_eq!(attr.sample_rate, 96000);
assert_eq!(attr.channels, 2);
assert_eq!(attr.language, "fr");
assert_eq!(attr.language, "fra");
}
// ─────────────────────────────────────────────────────────────────────
@@ -1393,7 +1638,7 @@ mod tests {
// 0h 0m 0s, 12 frames at 25fps → 12/25 = 0.48s.
let bcd = [0x00, 0x00, 0x00, 0b01_010010]; // frame BCD 0x12 = 12
let secs = bcd_to_secs(&bcd);
assert!((secs - 12.0 / 25.0).abs() < 0.001, "got {secs}");
assert!((secs - 12.0 / 25.0).abs() < 1e-9, "got {secs}");
}
/// BCD rate_flag 0b00 (and 0b10) → fps 0.0 → frame count ignored
@@ -1415,8 +1660,9 @@ mod tests {
#[test]
fn bcd_frame_count_masks_rate_bits() {
let bcd = [0x00, 0x00, 0x00, 0b11_100101]; // 0x25 BCD = 25 frames
assert_eq!(bcd_to_frames(&bcd).unwrap().0, 25);
let secs = bcd_to_secs(&bcd);
assert!((secs - 25.0 / 29.97).abs() < 0.001, "got {secs}");
assert!((secs - 25.0 * 1001.0 / 30000.0).abs() < 1e-9, "got {secs}");
}
/// BCD hours can exceed 12 (long titles): 0x12 BCD = 12 → but test a
@@ -1490,15 +1736,17 @@ mod tests {
}
/// Audio language bytes [offset+2..+4]: when both bytes are 0x00 the
/// language is the empty string (unspecified), per source.
/// on-disc code is unspecified, and `dvd_lang_to_iso639_2` maps that to
/// the valid ISO 639-2 "und" (undetermined) rather than an empty string,
/// which is not a legal Matroska `Language` element value.
#[test]
fn audio_attr_zero_language_is_empty() {
fn audio_attr_zero_language_becomes_und() {
let mut data = vec![0u8; 8];
data[0] = 0x00;
data[2] = 0x00;
data[3] = 0x00;
let attr = parse_audio_attr(&data, 0).unwrap();
assert_eq!(attr.language, "");
assert_eq!(attr.language, "und");
}
/// Audio sample_rate flag (b0>>3 & 0x03): 0=48kHz, 1=96kHz, else 48kHz.
@@ -1519,11 +1767,11 @@ mod tests {
data[2] = b'd';
data[3] = b'e';
let attr = parse_subtitle_attr(&data, 0).unwrap();
assert_eq!(attr.language, "de");
assert_eq!(attr.language, "deu");
let zero = vec![0u8; 6];
let attr2 = parse_subtitle_attr(&zero, 0).unwrap();
assert_eq!(attr2.language, "");
assert_eq!(attr2.language, "und");
}
/// assign_audio_sub_stream_ids: MP1/MP2 and other non-private-stream-1
@@ -1609,8 +1857,8 @@ mod tests {
assert_eq!(title.cells[1].first_sector, 20);
}
/// parse_pgc palette: at PGC+0xA4, 16 colors × 4 bytes [pad, Y, Cb, Cr].
/// A palette with at least one non-zero Y/Cb/Cr is returned as Some;
/// parse_pgc palette: at PGC+0xA4, 16 colors × 4 bytes [pad, Y, Cr, Cb].
/// A palette with at least one non-zero Y/Cr/Cb is returned as Some;
/// an all-zero palette returns None (source filters empty palettes).
#[test]
fn pgc_palette_present_and_empty() {
@@ -1630,14 +1878,14 @@ mod tests {
assert!(title2.palette.is_none());
}
/// parse_pgc palette layout: each color is [padding, Y, Cb, Cr] and the
/// parse_pgc palette layout: each color is [padding, Y, Cr, Cb] and the
/// "non-empty" test ignores the padding byte (index 0). A palette whose
/// ONLY non-zero bytes are padding must still be treated as empty (None).
#[test]
fn pgc_palette_padding_only_is_empty() {
let mut pgc = vec![0u8; 0xEA];
pgc[0x03] = 0;
// Set padding byte (index 0) of color 0 non-zero, but Y/Cb/Cr zero.
// Set padding byte (index 0) of color 0 non-zero, but Y/Cr/Cb zero.
pgc[0xA4] = 0xFF;
let title = parse_pgc(&pgc, 0, 1).unwrap();
assert!(
@@ -1688,6 +1936,93 @@ mod tests {
);
}
/// Regression test for issue freemkv#25 (NTSC chapter drift).
///
/// Builds a PGC from the real cell table of the DOLL_MASTERS_7 disc in the
/// bug report and checks the chapter marks against MakeMKV's values. Before
/// the fix, chapter 14 landed 4.019 s early; the drift was proportional to
/// elapsed time (0.1%), so a short synthetic fixture would not have caught it.
#[test]
fn pgc_chapter_times_ntsc_no_pulldown_drift() {
// (minutes, seconds, frames) of NTSC non-drop-frame timecode per cell.
let cells: [(u8, u8, u8); 13] = [
(1, 2, 0),
(4, 44, 12),
(1, 32, 11),
(13, 42, 1),
(0, 59, 28),
(0, 31, 8),
(1, 2, 0),
(19, 58, 24),
(0, 59, 28),
(0, 31, 8),
(1, 2, 0),
(19, 58, 26),
(0, 59, 28),
];
// Program map: 14 chapters, one per cell boundary (1-based first cell).
let pgm: [u8; 14] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14];
fn bcd(v: u8) -> u8 {
((v / 10) << 4) | (v % 10)
}
let mut pgc = vec![0u8; 0xEA];
pgc[0x02] = pgm.len() as u8;
pgc[0x03] = cells.len() as u8;
let pgm_off: u16 = 0xEA;
pgc[0xE6] = (pgm_off >> 8) as u8;
pgc[0xE7] = pgm_off as u8;
let cell_off: u16 = pgm_off + pgm.len() as u16;
pgc[0xE8] = (cell_off >> 8) as u8;
pgc[0xE9] = cell_off as u8;
pgc.resize(cell_off as usize + cells.len() * 24, 0);
pgc[pgm_off as usize..pgm_off as usize + pgm.len()].copy_from_slice(&pgm);
for (i, &(m, s, f)) in cells.iter().enumerate() {
let co = cell_off as usize + i * 24;
pgc[co + 4] = 0x00; // hours
pgc[co + 5] = bcd(m);
pgc[co + 6] = bcd(s);
pgc[co + 7] = 0xC0 | bcd(f); // rate flag 0b11 = NTSC
}
let title = parse_pgc(&pgc, 0, pgm.len() as u16).unwrap();
// MakeMKV / source truth, in seconds.
let expected = [
0.0,
62.062,
346.7464,
439.205_433_333,
1262.0608,
1_322.054_066_666,
1353.352,
1415.414,
2615.4128,
2_675.406_066_666,
2706.704,
2768.766,
3_968.831_533_333,
4028.8248,
];
assert_eq!(title.chapter_times.len(), expected.len());
for (i, (&got, &want)) in title.chapter_times.iter().zip(&expected).enumerate() {
assert!(
(got - want).abs() < 1e-6,
"chapter {} drifted: got {got}, want {want}",
i + 1
);
}
// Every mark must sit on an exact 30000/1001 frame boundary.
for (i, &t) in title.chapter_times.iter().enumerate() {
let frames = t * 30000.0 / 1001.0;
assert!(
(frames - frames.round()).abs() < 1e-6,
"chapter {} not frame-aligned: {t}",
i + 1
);
}
}
/// parse_pgc duration: when the PGC-level BCD time is NON-zero it is
/// used directly and NOT overwritten by cell-sum recomputation
/// (the recompute only fires when duration_secs == 0.0).
@@ -1988,10 +2323,13 @@ mod tests {
/// two bytes at +2. Only a pair of lowercase a-z bytes is taken verbatim;
/// anything else falls through to the ASCII-alphanumeric salvage, which
/// keeps only the usable characters. A byte outside a-z must never end up
/// in the language string.
/// in the raw salvaged string. This exercises `parse_raw_dvd_lang_bytes`
/// directly — the byte-level salvage step — separately from the ISO
/// 639-1 -> 639-2 mapping `parse_audio_attr`/`parse_subtitle_attr` apply
/// on top (see `dvd_two_letter_and_malformed_language_becomes_iso639_2`).
#[test]
fn language_code_rejects_non_lowercase_bytes() {
// (byte0, byte1, expected language)
// (byte0, byte1, expected raw salvage)
let cases: [(u8, u8, &str); 8] = [
(b'e', b'n', "en"), // both in range → verbatim
(0x21, b'n', "n"), // '!' is below 'a'
@@ -2002,6 +2340,31 @@ mod tests {
(0x00, b'E', "E"), // only the first byte is zero
(0x00, 0x00, ""), // both zero → unset
];
for (b0, b1, want) in cases {
assert_eq!(
parse_raw_dvd_lang_bytes(&[b0, b1]),
want,
"raw language salvage for ({b0:#04x}, {b1:#04x})"
);
}
}
/// The full pipeline `parse_audio_attr`/`parse_subtitle_attr` apply on
/// top of the raw salvage: a valid ISO 639-1 code maps to its ISO 639-2
/// equivalent, and anything the raw salvage does NOT produce a mapped
/// code for (empty, or a single leftover letter from a malformed byte
/// pair) degrades to "und" — never an invalid 2-letter/1-letter code and
/// never an empty string, both of which are illegal Matroska `Language`
/// element values (RFC 9559 §12).
#[test]
fn dvd_two_letter_and_malformed_language_becomes_iso639_2() {
// (byte0, byte1, expected final language)
let cases: [(u8, u8, &str); 4] = [
(b'e', b'n', "eng"), // valid ISO 639-1 → mapped
(0x21, b'n', "und"), // malformed → salvage "n", unmapped → und
(b'E', 0x00, "und"), // malformed → salvage "E", unmapped → und
(0x00, 0x00, "und"), // unspecified → und
];
for (b0, b1, want) in cases {
let mut audio = vec![0u8; 8];
audio[2] = b0;
@@ -2065,7 +2428,14 @@ mod tests {
d
}
/// BCD playback time of `secs` seconds (< 60) at the 29.97 fps flag.
/// BCD playback time of `secs` TIMECODE seconds (< 60) at the NTSC rate
/// flag.
///
/// The argument is timecode, not real time: every fixture built from this
/// helper carries the 0b11 flag, so `secs` timecode seconds are
/// `secs * 30` frames and last `secs * 1.001` real seconds. Expectations
/// below are written in real seconds and so read 1.001x the argument —
/// `bcd_secs(10)` is 10.01 s, not 10 s.
fn bcd_secs(secs: u8) -> [u8; 4] {
assert!(secs < 60);
[0, 0, ((secs / 10) << 4) | (secs % 10), 0b11_000000]
@@ -2088,12 +2458,13 @@ mod tests {
);
let title = parse_pgc(&pgc, 0, 3).unwrap();
assert_eq!(title.cells.len(), 3);
assert_eq!(title.cells[0].duration_secs, 10.0);
assert_eq!(title.cells[1].duration_secs, 20.0);
assert_eq!(title.cells[2].duration_secs, 31.0);
assert_eq!(
title.duration_secs, 61.0,
"recomputed duration must be the sum of the distinct cell times"
assert_eq!(title.cells[0].duration_secs, 10.01);
assert_eq!(title.cells[1].duration_secs, 20.02);
assert_eq!(title.cells[2].duration_secs, 31.031);
assert!(
(title.duration_secs - 61.061).abs() < 1e-9,
"recomputed duration must be the sum of the distinct cell times, got {}",
title.duration_secs
);
}
@@ -2114,7 +2485,7 @@ mod tests {
None,
);
let title = parse_pgc(&pgc, 0, 3).unwrap();
assert_eq!(title.chapter_times, vec![0.0, 10.0, 30.0]);
assert_eq!(title.chapter_times, vec![0.0, 10.01, 30.03]);
}
/// A program map offset of 0 means there is no program map, so no chapter
@@ -2169,7 +2540,7 @@ mod tests {
let title = parse_pgc(&pgc, 0, 2).unwrap();
assert_eq!(title.cells.len(), 2, "only the readable cells are kept");
// Program 2 starts at cell 2, so its time is cell 0's duration.
assert_eq!(title.chapter_times, vec![0.0, 10.0]);
assert_eq!(title.chapter_times, vec![0.0, 10.01]);
}
/// A program map that runs past the end of the data must stop at the
@@ -2194,11 +2565,11 @@ mod tests {
"the program map walk must stop exactly at the buffer end"
);
assert_eq!(title.chapter_times[0], 0.0);
assert_eq!(title.chapter_times[1], 10.0);
assert_eq!(title.chapter_times[1], 10.01);
}
/// The subtitle palette is 16 entries of 4 bytes at PGC+0xA4, each
/// `[padding, Y, Cb, Cr]`. Every byte of every entry is distinct here, so
/// `[padding, Y, Cr, Cb]`. Every byte of every entry is distinct here, so
/// a wrong stride, a wrong base or a shifted component shows up.
#[test]
// The loop variable is the DOMAIN VALUE being checked (a palette entry number), not a
@@ -2226,7 +2597,7 @@ mod tests {
}
}
/// A palette is "present" when ANY of Y, Cb or Cr is non-zero in ANY
/// A palette is "present" when ANY of Y, Cr or Cb is non-zero in ANY
/// entry — a single non-zero chroma component is enough. Only the
/// padding byte [0] is ignored.
#[test]
@@ -2284,19 +2655,19 @@ mod tests {
// vts_title_num is 1-based: title 2 → PGC index 1.
let titles = parse_pgcit(&data, 0, &[(5, 2)]).unwrap();
assert_eq!(titles.len(), 1);
assert_eq!(titles[0].duration_secs, 22.0);
assert_eq!(titles[0].duration_secs, 22.022);
assert_eq!(titles[0].cells[0].first_sector, 50);
let titles = parse_pgcit(&data, 0, &[(5, 1)]).unwrap();
assert_eq!(titles.len(), 1);
assert_eq!(titles[0].duration_secs, 11.0);
assert_eq!(titles[0].duration_secs, 11.011);
assert_eq!(titles[0].cells[0].first_sector, 0);
// Both titles, in order.
let titles = parse_pgcit(&data, 0, &[(5, 1), (7, 2)]).unwrap();
assert_eq!(titles.len(), 2);
assert_eq!(titles[0].duration_secs, 11.0);
assert_eq!(titles[1].duration_secs, 22.0);
assert_eq!(titles[0].duration_secs, 11.011);
assert_eq!(titles[1].duration_secs, 22.022);
}
/// A VTS_PGCIT whose 8-byte header ends exactly at the end of the data is
@@ -2341,4 +2712,132 @@ mod tests {
assert_eq!(map[&1], vec![(7u16, 1u8), (13, 2)]);
assert_eq!(map[&2], vec![(0x0102u16, 1u8)]);
}
// ── TT_SRPT: the title table (`parse_tt_srpt` / `parse_vmg`) ────────────
//
// This is the table that says which titles a DVD has, and it had no tests
// at all until 1.6.2 — while being at the centre of both DVD stories in
// 1.6.1. The CSS descramble bug destroyed the sector carrying it, so a disc
// enumerating 38 titles produced an image enumerating 10, silently, at exit
// 0. And `dir://` was pulled from the release on 2026-08-05 because a
// folder produced the wrong title list — the same table, reached through a
// synthesized image.
//
// Both were found on real discs. Nothing hermetic covered the table itself.
/// Build a VMG with a TT_SRPT at `tt_srpt_sector` declaring `entries`
/// of `(num_chapters, vts_number, vts_title_num)`.
fn vmg_with_tt_srpt(tt_srpt_sector: u32, entries: &[(u16, u8, u8)]) -> Vec<u8> {
let off = tt_srpt_sector as usize * crate::consts::SECTOR_BYTES;
let mut v = vec![0u8; off + 8 + entries.len() * 12 + 16];
v[0..12].copy_from_slice(b"DVDVIDEO-VMG");
v[0xC4..0xC8].copy_from_slice(&tt_srpt_sector.to_be_bytes());
v[off..off + 2].copy_from_slice(&(entries.len() as u16).to_be_bytes());
for (i, &(chapters, vts, title)) in entries.iter().enumerate() {
let b = off + 8 + i * 12;
v[b + 2..b + 4].copy_from_slice(&chapters.to_be_bytes());
v[b + 6] = vts;
v[b + 7] = title;
}
v
}
#[test]
fn tt_srpt_groups_titles_by_their_title_set() {
// Three titles: two in VTS 1, one in VTS 2 — the ordinary shape.
let vmg = vmg_with_tt_srpt(1, &[(12, 1, 1), (5, 1, 2), (20, 2, 1)]);
let map = parse_tt_srpt(&vmg, crate::consts::SECTOR_BYTES).expect("parse");
assert_eq!(map.len(), 2, "two title sets");
assert_eq!(
map[&1],
vec![(12, 1), (5, 2)],
"VTS 1 keeps both titles in order"
);
assert_eq!(map[&2], vec![(20, 1)], "VTS 2 keeps its one");
}
/// The pointer at 0xC4 is a SECTOR offset from the start of VIDEO_TS.IFO,
/// not a byte offset. Reading it as bytes would find the table at 1/2048th
/// of its real position — which is a zero-filled region on most discs, so
/// it would report a disc with no titles rather than fail loudly.
#[test]
fn tt_srpt_pointer_is_a_sector_offset_not_a_byte_offset() {
for sector in [1u32, 2, 5] {
let vmg = vmg_with_tt_srpt(sector, &[(3, 1, 1)]);
let at_sector = parse_tt_srpt(&vmg, sector as usize * crate::consts::SECTOR_BYTES)
.expect("reading at the sector offset works");
assert_eq!(at_sector[&1], vec![(3, 1)], "sector {sector}");
// Reading the same pointer as a BYTE offset lands 1/2048th of the
// way in — inside the header here — and yields a different table.
// It does not error: it silently reports the wrong titles, which is
// how 28 dropped titles looked like a scan difference rather than a
// failure.
let as_bytes = parse_tt_srpt(&vmg, sector as usize).expect("does not error");
assert_ne!(
as_bytes, at_sector,
"sector {sector}: a byte-offset read must not coincidentally \
agree with the correct one, or this test proves nothing"
);
}
}
/// A VTS number of 0 is not a title set; entries carrying it are padding.
#[test]
fn tt_srpt_skips_entries_with_no_title_set() {
let vmg = vmg_with_tt_srpt(1, &[(9, 0, 1), (4, 3, 1)]);
let map = parse_tt_srpt(&vmg, crate::consts::SECTOR_BYTES).expect("parse");
assert_eq!(map.len(), 1, "the vts=0 entry is dropped");
assert_eq!(map[&3], vec![(4, 1)]);
}
/// The same (title set, title) twice is one title, not two. Real discs
/// carry duplicate TT_SRPT rows.
#[test]
fn tt_srpt_collapses_a_duplicate_title() {
let vmg = vmg_with_tt_srpt(1, &[(7, 2, 1), (7, 2, 1), (8, 2, 2)]);
let map = parse_tt_srpt(&vmg, crate::consts::SECTOR_BYTES).expect("parse");
assert_eq!(
map[&2],
vec![(7, 1), (8, 2)],
"the repeat is dropped, the distinct one kept"
);
}
/// A truncated table parses what is there rather than failing the whole
/// disc — one unreadable row should not cost a user every other title.
#[test]
fn tt_srpt_truncated_keeps_the_entries_it_can_read() {
let mut vmg = vmg_with_tt_srpt(1, &[(1, 1, 1), (2, 2, 1), (3, 3, 1)]);
vmg.truncate(crate::consts::SECTOR_BYTES + 8 + 12 + 6); // 1 whole entry + a fragment
let map = parse_tt_srpt(&vmg, crate::consts::SECTOR_BYTES).expect("parse");
assert_eq!(map.len(), 1, "the complete entry survives");
assert_eq!(map[&1], vec![(1, 1)]);
}
/// The count is an untrusted u16 off the disc. DVD-Video caps a disc at 99
/// titles, and each entry re-parses a PGC into a full DvdTitle — so an
/// unclamped 65535 is a ~540 MB allocation from a ~800 KB crafted file.
#[test]
fn tt_srpt_clamps_an_absurd_declared_title_count() {
// The fixture must actually CONTAIN more entries than the cap, or the
// walk stops when the buffer runs out and the clamp is never what
// bounded it. An earlier version of this test declared u16::MAX over a
// two-kilobyte fixture: the loop broke on `base + 12 > len` after three
// iterations and the assertion held with the clamp deleted entirely.
const PRESENT: usize = MAX_TT_SRPT_TITLES + 40;
let entries: Vec<(u16, u8, u8)> = (0..PRESENT)
.map(|i| (1u16, 1u8, (i % 250 + 1) as u8))
.collect();
let mut vmg = vmg_with_tt_srpt(1, &entries);
let off = crate::consts::SECTOR_BYTES;
vmg[off..off + 2].copy_from_slice(&u16::MAX.to_be_bytes());
let map = parse_tt_srpt(&vmg, off).expect("parse");
let total: usize = map.values().map(|v| v.len()).sum();
assert!(
total <= MAX_TT_SRPT_TITLES,
"a declared count of u16::MAX over {PRESENT} real entries must be \
clamped to {MAX_TT_SRPT_TITLES}, got {total}"
);
}
}
+3 -3
View File
@@ -23,7 +23,7 @@
use std::fs::File;
use std::os::unix::io::AsRawFd;
pub(super) fn hint_sequential(file: &File, _len_bytes: u64) {
pub(crate) fn hint_sequential(file: &File, _len_bytes: u64) {
// Best-effort: return value ignored. A fadvise failure has no
// user-observable consequence.
unsafe {
@@ -34,7 +34,7 @@ pub(super) fn hint_sequential(file: &File, _len_bytes: u64) {
/// Drop pages in the half-open byte range `[start, start+len)` from
/// the page cache. Called periodically by `read_sectors` to bound the
/// read-side page cache pressure.
pub(super) fn drop_window(file: &File, start: u64, len: u64) {
pub(crate) fn drop_window(file: &File, start: u64, len: u64) {
unsafe {
libc::posix_fadvise(
file.as_raw_fd(),
@@ -58,7 +58,7 @@ pub(super) fn drop_window(file: &File, start: u64, len: u64) {
/// can only pre-stage a tiny slice of the next batch. An explicit
/// `readahead()` of the same size as the current batch tells the
/// kernel to queue the full next-batch read now.
pub(super) fn prefetch(file: &File, offset: u64, len: u64) {
pub(crate) fn prefetch(file: &File, offset: u64, len: u64) {
unsafe {
libc::readahead(file.as_raw_fd(), offset as i64, len as usize);
}
+3 -3
View File
@@ -15,7 +15,7 @@ use std::os::unix::io::AsRawFd;
/// pipeline depth.
const RDADVISE_MAX_BYTES: i64 = 64 * 1024 * 1024;
pub(super) fn hint_sequential(file: &File, len_bytes: u64) {
pub(crate) fn hint_sequential(file: &File, len_bytes: u64) {
let bytes = (len_bytes as i64).min(RDADVISE_MAX_BYTES);
let mut ra = libc::radvisory {
ra_offset: 0,
@@ -33,14 +33,14 @@ pub(super) fn hint_sequential(file: &File, len_bytes: u64) {
/// approximation: no-op. macOS's unified buffer cache is generally
/// less prone to the pin-everything pathology that triggers the
/// regression on Linux NFS clients.
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
pub(crate) fn drop_window(_file: &File, _start: u64, _len: u64) {}
/// Async-prefetch the byte range `[offset, offset+len)`. macOS uses
/// the same `fcntl(F_RDADVISE, &radvisory)` primitive as the open-
/// time sequential hint, just targeted at a moving window instead of
/// the whole file. The kernel queues I/O for the requested range and
/// returns immediately.
pub(super) fn prefetch(file: &File, offset: u64, len: u64) {
pub(crate) fn prefetch(file: &File, offset: u64, len: u64) {
let bytes = (len as i64).min(RDADVISE_MAX_BYTES);
let mut ra = libc::radvisory {
ra_offset: offset as libc::off_t,
+11 -8
View File
@@ -48,22 +48,25 @@
//! far smaller than our 16 MiB app-level batch.
#[cfg(target_os = "linux")]
mod linux;
pub(crate) mod linux;
#[cfg(target_os = "macos")]
mod macos;
pub(crate) mod macos;
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
mod other;
pub(crate) mod other;
#[cfg(target_os = "windows")]
mod windows;
pub(crate) mod windows;
// The page-cache hints are shared with any other file-backed sector source:
// `dirimage` reads host files the same way and needs the same eviction, or a
// large rip pins every byte it has read (see this module's DONTNEED note).
#[cfg(target_os = "linux")]
use linux as platform;
pub(crate) use linux as platform;
#[cfg(target_os = "macos")]
use macos as platform;
pub(crate) use macos as platform;
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
use other as platform;
pub(crate) use other as platform;
#[cfg(target_os = "windows")]
use windows as platform;
pub(crate) use windows as platform;
use std::fs::File;
use std::io::{Read, Seek, SeekFrom};
+3 -3
View File
@@ -4,8 +4,8 @@
use std::fs::File;
pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {}
pub(crate) fn hint_sequential(_file: &File, _len_bytes: u64) {}
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
pub(crate) fn drop_window(_file: &File, _start: u64, _len: u64) {}
pub(super) fn prefetch(_file: &File, _offset: u64, _len: u64) {}
pub(crate) fn prefetch(_file: &File, _offset: u64, _len: u64) {}
+3 -3
View File
@@ -9,7 +9,7 @@ use std::fs::File;
/// No-op stub. `FILE_FLAG_SEQUENTIAL_SCAN` can only be set at
/// `CreateFile` open time, which the plain `File::open` path does not
/// do, so there is no post-open hint to issue here.
pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {
pub(crate) fn hint_sequential(_file: &File, _len_bytes: u64) {
tracing::debug!(
target: "mux",
"FileSectorSource hint_sequential: windows no-op stub"
@@ -19,10 +19,10 @@ pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {
/// Windows page-cache eviction is not exposed via a posix_fadvise
/// equivalent. The kernel does its own working-set management. No-op
/// for now.
pub(super) fn drop_window(_file: &File, _start: u64, _len: u64) {}
pub(crate) fn drop_window(_file: &File, _start: u64, _len: u64) {}
/// Windows async-prefetch hint. With FILE_FLAG_SEQUENTIAL_SCAN at
/// open the kernel already prefetches aggressively, so there's no
/// per-range hint we'd add on top. No-op stub for parity with the
/// posix platforms.
pub(super) fn prefetch(_file: &File, _offset: u64, _len: u64) {}
pub(crate) fn prefetch(_file: &File, _offset: u64, _len: u64) {}
+283
View File
@@ -0,0 +1,283 @@
//! `write_image` — write an image-level source out as a sector image.
//!
//! This is the plain image writer: sectors in from any [`SectorSource`], bytes
//! out to a file, in order, once. It is what an `iso://` DESTINATION means when
//! the source is not a physical drive.
//!
//! # Why this is not `freemkv_engine::copy`
//!
//! The engine's `copy` is the RECOVERY path — mapfile sidecar, `--multipass`
//! sweep/patch, damage-jump, ECC-aware batching, auto-resume. Every one of those
//! exists because an optical drive returns read errors on marginal media. A
//! file-backed or synthesized source has no marginal media: a read either
//! succeeds or the underlying file is broken, and retrying it is pointless.
//!
//! Routing a non-drive source through the recovery path is not merely wasteful,
//! it is wrong. Its mapfile identity check compares AACS unit keys and the VID,
//! both of which are empty for an already-decrypted source, so identity passes
//! for ANY such source: a second run with a different input to the same output
//! path would resume over the previous image and produce wrong content at exit
//! zero. Keeping the two paths separate makes that unrepresentable.
//!
//! So: drive sources get `freemkv_engine::copy`. Everything else gets this.
use crate::consts::SECTOR_BYTES;
use crate::error::{Error, Result};
use crate::halt::Halt;
use crate::sector::SectorSource;
use std::fs::File;
use std::io::{BufWriter, Write};
use std::path::Path;
/// Sectors per read/write batch. 4 MiB — large enough that per-call overhead
/// disappears against a file-backed source, small enough that the buffer is not
/// a notable allocation and cancellation stays responsive.
const BATCH_SECTORS: u32 = 2048;
/// Write `total_sectors` sectors from `reader` to `dest`.
///
/// Reads sequentially from LBA 0 and writes in order, so the output is a faithful
/// image of whatever the source presents — decrypted if the caller wrapped the
/// source in a [`DecryptingSectorSource`](crate::sector::decrypting::DecryptingSectorSource),
/// ciphertext if it did not. This function performs no decryption itself and makes
/// no decryption decision; that belongs to the caller, which knows whether the run
/// is `--raw`.
///
/// `on_progress` is called after each batch with the cumulative byte count, for
/// front-end progress reporting. It must not block.
///
/// `halt` is checked once per batch. On cancellation the partial file is left in
/// place — the caller decides whether a partial image is worth keeping, and
/// deleting a multi-gigabyte file the user may want to inspect is not this
/// function's call to make.
///
/// Returns the number of bytes written.
///
/// # Errors
///
/// - [`Error::Halted`] if `halt` was cancelled.
/// - [`Error::IoError`] if the destination cannot be created or written.
/// - Whatever the source's `read_sectors` returns. A short read is an error, not
/// a zero-fill: silently padding a truncated source produces an image that
/// looks complete and is not.
pub fn write_image(
reader: &mut dyn SectorSource,
dest: &Path,
total_sectors: u32,
halt: &Halt,
mut on_progress: impl FnMut(u64),
) -> Result<u64> {
if total_sectors == 0 {
return Err(Error::EmptyImage);
}
let file = File::create(dest).map_err(|source| Error::IoError { source })?;
let mut out = BufWriter::with_capacity(BATCH_SECTORS as usize * SECTOR_BYTES, file);
let mut buf = vec![0u8; BATCH_SECTORS as usize * SECTOR_BYTES];
let mut written: u64 = 0;
let mut lba: u32 = 0;
while lba < total_sectors {
if halt.is_cancelled() {
return Err(Error::Halted);
}
let count = BATCH_SECTORS.min(total_sectors - lba);
let want = count as usize * SECTOR_BYTES;
// `recovery = false`: a file-backed source ignores the flag, and a
// retry loop over a local file would only re-read the same bytes.
let got = reader.read_sectors(lba, count as u16, &mut buf[..want], false)?;
if got != want {
return Err(Error::ShortImageRead {
lba,
expected: want as u32,
got: got as u32,
});
}
out.write_all(&buf[..want])
.map_err(|source| Error::IoError { source })?;
written += want as u64;
lba += count;
on_progress(written);
}
// flush() only pushes the BufWriter's bytes into the kernel via write(2).
// It makes no durability promise at all, so returning Ok here would report
// a finished image while up to several gigabytes of it still sit in the
// page cache. A crash, a power loss, or yanking the removable/network
// volume the image was written to then leaves a truncated or empty file
// that the caller was told was complete.
//
// For a 6-90 GB image that is exactly the failure this crate treats as
// worst: success reported over wrong output. `into_inner` is used rather
// than `flush` so a buffered-write error is surfaced instead of being
// dropped on the floor by BufWriter's Drop.
let file = out.into_inner().map_err(|e| Error::IoError {
source: e.into_error(),
})?;
file.sync_all()
.map_err(|source| Error::IoError { source })?;
Ok(written)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::error::Result as FmResult;
/// A source that yields a deterministic byte per sector, so the written
/// image can be checked positionally rather than just by length.
struct PatternSource {
sectors: u32,
/// Sectors after which `read_sectors` reports a short read.
short_after: Option<u32>,
}
impl SectorSource for PatternSource {
fn capacity_sectors(&self) -> u32 {
self.sectors
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> FmResult<usize> {
let want = count as usize * SECTOR_BYTES;
if self.short_after.is_some_and(|after| lba >= after) {
return Ok(want - 1);
}
for s in 0..count as usize {
let byte = ((lba as usize + s) % 251) as u8;
buf[s * SECTOR_BYTES..(s + 1) * SECTOR_BYTES].fill(byte);
}
Ok(want)
}
}
fn tmp(name: &str) -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!("fmkv-image-writer-{name}-{}", std::process::id()));
p
}
/// The written image is byte-for-byte what the source presented, at the
/// right offsets — not merely the right length.
#[test]
fn writes_every_sector_in_order() {
let dest = tmp("order");
let mut src = PatternSource {
sectors: 5000,
short_after: None,
};
let n = write_image(&mut src, &dest, 5000, &Halt::new(), |_| {}).expect("write");
assert_eq!(n, 5000 * SECTOR_BYTES as u64);
let data = std::fs::read(&dest).expect("read back");
assert_eq!(data.len(), 5000 * SECTOR_BYTES);
// Spot-check across batch boundaries (BATCH_SECTORS = 2048): the last
// sector of batch 0, the first of batch 1, and the final sector.
for lba in [0usize, 2047, 2048, 4095, 4096, 4999] {
let want = (lba % 251) as u8;
assert_eq!(
data[lba * SECTOR_BYTES],
want,
"sector {lba} head byte wrong — batching lost or duplicated a sector"
);
assert_eq!(
data[(lba + 1) * SECTOR_BYTES - 1],
want,
"sector {lba} tail"
);
}
let _ = std::fs::remove_file(&dest);
}
/// A tail shorter than a full batch must still be written whole — the
/// classic off-by-one when `total_sectors` is not a batch multiple.
#[test]
fn writes_a_partial_final_batch() {
let dest = tmp("tail");
let mut src = PatternSource {
sectors: 2049,
short_after: None,
};
let n = write_image(&mut src, &dest, 2049, &Halt::new(), |_| {}).expect("write");
assert_eq!(n, 2049 * SECTOR_BYTES as u64);
assert_eq!(
std::fs::metadata(&dest).expect("stat").len(),
2049 * SECTOR_BYTES as u64
);
let _ = std::fs::remove_file(&dest);
}
/// A short read is an error. Zero-filling would yield an image that looks
/// complete and is not — the single worst outcome for an archival copy.
#[test]
fn short_read_is_an_error_not_a_zero_fill() {
let dest = tmp("short");
let mut src = PatternSource {
sectors: 4096,
short_after: Some(2048),
};
let err = write_image(&mut src, &dest, 4096, &Halt::new(), |_| {}).expect_err("must fail");
assert!(
matches!(err, Error::ShortImageRead { lba: 2048, .. }),
"got {err:?}"
);
let _ = std::fs::remove_file(&dest);
}
/// Cancellation stops the run and reports it, rather than finishing quietly
/// or reporting success on a partial image.
#[test]
fn cancellation_halts_and_reports() {
let dest = tmp("halt");
let mut src = PatternSource {
sectors: 100_000,
short_after: None,
};
let halt = Halt::new();
halt.cancel();
let err = write_image(&mut src, &dest, 100_000, &halt, |_| {}).expect_err("must halt");
assert!(matches!(err, Error::Halted), "got {err:?}");
let _ = std::fs::remove_file(&dest);
}
/// Progress is cumulative and monotonic, and its final value equals the
/// returned byte count — a front-end that trusts the callback must not end
/// up disagreeing with the return value.
#[test]
fn progress_is_cumulative_and_ends_at_the_total() {
let dest = tmp("progress");
let mut src = PatternSource {
sectors: 5000,
short_after: None,
};
let mut seen: Vec<u64> = Vec::new();
let n = write_image(&mut src, &dest, 5000, &Halt::new(), |b| seen.push(b)).expect("write");
assert!(
seen.windows(2).all(|w| w[1] > w[0]),
"not monotonic: {seen:?}"
);
assert_eq!(*seen.last().expect("at least one callback"), n);
let _ = std::fs::remove_file(&dest);
}
/// A zero-sector source is a caller error, not a zero-byte image: an empty
/// ISO is never what anyone wanted, and failing here names the problem.
#[test]
fn zero_sectors_is_an_error() {
let dest = tmp("empty");
let mut src = PatternSource {
sectors: 0,
short_after: None,
};
let err = write_image(&mut src, &dest, 0, &Halt::new(), |_| {}).expect_err("must fail");
assert!(matches!(err, Error::EmptyImage), "got {err:?}");
// The destination must not have been created — a failed run leaves no
// stub for a later run to mistake for output.
assert!(!dest.exists(), "empty run created a file");
}
}
+1
View File
@@ -31,6 +31,7 @@ pub(crate) mod bounded;
pub mod byte_prefetcher;
pub mod file_sector_source;
pub mod fsync;
pub mod image_writer;
pub mod sink;
mod writeback;
mod writeback_file;
+43 -13
View File
@@ -325,17 +325,21 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
///
/// The thread is named `freemkv-pipeline-consumer` so it shows up
/// distinctly in stack traces and `top -H`. Callers that want a
/// more specific name (e.g. `freemkv-sweep-consumer`) should use
/// [`Pipeline::spawn_named`] instead. Returns an `Error::IoError`
/// if the OS refuses the thread spawn (resource exhaustion);
/// callers already operate in fallible context, so this is
/// propagated rather than panicked.
/// more specific name should use [`Pipeline::spawn_named`] instead.
/// Returns an `Error::IoError` if the OS refuses the thread spawn
/// (resource exhaustion); callers already operate in fallible context, so
/// this is propagated rather than panicked.
///
/// Sweep uses [`Pipeline::spawn_named`] directly so the consumer
/// thread shows up as `freemkv-sweep-consumer`; mux uses
/// `freemkv-mux-consumer`. `Pipeline::spawn` (this function, with
/// the default name) is used by `disc::patch` and by the unit
/// tests in this module.
/// Inside this crate the only [`Pipeline::spawn_named`] caller is the mux
/// driver, which names its thread `freemkv-mux-consumer`. `Pipeline::spawn`
/// (this function, with the default name) is used only by the unit tests in
/// this module.
///
/// This paragraph twice named a caller that had left the crate: first
/// `disc::patch`, then Sweep and its `freemkv-sweep-consumer` thread. Both
/// went to freemkv-engine with the recovery passes in 1.6.0, and each in
/// turn sent readers hunting a component that is not here. Name callers
/// that live in THIS crate, or none.
pub fn spawn<S: Sink<I, Output = R>>(depth: usize, sink: S) -> Result<Self, Error> {
Self::spawn_named("freemkv-pipeline-consumer", depth, sink)
}
@@ -343,7 +347,7 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
/// Like [`Pipeline::spawn`] but lets the caller supply the
/// consumer thread's name. Useful when several pipelines run in
/// the same process and stack traces / `top -H` need to tell them
/// apart (e.g. `freemkv-sweep-consumer`, `freemkv-mux-consumer`).
/// apart (e.g. `freemkv-mux-consumer`).
pub fn spawn_named<S: Sink<I, Output = R>>(
name: &str,
depth: usize,
@@ -628,6 +632,14 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
/// wedged inside an unkillable syscall, the producer can still
/// observe `/api/stop` and unwind within
/// [`SEND_HALT_CHECK_INTERVAL`].
/// NOT a `foo_with_X` variant of [`Pipeline::send`], despite the name.
/// The two encode OPPOSITE policies on the same event, each with its own
/// test: after the consumer's `apply` has failed, `send` still succeeds
/// (the consumer keeps draining, so the channel accepts the item), while
/// this one hands the item straight back — so a producer does not read an
/// hour of disc for a write that died on the first frame. Collapsing them
/// into one Option-parameterised method deletes one of those behaviours;
/// it was tried and `apply_error_drains_then_propagates` caught it.
pub fn send_with_halt(&self, item: I, halt: &Halt, deadline: Duration) -> Result<(), I> {
use crossbeam_channel::SendTimeoutError;
let end = Instant::now() + deadline;
@@ -739,6 +751,12 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
/// Plain [`Pipeline::finish`] is preserved for callers without a
/// halt-token plumbed through; that path still blocks indefinitely
/// on `join()`, matching pre-0.20.8 behaviour.
/// Also not a `foo_with_X` variant: [`Pipeline::finish`] joins and waits
/// however long the consumer needs, while this one gives up after
/// `JOIN_TIMEOUT_SECS` and reports halted. Which is right depends on
/// whether the caller has a user waiting to cancel — the mux driver does
/// and uses this; the unit tests do not and use the plain join. Merging
/// them means picking one of those policies for both.
pub fn finish_with_halt(self, halt: Option<&Halt>) -> Result<R, Error> {
let Pipeline {
tx,
@@ -1839,11 +1857,23 @@ mod tests {
thread::spawn(move || {
// Past the first grace window (and past the 250 ms poll cadence that
// bounds when the window is actually observed), inside the second.
thread::sleep(Duration::from_millis(600));
//
// These intervals used to be 600 ms against a 300 ms grace, which
// left NO margin: two 300 ms windows end at 600 ms, and the 250 ms
// poll cadence can push the observation later still, so on a loaded
// runner the second window expired first and the caller abandoned —
// failing with Err(Halted) against a race, not a defect.
//
// Scaled up so the jitter is small relative to the intervals: the
// first window ends at ~1.0-1.25 s and the second at ~2.0-2.25 s,
// so releasing at 1.6 s sits well inside the second with roughly
// 350 ms of slack on either side. The ordering under test is
// unchanged; only the margin is.
thread::sleep(Duration::from_millis(1600));
rel.store(true, Ordering::SeqCst);
});
let grace = Duration::from_millis(300);
let grace = Duration::from_secs(1);
let res = finish_with_grace(handle, &state, grace, Error::Halted);
assert!(
matches!(res, Ok(42)),
+8 -8
View File
@@ -85,16 +85,16 @@ pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult>
let enums = identify_master_enums(archive);
if enums.is_empty() {
tracing::info!(
jar = %entry_name,
jar = ?entry_name,
"deluxe: com/bydeluxe/ present but no master enum fingerprint matched"
);
return None;
}
for (label, m) in &enums {
tracing::info!(
jar = %entry_name,
jar = ?entry_name,
enum = %label,
class = %m.class_name,
class = ?m.class_name,
count = m.values.len(),
"deluxe master enum identified",
);
@@ -110,15 +110,15 @@ pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult>
let binding_classes = find_binding_classes(archive, &master_table.class_name_set());
if binding_classes.is_empty() {
tracing::info!(
jar = %entry_name,
jar = ?entry_name,
"deluxe: no binding class found (no class has enough getstatic refs to master enums)"
);
return None;
}
for (name, count) in &binding_classes {
tracing::info!(
jar = %entry_name,
binding_class = %name,
jar = ?entry_name,
binding_class = ?name,
getstatic_count = count,
"deluxe binding class candidate",
);
@@ -138,7 +138,7 @@ pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult>
}
if streams.is_empty() {
tracing::info!(
jar = %entry_name,
jar = ?entry_name,
"deluxe: binding classes found but produced 0 decoded streams"
);
return None;
@@ -149,7 +149,7 @@ pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult>
return None;
}
tracing::info!(
jar = %entry_name,
jar = ?entry_name,
audio = labels.iter().filter(|l| l.stream_type == StreamLabelType::Audio).count(),
subtitle = labels.iter().filter(|l| l.stream_type == StreamLabelType::Subtitle).count(),
"deluxe emitted labels",
+170 -6
View File
@@ -450,13 +450,22 @@ pub(crate) fn apply_labels(labels: &[StreamLabel], titles: &mut [DiscTitle]) {
}) else {
continue;
};
for clip in &title.clips {
// ONLY the anchor's first clip. `disc::bluray` builds a title's
// stream list from `play_items[0]`'s STN table, so that is the only
// clip in which these PIDs were ever observed — the same clip tier 3
// keys its derived ids on (`clip0`, below). Recording the fact
// against every clip the anchor plays claims knowledge of stream
// tables never read: a sibling playlist over a LATER clip then binds
// the anchor's editorial label onto whichever stream of that clip
// reuses the PID, which is a different physical stream (PIDs are
// unique only within a clip).
if let Some(clip) = title.clips.first() {
pid_map.insert((clip.clip_id.as_str(), *pid), pos);
}
}
tracing::info!(
stream_type = ?stream_type,
playlist = %titles[anchor].playlist,
playlist = ?titles[anchor].playlist,
slots = slots_of(&titles[anchor], stream_type).len(),
"label list anchored to a title by its stream-language sequence",
);
@@ -1418,11 +1427,20 @@ mod registry_tests {
/// straight from the disc's UDF directory records, with attacker-chosen
/// name lengths to inflate each comparison.
///
/// Proof is by deadline. With the linear scan this fixture measures well
/// past the deadline; with a set it is milliseconds. Bounded so a
/// regression fails fast instead of hanging CI.
/// This is a HANG GUARD, and the name says so: a return to the linear scan
/// makes this fixture run for minutes (120 000² / 2 comparisons over a
/// 180-byte shared prefix), which without the deadline would wedge CI
/// rather than fail it. It is not a complexity proof — no assertion here
/// can distinguish `BTreeSet` from any other sub-quadratic dedup, and the
/// clock-free half of the claim (dedup, sort, directory exclusion) belongs
/// to `jar_inventory_dedups_sorts_and_skips_dirs` below.
///
/// The deadline is a real margin, unlike the 6x one that made
/// `paramount.rs`'s wall-clock test flake under a loaded CI box: measured
/// at 0.14 s debug / 0.07 s release against 10 s, so ~70x. A shared CPU
/// does not close that; a quadratic dedup does not survive it.
#[test]
fn jar_inventory_dedup_is_not_quadratic() {
fn jar_inventory_dedup_does_not_hang_on_a_hostile_directory() {
const FILES: usize = 120_000;
let (tx, rx) = std::sync::mpsc::channel();
let worker = std::thread::spawn(move || {
@@ -1887,6 +1905,7 @@ mod apply_tests {
DiscTitle {
playlist: playlist.into(),
clips: vec![crate::disc::Clip {
feed_span: None,
clip_id: clip_id.into(),
in_time: 0,
out_time: 0,
@@ -1897,6 +1916,78 @@ mod apply_tests {
}
}
/// Sentinel embedded in the crafted playlist name below. The capture keeps
/// ONLY fields whose rendered form contains it, so the assertion cannot be
/// fooled by another test's log output.
const LOG_INJECTION_SENTINEL: &str = "FMKV-LOG-INJECTION-PROBE";
/// A playlist name is a raw UDF directory entry — disc-controlled bytes,
/// validated no further than a lossy UTF-8 decode. Logging it through
/// tracing's `%` (Display) sigil writes those bytes VERBATIM, so a crafted
/// `.mpls` filename carrying ANSI escapes or control characters forges
/// terminal output and log structure in any consumer rendering the event
/// (CWE-117). `?` (Debug) escapes them, and `str`'s Debug is exactly the
/// escaping this needs.
///
/// `info!` is not covered by the debug/trace-logging exemption: this fires
/// on an ordinary rip of an ordinary disc.
///
/// Mutation: put `%` back on `playlist` in `apply_labels` and this goes red.
#[test]
fn a_disc_derived_playlist_name_is_escaped_in_the_log_not_written_verbatim() {
// A name whose bytes would clear the line and repaint it.
let evil = format!("\u{1b}[2K\u{1b}[31m{LOG_INJECTION_SENTINEL}\u{7}\u{1b}[0m.mpls");
let labels = vec![
sub_label(1, "eng", LabelQualifier::None),
sub_label(2, "spa", LabelQualifier::None),
sub_label(3, "fra", LabelQualifier::None),
];
let mut titles = vec![title_on_clip(
&evil,
"00294",
vec![
subtitle(0x12A0, "eng"),
subtitle(0x12A1, "spa"),
subtitle(0x12A2, "fra"),
],
)];
// Capture through the crate's ONE global `tracing` subscriber:
// `testlog::capture` installs it exactly once and routes each event to a
// thread-local sink. A process-wide `set_global_default` HERE instead
// would poison every other test's callsite interest cache for the rest of
// the binary — `tracing` caches interest GLOBALLY — and only the first
// `set_global_default` in a process takes effect anyway. The shared
// subscriber answers interest for every callsite and isolates concurrent
// captures per thread, which is the invariant these log-accounting
// assertions rely on.
let ((), events) = crate::testlog::capture(|| {
apply_labels(&labels, &mut titles);
});
let playlist: Vec<&str> = events
.iter()
.filter(|e| e.target.starts_with("libfreemkv::labels"))
.filter_map(|e| e.field("playlist"))
.filter(|v| v.contains(LOG_INJECTION_SENTINEL))
.collect();
assert!(
!playlist.is_empty(),
"the anchoring event must actually have fired, or this test proves \
nothing; captured: {events:?}"
);
for rendered in playlist {
assert!(
!rendered.contains('\u{1b}') && !rendered.contains('\u{7}'),
"a disc-controlled playlist name reached the log with its raw \
control bytes intact: {rendered:?}"
);
assert!(
rendered.contains(LOG_INJECTION_SENTINEL),
"the name must still be legible once escaped: {rendered:?}"
);
}
}
fn sub_label(num: u16, lang: &str, qualifier: LabelQualifier) -> StreamLabel {
StreamLabel {
stream_id: None,
@@ -3093,6 +3184,79 @@ mod apply_tests {
);
}
/// A title that plays SEVERAL clips in order — the shape the anchor's
/// PID facts are harvested from.
fn title_on_clips(playlist: &str, clip_ids: &[&str], streams: Vec<Stream>) -> DiscTitle {
DiscTitle {
playlist: playlist.into(),
clips: clip_ids
.iter()
.map(|id| crate::disc::Clip {
feed_span: None,
clip_id: (*id).into(),
in_time: 0,
out_time: 0,
duration_secs: 3600.0,
source_packets: 0,
})
.collect(),
..title_with(streams)
}
}
/// Spec: the anchor proves a `(clip, PID)` fact only for the clip its
/// stream table was READ FROM — the first play item — never for every clip
/// the anchor happens to play.
///
/// `disc::bluray` builds a title's stream list from `play_items[0]`'s STN
/// table, and tier 3 fifty lines below says exactly that by keying its
/// derived ids on `clip0`. Tier 2's harvest contradicted it: it recorded
/// the anchor's slot PIDs against EVERY clip the anchor plays, so a sibling
/// playlist that plays a LATER clip of the anchor bound the anchor's
/// editorial label onto whatever stream in that clip happens to reuse the
/// PID — a different physical stream, in a different clip, whose own
/// language says so.
///
/// Mutation: harvest over `&title.clips` instead of its first clip — the
/// featurette wears the feature's SDH again.
#[test]
fn an_anchor_proves_pids_only_for_the_clip_its_table_came_from() {
let labels = vec![
sub_label(1, "eng", LabelQualifier::Sdh),
sub_label(2, "fra", LabelQualifier::None),
];
let mut titles = vec![
// The anchor: its stream table is clip 00082's (the first play
// item); it merely CONTINUES into 00090.
title_on_clips(
"00800.mpls",
&["00082", "00090"],
vec![subtitle(0x1200, "eng"), subtitle(0x1201, "fra")],
),
// A sibling playlist over the anchor's SECOND clip. Its subtitle
// reuses PID 0x1200 — PIDs are only unique within a clip — and it
// is Spanish, so nothing about the anchor's English SDH slot
// describes it.
title_on_clips("00451.mpls", &["00090"], vec![subtitle(0x1200, "spa")]),
];
apply_labels(&labels, &mut titles);
assert_eq!(
sub_state(&titles[0]),
vec![
(0x1200, false, LabelQualifier::Sdh),
(0x1201, false, LabelQualifier::None)
],
"the anchor itself keeps the qualifiers the list states for it"
);
assert_eq!(
sub_state(&titles[1]),
vec![(0x1200, false, LabelQualifier::None)],
"a PID in a clip the anchor's table never described is not that \
table's stream 1"
);
}
/// Spec: a vendor codec/variant claim does not follow the ordinal onto a
/// bonus clip that carries a different codec.
///
+194 -60
View File
@@ -98,6 +98,74 @@ enum ForcedSub {
ForcedNarrative,
}
/// The one number behind every cap in this parser: the highest CSV cell
/// position that can ever be addressed.
///
/// The labelling loops number cells 1-based into a `u16` and `break` at
/// `u16::try_from(i + 1)`, so cell `MAX_COM_INDICES` and everything past it is
/// never visited. Two different things are measured against that, and they are
/// not the same bound:
///
/// - **A VALUE at or beyond it cannot match any cell.** This is what caps the
/// set: values are filtered before insertion, so at most `MAX_COM_INDICES`
/// distinct entries can ever be stored, however long the attribute is. The
/// `HashSet` that replaced a linear scan fixed the LOOKUP cost; this is what
/// fixes the ALLOCATION, and a disc declaring half a billion indices no
/// longer costs half a billion entries.
/// - **A POSITION at or beyond it describes nothing new.** This caps the WORK,
/// not the memory — the value filter already made the set small, but without
/// it every one of those half a billion cells is still split and parsed. It
/// is an early exit at the first position whose contents provably cannot
/// matter, and it is why `forced_sub` — which holds no set at all, and so
/// gets no protection from the value rule — is bounded too.
///
/// Real authoring is nowhere near either limit: the BD STN table admits at
/// most 32 streams per playlist, so nothing legitimate is lost.
const MAX_COM_INDICES: usize = u16::MAX as usize;
/// Parse a `*_com1_idx` attribute into the set the labelling loops query.
///
/// Extracted so the BOUND is observable. Asserting it through
/// `labels_from_feature` is not possible: a `HashSet` collapses repeated
/// values, and an out-of-range index changes no label either way, so such a
/// test passes whether or not the cap exists — an assertion that cannot fail.
/// Returning the set lets a test hand in tens of thousands of DISTINCT
/// unaddressable indices and see them refused.
fn com_indices(attr: Option<String>) -> HashSet<usize> {
attr.map(|s| {
s.split(',')
.take(MAX_COM_INDICES)
.filter_map(|i| i.trim().parse().ok())
.filter(|&i| i < MAX_COM_INDICES)
.collect()
})
.unwrap_or_default()
}
/// Parse the `forced_sub` attribute into the cell list the subtitle loop
/// queries — the third attacker-controlled CSV in this file, and the last one
/// that was still unbounded.
///
/// Bounded by POSITION, and it has no other choice: a `*_com1_idx` list holds
/// values that can be filtered, and that filter is what caps its set, but a
/// `forced_sub` cell is a classification of the position it sits at, so there
/// is nothing to filter and nothing else would ever cap this. The Vec is read
/// only as `forced.get(i)` from a loop that stops at `MAX_COM_INDICES`, so
/// every cell past that is unreachable by construction.
///
/// Extracted, like [`com_indices`], so the bound is OBSERVABLE. Through
/// `labels_from_feature` it is not: the subtitle loop cannot reach those cells
/// either, so a label-level assertion passes whether or not the cap exists.
fn forced_subs(attr: Option<String>) -> Vec<ForcedSub> {
attr.map(|s| {
s.split(',')
.take(MAX_COM_INDICES)
.map(forced_sub_cell)
.collect()
})
.unwrap_or_default()
}
fn forced_sub_cell(cell: &str) -> ForcedSub {
match cell.trim() {
"1" => ForcedSub::ContainsForcedSegments,
@@ -123,9 +191,7 @@ fn labels_from_feature(feature: &str) -> Vec<StreamLabel> {
// linearly once per stream, so `aud="..."` and `aud_com1_idx="..."`
// both grown large make this quadratic in the size of one XML file.
// Membership is the only operation performed on it.
let com_indices: HashSet<usize> = xml::attr(feature, "aud_com1_idx")
.map(|s| s.split(',').filter_map(|i| i.trim().parse().ok()).collect())
.unwrap_or_default();
let com_indices = com_indices(xml::attr(feature, "aud_com1_idx"));
// The CSV *is* the STN list: one cell per stream, in stream order,
// and `aud_com1_idx` is a 0-based index into those same cells. So
@@ -172,15 +238,11 @@ fn labels_from_feature(feature: &str) -> Vec<StreamLabel> {
// Parse subtitle streams
if let Some(sub) = xml::attr(feature, "sub") {
let forced: Vec<ForcedSub> = xml::attr(feature, "forced_sub")
.map(|s| s.split(',').map(forced_sub_cell).collect())
.unwrap_or_default();
let forced = forced_subs(xml::attr(feature, "forced_sub"));
// HashSet for the same reason as the audio side above: unbounded
// parsed input, membership-only use, linear scan once per stream.
let com_indices: HashSet<usize> = xml::attr(feature, "sub_com1_idx")
.map(|s| s.split(',').filter_map(|i| i.trim().parse().ok()).collect())
.unwrap_or_default();
let com_indices = com_indices(xml::attr(feature, "sub_com1_idx"));
// As with audio: the cell position IS the STN slot. `forced_sub` and
// `sub_com1_idx` are indexed against those same cells, so an empty
@@ -315,61 +377,133 @@ mod tests {
);
}
/// `sub_com1_idx` is an unbounded index list parsed straight out of the
/// disc's `playlists.xml` and was membership-tested with a linear
/// `Vec::contains` once per subtitle stream — quadratic in the size of a
/// single attacker-supplied file.
/// `sub_com1_idx` is parsed straight out of the disc's `playlists.xml`,
/// which is attacker-controlled and has no length bound of its own.
///
/// Proof is by deadline rather than micro-benchmark. With the linear scan
/// the original fixture (200 000 streams x 1 000 001 indices) measured
/// 31 s in a release build and far longer in debug; with a set it measured
/// 0.03 s release / 0.56 s debug. A 10 s deadline sits ~18x above the
/// slowest passing measurement and ~3x below the fastest failing one, and
/// makes a regression fail fast instead of hanging CI.
/// This replaces a WALL-CLOCK test. That one built a 200 000 x 1 000 001
/// fixture and failed if it took over 10 s, to prove the membership test
/// was a set rather than a linear scan. Measured on the machine that
/// wrote this: 1.62 s alone, and OVER 10 s — a real failure — when the
/// suite's other 3 347 tests were running concurrently. A 6x margin
/// against a shared CPU is not a margin; it is a CI failure that looks
/// like a flake and gets re-run until it passes.
///
/// The CSV now stops at the end of the 1-based `u16` stream-numbering
/// space, so only the first 65 535 cells are scanned. `INDICES` is raised
/// to keep the linear-scan work product (`cells x indices`) at or above
/// the original fixture's, preserving that deadline margin.
/// It also measured the wrong thing. Making the lookup O(1) bounded the
/// QUERY, not the PARSE: the set was still built from every entry the
/// disc declared, so a hostile playlist could still force an unbounded
/// allocation before any lookup happened. `MAX_COM_INDICES` bounds that.
///
/// Correctness is pinned on fixture-derived literals: indices 0, 2 and 4
/// are the commentary tracks, 1 and 3 are not.
/// What THIS test guards is that bounding did not change what a
/// legitimate playlist MEANS: it goes red if the bound is set too LOW
/// (verified at 2 — the real indices `0,2,4` stop resolving and the
/// purposes change). It does NOT go red if the bound is deleted
/// entirely, because the out-of-range filler is unobservable at the
/// label level and a `HashSet` collapses the repeats. Enforcement is
/// proven separately, by
/// `distinct_unaddressable_indices_are_refused_not_stored`, which reads
/// the set itself. Two tests, two properties; neither pretends to the
/// other's job.
#[test]
fn commentary_index_lookup_is_not_quadratic() {
/// Cells offered. Everything past `u16::MAX` is unnumberable and the
/// parser stops there, so the scanned prefix is 65 535 cells.
const STREAMS: usize = 200_000;
const SCANNED: usize = u16::MAX as usize;
const INDICES: usize = 3_100_000;
let (tx, rx) = std::sync::mpsc::channel();
let worker = std::thread::spawn(move || {
let mut feature = String::from(r#"<playlist name="Feature" sub=""#);
feature.push_str(&"eng,".repeat(STREAMS));
feature.pop();
// Three real commentary indices, then a long run of one
// out-of-range value: nothing here is bounded by the stream count.
feature.push_str(r#"" sub_com1_idx="0,2,4,"#);
feature.push_str(&"9999999,".repeat(INDICES));
feature.pop();
feature.push_str(r#"" />"#);
let _ = tx.send(labels_from_feature(&feature));
});
match rx.recv_timeout(std::time::Duration::from_secs(10)) {
Ok(labels) => {
worker.join().expect("worker panicked");
assert_eq!(labels.len(), SCANNED);
assert_eq!(labels[0].purpose, LabelPurpose::Commentary);
assert_eq!(labels[1].purpose, LabelPurpose::Normal);
assert_eq!(labels[2].purpose, LabelPurpose::Commentary);
assert_eq!(labels[3].purpose, LabelPurpose::Normal);
assert_eq!(labels[4].purpose, LabelPurpose::Commentary);
}
Err(_) => panic!(
"labels_from_feature did not finish {STREAMS} streams x \
{INDICES} commentary indices within 10s — the membership \
test is still a linear scan"
),
}
fn bounding_the_parse_does_not_change_a_legitimate_playlist() {
// Three real indices, then far more entries than can address a cell.
const OVERSIZED: usize = MAX_COM_INDICES + 10_000;
let mut feature = String::from(r#"<playlist name="Feature" sub=""#);
feature.push_str(&"eng,".repeat(8));
feature.pop();
feature.push_str(r#"" sub_com1_idx="0,2,4,"#);
feature.push_str(&"9999999,".repeat(OVERSIZED));
feature.pop();
feature.push_str(r#"" />"#);
let labels = labels_from_feature(&feature);
// The fixture's real indices still decide the purposes: bounding the
// parse must not change what a legitimate playlist means.
assert_eq!(labels.len(), 8);
assert_eq!(labels[0].purpose, LabelPurpose::Commentary);
assert_eq!(labels[1].purpose, LabelPurpose::Normal);
assert_eq!(labels[2].purpose, LabelPurpose::Commentary);
assert_eq!(labels[3].purpose, LabelPurpose::Normal);
assert_eq!(labels[4].purpose, LabelPurpose::Commentary);
}
/// The set REFUSES unaddressable indices, so a hostile playlist cannot
/// inflate it. DISTINCT values on purpose: a `HashSet` collapses repeats,
/// so a million copies of one index costs one entry and would prove
/// nothing. Fifty thousand distinct out-of-range indices cost fifty
/// thousand entries without the filter, and none with it — so this test
/// goes red if the bound is removed, which the label-level assertions
/// below cannot do.
#[test]
fn distinct_unaddressable_indices_are_refused_not_stored() {
let hostile: String = (MAX_COM_INDICES..MAX_COM_INDICES + 50_000)
.map(|i| i.to_string())
.collect::<Vec<_>>()
.join(",");
let set = com_indices(Some(hostile));
assert!(
set.is_empty(),
"kept {} unaddressable indices — the parse is still unbounded",
set.len()
);
// The addressable ones are still kept.
assert_eq!(com_indices(Some("0,2,4".to_string())).len(), 3);
}
/// `forced_sub` is bounded too — the third CSV in the same function, and
/// the one that had no value filter to hide behind.
///
/// Read through `forced_subs` rather than through the labels for the same
/// reason the two tests above read the set: the subtitle loop stops at
/// `MAX_COM_INDICES`, so a label-level assertion cannot tell a bounded
/// parse from an unbounded one.
#[test]
fn forced_sub_cells_past_the_last_addressable_one_are_not_parsed() {
let hostile = "0,".repeat(MAX_COM_INDICES + 50_000);
let cells = forced_subs(Some(hostile));
assert_eq!(
cells.len(),
MAX_COM_INDICES,
"parsed {} cells — the forced_sub parse is still unbounded",
cells.len()
);
}
/// Bounding it must not change what a legitimate playlist means: the
/// cells that CAN address a stream still classify exactly as before.
#[test]
fn bounding_forced_sub_leaves_the_addressable_cells_alone() {
let cells = forced_subs(Some("0,1,3,2".to_string()));
assert_eq!(
cells,
vec![
ForcedSub::None,
ForcedSub::ContainsForcedSegments,
ForcedSub::ForcedNarrative,
ForcedSub::ForcedNarrative,
]
);
}
/// An index that cannot address any cell is dropped rather than STORED.
///
/// Asserted through `com_indices`, not through the labels: the labelling
/// loop never queries a cell position that high, so at the label level
/// retaining the value is unobservable and the assertion could not fail.
/// Reading the set is what makes the claim checkable.
#[test]
fn an_index_that_cannot_address_any_cell_is_not_retained() {
let set = com_indices(Some(format!(
"1,{},{}",
MAX_COM_INDICES,
MAX_COM_INDICES + 1
)));
assert_eq!(
set.len(),
1,
"only the addressable index belongs in the set, got {set:?}"
);
assert!(set.contains(&1));
}
/// Headroom: the BD STN_table admits at most 32 PG streams per playlist,
+326
View File
@@ -219,6 +219,237 @@ pub fn menu_lang(token: &str) -> Option<&'static str> {
Some(code)
}
// ── ISO 639-1 → ISO 639-2 ────────────────────────────────────────────────────
/// The complete ISO 639-1 set, paired with its ISO 639-2/**T** (terminological)
/// code. Every two-letter code ISO 639-1 defines appears exactly once.
///
/// /T is the variant the rest of this crate uses — [`lang`] and [`menu_lang`]
/// both normalize to it (`deu` not `ger`, `fra` not `fre`, `zho` not `chi`,
/// `ces`, `nld`, `ell`, `ron`, `slk`, `isl`, `eus`, `hrv`) — so the three
/// tables cannot disagree. `iso639_1_agrees_with_menu_lang` pins that.
///
/// For the 165 codes where 639-2/B and /T are identical this distinction does
/// not arise; it only matters for the 20-odd languages with a distinct
/// bibliographic code.
const ISO_639_1_TO_2: &[(&str, &str)] = &[
("aa", "aar"),
("ab", "abk"),
("ae", "ave"),
("af", "afr"),
("ak", "aka"),
("am", "amh"),
("an", "arg"),
("ar", "ara"),
("as", "asm"),
("av", "ava"),
("ay", "aym"),
("az", "aze"),
("ba", "bak"),
("be", "bel"),
("bg", "bul"),
("bh", "bih"),
("bi", "bis"),
("bm", "bam"),
("bn", "ben"),
("bo", "bod"),
("br", "bre"),
("bs", "bos"),
("ca", "cat"),
("ce", "che"),
("ch", "cha"),
("co", "cos"),
("cr", "cre"),
("cs", "ces"),
("cu", "chu"),
("cv", "chv"),
("cy", "cym"),
("da", "dan"),
("de", "deu"),
("dv", "div"),
("dz", "dzo"),
("ee", "ewe"),
("el", "ell"),
("en", "eng"),
("eo", "epo"),
("es", "spa"),
("et", "est"),
("eu", "eus"),
("fa", "fas"),
("ff", "ful"),
("fi", "fin"),
("fj", "fij"),
("fo", "fao"),
("fr", "fra"),
("fy", "fry"),
("ga", "gle"),
("gd", "gla"),
("gl", "glg"),
("gn", "grn"),
("gu", "guj"),
("gv", "glv"),
("ha", "hau"),
("he", "heb"),
("hi", "hin"),
("ho", "hmo"),
("hr", "hrv"),
("ht", "hat"),
("hu", "hun"),
("hy", "hye"),
("hz", "her"),
("ia", "ina"),
("id", "ind"),
("ie", "ile"),
("ig", "ibo"),
("ii", "iii"),
("ik", "ipk"),
("io", "ido"),
("is", "isl"),
("it", "ita"),
("iu", "iku"),
("ja", "jpn"),
("jv", "jav"),
("ka", "kat"),
("kg", "kon"),
("ki", "kik"),
("kj", "kua"),
("kk", "kaz"),
("kl", "kal"),
("km", "khm"),
("kn", "kan"),
("ko", "kor"),
("kr", "kau"),
("ks", "kas"),
("ku", "kur"),
("kv", "kom"),
("kw", "cor"),
("ky", "kir"),
("la", "lat"),
("lb", "ltz"),
("lg", "lug"),
("li", "lim"),
("ln", "lin"),
("lo", "lao"),
("lt", "lit"),
("lu", "lub"),
("lv", "lav"),
("mg", "mlg"),
("mh", "mah"),
("mi", "mri"),
("mk", "mkd"),
("ml", "mal"),
("mn", "mon"),
("mr", "mar"),
("ms", "msa"),
("mt", "mlt"),
("my", "mya"),
("na", "nau"),
("nb", "nob"),
("nd", "nde"),
("ne", "nep"),
("ng", "ndo"),
("nl", "nld"),
("nn", "nno"),
("no", "nor"),
("nr", "nbl"),
("nv", "nav"),
("ny", "nya"),
("oc", "oci"),
("oj", "oji"),
("om", "orm"),
("or", "ori"),
("os", "oss"),
("pa", "pan"),
("pi", "pli"),
("pl", "pol"),
("ps", "pus"),
("pt", "por"),
("qu", "que"),
("rm", "roh"),
("rn", "run"),
("ro", "ron"),
("ru", "rus"),
("rw", "kin"),
("sa", "san"),
("sc", "srd"),
("sd", "snd"),
("se", "sme"),
("sg", "sag"),
("si", "sin"),
("sk", "slk"),
("sl", "slv"),
("sm", "smo"),
("sn", "sna"),
("so", "som"),
("sq", "sqi"),
("sr", "srp"),
("ss", "ssw"),
("st", "sot"),
("su", "sun"),
("sv", "swe"),
("sw", "swa"),
("ta", "tam"),
("te", "tel"),
("tg", "tgk"),
("th", "tha"),
("ti", "tir"),
("tk", "tuk"),
("tl", "tgl"),
("tn", "tsn"),
("to", "ton"),
("tr", "tur"),
("ts", "tso"),
("tt", "tat"),
("tw", "twi"),
("ty", "tah"),
("ug", "uig"),
("uk", "ukr"),
("ur", "urd"),
("uz", "uzb"),
("ve", "ven"),
("vi", "vie"),
("vo", "vol"),
("wa", "wln"),
("wo", "wol"),
("xh", "xho"),
("yi", "yid"),
("yo", "yor"),
("za", "zha"),
("zh", "zho"),
("zu", "zul"),
];
/// The three two-letter codes ISO 639-1 has since withdrawn, mapped to their
/// replacements. DVD-Video froze its language list on the 1988 edition, so
/// discs authored to the spec carry these spellings and no other table sees
/// them: `iw` Hebrew (now `he`), `in` Indonesian (now `id`), `ji` Yiddish
/// (now `yi`).
const ISO_639_1_DEPRECATED: &[(&str, &str)] = &[("iw", "he"), ("in", "id"), ("ji", "yi")];
/// Map an ISO 639-1 two-letter language code to its ISO 639-2/T three-letter
/// code, accepting the withdrawn DVD-era spellings (`iw`, `in`, `ji`) as
/// aliases for their replacements.
///
/// Covers the WHOLE of ISO 639-1, unlike [`menu_lang`], whose table only spans
/// the languages that show up in Blu-ray menu-graphic filenames. Callers that
/// convert a spec field — a DVD IFO attribute block, say — need the whole set:
/// narrowing it to the menu vocabulary would fold every other language onto
/// one value and make a disc's tracks indistinguishable from each other.
///
/// Case-insensitive and trimmed. Returns `None` for anything that is not an
/// ISO 639-1 code, so callers decide the fallback rather than getting a guess.
pub fn iso639_1_to_iso639_2(code: &str) -> Option<&'static str> {
let c = code.trim().to_ascii_lowercase();
let c = ISO_639_1_DEPRECATED
.iter()
.find(|(old, _)| *old == c)
.map_or(c.as_str(), |(_, new)| new);
ISO_639_1_TO_2
.iter()
.find(|(two, _)| *two == c)
.map(|(_, three)| *three)
}
// ── Purpose ──────────────────────────────────────────────────────────────────
/// Classify a free-form English label string into a [`LabelPurpose`].
@@ -789,4 +1020,99 @@ mod tests {
assert_eq!(menu_lang("xyz"), None);
assert_eq!(menu_lang(""), None);
}
/// Structural invariants of `ISO_639_1_TO_2`: it must hold the complete
/// ISO 639-1 set (184 codes), every key a distinct pair of lowercase
/// letters and every value three lowercase letters. A typo'd or duplicated
/// row fails here rather than silently mislabelling a track.
#[test]
fn iso639_1_table_is_complete_and_well_formed() {
assert_eq!(
ISO_639_1_TO_2.len(),
184,
"ISO 639-1 defines 184 two-letter codes; the table must hold all \
of them"
);
let mut keys: Vec<&str> = ISO_639_1_TO_2.iter().map(|(two, _)| *two).collect();
keys.sort_unstable();
let unique = keys.len();
keys.dedup();
assert_eq!(unique, keys.len(), "no ISO 639-1 code may appear twice");
for (two, three) in ISO_639_1_TO_2 {
assert!(
two.len() == 2 && two.bytes().all(|b| b.is_ascii_lowercase()),
"{two:?} is not a two-letter lowercase ISO 639-1 code"
);
assert!(
three.len() == 3 && three.bytes().all(|b| b.is_ascii_lowercase()),
"{three:?} is not a three-letter lowercase ISO 639-2 code"
);
}
// The withdrawn DVD-era spellings resolve, and are not themselves
// rows in the main table (they are aliases, not codes).
for (old, new) in ISO_639_1_DEPRECATED {
assert!(
!ISO_639_1_TO_2.iter().any(|(two, _)| two == old),
"withdrawn code {old:?} must not be a table row"
);
assert_eq!(
iso639_1_to_iso639_2(old),
iso639_1_to_iso639_2(new),
"withdrawn code {old:?} must resolve exactly as {new:?}"
);
}
}
/// The two tables must not disagree. Every two-letter token `menu_lang`
/// accepts has to yield the same ISO 639-2/T code through
/// `iso639_1_to_iso639_2`, so a DVD-sourced language and a Blu-ray
/// menu-label language for the same tongue never produce different
/// `Language` elements.
#[test]
fn iso639_1_agrees_with_menu_lang() {
for (two, three) in ISO_639_1_TO_2 {
if let Some(via_menu) = menu_lang(two) {
assert_eq!(
via_menu, *three,
"menu_lang({two:?}) = {via_menu:?} disagrees with the ISO \
639-1 table's {three:?}"
);
}
}
// Spot-check the /T choice itself, on the languages where /B differs.
for (two, t_code) in [
("de", "deu"),
("fr", "fra"),
("zh", "zho"),
("cs", "ces"),
("nl", "nld"),
("el", "ell"),
("ro", "ron"),
("sk", "slk"),
("is", "isl"),
("hy", "hye"),
("ka", "kat"),
("fa", "fas"),
] {
assert_eq!(
iso639_1_to_iso639_2(two),
Some(t_code),
"the crate standardises on ISO 639-2/T, so {two:?} is \
{t_code:?} and never the bibliographic form"
);
}
}
/// Trimming, case-insensitivity, and the no-guess contract.
#[test]
fn iso639_1_normalizes_input_and_never_guesses() {
assert_eq!(iso639_1_to_iso639_2("RO"), Some("ron"));
assert_eq!(iso639_1_to_iso639_2(" Ro "), Some("ron"));
assert_eq!(iso639_1_to_iso639_2("IW"), Some("heb"));
assert_eq!(iso639_1_to_iso639_2("zz"), None);
assert_eq!(iso639_1_to_iso639_2(""), None);
assert_eq!(iso639_1_to_iso639_2("e"), None);
// A three-letter code is not ISO 639-1 input — that is menu_lang's job.
assert_eq!(iso639_1_to_iso639_2("eng"), None);
}
}
+11 -1
View File
@@ -106,6 +106,7 @@ pub mod consts;
pub mod css;
pub mod decrypt;
pub mod diag;
pub mod dirimage;
pub mod disc;
pub mod drive;
pub mod dvdnav;
@@ -128,6 +129,8 @@ pub mod progress;
pub mod scsi;
pub mod sector;
pub mod session;
#[cfg(test)]
pub(crate) mod testlog;
pub(crate) mod udf;
pub(crate) mod unlock_bridge;
@@ -148,7 +151,8 @@ pub use drive::{Drive, DriveStatus, extract_scsi_context, find_drive};
// Owns the `Drive` by value; forwards consumer-built key material into
// `ScanOptions` (the library derives no certs — see `KeySpec`).
pub use session::{
DeviceTarget, DiscSession, KeySourceFactory, KeySpec, ResolvedKeys, resolve_keys_for, scan_iso,
DeviceTarget, DiscSession, KeySourceFactory, KeySpec, ResolvedKeys, resolve_keys_for, scan_dir,
scan_iso,
};
// ─── Errors ─────────────────────────────────────────────────────────────────
@@ -191,6 +195,11 @@ pub use io::pipeline::{
// continuously instead of bursting. General I/O infra, not recovery policy;
// promoted to `pub` so freemkv-engine's relocated sweep/patch can use it too.
pub use io::WritebackFile;
/// Write an image-level source out as a sector image — what an `iso://`
/// DESTINATION means for any source that is not a physical drive. Drive sources
/// go through `freemkv_engine::copy`, which is the recovery path; see
/// [`io::image_writer`] for why the two are deliberately separate.
pub use io::image_writer::write_image;
// ─── Drive events (low-level callbacks) ─────────────────────────────────────
pub use event::{BatchSizeReason, Event, EventKind};
@@ -223,6 +232,7 @@ pub use decrypt::{AacsKeyMap, DecryptKeys, decrypt_sectors, decrypt_threads, set
// — not the `pes::Stream` trait re-exported below as `PesStream`. Two
// different concepts, the same short name; the trait gets the `Pes`
// prefix at the crate root to keep both addressable.
pub use dirimage::DirImage;
pub use disc::{
AacsState, AudioChannels, AudioStream, Clip, Codec, ColorSpace, ContentFormat, Disc,
DiscFormat, DiscId, DiscTitle, DriveCredentials, Extent, ExtractOptions, ExtractResult,
+487 -31
View File
@@ -62,8 +62,18 @@ const AC3_SAMPLES_PER_FRAME: u32 = 1536;
const MAX_AC3_BUF: usize = 1024 * 1024;
pub struct Ac3Parser {
/// Leftover bytes from previous PES (incomplete frame at end).
buf: Vec<u8>,
/// Leftover bytes from previous PES (incomplete frame at end), each still
/// attributable to the packet that carried it — so an access unit that
/// began in an earlier packet takes THAT packet's source offset, not the
/// one that happened to complete it.
acc: super::pesbuf::PesBuf,
/// Reused working copy of `acc`, kept across calls so the per-PES scan does
/// not allocate. `parse` runs once per PES packet on an audio track — of
/// the order of 10^5 times per title — and previously built a fresh `Vec`
/// each time. The copy itself is unavoidable without restructuring the
/// borrow relationship (the scanner needs `&mut self.tally` while it reads
/// these bytes); the ALLOCATION is not.
scratch: Vec<u8>,
/// PTS (ns) to stamp on the frame that begins the carry-over `buf` — i.e.
/// the running per-frame PTS at the point the partial tail was retained.
/// Used by `flush()` to time the final buffered frame at EOS.
@@ -81,6 +91,18 @@ pub struct Ac3Parser {
/// across the PES boundary because it may be the core of an AC-3-core +
/// E-AC-3-dependent frame set whose remaining substreams are in the next PES.
saw_extension: bool,
/// The access unit held open across the last PES boundary, ALREADY
/// scanned. The carry-over begins at its first byte, so without this the
/// next call re-scans and re-CRCs every syncframe of it from byte 0 — and
/// an access unit that keeps gaining substreams grows to [`MAX_AC3_BUF`]
/// (1 MiB) before the resync guard drops it, which on a ~2 KiB DVD PES is
/// three orders of magnitude of repeated work per packet.
held: Option<HeldAu>,
/// Test-only: syncframes examined (sized + CRC-gated) by
/// `scan_access_units`. Pins the resume above — the property it exists for
/// is a WORK bound, which no frame-level assertion can observe.
#[cfg(test)]
frames_scanned: u64,
}
impl Default for Ac3Parser {
@@ -92,10 +114,14 @@ impl Default for Ac3Parser {
impl Ac3Parser {
pub fn new() -> Self {
Self {
buf: Vec::with_capacity(4096),
scratch: Vec::new(),
acc: super::pesbuf::PesBuf::with_capacity(4096),
flush_pts_ns: 0,
tally: super::dropgate::DropTally::new("ac3"),
saw_extension: false,
held: None,
#[cfg(test)]
frames_scanned: 0,
}
}
@@ -137,20 +163,57 @@ impl Ac3Parser {
base_pts_ns: i64,
anchor: Option<PtsAnchor>,
at_eos: bool,
) -> (Vec<Frame>, usize, i64) {
marks: &[(usize, super::pesbuf::PesFacts)],
held: Option<HeldAu>,
) -> ScanOut {
let mut frames = Vec::new();
let mut pos = 0usize;
// Running PTS for the next access unit to emit in this call.
let mut frame_pts_ns = base_pts_ns;
let mut anchor = anchor;
let mut pending: Option<PendingAu> = None;
// How far this call has proved there is no further syncframe to
// process; carried over so the held access unit's own bytes (and the
// junk after them) are not searched again next call.
let mut scanned_to = 0usize;
// Resume a held access unit instead of re-deriving it. `keep_from` was
// its first byte, so it starts at 0 of this buffer, and every frame in
// it was sized and CRC-gated on the call that built it.
if let Some(h) = held {
let mut drop_reason = h.drop_reason;
// The one verdict that can have changed since: the track may have
// become poisoned while this access unit was held, and a re-scan
// would have picked that up.
if drop_reason.is_none() && self.tally.is_poisoned() {
drop_reason = Some("track-poisoned");
}
pending = Some(PendingAu {
start: 0,
end: h.end,
pts_ns: base_pts_ns,
duration_ns: h.duration_ns,
drop_reason,
bsid: h.bsid,
});
frame_pts_ns = base_pts_ns + h.duration_ns as i64;
pos = h.scanned_to;
scanned_to = h.scanned_to;
}
while pos < data.len() {
let sync = find_ac3_sync(&data[pos..]);
let start = match sync {
Some(offset) => pos + offset,
None => break,
None => {
// No syncword in `data[pos..]` at all: every byte but the
// last is proved sync-free (a syncword is two bytes and the
// second may still arrive).
scanned_to = data.len().saturating_sub(1).max(pos);
break;
}
};
scanned_to = start;
let remaining = &data[start..];
@@ -170,6 +233,7 @@ impl Ac3Parser {
// Invalid/sub-header frame size (e.g. an E-AC-3 frmsiz of 0/1
// sizing to a 2/4-byte fragment) — skip this sync word.
pos = start + 2;
scanned_to = pos;
continue;
}
@@ -179,6 +243,10 @@ impl Ac3Parser {
}
let frame = &data[start..start + frame_size];
#[cfg(test)]
{
self.frames_scanned += 1;
}
// Decodability gate: a syncframe with an out-of-range bsid (> 16) or
// a failed native CRC (payload corruption) poisons the access unit it
// belongs to — a dependent substream is useless without its parent and
@@ -219,7 +287,7 @@ impl Ac3Parser {
}
} else {
if let Some(au) = pending.take() {
close_access_unit(&mut self.tally, data, &au, &mut frames);
close_access_unit(&mut self.tally, data, &au, marks, &mut frames);
}
// First access unit that starts in this PES's own bytes: adopt
// this PES's timestamp so a genuine PTS jump is followed instead
@@ -246,6 +314,7 @@ impl Ac3Parser {
}
pos = start + frame_size;
scanned_to = pos;
}
// Close or HOLD the trailing access unit. The rest of its frame set — its
@@ -259,12 +328,24 @@ impl Ac3Parser {
// substream that extends an access unit — a plain AC-3 track keeps
// emitting every frame in-call.
let mut hold_from = None;
let mut held_out = None;
if let Some(au) = pending {
if !at_eos && (au.bsid >= 11 || self.saw_extension) {
frame_pts_ns = au.pts_ns;
hold_from = Some(au.start);
// Everything below `scanned_to` has been searched already, and
// the access unit's own frames have been sized and CRC-gated;
// record both, rebased onto the carry-over (which starts at
// `au.start`), so the next call resumes instead of redoing it.
held_out = Some(HeldAu {
end: au.end - au.start,
scanned_to: scanned_to.max(au.end) - au.start,
duration_ns: au.duration_ns,
drop_reason: au.drop_reason,
bsid: au.bsid,
});
} else {
close_access_unit(&mut self.tally, data, &au, &mut frames);
close_access_unit(&mut self.tally, data, &au, marks, &mut frames);
}
}
@@ -294,7 +375,12 @@ impl Ac3Parser {
None => data.len(),
};
(frames, keep_from, frame_pts_ns)
ScanOut {
frames,
keep_from,
frame_pts_ns,
held: held_out,
}
}
}
@@ -307,6 +393,36 @@ struct PtsAnchor {
pts_ns: i64,
}
/// What one `scan_access_units` pass produced: the access units it emitted,
/// the offset in the scanned buffer from which bytes must be carried over to
/// the next call, the PTS to stamp on the access unit that begins that
/// carry-over, and — when the trailing access unit was HELD — the state that
/// lets the next call resume rather than re-derive it.
struct ScanOut {
frames: Vec<Frame>,
keep_from: usize,
frame_pts_ns: i64,
held: Option<HeldAu>,
}
/// A trailing access unit held across the PES boundary, already scanned.
/// Offsets are relative to the carry-over, which begins at the access unit's
/// first byte — so the access unit occupies `0..end`.
#[derive(Clone, Copy)]
struct HeldAu {
/// End of the access unit's bytes.
end: usize,
/// How far the scan that built it had searched (`>= end`). Bytes below it
/// hold no further syncframe to process.
scanned_to: usize,
/// Duration contributed by the access unit's `substreamid`-0 substream.
duration_ns: u64,
/// Decodability verdict reached for it so far.
drop_reason: Option<&'static str>,
/// bsid of the substream that opened it.
bsid: u8,
}
/// An access unit (frame set) under construction: `data[start..end]` is the
/// `substreamid`-0 independent substream frame plus every substream appended to it
/// so far — its dependents, and any additional independent substreams 1..7 with
@@ -334,6 +450,7 @@ fn close_access_unit(
tally: &mut super::dropgate::DropTally,
data: &[u8],
au: &PendingAu,
marks: &[(usize, super::pesbuf::PesFacts)],
out: &mut Vec<Frame>,
) {
if let Some(reason) = au.drop_reason {
@@ -344,7 +461,9 @@ fn close_access_unit(
out.push(Frame {
discontinuity: false,
coding: None,
source: None,
// The packet covering this unit's FIRST byte — which is the packet its
// PTS came from too, when the unit began in an earlier PES.
source: super::pesbuf::facts_for(marks, au.start).source,
pts_ns: au.pts_ns,
keyframe: true,
data: data[au.start..au.end].to_vec(),
@@ -461,7 +580,9 @@ impl CodecParser for Ac3Parser {
// never be stranded by an empty post-gap PES (the demuxer only emits
// non-empty PES today; this is defensive for any future caller).
if pes.discontinuity {
self.buf.clear();
self.acc.clear();
// The held access unit's bytes went with it.
self.held = None;
}
if pes.data.is_empty() {
return Vec::new();
@@ -480,7 +601,7 @@ impl CodecParser for Ac3Parser {
// no anchor the running cadence simply continues. The discontinuity-
// carrying PES is a PUSI with a PTS in practice, so this is
// defense-in-depth.
let carry_len = self.buf.len();
let carry_len = self.acc.len();
let anchor = pes.pts.map(|p| PtsAnchor {
at: carry_len,
pts_ns: pts_to_ns(p),
@@ -489,11 +610,25 @@ impl CodecParser for Ac3Parser {
// Prepend leftover from previous PES, then take the whole buffer into a
// local so the scanner can call `self.tally` (the bytes are no longer
// borrowed from `self`). The unconsumed tail is written back at the end.
self.buf.extend_from_slice(&pes.data);
let buf = std::mem::take(&mut self.buf);
self.acc.push(pes);
// Copy the working bytes out so the scanner can borrow `self.tally`;
// the buffer keeps its marks, so the unconsumed tail stays attributed
// to the packet that carried it.
// Take the scratch OUT of `self` so the scanner can borrow `self.tally`
// while reading it; it is put back at the end of the call, keeping its
// capacity for the next PES. There is no early return after this point.
let mut buf = std::mem::take(&mut self.scratch);
buf.clear();
buf.extend_from_slice(self.acc.as_slice());
let marks = self.acc.marks_snapshot();
let data = &buf;
let (frames, keep_from, frame_pts_ns) =
self.scan_access_units(data, self.flush_pts_ns, anchor, false);
let held = self.held.take();
let ScanOut {
frames,
keep_from,
frame_pts_ns,
held: still_held,
} = self.scan_access_units(data, self.flush_pts_ns, anchor, false, &marks, held);
if keep_from < data.len() {
let tail = &data[keep_from..];
@@ -506,7 +641,8 @@ impl CodecParser for Ac3Parser {
"ac3: carry-over buffer exceeded {} bytes without a frame; dropping and resyncing",
MAX_AC3_BUF
);
self.buf.clear();
self.acc.clear();
self.held = None;
// Advance the cadence, as both sibling branches below do, so the
// three paths out of this block cannot disagree. Defensive: no
// input reaching this parser was found that both parses frames and
@@ -514,7 +650,8 @@ impl CodecParser for Ac3Parser {
// prevents is not currently reachable and has no regression test.
self.flush_pts_ns = frame_pts_ns;
} else {
self.buf = tail.to_vec();
self.acc.drain(keep_from);
self.held = still_held;
// The carried bytes, when later completed and emitted (next call
// or by flush() at EOS), are timed at the PTS the scanner reached
// here: the PTS of the next access unit in presentation order, or
@@ -523,13 +660,17 @@ impl CodecParser for Ac3Parser {
self.flush_pts_ns = frame_pts_ns;
}
} else {
self.buf.clear();
self.acc.clear();
self.held = None;
// Nothing carried, but keep the cadence so a following PES with no
// PTS (no anchor) continues the timeline instead of reusing a stale
// value.
self.flush_pts_ns = frame_pts_ns;
}
// Hand the working buffer back so the next PES reuses its capacity.
// `data` borrowed it; that borrow ends here, at its last use.
self.scratch = buf;
frames
}
@@ -539,10 +680,13 @@ impl CodecParser for Ac3Parser {
// with no following PES to close it, and without this drain the last
// ~32 ms of audio is lost. `at_eos` closes the trailing access unit
// instead of holding it; a partial/garbage tail yields nothing.
let buf = std::mem::take(&mut self.buf);
let buf = self.acc.as_slice().to_vec();
let marks = self.acc.marks_snapshot();
self.acc.clear();
let held = self.held.take();
let out = self
.scan_access_units(&buf, self.flush_pts_ns, None, true)
.0;
.scan_access_units(&buf, self.flush_pts_ns, None, true, &marks, held)
.frames;
// Aggregate drop report at end-of-stream (warn-level, always visible).
self.tally.log_summary();
out
@@ -794,6 +938,73 @@ mod tests {
frame[n - 1] = (c & 0xFF) as u8;
}
/// The per-PES working buffer must be REUSED, not reallocated.
///
/// `parse` runs once per PES packet — of the order of 10^5 times on a
/// feature's audio track — and used to build a fresh `Vec` every call. The
/// copy itself cannot go without restructuring the borrow relationship, but
/// the allocation can: the buffer is taken out of `self`, filled, and put
/// back with its capacity intact.
///
/// Asserting on capacity is deliberately white-box, because that is exactly
/// what regresses if someone reverts to `to_vec()` — the behaviour would be
/// identical and no other test would notice.
#[test]
fn the_working_buffer_is_reused_across_packets_not_reallocated() {
let mut parser = Ac3Parser::new();
let frame = make_ac3_frame(0, 0);
parser.parse(&PesPacket {
source: None,
pid: 0x1100,
pts: Some(90_000),
dts: None,
data: frame.clone(),
discontinuity: false,
});
let cap_after_first = parser.scratch.capacity();
assert!(
cap_after_first > 0,
"the buffer must be handed back to the parser, not dropped"
);
// The discriminator: a LARGE packet, then a small one. A reused buffer
// keeps the large capacity; a fresh `Vec` sized to each packet drops
// back to the small one. Feeding equal-sized packets cannot tell those
// apart — the first version of this test did exactly that and a
// to_vec()-per-call implementation passed it.
let big = [frame.clone(), frame.clone(), frame.clone(), frame.clone()].concat();
parser.parse(&PesPacket {
source: None,
pid: 0x1100,
pts: None,
dts: None,
data: big.clone(),
discontinuity: false,
});
let cap_after_big = parser.scratch.capacity();
assert!(
cap_after_big >= big.len(),
"the buffer must have grown to hold the large packet"
);
parser.parse(&PesPacket {
source: None,
pid: 0x1100,
pts: None,
dts: None,
data: frame.clone(),
discontinuity: false,
});
assert_eq!(
parser.scratch.capacity(),
cap_after_big,
"after a small packet the buffer must STILL hold the large \
capacity — a fresh allocation per call would have shrunk to the \
small packet's size"
);
}
#[test]
fn parse_empty_pes() {
let mut parser = Ac3Parser::new();
@@ -1030,15 +1241,15 @@ mod tests {
let frames = parser.parse(&pes);
assert!(frames.is_empty());
assert!(
parser.buf.len() <= MAX_AC3_BUF,
parser.acc.len() <= MAX_AC3_BUF,
"buffer grew to {} (cap {})",
parser.buf.len(),
parser.acc.len(),
MAX_AC3_BUF
);
}
// After all that garbage the retained tail is at most a single partial
// syncword byte — never an accumulation of whole PES packets.
assert!(parser.buf.len() <= 1, "retained {} bytes", parser.buf.len());
assert!(parser.acc.len() <= 1, "retained {} bytes", parser.acc.len());
}
#[test]
@@ -1057,7 +1268,11 @@ mod tests {
discontinuity: false,
};
assert!(parser.parse(&pes).is_empty());
assert_eq!(parser.buf, vec![0x0B], "lone trailing 0x0B retained");
assert_eq!(
parser.acc.as_slice(),
vec![0x0B],
"lone trailing 0x0B retained"
);
}
#[test]
@@ -1067,14 +1282,14 @@ mod tests {
// ac3 inherited the no-op default flush and dropped the last frame.
let mut parser = Ac3Parser::new();
let frame_data = make_ac3_frame(0, 2);
parser.buf = frame_data.clone();
parser.acc.seed(&frame_data.clone());
parser.flush_pts_ns = pts_to_ns(99000);
let f = parser.flush();
assert_eq!(f.len(), 1, "complete buffered frame drained at EOS");
assert_eq!(f[0].data.len(), 160);
assert_eq!(f[0].pts_ns, pts_to_ns(99000), "flush uses carried PTS");
assert!(f[0].duration_ns.is_some(), "flush sets duration");
assert!(parser.buf.is_empty(), "buffer consumed by flush");
assert!(parser.acc.is_empty(), "buffer consumed by flush");
}
#[test]
@@ -1107,7 +1322,7 @@ mod tests {
// emitted truncated.
let mut parser = Ac3Parser::new();
let frame_data = make_ac3_frame(0, 2);
parser.buf = frame_data[..80].to_vec(); // half a frame
parser.acc.seed(&frame_data[..80]); // half a frame
assert!(parser.flush().is_empty(), "partial tail dropped");
}
@@ -1511,7 +1726,7 @@ mod tests {
// 100 bytes.
let mut parser = Ac3Parser::new();
let frame = make_ac3_frame(0, 2); // sizes to 160
parser.buf = frame[..100].to_vec();
parser.acc.seed(&frame[..100]);
assert!(
parser.flush().is_empty(),
"incomplete frame must not be emitted truncated at flush"
@@ -1522,7 +1737,7 @@ mod tests {
fn flush_with_no_sync_is_empty() {
// flush on a buffer with no syncword yields nothing and clears.
let mut parser = Ac3Parser::new();
parser.buf = vec![0xAA, 0xBB, 0xCC];
parser.acc.seed(&[0xAA, 0xBB, 0xCC]);
assert!(parser.flush().is_empty());
}
@@ -1839,7 +2054,7 @@ mod tests {
assert_eq!(fr.pts_ns, pts_to_ns(90000) + i as i64 * 32_000_000);
assert_eq!(fr.duration_ns, Some(32_000_000));
}
assert!(parser.buf.is_empty(), "nothing held back for plain AC-3");
assert!(parser.acc.is_empty(), "nothing held back for plain AC-3");
assert!(parser.flush().is_empty(), "flush has nothing left to drain");
}
@@ -2099,6 +2314,125 @@ mod tests {
);
}
/// A 256-byte E-AC-3 syncframe with a valid CRC. `strmtyp`/`substreamid`
/// go into byte 2 (A/52 Annex E), which is what `substream_role` reads:
/// (0, 0) OPENS an access unit, (1, 0) is a dependent substream that
/// EXTENDS the open one.
fn eac3_substream_frame(strmtyp: u8, substreamid: u8) -> Vec<u8> {
const SIZE: usize = 256;
let frmsiz = SIZE / 2 - 1; // (frmsiz + 1) * 2 == SIZE
let mut f = vec![0u8; SIZE];
f[0] = 0x0B;
f[1] = 0x77;
f[2] = (strmtyp << 6) | (substreamid << 3) | ((frmsiz >> 8) as u8 & 0x07);
f[3] = (frmsiz & 0xFF) as u8;
f[5] = 16 << 3; // bsid 16 → E-AC-3
finalize_ac3_crc(&mut f);
f
}
/// An access unit closes only at the next `substreamid`-0 independent
/// substream, so one that keeps gaining dependent substreams stays OPEN
/// across PES boundaries and its bytes stay in the carry-over. The
/// carry-over must not be re-scanned — and re-CRCed — from the access
/// unit's first byte on every packet: the buffer only stops growing at
/// MAX_AC3_BUF (1 MiB), and a PES on a DVD is about 2 KiB, so re-deriving
/// the held access unit costs work quadratic in the packets fed.
///
/// Measured directly, because a work bound has no frame-level shadow:
/// `frames_scanned` counts the syncframes the scanner sizes and CRC-gates.
/// Re-scanning from byte 0 examines 1 + 2 + ... + (N+1) frames.
///
/// Mutation: pass `None` for `held` in `parse` (or drop the `if let
/// Some(h) = held` resume) — the count returns to the quadratic figure.
#[test]
fn a_held_access_unit_is_not_rescanned_from_its_first_frame_every_packet() {
const DEPENDENTS: usize = 200;
let mut parser = Ac3Parser::new();
// Opens the access unit.
let emitted = parser.parse(&make_eac3_pes(eac3_substream_frame(0, 0)));
assert!(
emitted.is_empty(),
"the access unit is held open, not emitted"
);
for _ in 0..DEPENDENTS {
let f = parser.parse(&make_eac3_pes(eac3_substream_frame(1, 0)));
assert!(f.is_empty(), "a dependent substream extends the open unit");
}
let fed = (DEPENDENTS + 1) as u64;
assert!(
parser.frames_scanned <= 2 * fed,
"the scanner examined {} syncframes for {fed} fed — a held access \
unit must be resumed, not re-derived",
parser.frames_scanned
);
// ...and the resume must not have cost correctness: the whole frame
// set is still one access unit, emitted intact at EOS.
let out = parser.flush();
assert_eq!(out.len(), 1, "the frame set is a single access unit");
assert_eq!(
out[0].data.len(),
256 * (DEPENDENTS + 1),
"every substream of the frame set belongs to it"
);
}
/// A concealed gap must drop the HELD access unit, not just the byte
/// buffer.
///
/// `parse` clears `self.acc` on a discontinuity because the buffered bytes
/// are a truncated frame. The held access unit is described by OFFSETS into
/// exactly those bytes, so it has to go with them. Without
/// `self.held = None`, the next packet resumes a HeldAu whose `start`/`end`
/// were computed against the pre-gap buffer but are applied to the
/// unrelated post-gap bytes — splicing audio across the gap at best, and
/// indexing past the end of the new, shorter buffer at worst.
///
/// The two existing discontinuity tests use plain AC-3 (bsid < 11), which
/// never holds an access unit open, so neither of them reaches this reset.
#[test]
fn a_discontinuity_drops_the_held_access_unit_with_its_bytes() {
let mut parser = Ac3Parser::new();
// Open an access unit and extend it, so a HeldAu exists describing
// offsets into a large buffer.
assert!(
parser
.parse(&make_eac3_pes(eac3_substream_frame(0, 0)))
.is_empty(),
"the access unit is held open, not emitted"
);
for _ in 0..8 {
assert!(
parser
.parse(&make_eac3_pes(eac3_substream_frame(1, 0)))
.is_empty(),
"a dependent substream extends the open unit"
);
}
// The gap. Its post-gap payload is deliberately far SHORTER than the
// held unit's bytes, so a stale HeldAu indexes past its end.
let mut gap = make_eac3_pes(eac3_substream_frame(0, 0));
gap.discontinuity = true;
let _ = parser.parse(&gap);
// Whatever comes out, nothing may carry pre-gap bytes: the truncated
// unit was dropped, so the only access unit that can be emitted is the
// one opened after the gap.
let out = parser.flush();
let total: usize = out.iter().map(|f| f.data.len()).sum();
assert!(
total <= 256,
"a post-gap access unit must not be spliced onto the 9 frames held \
before the gap; got {total} bytes across {} frame(s)",
out.len()
);
}
// helper: PES with a generic pts for E-AC-3 tests
fn make_eac3_pes(data: Vec<u8>) -> PesPacket {
PesPacket {
@@ -2110,4 +2444,126 @@ mod tests {
discontinuity: false,
}
}
/// An access unit that began in an earlier packet keeps THAT packet's
/// source offset. The packet that completes it is a different clip at a
/// seam, and taking its offset places the audio in the wrong one.
#[test]
fn an_access_unit_carries_the_source_of_the_packet_it_began_in() {
let mut parser = Ac3Parser::new();
let frame = make_ac3_frame(0, 4);
let mut p1 = PesPacket {
pid: 0x1100,
pts: Some(90_000),
dts: None,
data: frame[..frame.len() / 2].to_vec(),
source: Some(crate::pes::SourcePos::at_byte(1_000)),
discontinuity: false,
};
p1.data.truncate(frame.len() / 2);
assert!(parser.parse(&p1).is_empty(), "partial frame held");
let mut rest = frame[frame.len() / 2..].to_vec();
rest.extend_from_slice(&make_ac3_frame(0, 4));
let p2 = PesPacket {
pid: 0x1100,
pts: Some(180_000),
dts: None,
data: rest,
source: Some(crate::pes::SourcePos::at_byte(9_000)),
discontinuity: false,
};
let frames = parser.parse(&p2);
assert!(!frames.is_empty(), "the completed unit is emitted");
assert_eq!(
frames[0].source.map(|s| s.byte),
Some(1_000),
"the unit belongs to the packet its FIRST byte came from"
);
}
/// A track that becomes POISONED while an access unit is held open across a
/// PES boundary must not emit that access unit when it resumes.
///
/// Mutation this catches: deleting (or inverting) the resume-path re-check
/// `if drop_reason.is_none() && self.tally.is_poisoned()` at the top of
/// `scan_access_units`. The verdict on a held access unit is frozen at the
/// moment it was OPENED, and `ac3_drop_reason` reads the tally BEFORE the
/// previous access unit is closed — so the very drop that crosses the
/// poison threshold lands after the next unit's verdict was already taken
/// as `None`. Without the re-check that unit is emitted as an ordinary
/// frame after the track has been judged too damaged to mux: corrupt audio
/// passed through as success, and worse, it is the ONE frame that escapes
/// a whole-track fallback whose entire point is that nothing after the
/// verdict ships.
///
/// Neither existing held-AU test reaches this: both keep a pristine tally.
///
/// The fixture drives the exact interleaving above. `DropTally` poisons
/// once `verified_dropped * 2 > kept + dropped` past the 200-AU gate, so
/// one PES carries 200 CRC-failing `substreamid`-0 syncframes followed by a
/// clean one. Closing corrupt unit #200 (which happens only when the clean
/// frame is reached) latches the poison — after that clean frame's own
/// verdict was computed. Being an E-AC-3 unit that can still gain
/// substreams, it is then HELD across the boundary with `drop_reason:
/// None`, which is precisely the state the re-check exists for.
#[test]
fn a_held_access_unit_is_dropped_when_the_track_poisons_before_it_resumes() {
// One more than the verdict gate: the Nth close is what poisons.
const CORRUPT_AUS: usize = 200;
let mut parser = Ac3Parser::new();
let mut data = Vec::new();
for _ in 0..CORRUPT_AUS {
let mut f = eac3_substream_frame(0, 0);
// Corrupt a payload byte AFTER the CRC was finalized: the header
// (sizing, strmtyp/substreamid, bsid) stays intact so the frame is
// still parsed as a whole access unit and fails only the CRC —
// a VERIFIED drop, the only kind that feeds the poison verdict.
f[100] ^= 0xFF;
assert!(!frame_crc_ok(&f), "the fixture frame must fail its CRC");
data.extend_from_slice(&f);
}
// The clean access unit. Last in the PES, so it is held open.
let clean = eac3_substream_frame(0, 0);
assert!(
frame_crc_ok(&clean),
"the held unit is individually decodable"
);
data.extend_from_slice(&clean);
let emitted = parser.parse(&make_eac3_pes(data));
assert!(
emitted.is_empty(),
"every corrupt unit is dropped and the clean one is held; got {} frame(s)",
emitted.len()
);
// The state the re-check depends on: the track IS poisoned, and the
// held unit was opened before that verdict existed.
assert!(
parser.tally.is_poisoned(),
"fixture must actually cross the whole-track poison threshold"
);
// Resume. The next PES opens a new unit, which closes the held one.
let out = parser.parse(&make_eac3_pes(eac3_substream_frame(0, 0)));
assert!(
out.is_empty(),
"an access unit held across the boundary must not be emitted once \
the track is poisoned; got {} frame(s) totalling {} bytes",
out.len(),
out.iter().map(|f| f.data.len()).sum::<usize>()
);
// ...and nothing may leak at end of stream either.
let tail = parser.flush();
assert!(
tail.is_empty(),
"a poisoned track emits nothing at EOS; got {} frame(s)",
tail.len()
);
assert!(
parser.dropped_frames() > CORRUPT_AUS as u64,
"the held unit must be ACCOUNTED as a drop, not silently discarded"
);
}
}
+71 -3
View File
@@ -50,7 +50,15 @@ fn adts_verdict(data: &[u8]) -> AdtsVerdict {
// aac_frame_length: 13 bits = byte3[1:0] | byte4 | byte5[7:5].
let frame_length =
((u32::from(data[3]) & 0x03) << 11) | (u32::from(data[4]) << 3) | (u32::from(data[5]) >> 5);
if frame_length < 7 {
// The floor is the header the frame SAYS it carries, not a constant.
// protection_absent (byte1 bit0) clear means a 16-bit crc_check follows the
// 7-byte fixed+variable header, so the frame cannot be shorter than 9 —
// aac_frame_length counts the header and the CRC, not just the payload.
// Comparing against a flat 7 let a CRC-present frame declaring 7 or 8
// through as structurally Valid, and the muxer then carried a frame whose
// own header says it is impossible.
let header_bytes = if data[1] & 0x01 == 0 { 9 } else { 7 };
if frame_length < header_bytes {
return AdtsVerdict::Invalid;
}
AdtsVerdict::Valid
@@ -112,10 +120,14 @@ impl CodecParser for AdtsParser {
}
self.tally.record_kept();
// One PES is one unit here, so the unit's first byte is in THIS packet
// and its facts are this packet's -- the same rule the buffering
// parsers apply through `PesBuf::front`, with nothing carried over.
let facts = super::pesbuf::PesFacts::of(pes);
vec![Frame {
discontinuity: pes.discontinuity,
discontinuity: facts.discontinuity,
coding: None,
source: None,
source: facts.source,
pts_ns,
keyframe: true,
data: pes.data.clone(),
@@ -148,6 +160,51 @@ mod tests {
}
}
/// A header that CLAIMS a CRC (protection_absent = 0) but declares a
/// frame length too short to contain one.
///
/// `aac_frame_length` counts the header and the CRC, not just the payload,
/// so with a CRC present the smallest structurally possible frame is 9
/// bytes: the 7-byte fixed+variable header plus the 16-bit crc_check.
/// The gate compared against a flat 7 and never read protection_absent at
/// all, so a frame whose own header says it is impossible was classified
/// Valid and forwarded to the muxer.
#[test]
fn a_crc_present_header_shorter_than_its_own_crc_is_invalid() {
for declared in [7u32, 8] {
let mut f = adts_frame(16);
f[1] = 0xF0; // sync + MPEG-4, protection_absent = 0 => CRC present
f[3] = (f[3] & 0xFC) | ((declared >> 11) & 0x03) as u8;
f[4] = ((declared >> 3) & 0xFF) as u8;
f[5] = (f[5] & 0x1F) | ((declared & 0x07) << 5) as u8;
assert!(
matches!(adts_verdict(&f), AdtsVerdict::Invalid),
"protection_absent=0 declaring {declared} bytes cannot hold its \
own 7-byte header plus a 2-byte CRC"
);
}
// 9 is the smallest length that CAN hold header + CRC, so it must pass
// the structural gate — the floor moved, it did not become stricter
// than the spec.
let mut ok = adts_frame(16);
ok[1] = 0xF0;
let nine = 9u32;
ok[3] = (ok[3] & 0xFC) | ((nine >> 11) & 0x03) as u8;
ok[4] = ((nine >> 3) & 0xFF) as u8;
ok[5] = (ok[5] & 0x1F) | ((nine & 0x07) << 5) as u8;
assert!(matches!(adts_verdict(&ok), AdtsVerdict::Valid));
// And with NO CRC the floor is still 7, unchanged.
let mut no_crc = adts_frame(16);
no_crc[1] = 0xF1; // protection_absent = 1
let seven = 7u32;
no_crc[3] = (no_crc[3] & 0xFC) | ((seven >> 11) & 0x03) as u8;
no_crc[4] = ((seven >> 3) & 0xFF) as u8;
no_crc[5] = (no_crc[5] & 0x1F) | ((seven & 0x07) << 5) as u8;
assert!(matches!(adts_verdict(&no_crc), AdtsVerdict::Valid));
}
/// A valid ADTS header (AAC-LC, 44.1 kHz, stereo) + payload, with
/// aac_frame_length set to the total size.
fn adts_frame(payload: usize) -> Vec<u8> {
@@ -272,4 +329,15 @@ mod tests {
let f = p.parse(&make_pes(vec![0xFF, 0xF1, 0x50], Some(0)));
assert_eq!(f.len(), 1, "too short to validate → kept");
}
/// One PES is one unit here, so the frame carries that packet's offset.
#[test]
fn a_frame_carries_its_packets_source() {
let mut parser = AdtsParser::new();
let mut p = make_pes(adts_frame(64), Some(90_000));
p.source = Some(crate::pes::SourcePos::at_byte(4_242));
let frames = parser.parse(&p);
assert!(!frames.is_empty(), "a valid ADTS frame is emitted");
assert_eq!(frames[0].source.map(|s| s.byte), Some(4_242));
}
}
+18 -6
View File
@@ -29,10 +29,17 @@ pub(crate) fn crc16_ansi(data: &[u8]) -> u16 {
}
/// CRC-16 with polynomial 0x002D, init 0, MSB-first, used by the MLP / Dolby
/// TrueHD major-sync header checksum. NOTE: MLP's checksum is the "reversed"
/// scheme — the stored trailer word is the little-endian-read CRC, so this
/// standard CRC must be compared against the stored bytes read big-endian.
/// The caller handles that comparison (see `truehd::mlp_major_sync_ok`).
/// TrueHD major-sync header checksum.
///
/// NOTE: MLP's checksum is the "reversed" scheme. This function emits its two
/// bytes in the OPPOSITE order to a standard little-endian CRC readout, so the
/// caller swaps them back and compares against the stored trailer word read
/// LITTLE-endian — see `truehd::mlp_major_sync_crc_ok`, which is authoritative.
///
/// Comparing big-endian instead is precisely the bug that function was fixed
/// for: it could never validate a real extended major sync, so whole TrueHD
/// tracks were dropped silently. This comment used to prescribe exactly that,
/// and to point at a `truehd::mlp_major_sync_ok` that does not exist.
/// Verified against real MLP/TrueHD bitstreams (225/225 major-sync AUs).
pub(crate) fn crc16_mlp(data: &[u8]) -> u16 {
let mut crc: u16 = 0;
@@ -103,8 +110,13 @@ mod tests {
#[test]
fn crc16_mlp_residue_property_holds() {
// Appending the big-endian CRC zeroes the residue over message+crc — the
// scheme `truehd::mlp_major_sync_ok` relies on.
// Appending the big-endian CRC zeroes the residue over message+crc.
// This is a property of the CRC itself, pinned here so a change to the
// polynomial or the bit order is caught. It is NOT how the TrueHD
// caller validates a major sync: `truehd::mlp_major_sync_crc_ok` does a
// swap-and-XOR compare against the little-endian trailer word. (This
// comment used to claim the caller relied on the residue, and named a
// `truehd::mlp_major_sync_ok` that does not exist.)
let msg = [0xF8u8, 0x72, 0x6F, 0xBA];
let c = crc16_mlp(&msg);
let mut framed = msg.to_vec();
+157 -115
View File
@@ -6,7 +6,7 @@
//! are emitted complete.
use super::startcode::BitReader;
use super::{CodecParser, Frame, PesPacket, pts_to_ns};
use super::{CodecParser, Frame, PesPacket};
const DTS_CORE_SYNC: [u8; 4] = [0x7F, 0xFE, 0x80, 0x01];
/// DTS-HD extension substream syncword. An access unit is delimited by the next
@@ -21,23 +21,16 @@ const DTS_HD_EXT_SYNC: [u8; 4] = [0x64, 0x58, 0x20, 0x25];
/// This preserves the lossless extension data instead of downgrading to lossy
/// core (the lossy-core downgrade bug).
pub struct DtsParser {
buf: Vec<u8>,
/// Bytes assembled across PES packets, each attributable to the packet
/// that carried it. An emitted unit takes the facts of the packet covering
/// its FIRST byte, so an AU whose core arrived in an earlier PES keeps that
/// core's timestamp and source offset when its extensions arrive later.
acc: super::pesbuf::PesBuf,
/// PTS of the access unit currently being assembled in `buf` (the unit
/// starting at the first buffered core sync). Captured when that core
/// frame's PES first arrived; the trailing extension-substream PES
/// packets carry their own (later) PTS which must NOT override it.
pending_pts: i64,
/// PTS markers attributing buffer regions to their source PES. Each entry
/// is `(buffer_offset, pts_ns)` for the PES whose bytes begin at that
/// offset. When an access unit is emitted from the front of `buf`, its PTS
/// is the marker covering offset 0 — NOT the most recent PES's PTS. This is
/// what fixes multi-AU-per-call PTS attribution: if a PES carries the
/// extension substreams (and possibly the next core) for an AU whose own
/// core arrived in an earlier PES, the emitted AU keeps its own core PES's
/// timestamp instead of the later PES's. Offsets are kept relative to the
/// current `buf` start and rebased whenever bytes are drained from the
/// front.
pts_marks: std::collections::VecDeque<(usize, i64)>,
/// The `front_pts` of the PREVIOUS emitted access unit. When the current
/// AU's `front_pts` differs, it began a new PES → re-base to it. When it is
/// unchanged, this AU shares the previous AU's PES → advance one frame
@@ -66,9 +59,8 @@ impl Default for DtsParser {
impl DtsParser {
pub fn new() -> Self {
Self {
buf: Vec::with_capacity(32768),
acc: super::pesbuf::PesBuf::with_capacity(32768),
pending_pts: 0,
pts_marks: std::collections::VecDeque::new(),
last_front_pts: PTS_UNSET,
next_pts_ns: PTS_UNSET,
tally: super::dropgate::DropTally::new("dts"),
@@ -92,7 +84,14 @@ impl DtsParser {
/// PTS clock (which the caller advances whether or not the AU survives), so
/// a drop leaves the following audio on its true timeline — a gap, not a
/// shift. Every drop is logged (fail-loud, never silent).
fn emit_or_drop(&mut self, au: Vec<u8>, au_pts: i64, dur_ns: i64, out: &mut Vec<Frame>) {
fn emit_or_drop(
&mut self,
au: Vec<u8>,
au_pts: i64,
dur_ns: i64,
src: Option<crate::pes::SourcePos>,
out: &mut Vec<Frame>,
) {
let verdict = if self.tally.is_poisoned() {
Err(DropReason::TrackPoisoned)
} else {
@@ -104,7 +103,9 @@ impl DtsParser {
out.push(Frame {
discontinuity: false,
coding: None,
source: None,
// From the SAME packet as `au_pts` — both are the facts of
// the PES covering this unit's first byte.
source: src,
pts_ns: au_pts,
keyframe: true,
data: au,
@@ -148,41 +149,25 @@ impl DtsParser {
base
}
/// Drop `n` bytes from the front of `buf` and rebase the PTS markers so
/// their offsets stay relative to the new buffer start. A marker that now
/// sits at or before offset 0 is clamped to 0 (it still covers the front).
/// Redundant markers all at offset 0 collapse to the last one.
/// Drop `n` bytes from the front, rebasing attribution onto the new front.
fn drain_front(&mut self, n: usize) {
if n == 0 {
return;
}
self.buf.drain(..n);
for m in &mut self.pts_marks {
m.0 = m.0.saturating_sub(n);
}
// Collapse all leading markers that now sit at offset 0 to the last
// such marker — that is the PES whose data currently begins the buffer.
let last_zero = self
.pts_marks
.iter()
.rposition(|&(off, _)| off == 0)
.filter(|&i| i > 0);
if let Some(i) = last_zero {
self.pts_marks.drain(..i);
}
self.acc.drain(n);
}
/// PTS that should be stamped on an access unit currently at the front of
/// `buf` (offset 0): the most recent marker at offset 0, falling back to
/// `pending_pts`.
/// PTS for the access unit at the front of the buffer: the facts of the
/// packet covering offset 0, falling back to the unit's captured base.
fn front_pts(&self) -> i64 {
self.pts_marks
.iter()
.rev()
.find(|&&(off, _)| off == 0)
.map(|&(_, pts)| pts)
self.acc
.front()
.presentation_ns()
.unwrap_or(self.pending_pts)
}
/// Source offset for that same unit — from the SAME packet as its PTS,
/// which is the property the shared buffer exists to guarantee.
fn front_source(&self) -> Option<crate::pes::SourcePos> {
self.acc.front().source
}
}
/// Hard cap on a buffered access unit (core + all its extension substreams).
@@ -227,8 +212,7 @@ impl CodecParser for DtsParser {
// never be stranded by an empty post-gap PES (defensive; the demuxer only
// emits non-empty PES today).
if pes.discontinuity {
self.buf.clear();
self.pts_marks.clear();
self.acc.clear();
self.pending_pts = PTS_UNSET;
// A concealed gap is a timeline discontinuity: let the post-gap AU
// re-base to its own PES PTS rather than the pre-gap cursor.
@@ -242,17 +226,15 @@ impl CodecParser for DtsParser {
// guard at a post-gap continuation) must NOT reset the timeline to 0;
// continue from the most recent known base. Defense-in-depth: the
// discontinuity-carrying PES is a PUSI with a PTS in practice.
let pts_ns = pes.pts.map(pts_to_ns).unwrap_or_else(|| {
self.pts_marks
.back()
.map(|&(_, p)| p)
.filter(|&p| p >= 0)
.unwrap_or(if self.pending_pts >= 0 {
self.pending_pts
} else {
0
})
});
// A PES with no PTS (rare for audio, but legal) must NOT reset the
// timeline to 0 — continue from the most recent known base.
let pts_ns = super::pesbuf::PesFacts::of(pes)
.presentation_ns()
.unwrap_or(if self.pending_pts >= 0 {
self.pending_pts
} else {
0
});
// On Blu-ray, a DTS-HD MA/HRA access unit is a DTS core frame
// (sync 0x7FFE8001) followed by one or more DTS extension substreams
@@ -272,7 +254,7 @@ impl CodecParser for DtsParser {
// prior forced (safety-valve) flush left it invalidated — in the
// forced case the bytes still in `buf` are not a real core frame, so
// the first PES to arrive after the flush carries the correct base.
if self.buf.is_empty() || self.pending_pts == PTS_UNSET {
if self.acc.is_empty() || self.pending_pts == PTS_UNSET {
self.pending_pts = pts_ns;
}
// Mark where THIS PES's bytes begin in the buffer, with its PTS. The
@@ -280,21 +262,22 @@ impl CodecParser for DtsParser {
// (see `front_pts`), so an AU whose core arrived in an earlier PES keeps
// that core's timestamp even when its extensions / the following core
// arrive (with a later PTS) in this same parse() call.
// (pts_marks is bounded implicitly: an empty PES returns above without
// pushing a mark, and a non-empty run grows `buf`, which is cleared —
// along with pts_marks — once it exceeds MAX_AU_BYTES.)
self.pts_marks.push_back((self.buf.len(), pts_ns));
self.buf.extend_from_slice(&pes.data);
// `pts_ns` is this packet's own timestamp, or the carried-forward base
// when it had none; the source offset is always this packet's.
self.acc.push_with(
&pes.data,
super::pesbuf::PesFacts::of(pes).with_pts_ns(pts_ns),
);
let mut frames = Vec::new();
loop {
// Resync to the first core sync; drop any leading junk.
let Some(start) = find_sync(&self.buf, &DTS_CORE_SYNC) else {
let Some(start) = find_sync(self.acc.as_slice(), &DTS_CORE_SYNC) else {
// No core sync at all yet — keep at most a 3-byte tail so a
// sync split across PES packets can still be found next time.
if self.buf.len() > 3 {
let tail = self.buf.len() - 3;
if self.acc.len() > 3 {
let tail = self.acc.len() - 3;
self.drain_front(tail);
}
break;
@@ -305,17 +288,17 @@ impl CodecParser for DtsParser {
// offset 0 by construction, so a re-scan would be a redundant
// O(buf_len) walk per iteration; assert the invariant instead.
debug_assert_eq!(
find_sync(&self.buf, &DTS_CORE_SYNC),
find_sync(self.acc.as_slice(), &DTS_CORE_SYNC),
Some(0),
"drain_front(start) must leave the core sync at offset 0"
);
}
// Need the core header to size the core frame.
if self.buf.len() < CORE_HEADER_MIN_BYTES {
if self.acc.len() < CORE_HEADER_MIN_BYTES {
break;
}
let core_size = dts_core_frame_size(&self.buf);
let core_size = dts_core_frame_size(self.acc.as_slice());
// `dts_core_frame_size` returns a 14-bit `fsize + 1`, so it is
// always in [1, 16384]; the bare `== 0` / `> MAX_AU_BYTES` checks
// can never fire. A real DTS core frame is at least
@@ -330,7 +313,7 @@ impl CodecParser for DtsParser {
self.drain_front(4);
continue;
}
if self.buf.len() < core_size {
if self.acc.len() < core_size {
break; // core frame not fully buffered yet — wait
}
@@ -349,9 +332,9 @@ impl CodecParser for DtsParser {
// flush is an extension-substream PES, carrying its own later
// timestamp) must NOT become the next unit's PTS base.
let mut forced = false;
let (au_end, ext_clean) = match next_core_boundary(&self.buf, core_size) {
let (au_end, ext_clean) = match next_core_boundary(self.acc.as_slice(), core_size) {
NextCore::Found { end, ext_clean } => (end, ext_clean),
NextCore::NeedMore if self.buf.len() <= MAX_AU_BYTES => break,
NextCore::NeedMore if self.acc.len() <= MAX_AU_BYTES => break,
NextCore::NeedMore => {
// A candidate boundary exists but is not fully buffered. Normally
// we wait for more PES; but once the buffer exceeds the AU cap,
@@ -359,7 +342,7 @@ impl CodecParser for DtsParser {
// stream that keeps a boundary perpetually incomplete can't grow
// `buf` without bound (the `break` above never reaches it).
forced = true;
(self.buf.len(), true)
(self.acc.len(), true)
}
NextCore::None => {
// No next core sync buffered yet. The trailing extension
@@ -367,11 +350,11 @@ impl CodecParser for DtsParser {
// them rather than emit a core-only (lossy) frame — unless
// the buffer has grown unreasonably large, in which case
// emit what we have to guarantee forward progress.
if self.buf.len() <= MAX_AU_BYTES {
if self.acc.len() <= MAX_AU_BYTES {
break;
}
forced = true;
(self.buf.len(), true)
(self.acc.len(), true)
}
};
@@ -384,7 +367,7 @@ impl CodecParser for DtsParser {
// rather than shipping it. A recognized-but-unsizeable extension
// (`ext_clean == true`) is preserved in full (lossless).
let emit_end = if ext_clean { au_end } else { core_size };
let au: Vec<u8> = self.buf[..emit_end].to_vec();
let au: Vec<u8> = self.acc.as_slice()[..emit_end].to_vec();
// The AU's own core PES PTS (the PES covering its first byte, even if
// that PES preceded the one(s) carrying its extensions or the next
// core), stamped monotonically: honored when it advances past the
@@ -396,7 +379,10 @@ impl CodecParser for DtsParser {
// would: the following AU keeps its true PTS and the drop is a gap,
// never a shift. `emit_or_drop` decides whether to actually push it.
let au_pts = self.stamp_pts(self.front_pts(), dur_ns);
self.emit_or_drop(au, au_pts, dur_ns, &mut frames);
// Read BEFORE draining: after the drain the front is the NEXT
// unit's packet, not this one's.
let au_src = self.front_source();
self.emit_or_drop(au, au_pts, dur_ns, au_src, &mut frames);
self.drain_front(au_end);
// After draining, the marker covering the new front (if any) carries
// the next AU's PTS; `pending_pts` is only the fallback when no
@@ -408,15 +394,14 @@ impl CodecParser for DtsParser {
// regardless of buffer state, rather than inheriting this
// (non-core) PES's timestamp.
self.pending_pts = PTS_UNSET;
self.pts_marks.clear();
}
}
// Discard markers that no longer reference live buffer bytes (everything
// past the buffer end can't happen, but collapse duplicates at offset 0
// and drop a stale empty-buffer marker set).
if self.buf.is_empty() {
self.pts_marks.clear();
// An empty buffer holds no bytes for a mark to attribute, so drop the
// marks with them. `drain` deliberately keeps the mark covering the new
// front — correct while bytes remain, stale once none do.
if self.acc.is_empty() {
self.acc.clear();
}
frames
@@ -440,26 +425,28 @@ impl DtsParser {
/// streaming), gated through the decodability check. Require a complete core
/// frame; drop a bare partial sync tail.
fn flush_tail(&mut self) -> Vec<Frame> {
if find_sync(&self.buf, &DTS_CORE_SYNC) != Some(0) || self.buf.len() < CORE_HEADER_MIN_BYTES
if find_sync(self.acc.as_slice(), &DTS_CORE_SYNC) != Some(0)
|| self.acc.len() < CORE_HEADER_MIN_BYTES
{
self.buf.clear();
self.acc.clear();
return Vec::new();
}
let core_size = dts_core_frame_size(&self.buf);
let core_size = dts_core_frame_size(self.acc.as_slice());
// `dts_core_frame_size` returns a 14-bit `fsize + 1` (never 0), so the
// old `== 0` check was dead; reject a sub-minimum core like `parse()`.
if core_size < MIN_CORE_FRAME_BYTES || self.buf.len() < core_size {
self.buf.clear();
if core_size < MIN_CORE_FRAME_BYTES || self.acc.len() < core_size {
self.acc.clear();
return Vec::new();
}
// The final AU's PTS is the PES covering the buffer front (its core's
// PES). Fall back to pending_pts, clamping the sentinel to 0.
let au = std::mem::take(&mut self.buf);
let au = self.acc.as_slice().to_vec();
let dur_ns = dts_core_duration_ns(&au) as i64;
let pts_ns = self.stamp_pts(self.front_pts(), dur_ns);
self.pts_marks.clear();
let src = self.front_source();
self.acc.clear();
let mut out = Vec::new();
self.emit_or_drop(au, pts_ns, dur_ns, &mut out);
self.emit_or_drop(au, pts_ns, dur_ns, src, &mut out);
out
}
}
@@ -824,6 +811,7 @@ fn core_header_drop_reason(au: &[u8]) -> Option<DropReason> {
#[cfg(test)]
mod tests {
use super::*;
use crate::mux::codec::pts_to_ns;
use crate::mux::ts::PesPacket;
fn make_pes(data: Vec<u8>, pts: Option<i64>) -> PesPacket {
@@ -899,15 +887,19 @@ mod tests {
fn drain_front_collapses_offset_zero_markers_instead_of_leaking() {
let mut parser = DtsParser::new();
for i in 0..200i64 {
parser.buf.extend_from_slice(&[0u8; 5]);
parser.pts_marks.push_back((5, i));
parser.pts_marks.push_back((5, i));
parser.acc.append_unattributed(&[0u8; 5]);
parser
.acc
.mark_here(crate::mux::codec::pesbuf::PesFacts::default().with_pts_ns(i));
parser
.acc
.mark_here(crate::mux::codec::pesbuf::PesFacts::default().with_pts_ns(i));
parser.drain_front(5);
}
assert!(
parser.pts_marks.len() <= 2,
parser.acc.mark_count() <= 2,
"pts_marks must stay bounded across repeated drains, got {}",
parser.pts_marks.len()
parser.acc.mark_count()
);
}
@@ -1579,7 +1571,7 @@ mod tests {
"NeedMore past the AU cap must force-emit, not stall and balloon the buffer"
);
assert!(
parser.buf.is_empty(),
parser.acc.is_empty(),
"the forced flush drains the buffer instead of growing it unbounded"
);
}
@@ -1751,7 +1743,7 @@ mod tests {
// The first core's bytes are still buffered awaiting the verdict — not
// dropped, not emitted.
assert!(
parser.buf.len() >= 512,
parser.acc.len() >= 512,
"core1 retained while candidate boundary is undecided"
);
}
@@ -1809,8 +1801,8 @@ mod tests {
let mut parser = DtsParser::new();
let f = parser.parse(&make_pes(vec![0x11, 0x22, 0x33, 0x44], Some(90000)));
assert!(f.is_empty());
assert_eq!(parser.buf.len(), 3, "only a 3-byte resync tail retained");
assert_eq!(parser.buf, vec![0x22, 0x33, 0x44]);
assert_eq!(parser.acc.len(), 3, "only a 3-byte resync tail retained");
assert_eq!(parser.acc.as_slice(), &[0x22, 0x33, 0x44]);
}
#[test]
@@ -1825,7 +1817,7 @@ mod tests {
.parse(&make_pes(core[..3].to_vec(), Some(90000)))
.is_empty()
);
assert_eq!(parser.buf.len(), 3, "3-byte sync prefix retained");
assert_eq!(parser.acc.len(), 3, "3-byte sync prefix retained");
// PES 2: the 4th sync byte + the rest of core1, then a 2nd core to close.
let mut rest = core[3..].to_vec();
rest.extend_from_slice(&make_dts_core(640));
@@ -1847,7 +1839,7 @@ mod tests {
let mut data = DTS_CORE_SYNC.to_vec();
data.extend_from_slice(&[0x00, 0x00, 0x00]); // only 7 bytes total < 10
assert!(parser.parse(&make_pes(data, Some(90000))).is_empty());
assert!(!parser.buf.is_empty(), "partial core header retained");
assert!(!parser.acc.is_empty(), "partial core header retained");
}
#[test]
@@ -1859,7 +1851,7 @@ mod tests {
let mut d = vec![0u8; 17];
d[0..4].copy_from_slice(&DTS_CORE_SYNC);
d[6] = 0x01; // fsize → 16 → size 17
parser.buf = d;
parser.acc.seed(&d);
assert!(parser.flush().is_empty(), "sub-spec core rejected at flush");
}
@@ -1869,7 +1861,7 @@ mod tests {
// declared size must be dropped (never emit fewer bytes than declared).
let mut parser = DtsParser::new();
let core = make_dts_core(512);
parser.buf = core[..300].to_vec(); // header says 512, only 300 present
parser.acc.seed(&core[..300]); // header says 512, only 300 present
assert!(
parser.flush().is_empty(),
"incomplete core not emitted truncated"
@@ -1886,9 +1878,9 @@ mod tests {
fn flush_partial_sync_tail_dropped() {
// A bare partial-sync tail (not at offset 0 / not a full core) is dropped.
let mut parser = DtsParser::new();
parser.buf = vec![0x7F, 0xFE, 0x80]; // 3 of 4 sync bytes
parser.acc.seed(&[0x7F, 0xFE, 0x80]); // 3 of 4 sync bytes
assert!(parser.flush().is_empty());
assert!(parser.buf.is_empty(), "buffer cleared on flush");
assert!(parser.acc.is_empty(), "buffer cleared on flush");
}
#[test]
@@ -2311,13 +2303,13 @@ mod tests {
let out = parser.parse(&make_pes(d, Some(90_000)));
assert!(out.is_empty(), "a false sync emits nothing");
assert_eq!(
parser.buf.len(),
parser.acc.len(),
3,
"the false sync was decoded, drained and resynced past — leaving only \
the 3-byte split-sync carry-over"
);
assert_ne!(
find_sync(&parser.buf, &DTS_CORE_SYNC),
find_sync(parser.acc.as_slice(), &DTS_CORE_SYNC),
Some(0),
"and the bogus sync is no longer at the front of the buffer"
);
@@ -2331,14 +2323,14 @@ mod tests {
fn flush_emits_a_core_that_exactly_fills_the_buffer() {
let mut parser = DtsParser::new();
let core = make_dts_core(512);
parser.buf = core.clone();
parser.acc.seed(&core.clone());
parser.pending_pts = 90_000;
let out = parser.flush();
assert_eq!(out.len(), 1, "the final AU is emitted, not dropped");
assert_eq!(out[0].data, core, "and it is the whole core frame");
// One byte short is still refused — the bound is not simply absent.
let mut parser = DtsParser::new();
parser.buf = core[..511].to_vec();
parser.acc.seed(&core[..511]);
parser.pending_pts = 90_000;
assert!(
parser.flush().is_empty(),
@@ -2368,18 +2360,18 @@ mod tests {
Some(0),
"the fixture really has no core sync at the front"
);
parser.buf = broken;
parser.acc.seed(&broken);
parser.pending_pts = 90_000;
assert!(
parser.flush().is_empty(),
"a buffer whose front is not a core sync is discarded, not size-decoded"
);
assert!(parser.buf.is_empty(), "and the junk is dropped");
assert!(parser.acc.is_empty(), "and the junk is dropped");
// The other half of the disjunction: a buffer too short to size, whose
// front IS a core sync, is discarded too.
let mut parser = DtsParser::new();
parser.buf = DTS_CORE_SYNC.to_vec();
parser.acc.seed(DTS_CORE_SYNC.as_ref());
parser.pending_pts = 90_000;
assert!(parser.flush().is_empty(), "a bare sync tail is not an AU");
}
@@ -2416,4 +2408,54 @@ mod tests {
"the bare sync sizes nothing"
);
}
/// The whole point of the shared buffer: an access unit whose core arrived
/// in an EARLIER packet keeps that packet's source offset, not the offset
/// of whichever packet completed it. At a clip boundary the two belong to
/// different clips, and taking the later one puts the unit in the wrong
/// clip -- which is what left nine audio and subtitle tracks unplaceable.
#[test]
fn an_access_unit_carries_the_source_of_the_packet_its_core_arrived_in() {
let mut parser = DtsParser::new();
let core = make_dts_core(512);
// The core starts here, at byte 1000 of the feed.
let mut p1 = make_pes(core[..256].to_vec(), Some(90_000));
p1.source = Some(crate::pes::SourcePos::at_byte(1_000));
assert!(parser.parse(&p1).is_empty(), "partial core held");
// The rest arrives later, at byte 9_000, together with the next core
// that closes the unit.
let mut rest = core[256..].to_vec();
rest.extend_from_slice(&make_dts_core(512));
let mut p2 = make_pes(rest, Some(180_000));
p2.source = Some(crate::pes::SourcePos::at_byte(9_000));
let frames = parser.parse(&p2);
assert!(!frames.is_empty(), "the completed unit is emitted");
assert_eq!(
frames[0].source.map(|s| s.byte),
Some(1_000),
"the unit belongs to the packet its FIRST byte came from"
);
assert_eq!(
frames[0].pts_ns,
pts_to_ns(90_000),
"and its timestamp comes from that same packet"
);
}
/// A unit that begins and ends in one packet takes that packet's offset —
/// the ordinary case, which must not regress while fixing the spanning one.
#[test]
fn a_self_contained_access_unit_carries_its_own_packets_source() {
let mut parser = DtsParser::new();
let mut data = make_dts_core(512);
data.extend_from_slice(&make_dts_core(512));
let mut p = make_pes(data, Some(90_000));
p.source = Some(crate::pes::SourcePos::at_byte(4_242));
let frames = parser.parse(&p);
assert!(!frames.is_empty());
assert_eq!(frames[0].source.map(|s| s.byte), Some(4_242));
}
}
+99 -18
View File
@@ -25,7 +25,10 @@ pub struct DvdSubParser {
/// Pre-formatted VobSub .idx palette header for codec_private.
codec_data: Option<Vec<u8>>,
/// In-progress SPU reassembly: (head PTS in ns, declared SPU_size, bytes).
pending: Option<(i64, usize, Vec<u8>)>,
/// The SPU being accumulated, with the facts of the PES that STARTED it.
/// An SPU spans PES packets, so its timestamp and source offset are the
/// opening packet's — the same rule every other buffering parser applies.
pending: Option<(super::pesbuf::PesFacts, usize, Vec<u8>)>,
}
impl DvdSubParser {
@@ -41,11 +44,12 @@ impl DvdSubParser {
fn take_if_complete(&mut self, force: bool) -> Option<Frame> {
let (_, size, buf) = self.pending.as_ref()?;
if force || buf.len() >= *size {
let (pts_ns, _, data) = self.pending.take().unwrap();
let (facts, _, data) = self.pending.take().unwrap();
let pts_ns = facts.presentation_ns().unwrap_or(0);
return Some(Frame {
discontinuity: false,
coding: None,
source: None,
source: facts.source,
pts_ns,
keyframe: true,
data,
@@ -106,7 +110,7 @@ impl CodecParser for DvdSubParser {
out.push(Frame {
discontinuity: false,
coding: None,
source: None,
source: super::pesbuf::PesFacts::of(pes).source,
pts_ns,
keyframe: true,
data: pes.data.clone(),
@@ -120,7 +124,7 @@ impl CodecParser for DvdSubParser {
out.push(Frame {
discontinuity: false,
coding: None,
source: None,
source: super::pesbuf::PesFacts::of(pes).source,
pts_ns,
keyframe: true,
data: pes.data.clone(),
@@ -133,7 +137,7 @@ impl CodecParser for DvdSubParser {
if buf.len() > MAX_SPU_BYTES {
buf.truncate(MAX_SPU_BYTES);
}
self.pending = Some((pts_ns, declared, buf));
self.pending = Some((super::pesbuf::PesFacts::of(pes), declared, buf));
if let Some(frame) = self.take_if_complete(false) {
out.push(frame);
}
@@ -155,7 +159,13 @@ impl CodecParser for DvdSubParser {
/// Convert a single YCbCr color to RGB, clamping to [0, 255].
///
/// Input: `[padding, Y, Cb, Cr]` (as stored in DVD IFO PGC data).
/// Input: `[padding, Y, Cr, Cb]` (as stored in DVD IFO PGC data). Note the
/// chroma order: the on-disc PGC CLUT is **Cr before Cb** — byte 2 is Cr and
/// byte 3 is Cb. Reading byte 2 as Cb swaps red and blue on every chromatic
/// entry, and is invisible on the achromatic (white/black/grey, Cb = Cr = 128)
/// entries that dominate real palettes, which is how it survives casual
/// inspection. The order is fixed by the DVD-Video PGC format, not by us.
///
/// Returns `[R, G, B]`.
///
/// Range convention (deliberate): this uses the **full-range (JFIF) BT.601**
@@ -171,8 +181,8 @@ impl CodecParser for DvdSubParser {
/// side in lockstep.
pub fn ycbcr_to_rgb(color: &[u8; 4]) -> [u8; 3] {
let y = color[1] as f64;
let cb = color[2] as f64;
let cr = color[3] as f64;
let cr = color[2] as f64;
let cb = color[3] as f64;
let r = y + 1.402 * (cr - 128.0);
let g = y - 0.344 * (cb - 128.0) - 0.714 * (cr - 128.0);
@@ -194,7 +204,7 @@ fn clamp_u8(v: f64) -> u8 {
/// Format a 16-color YCbCr palette as a VobSub `.idx` header for S_VOBSUB
/// CodecPrivate.
///
/// Each entry is `[padding, Y, Cb, Cr]`. Output is a UTF-8 text block carrying
/// Each entry is `[padding, Y, Cr, Cb]`. Output is a UTF-8 text block carrying
/// the two `.idx` header lines mkvmerge / libvobsub expect:
///
/// ```text
@@ -428,24 +438,26 @@ mod tests {
#[test]
fn ycbcr_to_rgb_clamps_overflow() {
// Y=255, Cr=255 → R would be 255 + 1.402*127 = ~433, should clamp to 255
let color = [0x00, 255, 128, 255];
// Y=255, Cr=255 → R would be 255 + 1.402*127 = ~433, should clamp to 255.
// Cr is byte 2 in the on-disc [pad, Y, Cr, Cb] layout.
let color = [0x00, 255, 255, 128];
let [r, _g, _b] = ycbcr_to_rgb(&color);
assert_eq!(r, 255);
}
#[test]
fn ycbcr_to_rgb_clamps_underflow() {
// Y=0, Cr=0 → R = 0 + 1.402*(0-128) = -179, should clamp to 0
let color = [0x00, 0, 128, 0];
// Y=0, Cr=0 → R = 0 + 1.402*(0-128) = -179, should clamp to 0.
// Cr is byte 2 in the on-disc [pad, Y, Cr, Cb] layout.
let color = [0x00, 0, 0, 128];
let [r, _g, _b] = ycbcr_to_rgb(&color);
assert_eq!(r, 0);
}
#[test]
fn ycbcr_to_rgb_red() {
// Approximate red: Y=82, Cb=90, Cr=240
let color = [0x00, 82, 90, 240];
// Approximate red: Y=82, Cr=240, Cb=90 — on disc as [pad, Y, Cr, Cb].
let color = [0x00, 82, 240, 90];
let [r, g, b] = ycbcr_to_rgb(&color);
// R = 82 + 1.402*(240-128) = 82 + 156.9 ≈ 239
// G = 82 - 0.344*(90-128) - 0.714*(240-128) = 82 + 13.1 - 79.97 ≈ 15
@@ -455,6 +467,46 @@ mod tests {
assert!(b < 30, "B should be low for red, got {}", b);
}
/// On-disc DVD PGC CLUT byte order is `[0, Y, Cr, Cb]` — byte 2 is **Cr**
/// and byte 3 is **Cb**, per the DVD-Video PGC format.
///
/// This fixture uses a real on-disc red entry, so it fails if the two
/// chroma bytes are ever exchanged again. It is deliberately NOT built
/// from this crate's own doc comments: those described the order wrongly
/// for a long time, and the previous version of this test inherited the
/// error from them and therefore could not detect it.
///
/// A saturated RED entry therefore appears on disc as Y=76, Cr=255, Cb=85
/// (full-range BT.601 encoding of RGB #FF0000), i.e. bytes
/// `[0x00, 76, 255, 85]`. Reading byte 2 as Cb and byte 3 as Cr instead
/// turns this entry BLUE, which is the exact user-visible symptom.
///
/// The pre-existing `_white` / `_black` tests cannot catch this: they use
/// Cb = Cr = 128, so exchanging two equal bytes is a literal no-op.
#[test]
fn ycbcr_to_rgb_reads_byte2_as_cr_and_byte3_as_cb() {
// On-disc [pad, Y, Cr, Cb] for saturated red.
let on_disc_red = [0x00u8, 76, 255, 85];
let [r, g, b] = ycbcr_to_rgb(&on_disc_red);
assert!(
r > 200 && b < 60,
"on-disc red [0,Y=76,Cr=255,Cb=85] must render red-dominant, \
got R={r} G={g} B={b} (R and B swapped => byte 2/3 are transposed)"
);
assert_eq!([r, g, b], [254, 0, 0], "exact full-range BT.601 red");
// And the converse: a saturated BLUE on-disc entry (Y=29, Cr=107, Cb=255)
// must not come out red.
let on_disc_blue = [0x00u8, 29, 107, 255];
let [r2, g2, b2] = ycbcr_to_rgb(&on_disc_blue);
assert!(
b2 > 200 && r2 < 60,
"on-disc blue [0,Y=29,Cr=107,Cb=255] must render blue-dominant, \
got R={r2} G={g2} B={b2}"
);
}
// ── Palette formatting tests ──────────────────────────────────────────
#[test]
@@ -671,7 +723,17 @@ mod tests {
let f = parser.parse(&make_pes(vec![0x00, 0x10, 0xAA], None));
assert!(f.is_empty(), "incomplete sized segment held, not emitted");
assert!(parser.pending.is_some(), "started a new pending SPU");
assert_eq!(parser.pending.as_ref().unwrap().0, 0, "pts 0 (no PTS)");
assert_eq!(
parser
.pending
.as_ref()
.unwrap()
.0
.presentation_ns()
.unwrap_or(0),
0,
"pts 0 (no PTS)"
);
}
#[test]
@@ -685,7 +747,8 @@ mod tests {
#[test]
fn ycbcr_blue_channel_clamps_high() {
// B = Y + 1.772*(Cb-128). Y=128, Cb=255 → 128 + 1.772*127 ≈ 353 → clamp 255.
let [_r, _g, b] = ycbcr_to_rgb(&[0x00, 128, 255, 128]);
// Cb is byte 3 in the on-disc [pad, Y, Cr, Cb] layout.
let [_r, _g, b] = ycbcr_to_rgb(&[0x00, 128, 128, 255]);
assert_eq!(b, 255, "blue clamps at 255");
}
@@ -703,4 +766,22 @@ mod tests {
let result = format_palette(&[[0x00, 16, 128, 128]], 0, 0);
assert_eq!(String::from_utf8(result).unwrap(), "palette: 101010\n");
}
/// The text guard in `codec/mod.rs` scans for a literal `source: None` and
/// cannot see a parser that writes `source: facts.source` where the facts
/// carry no offset. Only a runtime check proves an emitted frame really
/// carries the byte it was read from, and without it a multi-clip title
/// places this track by timestamp inference instead of by byte.
#[test]
fn an_emitted_frame_carries_the_packets_source_offset() {
let mut parser = DvdSubParser::new(None);
let mut pes = make_pes(
vec![0x00, 0x0A, 0x00, 0x08, 0x01, 0xFF, 0x02, 0x03, 0x04, 0x05],
Some(90_000),
);
pes.source = Some(crate::pes::SourcePos::at_byte(7_777));
let frames = parser.parse(&pes);
assert!(!frames.is_empty(), "the segment is emitted");
assert_eq!(frames[0].source.map(|s| s.byte), Some(7_777));
}
}
+16 -1
View File
@@ -118,7 +118,7 @@ impl CodecParser for FlacParser {
vec![Frame {
discontinuity: pes.discontinuity,
coding: None,
source: None,
source: super::pesbuf::PesFacts::of(pes).source,
pts_ns,
keyframe: true,
data: pes.data.clone(),
@@ -271,4 +271,19 @@ mod tests {
);
assert_eq!(emitted.len() + tail.len(), 2);
}
/// The text guard in `codec/mod.rs` scans for a literal `source: None` and
/// cannot see a parser that writes `source: facts.source` where the facts
/// carry no offset. Only a runtime check proves an emitted frame really
/// carries the byte it was read from, and without it a multi-clip title
/// places this track by timestamp inference instead of by byte.
#[test]
fn an_emitted_frame_carries_the_packets_source_offset() {
let mut p = FlacParser::new();
let mut pes = make_pes(make_flac_frame(100), Some(90_000));
pes.source = Some(crate::pes::SourcePos::at_byte(7_777));
let f = p.parse(&pes);
assert!(!f.is_empty(), "the frame is emitted");
assert_eq!(f[0].source.map(|s| s.byte), Some(7_777));
}
}
+73 -1
View File
@@ -1,7 +1,8 @@
//! H.264 (AVC) elementary stream parser.
//!
//! Extracts SPS and PPS NAL units for MKV codecPrivate.
//! Detects keyframes (IDR slices).
//! Detects keyframes: IDR slices, and open-GOP intra-coded access units (the
//! non-IDR recovery points BD titles use — see the intra promotion in `parse`).
//! Each PES packet = one access unit = one frame.
use super::coding::{CodingType, PictureInfo};
@@ -351,6 +352,26 @@ impl CodecParser for H264Parser {
return Vec::new();
}
// Open-GOP resync anchor: an intra-coded access unit is a keyframe even
// when it is NOT an IDR. BD H.264 titles use open GOPs whose only
// random-access point is a non-IDR I-frame (the recovery point, which an
// encoder MAY additionally tag with a recovery_point SEI). Keying off the
// already-measured picture coding type covers every signalling — a
// recovery_point SEI, or a disc that marks the recovery point only by GOP
// structure with no SEI at all — where SEI parsing alone would miss the
// latter. Without this, the B1 resync gate (`mux/resync.rs`), once armed
// by a concealed-gap / clip-seam discontinuity, never sees a keyframe on
// such a tail and drops every frame to EOF, truncating the video. HEVC
// does not hit this: its open-GOP IRAP (CRA) is a distinct NAL type
// already flagged a keyframe (`mux/codec/hevc.rs`); H.264 has no such NAL
// tag, so the intra coding type is the equivalent signal. Consistent with
// `videomap.rs`'s model that a keyframe IS the intra decode-restart point.
// IDR is still flagged by NAL type in the loop above, so a slice whose
// header failed to parse (`coding_type == None`) is unaffected here.
if coding_type == Some(CodingType::I) {
keyframe = true;
}
// Every keyframe is self-contained: re-assert the active SPS/PPS in-band
// ahead of the slices (even when unchanged vs codecPrivate) so a decoder
// that dropped the set at a reset recovers, and a stale avcC re-apply
@@ -694,6 +715,57 @@ mod tests {
}
}
/// Open-GOP resync fix: a non-IDR intra-coded access unit MUST be flagged a
/// keyframe so the B1 resync gate (`mux/resync.rs`) can disarm on a BD open-
/// GOP tail instead of dropping every frame to EOF and truncating the video.
/// BD H.264 titles reach random access at a non-IDR I-frame (the recovery
/// point); HEVC gets this free from the CRA NAL type, H.264 does not, so the
/// measured intra coding type is the signal. Red before the intra promotion
/// in `parse` — a non-IDR I-slice used to yield `keyframe == false`.
#[test]
fn open_gop_intra_access_unit_is_a_keyframe() {
// Annex B AU: one non-IDR coded slice (NAL type 1, nal_ref_idc 3 → 0x61)
// whose header decodes first_mb_in_slice = 0 (ue = "1") and slice_type = 7
// (ue = "0001000"), i.e. an all-I slice. The two ue(v) fields pack into
// one byte: "1" + "0001000" = 0b1000_1000 = 0x88.
let au = vec![0x00, 0x00, 0x01, 0x61, 0x88, 0x00];
let mut parser = H264Parser::new();
let frames = parser.parse(&make_pes(au, Some(0)));
assert_eq!(frames.len(), 1, "one PES access unit → one frame");
assert_eq!(
frames[0].coding.map(|c| c.coding_type()),
Some(CodingType::I),
"the slice header measures an I picture",
);
assert!(
frames[0].keyframe,
"an open-GOP intra access unit is a resync-safe keyframe even without an IDR",
);
}
/// Guard the promotion's precision: a non-intra (P) access unit must NOT be
/// promoted to a keyframe, or the resync gate would resume on a frame that
/// carries inter-references to dropped data and the MKV cue index would gain
/// bogus random-access points.
#[test]
fn inter_coded_access_unit_is_not_a_keyframe() {
// Same NAL type 1, but slice_type = 0 (P): ue "1", so with first_mb = 0
// the byte is "1" + "1" padded = 0b1100_0000 = 0xC0.
let au = vec![0x00, 0x00, 0x01, 0x61, 0xC0, 0x00];
let mut parser = H264Parser::new();
let frames = parser.parse(&make_pes(au, Some(0)));
assert_eq!(frames.len(), 1);
assert_eq!(
frames[0].coding.map(|c| c.coding_type()),
Some(CodingType::P),
"the slice header measures a P picture",
);
assert!(
!frames[0].keyframe,
"an inter-coded access unit must never be a keyframe",
);
}
/// The slice-header parse must remove emulation-prevention bytes first
/// (ISO/IEC 14496-10 §7.3.1/§7.4.1): the payload after the NAL header is
/// EBSP, and a 0x03 following 0x00 0x00 is not part of the RBSP.
+18 -1
View File
@@ -74,7 +74,7 @@ impl CodecParser for LpcmParser {
vec![Frame {
discontinuity: false,
coding: None,
source: None,
source: super::pesbuf::PesFacts::of(pes).source,
pts_ns,
keyframe: true,
data: pes.data[offset..].to_vec(),
@@ -279,4 +279,21 @@ mod tests {
let f = parser.parse(&make_pes(vec![0xAA, 0xBB], None));
assert_eq!(f[0].pts_ns, 0);
}
/// The text guard in `codec/mod.rs` scans for a literal `source: None` and
/// cannot see a parser that writes `source: facts.source` where the facts
/// carry no offset. Only a runtime check proves an emitted frame really
/// carries the byte it was read from, and without it a multi-clip title
/// places this track by timestamp inference instead of by byte.
#[test]
fn an_emitted_frame_carries_the_packets_source_offset() {
let mut parser = LpcmParser::new();
let mut data = vec![0x00, 0x01, 0x00, 0b1001_0001];
data.extend_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE]);
let mut pes = make_pes(data, Some(90_000));
pes.source = Some(crate::pes::SourcePos::at_byte(7_777));
let frames = parser.parse(&pes);
assert!(!frames.is_empty(), "the frame is emitted");
assert_eq!(frames[0].source.map(|s| s.byte), Some(7_777));
}
}
+170 -1
View File
@@ -34,6 +34,11 @@ pub mod mpeg2;
pub mod mpegaudio;
/// HDMV PGS (Presentation Graphics Stream) subtitle parser.
pub mod pgs;
/// One accumulation buffer for parsers that assemble access units across PES
/// packets, so a unit's timestamp and its source offset always come from the
/// packet that carried its first byte -- and from the SAME packet.
pub(crate) mod pesbuf;
/// Display-order PTS reconstruction for sparse-PTS program-stream video.
pub(crate) mod reorder;
/// Shared MPEG/Annex-B start-code scanning helpers.
@@ -149,7 +154,7 @@ impl CodecParser for PassthroughParser {
// parsers must instead defer the flag to the next emitted frame.)
vec![Frame {
coding: None,
source: None,
source: pesbuf::PesFacts::of(pes).source,
pts_ns,
keyframe: self.keyframe,
discontinuity: pes.discontinuity,
@@ -354,3 +359,167 @@ mod tests {
}
}
}
#[cfg(test)]
mod provenance_guard {
//! Every emitted frame must carry the source byte offset of the packet it
//! came from. That invariant held only for video for as long as it existed:
//! `dts`, `ac3`, `adts`, `truehd`, `pgs`, `dvdsub`, `flac`, `lpcm`,
//! `mpegaudio` and the passthrough parser all built frames with
//! `source: None`, so a multi-clip title could not place audio or subtitles
//! by byte and fell back to inferring the clip from timestamps — which is
//! what made branched titles run minutes long.
//!
//! Nothing asserted it, so nothing caught it. Finding it took a
//! brace-balanced scan of the tree by hand, which also turned up five sites
//! a regex had missed. This is that scan, as a test.
/// Every parser module's source, checked for a `Frame` built without a
/// source. Paired with `every_codec_module_is_covered` below, which fails
/// if a module is added and not listed here.
const PARSER_SOURCES: &[(&str, &str)] = &[
("mod.rs", include_str!("mod.rs")),
("ac3.rs", include_str!("ac3.rs")),
("adts.rs", include_str!("adts.rs")),
("dts.rs", include_str!("dts.rs")),
("dvdsub.rs", include_str!("dvdsub.rs")),
("flac.rs", include_str!("flac.rs")),
("h264.rs", include_str!("h264.rs")),
("hevc.rs", include_str!("hevc.rs")),
("lpcm.rs", include_str!("lpcm.rs")),
("mpeg2.rs", include_str!("mpeg2.rs")),
("mpegaudio.rs", include_str!("mpegaudio.rs")),
("pgs.rs", include_str!("pgs.rs")),
("truehd.rs", include_str!("truehd.rs")),
("vc1.rs", include_str!("vc1.rs")),
];
/// Modules that declare no `Frame` and so need no entry above.
const NON_PARSER_MODULES: &[&str] = &[
"coding",
"crc",
"dropgate",
"pesbuf",
"reorder",
"startcode",
];
/// Walk `Frame { .. }` literals with balanced braces. A regex cannot do
/// this — `Frame` blocks contain nested braces (`coding: Some(..)`, closures)
/// and a non-greedy match stops at the first `}`, which is how five sites
/// survived the first pass.
/// Strip line comments before scanning. A comment that MENTIONS
/// `source: None` is prose, not a construction — this guard's own doc
/// comment tripped it on the first run, which is the same mistake as
/// grepping a file and matching its commentary.
fn code_only(src: &str) -> String {
// The guard itself constructs no Frame; it only talks about them, in
// prose and in string literals. Scanning its own body matches both.
let src = match src.find("mod provenance_guard") {
Some(i) => &src[..i],
None => src,
};
src.lines()
.map(|l| match l.find("//") {
// Not inside a string literal: no quote before the slashes.
Some(i) if !l[..i].contains('"') => &l[..i],
_ => l,
})
.collect::<Vec<_>>()
.join("\n")
}
fn frame_literals(src: &str) -> Vec<&str> {
let mut out = Vec::new();
let b = src.as_bytes();
let mut i = 0;
while let Some(p) = src[i..].find("Frame {") {
let start = i + p;
let open = start + src[start..].find('{').unwrap();
let (mut depth, mut k) = (0usize, open);
while k < b.len() {
match b[k] {
b'{' => depth += 1,
b'}' => {
depth -= 1;
if depth == 0 {
break;
}
}
_ => {}
}
k += 1;
}
out.push(&src[start..(k + 1).min(src.len())]);
i = k + 1;
}
out
}
#[test]
fn no_parser_emits_a_frame_without_provenance() {
let mut offenders = Vec::new();
for (name, src) in PARSER_SOURCES {
let src = &code_only(src);
for blk in frame_literals(src) {
// `PesFrame` in the tests of other modules is not ours; only
// codec `Frame` literals are scanned, and a test fixture that
// builds a PesPacket with `source: None` is legitimate.
// Two spellings, not one. `source: None` is the obvious way to
// lose provenance; OMITTING the field entirely is the quiet
// one, because `Frame` derives Default, so
// `Frame { pts_ns, .. Default::default() }` compiles and
// yields `source: None` while containing no such text. A guard
// that only knew the first spelling would have watched a
// parser be rewritten into the second and stayed green.
let explicit_none = blk.contains("source: None");
let no_source_field = !blk.contains("source:");
if explicit_none || no_source_field {
let line = src[..src.find(blk).unwrap_or(0)].lines().count() + 1;
let how = if explicit_none {
"source: None"
} else {
"no source field (Default fills in None)"
};
offenders.push(format!("{name}:{line} ({how})"));
}
}
}
assert!(
offenders.is_empty(),
"these parsers build a Frame with no source byte offset, so a \
multi-clip title cannot place their tracks by provenance and \
falls back to inferring the clip from timestamps: {offenders:?}"
);
}
/// A parser added without an entry in `PARSER_SOURCES` would be silently
/// unchecked, which is exactly how this class of gap persists.
#[test]
fn every_codec_module_is_covered() {
let modules: Vec<&str> = include_str!("mod.rs")
.lines()
.filter_map(|l| {
let l = l.trim();
let rest = l
.strip_prefix("pub mod ")
.or_else(|| l.strip_prefix("pub(crate) mod "))?;
rest.strip_suffix(';')
})
.collect();
let listed: Vec<&str> = PARSER_SOURCES
.iter()
.map(|(n, _)| n.trim_end_matches(".rs"))
.collect();
let unchecked: Vec<&&str> = modules
.iter()
.filter(|m| !listed.contains(m) && !NON_PARSER_MODULES.contains(m))
.collect();
assert!(
unchecked.is_empty(),
"codec module(s) not covered by the provenance guard — add them to \
PARSER_SOURCES (or to NON_PARSER_MODULES if they emit no Frame): \
{unchecked:?}"
);
}
}
+16 -1
View File
@@ -116,7 +116,7 @@ impl CodecParser for MpegAudioParser {
vec![Frame {
discontinuity: pes.discontinuity,
coding: None,
source: None,
source: super::pesbuf::PesFacts::of(pes).source,
pts_ns,
keyframe: true,
data: pes.data.clone(),
@@ -319,4 +319,19 @@ mod tests {
// a manufactured tail frame would break this even if it were non-empty.
assert_eq!(emitted.len() + tail.len(), 2);
}
/// The text guard in `codec/mod.rs` scans for a literal `source: None` and
/// cannot see a parser that writes `source: facts.source` where the facts
/// carry no offset. Only a runtime check proves an emitted frame really
/// carries the byte it was read from, and without it a multi-clip title
/// places this track by timestamp inference instead of by byte.
#[test]
fn an_emitted_frame_carries_the_packets_source_offset() {
let mut p = MpegAudioParser::new();
let mut pes = make_pes(mp3_frame(400), Some(90_000));
pes.source = Some(crate::pes::SourcePos::at_byte(7_777));
let f = p.parse(&pes);
assert!(!f.is_empty(), "the frame is emitted");
assert_eq!(f[0].source.map(|s| s.byte), Some(7_777));
}
}
+362
View File
@@ -0,0 +1,362 @@
//! One accumulation buffer for parsers that assemble access units across PES
//! packets.
//!
//! A buffering parser has to answer the same question for every unit it emits:
//! *which PES contributed this unit's FIRST byte?* Its timestamp comes from
//! that PES, and so does the source byte offset that identifies which clip of a
//! multi-clip title the unit belongs to. The trailing PES packets that complete
//! the unit carry their own, later, values which must not override it.
//!
//! That question was answered three different ways. DTS kept a deque of
//! `(offset, pts)` markers and took the one covering offset 0 — correct. AC-3
//! kept a single carry-over timestamp. TrueHD kept its own. None of them
//! carried the source offset at all, so provenance existed only for video, and
//! a title whose clip marks could not be read from timestamps alone had nine
//! audio and subtitle tracks with nothing to place them by.
//!
//! Three spellings of one rule is how they drifted, so this is the one place it
//! lives. The buffer owns the bytes AND the marks, and returns a PES's facts
//! together — a parser cannot take the timestamp from one PES and the source
//! from another, because it does not assemble them itself.
use super::PesPacket;
use super::pts_to_ns;
use crate::pes::SourcePos;
/// What a PES contributes to the bytes it carried.
///
/// Returned as a unit so a caller cannot mix fields from different packets.
/// The timestamps are carried RAW, as the packet had them. This type answers
/// *which packet* a unit's facts come from — the question that was being
/// answered three different ways. How a given codec derives a timestamp from
/// that packet stays the codec's business: DVD subtitles read `pts` only and
/// fall back to 0, most audio takes `pts.or(dts)`. Deriving it here would have
/// changed those semantics silently while fixing provenance.
#[derive(Debug, Clone, Copy, Default, PartialEq)]
pub(crate) struct PesFacts {
/// Presentation time in NANOSECONDS, already derived (see `of`). Stored
/// derived rather than raw so there is one derivation and no caller can
/// pick a different one.
pub pts_ns: Option<i64>,
/// Byte offset of this PES's first ES byte within the title's feed — what
/// identifies the clip a frame came from. `None` when the demuxer was fed
/// without a base offset.
pub source: Option<SourcePos>,
/// Packets for this stream were lost before this PES.
pub discontinuity: bool,
}
impl PesFacts {
/// The facts a PES packet carries, read straight off it.
///
/// Every parser reads its frame's timestamp, source and discontinuity from
/// a `PesFacts` — never off a `PesPacket` field directly — so the three can
/// never be taken from different packets. Parsers that assemble units
/// across packets get theirs from [`PesBuf::front`]; this is the same value
/// for a parser whose unit begins in the packet it is handed.
pub(crate) fn of(pes: &PesPacket) -> Self {
Self {
pts_ns: pes.pts.or(pes.dts).map(pts_to_ns),
source: pes.source,
discontinuity: pes.discontinuity,
}
}
/// The same facts with the presentation time replaced by one the parser
/// resolved itself — for a packet that carried no timestamp and whose unit
/// continues a base established earlier. The attribution is unchanged:
/// still this packet's bytes, still its source offset.
pub(crate) fn with_pts_ns(self, pts_ns: i64) -> Self {
Self {
pts_ns: Some(pts_ns),
..self
}
}
/// This unit's presentation time in nanoseconds — the ONE derivation.
///
/// PTS and DTS are not two spellings of one value: PTS is when to display,
/// DTS is when to decode, and for a stream that REORDERS (video carrying
/// B-frames) they differ, so reading DTS as a presentation time would be
/// wrong. Reordering is handled by the video path, which reconstructs
/// display order rather than calling this.
///
/// For everything that reaches here — audio and subtitles — there is no
/// reordering, so DTS *is* the presentation time, and falling back to it is
/// reading the same value from whichever field the packet used. A fallback
/// for a missing field, not a second rule: dvdsub read `pts` alone and
/// returned 0 for a packet that carried only DTS.
pub(crate) fn presentation_ns(&self) -> Option<i64> {
self.pts_ns
}
}
/// The facts of the packet covering `off` within a [`PesBuf::marks_snapshot`].
///
/// The same at-or-before rule as [`PesBuf::facts_at`], for a scanner holding a
/// snapshot rather than the buffer.
pub(crate) fn facts_for(marks: &[(usize, PesFacts)], off: usize) -> PesFacts {
let mut found = PesFacts::default();
for &(at, facts) in marks {
if at > off {
break;
}
found = facts;
}
found
}
/// Bytes accumulated across PES packets, each byte attributable to the packet
/// that carried it.
pub(crate) struct PesBuf {
buf: Vec<u8>,
/// `(offset of this PES's first byte within `buf`, its facts)`, ascending.
/// Offsets are relative to the current front and are rebased on `drain`.
marks: std::collections::VecDeque<(usize, PesFacts)>,
}
impl PesBuf {
pub(crate) fn with_capacity(n: usize) -> Self {
Self {
buf: Vec::with_capacity(n),
marks: std::collections::VecDeque::new(),
}
}
/// Append a PES's payload, recording where its bytes begin.
///
/// A PES carrying no payload records nothing: it contributed no byte, so it
/// can never be the answer to "which packet carried the byte at offset N",
/// and admitting a zero-length mark would let it shadow the packet that
/// actually did.
pub(crate) fn push(&mut self, pes: &PesPacket) {
if pes.data.is_empty() {
return;
}
self.marks.push_back((self.buf.len(), PesFacts::of(pes)));
self.buf.extend_from_slice(&pes.data);
}
/// Append a payload under facts the caller resolved — for a parser that
/// carries a timestamp forward across a packet that omitted one. Same
/// attribution rule; only the timestamp differs from what the packet said.
pub(crate) fn push_with(&mut self, data: &[u8], facts: PesFacts) {
if data.is_empty() {
return;
}
self.marks.push_back((self.buf.len(), facts));
self.buf.extend_from_slice(data);
}
/// The facts of the PES that carried the byte at `off`.
///
/// The last mark at or before `off`: bytes belong to the most recent packet
/// that started at or before them. Defaults when the buffer holds bytes
/// that predate any mark (a parser that seeded it directly).
pub(crate) fn facts_at(&self, off: usize) -> PesFacts {
let mut found = PesFacts::default();
for &(at, facts) in &self.marks {
if at > off {
break;
}
found = facts;
}
found
}
/// The facts of the PES that carried the byte at the FRONT of the buffer —
/// the first byte of the access unit a parser is about to emit, which is
/// the whole point of this type.
pub(crate) fn front(&self) -> PesFacts {
self.facts_at(0)
}
pub(crate) fn as_slice(&self) -> &[u8] {
&self.buf
}
pub(crate) fn len(&self) -> usize {
self.buf.len()
}
pub(crate) fn is_empty(&self) -> bool {
self.buf.is_empty()
}
/// Consume `n` bytes from the front, rebasing the marks onto the new front.
///
/// The mark covering the new front byte is RETAINED at offset 0 even though
/// its packet started earlier: those bytes are still that packet's. Dropping
/// it would attribute the remainder of a straddling unit to whichever packet
/// happened to start next — precisely the misattribution this type exists to
/// prevent, and it would land at a clip boundary, the one place it matters.
pub(crate) fn drain(&mut self, n: usize) {
let n = n.min(self.buf.len());
if n == 0 {
return;
}
self.buf.drain(..n);
let covering = self.facts_at(n);
self.marks.retain(|&(at, _)| at > n);
for m in &mut self.marks {
m.0 -= n;
}
if self.marks.front().map(|&(at, _)| at) != Some(0) {
self.marks.push_front((0, covering));
}
}
/// Seed the buffer directly with bytes carrying no packet attribution —
/// for tests that drive a parser's scanner without a demuxer in front of
/// it. Facts for these bytes default to absent, which is what an
/// unattributed byte honestly is.
#[cfg(test)]
pub(crate) fn seed(&mut self, data: &[u8]) {
self.buf.clear();
self.marks.clear();
self.buf.extend_from_slice(data);
}
/// Append unattributed bytes, keeping existing content and marks.
#[cfg(test)]
pub(crate) fn append_unattributed(&mut self, data: &[u8]) {
self.buf.extend_from_slice(data);
}
/// How many packet marks are held — a test hook for the bound on marks.
#[cfg(test)]
pub(crate) fn mark_count(&self) -> usize {
self.marks.len()
}
/// Record a mark at the current end without appending bytes — a test hook
/// for exercising mark bookkeeping directly.
#[cfg(test)]
pub(crate) fn mark_here(&mut self, facts: PesFacts) {
self.marks.push_back((self.buf.len(), facts));
}
/// The marks, for a scanner that must resolve facts at several offsets
/// while the buffer's bytes are borrowed elsewhere. Use with
/// [`facts_for`], which applies the same at-or-before rule as
/// [`PesBuf::facts_at`].
pub(crate) fn marks_snapshot(&self) -> Vec<(usize, PesFacts)> {
self.marks.iter().copied().collect()
}
pub(crate) fn clear(&mut self) {
self.buf.clear();
self.marks.clear();
}
}
#[cfg(test)]
mod tests {
use super::*;
fn pes(data: &[u8], pts: Option<i64>, byte: Option<u64>) -> PesPacket {
PesPacket {
pid: 0x1100,
pts,
dts: None,
data: data.to_vec(),
source: byte.map(SourcePos::at_byte),
discontinuity: false,
}
}
/// The point of the type: a unit assembled from two packets is attributed
/// to the one that carried its FIRST byte, not the one that completed it.
#[test]
fn a_unit_spanning_two_packets_belongs_to_the_packet_it_started_in() {
let mut b = PesBuf::with_capacity(64);
b.push(&pes(&[1, 2, 3], Some(90_000), Some(1_000)));
b.push(&pes(&[4, 5, 6], Some(180_000), Some(2_000)));
let f = b.front();
assert_eq!(f.presentation_ns(), Some(pts_to_ns(90_000)));
assert_eq!(f.source.unwrap().byte, 1_000, "the packet it STARTED in");
}
/// And the timestamp and the source come from the SAME packet — the
/// property that cannot hold when each is derived separately.
#[test]
fn timestamp_and_source_always_come_from_one_packet() {
let mut b = PesBuf::with_capacity(64);
b.push(&pes(&[1, 2], Some(90_000), Some(10)));
b.push(&pes(&[3, 4], Some(180_000), Some(20)));
b.push(&pes(&[5, 6], Some(270_000), Some(30)));
for off in 0..6usize {
let f = b.facts_at(off);
let want = match off {
0 | 1 => (Some(pts_to_ns(90_000)), 10),
2 | 3 => (Some(pts_to_ns(180_000)), 20),
_ => (Some(pts_to_ns(270_000)), 30),
};
assert_eq!(
(f.presentation_ns(), f.source.unwrap().byte),
want,
"offset {off}"
);
}
}
/// Draining a completed unit must leave the REMAINDER attributed to the
/// packet that carried it, not to whichever packet starts next.
#[test]
fn draining_keeps_the_remainder_attributed_to_its_own_packet() {
let mut b = PesBuf::with_capacity(64);
b.push(&pes(&[1, 2, 3, 4], Some(90_000), Some(1_000)));
b.push(&pes(&[5, 6], Some(180_000), Some(2_000)));
// Emit a 2-byte unit: bytes 3 and 4 are still the FIRST packet's.
b.drain(2);
assert_eq!(b.front().source.unwrap().byte, 1_000);
// Emit those two: now the front is genuinely the second packet's.
b.drain(2);
assert_eq!(b.front().source.unwrap().byte, 2_000);
assert_eq!(b.front().presentation_ns(), Some(pts_to_ns(180_000)));
}
/// A payload-less PES contributed no byte, so it must not shadow the packet
/// that did — otherwise the next unit takes a timestamp from a packet whose
/// bytes are not in it.
#[test]
fn an_empty_packet_does_not_claim_the_next_units_bytes() {
let mut b = PesBuf::with_capacity(64);
b.push(&pes(&[1, 2], Some(90_000), Some(1_000)));
b.push(&pes(&[], Some(180_000), Some(2_000)));
assert_eq!(b.front().source.unwrap().byte, 1_000);
assert_eq!(b.len(), 2, "an empty packet adds no bytes");
}
/// Draining everything and refilling must not resurrect a stale mark.
#[test]
fn a_fully_drained_buffer_takes_its_next_packets_facts() {
let mut b = PesBuf::with_capacity(64);
b.push(&pes(&[1, 2], Some(90_000), Some(1_000)));
b.drain(2);
assert!(b.is_empty());
b.push(&pes(&[9], Some(450_000), Some(9_000)));
assert_eq!(b.front().source.unwrap().byte, 9_000);
assert_eq!(b.front().presentation_ns(), Some(pts_to_ns(450_000)));
}
/// The two ways a parser can obtain facts must agree when the unit begins
/// in the packet just handed over — otherwise "one pattern" is two.
#[test]
fn a_unit_starting_in_this_packet_reads_the_same_either_way() {
let p = pes(&[1, 2, 3], Some(90_000), Some(4_242));
let mut b = PesBuf::with_capacity(16);
b.push(&p);
assert_eq!(b.front(), PesFacts::of(&p));
}
/// Over-draining is clamped rather than panicking: a parser that
/// mis-sizes a unit must not take the process down.
#[test]
fn draining_past_the_end_is_clamped() {
let mut b = PesBuf::with_capacity(16);
b.push(&pes(&[1, 2], Some(90_000), Some(1)));
b.drain(99);
assert!(b.is_empty());
}
}
+56 -17
View File
@@ -214,7 +214,12 @@ impl ForcedTracker {
/// Stateful parser that collapses PGS display/clear PCS pairs into
/// duration-bearing Matroska frames. Implements [`CodecParser`].
pub struct PgsParser {
pending: Option<(i64, Vec<u8>)>,
/// The display set being accumulated, with the facts of the PES that
/// STARTED it. A set spans PES packets — it opens on a display PCS and
/// closes on the next one — so its timestamp and its source offset are the
/// opening packet's, never the closing packet's. Same rule the other
/// buffering parsers get from `PesBuf::front`.
pending: Option<(super::pesbuf::PesFacts, Vec<u8>)>,
}
impl Default for PgsParser {
@@ -234,12 +239,13 @@ impl PgsParser {
/// closes or replaces it), clamped to >= 0. Shared by the clear-PCS and
/// replace-PCS arms so the Frame shape stays in one place.
fn emit_pending(&mut self, end_pts_ns: i64) -> Option<Frame> {
let (start_pts, data) = self.pending.take()?;
let (facts, data) = self.pending.take()?;
let start_pts = facts.presentation_ns().unwrap_or(0);
let duration = end_pts_ns.saturating_sub(start_pts).max(0) as u64;
Some(Frame {
discontinuity: false,
coding: None,
source: None,
source: facts.source,
pts_ns: start_pts,
keyframe: true,
data,
@@ -271,12 +277,12 @@ impl CodecParser for PgsParser {
return self
.pending
.take()
.map(|(start_pts, data)| {
.map(|(facts, data)| {
vec![Frame {
discontinuity: false,
coding: None,
source: None,
pts_ns: start_pts,
source: facts.source,
pts_ns: facts.presentation_ns().unwrap_or(0),
keyframe: true,
data,
duration_ns: None,
@@ -300,11 +306,11 @@ impl CodecParser for PgsParser {
Some(0) => {
let frame = match pts {
Some(end) => self.emit_pending(end),
None => self.pending.take().map(|(start_pts, data)| Frame {
None => self.pending.take().map(|(facts, data)| Frame {
discontinuity: false,
coding: None,
source: None,
pts_ns: start_pts,
source: facts.source,
pts_ns: facts.presentation_ns().unwrap_or(0),
keyframe: true,
data,
duration_ns: None,
@@ -318,17 +324,24 @@ impl CodecParser for PgsParser {
Some(_) => match pts {
Some(start) => {
out.extend(self.emit_pending(start));
self.pending = Some((start, pes.data.clone()));
// The set's facts are THIS packet's — the one that opened
// it. `start` is that packet's PTS by construction.
self.pending = Some((super::pesbuf::PesFacts::of(pes), pes.data.clone()));
debug_assert_eq!(
super::pesbuf::PesFacts::of(pes).presentation_ns(),
Some(start),
"the opening packet's PTS is the set's start"
);
}
// A display PCS with no PTS has an unknown start time. Don't
// store it with a 0 sentinel (wrong start, absurd duration).
// Flush any prior pending undurated and skip storing this one.
None => {
out.extend(self.pending.take().map(|(start_pts, data)| Frame {
out.extend(self.pending.take().map(|(facts, data)| Frame {
discontinuity: false,
coding: None,
source: None,
pts_ns: start_pts,
source: facts.source,
pts_ns: facts.presentation_ns().unwrap_or(0),
keyframe: true,
data,
duration_ns: None,
@@ -353,7 +366,10 @@ impl CodecParser for PgsParser {
out.push(Frame {
discontinuity: false,
coding: None,
source: None,
// Emitted straight from THIS packet, so its facts are
// this packet's -- the same rule as a pending set,
// which takes the facts of the packet that opened it.
source: super::pesbuf::PesFacts::of(pes).source,
pts_ns: pts.unwrap_or(0),
keyframe: true,
data: pes.data.clone(),
@@ -379,11 +395,11 @@ impl CodecParser for PgsParser {
// until end of file, which is exactly the desired behavior for
// the final on-screen subtitle (see the module doc).
match self.pending.take() {
Some((start_pts, data)) => vec![Frame {
Some((facts, data)) => vec![Frame {
discontinuity: false,
coding: None,
source: None,
pts_ns: start_pts,
source: facts.source,
pts_ns: facts.presentation_ns().unwrap_or(0),
keyframe: true,
data,
duration_ns: None,
@@ -901,4 +917,27 @@ mod tests {
assert_eq!(t.facts().displays, u32::MAX);
assert_eq!(t.facts().forced_displays, u32::MAX);
}
/// A lone non-PCS segment with a PTS is emitted straight through rather
/// than accumulated, and it must still carry provenance. This path was
/// missed on the first pass and showed up on a real disc as a subtitle
/// track with no source offset -- the one track out of forty that could
/// not be placed by byte.
#[test]
fn a_lone_segment_emitted_directly_still_carries_provenance() {
let mut parser = PgsParser::new();
// A non-PCS segment (type 0x15 = ODS) with a PTS and no pending set.
let mut p = make_pes(vec![0x15, 0x00, 0x00, 0x00, 0x04, 1, 2, 3, 4], Some(90_000));
p.source = Some(crate::pes::SourcePos::at_byte(7_777));
let frames = parser.parse(&p);
assert!(
!frames.is_empty(),
"a lone segment with a PTS is passed through"
);
assert_eq!(
frames[0].source.map(|s| s.byte),
Some(7_777),
"emitted straight from this packet, so it carries this packet's offset"
);
}
}
+65 -34
View File
@@ -63,7 +63,10 @@ const AU_DURATION_NS_441: i64 = 907_029;
const MAX_TRUEHD_BUF: usize = 256 * 1024;
pub struct TrueHdParser {
buf: Vec<u8>,
/// Bytes assembled across PES packets, each attributable to the packet
/// that carried it, so an access unit takes the timestamp AND the source
/// offset of the packet covering its first byte.
acc: super::pesbuf::PesBuf,
next_pts_ns: i64,
/// Per-AU PTS increment. Defaults to the 48 kHz-family value (833_333) and
/// is refined to the 44.1 kHz-family value once the first major sync reveals
@@ -94,7 +97,7 @@ impl Default for TrueHdParser {
impl TrueHdParser {
pub fn new() -> Self {
Self {
buf: Vec::with_capacity(32768),
acc: super::pesbuf::PesBuf::with_capacity(32768),
next_pts_ns: 0,
au_duration_ns: AU_DURATION_NS,
tally: DropTally::new("truehd"),
@@ -195,15 +198,15 @@ impl TrueHdParser {
/// single source of truth shared with the AC-3 parser; a returned `0` there
/// (reserved fscod or out-of-range frmsizecod) is the unmappable case.
fn ac3_frame_at_head(&self) -> Ac3Size {
if self.buf.len() < 6 {
if self.acc.len() < 6 {
return Ac3Size::NeedMore;
}
let frame_bytes = super::ac3::ac3_frame_size(&self.buf);
let frame_bytes = super::ac3::ac3_frame_size(self.acc.as_slice());
if frame_bytes == 0 {
// Reserved fscod or out-of-range frmsizecod → unmappable header.
return Ac3Size::Unmappable;
}
if self.buf.len() < frame_bytes {
if self.acc.len() < frame_bytes {
return Ac3Size::NeedMore;
}
Ac3Size::Frame(frame_bytes)
@@ -363,7 +366,7 @@ impl CodecParser for TrueHdParser {
// never be stranded by an empty post-gap PES (defensive; the demuxer only
// emits non-empty PES today).
if pes.discontinuity {
self.buf.clear();
self.acc.clear();
}
if pes.data.is_empty() {
return Vec::new();
@@ -376,7 +379,7 @@ impl CodecParser for TrueHdParser {
// mid-AU would snap that AU's PTS backward/forward and break the
// monotonic +AU_DURATION_NS cadence (A/V drift). Once the buffer is empty
// the next PES legitimately begins a new AU and seeds the base.
if self.buf.is_empty()
if self.acc.is_empty()
&& let Some(pts) = pes.pts
{
// Resync to the authoritative PES PTS. TrueHD AUs are a fixed
@@ -421,12 +424,12 @@ impl CodecParser for TrueHdParser {
}
}
self.buf.extend_from_slice(&pes.data);
self.acc.push(pes);
let mut frames = Vec::new();
loop {
if self.buf.len() < 4 {
if self.acc.len() < 4 {
break;
}
@@ -438,19 +441,19 @@ impl CodecParser for TrueHdParser {
// when its computed end is corroborated by what follows: end of
// buffer (frame fills the rest), another AC-3 sync, or a plausible
// TrueHD AU header. If none holds, this is treated as a TrueHD AU.
if self.buf[0] == 0x0B && self.buf[1] == 0x77 {
if self.acc.as_slice()[0] == 0x0B && self.acc.as_slice()[1] == 0x77 {
match self.ac3_frame_at_head() {
Ac3Size::Unmappable => {
// Permanently unmappable header at the head would stall
// the parser forever; resync by dropping 2 bytes so one
// bad frame costs one frame, not the whole buffer.
self.buf.drain(..2);
self.acc.drain(2);
continue;
}
Ac3Size::NeedMore => break, // wait for the rest of the frame
Ac3Size::Frame(skip) => {
if ac3_boundary_corroborated(&self.buf, skip) {
self.buf.drain(..skip);
if ac3_boundary_corroborated(self.acc.as_slice(), skip) {
self.acc.drain(skip);
continue;
}
// Not corroborated — fall through and interpret the
@@ -460,19 +463,21 @@ impl CodecParser for TrueHdParser {
}
// TrueHD access unit: lower 12 bits of first 2 bytes = length in words
let unit_words = (((self.buf[0] as usize) << 8) | self.buf[1] as usize) & 0xFFF;
let unit_words = (((self.acc.as_slice()[0] as usize) << 8)
| self.acc.as_slice()[1] as usize)
& 0xFFF;
if unit_words == 0 {
// A zero-length AU is malformed/padding. The AU header is 4 bytes
// (length + timing); drain the whole header, not just the length
// word, otherwise the timing bytes get misread as the next
// length word and produce a spurious parse on the next iteration.
self.buf.drain(..4);
self.acc.drain(4);
continue;
}
// unit_words is masked to 12 bits, so unit_bytes <= 4095 * 2 = 8190;
// no separate oversize-resync guard is reachable.
let unit_bytes = unit_words * 2;
if self.buf.len() < unit_bytes {
if self.acc.len() < unit_bytes {
break; // incomplete access unit, wait for more data
}
@@ -482,10 +487,10 @@ impl CodecParser for TrueHdParser {
// gated on 0xBA alone in `au_check`.
let is_major_sync = unit_bytes >= 8
&& is_mlp_major_sync(u32::from_be_bytes([
self.buf[4],
self.buf[5],
self.buf[6],
self.buf[7],
self.acc.as_slice()[4],
self.acc.as_slice()[5],
self.acc.as_slice()[6],
self.acc.as_slice()[7],
]));
// Decodability gate. MLP/TrueHD decode state persists across access
@@ -493,8 +498,11 @@ impl CodecParser for TrueHdParser {
// major sync (the clean re-init point) rather than excised in place.
// The PTS clock advances across every dropped AU so a drop is a
// silence gap, never a shift.
let au = self.buf[..unit_bytes].to_vec();
let au = self.acc.as_slice()[..unit_bytes].to_vec();
let pts = self.next_pts_ns;
// Read BEFORE the drain below: this unit's source is the packet
// covering the CURRENT front, not the next unit's.
let au_src = self.acc.front().source;
let mut emit_keyframe: Option<bool> = None; // Some(is_keyframe) => emit
let mut drop_reason: Option<(&'static str, bool)> = None; // (reason, verified)
@@ -560,7 +568,7 @@ impl CodecParser for TrueHdParser {
frames.push(Frame {
discontinuity: false,
coding: None,
source: None,
source: au_src,
pts_ns: pts,
keyframe,
data: au,
@@ -575,14 +583,14 @@ impl CodecParser for TrueHdParser {
.record_collateral_drop(pts, self.au_duration_ns, au.len(), reason);
}
}
self.buf.drain(..unit_bytes);
self.acc.drain(unit_bytes);
self.next_pts_ns += self.au_duration_ns;
}
// Bound memory on malformed input: a stream that never yields a
// complete frame must not grow the buffer without limit.
if self.buf.len() > MAX_TRUEHD_BUF {
self.buf.clear();
if self.acc.len() > MAX_TRUEHD_BUF {
self.acc.clear();
}
frames
@@ -1455,7 +1463,7 @@ mod tests {
"TrueHD AU behind a bad header is recovered"
);
assert_eq!(frames[0].data.len(), 200);
assert!(parser.buf.is_empty(), "buffer fully consumed, no stall");
assert!(parser.acc.is_empty(), "buffer fully consumed, no stall");
}
#[test]
@@ -1707,7 +1715,7 @@ mod tests {
f.is_empty(),
"must not emit fewer bytes than the length field"
);
assert_eq!(parser.buf.len(), 100, "partial AU retained");
assert_eq!(parser.acc.len(), 100, "partial AU retained");
}
/// Largest AU the 12-bit length field can declare: 0xFFF words × 2.
@@ -1720,7 +1728,7 @@ mod tests {
//
// The bound that actually holds is MAX_AU_BYTES, not MAX_TRUEHD_BUF:
// `parse`'s loop only breaks with data retained when
// `self.buf.len() < unit_bytes`, and `unit_bytes` is
// `self.acc.len() < unit_bytes`, and `unit_bytes` is
// `((buf[0] << 8 | buf[1]) & 0xFFF) * 2 <= 8190`. Every other exit
// drains. So the post-loop `buf.len() > MAX_TRUEHD_BUF` cap (256 KiB) is
// an unreachable backstop — an exhaustive sweep of all 65536 two-byte
@@ -1740,17 +1748,17 @@ mod tests {
frag[0] = 0xFF;
frag[1] = 0xFF;
let _ = parser.parse(&make_pes(frag, Some(0)));
worst = worst.max(parser.buf.len());
worst = worst.max(parser.acc.len());
assert!(
parser.buf.len() < MAX_AU_BYTES,
parser.acc.len() < MAX_AU_BYTES,
"reassembly buffer exceeded the AU-length ceiling: {} >= {}",
parser.buf.len(),
parser.acc.len(),
MAX_AU_BYTES
);
assert!(
parser.buf.len() <= MAX_TRUEHD_BUF,
parser.acc.len() <= MAX_TRUEHD_BUF,
"reassembly buffer exceeded cap: {} > {}",
parser.buf.len(),
parser.acc.len(),
MAX_TRUEHD_BUF
);
}
@@ -1813,7 +1821,7 @@ mod tests {
fn ac3_frame_at_head_needs_more_when_buffer_short() {
// < 6 bytes buffered → NeedMore (can't read the AC-3 header).
let mut parser = TrueHdParser::new();
parser.buf = vec![0x0B, 0x77, 0x00];
parser.acc.seed(&[0x0B, 0x77, 0x00]);
// Drive through parse: a short 0x0B77 head must wait, not emit.
let f = parser.parse(&make_pes(vec![0x0B, 0x77, 0x00], Some(0)));
assert!(f.is_empty());
@@ -2014,4 +2022,27 @@ mod tests {
assert_eq!(truehd_sample_rate_hz(info.format_info), Some(96000));
assert_eq!(info.is_atmos, Some(true));
}
/// Same rule for TrueHD: a unit spanning two packets belongs to the one
/// that carried its first byte.
#[test]
fn an_access_unit_carries_the_source_of_the_packet_it_began_in() {
let mut parser = TrueHdParser::new();
let unit = make_truehd_unit(512);
let mut p1 = make_pes(unit[..200].to_vec(), Some(90_000));
p1.source = Some(crate::pes::SourcePos::at_byte(1_000));
let first = parser.parse(&p1);
assert!(first.is_empty(), "partial unit held");
let mut p2 = make_pes(unit[200..].to_vec(), Some(180_000));
p2.source = Some(crate::pes::SourcePos::at_byte(9_000));
let frames = parser.parse(&p2);
assert!(!frames.is_empty(), "the completed unit is emitted");
assert_eq!(
frames[0].source.map(|s| s.byte),
Some(1_000),
"the unit belongs to the packet its FIRST byte came from"
);
}
}
+202 -9
View File
@@ -190,7 +190,13 @@ pub(crate) fn codec_label(codec: Codec) -> &'static str {
/// to `.sup`, VobSub records `.idx` entries and emits the sidecar at finish.
trait EsWriter: Send {
/// Write one frame's payload to `w`. Returns the number of bytes written to
/// the main file (used by the VobSub writer for `.idx` filepos tracking).
/// the main file.
///
/// Nothing in production reads this count: the sole caller discards it with
/// `?`, and `VobSubWriter` tracks `.idx` fileposes from its own `pos` field.
/// This previously claimed the VobSub writer depended on the return value,
/// which would have a maintainer believe a wrong count shows up as broken
/// `.idx` output. It does not.
fn write_frame(&mut self, w: &mut dyn Write, f: &PesFrame, pts_ns: i64) -> io::Result<usize>;
/// Finalize. Default: no-op. The VobSub writer serializes its `.idx` here.
@@ -631,10 +637,18 @@ fn xml_escape(s: &str) -> String {
}
/// Replace path-hostile characters in a filename component.
///
/// Control characters included, NUL above all. Every string this touches is
/// disc bytes, and a language code is three raw STN bytes run through
/// `from_utf8_lossy` with no validation — `00 00 00` is the ordinary
/// "undefined" encoding on real Blu-rays. A NUL in a path aborts `File::create`
/// with `InvalidInput`, which took the whole demux export down before a single
/// track file was opened.
fn sanitize(s: &str) -> String {
s.chars()
.map(|c| match c {
'/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|' => '_',
c if c.is_control() => '_',
_ => c,
})
.collect()
@@ -660,6 +674,12 @@ pub struct DemuxSink {
/// Index = track id; `None` for unselected tracks.
tracks: Vec<Option<TrackOut>>,
ref_video_track: Option<usize>,
/// Every VIDEO track index, recorded before the kind filter.
///
/// The filtered `tracks` slots are `None` for a class this export drops, so
/// they cannot answer "is this video" for a frame that still flows through
/// `write()`.
video_tracks: std::collections::HashSet<usize>,
/// First PTS observed on `ref_video_track`, recorded in `write()` REGARDLESS
/// of whether that track has a `TrackOut`. The DELAY reference cannot live in
/// `TrackOut::first_pts_ns`: `audio://` / `sub://` filter the video track's
@@ -671,6 +691,15 @@ pub struct DemuxSink {
ref_first_pts_ns: Option<i64>,
timeline: TimelineContinuity,
finished: bool,
/// Frames the timeline PLACED (not necessarily persisted).
///
/// A frame for a track this export filters out still flows through
/// `write()` and still counts here, so this is "the timeline placed
/// something" rather than "a file received bytes". That is the right
/// denominator for the drop gate below — it asks whether placement worked
/// at all — but it is deliberately NOT a per-track written count, so a
/// single lost track among many is not visible to it.
frames_mapped: u64,
}
impl DemuxSink {
@@ -679,6 +708,7 @@ impl DemuxSink {
std::fs::create_dir_all(dir)?;
let mut tracks: Vec<Option<TrackOut>> = Vec::with_capacity(title.streams.len());
let mut ref_video_track = None;
let mut video_tracks: std::collections::HashSet<usize> = std::collections::HashSet::new();
for (idx, stream) in title.streams.iter().enumerate() {
let selected = opts
@@ -705,6 +735,15 @@ impl DemuxSink {
if kind == TrackKind::Video && ref_video_track.is_none() {
ref_video_track = Some(idx);
}
// Also BEFORE the kind filter, and for the same reason: the seam
// crossing rule differs for a track that carries B-frame reorder,
// and a video track still flows through write() on an `audio://`
// or `sub://` export even though nothing persists it. Reading the
// kind back out of the filtered slot would call it non-video and
// hand it the permissive rule.
if kind == TrackKind::Video {
video_tracks.insert(idx);
}
// Kind filter: `audio://` / `sub://` keep only their class.
if opts.kind_filter.is_some_and(|k| k != kind) {
tracks.push(None);
@@ -741,9 +780,11 @@ impl DemuxSink {
opts: opts.clone(),
tracks,
ref_video_track,
video_tracks,
ref_first_pts_ns: None,
timeline: TimelineContinuity::new(),
timeline: TimelineContinuity::with_clips(&title.clips, title.content_format),
finished: false,
frames_mapped: 0,
})
}
@@ -754,8 +795,11 @@ impl DemuxSink {
Naming::Pid => format!("{} {:04x}", sanitize(&opts.base), pid),
Naming::Friendly => {
let mut parts = vec![sanitize(&opts.base), format!("t{idx:02}")];
// The language is disc bytes too, and got none of the
// treatment `opts.base` two lines up already had.
let lang = sanitize(lang);
if !lang.is_empty() {
parts.push(lang.to_string());
parts.push(lang);
}
parts.push(codec_label(codec).to_string());
parts.join(" ")
@@ -866,7 +910,23 @@ impl Stream for DemuxSink {
// an audio ES before the video ES, in which case track 0 is audio and a
// non-video epoch driver would ratchet the frontier on sparse/lagging PTS.
let drives = Some(frame.track) == self.ref_video_track;
let pts = self.timeline.adjust(frame.pts, drives);
// Keyed on the track KIND, not on driving epochs: a second video track
// (a Dolby Vision enhancement layer) does not drive epochs but does
// carry B-frame reorder, and the seam-crossing rule differs on exactly
// that property.
let is_video = self.video_tracks.contains(&frame.track);
// See `MkvMuxer::write_frame`: `None` is material outside the
// playlist's clip marks and is dropped rather than emitted.
let Some(pts) = self.timeline.map(
frame.pts,
drives,
frame.track,
is_video,
frame.source.map(|s| s.byte),
) else {
return Ok(());
};
self.frames_mapped = self.frames_mapped.saturating_add(1);
if drives {
// Delay reference: recorded here, not in the track's `TrackOut`, so
// it survives the `audio://` / `sub://` kind filter dropping the
@@ -885,6 +945,34 @@ impl Stream for DemuxSink {
return Ok(());
}
self.finished = true;
// Same reporting as the MKV muxer's finish. Frames the playlist's clip
// marks exclude are dropped on purpose, but the count must not be
// write-only in one sink and reported in the other — an unexpected
// volume here is how a demux ends up quietly short.
let seam_dropped = self.timeline.dropped_total();
// Frames ARRIVED and every one was dropped. That is a fully-dropped
// title, which this sink used to finish cleanly: a directory of
// zero-byte track files beside a populated chapters document, at exit
// 0. Keyed on frames having been offered, because a sink that is never
// given any — a chapters-only export, or a track class the title does
// not carry — legitimately writes none.
if self.frames_mapped == 0 && seam_dropped > 0 {
return Err(crate::error::Error::SinkWroteNothing.into());
}
if seam_dropped > self.frames_mapped {
return Err(crate::error::Error::SeamPlanDroppedMost {
dropped: seam_dropped,
written: self.frames_mapped,
}
.into());
}
if seam_dropped > 0 {
tracing::info!(
target: "mux",
dropped = seam_dropped,
"frames outside the playlist's clip marks were dropped at clip joins"
);
}
// Flush each track's codec writer, then the buffered file.
for slot in self.tracks.iter_mut() {
if let Some(t) = slot.as_mut() {
@@ -910,6 +998,52 @@ mod tests {
Resolution, SampleRate, VideoStream,
};
// ── The language component is disc bytes ──────────────────────────────
//
// `opts.base` was sanitised and the language beside it was not, though a
// language code is three raw STN bytes run through `from_utf8_lossy` with
// no validation. `00 00 00` is the ordinary "undefined" encoding on real
// Blu-rays, and a NUL in a path fails `File::create` with `InvalidInput` —
// which aborted the whole export before a single track file was opened.
/// The stem stays one usable filename component whatever the disc says.
#[test]
fn a_hostile_language_code_cannot_break_the_output_filename() {
let opts = DemuxOptions {
base: "Movie".to_string(),
..Default::default()
};
assert!(matches!(opts.naming, Naming::Friendly), "default naming");
for lang in ["\u{0}\u{0}\u{0}", "a/b", "..", "a\nb", "e:s"] {
let stem = DemuxSink::stem_for(&opts, 1, 0x1100, lang, Codec::Ac3);
assert!(
!stem.chars().any(|c| c.is_control()),
"control character survived into the filename for {lang:?}: {stem:?}"
);
assert!(
!stem.contains('/') && !stem.contains('\\') && !stem.contains(':'),
"a path separator survived for {lang:?}: {stem:?}"
);
assert!(
std::path::Path::new(&stem).components().count() == 1,
"the stem must stay ONE component for {lang:?}: {stem:?}"
);
}
}
/// A legitimate language is still carried through untouched — the
/// sanitiser must not be so lossy that it stops naming the track.
#[test]
fn an_ordinary_language_code_survives_sanitising() {
let opts = DemuxOptions {
base: "Movie".to_string(),
..Default::default()
};
let stem = DemuxSink::stem_for(&opts, 1, 0x1100, "eng", Codec::Ac3);
assert!(stem.contains("eng"), "got {stem:?}");
}
fn video_stream(codec: Codec) -> DiscStream {
DiscStream::Video(VideoStream {
pid: 0x1011,
@@ -1466,14 +1600,14 @@ mod tests {
fn timeline_track0_drives_epoch_others_ride() {
let mut tl = TimelineContinuity::new();
// Clip 1: video 0..10s (track 0 drives the epoch).
assert_eq!(tl.adjust(0, true), 0);
assert_eq!(tl.adjust(0, false), 0); // audio rides the same offset
assert_eq!(tl.adjust(10_000_000_000, true), 10_000_000_000);
assert_eq!(tl.adjust(0, true, 0), 0);
assert_eq!(tl.adjust(0, false, 1), 0); // audio rides the same offset
assert_eq!(tl.adjust(10_000_000_000, true, 0), 10_000_000_000);
// Clip 2 seam: video PTS jumps back to ~0 (> 3s back) → new epoch.
let out = tl.adjust(0, true);
let out = tl.adjust(0, true, 0);
assert!(out >= 10_000_000_000, "epoch must advance past prev high");
// Audio in clip 2 (non-epoch) gets the SAME offset (A/V sync preserved).
let a = tl.adjust(0, false);
let a = tl.adjust(0, false, 1);
assert_eq!(a, out);
}
@@ -1609,6 +1743,65 @@ mod tests {
let _ = std::fs::remove_dir_all(&dir);
}
/// A demux export whose seam plan drops every frame must FAIL, not write a
/// directory of empty track files and report success.
///
/// Audit finding: this sink builds its timeline with
/// `TimelineContinuity::with_clips` but no test ever gave it a title with
/// usable PlayItem marks, so `frames_mapped` and `dropped_total()` were
/// always zero and BOTH gates in `finish` were unreachable. Deleting either
/// left the whole suite green while a `demux://` export of a
/// seamless-branching title wrote zero-byte track files beside a populated
/// chapters document, at exit 0.
#[test]
fn a_demux_export_the_seam_plan_emptied_fails() {
let dir = tempdir();
let mut title = title_with(vec![video_stream(Codec::H264)], vec![None]);
// Two clips (SeamPlan needs >= 2 with strictly increasing IN marks)
// covering 100s..200s and 200s..300s in 45 kHz ticks.
title.clips = vec![
crate::disc::Clip {
feed_span: None,
clip_id: "00000".into(),
in_time: 100 * 45_000,
out_time: 200 * 45_000,
duration_secs: 100.0,
source_packets: 0,
},
crate::disc::Clip {
feed_span: None,
clip_id: "00001".into(),
in_time: 200 * 45_000,
out_time: 300 * 45_000,
duration_secs: 100.0,
source_packets: 0,
},
];
let mut sink = DemuxSink::create(&dir, &title, &DemuxOptions::default()).unwrap();
// Frames far before the first IN mark: the plan places none of them.
for (i, pts) in [0i64, 1_000_000_000].iter().enumerate() {
let f = PesFrame {
coding: None,
source: None,
track: 0,
pts: *pts,
keyframe: true,
data: vec![0x00, 0x00, 0x00, 0x01, 0x09, 0x10, i as u8],
duration_ns: None,
};
let _ = Stream::write(&mut sink, &f);
}
let err = Stream::finish(&mut sink).expect_err("an emptied export must fail");
assert_eq!(
crate::error::error_code(&err),
Some(crate::error::E_SINK_WROTE_NOTHING),
"a fully-dropped demux export must report SinkWroteNothing, not success"
);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn sink_read_returns_write_only() {
let dir = tempdir();
+148
View File
@@ -566,6 +566,20 @@ impl DiscStream {
.into());
}
// The read SOURCE is gone (a prefetch producer thread that
// terminated), not one range of media. Shrinking and retrying at
// the same LBA asks a dead source for data it can never produce,
// and the `skip_errors` branch below would then zero-fill and
// advance over every remaining sector of the title and still
// return success. Abort with the terminal error itself — a
// fabricated SCSI status would be a lie, so this is deliberately
// NOT folded into the transport-failure arm above.
if let Some(e) = res.as_ref().err()
&& e.is_source_terminated()
{
return Err(crate::error::Error::SourceTerminated.into());
}
if (sectors as u32) <= align {
// Bottomed out at one unit (AACS) / one sector (CSS) / the
// extent tail. This is single-pass disc→MKV, which has NO Pass N
@@ -621,6 +635,16 @@ impl DiscStream {
.into());
}
// Same rule as after the first-attempt read: the 60s recovery
// read goes through the same source, so it can be the call
// that discovers the source is dead. Skipping the unit would
// zero-fill the rest of the title as fabricated content.
if let Some(e) = rec.as_ref().err()
&& e.is_source_terminated()
{
return Err(crate::error::Error::SourceTerminated.into());
}
// Recovery read also failed. Skip the WHOLE failed unit or bail.
// Zero-filling and advancing by the full unit keeps
// current_offset unit-aligned, so the next read still begins on a
@@ -1859,6 +1883,130 @@ mod tests {
);
}
/// REGRESSION (round-4 audit): an ordinary MEDIUM ERROR bad sector must
/// keep its identity when it crosses the prefetch producer channel — the
/// same `DiscRead` with its SCSI status, NOT a transport failure.
///
/// `PrefetchedSectorSource::read_sectors` re-wrapped every error that
/// crossed the channel as `Error::IoError`, and `is_scsi_transport_failure`
/// matches `IoError` (the wedged-USB-bridge arm). So a bad sector reached
/// `fill_extents` looking like a dead bus and aborted the pass with a
/// fabricated status 0xFF — the exact inverse of what that short-circuit
/// exists for, and it told the user to power-cycle a healthy drive.
///
/// Asserted on the source, not on a `fill_extents` skip: the producer
/// thread exits for good after sending an error, so nothing downstream of
/// it can genuinely recover the rest of the title (see
/// `dead_prefetch_producer_does_not_silently_zero_fill_the_title`). An
/// assertion that the pass continues could only ever have been satisfied
/// by fabricated zeros.
#[test]
fn bad_sector_keeps_its_identity_across_the_prefetch_channel() {
const COUNT: u32 = 9;
let log = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
let reader = RecordingReader {
capacity: COUNT,
bad_sector: 4,
log: log.clone(),
};
let mut prefetched = crate::sector::PrefetchedSectorSource::new_with_events(
reader,
vec![crate::disc::Extent {
start_lba: 0,
sector_count: COUNT,
}],
8,
1,
None,
None,
)
.expect("spawn producer");
let mut buf = vec![0u8; 8 * 2048];
let err = crate::sector::SectorSource::read_sectors(&mut prefetched, 0, 8, &mut buf, false)
.expect_err("the batch covering the bad sector must fail");
assert!(
!err.is_scsi_transport_failure(),
"a MEDIUM ERROR bad sector is not a dead bus; got {err:?}"
);
assert!(
matches!(
err,
crate::error::Error::DiscRead {
sector: 4,
status: Some(0x02),
..
}
),
"the producer's typed error must survive the channel intact; got {err:?}"
);
}
/// The prefetch producer thread terminates PERMANENTLY on its first read
/// error, so once one bad sector has crossed the channel the source can
/// never deliver another byte. Driving `fill_extents` to exhaustion after
/// that must NOT look like a completed pass: every remaining sector would
/// be fabricated zeros, and DATA LOSS MUST NEVER LOOK LIKE SUCCESS.
///
/// The expectation is the product rule, not the code: a source that is
/// permanently out of data must report that, not answer `Ok(0)` forever —
/// which `commit_read` legitimately reads as an ordinary short read and
/// zero-fills.
#[test]
fn dead_prefetch_producer_does_not_silently_zero_fill_the_title() {
const COUNT: u32 = 30;
let log = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
let reader = RecordingReader {
capacity: COUNT,
bad_sector: 4,
log: log.clone(),
};
let prefetched = crate::sector::PrefetchedSectorSource::new_with_events(
reader,
vec![crate::disc::Extent {
start_lba: 0,
sector_count: COUNT,
}],
8,
1,
None,
None,
)
.expect("spawn producer");
let mut stream = DiscStream::new(
Box::new(prefetched),
synthetic_title(COUNT),
crate::decrypt::DecryptKeys::None,
8,
ContentFormat::BdTs,
false,
None,
)
.unwrap();
stream.skip_errors = true;
// Drive the whole title. Bounded so a regression cannot hang the suite.
let mut completed_clean = false;
for _ in 0..(COUNT as usize * 4) {
match stream.fill_extents() {
Ok(true) => continue,
Ok(false) => {
completed_clean = true;
break;
}
Err(_) => break,
}
}
assert!(
!completed_clean,
"the producer died at sector 4, so sectors 4..{COUNT} were never \
read — reporting the pass as complete zero-fills {} of {} bytes \
and calls it success",
stream.lost_bytes,
COUNT as u64 * 2048
);
}
/// AACS unit-alignment skip (the #1 coverage gap). With `unit_align=3`
/// (DecryptKeys::Aacs) and `skip_errors=true`, a single bad mid-extent
/// sector must NOT desync the rest of the title: every `read_sectors`
+476 -10
View File
@@ -359,6 +359,22 @@ fn write_hdr10<W: Write + Seek>(w: &mut W, h: &crate::mux::codec::Hdr10Metadata)
Ok(())
}
/// The Matroska `Language` element the muxer writes for a stream whose source
/// reported `lang`. RFC 9559 §12 defines the element as an ISO 639-2 code and
/// gives no meaning to an empty one; the code for "no language stated" is
/// `und`, and that is what a source with no language table (the HD-DVD EVO
/// stream probe, a Blu-ray STN slot with no language bytes) has to emit. The
/// element is written unconditionally by `MkvMuxer::new`, so this is the one
/// place that decides it — a source-side default would have to be repeated in
/// every scanner and would still leave the muxer able to ship an invalid file.
fn language_or_und(lang: &str) -> String {
if lang.is_empty() {
"und".to_string()
} else {
lang.to_string()
}
}
impl MkvTrack {
/// Build a video track from a [`VideoStream`]. Language defaults to `"und"`;
/// colour metadata is derived from the stream's colour space and HDR format
@@ -530,7 +546,7 @@ impl MkvTrack {
Self {
track_type: ebml::TRACK_TYPE_AUDIO,
codec_id,
language: a.language.clone(),
language: language_or_und(&a.language),
name,
codec_private: None,
is_default: !a.secondary,
@@ -587,7 +603,7 @@ impl MkvTrack {
Self {
track_type: ebml::TRACK_TYPE_SUBTITLE,
codec_id,
language: s.language.clone(),
language: language_or_und(&s.language),
name: String::new(),
codec_private: s.codec_data.clone(),
is_default: false,
@@ -680,8 +696,9 @@ pub struct MkvMuxer<W: Write + Seek> {
cues: Vec<CuePoint>,
frame_count: u64,
/// Frames handed to `write_frame` that were dropped because no cluster was
/// open yet (a cluster only opens on a track-0 video keyframe). See
/// `write_frame` for the track-0 invariant.
/// open yet (a cluster only opens on a keyframe from the primary video
/// track, whatever index that is — not necessarily track 0). See
/// `write_frame` for the cluster-driver invariant.
///
/// The ALL-dropped case is surfaced as an error by `finish()`, but via
/// `frame_count == 0`, not via this counter. A PARTIAL drop — leading audio /
@@ -1323,6 +1340,25 @@ impl<W: Write + Seek> MkvMuxer<W> {
pub fn set_opening_capture(&mut self, capture: Option<crate::diag::OpeningCapture>) {
self.opening_capture = capture;
}
/// Drive seam correction from the title's PlayItem marks instead of
/// inferring it from PTS jumps.
///
/// A multi-clip Blu-ray playlist joins its clips with overlaps and skips
/// that PTS inspection cannot recover: a forward jump is indistinguishable
/// from frames lost to damaged media, and an overlap smaller than the
/// reorder threshold is invisible. Given the marks, each clip is placed at
/// the sum of the earlier clips' durations, so the output runs exactly as
/// long as the playlist says the title is.
///
/// No-op for a title with fewer than two clips or without usable marks —
/// DVD, HD-DVD and file sources keep the inference path.
pub fn set_clips(
&mut self,
clips: &[crate::disc::Clip],
content_format: crate::disc::ContentFormat,
) {
self.continuity = TimelineContinuity::with_clips(clips, content_format);
}
/// Write a single frame.
///
@@ -1348,6 +1384,14 @@ impl<W: Write + Seek> MkvMuxer<W> {
/// Block and the dependent (right-eye) access unit rides as the
/// BlockAdditional under the track's `mvcC` mapping. Such a frame is always a
/// `BlockGroup` (never a SimpleBlock), with a `ReferenceBlock` when it is not
/// TEST-ONLY. Every production path writes through [`MkvMuxer::write_frame_at`]
/// and passes the byte offset the frame was read from; the deferred-mux
/// replay of buffered frames was the last one that did not, and it now
/// does. Keeping this compiled out of the library means there is no
/// production API through which a frame can reach the file with its
/// provenance discarded — the placement logic stopped relying on
/// timestamps precisely so that could not happen silently.
#[cfg(test)]
/// a keyframe. `None` for every non-3D frame.
pub fn write_frame(
&mut self,
@@ -1357,6 +1401,36 @@ impl<W: Write + Seek> MkvMuxer<W> {
data: &[u8],
duration_ns: Option<u64>,
block_additional: Option<&[u8]>,
) -> io::Result<()> {
self.write_frame_at(
track_idx,
pts_ns,
keyframe,
data,
duration_ns,
block_additional,
None,
)
}
/// [`Self::write_frame`] with the frame's SOURCE BYTE OFFSET.
///
/// Under a seam plan that offset identifies which clip the frame came from
/// by lookup rather than by inferring it from the timestamp — which is
/// ambiguous inside an overlap, where two clips' mark ranges both contain
/// the same instant. Callers that have a `PesFrame` should pass
/// `frame.source.map(|s| s.byte)`; `write_frame` is the same call with no
/// provenance, which falls back to the mark heuristics.
#[allow(clippy::too_many_arguments)]
pub fn write_frame_at(
&mut self,
track_idx: usize,
pts_ns: i64,
keyframe: bool,
data: &[u8],
duration_ns: Option<u64>,
block_additional: Option<&[u8]>,
src_byte: Option<u64>,
) -> io::Result<()> {
// --log-level 3: capture the first ~100 coded frames per track to the
// side file BEFORE any timeline mangling, with the codec parser's own
@@ -1390,8 +1464,22 @@ impl<W: Write + Seek> MkvMuxer<W> {
// all tracks, A/V-sync-preserving) keeps the boundary from becoming a
// band of non-monotonic block timestamps. Only the PRIMARY video track
// drives the boundary decision; every other track (audio, subtitle, DV
// EL) rides the current offset. No-op for single-clip titles.
let pts_ns = self.continuity.adjust(pts_ns, drives_epoch);
// EL) rides the current offset. A single-clip BD title still trims to
// its one clip's [in, out] marks (dropping trailing audio the m2ts
// carries past OUT); only a source with no marks at all is a true no-op.
// `None` means the playlist does not include this frame — material
// outside every clip's IN/OUT marks, which only a seam-plan-driven
// title can report. Dropping it is the point: emitting it is what put
// duplicate content on the timeline at a join.
let Some(pts_ns) = self.continuity.map(
pts_ns,
drives_epoch,
track_idx,
self.track_is_video.get(track_idx).copied().unwrap_or(false),
src_byte,
) else {
return Ok(());
};
let raw_ticks = pts_ns / TIMESTAMP_SCALE_NS;
// Cluster boundaries normally coincide with a video keyframe so every
@@ -1653,11 +1741,17 @@ impl<W: Write + Seek> MkvMuxer<W> {
/// Finish the MKV file: write Cues element.
///
/// # Track-0 invariant
/// # Cluster-driver invariant
///
/// A cluster only opens on a track-0 video keyframe, so the caller must
/// supply track 0 as the video track and deliver a keyframe on it before
/// (or alongside) other-track data. If no track-0 keyframe ever arrives,
/// A cluster only opens on a keyframe from the PRIMARY VIDEO TRACK — the
/// first track whose type is video, at whatever index it occupies (see
/// `cluster_driver`, which is `primary_video_track.unwrap_or(0)`; index 0
/// is only the fallback for a file with no video track at all). The caller
/// must deliver a keyframe on that track before (or alongside) other-track
/// data. This said "track-0" and required the caller to place video at
/// index 0, which the code has never actually required — so a reader
/// debugging dropped frames would suspect track ordering, which is not it.
/// If no such keyframe ever arrives,
/// every `write_frame` is silently dropped; rather than emit a structurally
/// valid but empty MKV (zero clusters, zero frames), `finish` returns
/// `Error::MkvInvalid` when frames were submitted but none were written.
@@ -1667,6 +1761,15 @@ impl<W: Write + Seek> MkvMuxer<W> {
// would otherwise yield a structurally-empty MKV with no clusters or
// cues. Surface that as an error rather than writing valid-but-empty
// output.
// Order matters. A title the seam plan dropped ENTIRELY also has a zero
// frame count, and `MkvInvalid` is classified by
// `is_skippable_title_stub` as an empty nav/menu stub — so reporting it
// that way would make an all-titles rip drop a real feature and finish
// the rest at exit 0. Decide the seam case FIRST, with a code that is
// not skippable.
if self.frame_count == 0 && self.continuity.dropped_total() > 0 {
return Err(crate::error::Error::SinkWroteNothing.into());
}
if self.frame_count == 0 {
return Err(crate::error::Error::MkvInvalid.into());
}
@@ -1682,6 +1785,36 @@ impl<W: Write + Seek> MkvMuxer<W> {
"frames were discarded before the first cluster opened (no track-0 video keyframe had arrived yet); they are absent from the output"
);
}
// Frames the playlist's clip marks excluded are dropped on purpose — a
// join legitimately discards the material a disc stores twice — but the
// count must not be write-only. Same reasoning as the pre-cluster
// counter above: an unexpected VOLUME here is how a title ends up
// quietly short while the run reports success.
let seam_dropped = self.continuity.dropped_total();
// Counting a drop is not the same as bounding it. A join legitimately
// discards the material a disc stores twice — tens of frames — but if a
// title's marks do not line up with its PES clock the plan can discard
// most of it, and the only other gate is a GLOBAL zero-frame check
// whose error is additionally classified as a skippable stub. Between
// them, a title emitting seconds of a two-hour feature exits 0. That is
// the defect this change set already shipped once.
//
// More dropped than kept is never a real join, so it fails.
if seam_dropped > self.frame_count {
return Err(crate::error::Error::SeamPlanDroppedMost {
dropped: seam_dropped,
written: self.frame_count,
}
.into());
}
if seam_dropped > 0 {
tracing::info!(
target: "mux",
dropped = seam_dropped,
frames_written = self.frame_count,
"frames outside the playlist's clip marks were dropped at clip joins"
);
}
// The source declared no duration up-front (DURATION was reserved as a
// placeholder). Derive the real runtime from the muxed timeline so the
// Segment declares it — and so the BPS tags below can be computed.
@@ -3190,6 +3323,285 @@ mod tests {
);
}
/// Read back the value of the FIRST `Language` element in `data` as a
/// UTF-8 string. `LANGUAGE` (0x22B59C) is a 3-byte EBML ID; its size is
/// always a 1-byte VINT for the short strings this writer emits.
fn first_language_value(data: &[u8]) -> &str {
let pos = find_id(data, ebml::LANGUAGE).expect("Language element must be present");
let len = (data[pos + 3] & 0x7F) as usize;
std::str::from_utf8(&data[pos + 4..pos + 4 + len]).unwrap()
}
/// RFC 9559 §12 / the Matroska `Language` element spec restrict the
/// legacy `Language` element to the Matroska language form (ISO 639-2,
/// three lowercase letters), never ISO 639-1 (two letters). The DVD
/// IFO audio-attribute block itself carries a raw ISO 639-1 code (e.g.
/// "en") on disc — `ifo::parse_audio_attr` converts it to ISO 639-2
/// before returning, via `ifo::dvd_lang_to_iso639_2`. This mimics the
/// real DVD pipeline (`disc/dvd.rs`'s `Stream::Audio` construction) end
/// to end: real on-disc IFO bytes -> `ifo::parse_audio_attr` ->
/// `disc::AudioStream` -> `MkvTrack::audio` -> the muxer -> the emitted
/// `Language` element.
#[test]
fn dvd_two_letter_language_becomes_iso_639_2_in_language_element() {
// AC-3 (coding_mode=0), 48 kHz, 6 channels, on-disc language "en" —
// the exact byte layout `ifo::audio_attr_parsing` pins.
let mut attr_bytes = vec![0u8; 8];
attr_bytes[0] = 0x00;
attr_bytes[1] = 0x05;
attr_bytes[2] = b'e';
attr_bytes[3] = b'n';
let attr = crate::ifo::parse_audio_attr(&attr_bytes, 0).unwrap();
assert_eq!(
attr.language, "eng",
"parse_audio_attr must already return the ISO 639-2 form"
);
let audio_stream = crate::disc::AudioStream {
pid: 0xBD80,
codec: attr.codec,
channels: crate::disc::AudioChannels::from_count(attr.channels),
language: attr.language,
sample_rate: crate::disc::SampleRate::from_hz(attr.sample_rate),
secondary: false,
purpose: crate::disc::LabelPurpose::Normal,
label: String::new(),
};
let track = MkvTrack::audio(&audio_stream);
let buf = Cursor::new(Vec::new());
let tracks = [track];
let muxer = MkvMuxer::new(buf, &tracks, None, 60.0, &[]).unwrap();
let data = muxer.writer.into_inner();
assert_eq!(
first_language_value(&data),
"eng",
"a DVD-sourced ISO 639-1 code must be written as its ISO 639-2 \
equivalent in the Matroska Language element, per RFC 9559 §12"
);
}
/// A source that knows no language at all leaves `language` EMPTY, and the
/// muxer writes the `Language` element unconditionally — so an empty
/// string becomes a zero-length `Language` in the shipped file, which is
/// not a Matroska language form (RFC 9559 §12 wants three ISO 639-2
/// letters) and is not the ISO 639-2 code for "unknown" either.
///
/// This is the state every HD-DVD rip is in: `disc::hddvd`'s EVO stream
/// probe has no language table to read and sets `language: String::new()`
/// on every audio stream it finds. The DVD path normalises to "und" in
/// `ifo::parse_audio_attr`; the guard has to exist at the muxer too, which
/// is the one place every source funnels through.
#[test]
fn a_source_with_no_language_emits_und_not_an_empty_language_element() {
// Exactly what `disc::hddvd::probe_evo_streams` builds.
let audio_stream = crate::disc::AudioStream {
pid: 0xBD80,
codec: Codec::Ac3Plus,
channels: crate::disc::AudioChannels::Surround51,
language: String::new(),
sample_rate: crate::disc::SampleRate::S48,
secondary: false,
purpose: crate::disc::LabelPurpose::Normal,
label: String::new(),
};
let subtitle_stream = crate::disc::SubtitleStream {
pid: 0x1200,
codec: Codec::Pgs,
language: String::new(),
forced: false,
qualifier: crate::disc::LabelQualifier::None,
codec_data: None,
};
for track in [
MkvTrack::audio(&audio_stream),
MkvTrack::subtitle(&subtitle_stream),
] {
let track_type = track.track_type;
let buf = Cursor::new(Vec::new());
let tracks = [track];
let muxer = MkvMuxer::new(buf, &tracks, None, 60.0, &[]).unwrap();
let data = muxer.writer.into_inner();
assert_eq!(
first_language_value(&data),
"und",
"track type {track_type}: a stream with no known language must \
emit the ISO 639-2 'undetermined' code, never a zero-length \
Language element"
);
}
}
/// An unmapped or absent DVD language code (bytes 0x00 0x00 in the IFO
/// attribute block) must degrade to the valid Matroska "undetermined"
/// code `und`, never to an empty string or a raw 2-letter code — both of
/// which violate the Matroska language form.
#[test]
fn dvd_unmapped_or_empty_language_becomes_und_in_language_element() {
// Empty IFO language bytes (0x00 0x00).
let mut empty_bytes = vec![0u8; 8];
empty_bytes[0] = 0x00; // AC-3, 48k
empty_bytes[1] = 0x05; // 6ch
let empty_attr = crate::ifo::parse_audio_attr(&empty_bytes, 0).unwrap();
assert_eq!(
empty_attr.language, "und",
"IFO zero bytes (unspecified) must resolve to 'und', not empty"
);
// An IFO code with no known ISO 639-1 -> 639-2 mapping (e.g. a
// fictitious "zz").
let mut unmapped_bytes = vec![0u8; 8];
unmapped_bytes[0] = 0x00;
unmapped_bytes[1] = 0x05;
unmapped_bytes[2] = b'z';
unmapped_bytes[3] = b'z';
let unmapped_attr = crate::ifo::parse_audio_attr(&unmapped_bytes, 0).unwrap();
assert_eq!(
unmapped_attr.language, "und",
"an on-disc code with no known ISO 639-1 -> 639-2 mapping must \
degrade to 'und', never pass through raw"
);
for attr in [empty_attr, unmapped_attr] {
let audio_stream = crate::disc::AudioStream {
pid: 0xBD80,
codec: attr.codec,
channels: crate::disc::AudioChannels::from_count(attr.channels),
language: attr.language,
sample_rate: crate::disc::SampleRate::from_hz(attr.sample_rate),
secondary: false,
purpose: crate::disc::LabelPurpose::Normal,
label: String::new(),
};
let track = MkvTrack::audio(&audio_stream);
let buf = Cursor::new(Vec::new());
let tracks = [track];
let muxer = MkvMuxer::new(buf, &tracks, None, 60.0, &[]).unwrap();
let data = muxer.writer.into_inner();
assert_eq!(
first_language_value(&data),
"und",
"an empty or unmapped DVD language code must degrade to 'und', \
never an empty string or an invalid ISO 639-1 code"
);
}
}
/// Build an 8-byte DVD IFO audio-attribute block (AC-3, 48 kHz, 6ch)
/// carrying `code` in the language bytes, run it through the real DVD
/// pipeline (`ifo::parse_audio_attr` -> `disc::AudioStream` ->
/// `MkvTrack::audio` -> the muxer) and return the value that actually
/// lands in the emitted Matroska `Language` element.
fn emitted_language_for_dvd_code(code: &[u8; 2]) -> String {
let mut attr_bytes = vec![0u8; 8];
attr_bytes[0] = 0x00; // AC-3, 48 kHz
attr_bytes[1] = 0x05; // 6 channels
attr_bytes[2] = code[0];
attr_bytes[3] = code[1];
let attr = crate::ifo::parse_audio_attr(&attr_bytes, 0).unwrap();
let audio_stream = crate::disc::AudioStream {
pid: 0xBD80,
codec: attr.codec,
channels: crate::disc::AudioChannels::from_count(attr.channels),
language: attr.language,
sample_rate: crate::disc::SampleRate::from_hz(attr.sample_rate),
secondary: false,
purpose: crate::disc::LabelPurpose::Normal,
label: String::new(),
};
let track = MkvTrack::audio(&audio_stream);
let buf = Cursor::new(Vec::new());
let tracks = [track];
let muxer = MkvMuxer::new(buf, &tracks, None, 60.0, &[]).unwrap();
let data = muxer.writer.into_inner();
first_language_value(&data).to_string()
}
/// The ISO 639-1 -> ISO 639-2 conversion must cover the WHOLE of ISO
/// 639-1, not just the handful of languages that happen to appear in
/// Blu-ray menu-graphic filenames. A Region-2 disc routinely carries
/// Romanian, Bulgarian, Croatian, Serbian, Slovak, Slovenian, Hebrew,
/// Estonian, Latvian, Lithuanian, Icelandic and so on; if those all
/// collapse to `und`, every one of a disc's subtitle tracks emits the
/// same `Language` value and nothing else tells them apart (DVD streams
/// carry an empty `label`). A valid-but-identical code is worse for the
/// user than the invalid one it replaced, so each of these must reach the
/// emitted `Language` element as its own correct three-letter code.
#[test]
fn dvd_language_outside_the_menu_vocabulary_is_still_mapped() {
assert_eq!(
emitted_language_for_dvd_code(b"ro"),
"ron",
"Romanian ('ro'), common on Region-2 discs, must reach the \
Matroska Language element as 'ron' — not 'und'"
);
// The rest of the set the menu-label table never knew, one per
// language so a single missing table row fails loudly.
for (code, expected) in [
(b"bg", "bul"),
(b"hr", "hrv"),
(b"sr", "srp"),
(b"sk", "slk"),
(b"sl", "slv"),
(b"he", "heb"),
(b"et", "est"),
(b"lv", "lav"),
(b"lt", "lit"),
(b"is", "isl"),
(b"id", "ind"),
(b"vi", "vie"),
(b"fa", "fas"),
] {
assert_eq!(
emitted_language_for_dvd_code(code),
expected,
"DVD language {:?} must map to {expected:?} in the emitted \
Language element",
std::str::from_utf8(code).unwrap()
);
}
}
/// DVD-Video froze its language list on the 1988 edition of ISO 639-1,
/// which spelled Hebrew `iw`, Indonesian `in` and Yiddish `ji`. Real
/// discs authored to that list carry those bytes, so they must map to the
/// same ISO 639-2 codes as the modern `he` / `id` / `yi` spellings rather
/// than degrading to `und`.
#[test]
fn dvd_era_language_aliases_map_to_the_modern_code() {
assert_eq!(
emitted_language_for_dvd_code(b"iw"),
"heb",
"the DVD-era spelling of Hebrew ('iw') must emit 'heb'"
);
assert_eq!(emitted_language_for_dvd_code(b"in"), "ind");
assert_eq!(emitted_language_for_dvd_code(b"ji"), "yid");
// ...and agree with the modern spellings.
assert_eq!(emitted_language_for_dvd_code(b"he"), "heb");
assert_eq!(emitted_language_for_dvd_code(b"id"), "ind");
assert_eq!(emitted_language_for_dvd_code(b"yi"), "yid");
}
/// Widening the table must not weaken the degradation guarantee: a code
/// that is not ISO 639-1 at all still has to yield exactly `und`, a valid
/// Matroska language value, and never a passed-through two-letter code,
/// an empty string, or a guess.
#[test]
fn unknown_dvd_language_still_yields_exactly_und() {
for code in [b"zz", b"qq", b"xx"] {
assert_eq!(
emitted_language_for_dvd_code(code),
"und",
"a code outside ISO 639-1 must degrade to exactly 'und'"
);
}
// Empty language bytes (0x00 0x00) likewise.
assert_eq!(emitted_language_for_dvd_code(&[0x00, 0x00]), "und");
}
#[test]
fn mkv_forced_flag_on_forced_subtitle() {
use crate::disc::SubtitleStream;
@@ -4376,6 +4788,60 @@ mod tests {
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
}
/// A seam-plan clip list whose marks exclude every frame must fail with
/// `SinkWroteNothing`, NOT `MkvInvalid`.
///
/// Audit finding: both gates in `finish()` were dead code under test —
/// `set_clips` was never called anywhere in the suite, so
/// `continuity.dropped_total()` was always 0 and neither branch could be
/// reached. Swapping the two `frame_count == 0` checks left the whole suite
/// green, while a title the seam plan emptied came back as `MkvInvalid` —
/// which `is_skippable_title_stub` classifies as an empty nav/menu stub, so
/// an all-titles rip would silently omit a real feature and exit 0.
#[test]
fn a_title_the_seam_plan_emptied_is_not_reported_as_a_skippable_stub() {
let tracks = [make_video_track()];
// TWO clips (SeamPlan::from_clips requires at least two, with strictly
// increasing IN marks), covering 100s..200s and 200s..300s in 45 kHz
// ticks. Every frame below sits far before the first IN, so the plan
// places none of them and drops them all.
let clips = [
crate::disc::Clip {
feed_span: None,
clip_id: "00000".into(),
in_time: 100 * 45_000,
out_time: 200 * 45_000,
duration_secs: 100.0,
source_packets: 0,
},
crate::disc::Clip {
feed_span: None,
clip_id: "00001".into(),
in_time: 200 * 45_000,
out_time: 300 * 45_000,
duration_secs: 100.0,
source_packets: 0,
},
];
let mut muxer = MkvMuxer::new(Cursor::new(Vec::new()), &tracks, None, 100.0, &[]).unwrap();
muxer.set_clips(&clips, crate::disc::ContentFormat::BdTs);
// 0s and 1s — both before the clip's IN mark.
let _ = muxer.write_frame(0, 0, true, &[0x01; 16], None, None);
let _ = muxer.write_frame(0, 1_000_000_000, true, &[0x02; 16], None, None);
let err = muxer.finish().unwrap_err();
assert_eq!(
crate::error::error_code(&err),
Some(crate::error::E_SINK_WROTE_NOTHING),
"an emptied seam title must report SinkWroteNothing"
);
assert!(
!crate::error::is_skippable_title_stub(&err),
"it must NOT be classified as a skippable stub — that is what makes an \
all-titles rip drop a real feature and still exit 0"
);
}
#[test]
fn backjumped_audio_handled_by_i16_split_no_wrap() {
// A NON-VIDEO (audio) frame whose PTS back-jumps far below the open
+98 -3
View File
@@ -12,8 +12,19 @@ use super::{WriteSeek, ebml};
type MkvHeaderResult = io::Result<(crate::disc::DiscTitle, Vec<(u16, Vec<u8>)>, i64, TrackTable)>;
/// Skip `n` bytes on a forward-only reader (no Seek required).
///
/// A skip that runs out of input before `n` bytes is a TRUNCATED element, and is
/// reported the same way `ebml::read_binary_val` reports a truncated body: as
/// `MkvSourceInvalid`. Discarding `io::copy`'s byte count instead made a skip
/// that hit EOF look like a success, so one corrupt size field mid-Clusters
/// drained the rest of the file, the next element header raised
/// `UnexpectedEof`, and `Stream::read` mapped that to `Ok(None)` — half the
/// title missing, `errors = 0`, `completed = true`.
fn skip_bytes(r: &mut impl Read, n: u64) -> io::Result<()> {
io::copy(&mut r.take(n), &mut io::sink())?;
let skipped = io::copy(&mut r.take(n), &mut io::sink())?;
if skipped != n {
return Err(crate::error::Error::MkvSourceInvalid.into());
}
Ok(())
}
@@ -302,13 +313,15 @@ fn emit_to_muxer(
frame: &crate::pes::PesFrame,
additional: Option<&[u8]>,
) -> io::Result<()> {
m.write_frame(
m.write_frame_at(
frame.track,
frame.pts,
frame.keyframe,
&frame.data,
frame.duration_ns,
additional,
// Provenance: which clip this frame came from is a lookup, not a guess.
frame.source.map(|s| s.byte),
)
}
@@ -504,17 +517,27 @@ impl MkvStream {
self.disc_title.duration_secs,
&self.disc_title.chapters,
)?;
// Seam correction from the playlist's marks where the title has them.
muxer.set_clips(&self.disc_title.clips, self.disc_title.content_format);
if let Some(path) = &pending.opening_capture_path {
muxer.set_opening_capture(crate::diag::OpeningCapture::new(path, pending.tracks.len()));
}
for (f, additional) in pending.buffered.drain(..) {
muxer.write_frame(
// Provenance must survive the replay. These frames were buffered
// before the muxer existed, but each still carries the byte offset
// it was read from — dropping it here sent them down the timestamp
// heuristic instead, the one path the rest of this change set
// exists to stop relying on. They sit at the head of the title, so
// clip 0 is usually the right guess and the damage was bounded;
// "usually right" is exactly what provenance replaced.
muxer.write_frame_at(
f.track,
f.pts,
f.keyframe,
&f.data,
f.duration_ns,
additional.as_deref(),
f.source.map(|s| s.byte),
)?;
}
self.mode = Mode::Write(WriteMode::Active(Box::new(muxer)));
@@ -3181,6 +3204,78 @@ mod tests {
assert!(stream.read().unwrap().is_none(), "clean EOF → None");
}
/// A skipped element whose declared size runs PAST the end of the file is a
/// truncated element, exactly like a truncated `read_binary_val` body — and
/// must be reported the same way, as `MkvSourceInvalid`.
///
/// `skip_bytes` used to discard `io::copy`'s returned count, so the skip
/// "succeeded" having drained the rest of the file. The next element header
/// then hit `UnexpectedEof`, which `read()` maps to `Ok(None)` — a clean end
/// of stream. One corrupt size field mid-Clusters therefore threw away every
/// remaining frame of the title and reported `errors = 0`, `complete = true`.
#[test]
fn a_skip_past_eof_is_an_error_not_a_clean_end_of_stream() {
let mut cluster = Vec::new();
ebml::write_id(&mut cluster, ebml::CLUSTER).unwrap();
ebml::write_unknown_size(&mut cluster).unwrap();
// Frame 1 — read normally.
let block = [0x81u8, 0x00, 0x00, 0x80, 0xAA];
ebml::write_id(&mut cluster, ebml::SIMPLE_BLOCK).unwrap();
ebml::write_size(&mut cluster, block.len() as u64).unwrap();
cluster.extend_from_slice(&block);
// A VOID whose size field is corrupt: it claims 1 MiB, and the file
// holds only the handful of bytes below. This is the "corrupt size
// field mid-Clusters" case.
ebml::write_id(&mut cluster, ebml::VOID).unwrap();
ebml::write_size(&mut cluster, 1024 * 1024).unwrap();
// Frame 2 — the rest of the title, swallowed by the bad skip.
let block2 = [0x81u8, 0x00, 0x01, 0x80, 0xBB];
ebml::write_id(&mut cluster, ebml::SIMPLE_BLOCK).unwrap();
ebml::write_size(&mut cluster, block2.len() as u64).unwrap();
cluster.extend_from_slice(&block2);
let bytes = mkv_with_track_and_cluster(1, 1, &cluster);
let mut stream = MkvStream::open(Cursor::new(bytes)).unwrap();
assert!(stream.read().unwrap().is_some(), "first frame reads");
let e = match stream.read() {
Err(e) => e,
Ok(None) => panic!(
"a skip that hit EOF was reported as a CLEAN END OF STREAM: the \
rest of the title is gone and the caller sees errors = 0, \
complete = true"
),
Ok(Some(_)) => panic!("the truncated skip must not yield a frame"),
};
assert!(is_mkv_source_invalid(&e), "{e:?}");
}
/// The honest path this fix must not break: a skipped element whose declared
/// size is exactly satisfied by the bytes present is still skipped cleanly,
/// and the genuine EOF that follows is still `Ok(None)`.
#[test]
fn a_fully_satisfied_skip_still_ends_at_a_clean_eof() {
let mut cluster = Vec::new();
ebml::write_id(&mut cluster, ebml::CLUSTER).unwrap();
ebml::write_unknown_size(&mut cluster).unwrap();
// A VOID that is fully present.
ebml::write_id(&mut cluster, ebml::VOID).unwrap();
ebml::write_size(&mut cluster, 8).unwrap();
cluster.extend_from_slice(&[0u8; 8]);
let block = [0x81u8, 0x00, 0x00, 0x80, 0xAA];
ebml::write_id(&mut cluster, ebml::SIMPLE_BLOCK).unwrap();
ebml::write_size(&mut cluster, block.len() as u64).unwrap();
cluster.extend_from_slice(&block);
let bytes = mkv_with_track_and_cluster(1, 1, &cluster);
let mut stream = MkvStream::open(Cursor::new(bytes)).unwrap();
let f = stream.read().unwrap().expect("the frame after the VOID");
assert_eq!(f.data, vec![0xAA]);
assert!(
stream.read().unwrap().is_none(),
"a genuine EOF at a record boundary is still a clean end"
);
}
// ============================================================
// Block LACING (RFC 9559 §10.3) and TrackNumber→stream routing
// (RFC 9559 §5.1.4.1.1).
+22 -2
View File
@@ -627,16 +627,36 @@ mod tests {
/// accept_from() must reject a connection whose first bytes are NOT the
/// FMKV magic — there is no metadata to drive muxing, so it surfaces
/// NoMetadata rather than proceeding with an empty/garbage title.
///
/// FLAKE FIXED, not the behaviour under test: this used to
/// `shutdown(Shutdown::Both)` the instant the bytes were written. Closing
/// the READ half while the server had not yet read makes the kernel answer
/// the server's in-flight data with an RST, so `accept_from` came back
/// `ConnectionReset` instead of the `InvalidInput` this asserts — rarely
/// when run alone, reproducibly under the loaded concurrent suite, where
/// the server thread is descheduled long enough for the race to open. A
/// logging/protocol assertion that fails at random teaches the next person
/// to re-run until green, which is how a real regression gets waved
/// through.
///
/// Half-closing (`Shutdown::Write`) delivers the same EOF the test needs
/// while leaving the read half open, and blocking on a read until the
/// server drops its end keeps the socket alive for as long as the server
/// is looking at it. The port was already ephemeral (`:0`), so it was
/// never a port collision.
#[test]
fn accept_from_rejects_stream_without_fmkv_header() {
use std::io::Read as _;
use std::io::Write as _;
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let handle = std::thread::spawn(move || {
// Raw non-FMKV bytes (not starting with 'F') then close.
// Raw non-FMKV bytes (not starting with 'F') then EOF.
let mut s = TcpStream::connect(addr).unwrap();
s.write_all(&[0x47u8; 64]).unwrap(); // TS sync bytes, no FMKV magic
s.shutdown(std::net::Shutdown::Both).unwrap();
s.shutdown(std::net::Shutdown::Write).unwrap();
// Park until the server closes, so no RST can overtake the data.
let _ = s.read(&mut [0u8; 1]);
});
let err = match NetworkStream::accept_from(listener) {
Ok(_) => panic!("missing FMKV header must error, not silently accept"),
+256 -10
View File
@@ -17,7 +17,7 @@ use super::codec::startcode::find_start_code;
const PACK_HEADER_ID: u8 = 0xBA;
/// System header start code suffix.
const SYSTEM_HEADER_ID: u8 = 0xBB;
const SYSTEM_HEADER_ID: u8 = crate::consts::pes_stream_id::SYSTEM_HEADER;
/// Program end start code suffix.
const PROGRAM_END_ID: u8 = 0xB9;
@@ -169,6 +169,24 @@ pub struct PsDemuxer {
/// stay byte-identical.
buffer_base: u64,
has_base: bool,
/// Boundary-scan cursor for an unbounded (length-0) PES still waiting for
/// its terminating PS-layer unit: `(buffer offset of the PES start code,
/// buffer offset up to which the search has already proved there is no
/// boundary)`. Both are buffer-relative and are rebased when the buffer
/// drains.
///
/// Without it, every `feed` re-searches the WHOLE accumulated payload from
/// the PES header: the buffer only stops growing at [`MAX_PS_BUFFER`], so a
/// stream that declares an unbounded PES and then never emits a PS-layer
/// start code (a corrupt or crafted VOB) makes the demuxer scan up to 4 MiB
/// per call, quadratic in the bytes fed. Cleared whenever the PES is
/// emitted, so it can never outlive the packet it describes.
pending_scan: Option<(usize, usize)>,
/// Test-only: total bytes examined by `find_ps_boundary`. Pins the cursor
/// above — the property it exists for is a WORK bound, which no
/// packet-level assertion can observe.
#[cfg(test)]
boundary_bytes_scanned: u64,
}
impl Default for PsDemuxer {
@@ -184,6 +202,9 @@ impl PsDemuxer {
buffer: Vec::with_capacity(64 * 1024),
buffer_base: 0,
has_base: false,
pending_scan: None,
#[cfg(test)]
boundary_bytes_scanned: 0,
}
}
@@ -220,6 +241,8 @@ impl PsDemuxer {
// discarded.
let packets = self.extract_packets(true);
self.buffer.clear();
// The buffer the cursor indexes into is gone.
self.pending_scan = None;
packets
}
@@ -288,11 +311,29 @@ impl PsDemuxer {
// start code — the video ES payload is itself full of
// 00 00 01 xx codes that would otherwise cut the PES short.
let end = if pes_packet_len == 0 {
match find_ps_boundary(&self.buffer, sc + 4) {
Some(next) => next,
// Resume where the last call stopped searching for
// THIS PES's terminating unit; anything before that is
// already proved boundary-free.
let from = match self.pending_scan {
Some((pes_at, searched_to)) if pes_at == sc => searched_to,
_ => sc + 4,
};
let (found, searched_to) = find_ps_boundary(&self.buffer, from);
#[cfg(test)]
{
self.boundary_bytes_scanned += searched_to.saturating_sub(from) as u64;
}
match found {
Some(next) => {
self.pending_scan = None;
next
}
// At EOF the rest of the buffer is this PES's
// payload — emit it.
None if flushing => self.buffer.len(),
None if flushing => {
self.pending_scan = None;
self.buffer.len()
}
None => {
// No boundary buffered yet. Normally wait for
// more data, but a corrupt stream could declare
@@ -301,8 +342,10 @@ impl PsDemuxer {
// stops untrusted input forcing unbounded
// allocation. Past the cap, flush what we have.
if self.buffer.len() - sc > MAX_PS_BUFFER {
self.pending_scan = None;
self.buffer.len()
} else {
self.pending_scan = Some((sc, searched_to));
break; // wait for more data
}
}
@@ -338,6 +381,13 @@ impl PsDemuxer {
if self.has_base {
self.buffer_base += pos as u64;
}
// The cursor is a BUFFER offset, so it moves with the drain. A
// pending PES always starts at or after `pos` (the loop broke on
// it, having already consumed everything before it), so neither
// component can underflow.
self.pending_scan = self
.pending_scan
.map(|(pes_at, searched_to)| (pes_at - pos, searched_to - pos));
}
// Trim a start-code-free tail. Every other exit from the loop above
@@ -359,6 +409,10 @@ impl PsDemuxer {
if self.has_base {
self.buffer_base += drop as u64;
}
// A pending PES implies a start code IS in the buffer, so this
// branch cannot run while one is open; drop the cursor anyway
// rather than leave a stale offset behind this drain.
self.pending_scan = None;
}
packets
@@ -380,11 +434,19 @@ const START_CODE_PREFIX_KEEP: usize = 2;
/// PES inside its own payload and re-scan the discarded video bytes as bogus PS
/// units. Restricting the search to PS-layer IDs (>= 0xB9, excluding the video
/// ES codes below it) frames the unbounded PES at the right boundary.
fn find_ps_boundary(data: &[u8], from: usize) -> Option<usize> {
/// Returns `(boundary, searched_to)`. `searched_to` is the offset up to which
/// every byte has been PROVED not to begin a PS-layer boundary start code, so
/// a later call over the same buffer (grown at the tail) may resume there
/// instead of re-scanning the payload from the PES header. When the scan runs
/// off the end, the last two bytes are NOT proved: a `00 00 01` prefix can
/// straddle the next feed's boundary by up to two bytes.
fn find_ps_boundary(data: &[u8], from: usize) -> (Option<usize>, usize) {
let mut pos = from;
while let Some(sc) = find_start_code(data, pos) {
if sc + 3 >= data.len() {
return None;
// A start code whose ID byte has not arrived yet: undecided, so
// the next scan must look at it again.
return (None, sc);
}
let id = data[sc + 3];
if id == PACK_HEADER_ID
@@ -392,11 +454,11 @@ fn find_ps_boundary(data: &[u8], from: usize) -> Option<usize> {
|| id == PROGRAM_END_ID
|| is_pes_stream_id(id)
{
return Some(sc);
return (Some(sc), sc);
}
pos = sc + 4;
}
None
(None, data.len().saturating_sub(2).max(from))
}
/// Check whether a start code byte is a valid PES stream ID that carries payload.
@@ -1752,7 +1814,191 @@ mod tests {
/// after it — exactly the tail a real feed can end on.
#[test]
fn find_ps_boundary_handles_a_bare_start_code_at_the_buffer_head() {
assert_eq!(find_ps_boundary(&[0x00, 0x00, 0x01], 0), None);
assert_eq!(
find_ps_boundary(&[0x00, 0x00, 0x01], 0),
(None, 0),
"an undecided trailing start code is not proved boundary-free"
);
}
/// An unbounded (length-0) PES is terminated by the next PS-LAYER unit, and
/// until one arrives the payload accumulates in the buffer. The search for
/// that unit must not restart at the PES header on every feed: the buffer
/// only stops growing at `MAX_PS_BUFFER` (4 MiB), and a feed is one read
/// batch (at most 510 sectors ≈ 1 MiB, 60 sectors ≈ 120 KiB on an optical
/// drive), so re-scanning from byte 0 costs work quadratic in the bytes
/// fed — up to 4 MiB of scanning per call, for as long as a corrupt or
/// crafted VOB withholds the boundary. A conformant DVD ends every pack
/// within 2048 bytes and never reaches this state.
///
/// Measured directly, because a work bound has no packet-level shadow:
/// `boundary_bytes_scanned` counts the bytes `find_ps_boundary` examines.
/// 256 chunks x 4 KiB of boundary-free payload is 1 MiB of input;
/// re-scanning from the header on every call examines
/// 4 KiB * 256*257/2 = ~128 MiB.
///
/// Mutation: drop the `Some((pes_at, searched_to)) if pes_at == sc` arm so
/// `from` is always `sc + 4`.
#[test]
fn an_unterminated_pes_is_not_rescanned_from_its_header_every_feed() {
const CHUNKS: usize = 256;
const CHUNK: usize = 4096;
let mut demuxer = PsDemuxer::new();
// Unbounded PES header (length 0), then payload that carries no start
// code at all, so no PS-layer boundary is ever found.
demuxer.feed(&[0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x00, 0x00]);
for _ in 0..CHUNKS {
assert!(
demuxer.feed(&[0xFFu8; CHUNK]).is_empty(),
"no boundary yet, so no PES can be emitted"
);
}
let fed = (CHUNKS * CHUNK) as u64;
assert!(
demuxer.boundary_bytes_scanned <= 2 * fed,
"boundary search examined {} bytes over {fed} bytes of payload — \
the scan must advance with the buffer, not restart at the PES header",
demuxer.boundary_bytes_scanned
);
// ...and the cursor must not have cost correctness: the PES still ends
// at the pack header that finally arrives, with its whole payload.
let pack = [
0x00,
0x00,
0x01,
PACK_HEADER_ID,
0x44,
0x00,
0x04,
0x00,
0x04,
0x01,
0x00,
0x00,
0x03,
0xF8,
];
let packets = demuxer.feed(&pack);
assert_eq!(packets.len(), 1, "the pack header terminates the PES");
assert_eq!(
packets[0].data.len(),
CHUNKS * CHUNK,
"the whole accumulated payload belongs to the PES"
);
}
/// The resume cursor is a BUFFER offset, so it must be rebased when the
/// buffer drains — and this is the only test in which a drain actually
/// happens while a cursor is live.
///
/// Mutations this catches, both halves of
/// `self.pending_scan.map(|(pes_at, searched_to)| (pes_at - pos, searched_to - pos))`
/// in `extract_packets`:
/// * `searched_to - pos` -> `searched_to`: the next call resumes `pos`
/// bytes PAST where the previous scan actually stopped, so that window
/// is never examined. Here the terminating pack header lands inside it
/// and is missed outright — the unbounded PES runs on past its real
/// end, swallowing the following unit. That is a CORRECTNESS failure,
/// not a slow path, and the assertion on the emitted packet catches it.
/// * `pes_at - pos` -> `pes_at`: the stale offset no longer equals the
/// PES's post-drain `sc`, the resume arm stops matching and the search
/// restarts at the PES header. Caught by `boundary_bytes_scanned`,
/// which is why the fixture puts 64 KiB of payload in the SAME chunk
/// that opens the PES: that is exactly the span a restart re-examines,
/// so the mutant roughly doubles the bytes scanned.
///
/// `an_unterminated_pes_is_not_rescanned_from_its_header_every_feed` cannot
/// reach either: it opens the unbounded PES as the very FIRST bytes of the
/// very first feed, so nothing ever drains ahead of it, `pos` stays 0 and
/// the subtraction is a no-op. The comment above it asserts neither
/// component can underflow; nothing exercised the arithmetic at all.
///
/// So this fixture puts COMPLETE PS units — a pack header and a
/// length-bounded PES — ahead of the unbounded video PES *in the same
/// chunk*. The loop consumes them, breaks on the unbounded PES, and drains
/// `pos` bytes with `pending_scan` live: exactly the real DVD shape, where
/// a video PES opens partway through a read batch.
#[test]
fn a_resume_cursor_survives_the_drain_of_units_ahead_of_the_unbounded_pes() {
// Payload fed in the SAME chunk that opens the PES. Large enough that
// re-scanning it is unmistakable in `boundary_bytes_scanned`, and it is
// the exact span the un-rebased `pes_at` mutant re-examines.
const PAYLOAD: usize = 64 * 1024;
// A 14-byte MPEG-2 pack header with pack_stuffing_length 0.
const PACK: [u8; 14] = [
0x00,
0x00,
0x01,
PACK_HEADER_ID,
0x44,
0x00,
0x04,
0x00,
0x04,
0x01,
0x00,
0x00,
0x03,
0xF8,
];
// A length-BOUNDED PES — a complete unit, so the loop consumes it and
// `pos` advances past it before breaking on the unbounded PES.
const BOUNDED_PES: [u8; 11] = [
0x00, 0x00, 0x01, 0xE0, 0x00, 0x05, 0x80, 0x00, 0x00, 0xAA, 0xBB,
];
// The unbounded (length-0) video PES whose scan must be resumed.
const OPEN_PES: [u8; 9] = [0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x00, 0x00];
// Bytes drained ahead of the unbounded PES on the first feed — the
// `pos` the cursor must be rebased by.
const DRAINED: usize = PACK.len() + BOUNDED_PES.len();
let mut demuxer = PsDemuxer::new();
let mut first = PACK.to_vec();
first.extend_from_slice(&BOUNDED_PES);
first.extend_from_slice(&OPEN_PES);
first.extend_from_slice(&[0xFFu8; PAYLOAD]);
let head = demuxer.feed(&first);
assert_eq!(
head.len(),
1,
"the bounded PES ahead of the open one is emitted immediately, \
which is what makes the buffer drain with a cursor live"
);
// The terminating pack arrives at the head of the next feed — i.e.
// within `DRAINED` bytes of where the previous scan stopped, which is
// precisely the window an un-rebased `searched_to` skips over.
assert!(
PACK.len() <= DRAINED,
"the terminating pack must fit inside the window a stale \
`searched_to` would skip, or the mutant survives"
);
let packets = demuxer.feed(&PACK);
assert_eq!(
packets.len(),
1,
"the pack header terminates the open PES; a scan resumed past it \
never sees it and the PES runs on"
);
assert_eq!(
packets[0].data.len(),
PAYLOAD,
"exactly the payload fed belongs to the PES"
);
// Work bound: the payload is proved boundary-free ONCE. Re-scanning it
// after the drain roughly doubles this.
assert!(
demuxer.boundary_bytes_scanned <= (PAYLOAD + 1024) as u64,
"boundary search examined {} bytes over {PAYLOAD} bytes of payload — \
a cursor left un-rebased across the drain never matches the PES's \
new offset, so the scan restarts at the header",
demuxer.boundary_bytes_scanned
);
}
/// The boundary-ID check is a 4-way `||`; a mutant that turns the FIRST
@@ -1763,7 +2009,7 @@ mod tests {
let data = [0x00, 0x00, 0x01, PACK_HEADER_ID, 0xAA];
assert_eq!(
find_ps_boundary(&data, 0),
Some(0),
(Some(0), 0),
"a pack header start code alone must register as a PS-layer boundary"
);
}
+294 -184
View File
@@ -55,9 +55,15 @@ pub enum StreamUrl {
Stdio,
/// ISO disc image file.
Iso { path: PathBuf },
/// Decrypted file-tree output directory (`dir://`). A sink that writes
/// per-file decrypted bytes (not muxed PES frames), so it never flows
/// through `output()`; the CLI routes a `Dir` dest to `Disc::extract_tree`.
/// An extracted disc file tree (`dir://`) — a source AND a sink.
///
/// As a SINK it writes per-file decrypted bytes rather than muxed PES
/// frames, so it never flows through `output()`; the CLI routes a `Dir`
/// dest to `Disc::extract_tree`.
///
/// As a SOURCE (1.6.1) it is an image-level source: `crate::dirimage`
/// synthesizes a real UDF volume over the folder, so it reaches the same
/// scan/mux path `iso://` does and every destination follows.
Dir { path: PathBuf },
/// Null sink (write-only, discards data).
Null,
@@ -142,9 +148,22 @@ impl StreamUrl {
}
}
/// Whether this URL represents a disc source (disc:// or iso://).
/// Whether this URL is an IMAGE-level source — one that carries a UDF
/// filesystem, so it can be scanned into a title list, have `-t`/`-a`/`-s`
/// applied, and feed either a PES sink or an image sink.
///
/// `dir://` joined in 1.6.1, when `crate::dirimage` gave a folder a real
/// synthesized UDF volume.
///
/// NOT the same predicate as the CLI's `engine::is_disc_source`, which
/// means "is a live drive" and drives tray/eject behaviour. That one must
/// never gain `Dir` — a directory routed down the live-drive rip path
/// would open, lock and eject a drive that has nothing to do with it.
pub fn is_disc_source(&self) -> bool {
matches!(self, StreamUrl::Disc { .. } | StreamUrl::Iso { .. })
matches!(
self,
StreamUrl::Disc { .. } | StreamUrl::Iso { .. } | StreamUrl::Dir { .. }
)
}
}
@@ -371,170 +390,32 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
// (so the kernel readahead window widens) and the periodic
// DONTNEED page-cache eviction that bounds memory pressure
// when the mux output is being written to the same disk.
let mut reader = crate::io::file_sector_source::FileSectorSource::open(path)?;
let capacity = reader.capacity_sectors();
let mut disc = crate::disc::Disc::scan_image(
&mut reader,
capacity,
&crate::disc::ScanOptions::default(),
)
.map_err(|e| -> io::Error { e.into() })?;
// Apply the caller-resolved keys (lookup-free); decrypt_keys() then
// yields them for the stream below. Propagate a failed application
// rather than silently muxing an undecryptable stream.
if !opts.unit_keys.is_empty() {
// These UKs were already resolved AND validated by the caller
// (the CLI's keydb loop), so no re-validation sample is needed.
disc.decrypt_with(crate::disc::Key::Unit(opts.unit_keys.clone()), &[])
.map_err(|e| -> io::Error { e.into() })?;
}
// Pre-flight decrypt gate (the single, system-wide verdict — see
// `Disc::ensure_decryptable`). Fails fast BEFORE any mux work when
// decryption is needed and unavailable: a scrambled-but-uncracked
// CSS disc (`css_error` set), or an AACS-encrypted disc with no
// usable key (would mux ~100 MB of garbage — encrypted m2ts → no TS
// syncs → demuxer emits nothing → empty/garbage output at exit 0).
// `--raw` and unencrypted/CSS-keyless-success discs pass. This is the
// disc-wide check; the per-title (multi-VTS CSS) check is below, once
// the chosen title's key is resolved.
disc.ensure_decryptable(opts.raw)
.map_err(|e| -> io::Error { e.into() })?;
if disc.titles.is_empty() {
return Err(crate::error::Error::NoStreams.into());
}
let idx = opts.title_index.unwrap_or(0);
if idx >= disc.titles.len() {
return Err(crate::error::Error::DiscTitleRange {
index: idx,
count: disc.titles.len(),
}
.into());
}
// Prune to the selected audio/subtitle streams now, on the scanned
// (pre-`probe_and_remap`) title, so everything downstream — the
// TrueHD channel-correction probe, the final title clone, and
// `build_iso_pipeline`'s demux/track construction — sees the pruned
// list. Video is always kept; a no-op for the default All/All.
opts.selection
.apply(&mut disc.titles[idx])
.map_err(|e| -> io::Error { e.into() })?;
// Per-title key resolution. DVD CSS is resolved at exactly ONE site —
// `build_iso_pipeline`'s per-title crack (below), which decrypts a
// crackable title, passes a genuinely-clear one through, and
// hard-fails an uncrackable one with CssKeyMissing. So for a DVD we do
// NOT pre-crack here: pass `None` and let the pipeline own it.
// Pre-cracking would re-open the ISO and re-scan every clear title
// (`decrypt_keys_for_title` → None → the pipeline re-cracks anyway).
// AACS / unencrypted resolve from `decrypt_keys()` with NO read; `--raw`
// (any format) is deliberate ciphertext passthrough — also `None`.
let is_dvd = disc.format == crate::disc::DiscFormat::Dvd;
let (keys, title_is_clear) = if opts.raw || is_dvd {
(crate::decrypt::DecryptKeys::None, false)
} else {
(disc.decrypt_keys(), false)
};
// Decrypt gate for the AACS / non-DVD path: a None key means no usable
// disc key, which would mux scrambled ciphertext verbatim — fail loudly
// (NoDiscKey). The DVD path is gated inside `build_iso_pipeline` (its
// CSS hard-fail), and `--raw` passes.
if !is_dvd {
disc.ensure_title_decryptable(opts.raw, &keys, title_is_clear)
.map_err(|e| -> io::Error { e.into() })?;
}
// FMTS (AACS 2.1) forensic segments are sourced + fail-loud-checked
// downstream by `resolve_mux_key_map`/`resolve_fmts_key_map`, which hold
// the key-fetch closure and can actually attempt resolution. (An older
// upfront blanket-reject gate lived here; it predated the resolver and
// rejected every 2.1 disc before a source could be tried.)
// Correct TrueHD channel counts (MPLS understates 7.1/Atmos as 5.1)
// by probing the first DECRYPTED access units of the chosen title.
// A fresh reader avoids disturbing the mux reader below. Skipped in
// --raw mode: the probe would re-open + decrypt for nothing (on an
// AACS disc with no key the correction is a no-op on ciphertext, and
// raw output isn't decoded anyway).
if !opts.raw {
match crate::io::file_sector_source::FileSectorSource::open(path) {
Ok(mut probe) => {
// The probe DECRYPTS the title head, so it needs the SAME
// up-front key map the mux read installs below. An AACS
// `DecryptingSectorSource` with no map fails loud on the
// first unit (`decrypt_sectors_mapped` is the only AACS
// decrypt path) — without resolving one here the correction
// is silently skipped on every AACS disc and 7.1/Atmos stays
// understated as the MPLS-declared 5.1. Resolution failure is
// non-fatal (`.ok()`): leave channels uncorrected, never
// fail the mux.
let mut probe_keys = keys.clone();
let probe_title = disc.titles[idx].clone();
let probe_map = match &probe_keys {
crate::decrypt::DecryptKeys::Aacs { .. } => resolve_mux_key_map(
&mut probe,
&probe_title,
&mut probe_keys,
opts.key_fetch.as_ref(),
disc.content_format,
// File-backed, bounded probe (best-effort `.ok()`);
// no live drive to protect from a stuck stop here.
None,
)
.ok()
.map(std::sync::Arc::new),
_ => None,
};
let mut dec = crate::sector::DecryptingSectorSource::new(probe, probe_keys);
if let Some(map) = probe_map {
dec = dec.with_key_map(map);
}
crate::disc::correct_truehd_channels(&mut dec, &mut disc.titles[idx]);
}
Err(e) => {
// Non-fatal: a failed re-open just leaves MPLS 7.1/Atmos
// channel counts uncorrected (understated as 5.1). Log so
// the uncorrected path is diagnosable rather than silent.
tracing::debug!(
target: "mux",
"TrueHD channel-correction probe re-open failed: {e}"
);
}
}
}
let title = disc.titles[idx].clone();
let format = disc.content_format;
// ISO file: 8192-sector batch (16 MiB at 2048 B/sector) —
// sequential read from fast storage, no bad sectors. Empirically
// optimal; bumping to 16384 sectors (32 MiB) regressed (more cache
// pressure, longer per-batch latency starves the consumer between
// iterations). Physical drives keep smaller batches for adaptive
// error handling.
const ISO_MUX_BATCH_SECTORS: u16 = 8192;
// Pass `DecryptKeys::None` to the decrypt decorator when
// --raw is set — the read stack still flows through the
// same producer+demux+parse pipeline, just without the
// AACS / CSS step. Single highway for all ISO reads.
let effective_keys = if opts.raw {
crate::decrypt::DecryptKeys::None
} else {
keys
};
// Install the shared fetch closure (if the app supplied one) so a
// unit no held key decrypts is re-tried via the app's key source.
// Suppressed in --raw (no decrypt step to recover).
let fetch = if opts.raw {
None
} else {
opts.key_fetch.clone()
};
let stream = build_iso_pipeline(
let reader = crate::io::file_sector_source::FileSectorSource::open(path)?;
let probe_path = path.clone();
let stream = image_input(
reader,
title,
effective_keys,
ISO_MUX_BATCH_SECTORS,
format,
opts.raw,
None,
None,
fetch,
opts,
move || crate::io::file_sector_source::FileSectorSource::open(&probe_path).ok(),
false,
)?;
Ok(Box::new(stream))
}
// `dir://` as a SOURCE: an extracted disc folder, presented as a
// synthetic UDF image (`crate::dirimage`). It reaches EXACTLY the same
// body as `iso://` above — scan, title select, key resolution, mux —
// because by the time `DirImage` exists it is just another
// `SectorSource`. That is the point of synthesizing a real filesystem
// rather than faking a tree: every destination follows for free, with
// no per-scheme mux path to keep in step.
StreamUrl::Dir { ref path } => {
validate_file_path(path, "dir")?;
let reader = crate::dirimage::DirImage::open(path)?;
let probe_path = path.clone();
let stream = image_input(
reader,
opts,
move || crate::dirimage::DirImage::open(&probe_path).ok(),
true,
)?;
Ok(Box::new(stream))
}
@@ -556,9 +437,6 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
Ok(Box::new(NetworkStream::listen(addr)?))
}
StreamUrl::Stdio => Ok(Box::new(StdioStream::input())),
// `dir://` is an output-only sink (decrypted file tree); it is never a
// PES source. Mirror `null://` → write-only.
StreamUrl::Dir { .. } => Err(crate::error::Error::StreamWriteOnly.into()),
StreamUrl::Null => Err(crate::error::Error::StreamWriteOnly.into()),
// `mp4://` as a source: demux a progressive MP4 back into PES frames, so
// `mp4://` flows to every sink (mkv://, audio://, json://, …).
@@ -578,6 +456,202 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
}
}
/// Shared body of every IMAGE-level PES source.
///
/// `iso://` and `dir://` differ only in how the sectors are produced — a file
/// versus a synthesized UDF volume over a folder — and not at all in what is
/// done with them: scan, apply caller-resolved keys, gate on decryptability,
/// select the title, prune streams, correct TrueHD channel counts, mux. One
/// body means a `dir://` source cannot drift away from the `iso://` behaviour
/// that has years of fixes in it.
///
/// `reopen` yields a SECOND, independent reader for the TrueHD channel probe,
/// which must not disturb the mux reader's position. It returns `Option`
/// because a failed re-open is non-fatal: the correction is skipped, not the
/// mux.
fn image_input<S, F>(
mut reader: S,
opts: &InputOptions,
reopen: F,
// A FOLDER's `encrypted` flag comes from tree shape and can be wrong; an
// image's cannot. See `session::apply_folder_encryption_verdict`.
is_folder: bool,
) -> io::Result<PipelinedPesStream>
where
S: SectorSource + Send + 'static,
F: FnOnce() -> Option<S>,
{
let capacity = reader.capacity_sectors();
let mut disc =
crate::disc::Disc::scan_image(&mut reader, capacity, &crate::disc::ScanOptions::default())
.map_err(|e| -> io::Error { e.into() })?;
// Without this a folder reached here with a tree-shape verdict while
// `session::scan_dir` reached the opposite one from its CONTENT, so the
// same folder ripped through one door and failed through the other.
if is_folder {
crate::session::apply_folder_encryption_verdict(&mut reader, &mut disc)
.map_err(|e| -> io::Error { e.into() })?;
}
// Apply the caller-resolved keys (lookup-free); decrypt_keys() then
// yields them for the stream below. Propagate a failed application
// rather than silently muxing an undecryptable stream.
if !opts.unit_keys.is_empty() {
// These UKs were already resolved AND validated by the caller
// (the CLI's keydb loop), so no re-validation sample is needed.
disc.decrypt_with(crate::disc::Key::Unit(opts.unit_keys.clone()), &[])
.map_err(|e| -> io::Error { e.into() })?;
}
// Pre-flight decrypt gate (the single, system-wide verdict — see
// `Disc::ensure_decryptable`). Fails fast BEFORE any mux work when
// decryption is needed and unavailable: a scrambled-but-uncracked
// CSS disc (`css_error` set), or an AACS-encrypted disc with no
// usable key (would mux ~100 MB of garbage — encrypted m2ts → no TS
// syncs → demuxer emits nothing → empty/garbage output at exit 0).
// `--raw` and unencrypted/CSS-keyless-success discs pass. This is the
// disc-wide check; the per-title (multi-VTS CSS) check is below, once
// the chosen title's key is resolved.
disc.ensure_decryptable(opts.raw)
.map_err(|e| -> io::Error { e.into() })?;
if disc.titles.is_empty() {
return Err(crate::error::Error::NoStreams.into());
}
let idx = opts.title_index.unwrap_or(0);
if idx >= disc.titles.len() {
return Err(crate::error::Error::DiscTitleRange {
index: idx,
count: disc.titles.len(),
}
.into());
}
// Prune to the selected audio/subtitle streams now, on the scanned
// (pre-`probe_and_remap`) title, so everything downstream — the
// TrueHD channel-correction probe, the final title clone, and
// `build_iso_pipeline`'s demux/track construction — sees the pruned
// list. Video is always kept; a no-op for the default All/All.
opts.selection
.apply(&mut disc.titles[idx])
.map_err(|e| -> io::Error { e.into() })?;
// Per-title key resolution. DVD CSS is resolved at exactly ONE site —
// `build_iso_pipeline`'s per-title crack (below), which decrypts a
// crackable title, passes a genuinely-clear one through, and
// hard-fails an uncrackable one with CssKeyMissing. So for a DVD we do
// NOT pre-crack here: pass `None` and let the pipeline own it.
// Pre-cracking would re-open the ISO and re-scan every clear title
// (`decrypt_keys_for_title` → None → the pipeline re-cracks anyway).
// AACS / unencrypted resolve from `decrypt_keys()` with NO read; `--raw`
// (any format) is deliberate ciphertext passthrough — also `None`.
let is_dvd = disc.format == crate::disc::DiscFormat::Dvd;
let (keys, title_is_clear) = if opts.raw || is_dvd {
(crate::decrypt::DecryptKeys::None, false)
} else {
(disc.decrypt_keys(), false)
};
// Decrypt gate for the AACS / non-DVD path: a None key means no usable
// disc key, which would mux scrambled ciphertext verbatim — fail loudly
// (NoDiscKey). The DVD path is gated inside `build_iso_pipeline` (its
// CSS hard-fail), and `--raw` passes.
if !is_dvd {
disc.ensure_title_decryptable(opts.raw, &keys, title_is_clear)
.map_err(|e| -> io::Error { e.into() })?;
}
// FMTS (AACS 2.1) forensic segments are sourced + fail-loud-checked
// downstream by `resolve_mux_key_map`/`resolve_fmts_key_map`, which hold
// the key-fetch closure and can actually attempt resolution. (An older
// upfront blanket-reject gate lived here; it predated the resolver and
// rejected every 2.1 disc before a source could be tried.)
// Correct TrueHD channel counts (MPLS understates 7.1/Atmos as 5.1)
// by probing the first DECRYPTED access units of the chosen title.
// A fresh reader avoids disturbing the mux reader below. Skipped in
// --raw mode: the probe would re-open + decrypt for nothing (on an
// AACS disc with no key the correction is a no-op on ciphertext, and
// raw output isn't decoded anyway).
if !opts.raw {
match reopen() {
Some(mut probe) => {
// The probe DECRYPTS the title head, so it needs the SAME
// up-front key map the mux read installs below. An AACS
// `DecryptingSectorSource` with no map fails loud on the
// first unit (`decrypt_sectors_mapped` is the only AACS
// decrypt path) — without resolving one here the correction
// is silently skipped on every AACS disc and 7.1/Atmos stays
// understated as the MPLS-declared 5.1. Resolution failure is
// non-fatal (`.ok()`): leave channels uncorrected, never
// fail the mux.
let mut probe_keys = keys.clone();
let probe_title = disc.titles[idx].clone();
let probe_map = match &probe_keys {
crate::decrypt::DecryptKeys::Aacs { .. } => resolve_mux_key_map(
&mut probe,
&probe_title,
&mut probe_keys,
opts.key_fetch.as_ref(),
disc.content_format,
// File-backed, bounded probe (best-effort `.ok()`);
// no live drive to protect from a stuck stop here.
None,
)
.ok()
.map(std::sync::Arc::new),
_ => None,
};
let mut dec = crate::sector::DecryptingSectorSource::new(probe, probe_keys);
if let Some(map) = probe_map {
dec = dec.with_key_map(map);
}
crate::disc::correct_truehd_channels(&mut dec, &mut disc.titles[idx]);
}
None => {
// Non-fatal: a failed re-open just leaves MPLS 7.1/Atmos
// channel counts uncorrected (understated as 5.1). Log so
// the uncorrected path is diagnosable rather than silent.
tracing::debug!(
target: "mux",
"TrueHD channel-correction probe re-open failed"
);
}
}
}
let title = disc.titles[idx].clone();
let format = disc.content_format;
// ISO file: 8192-sector batch (16 MiB at 2048 B/sector) —
// sequential read from fast storage, no bad sectors. Empirically
// optimal; bumping to 16384 sectors (32 MiB) regressed (more cache
// pressure, longer per-batch latency starves the consumer between
// iterations). Physical drives keep smaller batches for adaptive
// error handling.
const ISO_MUX_BATCH_SECTORS: u16 = 8192;
// Pass `DecryptKeys::None` to the decrypt decorator when
// --raw is set — the read stack still flows through the
// same producer+demux+parse pipeline, just without the
// AACS / CSS step. Single highway for all ISO reads.
let effective_keys = if opts.raw {
crate::decrypt::DecryptKeys::None
} else {
keys
};
// Install the shared fetch closure (if the app supplied one) so a
// unit no held key decrypts is re-tried via the app's key source.
// Suppressed in --raw (no decrypt step to recover).
let fetch = if opts.raw {
None
} else {
opts.key_fetch.clone()
};
let stream = build_iso_pipeline(
reader,
title,
effective_keys,
ISO_MUX_BATCH_SECTORS,
format,
opts.raw,
None,
None,
fetch,
)?;
Ok(stream)
}
/// Open a PES output stream (consumes PES frames).
///
/// `source` is the provenance of the material being written — the INPUT the
@@ -1447,8 +1521,11 @@ fn forensic_clip_extents(
tracing::warn!(target: "freemkv::keysource", "fmts: more than one forensic clip on the disc — segment byte space is ambiguous");
return Ok(None);
}
// Addressing variant: these extents are a byte-space map for the forensic
// segment table (`clip_byte_to_lba`), not a read plan — an unrecorded
// extent must stay in place here or every later segment offset shifts.
let exts: Vec<crate::disc::Extent> = udf
.file_extents(reader, &format!("/BDMV/STREAM/{name}"))
.file_extents_addressing(reader, &format!("/BDMV/STREAM/{name}"))
.map_err(io::Error::from)?
.into_iter()
.filter(|&(lba, sectors)| lba > 0 && sectors > 0)
@@ -2106,10 +2183,25 @@ pub fn build_iso_pipeline<S: SectorSource + Send + 'static>(
// decrypted, or handed to the demux, so the demux sees one gapless our-variant
// stream (no ciphertext to trip a concealed-gap resync). A non-forensic map
// returns the extents unchanged, so the common disc reads exactly as before.
let full_extents = extents.clone();
let extents = match &key_map {
Some(map) => map.read_plan(&extents, unit_align as u32),
None => extents,
};
// The plan and the clips' feed spans must describe the SAME bytes.
//
// A clip's span was measured over the title's full extents at scan time,
// and a frame is placed by the offset it was read from. When a forensic
// segment makes the plan drop alternate-phase units, the mux feeds fewer
// bytes than the spans describe and the two drift apart cumulatively —
// every frame after the first segment looks earlier than it is, and near a
// join it is placed in the wrong clip or dropped. The spans still tile each
// other perfectly, so the trust check cannot see it.
//
// Provenance is only meaningful when the feed matches. When it does not,
// say so and let placement fall back to timestamps, which is what the
// untrusted path exists for.
let feed_matches_spans = extents == full_extents;
let mut decrypting =
crate::sector::DecryptingSectorSource::new(Box::new(reader) as Box<dyn SectorSource>, keys);
if let Some(map) = key_map {
@@ -2128,6 +2220,17 @@ pub fn build_iso_pipeline<S: SectorSource + Send + 'static>(
// correct `0x8x` sub-streams. No-op for non-DVD or an empty probe. Reset the
// unit base afterward so the prefetcher's first batch starts clean.
let mut title = title;
if !feed_matches_spans {
tracing::info!(
target: "freemkv::mux",
planned = extents.len(),
full = full_extents.len(),
"read plan omits units the clip spans include; placing by timestamps"
);
for c in &mut title.clips {
c.feed_span = None;
}
}
crate::disc::dvd_audio_probe::probe_and_remap(&mut decrypting, &mut title);
decrypting.set_unit_base(0);
@@ -2547,9 +2650,12 @@ mod tests {
}
/// `dir://PATH/` parses to `StreamUrl::Dir` with the raw remainder as the
/// path; it is a SINK (not a disc source), so `is_disc_source()` is false.
/// path. Unlike the other directory schemes it IS an image-level source
/// (1.6.1): `crate::dirimage` synthesizes a UDF volume over the folder, so
/// `is_disc_source()` — "has a filesystem to scan" — is true for it and
/// false for the write-only `demux://` / `fvi://` directory sinks.
#[test]
fn parse_dir_url_is_sink_not_disc_source() {
fn parse_dir_url_is_an_image_source_unlike_the_directory_sinks() {
match parse_url("dir://out/movie/") {
StreamUrl::Dir { path } => {
assert_eq!(path, PathBuf::from("out/movie/"));
@@ -2565,8 +2671,8 @@ mod tests {
"demux:// is a sink, never a disc source"
);
assert!(
!parse_url("dir://x").is_disc_source(),
"dir:// is a sink, never a disc source"
parse_url("dir://x").is_disc_source(),
"dir:// carries a filesystem, so selection flags and image sinks apply"
);
// fvi:// parses to Fvi with the raw remainder as the path, and is a
// sink (never a disc source) — parallel to the demux:// coverage above.
@@ -2594,15 +2700,19 @@ mod tests {
);
}
/// `dir://` is output-only: `input()` rejects it (StreamWriteOnly →
/// Unsupported), and `output()` rejects it too (StreamReadOnly →
/// Unsupported) because it is NOT a PES sink — the CLI routes it to
/// `Disc::extract_tree` before the mux path.
/// `dir://` is never a PES SINK — it writes raw decrypted files, not muxed
/// frames, so `output()` still rejects it (StreamReadOnly → Unsupported)
/// and the CLI routes a `dir://` dest to `Disc::extract_tree`.
///
/// As a SOURCE it is no longer rejected out of hand: it is an image source,
/// and a missing folder now fails as a missing folder (NotFound) rather
/// than as "this scheme cannot be read".
#[test]
fn dir_url_is_not_a_pes_stream_either_direction() {
fn dir_url_is_an_input_but_never_a_pes_sink() {
assert_eq!(
input_err_kind("dir://out/"),
std::io::ErrorKind::Unsupported
input_err_kind("dir://definitely/not/here/"),
std::io::ErrorKind::NotFound,
"a dir:// source that does not exist must report a missing path"
);
let t = DiscTitle::empty();
assert_eq!(
@@ -4333,7 +4443,7 @@ mod tests {
"00001.fmts",
20,
FMTS_CONTENT_LBA - PART_START,
FMTS_CONTENT_SECTORS * 2048,
u64::from(FMTS_CONTENT_SECTORS) * 2048,
true,
)],
subdirs: Vec::new(),
+2218 -31
View File
File diff suppressed because it is too large Load Diff
+3
View File
@@ -682,6 +682,9 @@ mod tests {
for (i, b) in template.iter_mut().enumerate() {
*b = (i as u8).wrapping_mul(29).wrapping_add(3);
}
// Real scrambled DVD sectors are MPEG-2 PS packs; the scramble policy
// requires the pack start code as well as the flag bits.
template[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
template[0x14] = 0x30; // scramble bits (4-5) set → flags == 0x03
let pristine = template;
+200 -9
View File
@@ -85,6 +85,12 @@ pub struct PrefetchedSectorSource {
///
/// [`capacity_sectors`]: SectorSource::capacity_sectors
total_sectors: u32,
/// Latched the moment a terminal error crosses the channel. The
/// producer NEVER resumes after sending one (every error arm
/// `return`s), so the closed channel that follows is a dead source,
/// not end-of-stream — and `read_sectors` must keep saying so instead
/// of answering `Ok(0)` for the rest of the title.
producer_failed: bool,
}
impl PrefetchedSectorSource {
@@ -300,6 +306,22 @@ impl PrefetchedSectorSource {
return;
}
let sectors_read = (n / 2048) as u32;
// A genuine zero-byte read with no error
// would otherwise spin this loop forever.
// It is not end-of-stream either: the
// extent list still has `remaining`
// sectors to serve, so the inner source
// has quit early. Send a terminal
// sentinel — dropping `tx` here instead
// would reach the consumer as a clean EOF
// and finalize a TRUNCATED title as
// success, exactly as the panic sentinel
// below exists to prevent.
if sectors_read == 0 {
let _ =
tx.send(Err(crate::error::Error::SourceTerminated.into()));
return;
}
buf.truncate(n);
bytes_read_total = bytes_read_total.saturating_add(n as u64);
if let Some(ref f) = event_fn {
@@ -313,12 +335,6 @@ impl PrefetchedSectorSource {
if tx.send(Ok(buf)).is_err() {
return; // consumer dropped
}
// A genuine zero-byte read with no error would
// otherwise spin this loop forever; treat it
// as end-of-source.
if sectors_read == 0 {
return;
}
offset = offset.saturating_add(sectors_read);
}
Err(e) => {
@@ -344,6 +360,7 @@ impl PrefetchedSectorSource {
recycle_tx,
producer: Some(producer),
total_sectors,
producer_failed: false,
})
}
@@ -486,8 +503,29 @@ impl SectorSource for PrefetchedSectorSource {
let _ = self.recycle_tx.send(filled);
Ok(n)
}
Ok(Err(e)) => Err(crate::error::Error::IoError { source: e }),
// Channel closed (producer finished or panicked).
// Recover the producer's TYPED error rather than blanket-wrapping
// it as `Error::IoError`. That wrapper is a classification, not a
// container: `is_scsi_transport_failure` matches `IoError`, so a
// wrapped MEDIUM ERROR bad sector reached `fill_extents` looking
// like a wedged bridge and aborted the pass instead of being
// skipped under `skip_errors`. `From<io::Error> for Error`
// downcasts the boxed payload back to the exact variant the
// producer sent (status + sense intact); a genuine OS-level
// `io::Error` — the real dead-bus case — still becomes `IoError`.
Ok(Err(e)) => {
// The producer `return`s after every error it sends, so
// this is also the moment the source dies. Latch it: the
// closed channel that follows must not read as EOF.
self.producer_failed = true;
Err(crate::error::Error::from(e))
}
// Channel closed. Clean EOF only if the producer never
// signalled a failure — otherwise it exited without
// delivering the rest of the extents, and answering `Ok(0)`
// would let `fill_extents` mistake a dead source for a short
// read, zero-fill every remaining sector of the title and
// still report the pass as complete.
Err(_) if self.producer_failed => Err(crate::error::Error::SourceTerminated),
Err(_) => Ok(0),
}
}
@@ -1214,7 +1252,12 @@ mod tests {
/// an error (not Ok(0)/EOF), and its ErrorKind must survive the
/// round-trip through the channel. Grounding: the producer's
/// `Err(e) => tx.send(Err(e.into()))` arm, and `read_sectors`'
/// `Ok(Err(e)) => Err(IoError{source:e})`.
/// `Ok(Err(e)) => { self.producer_failed = true; Err(Error::from(e)) }`.
///
/// That arm recovers the producer's TYPED error by downcast rather than
/// blanket-wrapping it as `Error::IoError`, so the kind survives; the
/// `producer_failed` latch it also sets is what turns the channel close
/// that follows into `SourceTerminated` instead of a clean EOF.
#[test]
fn reader_error_propagates_with_kind() {
with_watchdog(Duration::from_secs(10), || {
@@ -1235,6 +1278,63 @@ mod tests {
});
}
/// An inner source that answers a mid-extent read with `Ok(0)` has quit
/// early: the extent list still has sectors to serve. The producer must
/// say so, not simply drop `tx` — a closed channel reads as clean
/// end-of-stream, and `DiscStream::fill_extents` then fabricates zeros
/// for every remaining sector of the title and reports the pass
/// complete. Same rule as the panic sentinel: a truncated title must
/// never be finalized as success.
struct QuitsEarlySource;
impl SectorSource for QuitsEarlySource {
fn read_sectors(
&mut self,
_lba: u32,
_count: u16,
_buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
Ok(0)
}
fn capacity_sectors(&self) -> u32 {
9
}
}
#[test]
fn inner_source_quitting_early_is_not_reported_as_end_of_stream() {
with_watchdog(Duration::from_secs(10), || {
let extents = vec![Extent {
start_lba: 0,
sector_count: 9,
}];
let mut pf =
PrefetchedSectorSource::new(QuitsEarlySource, extents, 3, None).expect("spawn");
let mut buf = vec![0u8; 3 * 2048];
let mut last = pf.read_sectors(0, 3, &mut buf, false);
// Whatever the first answer, no call may ever settle on a clean
// `Ok(0)`: 9 sectors were promised and none were delivered.
for _ in 0..4 {
if last.is_err() {
break;
}
assert_eq!(
*last.as_ref().unwrap(),
0,
"the source delivered no bytes, so nothing can be Ok(n>0)"
);
last = pf.read_sectors(0, 3, &mut buf, false);
}
let err =
last.expect_err("an undelivered extent list must surface as an error, not as EOF");
assert!(
err.is_source_terminated(),
"the source is gone for good — retrying or skipping cannot \
recover anything; got {err:?}"
);
});
}
/// A read returning a byte count that is not a whole number of
/// sectors (n % 2048 != 0) must be rejected — never truncated and
/// advanced, which would split a sector and hand decrypt a partial
@@ -1472,4 +1572,95 @@ mod tests {
assert_eq!(got.len(), 30 * 2048, "all 10 extents must be drained");
});
}
/// Source whose every read fails with a `DiscRead` carrying the given
/// SCSI status (and optional sense) — an ordinary MEDIUM ERROR bad
/// sector (0x02 + 03/11/00) or the transport-failure sentinel (0xFF).
struct FailingSource {
status: u8,
sense: Option<crate::scsi::ScsiSense>,
}
impl SectorSource for FailingSource {
fn capacity_sectors(&self) -> u32 {
9999
}
fn read_sectors(
&mut self,
lba: u32,
_count: u16,
_buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
Err(crate::error::Error::DiscRead {
sector: lba as u64,
status: Some(self.status),
sense: self.sense,
})
}
}
fn read_one_err(status: u8, sense: Option<crate::scsi::ScsiSense>) -> crate::error::Error {
let extents = vec![Extent {
start_lba: 100,
sector_count: 9,
}];
let mut pf = PrefetchedSectorSource::new(FailingSource { status, sense }, extents, 3, None)
.expect("spawn");
let mut buf = vec![0u8; 3 * 2048];
pf.read_sectors(100, 3, &mut buf, false)
.expect_err("the producer's read failure must surface")
}
/// REGRESSION: an ordinary MEDIUM ERROR bad sector that crosses the
/// prefetch channel must NOT be classified as a SCSI transport failure.
///
/// `read_sectors` used to re-wrap EVERY channel error as
/// `Error::IoError { source }`, and `is_scsi_transport_failure` matches
/// `IoError` (the wedged-USB-bridge arm). So a skippable bad sector
/// arrived at `DiscStream::fill_extents` looking like a dead bus and
/// aborted the whole pass instead of honouring `skip_errors`.
#[test]
fn bad_sector_across_channel_is_not_a_transport_failure() {
with_watchdog(Duration::from_secs(10), || {
let sense = crate::scsi::ScsiSense {
sense_key: 0x03,
asc: 0x11,
ascq: 0x00,
};
let err = read_one_err(crate::scsi::SCSI_STATUS_CHECK_CONDITION, Some(sense));
assert!(
!err.is_scsi_transport_failure(),
"a MEDIUM ERROR bad sector must stay a bad sector across the \
prefetch channel, got {err:?}"
);
// The classification survives because the typed variant does.
assert!(
matches!(err, crate::error::Error::DiscRead { status: Some(s), .. } if s == 0x02),
"expected the producer's DiscRead to survive the channel, got {err:?}"
);
assert_eq!(
err.scsi_sense().map(|s| (s.sense_key, s.asc, s.ascq)),
Some((0x03, 0x11, 0x00)),
"the drive's sense triple must survive the channel"
);
});
}
/// OPPOSITE-DIRECTION CONTROL: a genuine transport failure (status 0xFF,
/// wedged USB bridge) crossing the same channel MUST still classify as a
/// transport failure, so `fill_extents` / sweep keep aborting the pass
/// instead of zero-filling every read against a dead bus.
#[test]
fn transport_failure_across_channel_still_classifies() {
with_watchdog(Duration::from_secs(10), || {
let err = read_one_err(crate::scsi::SCSI_STATUS_TRANSPORT_FAILURE, None);
assert!(
err.is_scsi_transport_failure(),
"a 0xFF transport failure must remain one across the prefetch \
channel, got {err:?}"
);
});
}
}
+152
View File
@@ -439,6 +439,158 @@ pub fn scan_iso(path: &Path, opts: ScanOptions) -> Result<(Disc, Box<dyn SectorS
Ok((disc, Box::new(reader)))
}
/// Sampled 6144-byte aligned units when deciding whether a folder that still
/// carries `AACS/` actually holds encrypted content. Enough to survive a clip
/// whose opening units happen to be unflagged (a clear leader), few enough to
/// stay a handful of reads.
const AACS_PROBE_UNITS: usize = 8;
/// [`Disc`] together with a [`SectorSource`] over a synthesized image of an
/// extracted disc FOLDER — the `dir://` counterpart to [`scan_iso`].
///
/// The extra step over `scan_iso` is the encryption verdict.
/// `Disc::scan_with` decides `encrypted` STRUCTURALLY, from the presence of an
/// `/AACS` or `/BDMV/AACS` directory (see `disc::aacs_dir_present`). For the common
/// case — a MakeMKV-style backup, which strips `AACS/` — that already gives the
/// right answer, and `DecryptKeys::None` is a pass-through. But a folder copied
/// verbatim from a decrypted disc keeps `AACS/`, and the tree shape then claims
/// encryption over content that is already in the clear: the rip would fail
/// asking for a key it does not need.
///
/// So for a folder, tree shape is not the evidence — CONTENT is. Several
/// aligned units at the largest title's start are sampled and judged by
/// `aacs_unit_needs_decrypt`, the same authority the mux read path uses:
///
/// * none need decryption → the folder is decrypted; `encrypted` is forced
/// false and the reason is logged.
/// * any unit does → the folder is a raw encrypted copy, which `dir://` does
/// not support; [`Error::DirImageEncrypted`].
///
/// This lives HERE and not in `Disc::scan_image`, which is shared with the ISO
/// and drive paths: an ISO that carries `AACS/` and clear content is a
/// different situation (it may be mid-decrypt, or `--raw` output), and the
/// verdict must not change underneath those callers.
pub fn scan_dir(path: &Path, opts: ScanOptions) -> Result<(Disc, Box<dyn SectorSource>)> {
let mut reader = crate::dirimage::DirImage::open(path)?;
let capacity = reader.capacity_sectors();
let mut disc = Disc::scan_image(&mut reader, capacity, &opts)?;
apply_folder_encryption_verdict(&mut reader, &mut disc)?;
Ok((disc, Box::new(reader)))
}
/// Re-judge a FOLDER's encryption verdict from its CONTENT.
///
/// `Disc::scan_with` decides `encrypted` from tree shape — whether an `AACS/`
/// directory is present. That is right for an image and wrong for a verbatim
/// copy of an already-decrypted disc that kept the directory: the rip would
/// fail asking for a key it does not need.
///
/// Shared by [`scan_dir`] and by the `dir://` PES input path in
/// `mux::resolve`. It lives in one place because the two disagreed: a folder
/// that ripped through `scan_dir` failed through `input()`, which is the exact
/// failure this probe was written to prevent, reachable by the other door.
pub(crate) fn apply_folder_encryption_verdict(
reader: &mut dyn SectorSource,
disc: &mut Disc,
) -> Result<()> {
// `css.is_some()` is the DVD path, and that verdict came from actually
// cracking scrambled sectors — real evidence about content, not tree shape.
// Only the AACS-by-tree-shape verdict is re-judged here.
if disc.encrypted && disc.css.is_none() && disc.css_error.is_none() {
match probe_folder_encryption(reader, disc)? {
true => return Err(Error::DirImageEncrypted),
false => {
tracing::warn!(
target: "freemkv::scan",
phase = "folder_verdict",
"folder carries an AACS directory but its sampled content units \
are already in the clear; treating it as decrypted"
);
disc.encrypted = false;
disc.aacs = None;
disc.aacs_error = None;
}
}
}
Ok(())
}
/// `true` when any sampled content unit still needs decryption.
///
/// Anchored at the largest title's first extent, because AACS unit alignment
/// is measured from the clip FILE's start (`aacs::content::is_unit_aligned`),
/// not from an absolute `lba % 3` — sampling off a boundary would mis-judge a
/// perfectly clear unit.
fn probe_folder_encryption(reader: &mut dyn SectorSource, disc: &Disc) -> Result<bool> {
use crate::aacs::content::{aacs_unit_needs_decrypt, is_unit_aligned};
use crate::consts::SECTOR_BYTES;
const UNIT_SECTORS: u32 = 3;
// Anchor on the largest TITLE's FIRST extent, not on the largest extent
// anywhere.
//
// AACS units are 3 sectors, and a unit boundary is only guaranteed at the
// START of a clip. `max_by_key` over every extent picked a mid-file one for
// any clip big enough to be split: the planner caps an allocation
// descriptor at MAX_AD_BYTES = 524287 sectors, every full piece of a split
// file therefore ties on sector_count, and `max_by_key` returns the LAST
// tie — an extent starting (k-1)*524287 sectors in. 524287 % 3 == 1, so
// that start is off the unit boundary for two file sizes in three.
//
// The sampling then reads 6144-byte windows that begin mid-source-packet,
// so the CPI byte it thinks it is testing is content. Both verdicts are
// wrong in a costly direction: a decrypted folder gets rejected as
// encrypted (DirImageEncrypted on something perfectly rippable), or
// genuine ciphertext reads as clear and the mux writes it out as video at
// exit 0. `is_unit_aligned` cannot catch it, because it measures against
// this same wrong base.
let Some(extent) = disc
.titles
.iter()
.max_by_key(|t| {
t.extents
.iter()
.fold(0u64, |a, e| a.saturating_add(e.sector_count as u64))
})
.and_then(|t| t.extents.first())
else {
// No content to judge. A folder with an AACS directory and no titles
// has nothing to rip either way; leave the structural verdict alone.
return Ok(true);
};
let base = extent.start_lba;
let mut unit = vec![0u8; UNIT_SECTORS as usize * SECTOR_BYTES];
let mut sampled = 0u32;
for i in 0..AACS_PROBE_UNITS as u32 {
// Saturating: `start_lba` and `sector_count` come off the medium, and a
// crafted or corrupt extent must not wrap this bound into a read past
// the end of the content.
let Some(lba) = base.checked_add(i.saturating_mul(UNIT_SECTORS)) else {
break;
};
let end = base.saturating_add(extent.sector_count);
if lba.saturating_add(UNIT_SECTORS) > end {
break;
}
debug_assert!(is_unit_aligned(lba, base));
reader.read_sectors(lba, UNIT_SECTORS as u16, &mut unit, false)?;
sampled += 1;
if aacs_unit_needs_decrypt(&unit, disc.content_format) {
return Ok(true);
}
}
// Nothing was actually sampled — the largest title is shorter than one
// aligned unit, so there is no evidence either way. "Not encrypted" is the
// dangerous default here: it would clear the structural verdict an `AACS`
// directory raised and rip ciphertext as though it were video, at exit 0.
// With no evidence, keep the structural verdict.
if sampled == 0 {
return Ok(true);
}
Ok(false)
}
#[cfg(test)]
mod tests {
use super::*;
+226
View File
@@ -0,0 +1,226 @@
//! Test-only capture of `tracing` events, so the crate's logging contract is
//! ENFORCED rather than merely commented.
//!
//! # Why this exists
//!
//! The error contract is "Account / Log / Classify", and the round-2 audit
//! found the third leg unverifiable: three separate sites carry long comments
//! insisting they log **the error's OWN code, not a fixed one** — because
//! flattening a scratched sector (E6000) or an over-long allocation-descriptor
//! chain (E6016) into E6017 sends whoever triages them after authoring holes
//! and hides the population that actually exists. Nothing tested that. Putting
//! a literal back at `bluray.rs`'s or `hddvd.rs`'s warn sites broke no test, so
//! the guarantee was a convention one careless edit away from being false. Two
//! of those very sites were changed in round 1, and a third still carried a
//! hardcoded `code = 6017`.
//!
//! Likewise "absence of a log is itself a bug": a refusal that returns the
//! right error but says nothing produces the wrong population downstream (a
//! residual-underrunning drive is indistinguishable from a scratched disc).
//! That is only checkable by looking at what was emitted.
//!
//! # Why a hand-rolled subscriber and not `tracing-subscriber`
//!
//! Same posture as [`crate::harness`]: this crate has exactly one
//! dev-dependency on purpose. `tracing-subscriber` would pull a tree of them to
//! do what forty lines of [`tracing::Subscriber`] does here. The capture is
//! installed with [`tracing::subscriber::with_default`], which is
//! THREAD-LOCAL — so it composes with `cargo test`'s parallel harness and two
//! capturing tests cannot see each other's events.
//!
//! Field values are stringified through [`std::fmt::Debug`]/`Display` because
//! that is all the visitor API offers without a typed schema; tests compare
//! against the string form of the expected constant, which is exactly the
//! comparison that catches a hardcoded code.
#![cfg(test)]
use std::sync::{Arc, Mutex};
/// One captured `tracing` event: its target, level, message and fields.
#[derive(Debug, Clone)]
pub(crate) struct CapturedEvent {
pub target: String,
pub level: tracing::Level,
/// Every field, in emission order, stringified. The implicit `message`
/// field (the format string) is included under the name `message`.
pub fields: Vec<(String, String)>,
}
impl CapturedEvent {
/// The stringified value of `name`, or `None` if the event has no such
/// field.
pub fn field(&self, name: &str) -> Option<&str> {
self.fields
.iter()
.find(|(k, _)| k == name)
.map(|(_, v)| v.as_str())
}
/// The event's message (the `tracing` format string), or `""`.
pub fn message(&self) -> &str {
self.field("message").unwrap_or("")
}
}
#[derive(Default)]
struct Visitor(Vec<(String, String)>);
impl tracing::field::Visit for Visitor {
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
self.0
.push((field.name().to_string(), format!("{value:?}")));
}
fn record_str(&mut self, field: &tracing::field::Field, value: &str) {
self.0.push((field.name().to_string(), value.to_string()));
}
fn record_u64(&mut self, field: &tracing::field::Field, value: u64) {
self.0.push((field.name().to_string(), value.to_string()));
}
fn record_i64(&mut self, field: &tracing::field::Field, value: i64) {
self.0.push((field.name().to_string(), value.to_string()));
}
fn record_bool(&mut self, field: &tracing::field::Field, value: bool) {
self.0.push((field.name().to_string(), value.to_string()));
}
}
type Sink = Arc<Mutex<Vec<CapturedEvent>>>;
thread_local! {
/// The sink for a capture ACTIVE ON THIS THREAD, if any. Thread-local so
/// concurrent captures across `cargo test`'s parallel harness never see
/// each other's events, and so a non-capturing thread simply has `None`.
static SINK: std::cell::RefCell<Option<Sink>> = const { std::cell::RefCell::new(None) };
}
/// The ONE process-wide subscriber. Installed once and left installed; it is
/// offered every event and records into whichever thread's sink is active,
/// dropping the event when none is.
struct Capture;
impl tracing::Subscriber for Capture {
// `sometimes`, deliberately NOT the default `always`/`never`. A cacheable
// interest lets `tracing`'s GLOBAL per-callsite cache short-circuit a
// callsite to "off", and under `cargo test`'s parallel harness a cache
// rebuild triggered by ANY other thread can leave it there while a capture
// is live — the capture then observes NOTHING. That is the race that made
// `parse_playlist_unreadable_clip_icb_yields_no_title` flake ~1 run in 15.
// `sometimes` forces `enabled` to be consulted on the emitting thread for
// every event, so a capture always sees its own.
fn register_callsite(
&self,
_meta: &'static tracing::Metadata<'static>,
) -> tracing::subscriber::Interest {
tracing::subscriber::Interest::sometimes()
}
fn enabled(&self, _metadata: &tracing::Metadata<'_>) -> bool {
SINK.with(|s| s.borrow().is_some())
}
// Spans are irrelevant here — nothing in this crate asserts on span
// structure, only on events — so they get a constant id and no storage.
fn new_span(&self, _span: &tracing::span::Attributes<'_>) -> tracing::span::Id {
tracing::span::Id::from_u64(1)
}
fn record(&self, _span: &tracing::span::Id, _values: &tracing::span::Record<'_>) {}
fn record_follows_from(&self, _span: &tracing::span::Id, _follows: &tracing::span::Id) {}
fn event(&self, event: &tracing::Event<'_>) {
SINK.with(|s| {
if let Some(sink) = s.borrow().as_ref() {
let mut v = Visitor::default();
event.record(&mut v);
let meta = event.metadata();
sink.lock().expect("capture mutex").push(CapturedEvent {
target: meta.target().to_string(),
level: *meta.level(),
fields: v.0,
});
}
});
}
fn enter(&self, _span: &tracing::span::Id) {}
fn exit(&self, _span: &tracing::span::Id) {}
}
/// Install the single global capturing subscriber, exactly once.
fn install() {
static INSTALLED: std::sync::OnceLock<()> = std::sync::OnceLock::new();
INSTALLED.get_or_init(|| {
// This is the crate's ONLY `set_global_default`; a second call cannot
// happen. Tolerate it via `.ok()` rather than panic if that ever
// changes — capture then no-ops, which the non-empty assertions catch.
let _ = tracing::subscriber::set_global_default(Capture);
});
}
/// Run `f` with every `tracing` event it emits ON THIS THREAD captured.
///
/// Returns `f`'s value alongside the events, in emission order.
///
/// # Why one global subscriber, not scoped `with_default`
///
/// `tracing`'s per-callsite INTEREST CACHE is GLOBAL, but `with_default` is
/// thread-local. Under `cargo test`'s parallel harness a rebuild of that cache
/// — triggered by ANY other thread registering any callsite for the first time
/// — re-evaluates the target callsite against the global dispatcher, which a
/// scoped subscriber is NOT part of, and can leave it cached "off" while a
/// capture is live, so the capture observes NOTHING. Serialising captures
/// against each other does not help: the poisoning thread is not itself
/// capturing. `parse_playlist_unreadable_clip_icb_yields_no_title` flaked ~1
/// run in 15 on exactly this — an empty event list.
///
/// The robust shape is a single subscriber installed globally for the whole
/// run, whose `register_callsite` returns `sometimes` (so no callsite is ever
/// hard-cached) and which routes each event to the emitting thread's own sink.
/// No scoped-dispatcher transitions, no cross-thread cache race, and concurrent
/// captures on different threads stay isolated by the thread-local sink.
pub(crate) fn capture<T>(f: impl FnOnce() -> T) -> (T, Vec<CapturedEvent>) {
install();
let sink: Sink = Arc::default();
// Save/restore any outer sink so a nested capture on one thread still works.
let prev = SINK.with(|s| s.borrow_mut().replace(sink.clone()));
let out = f();
SINK.with(|s| *s.borrow_mut() = prev);
let events = std::mem::take(&mut *sink.lock().expect("capture mutex"));
(out, events)
}
#[cfg(test)]
mod tests {
use super::*;
/// The capture must actually see events and their field VALUES — if it
/// silently recorded nothing, every logging assertion built on it would
/// pass vacuously, which is worse than having no harness at all.
///
/// Mutation: an `enabled()` returning `false`, or an `event()` that drops
/// the visitor's fields, fails here.
#[test]
fn capture_records_target_level_and_fields() {
let ((), events) = capture(|| {
tracing::warn!(target: "freemkv::testlog", code = 6017u16, clip = ?"A.EVO", "E6017");
});
assert_eq!(events.len(), 1, "exactly one event: {events:?}");
assert_eq!(events[0].target, "freemkv::testlog");
assert_eq!(events[0].level, tracing::Level::WARN);
assert_eq!(events[0].field("code"), Some("6017"));
assert_eq!(events[0].field("clip"), Some("\"A.EVO\""));
assert_eq!(events[0].message(), "E6017");
}
/// A field that is absent must read as `None`, not as an empty string — an
/// assertion of the shape `field("code") == Some(..)` has to be able to
/// fail when the site stops logging the code at all.
#[test]
fn missing_field_is_none_and_capture_is_scoped() {
let ((), events) = capture(|| tracing::warn!(target: "freemkv::testlog", "no fields"));
assert_eq!(events[0].field("code"), None);
// Emitted outside `capture`, so it must not appear in a later capture.
tracing::warn!(target: "freemkv::testlog", code = 1u16, "outside");
let ((), later) = capture(|| {});
assert!(
later.is_empty(),
"capture is scoped to its closure: {later:?}"
);
}
}
+778 -133
View File
File diff suppressed because it is too large Load Diff
+34 -20
View File
@@ -20,6 +20,10 @@ fn css_descramble_sector_roundtrip_via_public_api() {
};
let mut sector = vec![0xAAu8; 2048];
// Real scrambled DVD sectors are MPEG-2 PS packs, and the public API now
// requires the pack start code as well as the flag bits — byte 0x14 means
// something else entirely in an IFO or UDF sector.
sector[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
sector[0x14] = 0x30;
sector[0x54..0x59].copy_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF, 0x42]);
let original = sector.clone();
@@ -37,30 +41,36 @@ fn css_descramble_sector_roundtrip_via_public_api() {
assert_ne!(&sector[128..256], &original[128..256]);
}
/// Test: css_is_scrambled detects scramble flags correctly.
/// Test: has_scramble_flag_bits detects the raw flag bits correctly.
#[test]
fn css_is_scrambled_detection() {
let mut sector = vec![0u8; 2048];
assert!(
!css::is_scrambled(&sector),
!css::has_scramble_flag_bits(&sector),
"empty sector should not be scrambled"
);
sector[0x14] = 0x10; // bit 4 set
assert!(css::is_scrambled(&sector), "bit 4 set should be detected");
assert!(
css::has_scramble_flag_bits(&sector),
"bit 4 set should be detected"
);
sector[0x14] = 0x20; // bit 5 set
assert!(css::is_scrambled(&sector), "bit 5 set should be detected");
assert!(
css::has_scramble_flag_bits(&sector),
"bit 5 set should be detected"
);
sector[0x14] = 0x30; // both bits set
assert!(
css::is_scrambled(&sector),
css::has_scramble_flag_bits(&sector),
"both bits set should be detected"
);
sector[0x14] = 0xCF; // bits 4-5 clear, other bits set
assert!(
!css::is_scrambled(&sector),
!css::has_scramble_flag_bits(&sector),
"bits 4-5 clear should not be scrambled"
);
}
@@ -75,7 +85,7 @@ fn css_is_scrambled_detection() {
#[test]
fn aacs_decrypt_unit_roundtrip() {
use aes::Aes128;
use aes::cipher::{BlockEncrypt, KeyInit, generic_array::GenericArray};
use aes::cipher::{Array, BlockCipherEncrypt, KeyInit};
let unit_key = [0xAAu8; 16];
let aacs_iv: [u8; 16] = [
@@ -103,8 +113,8 @@ fn aacs_decrypt_unit_roundtrip() {
let header: [u8; 16] = plain[..16].try_into().unwrap();
// Step 1: AES-ECB encrypt header with unit key
let cipher_header = Aes128::new(GenericArray::from_slice(&unit_key));
let mut block = GenericArray::clone_from_slice(&header);
let cipher_header = Aes128::new(&unit_key.into());
let mut block: Array<u8, _> = header.into();
cipher_header.encrypt_block(&mut block);
let mut derived = [0u8; 16];
derived.copy_from_slice(&block);
@@ -116,7 +126,7 @@ fn aacs_decrypt_unit_roundtrip() {
}
// Step 3: AES-CBC encrypt bytes 16..6144
let cipher = Aes128::new(GenericArray::from_slice(&encrypt_key));
let cipher = Aes128::new(&encrypt_key.into());
let mut prev = aacs_iv;
let num_blocks = (aacs::content::ALIGNED_UNIT_LEN - 16) / 16;
for i in 0..num_blocks {
@@ -124,7 +134,9 @@ fn aacs_decrypt_unit_roundtrip() {
for j in 0..16 {
plain[off + j] ^= prev[j];
}
let mut blk = GenericArray::clone_from_slice(&plain[off..off + 16]);
let mut c_blk = [0u8; 16];
c_blk.copy_from_slice(&plain[off..off + 16]);
let mut blk: Array<u8, _> = c_blk.into();
cipher.encrypt_block(&mut blk);
plain[off..off + 16].copy_from_slice(&blk);
prev.copy_from_slice(&plain[off..off + 16]);
@@ -205,7 +217,7 @@ fn aacs_disc_hash_deterministic() {
#[test]
fn aacs_decrypt_unit_key_roundtrip() {
use aes::Aes128;
use aes::cipher::{BlockEncrypt, KeyInit, generic_array::GenericArray};
use aes::cipher::{Array, BlockCipherEncrypt, KeyInit};
let vuk = [
0x11u8, 0x14, 0x36, 0x0B, 0x10, 0xEE, 0x6E, 0xAC, 0x78, 0xAA, 0x4A, 0xC0, 0xB7, 0x52, 0xEA,
@@ -217,8 +229,8 @@ fn aacs_decrypt_unit_key_roundtrip() {
];
// Encrypt: AES-ECB encrypt the unit key with VUK
let cipher = Aes128::new(GenericArray::from_slice(&vuk));
let mut block = GenericArray::clone_from_slice(&original_unit_key);
let cipher = Aes128::new(&vuk.into());
let mut block: Array<u8, _> = original_unit_key.into();
cipher.encrypt_block(&mut block);
let mut encrypted_uk = [0u8; 16];
encrypted_uk.copy_from_slice(&block);
@@ -330,9 +342,9 @@ fn aacs_clear_unit_reports_not_encrypted() {
/// Independent AES-128-ECB encrypt (uses `aes` crate directly, NOT our library).
fn ref_aes_ecb_encrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
use aes::Aes128;
use aes::cipher::{BlockEncrypt, KeyInit, generic_array::GenericArray};
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut block = GenericArray::clone_from_slice(data);
use aes::cipher::{Array, BlockCipherEncrypt, KeyInit};
let cipher = Aes128::new(&(*key).into());
let mut block: Array<u8, _> = (*data).into();
cipher.encrypt_block(&mut block);
let mut out = [0u8; 16];
out.copy_from_slice(&block);
@@ -342,8 +354,8 @@ fn ref_aes_ecb_encrypt(key: &[u8; 16], data: &[u8; 16]) -> [u8; 16] {
/// Independent AES-128-CBC encrypt (uses `aes` crate directly, NOT our library).
fn ref_aes_cbc_encrypt(key: &[u8; 16], iv: &[u8; 16], data: &mut [u8]) {
use aes::Aes128;
use aes::cipher::{BlockEncrypt, KeyInit, generic_array::GenericArray};
let cipher = Aes128::new(GenericArray::from_slice(key));
use aes::cipher::{Array, BlockCipherEncrypt, KeyInit};
let cipher = Aes128::new(&(*key).into());
let mut prev = *iv;
let num_blocks = data.len() / 16;
for i in 0..num_blocks {
@@ -351,7 +363,9 @@ fn ref_aes_cbc_encrypt(key: &[u8; 16], iv: &[u8; 16], data: &mut [u8]) {
for j in 0..16 {
data[off + j] ^= prev[j];
}
let mut block = GenericArray::clone_from_slice(&data[off..off + 16]);
let mut c_block = [0u8; 16];
c_block.copy_from_slice(&data[off..off + 16]);
let mut block: Array<u8, _> = c_block.into();
cipher.encrypt_block(&mut block);
data[off..off + 16].copy_from_slice(&block);
prev.copy_from_slice(&data[off..off + 16]);
+4
View File
@@ -42,6 +42,10 @@ fn css_decrypt_of_an_uncrackable_sector_still_descrambles() {
// unit key either opens a unit or does not, whereas a CSS title key is
// recovered from data whose recoverability varies sector by sector.
let mut sector = vec![0xFFu8; 2048];
// Scrambled DVD sectors are MPEG-2 PS packs. The descramble policy requires
// the pack start code as well as the flag bits, because byte 0x14 means
// something else entirely in an IFO, UDF or ISO 9660 sector.
sector[0x00..0x04].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]);
sector[0x14] |= 0x30; // CSS scramble flag, bits 4-5
let title_key: [u8; 5] = [0x42, 0x13, 0x37, 0xBE, 0xEF];